Cluster in AWS can not have more than 100 nodes?

classic Classic list List threaded Threaded
8 messages Options
codeboyyong codeboyyong
Reply | Threaded
Open this post in threaded view
|

Cluster in AWS can not have more than 100 nodes?

Hi Friends,
I have a big ignite cluster running in private was like cloud, I use
TcpDiscoverySpi with TcpDiscoveryS3IpFinder and run with 96 nodes, it works
fine.
Today I redeploy it with 128 nodes and found in the S3 buckets, there are
always show 100 nodes. Is there any config or hard limit node numbers on
TcpDiscoverySpi ?
Can someone pls let us know how to expand more than 100 nodes ?


Thank You
Very much!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Jörn Franke Jörn Franke
Reply | Threaded
Open this post in threaded view
|

Re: Cluster in AWS can not have more than 100 nodes?

Maybe you have an instance limit of 100 on aws side ?

> Am 31.10.2019 um 19:09 schrieb codeboyyong <[hidden email]>:
>
> Hi Friends,
> I have a big ignite cluster running in private was like cloud, I use
> TcpDiscoverySpi with TcpDiscoveryS3IpFinder and run with 96 nodes, it works
> fine.
> Today I redeploy it with 128 nodes and found in the S3 buckets, there are
> always show 100 nodes. Is there any config or hard limit node numbers on
> TcpDiscoverySpi ?
> Can someone pls let us know how to expand more than 100 nodes ?
>
>
> Thank You
> Very much!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
codeboyyong codeboyyong
Reply | Threaded
Open this post in threaded view
|

Re: Cluster in AWS can not have more than 100 nodes?

No . The thing is we started ignite in embedded mode as web app, we can see
there are 128 instances.
But when we check he S3 bucket , it always only have 100 its, which means
only 100 nodes joined the ignite cluster.  Not sure if there are another way
to get the cluster size.
Since we are running as web app, we actually dont have all the access of
command line tools.

Thank You



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Cluster in AWS can not have more than 100 nodes?

There is no hard limit of this kind, here is the source code of the IP finder: 

You can enable DEBUG level for Discovery and the IP Finder to collect more details. 
-
Denis


On Thu, Oct 31, 2019 at 4:41 PM codeboyyong <[hidden email]> wrote:
No . The thing is we started ignite in embedded mode as web app, we can see
there are 128 instances.
But when we check he S3 bucket , it always only have 100 its, which means
only 100 nodes joined the ignite cluster.  Not sure if there are another way
to get the cluster size.
Since we are running as web app, we actually dont have all the access of
command line tools.

Thank You



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

Re: Cluster in AWS can not have more than 100 nodes?

In reply to this post by codeboyyong
Each node is supposed to add its own IP and port to the S3 bucket when it starts. That said, I wouldn't check the cluster state based on the contents of the bucket alone.
Check your logs for errors. Try using some tools (e.g. check out Web Console - either the one in Ignite https://apacheignite-tools.readme.io/docs/getting-started or from GridGain https://www.gridgain.com/products/software/enterprise-edition/management-tool) to see whether all nodes are there.

If you don't see any clues, please share your logs from any of the running servers. Ideally, a log from one of the 100 servers which joined the cluster, and a log from one of the 28 that didn't.

Stan

On Fri, Nov 1, 2019 at 2:41 AM codeboyyong <[hidden email]> wrote:
No . The thing is we started ignite in embedded mode as web app, we can see
there are 128 instances.
But when we check he S3 bucket , it always only have 100 its, which means
only 100 nodes joined the ignite cluster.  Not sure if there are another way
to get the cluster size.
Since we are running as web app, we actually dont have all the access of
command line tools.

Thank You



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Cluster in AWS can not have more than 100 nodes?

Hello!

List of hosts present in IP finder has very little bearing on list of nodes in cluster.

Server node's host do not have to be present in IP finder in order for this node to function. Cluster will be operational until there exists at least one node which is a) alive, b) reachable, and c) listed in IP finder. In your case there are 100 of such nodes.

Please share the latest "Topology snapshot" messages from your nodes.

Regards,
--
Ilya Kasnacheev


вт, 5 нояб. 2019 г. в 18:51, Stanislav Lukyanov <[hidden email]>:
Each node is supposed to add its own IP and port to the S3 bucket when it starts. That said, I wouldn't check the cluster state based on the contents of the bucket alone.
Check your logs for errors. Try using some tools (e.g. check out Web Console - either the one in Ignite https://apacheignite-tools.readme.io/docs/getting-started or from GridGain https://www.gridgain.com/products/software/enterprise-edition/management-tool) to see whether all nodes are there.

If you don't see any clues, please share your logs from any of the running servers. Ideally, a log from one of the 100 servers which joined the cluster, and a log from one of the 28 that didn't.

Stan

On Fri, Nov 1, 2019 at 2:41 AM codeboyyong <[hidden email]> wrote:
No . The thing is we started ignite in embedded mode as web app, we can see
there are 128 instances.
But when we check he S3 bucket , it always only have 100 its, which means
only 100 nodes joined the ignite cluster.  Not sure if there are another way
to get the cluster size.
Since we are running as web app, we actually dont have all the access of
command line tools.

Thank You



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
codeboyyong codeboyyong
Reply | Threaded
Open this post in threaded view
|

Re: Cluster in AWS can not have more than 100 nodes?

Hi Ilya
yes from the message I can see there  are more than 100 server nodes :

Topology snapshot [ver=2104, locNode=63120c8d, servers=117, clients=129,
state=ACTIVE, CPUs=9240, offheap=2300.0GB, heap=7300.0GB]

We are running a spark job keep writing to the database every minute.
At least from the long I can see there are 117 server nodes, comparing with
the 100 ip in S3 bucket, it is better but still no sure why it is not 128
nodes we started .

Thank You
Yong



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Cluster in AWS can not have more than 100 nodes?

Hello!

Please check logs from all nodes to determine if there are any nodes which did not join, and what is their reasoning for this.

Regards,
--
Ilya Kasnacheev


сб, 9 нояб. 2019 г. в 02:57, codeboyyong <[hidden email]>:
Hi Ilya
yes from the message I can see there  are more than 100 server nodes :

Topology snapshot [ver=2104, locNode=63120c8d, servers=117, clients=129,
state=ACTIVE, CPUs=9240, offheap=2300.0GB, heap=7300.0GB]

We are running a spark job keep writing to the database every minute.
At least from the long I can see there are 117 server nodes, comparing with
the 100 ip in S3 bucket, it is better but still no sure why it is not 128
nodes we started .

Thank You
Yong



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/