Nodes distribution/nodes in parallel on one host using Yarn

classic Classic list List threaded Threaded
3 messages Options
rpinzon rpinzon
Reply | Threaded
Open this post in threaded view
|

Nodes distribution/nodes in parallel on one host using Yarn

Hi there,

I've been using Apache Ignite 1.8 in standalone mode for a while and as I'm
upgrading to 2.5.0 I'm trying to use YARN deployment
(https://apacheignite.readme.io/docs/yarn-deployment).

I deployed Ignite and it is running fine on YARN using ZooKeeper discovery
but when I increase the number of nodes (IGNITE_NODE_COUNT) seemingly they
start in the same host as the first node started until the host is out of
resources to start another one or when it reaches a total of 5 nodes started
in sequence.

As I didn't found any documentation about node distribution or number of
nodes in parallel on one host I'm concluding this based on some experiments
that I've made.

I found a parameter (/-m/) used with /start/ command
(https://apacheignite-tools.readme.io/v2.5/docs/start-command) in
ignitevisor that apparently do the trick and reinforces what I concluded
about reaching 5 nodes (it's the default value for /-m/).

I'd like to know if there is any option to change this policy when starting
the cluster or if the only option is starting manually through the visor.

Thanks in advance!





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Nodes distribution/nodes in parallel on one host using Yarn

Hello!

I'm afraid you will have to debug it and fill and issue since there's not much expertise in YARN deployments here.

One thing I see in the source that additional nodes might be brought up on the same host if it has extra resources (as determined by YARN resource management)

I suggest increasing resources used by instance so that one server can only hose one instance.

Note that you have to put in IGNITE_MEMORY_OVERHEAD_PER_NODE memory used by all Ignite off-heap memory regions, i.e., all data stored in caches, as well as some padding.

Regards,

--
Ilya Kasnacheev

2018-07-13 2:13 GMT+03:00 rpinzon <[hidden email]>:
Hi there,

I've been using Apache Ignite 1.8 in standalone mode for a while and as I'm
upgrading to 2.5.0 I'm trying to use YARN deployment
(https://apacheignite.readme.io/docs/yarn-deployment).

I deployed Ignite and it is running fine on YARN using ZooKeeper discovery
but when I increase the number of nodes (IGNITE_NODE_COUNT) seemingly they
start in the same host as the first node started until the host is out of
resources to start another one or when it reaches a total of 5 nodes started
in sequence.

As I didn't found any documentation about node distribution or number of
nodes in parallel on one host I'm concluding this based on some experiments
that I've made.

I found a parameter (/-m/) used with /start/ command
(https://apacheignite-tools.readme.io/v2.5/docs/start-command) in
ignitevisor that apparently do the trick and reinforces what I concluded
about reaching 5 nodes (it's the default value for /-m/).

I'd like to know if there is any option to change this policy when starting
the cluster or if the only option is starting manually through the visor.

Thanks in advance!





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

rpinzon rpinzon
Reply | Threaded
Open this post in threaded view
|

Re: Nodes distribution/nodes in parallel on one host using Yarn

Hi Ilya,

First of all thanks for your response!

I thought about increasing the resources to prevent more than one node on a host but I believe that in this case it will be better to keep the cluster running in standalone mode. Firstly because I won't make use of affinity between Ignite and Spark cause I won't have resources to run Spark executors on those hosts. Secondly because I'll decrease the capacity of resources for data processing with Spark, so I believe it's better to give resources that are not part of my Hadoop/Spark cluster.

Another solution that I thought about is using YARN's resource pool but it's somewhat limited in relation to resource management per host.

I'll try to find which property is used during nodes startup to define the maximum number of parallel nodes per host and fill an issue.

Regards,

Renan T. Pinzon


On Fri, Jul 13, 2018 at 6:02 AM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

I'm afraid you will have to debug it and fill and issue since there's not much expertise in YARN deployments here.

One thing I see in the source that additional nodes might be brought up on the same host if it has extra resources (as determined by YARN resource management)

I suggest increasing resources used by instance so that one server can only hose one instance.

Note that you have to put in IGNITE_MEMORY_OVERHEAD_PER_NODE memory used by all Ignite off-heap memory regions, i.e., all data stored in caches, as well as some padding.

Regards,

--
Ilya Kasnacheev

2018-07-13 2:13 GMT+03:00 rpinzon <[hidden email]>:
Hi there,

I've been using Apache Ignite 1.8 in standalone mode for a while and as I'm
upgrading to 2.5.0 I'm trying to use YARN deployment
(https://apacheignite.readme.io/docs/yarn-deployment).

I deployed Ignite and it is running fine on YARN using ZooKeeper discovery
but when I increase the number of nodes (IGNITE_NODE_COUNT) seemingly they
start in the same host as the first node started until the host is out of
resources to start another one or when it reaches a total of 5 nodes started
in sequence.

As I didn't found any documentation about node distribution or number of
nodes in parallel on one host I'm concluding this based on some experiments
that I've made.

I found a parameter (/-m/) used with /start/ command
(https://apacheignite-tools.readme.io/v2.5/docs/start-command) in
ignitevisor that apparently do the trick and reinforces what I concluded
about reaching 5 nodes (it's the default value for /-m/).

I'd like to know if there is any option to change this policy when starting
the cluster or if the only option is starting manually through the visor.

Thanks in advance!





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/