Setting the number of cores and heap size on a node

classic Classic list List threaded Threaded
8 messages Options
kohlisimranjit kohlisimranjit
Reply | Threaded
Open this post in threaded view
|

Setting the number of cores and heap size on a node

Hi,

When I start the ignite server I get the following:
 Topology snapshot [ver=1, server nodes=1, client nodes=0, CPUs=8, heap=3.6GB]
I want to know is there anyway I can set the number of cores in the configuration.xml file which we pass during the server startup.Also how to set minimum and max heap size via the configuration.xml file.

Thanks in advance.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Setting the number of cores and heap size on a node

Hi,

This log line simply informs you how many CPUs and heap are available to the grid after the topology changed. For example, if you have two boxes with 4 CPUs each and you start a node with 4GB heap on each box, it will show the totals of 8 CPUs and 8GB heap.

There is no way to specify number of cores used by a node. But you can control the load by changing the sizes of internal thread pools via IgniteConfiguration.setPublicThreadPoolSize and IgniteConfiguration.setSystemThreadPoolSize configuration properties. Can you clarify what you're trying to achieve here?

To specify the amount of heap memory use '-Xms' and '-Xmx' JVM properties. If you're starting a node using ignite.sh script, you can do it like this:

  ./ignite.sh -J-Xms8g -J-Xmx8g

-Val
kohlisimranjit kohlisimranjit
Reply | Threaded
Open this post in threaded view
|

Re: Setting the number of cores and heap size on a node

Hi Val,

So I want to use apache ignite for standalone caching. I don't want to allocate 8 CPUS perhaps something lower. Also I want specify the amount of useable heap via code or in the configuration.xml file. I wanted to know how to go about it.

Thanks
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Setting the number of cores and heap size on a node

Heap memory is managed by JMV, not by application, so there is no way to allocate it on per-node basis. But you can use off-heap memory instead - it's fully managed by Ignite and you're allowed to limit the amount of memory used for particular cache. Please refer to this documentation page for more information and configuration examples: https://apacheignite.readme.io/docs/off-heap-memory

As for CPUs, I still don't see why do you need this. These CPUs are not "allocated" by an Ignite node, they are just available to it. So if there is enough CPU time for both Ignite and your application which share one box, it will work. In other words, let operating system properly manage the resources.

Makes sense?

-Val
Aleksei Valikov Aleksei Valikov
Reply | Threaded
Open this post in threaded view
|

Re: Setting the number of cores and heap size on a node

Hi,


As for CPUs, I still don't see why do you need this. These CPUs are not
"allocated" by an Ignite node, they are just available to it. So if there is
enough CPU time for both Ignite and your application which share one box, it
will work. In other words, let operating system properly manage the
resources.

I think what counts is the number of jobs/tasks which can be executed by one node at a given time. I think this should be configurable within ignite.

Here's my use case. We want to run a distributed computation using a third-party tool which, when executed, takes an entire CPU core. We'll wrap that execution in Java in an Ignite task/job (sorry, I alwas forget which is which). On a multi-core machines we'd like to be able to run N-X tasks simultaneously where N is a number of cores and X is 1 or 2 (for 4-core or >8-core machines) reserved for administrative/management tasks. Like, 14 cores for a 16-core machine.

At the moment my understanding is that we'll need to start N-X nodes per server, to handle N-X jobs/tasks at once, correct?

Best wishes,
Alexey
alexey.goncharuk alexey.goncharuk
Reply | Threaded
Open this post in threaded view
|

Re: Setting the number of cores and heap size on a node


I think what counts is the number of jobs/tasks which can be executed by one node at a given time. I think this should be configurable within ignite.

Here's my use case. We want to run a distributed computation using a third-party tool which, when executed, takes an entire CPU core. We'll wrap that execution in Java in an Ignite task/job (sorry, I alwas forget which is which). On a multi-core machines we'd like to be able to run N-X tasks simultaneously where N is a number of cores and X is 1 or 2 (for 4-core or >8-core machines) reserved for administrative/management tasks. Like, 14 cores for a 16-core machine.

At the moment my understanding is that we'll need to start N-X nodes per server, to handle N-X jobs/tasks at once, correct?

You do not need to start multiple nodes in order to achieve multi-threaded job execution. By default Ignite will run tasks in parallel in the public thread pool of a fixed size in the order in which they arrive on a node. The default number of threads in this pool is 2*number of cores reported by Runtime#availableProcessors(). So if you do need any fancy job execution ordering or job stealing, it will be enough to set the number of threads in the public thread pool to N-X (see IgniteConfiguration#setPublicThreadPoolSize). 

If you want a more fine-grained control over how jobs are executed, you might want to use one of the CollisionSPIs shipped with Ignite, for example, PriorityQueueCollisionSpi or JobStealingCollisionSpi, or implement your own collision SPI. 
 
Hope that helps,
Alexey
Aleksei Valikov Aleksei Valikov
Reply | Threaded
Open this post in threaded view
|

Re: Setting the number of cores and heap size on a node

It does, thank you. I've just dropped a few person-days from our current effort estimation.

Am 07.08.2015 um 07:13 schrieb Alexey Goncharuk <[hidden email]>:


I think what counts is the number of jobs/tasks which can be executed by one node at a given time. I think this should be configurable within ignite.

Here's my use case. We want to run a distributed computation using a third-party tool which, when executed, takes an entire CPU core. We'll wrap that execution in Java in an Ignite task/job (sorry, I alwas forget which is which). On a multi-core machines we'd like to be able to run N-X tasks simultaneously where N is a number of cores and X is 1 or 2 (for 4-core or >8-core machines) reserved for administrative/management tasks. Like, 14 cores for a 16-core machine.

At the moment my understanding is that we'll need to start N-X nodes per server, to handle N-X jobs/tasks at once, correct?

You do not need to start multiple nodes in order to achieve multi-threaded job execution. By default Ignite will run tasks in parallel in the public thread pool of a fixed size in the order in which they arrive on a node. The default number of threads in this pool is 2*number of cores reported by Runtime#availableProcessors(). So if you do need any fancy job execution ordering or job stealing, it will be enough to set the number of threads in the public thread pool to N-X (see IgniteConfiguration#setPublicThreadPoolSize). 

If you want a more fine-grained control over how jobs are executed, you might want to use one of the CollisionSPIs shipped with Ignite, for example, PriorityQueueCollisionSpi or JobStealingCollisionSpi, or implement your own collision SPI. 
 
Hope that helps,
Alexey
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Setting the number of cores and heap size on a node

I also want to add that by using a CollisionSpi you can limit the number of active jobs and waiting jobs without changing thread pool parameters. More information here:

https://apacheignite.readme.io/docs/job-scheduling