Ignite thread pool configuration

classic Classic list List threaded Threaded
7 messages Options
Dominik Przybysz Dominik Przybysz
Reply | Threaded
Open this post in threaded view
|

Ignite thread pool configuration

Hi,
I read the thread pool documentation (https://apacheignite.readme.io/docs/thread-pools) but  I need more information which thread pools I should configure/tune for my setup.

I have 2 server nodes (partitioned cache) and some clients (without near or local chaches) which add cache entries via data streamer and get data via getAsync operation.

As far as I understand the documentation:
- on server nodes I should configure especially publicThreadPoolSize on which put and get operation occur? 
- on client nodes I should configure dataStreamerThreadPoolSize (for data streamer) and publicThreadPoolSize (on which put and get occur)?

--
Pozdrawiam / Regards,
Dominik Przybysz
Mikhail Mikhail
Reply | Threaded
Open this post in threaded view
|

Re: Ignite thread pool configuration

Hi Dominik,

You don't need to configure thread pools manually, you should do this only
if you have some particular problem and think that this can be resolved by
changing pool sizes.
Ignite in 95% of cases automatically configure correct pools' sizes.

So you didn't understand it correctly:

The public pool is used the only for compute tasks:
https://apacheignite.readme.io/docs/compute-grid
all cache operations like get and put will be executed in system pool.
https://apacheignite.readme.io/docs/thread-pools#section-system-pool

StreamerThreadPoolSize - this pool process all incoming messages from Data
Streamers, but again, usually you shouldn't set it manually.

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: Ignite thread pool configuration

I noticed that the DataStreamer thread pool size seems to be around 32 in my
local machine. In our cluster setup we dont stream data from any external db
. We use native persistence. Do you think it makes sense to reduce this pool
size for both server and client configuraiton. I am not sure if this pool
will ever be used in our project.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Ignite thread pool configuration

You can certainly reduce its size (https://ignite.apache.org/docs/latest/perf-and-troubleshooting/thread-pools-tuning#data-streamer-pool). But at the same time created but unused threads are just some objects that consume a bit of memory but don't use any CPU resources (until you start using those).

-
Denis


On Sat, Oct 3, 2020 at 6:12 AM VeenaMithare <[hidden email]> wrote:
I noticed that the DataStreamer thread pool size seems to be around 32 in my
local machine. In our cluster setup we dont stream data from any external db
. We use native persistence. Do you think it makes sense to reduce this pool
size for both server and client configuraiton. I am not sure if this pool
will ever be used in our project.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: Ignite thread pool configuration

Thanks Denis,

I am thinking of setting the below thread pools as this on both client and
server since we dont use data streamer, IGFS or Peer Class loading:

        <property name="dataStreamerThreadPoolSize" value="4"/>
        <property name="igfsThreadPoolSize" value="4"/>
        <property name="peerClassLoadingThreadPoolSize" value="4"/>

Also our thick clients dont connect using REST . So thinking of adding the
below configuration on our thick client configuration.

        <property name="connectorConfiguration">
            <bean
class="org.apache.ignite.configuration.ConnectorConfiguration">
                <property name="threadPoolSize" value="4"/>
            </bean>
        </property>

Hope that is okay ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Ignite thread pool configuration

Give it a try but do some load testing close to your production workload. And then ramp the numbers up if needed.

-
Denis


On Mon, Oct 5, 2020 at 12:56 AM VeenaMithare <[hidden email]> wrote:
Thanks Denis,

I am thinking of setting the below thread pools as this on both client and
server since we dont use data streamer, IGFS or Peer Class loading:

        <property name="dataStreamerThreadPoolSize" value="4"/>
        <property name="igfsThreadPoolSize" value="4"/>
        <property name="peerClassLoadingThreadPoolSize" value="4"/>

Also our thick clients dont connect using REST . So thinking of adding the
below configuration on our thick client configuration.

        <property name="connectorConfiguration">
            <bean
class="org.apache.ignite.configuration.ConnectorConfiguration">
                <property name="threadPoolSize" value="4"/>
            </bean>
        </property>

Hope that is okay ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: Ignite thread pool configuration

Thanks Denis,

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/