maxSize - per node or per cluster? - only 2k/sec in 11 node dedicated cluster

classic Classic list List threaded Threaded
4 messages Options
devinbost devinbost
Reply | Threaded
Open this post in threaded view
|

maxSize - per node or per cluster? - only 2k/sec in 11 node dedicated cluster

We're trying to figure out how to get more throughput from our Ignite cluster. 
We have 11 dedicated Ignite VM nodes, each with 32 GB of RAM. Yet, we're only writing at 2k/sec max, even when we parallelize the writes to Ignite. 
We're using native persistence, but it just seems way slower than expected.

We have our cache maxSize set to 16 GB. Since the cluster has a total of 352 GB of RAM, should our maxSize be set per node, or per cluster? If it's per cluster, then I'm thinking we'd want to increase maxSize to at least 300 GB. Looking for feedback on this.

We've already turned off swapping on the boxes. 

What else can we tune to increase throughput? It seems like we should be getting 100x the throughput at least, so I'm hoping something is just misconfigured. 

Devin G. Bost
Pavel Vinokurov Pavel Vinokurov
Reply | Threaded
Open this post in threaded view
|

Re: maxSize - per node or per cluster? - only 2k/sec in 11 node dedicated cluster

Hi Devin,

MaxSize is set per node for the specified data region.
A few clarifying questions:
What cache configuration are you using? The performance could depend on the type of a cache and the number of backups.
How many clients and threads are writing to the cluster? Because it is a possible bottleneck on the client side.
Have you tried  IgniteDataStreamer[2]? It's recommended approach to load the massive amount of data.

You could refer to persistence tuning documentation[1].


пт, 13 нояб. 2020 г. в 07:26, Devin Bost <[hidden email]>:
We're trying to figure out how to get more throughput from our Ignite cluster. 
We have 11 dedicated Ignite VM nodes, each with 32 GB of RAM. Yet, we're only writing at 2k/sec max, even when we parallelize the writes to Ignite. 
We're using native persistence, but it just seems way slower than expected.

We have our cache maxSize set to 16 GB. Since the cluster has a total of 352 GB of RAM, should our maxSize be set per node, or per cluster? If it's per cluster, then I'm thinking we'd want to increase maxSize to at least 300 GB. Looking for feedback on this.

We've already turned off swapping on the boxes. 

What else can we tune to increase throughput? It seems like we should be getting 100x the throughput at least, so I'm hoping something is just misconfigured. 

Devin G. Bost


--

Regards

Pavel Vinokurov

devinbost devinbost
Reply | Threaded
Open this post in threaded view
|

Re: maxSize - per node or per cluster? - only 2k/sec in 11 node dedicated cluster

Hi Pavel,

Thanks for your reply.

Here's one of our configs:


<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="bot"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="backups" value="2"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="statisticsEnabled" value="true"/>
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="session"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="backups" value="2"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="statisticsEnabled" value="true"/>
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="placements"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="2"/>
<property name="eagerTtl" value="true"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="statisticsEnabled" value="true"/>
<property name="expiryPolicyFactory">
<bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
<constructor-arg>
<bean class="javax.cache.expiry.Duration">
<constructor-arg value="HOURS"/>
<constructor-arg value="48"/>
</bean>
</constructor-arg>
</bean>
</property>
</bean>
</list>
</property>

<property name="authenticationEnabled" value="true"/>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
<property name="maxSize" value="#{16L * 1024 * 1024 * 1024}"/>
</bean>
</property>
</bean>
</property>

<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses" value="[list of hosts]"/>
</bean>
</property>
</bean>
</property>

<property name="connectorConfiguration">
<bean class="org.apache.ignite.configuration.ConnectorConfiguration">
<property name="host" value="0.0.0.0"/>
<property name="port" value="[port]"/>
<property name="idleQueryCursorTimeout" value="1000"/>
</bean>
</property>

<property name="clientConnectorConfiguration">
<bean class="org.apache.ignite.configuration.ClientConnectorConfiguration">
<property name="maxOpenCursorsPerConnection" value="1200"/>
</bean>
</property>

<property name="includeEventTypes">
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_EXPIRED"/>
</property>

</bean>
</beans>

There are about 20 clients, and I'm not seeing any threading in the client code; however, what's strange is that we're seeing a very large number of connections (thousands), and we can't figure out what's causing the large number of connections. It does seem that performance drops even further when the connections increases over 2,000, but even when connections are in the hundreds, throughput is still below 3,000 / sec at best. 
One of our dev's said that IgniteDataStreamer didn't support our use case, I think because we're doing a lot of updates.

Devin G. Bost


On Thu, Nov 12, 2020 at 10:04 PM Pavel Vinokurov <[hidden email]> wrote:
Hi Devin,

MaxSize is set per node for the specified data region.
A few clarifying questions:
What cache configuration are you using? The performance could depend on the type of a cache and the number of backups.
How many clients and threads are writing to the cluster? Because it is a possible bottleneck on the client side.
Have you tried  IgniteDataStreamer[2]? It's recommended approach to load the massive amount of data.

You could refer to persistence tuning documentation[1].


пт, 13 нояб. 2020 г. в 07:26, Devin Bost <[hidden email]>:
We're trying to figure out how to get more throughput from our Ignite cluster. 
We have 11 dedicated Ignite VM nodes, each with 32 GB of RAM. Yet, we're only writing at 2k/sec max, even when we parallelize the writes to Ignite. 
We're using native persistence, but it just seems way slower than expected.

We have our cache maxSize set to 16 GB. Since the cluster has a total of 352 GB of RAM, should our maxSize be set per node, or per cluster? If it's per cluster, then I'm thinking we'd want to increase maxSize to at least 300 GB. Looking for feedback on this.

We've already turned off swapping on the boxes. 

What else can we tune to increase throughput? It seems like we should be getting 100x the throughput at least, so I'm hoping something is just misconfigured. 

Devin G. Bost


--

Regards

Pavel Vinokurov

Pavel Vinokurov Pavel Vinokurov
Reply | Threaded
Open this post in threaded view
|

Re: maxSize - per node or per cluster? - only 2k/sec in 11 node dedicated cluster

Devin,

Could you share the code block  making puts.
Do you use thick or thin clients?

Thanks,
Pavel

сб, 14 нояб. 2020 г. в 06:29, Devin Bost <[hidden email]>:
Hi Pavel,

Thanks for your reply.

Here's one of our configs:


<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="bot"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="backups" value="2"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="statisticsEnabled" value="true"/>
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="session"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="backups" value="2"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="statisticsEnabled" value="true"/>
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="placements"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="2"/>
<property name="eagerTtl" value="true"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="statisticsEnabled" value="true"/>
<property name="expiryPolicyFactory">
<bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
<constructor-arg>
<bean class="javax.cache.expiry.Duration">
<constructor-arg value="HOURS"/>
<constructor-arg value="48"/>
</bean>
</constructor-arg>
</bean>
</property>
</bean>
</list>
</property>

<property name="authenticationEnabled" value="true"/>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
<property name="maxSize" value="#{16L * 1024 * 1024 * 1024}"/>
</bean>
</property>
</bean>
</property>

<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses" value="[list of hosts]"/>
</bean>
</property>
</bean>
</property>

<property name="connectorConfiguration">
<bean class="org.apache.ignite.configuration.ConnectorConfiguration">
<property name="host" value="0.0.0.0"/>
<property name="port" value="[port]"/>
<property name="idleQueryCursorTimeout" value="1000"/>
</bean>
</property>

<property name="clientConnectorConfiguration">
<bean class="org.apache.ignite.configuration.ClientConnectorConfiguration">
<property name="maxOpenCursorsPerConnection" value="1200"/>
</bean>
</property>

<property name="includeEventTypes">
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_EXPIRED"/>
</property>

</bean>
</beans>

There are about 20 clients, and I'm not seeing any threading in the client code; however, what's strange is that we're seeing a very large number of connections (thousands), and we can't figure out what's causing the large number of connections. It does seem that performance drops even further when the connections increases over 2,000, but even when connections are in the hundreds, throughput is still below 3,000 / sec at best. 
One of our dev's said that IgniteDataStreamer didn't support our use case, I think because we're doing a lot of updates.

Devin G. Bost


On Thu, Nov 12, 2020 at 10:04 PM Pavel Vinokurov <[hidden email]> wrote:
Hi Devin,

MaxSize is set per node for the specified data region.
A few clarifying questions:
What cache configuration are you using? The performance could depend on the type of a cache and the number of backups.
How many clients and threads are writing to the cluster? Because it is a possible bottleneck on the client side.
Have you tried  IgniteDataStreamer[2]? It's recommended approach to load the massive amount of data.

You could refer to persistence tuning documentation[1].


пт, 13 нояб. 2020 г. в 07:26, Devin Bost <[hidden email]>:
We're trying to figure out how to get more throughput from our Ignite cluster. 
We have 11 dedicated Ignite VM nodes, each with 32 GB of RAM. Yet, we're only writing at 2k/sec max, even when we parallelize the writes to Ignite. 
We're using native persistence, but it just seems way slower than expected.

We have our cache maxSize set to 16 GB. Since the cluster has a total of 352 GB of RAM, should our maxSize be set per node, or per cluster? If it's per cluster, then I'm thinking we'd want to increase maxSize to at least 300 GB. Looking for feedback on this.

We've already turned off swapping on the boxes. 

What else can we tune to increase throughput? It seems like we should be getting 100x the throughput at least, so I'm hoping something is just misconfigured. 

Devin G. Bost


--

Regards

Pavel Vinokurov



--

Regards

Pavel Vinokurov