Cache group setting not taking effect

classic Classic list List threaded Threaded
4 messages Options
joaogoncalves1 joaogoncalves1
Reply | Threaded
Open this post in threaded view
|

Cache group setting not taking effect

I am trying to take an Ignite cluster to production but so far we have hit a
lot of performance issues and are in the process of tuning configurations.

Use case: Our application uses the JdbcThinClient to create temporary tables
that hold partial results. We then join all those tables into one 'final
table' and delete all temporary tables. All tables are kept off-heap, in a
dataRegion.

When under load, Ignite starts being really slow to execute simple INSERT
and DROP TABLE statements, sometimes taking <10 seconds. We never see any
memory issues, which makes sense because most tables have a very small
number of rows. Cpu utilization also seems to not be a problem.
We do see that Ignite takes a lot of time to exchange partition information
between nodes, sometimes >4 seconds.

My setting is as follows:

Cluster: 4 nodes with 4 cores and 16GB RAM each
Version:  Ignite 2.8.0
Heap: -Xms6g -Xmx6g
Persistence is disabled.

Config.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd">

  <bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">

    <property name="discoverySpi">
      <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
        <property name="ipFinder">
          <bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
            <property name="addresses">
              <list>
               
                <value>10.35.2.21</value>
                <value>10.35.2.22</value>
                <value>10.35.2.23</value>
                <value>10.35.2.24</value>
              </list>
            </property>
          </bean>
        </property>
      </bean>
    </property>

   
    <property name="dataStorageConfiguration">
      <bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
        <property name="defaultDataRegionConfiguration">
          <bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
            <property name="name" value="Default_Cache_Region"/>
            <property name="initialSize" value="#{100L * 1024 * 1024}"/>
            <property name="maxSize" value="#{6L * 1024 * 1024 * 1024}"/>
            <property name="persistenceEnabled" value="false"/>
          </bean>
        </property>
      </bean>
    </property>

    <property name="cacheConfiguration">
      <list>
       
        <bean id="cache-template-bean" abstract="true"
class="org.apache.ignite.configuration.CacheConfiguration">
          <property name="name" value="table_cache_template*"/>
          <property name="sqlSchema" value="PUBLIC"/>
          <property name="dataRegionName" value="Default_Cache_Region"/>
          <property name="groupName" value="default_group"/>
          <property name="eventsDisabled" value="true"/>
          <property name="backups" value="0"/>
          <property name="expiryPolicyFactory">
            <bean class="javax.cache.expiry.ModifiedExpiryPolicy"
factory-method="factoryOf">
              <constructor-arg>
                <bean class="javax.cache.expiry.Duration">
                  <constructor-arg value="MINUTES"/>
                  <constructor-arg value="6"/>
                </bean>
              </constructor-arg>
            </bean>
          </property>
        </bean>

       
        <bean id="cache-template-bean2" abstract="true"
class="org.apache.ignite.configuration.CacheConfiguration">
          <property name="name" value="debug*"/>
          <property name="sqlSchema" value="PUBLIC"/>
          <property name="dataRegionName" value="Default_Cache_Region"/>
          <property name="groupName" value="debug_group"/>
          <property name="eventsDisabled" value="true"/>
          <property name="backups" value="0"/>
        </bean>

      </list>
    </property>
  </bean>
</beans>

We tried to set 'groupName' to put all tables under the same cacheGroup, but
this setting appears to not be taking effect, because when I inspect the
caches with the *control.sh* script is shows that all caches (except for
system caches) have grpName=null and all have distinct grId, which matches
their cacheId. Does this setting not work with cache templates? I've also
tried setting it in the CREATE TABLE statement, but still no effect.

Would appreciate some insight into how to best configure ignite for this use
case.

Thanks in advance,
João Gonçalves




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Cache group setting not taking effect

Hello!

Are you sure that you have recreated your tables after setting group name in their configuration? Do you have a reproducer code where you specify grpName on a fresh cluster but it is not applied?

PME on cache create/destroy operation is blocking: it has to wait for all operations to stop, only then may new operations start.
This would explain why you see multi-second delays.

Regards,
--
Ilya Kasnacheev


ср, 10 мар. 2021 г. в 00:13, joaogoncalves1 <[hidden email]>:
I am trying to take an Ignite cluster to production but so far we have hit a
lot of performance issues and are in the process of tuning configurations.

Use case: Our application uses the JdbcThinClient to create temporary tables
that hold partial results. We then join all those tables into one 'final
table' and delete all temporary tables. All tables are kept off-heap, in a
dataRegion.

When under load, Ignite starts being really slow to execute simple INSERT
and DROP TABLE statements, sometimes taking <10 seconds. We never see any
memory issues, which makes sense because most tables have a very small
number of rows. Cpu utilization also seems to not be a problem.
We do see that Ignite takes a lot of time to exchange partition information
between nodes, sometimes >4 seconds.

My setting is as follows:

Cluster: 4 nodes with 4 cores and 16GB RAM each
Version:  Ignite 2.8.0
Heap: -Xms6g -Xmx6g
Persistence is disabled.

Config.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd">

  <bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">

    <property name="discoverySpi">
      <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
        <property name="ipFinder">
          <bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
            <property name="addresses">
              <list>

                <value>10.35.2.21</value>
                <value>10.35.2.22</value>
                <value>10.35.2.23</value>
                <value>10.35.2.24</value>
              </list>
            </property>
          </bean>
        </property>
      </bean>
    </property>


    <property name="dataStorageConfiguration">
      <bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
        <property name="defaultDataRegionConfiguration">
          <bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
            <property name="name" value="Default_Cache_Region"/>
            <property name="initialSize" value="#{100L * 1024 * 1024}"/>
            <property name="maxSize" value="#{6L * 1024 * 1024 * 1024}"/>
            <property name="persistenceEnabled" value="false"/>
          </bean>
        </property>
      </bean>
    </property>

    <property name="cacheConfiguration">
      <list>

        <bean id="cache-template-bean" abstract="true"
class="org.apache.ignite.configuration.CacheConfiguration">
          <property name="name" value="table_cache_template*"/>
          <property name="sqlSchema" value="PUBLIC"/>
          <property name="dataRegionName" value="Default_Cache_Region"/>
          <property name="groupName" value="default_group"/>
          <property name="eventsDisabled" value="true"/>
          <property name="backups" value="0"/>
          <property name="expiryPolicyFactory">
            <bean class="javax.cache.expiry.ModifiedExpiryPolicy"
factory-method="factoryOf">
              <constructor-arg>
                <bean class="javax.cache.expiry.Duration">
                  <constructor-arg value="MINUTES"/>
                  <constructor-arg value="6"/>
                </bean>
              </constructor-arg>
            </bean>
          </property>
        </bean>


        <bean id="cache-template-bean2" abstract="true"
class="org.apache.ignite.configuration.CacheConfiguration">
          <property name="name" value="debug*"/>
          <property name="sqlSchema" value="PUBLIC"/>
          <property name="dataRegionName" value="Default_Cache_Region"/>
          <property name="groupName" value="debug_group"/>
          <property name="eventsDisabled" value="true"/>
          <property name="backups" value="0"/>
        </bean>

      </list>
    </property>
  </bean>
</beans>

We tried to set 'groupName' to put all tables under the same cacheGroup, but
this setting appears to not be taking effect, because when I inspect the
caches with the *control.sh* script is shows that all caches (except for
system caches) have grpName=null and all have distinct grId, which matches
their cacheId. Does this setting not work with cache templates? I've also
tried setting it in the CREATE TABLE statement, but still no effect.

Would appreciate some insight into how to best configure ignite for this use
case.

Thanks in advance,
João Gonçalves




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
joaogoncalves1 joaogoncalves1
Reply | Threaded
Open this post in threaded view
|

Re: Cache group setting not taking effect

Hi Ilya,

Sorry for the delay, we have been heavy testing our cluster since we were
able to set the cache groups.
A difference between development and production configs was making me
believe that cache groups setting didn't work, but it does. We have noticed
some improvement in performance, but it's still not what we had hoped for.

Meanwhile, we found that caches have 1024 partitions by default. We are
considering setting the following in our config:

<property name="affinity">
            <bean
class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
              <property name="excludeNeighbors" value="true"/>
              <property name="partitions" value="64"/>
            </bean>
</property>

Could this help make PME's faster?
Are there other settings we can tune to make cache creates/destroys not slow
down the cluster as much, or overall improve performance?
As I said before, all our data is temporary, we use Ignite to store some
results for 5 minutes and then delete them. Losing this data is not a
problem, so if there are other safety measures we can turn off that will
help us get as fast as Ignite can go, please point them out.

Best regards,
João Gonçalves



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Cache group setting not taking effect

Hello!

You may choose to avoid destroying cache, instead just try to clean & reuse them.

Number of partitions will likely not affect PME length because it is slow due to its blocking nature, which forces all operations on the cluster to finish before any new ops may be started.

Regards,
--
Ilya Kasnacheev


пт, 12 мар. 2021 г. в 00:19, joaogoncalves1 <[hidden email]>:
Hi Ilya,

Sorry for the delay, we have been heavy testing our cluster since we were
able to set the cache groups.
A difference between development and production configs was making me
believe that cache groups setting didn't work, but it does. We have noticed
some improvement in performance, but it's still not what we had hoped for.

Meanwhile, we found that caches have 1024 partitions by default. We are
considering setting the following in our config:

<property name="affinity">
            <bean
class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
              <property name="excludeNeighbors" value="true"/>
              <property name="partitions" value="64"/>
            </bean>
</property>

Could this help make PME's faster?
Are there other settings we can tune to make cache creates/destroys not slow
down the cluster as much, or overall improve performance?
As I said before, all our data is temporary, we use Ignite to store some
results for 5 minutes and then delete them. Losing this data is not a
problem, so if there are other safety measures we can turn off that will
help us get as fast as Ignite can go, please point them out.

Best regards,
João Gonçalves



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/