Data load is very slow in ignite 2.3 compare to ignite 1.9

classic Classic list List threaded Threaded
12 messages Options
Tejashwa Kumar Verma Tejashwa Kumar Verma
Reply | Threaded
Open this post in threaded view
|

Data load is very slow in ignite 2.3 compare to ignite 1.9

Hi All,

We are migrating from 1.9 to 2.3 and following are the respective configs .

Cache Config in 1.9:

<bean class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="INSTALLBASE_CACHE" /> <property name="cacheMode" value="PARTITIONED" /> <property name="atomicityMode" value="TRANSACTIONAL" /> <property name="backups" value="0" /> <property name="statisticsEnabled" value="true" /> <!--property name="swapEnabled" value="true"/ --> <property name="copyOnRead" value="false" /> <property name="memoryMode" value="OFFHEAP_TIERED"/> <property name="offHeapMaxMemory" value="#{5 * 1024L * 1024L * 1024L}"/ > <property name="indexedTypes"> <list> <value>java.lang.String</value> <value>net.juniper.cs.entity.InstallBase</value> </list> </property> </bean>


Cache Config in 2.3:

Here we have to configure DataRegion and then that data region needs to be assigned to Cache.

Data Region Conf:

<property name="dataStorageConfiguration"> <bean class="org.apache.ignite.configuration.DataStorageConfiguration"> <property name="defaultDataRegionConfiguration"> <bean class="org.apache.ignite.configuration.DataRegionConfiguration"> <property name="name" value="5GB_SWAP_Region_Swapping" /> <!-- 100 MB initial size. --> <property name="initialSize" value="#{100L * 1024 * 1024}" /> <!-- Setting the size of the default region to 5GB. --> <property name="maxSize" value="#{5 * 1024L * 1024L * 1024L}" /> </bean> </property> </bean> </property>

Cache Conf:

<?xml version="1.0" encoding="UTF-8"?> <bean class="org.apache.ignite.configuration.CacheConfiguration"> <property name="dataRegionName" value="5GB_SWAP_Region_Swapping" /> <property name="name" value="INSTALLBASE_CACHE" /> <property name="cacheMode" value="PARTITIONED" /> <property name="atomicityMode" value="TRANSACTIONAL" /> <property name="backups" value="0" /> <property name="statisticsEnabled" value="true" /> <!--property name="swapEnabled" value="true"/ --> <property name="copyOnRead" value="false" /> <!--property name="memoryMode" value="OFFHEAP_TIERED"/> <property name="offHeapMaxMemory" value="#{5 * 1024L * 1024L * 1024L}"/ --> <property name="indexedTypes"> <list> <value>java.lang.String</value> <value>net.juniper.cs.entity.InstallBase</value> </list> </property> </bean>



In both cases we are loading data in Offheap memory but still we are observing noticeable increase in the load time(from 40 min to 90 min) of Cache.
Can anyone help me to understand that why this is happening ?

Note:
Data Persistence is not enabled.  


Thanks & Regards
Tejas
Alexey Kukushkin Alexey Kukushkin
Reply | Threaded
Open this post in threaded view
|

Re: Data load is very slow in ignite 2.3 compare to ignite 1.9

Specifying "swapPath" makes Ignite use memory mapped file to store data. Memory mapped files are slower than "raw" RAM. 

Remove "swapPath" setting to have same configuration as you had in Ignite 1.9 (data stored in off-heap RAM).  
Tejashwa Kumar Verma Tejashwa Kumar Verma
Reply | Threaded
Open this post in threaded view
|

Re: Data load is very slow in ignite 2.3 compare to ignite 1.9

Hi Alexei, 

In this case I have already removed swapPath configuration. Please see Data Region config

Still it is slow


Thanks and Regards 
Tejas

On 20 Dec 2017 9:44 pm, "Alexey Kukushkin" <[hidden email]> wrote:
Specifying "swapPath" makes Ignite use memory mapped file to store data. Memory mapped files are slower than "raw" RAM. 

Remove "swapPath" setting to have same configuration as you had in Ignite 1.9 (data stored in off-heap RAM).  
Alexey Kukushkin Alexey Kukushkin
Reply | Threaded
Open this post in threaded view
|

Re: Data load is very slow in ignite 2.3 compare to ignite 1.9

Tejas, how do you load the cache - are you using DataStreamer or SQL, JDBC or put/putAll or something else? Can you confirm - are you saying you have same cluster (same number of nodes and hardware) and after the upgrade the cache load time increased from 40 to 90 minutes?
Tejashwa Kumar Verma Tejashwa Kumar Verma
Reply | Threaded
Open this post in threaded view
|

Re: Data load is very slow in ignite 2.3 compare to ignite 1.9

Yes, I have same cluster, env and no of nodes. 

I am using DataStreamer to load data. 

Thanks and Regards
Tejas 

On 21 Dec 2017 12:11 am, "Alexey Kukushkin" <[hidden email]> wrote:
Tejas, how do you load the cache - are you using DataStreamer or SQL, JDBC or put/putAll or something else? Can you confirm - are you saying you have same cluster (same number of nodes and hardware) and after the upgrade the cache load time increased from 40 to 90 minutes?
Alexey Kukushkin Alexey Kukushkin
Reply | Threaded
Open this post in threaded view
|

Re: Data load is very slow in ignite 2.3 compare to ignite 1.9

Memory architecture changed in release 2 and one of the consequences was Ignite now allocates memory more eagerly. Do you have enough memory on the servers? Is it possible that the amount of memory you allocated for Ignite results in Ignite reserving too much RAM on the servers so that the remaining RAM is not enough for OS and other apps and that causes intensive swapping?



Denis Magda-2 Denis Magda-2
Reply | Threaded
Open this post in threaded view
|

Re: Data load is very slow in ignite 2.3 compare to ignite 1.9

In reply to this post by Tejashwa Kumar Verma
Tejas,

The new memory architecture of Ignite 2.x might require an extra tuning. I find this doc as a good starting point of the scrutiny:

Denis

On Dec 20, 2017, at 10:43 AM, Tejashwa Kumar Verma <[hidden email]> wrote:

Yes, I have same cluster, env and no of nodes. 

I am using DataStreamer to load data. 

Thanks and Regards
Tejas 

On 21 Dec 2017 12:11 am, "Alexey Kukushkin" <[hidden email]> wrote:
Tejas, how do you load the cache - are you using DataStreamer or SQL, JDBC or put/putAll or something else? Can you confirm - are you saying you have same cluster (same number of nodes and hardware) and after the upgrade the cache load time increased from 40 to 90 minutes?

Tejashwa Kumar Verma Tejashwa Kumar Verma
Reply | Threaded
Open this post in threaded view
|

Re: Data load is very slow in ignite 2.3 compare to ignite 1.9

HI Alexey,

We have enough memory(around 48 GB) on server whereas allocation wise we are assigning/utilizing only 15GB memory.


@Denis, I have tried all the configs given in mentioned link. But its not helping out. 


Thanks & regards
Tejas

On Thu, Dec 21, 2017 at 5:44 AM, Denis Magda <[hidden email]> wrote:
Tejas,

The new memory architecture of Ignite 2.x might require an extra tuning. I find this doc as a good starting point of the scrutiny:

Denis

On Dec 20, 2017, at 10:43 AM, Tejashwa Kumar Verma <[hidden email]> wrote:

Yes, I have same cluster, env and no of nodes. 

I am using DataStreamer to load data. 

Thanks and Regards
Tejas 

On 21 Dec 2017 12:11 am, "Alexey Kukushkin" <[hidden email]> wrote:
Tejas, how do you load the cache - are you using DataStreamer or SQL, JDBC or put/putAll or something else? Can you confirm - are you saying you have same cluster (same number of nodes and hardware) and after the upgrade the cache load time increased from 40 to 90 minutes?


Denis Magda-2 Denis Magda-2
Reply | Threaded
Open this post in threaded view
|

Re: Data load is very slow in ignite 2.3 compare to ignite 1.9

Does it mean that you have 3 cluster nodes and all of them are running on a single server? Is data preloaded from a different machine?

Denis

On Dec 20, 2017, at 8:09 PM, Tejashwa Kumar Verma <[hidden email]> wrote:

HI Alexey,

We have enough memory(around 48 GB) on server whereas allocation wise we are assigning/utilizing only 15GB memory.


@Denis, I have tried all the configs given in mentioned link. But its not helping out. 


Thanks & regards
Tejas

On Thu, Dec 21, 2017 at 5:44 AM, Denis Magda <[hidden email]> wrote:
Tejas,

The new memory architecture of Ignite 2.x might require an extra tuning. I find this doc as a good starting point of the scrutiny:

Denis

On Dec 20, 2017, at 10:43 AM, Tejashwa Kumar Verma <[hidden email]> wrote:

Yes, I have same cluster, env and no of nodes. 

I am using DataStreamer to load data. 

Thanks and Regards
Tejas 

On 21 Dec 2017 12:11 am, "Alexey Kukushkin" <[hidden email]> wrote:
Tejas, how do you load the cache - are you using DataStreamer or SQL, JDBC or put/putAll or something else? Can you confirm - are you saying you have same cluster (same number of nodes and hardware) and after the upgrade the cache load time increased from 40 to 90 minutes?



Tejashwa Kumar Verma Tejashwa Kumar Verma
Reply | Threaded
Open this post in threaded view
|

Re: Data load is very slow in ignite 2.3 compare to ignite 1.9

Hi Denis,

I dont know that i got your question correctly or not.
But still attempting to ans.

For now i have 2 node cluster and both have 48-48 GB RAM available. And data is not Preloaded .


Thanks & Regards
Tejas

On Thu, Dec 21, 2017 at 9:55 AM, Denis Magda <[hidden email]> wrote:
Does it mean that you have 3 cluster nodes and all of them are running on a single server? Is data preloaded from a different machine?

Denis

On Dec 20, 2017, at 8:09 PM, Tejashwa Kumar Verma <[hidden email]> wrote:

HI Alexey,

We have enough memory(around 48 GB) on server whereas allocation wise we are assigning/utilizing only 15GB memory.


@Denis, I have tried all the configs given in mentioned link. But its not helping out. 


Thanks & regards
Tejas

On Thu, Dec 21, 2017 at 5:44 AM, Denis Magda <[hidden email]> wrote:
Tejas,

The new memory architecture of Ignite 2.x might require an extra tuning. I find this doc as a good starting point of the scrutiny:

Denis

On Dec 20, 2017, at 10:43 AM, Tejashwa Kumar Verma <[hidden email]> wrote:

Yes, I have same cluster, env and no of nodes. 

I am using DataStreamer to load data. 

Thanks and Regards
Tejas 

On 21 Dec 2017 12:11 am, "Alexey Kukushkin" <[hidden email]> wrote:
Tejas, how do you load the cache - are you using DataStreamer or SQL, JDBC or put/putAll or something else? Can you confirm - are you saying you have same cluster (same number of nodes and hardware) and after the upgrade the cache load time increased from 40 to 90 minutes?




Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Data load is very slow in ignite 2.3 compare to ignite 1.9

Why are you giving only 5GB of RAM to every node then (referring to your data region configuration)? You mentioned that it’s fine to assign 15GB of RAM. Does it mean there are another processes running on the server that use the rest of RAM heavily.

To make the troubleshooting of your problem more effectively, please upload your complete configuration and the code of preloader that calls Ignite data streamer on GitHub and share with us.

Denis

On Dec 20, 2017, at 8:34 PM, Tejashwa Kumar Verma <[hidden email]> wrote:

Hi Denis,

I dont know that i got your question correctly or not.
But still attempting to ans.

For now i have 2 node cluster and both have 48-48 GB RAM available. And data is not Preloaded .


Thanks & Regards
Tejas

On Thu, Dec 21, 2017 at 9:55 AM, Denis Magda <[hidden email]> wrote:
Does it mean that you have 3 cluster nodes and all of them are running on a single server? Is data preloaded from a different machine?

Denis

On Dec 20, 2017, at 8:09 PM, Tejashwa Kumar Verma <[hidden email]> wrote:

HI Alexey,

We have enough memory(around 48 GB) on server whereas allocation wise we are assigning/utilizing only 15GB memory.


@Denis, I have tried all the configs given in mentioned link. But its not helping out. 


Thanks & regards
Tejas

On Thu, Dec 21, 2017 at 5:44 AM, Denis Magda <[hidden email]> wrote:
Tejas,

The new memory architecture of Ignite 2.x might require an extra tuning. I find this doc as a good starting point of the scrutiny:

Denis

On Dec 20, 2017, at 10:43 AM, Tejashwa Kumar Verma <[hidden email]> wrote:

Yes, I have same cluster, env and no of nodes. 

I am using DataStreamer to load data. 

Thanks and Regards
Tejas 

On 21 Dec 2017 12:11 am, "Alexey Kukushkin" <[hidden email]> wrote:
Tejas, how do you load the cache - are you using DataStreamer or SQL, JDBC or put/putAll or something else? Can you confirm - are you saying you have same cluster (same number of nodes and hardware) and after the upgrade the cache load time increased from 40 to 90 minutes?





dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Data load is very slow in ignite 2.3 compare to ignite 1.9

Hi Tejas,

Were you able to resolve your issue? If yes, it would be nice to share it with the community.

D.

On Wed, Dec 20, 2017 at 11:09 PM, Denis Magda <[hidden email]> wrote:
Why are you giving only 5GB of RAM to every node then (referring to your data region configuration)? You mentioned that it’s fine to assign 15GB of RAM. Does it mean there are another processes running on the server that use the rest of RAM heavily.

To make the troubleshooting of your problem more effectively, please upload your complete configuration and the code of preloader that calls Ignite data streamer on GitHub and share with us.

Denis

On Dec 20, 2017, at 8:34 PM, Tejashwa Kumar Verma <[hidden email]> wrote:

Hi Denis,

I dont know that i got your question correctly or not.
But still attempting to ans.

For now i have 2 node cluster and both have 48-48 GB RAM available. And data is not Preloaded .


Thanks & Regards
Tejas

On Thu, Dec 21, 2017 at 9:55 AM, Denis Magda <[hidden email]> wrote:
Does it mean that you have 3 cluster nodes and all of them are running on a single server? Is data preloaded from a different machine?

Denis

On Dec 20, 2017, at 8:09 PM, Tejashwa Kumar Verma <[hidden email]> wrote:

HI Alexey,

We have enough memory(around 48 GB) on server whereas allocation wise we are assigning/utilizing only 15GB memory.


@Denis, I have tried all the configs given in mentioned link. But its not helping out. 


Thanks & regards
Tejas

On Thu, Dec 21, 2017 at 5:44 AM, Denis Magda <[hidden email]> wrote:
Tejas,

The new memory architecture of Ignite 2.x might require an extra tuning. I find this doc as a good starting point of the scrutiny:

Denis

On Dec 20, 2017, at 10:43 AM, Tejashwa Kumar Verma <[hidden email]> wrote:

Yes, I have same cluster, env and no of nodes. 

I am using DataStreamer to load data. 

Thanks and Regards
Tejas 

On 21 Dec 2017 12:11 am, "Alexey Kukushkin" <[hidden email]> wrote:
Tejas, how do you load the cache - are you using DataStreamer or SQL, JDBC or put/putAll or something else? Can you confirm - are you saying you have same cluster (same number of nodes and hardware) and after the upgrade the cache load time increased from 40 to 90 minutes?