Regarding Ignite memory going beyond specified memory

classic Classic list List threaded Threaded
4 messages Options
dhiman_nikhil dhiman_nikhil
Reply | Threaded
Open this post in threaded view
|

Regarding Ignite memory going beyond specified memory

I am running java application having Ignite 2.7.5 as dependency
java -Xms4g -Xmx4g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC  -XX:+DisableExplicitGC -server -XX:MaxMetaspaceSize=256m -XX:+HeapDumpOnOutOfMemoryError  -XX:+ExitOnOutOfMemoryError -jar target/java_simple-1.0-SNAPSHOT-jar-with-dependencies.jar

This application does nothing except running Ignite server.

I have specified max memory as 4g and also internal data region memory as ~ 4g with eviction policy defined.
DataRegionConfiguration regionCfg = new DataRegionConfiguration();
          regionCfg.setName("chat");
          regionCfg.setMaxSize(2000L * 1024*1024);
          regionCfg.setInitialSize(1100L * 1024 *1024);
          regionCfg.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);
          regionCfg.setEvictionThreshold(.7);
          regionCfg.setMetricsEnabled(true);
          DataRegionConfiguration regionCfg2 = new DataRegionConfiguration();
          regionCfg2.setName("btc");
          regionCfg2.setMaxSize(2000L * 1024*1024);
          regionCfg2.setInitialSize(1100L * 1024 *1024);
          regionCfg2.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);
          regionCfg2.setEvictionThreshold(.7);
          regionCfg2.setMetricsEnabled(true);
          storageCfg.setDataRegionConfigurations(regionCfg2, regionCfg);
          cfg.setDataStorageConfiguration(storageCfg);


I am hitting ignite server with 18 clients with average get/put/remove operation of 2k hits per second.
System on which ignite is running has around 8G of ram.

When I run my clients[From different server] that hit the Ignite server for few minutes, it consumes all my system RAM[Ignite server] and it stops with  OOM error.

I don't understand why it goes beyond the memory specified in the data region.

Thanks & Regards,
Nikhil Dhiman
ibelyakov ibelyakov
Reply | Threaded
Open this post in threaded view
|

Re: Regarding Ignite memory going beyond specified memory

Hi Nikhil,

You've specified 4gb for heap size and 4gb for data regions in off heap memory, it's 8gb in total. Since operation system also consumes some amount of memory, 8gb of RAM isn't enough for your case. You should try to reduce heap size/data regions size or increase amount of RAM in your system.

Regards,
Igor

On Fri, Jul 26, 2019 at 1:20 PM nikhil dhiman <[hidden email]> wrote:
I am running java application having Ignite 2.7.5 as dependency
java -Xms4g -Xmx4g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC  -XX:+DisableExplicitGC -server -XX:MaxMetaspaceSize=256m -XX:+HeapDumpOnOutOfMemoryError  -XX:+ExitOnOutOfMemoryError -jar target/java_simple-1.0-SNAPSHOT-jar-with-dependencies.jar

This application does nothing except running Ignite server.

I have specified max memory as 4g and also internal data region memory as ~ 4g with eviction policy defined.
DataRegionConfiguration regionCfg = new DataRegionConfiguration();
          regionCfg.setName("chat");
          regionCfg.setMaxSize(2000L * 1024*1024);
          regionCfg.setInitialSize(1100L * 1024 *1024);
          regionCfg.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);
          regionCfg.setEvictionThreshold(.7);
          regionCfg.setMetricsEnabled(true);
          DataRegionConfiguration regionCfg2 = new DataRegionConfiguration();
          regionCfg2.setName("btc");
          regionCfg2.setMaxSize(2000L * 1024*1024);
          regionCfg2.setInitialSize(1100L * 1024 *1024);
          regionCfg2.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);
          regionCfg2.setEvictionThreshold(.7);
          regionCfg2.setMetricsEnabled(true);
          storageCfg.setDataRegionConfigurations(regionCfg2, regionCfg);
          cfg.setDataStorageConfiguration(storageCfg);


I am hitting ignite server with 18 clients with average get/put/remove operation of 2k hits per second.
System on which ignite is running has around 8G of ram.

When I run my clients[From different server] that hit the Ignite server for few minutes, it consumes all my system RAM[Ignite server] and it stops with  OOM error.

I don't understand why it goes beyond the memory specified in the data region.

Thanks & Regards,
Nikhil Dhiman
dhiman_nikhil dhiman_nikhil
Reply | Threaded
Open this post in threaded view
|

Re: Regarding Ignite memory going beyond specified memory

Thank Igor,
But the problem remains the same. Just to give more clarity.
I am doing load testing on 2 cache keys, with almost 500 Hits per
second which includes get, put and remove.
So when I start with 1 server and my clients, it works fine, but when
I try to add more nodes to my cluster, my initial node fails.

On Fri, Jul 26, 2019 at 5:29 PM Igor Belyakov <[hidden email]> wrote:

>
> Hi Nikhil,
>
> You've specified 4gb for heap size and 4gb for data regions in off heap memory, it's 8gb in total. Since operation system also consumes some amount of memory, 8gb of RAM isn't enough for your case. You should try to reduce heap size/data regions size or increase amount of RAM in your system.
>
> Regards,
> Igor
>
> On Fri, Jul 26, 2019 at 1:20 PM nikhil dhiman <[hidden email]> wrote:
>>
>> I am running java application having Ignite 2.7.5 as dependency
>> java -Xms4g -Xmx4g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC  -XX:+DisableExplicitGC -server -XX:MaxMetaspaceSize=256m -XX:+HeapDumpOnOutOfMemoryError  -XX:+ExitOnOutOfMemoryError -jar target/java_simple-1.0-SNAPSHOT-jar-with-dependencies.jar
>>
>> This application does nothing except running Ignite server.
>>
>> I have specified max memory as 4g and also internal data region memory as ~ 4g with eviction policy defined.
>> DataRegionConfiguration regionCfg = new DataRegionConfiguration();
>>           regionCfg.setName("chat");
>>           regionCfg.setMaxSize(2000L * 1024*1024);
>>           regionCfg.setInitialSize(1100L * 1024 *1024);
>>           regionCfg.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);
>>           regionCfg.setEvictionThreshold(.7);
>>           regionCfg.setMetricsEnabled(true);
>>           DataRegionConfiguration regionCfg2 = new DataRegionConfiguration();
>>           regionCfg2.setName("btc");
>>           regionCfg2.setMaxSize(2000L * 1024*1024);
>>           regionCfg2.setInitialSize(1100L * 1024 *1024);
>>           regionCfg2.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);
>>           regionCfg2.setEvictionThreshold(.7);
>>           regionCfg2.setMetricsEnabled(true);
>>           storageCfg.setDataRegionConfigurations(regionCfg2, regionCfg);
>>           cfg.setDataStorageConfiguration(storageCfg);
>>
>> I am hitting ignite server with 18 clients with average get/put/remove operation of 2k hits per second.
>> System on which ignite is running has around 8G of ram.
>>
>> When I run my clients[From different server] that hit the Ignite server for few minutes, it consumes all my system RAM[Ignite server] and it stops with  OOM error.
>>
>> I don't understand why it goes beyond the memory specified in the data region.
>>
>> Thanks & Regards,
>> Nikhil Dhiman
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Regarding Ignite memory going beyond specified memory

Hello!

Unfortunately it's hard to reason what happens here. Does it also happen if you limit heap to 2G? Do you have more than one node on the same machine?

Regards,
--
Ilya Kasnacheev


пт, 26 июл. 2019 г. в 22:00, nikhil dhiman <[hidden email]>:
Thank Igor,
But the problem remains the same. Just to give more clarity.
I am doing load testing on 2 cache keys, with almost 500 Hits per
second which includes get, put and remove.
So when I start with 1 server and my clients, it works fine, but when
I try to add more nodes to my cluster, my initial node fails.

On Fri, Jul 26, 2019 at 5:29 PM Igor Belyakov <[hidden email]> wrote:
>
> Hi Nikhil,
>
> You've specified 4gb for heap size and 4gb for data regions in off heap memory, it's 8gb in total. Since operation system also consumes some amount of memory, 8gb of RAM isn't enough for your case. You should try to reduce heap size/data regions size or increase amount of RAM in your system.
>
> Regards,
> Igor
>
> On Fri, Jul 26, 2019 at 1:20 PM nikhil dhiman <[hidden email]> wrote:
>>
>> I am running java application having Ignite 2.7.5 as dependency
>> java -Xms4g -Xmx4g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC  -XX:+DisableExplicitGC -server -XX:MaxMetaspaceSize=256m -XX:+HeapDumpOnOutOfMemoryError  -XX:+ExitOnOutOfMemoryError -jar target/java_simple-1.0-SNAPSHOT-jar-with-dependencies.jar
>>
>> This application does nothing except running Ignite server.
>>
>> I have specified max memory as 4g and also internal data region memory as ~ 4g with eviction policy defined.
>> DataRegionConfiguration regionCfg = new DataRegionConfiguration();
>>           regionCfg.setName("chat");
>>           regionCfg.setMaxSize(2000L * 1024*1024);
>>           regionCfg.setInitialSize(1100L * 1024 *1024);
>>           regionCfg.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);
>>           regionCfg.setEvictionThreshold(.7);
>>           regionCfg.setMetricsEnabled(true);
>>           DataRegionConfiguration regionCfg2 = new DataRegionConfiguration();
>>           regionCfg2.setName("btc");
>>           regionCfg2.setMaxSize(2000L * 1024*1024);
>>           regionCfg2.setInitialSize(1100L * 1024 *1024);
>>           regionCfg2.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);
>>           regionCfg2.setEvictionThreshold(.7);
>>           regionCfg2.setMetricsEnabled(true);
>>           storageCfg.setDataRegionConfigurations(regionCfg2, regionCfg);
>>           cfg.setDataStorageConfiguration(storageCfg);
>>
>> I am hitting ignite server with 18 clients with average get/put/remove operation of 2k hits per second.
>> System on which ignite is running has around 8G of ram.
>>
>> When I run my clients[From different server] that hit the Ignite server for few minutes, it consumes all my system RAM[Ignite server] and it stops with  OOM error.
>>
>> I don't understand why it goes beyond the memory specified in the data region.
>>
>> Thanks & Regards,
>> Nikhil Dhiman