Heap memory getting increase while doing cache.get()

classic Classic list List threaded Threaded
4 messages Options
TaqniaTech TaqniaTech
Reply | Threaded
Open this post in threaded view
|

Heap memory getting increase while doing cache.get()

Hi,

 

We have a simple client server mode of apache ignite and we see a memory increase in server node heap while doing “cache.get(cacheKey)”.

 

Below is our scenario.

 

1.       First we insert a one object list with “cacheKey”.

2.       Then we get it for like 200 times. In that case server node heap memory grows and finally it shuts down.

 

Why the heap is growing while just doing the “get(cacheKey)”.

 

Below is our cache configuration.

 

CacheConfiguration cacheCfg = new CacheConfiguration(“TEST CACHE”);

       cacheCfg.setExpiryPolicyFactory(FactoryBuilder.factoryOf(new CreatedExpiryPolicy(new Duration(TimeUnit.MINUTES, 15))));

       cacheCfg.setEagerTtl(true);

       cacheCfg.setDefaultLockTimeout(2000);

       cacheCfg.setBackups(1);

       cacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);

       cacheCfg.setCacheMode(CacheMode.PARTITIONED);

       cacheCfg.setMemoryMode(CacheMemoryMode.ONHEAP_TIERED);

       cacheCfg.setOffHeapMaxMemory(1*1024*1024*1024);

       cacheCfg.setEvictSynchronized(true);

   cacheCfg.setEvictionPolicy(new LruEvictionPolicy(500));

 

 

Below is the sample code

 

IgniteCache igniteCache = Ignition.ignite("CACHE_GRID").getOrCreateCache(cacheCfg,nearCfg);

                    

                    

       List<CacheData> cacheList = new ArrayList<CacheData>();

       for(int y =0;y<100;y++)

       {

              cacheList.add(new CacheData() );

       }

       System.out.println("100 objects added");

                    

       igniteCache.putIfAbsent("A", cacheList);

 

       int k =100;

       for(int i=0;i<k;i++)

       {

          igniteCache.get("A");

       }

 

Thanks and Regards,

Prabuddha

Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Heap memory getting increase while doing cache.get()

Hi Prabuddha,

Cache.get() operation returns copy of an object stored in a cache by default. If you set CacheConfiguration.setCopyOnRead to false then a copy won't be created but you mustn't modify returned object directly because it can effect consistency.

In any case the objects returned by cache.get() must be cleaned by Garbage Collector. Probably you should modify your GC settings to overcome application shutdown due to insufficient memory or long GC.
Please set this generic recommended settings to see the difference
https://apacheignite.readme.io/docs/performance-tips#tune-garbage-collection

--
Denis
TaqniaTech TaqniaTech
Reply | Threaded
Open this post in threaded view
|

RE: Heap memory getting increase while doing cache.get()

Hi,

Thank you for the response and it reduces the growing of heap.

Now we have got below exception when there are two server nodes and both
servers shuts down because of this.

[19:00:49,685][SEVERE][tcp-disco-msg-worker-#2%IBECACHE_GRID%][TcpDiscoveryS
pi] TcpDiscoverSpi's message worker thread failed abnormally. Stopping the
node in order to prevent cluster wide instability.
java.lang.InterruptedException
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.report
InterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:2095)
    at
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:
519)
    at
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:682)
    at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(Ser
verImpl.java:5779)
    at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(Server
Impl.java:2161)
    at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[19:00:53] Topology snapshot [ver=5, servers=1, clients=0, CPUs=8,
heap=4.0GB]
[19:01:06] Ignite node stopped OK [name=IBECACHE_GRID, uptime=00:14:33:947]

See our concerns below and correct us if anything misunderstood.

        1. In a PARTITIONED cache how the data is divided within the cache
nodes ?
             If the Total data size is 1GB, does it divide 500 MB for each ?
               
            In our case we have seen both nodes' heap size is looks equal to
1GB.
       
        2. When there are multiple server nodes to which node client
connects ? Is it the oldest node ?


Also could you let us know is there any other document that we can refer on
cluster configuration.


Regards,
Prabuddha.



-----Original Message-----
From: Denis Magda [mailto:[hidden email]]
Sent: Tuesday, April 05, 2016 12:34 PM
To: [hidden email]
Subject: Re: Heap memory getting increase while doing cache.get()

Hi Prabuddha,

Cache.get() operation returns copy of an object stored in a cache by
default. If you set CacheConfiguration.setCopyOnRead to false then a copy
won't be created but you mustn't modify returned object directly because it
can effect consistency.

In any case the objects returned by cache.get() must be cleaned by Garbage
Collector. Probably you should modify your GC settings to overcome
application shutdown due to insufficient memory or long GC.
Please set this generic recommended settings to see the difference
https://apacheignite.readme.io/docs/performance-tips#tune-garbage-collection

--
Denis



--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Heap-memory-getting-increase-
while-doing-cache-get-tp3925p3930.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

RE: Heap memory getting increase while doing cache.get()

Hi,

Can you please attach the whole log? Most likely the node was still suffering from GC pauses or OOM and was segmented.

Answering your questions:

1. Yes, data in partitioned cache is evenly split across nodes. But note that there is always Ignite overhead on top of the data itself. Refer to [1] for some details and hints. Also keep in mind that memory is managed by JVM, so part of the heap is usually consumed by garbage which will be eventually collected. So the reported amount of consumed memory is not always accurate. To check this you can use any tool like VisualVM to force garbage collection. After it's done, you will get more informative picture.
2. Client node will connect to one of the addresses provided by the IP finder. There is no single point where all clients connect to.

-Val