Unable to cache beyond RAM memory. Help me PLease!!!

classic Classic list List threaded Threaded
12 messages Options
bhargav bhargav
Reply | Threaded
Open this post in threaded view
|

Unable to cache beyond RAM memory. Help me PLease!!!

This post has NOT been accepted by the mailing list yet.
Hi,

I am trying to check the option of ignite, to store data beyond physical RAM. I have started a ignite server  in amazon instance (specifications 1gb RAM, disk space - 60gb). Through another application started ignite as CLIENT Only and started storing data using cache.put(k,v).

The following are the configurations setup maintained for the server

<property name="cacheMode" value="PARTITIONED" />
<property name="atomicityMode" value="ATOMIC" />
<property name="backups" value="0" />
<property name="memoryMode" value="OFFHEAP_TIERED"/>
<property name="swapEnabled" value="true"/>
<property name="offHeapMaxMemory" value="#{10 * 1024L * 1024L * 1024L}" />

The following is the exception I see in the server node, when storing data

Exception in thread "sys-#28%null%" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2367)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535)
        at java.lang.StringBuilder.append(StringBuilder.java:204)
        at org.apache.ignite.internal.util.io.GridUnsafeDataInput.readUTFSpan(GridUnsafeDataInput.java:598)
        at org.apache.ignite.internal.util.io.GridUnsafeDataInput.readUTFBody(GridUnsafeDataInput.java:500)
        at org.apache.ignite.internal.util.io.GridUnsafeDataInput.readUTF(GridUnsafeDataInput.java:479)
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readString(OptimizedObjectInputStream.java:864)
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:272)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at org.apache.ignite.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:248)
        at org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.unmarshal(IgniteCacheObjectProcessorImpl.java:110)
        at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.unmarshal(CacheObjectBinaryProcessorImpl.java:700)
        at org.apache.ignite.internal.processors.cache.CacheObjectImpl.finishUnmarshal(CacheObjectImpl.java:109)
        at org.apache.ignite.internal.processors.cache.GridCacheMessage.finishUnmarshalCacheObjects(GridCacheMessage.java:562)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateRequest.finishUnmarshal(GridNearAtomicUpdateRequest.java:623)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1038)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:275)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:204)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:80)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:163)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:821)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:103)
        at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:784)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


I understood that application is out of heap memory, but server node is started with off heap settings and swap - so how come it is out of heap memory. There are no other applications running in the instance. Might be missing something.

Regards,
Bhargav
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Unable to cache beyond RAM memory. Help me PLease!!!

Hi Bhargav,

Can you please properly subscribe to the mailing list so that community receives email notifications? Follow the instructions here: http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1

bhargav wrote
I am trying to check the option of ignite, to store data beyond physical RAM. I have started a ignite server  in amazon instance (specifications 1gb RAM, disk space - 60gb). Through another application started ignite as CLIENT Only and started storing data using cache.put(k,v).
Several points to consider:

1. Besides the data, Ignite creates several internal structures which also require memory. I would give it at least 2-4GB of heap memory just to make in fully functional.
2. Usually it makes sense to switch to OFFHEAP_TIERED when you data set is large and goes beyond 8-10GB per node, because large heaps can cause large GC pauses. For small data sets ONHEAP_TIERED is enough.
3. Most important: Ignite is in-memory system and having all the data in swap space will most likely slow you down instead of giving performance benefits. What are you trying to achieve with this configuration?

-Val
-Val
bhargav bhargav
Reply | Threaded
Open this post in threaded view
|

Re: Unable to cache beyond RAM memory. Help me PLease!!!

This post has NOT been accepted by the mailing list yet.
As suggested, I have used a 4gb RAM machine as ignite server node. And pushed 1gb data from client node to store it in cache. Again I see that the heap memory for JVM in the server node is completely filled. Please be noted that I have increased the heap memory settings in ignite.sh to consider 4gb (entire RAM).

And it was suggested by you to use ONHEAP rather OFFHEAP as the cache data size is lesser (1gb). But I m testing the case to see how off heap memory behaves. And I am stuck with this.

Definitely there is something in ignite process, which is consuming hell lot of heap space!!!

Just after starting ignite in server node - the memory occupied is around 330- 350 mb. As the data is being pushed from client to be stored in server, the ram space is increasing and finally the server node is getting killed (as per the settings in the ingnite.sh when ignite process is out of heap space)

And to mention that, I have uncommented GC settings in ignite.sh file (suggested to do so in case of sudden spikes in through put due to GC)
 
I have tried using datastreamer and IGFS as well, to see if I get lucky. But still facing the same issue. Please let me know is there a way to over come this issue? or Am I missing something here??

Regards,
Bhargav
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Unable to cache beyond RAM memory. Help me PLease!!!

Hi Bhargav,

Are you sure that 4GB is enough for your data? How many entries do you have and what is the average size of each entry in bytes?

-Val
bhargav bhargav
Reply | Threaded
Open this post in threaded view
|

Re: Unable to cache beyond RAM memory. Help me PLease!!!

This post has NOT been accepted by the mailing list yet.
Hi,

There are totally 124 entries, and each entry is around 10mb.
Ignite server node has 4gb RAM and I am trying to cache 1gb (approx - 124*10mb) of data from my ignite client node which has 128 GB RAM. Since there is only one ignite server node running, the data pushed from client will be cached only in ignite server node(4gb machine)

And I see that somewhere in middle of my process in pushing data from client to server, the server node stops responding and when checked I see that ignite process is killed in the server node (as the heap has reached its limit).

Please be noted that there are no other process running in the server node machine. It is dedicated only for ignite server node.

Regards,
Bhargav
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Unable to cache beyond RAM memory. Help me PLease!!!

Bhargav,

Can you show the full log of the server node?

-Val
bhargav bhargav
Reply | Threaded
Open this post in threaded view
|

Re: Unable to cache beyond RAM memory. Help me PLease!!!

This post has NOT been accepted by the mailing list yet.
Hi,

Here is the link for the requested server node log

https://gist.github.com/anonymous/86009408053a778f6f13

Please have a look and suggest incase i m missing anything.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Unable to cache beyond RAM memory. Help me PLease!!!

Bhargav,

Looks like you're still not subscribed to the mailing list. Can you please do this? Here is the short instruction: http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1

bhargav wrote
Hi,

Here is the link for the requested server node log

https://gist.github.com/anonymous/86009408053a778f6f13

Please have a look and suggest incase i m missing anything.
I'm confused. There are no out of memory in the logs and the latest metrics print out shows that there are 98% free heap:

Heap [used=138MB, free=98.65%, comm=1023MB]

Is it a full log?

I would also recommend to disable FINE logging for Ignite, it's much more verbose than needed right now.

-Val
bhargav bhargav
Reply | Threaded
Open this post in threaded view
|

Re: Unable to cache beyond RAM memory. Help me PLease!!!

This post has NOT been accepted by the mailing list yet.
Sorry sent wrong log...please access the following link to view the log..

https://gist.github.com/anonymous/2be481f7314cfbd6391f.js

The ignite process is automatically killed. Attaching screen shot for reference. The screen refers to the status of the server node. If observed you can see that the ignite server node is killed automatically when data is being stored in ignite server node from client node.

NO other process running in the ignite server node machine. Its config is 4gb and I am trying to cache a data of 1gb.



Regards,
Bhargav

vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Unable to cache beyond RAM memory. Help me PLease!!!

Bhargav,

I don't see any issues in this log either. The process is killed externally by the OS, this is not due to memory issues. This can happen, for example, if you logout from the system. Can this be the case?

-Val
bhargav bhargav
Reply | Threaded
Open this post in threaded view
|

Re: Unable to cache beyond RAM memory. Help me PLease!!!

This post has NOT been accepted by the mailing list yet.
No. I am just waiting for the client to push 1gb data to client server node and all of a sudden I see the killed process details in the server node screen. None of them has killed the process. That is not the case.

And I was monitoring the free memory using free -m command in the server node machine and I see the free memory reducing faster and as it reaches to less than 100mb, the caching process is halted, killing the ignite service

Regards,
Bhargav
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Unable to cache beyond RAM memory. Help me PLease!!!

Bhargav,

How much physical memory do you have?

-Val