Ignite Durable Memory Runs out of Heap Space

classic Classic list List threaded Threaded
2 messages Options
jackluo923 jackluo923
Reply | Threaded
Open this post in threaded view
|

Ignite Durable Memory Runs out of Heap Space

Hi,
    I am trying to store many kv pairs into ignite than the available ram.
At this moment, the primary purpose of using ignite is a distributed kv
database.

*My current configuration is: *
<property name="defaultDataRegionConfiguration">                    
   <bean class="org.apache.ignite.configuration.DataRegionConfiguration">                        
                             
      <property name="persistenceEnabled" value="true"/>                        
                                 
         <property name="initialSize" value="#{500L * 1024 * 1024}"/>                        
                                 
         <property name="maxSize" value="#{32L * 1024 * 1024 * 1024}"/>                    
   </bean>                
</property>

The physical machine has 64GB of ram. The data which I am trying to insert
into ignite is spread across multiple caches (around 20). From my current
understanding, we shouldn't receive any "Out of Heap Space" since the ignite
uses "off heap storage" and anything which does not fit in the allocated
cache space will be swapped to disk. Do I have any obvious flaw in my
understanding or incorrect configuration?

A small snippet of log message:
[01:40:16,006][SEVERE][grid-timeout-worker-#31][G] Blocked system-critical
thread has been detected. This can lead to cluster-wide undefined behaviour
[threadName=tcp-disco-msg-worker, blockedFor=119s]
[01:40:16,006][SEVERE][exchange-worker-#53][] Critical system error
detected. Will be handled accordingly toconfigured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
o.a.i.IgniteException: GridWorker [name=partition-exchanger,
igniteInstanceName=null, finished=true,heartbeatTs=1562030913742]]]class
org.apache.ignite.IgniteException: GridWorker [name=partition-exchanger,
igniteInstanceName=null, finished=true, heartbeatTs=1562030913742]        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)      
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)      
at
org.apache.ignite.internal.worker.WorkersRegistry.onStopped(WorkersRegistry.java:169)      
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:153)      
at java.lang.Thread.run(Thread.java:748)Exception in thread
"exchange-worker-#53" java.lang.OutOfMemoryError: Java heap
space[01:40:42,376][SEVERE][tcp-disco-msg-worker-#2][] Critical system error
detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=CRITICAL_ERROR,
err=java.lang.OutOfMemoryError: Java heap space]]java.lang.OutOfMemoryError:
Java heap space[01:40:47,316][SEVERE][tcp-disco-msg-worker-#2][] JVM will be
halted immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Durable Memory Runs out of Heap Space

Hi, 

Ignite stores data off-heap (and that's what you've configured) while the out-of-memory exception reports that Java Heap usage goes beyond some bar.

Please double check you have optimal heap settings and there are no memory leaks:

-
Denis


On Wed, Jul 3, 2019 at 2:55 PM jackluo923 <[hidden email]> wrote:
Hi,
    I am trying to store many kv pairs into ignite than the available ram.
At this moment, the primary purpose of using ignite is a distributed kv
database.

*My current configuration is: *
<property name="defaultDataRegionConfiguration">                   
   <bean class="org.apache.ignite.configuration.DataRegionConfiguration">                       

      <property name="persistenceEnabled" value="true"/>                       

         <property name="initialSize" value="#{500L * 1024 * 1024}"/>                       

         <property name="maxSize" value="#{32L * 1024 * 1024 * 1024}"/>                   
   </bean>               
</property>

The physical machine has 64GB of ram. The data which I am trying to insert
into ignite is spread across multiple caches (around 20). From my current
understanding, we shouldn't receive any "Out of Heap Space" since the ignite
uses "off heap storage" and anything which does not fit in the allocated
cache space will be swapped to disk. Do I have any obvious flaw in my
understanding or incorrect configuration?

A small snippet of log message:
[01:40:16,006][SEVERE][grid-timeout-worker-#31][G] Blocked system-critical
thread has been detected. This can lead to cluster-wide undefined behaviour
[threadName=tcp-disco-msg-worker, blockedFor=119s]
[01:40:16,006][SEVERE][exchange-worker-#53][] Critical system error
detected. Will be handled accordingly toconfigured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
o.a.i.IgniteException: GridWorker [name=partition-exchanger,
igniteInstanceName=null, finished=true,heartbeatTs=1562030913742]]]class
org.apache.ignite.IgniteException: GridWorker [name=partition-exchanger,
igniteInstanceName=null, finished=true, heartbeatTs=1562030913742]        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)       
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)       
at
org.apache.ignite.internal.worker.WorkersRegistry.onStopped(WorkersRegistry.java:169)       
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:153)       
at java.lang.Thread.run(Thread.java:748)Exception in thread
"exchange-worker-#53" java.lang.OutOfMemoryError: Java heap
space[01:40:42,376][SEVERE][tcp-disco-msg-worker-#2][] Critical system error
detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=CRITICAL_ERROR,
err=java.lang.OutOfMemoryError: Java heap space]]java.lang.OutOfMemoryError:
Java heap space[01:40:47,316][SEVERE][tcp-disco-msg-worker-#2][] JVM will be
halted immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/