[2.9.1] Out of memory in data region

[jjimeno]

Hello,

I've got this simple configuration file:

<bean class="org.apache.ignite.configuration.IgniteConfiguration" id="VDS">
        <property name="dataStorageConfiguration">
                <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
                        <property name="defaultDataRegionConfiguration">
                                <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                                        <property name="initialSize" value="#{1L * 1024 * 1024 * 1024}"/>
                                        <property name="maxSize" value="#{1L * 1024 * 1024 * 1024}"/>
                                        <property name="persistenceEnabled" value="true"/>
                                </bean>
                        </property>
                        <property name="pageSize" value="#{4 * 1024}"/>
                </bean>
        </property>
        <property name="cacheConfiguration">
                <list>
                        <bean class="org.apache.ignite.configuration.CacheConfiguration">
                                <property name="name" value="vds"/>
                                <property name="cacheMode" value="PARTITIONED"/>
                                <property name="atomicityMode" value="TRANSACTIONAL"/>
                                <property name="backups" value="1"/>
                                <property name="writeSynchronizationMode" value="PRIMARY_SYNC"/>
                                <property name="onheapCacheEnabled" value="false"/>
                        </bean>
                </list>
        </property>
</bean>

and, at some point during a transaction of ~1GB, I get this error:

[15:01:23,559][SEVERE][client-connector-#134][] JVM will be halted
immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException: Out
of memory in data region [name=default, initSize=1.0 GiB, maxSize=1.0 GiB,
*persistenceEnabled=true*] Try the following:
  ^-- Increase maximum off-heap memory size
(DataRegionConfiguration.maxSize)
  ^-- *Enable Ignite persistence*
(DataRegionConfiguration.persistenceEnabled)
  ^-- Enable eviction or expiration policies]]

Maybe the off-heap memory must be, at least, as large as the transaction we
want to commit?

Thanks in advance!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[Charlin]

Hello,

Please increase maxsize value based on your data volume and try.

example: <property name="maxSize" value="#{5L * 1024 * 1024 * 1024}"/> -- 5GB is not an approximation.

Thanks & Regards,

Charlin

On Fri, 12 Feb 2021 at 20:52, jjimeno <[hidden email]> wrote:

Hello,

I've got this simple configuration file:

<bean class="org.apache.ignite.configuration.IgniteConfiguration" id="VDS">
        <property name="dataStorageConfiguration">
                <bean class="org.apache.ignite.configuration.DataStorageConfiguration">                         
                        <property name="defaultDataRegionConfiguration">
                                <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                                        <property name="initialSize" value="#{1L * 1024 * 1024 * 1024}"/>
                                        <property name="maxSize" value="#{1L * 1024 * 1024 * 1024}"/>
                                        <property name="persistenceEnabled" value="true"/>
                                </bean>
                        </property>
                        <property name="pageSize" value="#{4 * 1024}"/>
                </bean>
        </property>
        <property name="cacheConfiguration">
                <list>
                        <bean class="org.apache.ignite.configuration.CacheConfiguration">
                                <property name="name" value="vds"/>
                                <property name="cacheMode" value="PARTITIONED"/>
                                <property name="atomicityMode" value="TRANSACTIONAL"/>
                                <property name="backups" value="1"/>
                                <property name="writeSynchronizationMode" value="PRIMARY_SYNC"/>
                                <property name="onheapCacheEnabled" value="false"/>
                        </bean>                         
                </list>
        </property>
</bean>

and, at some point during a transaction of ~1GB, I get this error:

[15:01:23,559][SEVERE][client-connector-#134][] JVM will be halted
immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException: Out
of memory in data region [name=default, initSize=1.0 GiB, maxSize=1.0 GiB,
*persistenceEnabled=true*] Try the following:
  ^-- Increase maximum off-heap memory size
(DataRegionConfiguration.maxSize)
  ^-- *Enable Ignite persistence*
(DataRegionConfiguration.persistenceEnabled)
  ^-- Enable eviction or expiration policies]]

Maybe the off-heap memory must be, at least, as large as the transaction we
want to commit?

Thanks in advance!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[jjimeno]

Hello

Thanks for your help.

I know that fixes the problem, but my question was about why I’m getting
that error when persistence is on.

As far as I know, if persistence is on, off-heap memory holds a subset of
data when they don’t fit in memory, and no OOM error should be thrown...
unless there are some requirements as the one I pointed out in my previous
mail I’m not aware of.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[Ilya Kazakov]

Hello!

Ignite uses off-heap memory, whether the persistence is on or not (it depends on another parameter [1]), but only for a data storing.

Transaction processing is performing on heap, therefore an OOM may occur here.

[1]. https://ignite.apache.org/docs/latest/configuring-caches/on-heap-caching

------------------
Ilya

вс, 14 февр. 2021 г. в 22:32, jjimeno <[hidden email]>:

Hello

Thanks for your help.

I know that fixes the problem, but my question was about why I’m getting
that error when persistence is on.

As far as I know, if persistence is on, off-heap memory holds a subset of
data when they don’t fit in memory, and no OOM error should be thrown...
unless there are some requirements as the one I pointed out in my previous
mail I’m not aware of.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[jjimeno]

Hello!

Ok but, according to the error:

[15:01:23,559][SEVERE][client-connector-#134][] JVM will be halted
immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException: *Out
of memory in data region* [name=default, initSize=1.0 GiB, maxSize=1.0 GiB,
*persistenceEnabled=true*]

the OOM is occurring in data region, and that's off-heap memory, as you
pointed out.  

On the other hand, on-heap memory in the test is 4GB, more than enough for a
1GB transaction...



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[ilya.kasnacheev]

Hello!

I think it's quite possible to run out of off-heap if your transactions are comparable with your data region in time. Ignite needs to do page eviction to be able to reuse off-heap pages, but I guess that pages relevant to currently running transactions can't be evicted.

Regards,

пн, 15 февр. 2021 г. в 09:49, jjimeno <[hidden email]>:

Hello!

Ok but, according to the error:

[15:01:23,559][SEVERE][client-connector-#134][] JVM will be halted
immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException: *Out
of memory in data region* [name=default, initSize=1.0 GiB, maxSize=1.0 GiB,
*persistenceEnabled=true*]

the OOM is occurring in data region, and that's off-heap memory, as you
pointed out. 

On the other hand, on-heap memory in the test is 4GB, more than enough for a
1GB transaction...



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[jjimeno]

Hello,

That's what I was thinking of... thanks Ilya



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/