Out-of-memory issue on single node cache with durable persistence

classic Classic list List threaded Threaded
3 messages Options
Scott Scott
Reply | Threaded
Open this post in threaded view
|

Out-of-memory issue on single node cache with durable persistence

Hello,

I recently ran into an out-of-memory error on a durable persistent cache I set up a few weeks ago.  I have a single node, with durable persistence enabled, as well as WAL archiving.  I'm running Ignite ver. 2.8.1#20200521-sha1:86422096.

I looked at the stack trace, but I couldn't get a clear fix on what part of the system ran out of memory, or what parameters I should change to fix the problem.  From what I could tell of the stack dump, it looks like the WAL archive ran out of memory;  but the memory usage report that occurred just a minute before the exception showed plenty of memory was available.

Can someone with more experience tuning Ignite memory point me towards the configuration parameters I should adjust?  Below are my log and my configuration.  ( I have read the wiki page on memory tuning, but I'm happy to be referred back to it.)

The log, with the metrics right before the OOM exception, then the OOM exception:

[2020-11-22T19:20:39,787][INFO ][grid-timeout-worker-#22][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=2845fe3e, uptime=5 days, 15:08:38.033]
    ^-- Cluster [hosts=1, CPUs=4, servers=1, clients=0, topVer=1, minorTopVer=1]
    ^-- Network [addrs=[0:0:0:0:0:0:0:1%lo, xxx.xxx.xxx.xxx, 127.0.0.1, yyy.yyy.yyy.yyy], discoPort=47500, commPort=47100]
    ^-- CPU [CPUs=4, curLoad=0.33%, avgLoad=0.29%, GC=0%]
    ^-- Heap [used=316MB, free=62.34%, comm=812MB]
    ^-- Off-heap memory [used=4288MB, free=33.45%, allocated=6344MB]
    ^-- Page memory [pages=1085139]
    ^--   sysMemPlc region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, allocRam=100MB, allocTotal=0MB]
    ^--   default_region region [type=default, persistence=true, lazyAlloc=true,
      ...  initCfg=256MB, maxCfg=6144MB, usedRam=4288MB, freeRam=30.2%, allocRam=6144MB, allocTotal=4240MB]
    ^--   metastoreMemPlc region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.94%, allocRam=0MB, allocTotal=0MB]
    ^--   TxLog region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=100MB, allocTotal=0MB]
    ^-- Ignite persistence [used=4240MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=6, qSize=0]
[2020-11-22T19:21:15,585][ERROR][db-checkpoint-thread-#63][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError]]
java.lang.OutOfMemoryError: null
        at sun.misc.Unsafe.allocateMemory(Native Method) ~[?:1.8.0_121]
        at org.apache.ignite.internal.util.GridUnsafe.allocateMemory(GridUnsafe.java:1205) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.util.GridUnsafe.allocateBuffer(GridUnsafe.java:264) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.ByteBufferExpander.<init>(ByteBufferExpander.java:36) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.<init>(AbstractWalRecordsIterator.java:125) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2701) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2637) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:944) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:920) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry$GroupStateLazyStore.initIfNeeded(CheckpointEntry.java:347) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry$GroupStateLazyStore.access$300(CheckpointEntry.java:243) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry.initIfNeeded(CheckpointEntry.java:122) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry.groupState(CheckpointEntry.java:104) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointHistory.addCpToEarliestCpMap(CheckpointHistory.java:242) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointHistory.addCheckpoint(CheckpointHistory.java:175) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:3952) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3515) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3404) [ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.9.0.jar:2.9.0]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

My configuration:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
      <property name="gridLogger">
        <bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
          <constructor-arg type="java.lang.String" value="/etc/apache-ignite/log4j2.xml"/>
        </bean>
      </property>
      <property name="dataStorageConfiguration">
        <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
          <property name="storagePath" value="/data/ignite/persistent-cache"/>
          <property name="walPath" value="/data/ignite/wal"/>
          <property name="walArchivePath" value="/data/ignite/wal/archive"/>
          <property name="defaultDataRegionConfiguration">
            <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
              <property name="name" value="default_region"/>
              <property name="persistenceEnabled" value="true"/>
              <property name="maxSize" value="#{6L * 1024 * 1024 * 1024}"/>
            </bean>
          </property>
          <property name="pageSize" value="#{4 * 1024}"/>
          <property name="maxWalArchiveSize" value="#{100L * 1024 * 1024 * 1024}"/>
        </bean>
      </property>
      <property name="cacheConfiguration">
        <list>
          <bean class="org.apache.ignite.configuration.CacheConfiguration">
            <property name="name" value="default" />
            <property name="atomicityMode" value="ATOMIC" />
            <property name="backups" value="1"/>
            <property name="dataRegionName" value="default_region"/>
          </bean>
        </list>
      </property>
      <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
          <property name="ipFinder">
            <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
              <property name="addresses">
                <list>
                  <value>127.0.0.1</value>
                </list>
              </property>
            </bean>
          </property>
        </bean>
      </property>
      <property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
          <property name="idleConnectionTimeout" value="60000"/>
        </bean>
      </property>
    </bean>
</beans>

Thanks in advance,

-- Scott
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Out-of-memory issue on single node cache with durable persistence

Hello!

It seems that you have run out of available memory. I.e., your operating system could not allocate more memory even though the demand was still in the range permitted by data region configuration. How much RAM do you have on that machine?

That you still have heap left is irrelevant here, since the allocation is for non-heap memory.

Regards,
--
Ilya Kasnacheev


пн, 23 нояб. 2020 г. в 21:44, Scott Prater <[hidden email]>:
Hello,

I recently ran into an out-of-memory error on a durable persistent cache I set up a few weeks ago.  I have a single node, with durable persistence enabled, as well as WAL archiving.  I'm running Ignite ver. 2.8.1#20200521-sha1:86422096.

I looked at the stack trace, but I couldn't get a clear fix on what part of the system ran out of memory, or what parameters I should change to fix the problem.  From what I could tell of the stack dump, it looks like the WAL archive ran out of memory;  but the memory usage report that occurred just a minute before the exception showed plenty of memory was available.

Can someone with more experience tuning Ignite memory point me towards the configuration parameters I should adjust?  Below are my log and my configuration.  ( I have read the wiki page on memory tuning, but I'm happy to be referred back to it.)

The log, with the metrics right before the OOM exception, then the OOM exception:

[2020-11-22T19:20:39,787][INFO ][grid-timeout-worker-#22][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=2845fe3e, uptime=5 days, 15:08:38.033]
    ^-- Cluster [hosts=1, CPUs=4, servers=1, clients=0, topVer=1, minorTopVer=1]
    ^-- Network [addrs=[0:0:0:0:0:0:0:1%lo, xxx.xxx.xxx.xxx, 127.0.0.1, yyy.yyy.yyy.yyy], discoPort=47500, commPort=47100]
    ^-- CPU [CPUs=4, curLoad=0.33%, avgLoad=0.29%, GC=0%]
    ^-- Heap [used=316MB, free=62.34%, comm=812MB]
    ^-- Off-heap memory [used=4288MB, free=33.45%, allocated=6344MB]
    ^-- Page memory [pages=1085139]
    ^--   sysMemPlc region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, allocRam=100MB, allocTotal=0MB]
    ^--   default_region region [type=default, persistence=true, lazyAlloc=true,
      ...  initCfg=256MB, maxCfg=6144MB, usedRam=4288MB, freeRam=30.2%, allocRam=6144MB, allocTotal=4240MB]
    ^--   metastoreMemPlc region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.94%, allocRam=0MB, allocTotal=0MB]
    ^--   TxLog region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=100MB, allocTotal=0MB]
    ^-- Ignite persistence [used=4240MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=6, qSize=0]
[2020-11-22T19:21:15,585][ERROR][db-checkpoint-thread-#63][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError]]
java.lang.OutOfMemoryError: null
        at sun.misc.Unsafe.allocateMemory(Native Method) ~[?:1.8.0_121]
        at org.apache.ignite.internal.util.GridUnsafe.allocateMemory(GridUnsafe.java:1205) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.util.GridUnsafe.allocateBuffer(GridUnsafe.java:264) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.ByteBufferExpander.<init>(ByteBufferExpander.java:36) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.<init>(AbstractWalRecordsIterator.java:125) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2701) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2637) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:944) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:920) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry$GroupStateLazyStore.initIfNeeded(CheckpointEntry.java:347) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry$GroupStateLazyStore.access$300(CheckpointEntry.java:243) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry.initIfNeeded(CheckpointEntry.java:122) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry.groupState(CheckpointEntry.java:104) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointHistory.addCpToEarliestCpMap(CheckpointHistory.java:242) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointHistory.addCheckpoint(CheckpointHistory.java:175) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:3952) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3515) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3404) [ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.9.0.jar:2.9.0]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

My configuration:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
      <property name="gridLogger">
        <bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
          <constructor-arg type="java.lang.String" value="/etc/apache-ignite/log4j2.xml"/>
        </bean>
      </property>
      <property name="dataStorageConfiguration">
        <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
          <property name="storagePath" value="/data/ignite/persistent-cache"/>
          <property name="walPath" value="/data/ignite/wal"/>
          <property name="walArchivePath" value="/data/ignite/wal/archive"/>
          <property name="defaultDataRegionConfiguration">
            <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
              <property name="name" value="default_region"/>
              <property name="persistenceEnabled" value="true"/>
              <property name="maxSize" value="#{6L * 1024 * 1024 * 1024}"/>
            </bean>
          </property>
          <property name="pageSize" value="#{4 * 1024}"/>
          <property name="maxWalArchiveSize" value="#{100L * 1024 * 1024 * 1024}"/>
        </bean>
      </property>
      <property name="cacheConfiguration">
        <list>
          <bean class="org.apache.ignite.configuration.CacheConfiguration">
            <property name="name" value="default" />
            <property name="atomicityMode" value="ATOMIC" />
            <property name="backups" value="1"/>
            <property name="dataRegionName" value="default_region"/>
          </bean>
        </list>
      </property>
      <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
          <property name="ipFinder">
            <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
              <property name="addresses">
                <list>
                  <value>127.0.0.1</value>
                </list>
              </property>
            </bean>
          </property>
        </bean>
      </property>
      <property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
          <property name="idleConnectionTimeout" value="60000"/>
        </bean>
      </property>
    </bean>
</beans>

Thanks in advance,

-- Scott
Scott Scott
Reply | Threaded
Open this post in threaded view
|

Re: Out-of-memory issue on single node cache with durable persistence

D'oh!  I thought our production machine (where the error occurred) had the same amount of memory as our test machine, but it doesn't:  I configured the test machine with 15GB of memory, but the production machine only has 4GB.  That would explain it.

Thank you very much!

-- Scott

On Tue, Nov 24, 2020 at 9:23 AM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

It seems that you have run out of available memory. I.e., your operating system could not allocate more memory even though the demand was still in the range permitted by data region configuration. How much RAM do you have on that machine?

That you still have heap left is irrelevant here, since the allocation is for non-heap memory.

Regards,
--
Ilya Kasnacheev


пн, 23 нояб. 2020 г. в 21:44, Scott Prater <[hidden email]>:
Hello,

I recently ran into an out-of-memory error on a durable persistent cache I set up a few weeks ago.  I have a single node, with durable persistence enabled, as well as WAL archiving.  I'm running Ignite ver. 2.8.1#20200521-sha1:86422096.

I looked at the stack trace, but I couldn't get a clear fix on what part of the system ran out of memory, or what parameters I should change to fix the problem.  From what I could tell of the stack dump, it looks like the WAL archive ran out of memory;  but the memory usage report that occurred just a minute before the exception showed plenty of memory was available.

Can someone with more experience tuning Ignite memory point me towards the configuration parameters I should adjust?  Below are my log and my configuration.  ( I have read the wiki page on memory tuning, but I'm happy to be referred back to it.)

The log, with the metrics right before the OOM exception, then the OOM exception:

[2020-11-22T19:20:39,787][INFO ][grid-timeout-worker-#22][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=2845fe3e, uptime=5 days, 15:08:38.033]
    ^-- Cluster [hosts=1, CPUs=4, servers=1, clients=0, topVer=1, minorTopVer=1]
    ^-- Network [addrs=[0:0:0:0:0:0:0:1%lo, xxx.xxx.xxx.xxx, 127.0.0.1, yyy.yyy.yyy.yyy], discoPort=47500, commPort=47100]
    ^-- CPU [CPUs=4, curLoad=0.33%, avgLoad=0.29%, GC=0%]
    ^-- Heap [used=316MB, free=62.34%, comm=812MB]
    ^-- Off-heap memory [used=4288MB, free=33.45%, allocated=6344MB]
    ^-- Page memory [pages=1085139]
    ^--   sysMemPlc region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, allocRam=100MB, allocTotal=0MB]
    ^--   default_region region [type=default, persistence=true, lazyAlloc=true,
      ...  initCfg=256MB, maxCfg=6144MB, usedRam=4288MB, freeRam=30.2%, allocRam=6144MB, allocTotal=4240MB]
    ^--   metastoreMemPlc region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.94%, allocRam=0MB, allocTotal=0MB]
    ^--   TxLog region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=100MB, allocTotal=0MB]
    ^-- Ignite persistence [used=4240MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=6, qSize=0]
[2020-11-22T19:21:15,585][ERROR][db-checkpoint-thread-#63][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError]]
java.lang.OutOfMemoryError: null
        at sun.misc.Unsafe.allocateMemory(Native Method) ~[?:1.8.0_121]
        at org.apache.ignite.internal.util.GridUnsafe.allocateMemory(GridUnsafe.java:1205) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.util.GridUnsafe.allocateBuffer(GridUnsafe.java:264) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.ByteBufferExpander.<init>(ByteBufferExpander.java:36) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.<init>(AbstractWalRecordsIterator.java:125) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2701) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2637) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:944) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:920) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry$GroupStateLazyStore.initIfNeeded(CheckpointEntry.java:347) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry$GroupStateLazyStore.access$300(CheckpointEntry.java:243) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry.initIfNeeded(CheckpointEntry.java:122) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry.groupState(CheckpointEntry.java:104) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointHistory.addCpToEarliestCpMap(CheckpointHistory.java:242) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointHistory.addCheckpoint(CheckpointHistory.java:175) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:3952) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3515) ~[ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3404) [ignite-core-2.9.0.jar:2.9.0]
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.9.0.jar:2.9.0]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

My configuration:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
      <property name="gridLogger">
        <bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
          <constructor-arg type="java.lang.String" value="/etc/apache-ignite/log4j2.xml"/>
        </bean>
      </property>
      <property name="dataStorageConfiguration">
        <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
          <property name="storagePath" value="/data/ignite/persistent-cache"/>
          <property name="walPath" value="/data/ignite/wal"/>
          <property name="walArchivePath" value="/data/ignite/wal/archive"/>
          <property name="defaultDataRegionConfiguration">
            <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
              <property name="name" value="default_region"/>
              <property name="persistenceEnabled" value="true"/>
              <property name="maxSize" value="#{6L * 1024 * 1024 * 1024}"/>
            </bean>
          </property>
          <property name="pageSize" value="#{4 * 1024}"/>
          <property name="maxWalArchiveSize" value="#{100L * 1024 * 1024 * 1024}"/>
        </bean>
      </property>
      <property name="cacheConfiguration">
        <list>
          <bean class="org.apache.ignite.configuration.CacheConfiguration">
            <property name="name" value="default" />
            <property name="atomicityMode" value="ATOMIC" />
            <property name="backups" value="1"/>
            <property name="dataRegionName" value="default_region"/>
          </bean>
        </list>
      </property>
      <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
          <property name="ipFinder">
            <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
              <property name="addresses">
                <list>
                  <value>127.0.0.1</value>
                </list>
              </property>
            </bean>
          </property>
        </bean>
      </property>
      <property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
          <property name="idleConnectionTimeout" value="60000"/>
        </bean>
      </property>
    </bean>
</beans>

Thanks in advance,

-- Scott