Quantcast

OutOfMemoryException

classic Classic list List threaded Threaded
2 messages Options
Vadim Dedkov Vadim Dedkov
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

OutOfMemoryException

Hello to all.
I use GG version 8.0.1.ea7 & ignite version 1.10.1.ea7.
On my cluster I have 4 server nodes.
Each node have 8 cpu and 16gb memory.
GG visor info on each node: Heap memory initialized: 10gb, Heap memory maximum: 10gb.
I tried to process file (~4.7gb) with Hadoop MR: parse and write file data to GG cache.

And after a while since start job I got errors (end of letter).
I tried to analise error log & source code and one of my versions is: GG tries to swap memory on hard disk --> I don't have enough free space in GG directory --> error.
But I didn't setup cache config "swapEnabled" and I cannot find such config option in Ignite 1.10.ea7 source code.
And in this hypothesis I cannot realise, why GG tries to use swap: 10gb * 4node is not enough for 4.7 gb data?


On one of my GG node:
[16:35:08,442][SEVERE][sys-#10%null%][GridDhtAtomicCache] <calcScorePointsCacheIn_2001> Unexpected exception during cache update
class org.apache.ignite.internal.mem.OutOfMemoryException: null
        at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:242)
        at org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:295)
        at org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.insertDataRow(FreeListImpl.java:325)
        at org.apache.ignite.internal.processors.cache.database.RowStore.addRow(RowStore.java:80)
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(IgniteCacheOffheapManagerImpl.java:842)
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.java:302)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2353)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1755)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1593)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3059)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:131)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:288)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:283)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:758)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:287)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:89)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1369)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:997)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:113)
        at org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:950)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

And in MR job log:
Error: org.apache.ignite.cache.CachePartialUpdateException: Failed to update keys (retry update if possible).: [00-1-10060940]
         at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1478)
         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2118)
         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1315)
         at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.GridGainLineCacheFiller.fillCache(GridGainLineCacheFiller.scala:10)
         at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.ValueGroupingReducer.reduce(ValueGroupingReducer.scala:36)
         at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.ValueGroupingReducer.reduce(ValueGroupingReducer.scala:16)
         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
         at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:422)
         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: class org.apache.ignite.internal.processors.cache.CachePartialUpdateCheckedException: Failed to update keys (retry update if possible).: [00-1-10060940]
         at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.onResult(GridNearAtomicSingleUpdateFuture.java:227)
         at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateResponse(GridDhtAtomicCache.java:3077)
         at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$700(GridDhtAtomicCache.java:131)
         at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:306)
         at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:301)
         at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:758)
         at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
         at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:287)
         at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:89)
         at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
         at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1369)
         at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:997)
         at org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:113)
         at org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:950)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)
         Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node.
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKeys(GridNearAtomicUpdateResponse.java:370)
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1828)
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1593)
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3059)
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:131)
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:288)
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:283)
                 ... 12 more
                 Suppressed: class org.apache.ignite.internal.mem.OutOfMemoryException: null
                          at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:242)
                          at org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:295)
                          at org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.insertDataRow(FreeListImpl.java:325)
                          at org.apache.ignite.internal.processors.cache.database.RowStore.addRow(RowStore.java:80)
                          at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(IgniteCacheOffheapManagerImpl.java:842)
                          at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.java:302)
                          at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576)
                          at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093)
                          at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2353)
                          at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1755)
                          ... 17 more



--
_______________             _______________
Best regards,                    С уважением
Vadim Dedkov.                  Вадим Дедков.
Andrey Mashenkov Andrey Mashenkov
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OutOfMemoryException

Hi Vadim,

You wrote to Apache Ignite Community dev list.
Ignite does not have versions you mentioned.

On Thu, Feb 16, 2017 at 7:58 PM, Vadim Dedkov <[hidden email]> wrote:
Hello to all.
I use GG version 8.0.1.ea7 & ignite version 1.10.1.ea7.
On my cluster I have 4 server nodes.
Each node have 8 cpu and 16gb memory.
GG visor info on each node: Heap memory initialized: 10gb, Heap memory maximum: 10gb.
I tried to process file (~4.7gb) with Hadoop MR: parse and write file data to GG cache.

And after a while since start job I got errors (end of letter).
I tried to analise error log & source code and one of my versions is: GG tries to swap memory on hard disk --> I don't have enough free space in GG directory --> error.
But I didn't setup cache config "swapEnabled" and I cannot find such config option in Ignite 1.10.ea7 source code.
And in this hypothesis I cannot realise, why GG tries to use swap: 10gb * 4node is not enough for 4.7 gb data?


On one of my GG node:
[16:35:08,442][SEVERE][sys-#10%null%][GridDhtAtomicCache] <calcScorePointsCacheIn_2001> Unexpected exception during cache update
class org.apache.ignite.internal.mem.OutOfMemoryException: null
        at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:242)
        at org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:295)
        at org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.insertDataRow(FreeListImpl.java:325)
        at org.apache.ignite.internal.processors.cache.database.RowStore.addRow(RowStore.java:80)
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(IgniteCacheOffheapManagerImpl.java:842)
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.java:302)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2353)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1755)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1593)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3059)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:131)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:288)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:283)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:758)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:287)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:89)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1369)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:997)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:113)
        at org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:950)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

And in MR job log:
Error: org.apache.ignite.cache.CachePartialUpdateException: Failed to update keys (retry update if possible).: [00-1-10060940]
         at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1478)
         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2118)
         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1315)
         at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.GridGainLineCacheFiller.fillCache(GridGainLineCacheFiller.scala:10)
         at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.ValueGroupingReducer.reduce(ValueGroupingReducer.scala:36)
         at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.ValueGroupingReducer.reduce(ValueGroupingReducer.scala:16)
         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
         at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:422)
         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: class org.apache.ignite.internal.processors.cache.CachePartialUpdateCheckedException: Failed to update keys (retry update if possible).: [00-1-10060940]
         at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.onResult(GridNearAtomicSingleUpdateFuture.java:227)
         at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateResponse(GridDhtAtomicCache.java:3077)
         at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$700(GridDhtAtomicCache.java:131)
         at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:306)
         at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:301)
         at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:758)
         at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
         at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:287)
         at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:89)
         at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
         at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1369)
         at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:997)
         at org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:113)
         at org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:950)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)
         Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node.
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKeys(GridNearAtomicUpdateResponse.java:370)
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1828)
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1593)
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3059)
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:131)
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:288)
                 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:283)
                 ... 12 more
                 Suppressed: class org.apache.ignite.internal.mem.OutOfMemoryException: null
                          at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:242)
                          at org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:295)
                          at org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.insertDataRow(FreeListImpl.java:325)
                          at org.apache.ignite.internal.processors.cache.database.RowStore.addRow(RowStore.java:80)
                          at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(IgniteCacheOffheapManagerImpl.java:842)
                          at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.java:302)
                          at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576)
                          at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093)
                          at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2353)
                          at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1755)
                          ... 17 more



--
_______________             _______________
Best regards,                    С уважением
Vadim Dedkov.                  Вадим Дедков.

Loading...