Cluster keeps crashing with SEVERE message

classic Classic list List threaded Threaded
4 messages Options
Josh Katz Josh Katz
Reply | Threaded
Open this post in threaded view
|

Cluster keeps crashing with SEVERE message

After we load 35 million records from the database into the cluster. When we query it using a scan query we get the following SEVERE error,

After that the node are restarting and the caches all show 0 keys and we have to restart the initial load.

Is there a way to salvage the data so we don’t have to reload every time? For example, if we add persistence at this stage the initial load will be slower correct?

How can we avoid seeing this error? “[SEVERE] Failed to process selector key”

 

 

Thanks,

Josh

 

 


Please follow the hyperlink to important disclosures.
https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html

stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Cluster keeps crashing with SEVERE message

We’d need to see more of the stack trace and see your configuration before this could be diagnosed.

It looks like a node abruptly left the cluster. If you’re losing data it seems like you’ve not configured any backups.

On 10 Mar 2021, at 16:53, Josh Katz <[hidden email]> wrote:

After we load 35 million records from the database into the cluster. When we query it using a scan query we get the following SEVERE error,
After that the node are restarting and the caches all show 0 keys and we have to restart the initial load.
Is there a way to salvage the data so we don’t have to reload every time? For example, if we add persistence at this stage the initial load will be slower correct?
How can we avoid seeing this error? “[SEVERE] Failed to process selector key”
 
<image001.png>
 
Thanks,
Josh
 
 

Please follow the hyperlink to important disclosures.
https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html


Josh Katz Josh Katz
Reply | Threaded
Open this post in threaded view
|

RE: Cluster keeps crashing with SEVERE message

Please see the log, it seems to be OutOfMemory from JVM on Windows Server 2016.

How can we address this issue?

 

^-- System thread pool [active=0, idle=7, qSize=0]

[08:35:20,163][INFO][grid-timeout-worker-#22%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][IgniteKernal%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56] FreeList [name=default##FreeList, buckets=256, dataPages=2657, reusePages=0]

[08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheDistributedQueryManager] <NonEquitiesYoY> Failed to run query [qry=GridCacheQueryInfo [loc=false, trans=null, rdc=null, qry=GridCacheQueryAdapter [type=SCAN, clsName=null, clause=null, filter=null, limit=0, transform=null, part=null, incMeta=false, pageSize=1024, timeout=0, incBackups=false, forceLocal=false, dedup=false, prj=null, keepBinary=true, subjId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, taskHash=0, mvccSnapshot=null, dataPageScanEnabled=null], locFut=null, sndId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, reqId=9, incMeta=false, all=false], node=10276c3c-a550-490e-bc48-edf9596dd1dd]

java.lang.OutOfMemoryError: Java heap space

                at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)

                at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)

                at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)

                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)

                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)

                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)

                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)

                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)

                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)

                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)

[08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheDistributedQueryManager] <NonEquitiesYoY> Failed to run query.

java.lang.OutOfMemoryError: Java heap space

                at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)

                at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)

                at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)

                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)

                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)

                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)

                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)

                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)

                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)

                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)

[08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheIoManager] Failed processing message [senderId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, msg=GridCacheQueryRequest [id=9, cacheName=NonEquitiesYoY, type=SCAN, fields=false, clause=null, limit=0, clsName=null, keyValFilter=null, rdc=null, trans=null, pageSize=1024, incBackups=false, cancel=false, incMeta=false, all=false, keepBinary=true, subjId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, taskHash=0, part=-1, topVer=AffinityTopologyVersion [topVer=163, minorTopVer=0], mvccSnapshot=null, flags=0, super=GridCacheIdMessage [cacheId=-1867634189, super=GridCacheMessage [msgId=10, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=158, minorTopVer=1], err=null, skipPrepare=false]]]]

java.lang.OutOfMemoryError: Java heap space

                at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)

                at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)

                at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)

                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)

                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)

                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)

                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)

                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)

                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)

                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)

[08:35:22,163][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]

java.lang.OutOfMemoryError: Java heap space

                at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)

                at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)

                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)

                at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)

                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)

                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)

                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)

                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)

                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)

                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)

                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)

                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)

                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)

                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)

[08:35:22,179][WARNING][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][CacheDiagnosticManager] Page locks dump:

 

Thread=[name=exchange-worker-#48%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=66], state=TIMED_WAITING

Locked pages = []

Locked pages log: name=exchange-worker-#48%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)

 

 

Thread=[name=query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=7400], state=RUNNABLE

Locked pages = []

Locked pages log: name=query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)

 

 

Thread=[name=sys-stripe-0-#1%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=11], state=WAITING

Locked pages = []

Locked pages log: name=sys-stripe-0-#1%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)

 

 

Thread=[name=sys-stripe-1-#2%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=12], state=WAITING

Locked pages = []

Locked pages log: name=sys-stripe-1-#2%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)

 

 

Thread=[name=sys-stripe-2-#3%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=13], state=WAITING

Locked pages = []

Locked pages log: name=sys-stripe-2-#3%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)

 

 

Thread=[name=sys-stripe-3-#4%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=14], state=WAITING

Locked pages = []

Locked pages log: name=sys-stripe-3-#4%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)

 

 

Thread=[name=sys-stripe-4-#5%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=15], state=WAITING

Locked pages = []

Locked pages log: name=sys-stripe-4-#5%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)

 

 

Thread=[name=sys-stripe-5-#6%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=16], state=WAITING

Locked pages = []

Locked pages log: name=sys-stripe-5-#6%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)

 

 

Thread=[name=sys-stripe-6-#7%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=17], state=WAITING

Locked pages = []

Locked pages log: name=sys-stripe-6-#7%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)

 

 

Thread=[name=sys-stripe-7-#8%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=18], state=WAITING

Locked pages = []

Locked pages log: name=sys-stripe-7-#8%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)

 

 

 

[08:35:22,179][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]

 

 

 

From: Stephen Darlington <[hidden email]>
Sent: Wednesday, March 10, 2021 9:35 AM
To: [hidden email]
Subject: Re: Cluster keeps crashing with SEVERE message

 

This is an EXTERNAL EMAIL. Stop and think before clicking a link or opening attachments.

We’d need to see more of the stack trace and see your configuration before this could be diagnosed.

 

It looks like a node abruptly left the cluster. If you’re losing data it seems like you’ve not configured any backups.



On 10 Mar 2021, at 16:53, Josh Katz <[hidden email]> wrote:

 

After we load 35 million records from the database into the cluster. When we query it using a scan query we get the following SEVERE error,

After that the node are restarting and the caches all show 0 keys and we have to restart the initial load.

Is there a way to salvage the data so we don’t have to reload every time? For example, if we add persistence at this stage the initial load will be slower correct?

How can we avoid seeing this error? “[SEVERE] Failed to process selector key”

 

<image001.png>

 

Thanks,

Josh

 

 


Please follow the hyperlink to important disclosures.
https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html

 

stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Cluster keeps crashing with SEVERE message

More heap-space. See the documentation for memory tuning: memory-tuning

You might also want to check that your query is iterating over the records in your cache rather than trying to read the whole thing into memory.

Regards,
Stephen

On 10 Mar 2021, at 17:44, Josh Katz <[hidden email]> wrote:

Please see the log, it seems to be OutOfMemory from JVM on Windows Server 2016.
How can we address this issue?
 
^-- System thread pool [active=0, idle=7, qSize=0]
[08:35:20,163][INFO][grid-timeout-worker-#22%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][IgniteKernal%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56] FreeList [name=default##FreeList, buckets=256, dataPages=2657, reusePages=0]
[08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheDistributedQueryManager] <NonEquitiesYoY> Failed to run query [qry=GridCacheQueryInfo [loc=false, trans=null, rdc=null, qry=GridCacheQueryAdapter [type=SCAN, clsName=null, clause=null, filter=null, limit=0, transform=null, part=null, incMeta=false, pageSize=1024, timeout=0, incBackups=false, forceLocal=false, dedup=false, prj=null, keepBinary=true, subjId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, taskHash=0, mvccSnapshot=null, dataPageScanEnabled=null], locFut=null, sndId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, reqId=9, incMeta=false, all=false], node=10276c3c-a550-490e-bc48-edf9596dd1dd]
java.lang.OutOfMemoryError: Java heap space
                at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
                at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)
                at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)
                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)
                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)
                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)
[08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheDistributedQueryManager] <NonEquitiesYoY> Failed to run query.
java.lang.OutOfMemoryError: Java heap space
                at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
                at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)
                at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)
                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)
                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)
                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)
[08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheIoManager] Failed processing message [senderId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, msg=GridCacheQueryRequest [id=9, cacheName=NonEquitiesYoY, type=SCAN, fields=false, clause=null, limit=0, clsName=null, keyValFilter=null, rdc=null, trans=null, pageSize=1024, incBackups=false, cancel=false, incMeta=false, all=false, keepBinary=true, subjId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, taskHash=0, part=-1, topVer=AffinityTopologyVersion [topVer=163, minorTopVer=0], mvccSnapshot=null, flags=0, super=GridCacheIdMessage [cacheId=-1867634189, super=GridCacheMessage [msgId=10, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=158, minorTopVer=1], err=null, skipPrepare=false]]]]
java.lang.OutOfMemoryError: Java heap space
                at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
                at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)
                at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)
                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)
                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)
                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)
[08:35:22,163][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]
java.lang.OutOfMemoryError: Java heap space
                at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
                at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)
                at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)
                at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)
                at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)
                at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)
                at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)
                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)
                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)
                at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
                at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
                at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)
[08:35:22,179][WARNING][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][CacheDiagnosticManager] Page locks dump:
 
Thread=[name=exchange-worker-#48%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=66], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=exchange-worker-#48%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
 
 
Thread=[name=query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=7400], state=RUNNABLE
Locked pages = []
Locked pages log: name=query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
 
 
Thread=[name=sys-stripe-0-#1%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=11], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-0-#1%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
 
 
Thread=[name=sys-stripe-1-#2%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=12], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-1-#2%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
 
 
Thread=[name=sys-stripe-2-#3%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=13], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-2-#3%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
 
 
Thread=[name=sys-stripe-3-#4%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=14], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-3-#4%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
 
 
Thread=[name=sys-stripe-4-#5%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=15], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-4-#5%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
 
 
Thread=[name=sys-stripe-5-#6%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=16], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-5-#6%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
 
 
Thread=[name=sys-stripe-6-#7%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=17], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-6-#7%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
 
 
Thread=[name=sys-stripe-7-#8%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=18], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-7-#8%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
 
 
 
[08:35:22,179][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]
 
 
 
From: Stephen Darlington <[hidden email]> 
Sent: Wednesday, March 10, 2021 9:35 AM
To: [hidden email]
Subject: Re: Cluster keeps crashing with SEVERE message
 

This is an EXTERNAL EMAIL. Stop and think before clicking a link or opening attachments.

We’d need to see more of the stack trace and see your configuration before this could be diagnosed.
 
It looks like a node abruptly left the cluster. If you’re losing data it seems like you’ve not configured any backups.


On 10 Mar 2021, at 16:53, Josh Katz <[hidden email]> wrote:
 
After we load 35 million records from the database into the cluster. When we query it using a scan query we get the following SEVERE error,
After that the node are restarting and the caches all show 0 keys and we have to restart the initial load.
Is there a way to salvage the data so we don’t have to reload every time? For example, if we add persistence at this stage the initial load will be slower correct?
How can we avoid seeing this error? “[SEVERE] Failed to process selector key”
 
<image001.png>
 
Thanks,
Josh
 
 

Please follow the hyperlink to important disclosures.
https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html