ignite 2.8.1: Failed to resolve node topology

classic Classic list List threaded Threaded
2 messages Options
Mahesh Renduchintala Mahesh Renduchintala
Reply | Threaded
Open this post in threaded view
|

ignite 2.8.1: Failed to resolve node topology

 Hi, 

We have a crash in our environment with the below error. 
Any insight into what might have gone wrong?

regards
Mahesh


^-- Heap [used=17211MB, free=64.98%, comm=49152MB]
    ^-- Off-heap [used=48448MB, free=26.41%, comm=65736MB]
    ^--   sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
    ^--   default region [used=48447MB, free=26.07%, comm=65536MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.21%, comm=0MB]
    ^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=48647MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=48646MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=4, qSize=0]
    ^-- System thread pool [active=11, idle=12, qSize=0]
[18:44:06,313][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], err=null, rebalanced=true, wasRebalanced=true]
[18:44:06,316][WARNING][sys-stripe-21-#22][finish] Received finish request for completed transaction (the message may be too late) [txId=GridCacheVersion [topVer=205532475, order=1594223109470, nodeOrder=51], dhtTxId=null, node=35c37b55-ec30-4457-b861-403bcfc20c12, commit=false]
[18:44:06,317][WARNING][sys-stripe-18-#19][finish] Received finish request for completed transaction (the message may be too late) [txId=GridCacheVersion [topVer=205532475, order=1594223109273, nodeOrder=51], dhtTxId=null, node=35c37b55-ec30-4457-b861-403bcfc20c12, commit=false]
[18:44:06,317][WARNING][sys-stripe-23-#24][finish] Received finish request for completed transaction (the message may be too late) [txId=GridCacheVersion [topVer=205532475, order=1594223109436, nodeOrder=51], dhtTxId=null, node=35c37b55-ec30-4457-b861-403bcfc20c12, commit=false]
[18:44:06,321][WARNING][sys-stripe-21-#22][finish] Received finish request for completed transaction (the message may be too late) [txId=GridCacheVersion [topVer=205532475, order=1594223109502, nodeOrder=51], dhtTxId=null, node=35c37b55-ec30-4457-b861-403bcfc20c12, commit=false]
[18:44:06,327][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=c19d4735-2b52-487f-9d6d-0574b8f15858, consistentId=L1APISERVICE_7aa60a92371, addrs=ArrayList [10.244.0.93, 127.0.0.1], sockAddrs=HashSet [/10.244.0.93:0, /127.0.0.1:0], discPort=0, order=68, intOrder=39, lastExchangeTime=1594131247817, loc=false, ver=2.8.1#20200521-sha1:86422096, isClient=true], done=true, newCrdFut=null], topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0]]
[18:44:06,327][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture] Exchange timings [startVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], stage="Waiting in exchange queue" (0 ms), stage="Exchange parameters initialization" (0 ms), stage="Determine exchange type" (16 ms), stage="Exchange done" (71053 ms), stage="Total time" (71069 ms)]
[18:44:06,327][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture] Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=83, minorTopVer=0]]
[18:44:06,327][INFO][exchange-worker-#70][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], crd=false]
[18:44:06,345][INFO][db-checkpoint-thread-#112][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=92718d25-2db4-4691-886c-ec26b8b6ecba, startPtr=FileWALPointer [idx=389, fileOff=2151853, len=17162371], checkpointBeforeLockTime=869ms, checkpointLockWait=86163ms, checkpointListenersExecuteTime=784ms, checkpointLockHoldTime=859ms, walCpRecordFsyncDuration=20ms, writeCheckpointEntryDuration=1ms, splitAndSortCpPagesDuration=24ms,  pages=28028, reason='timeout']
[18:44:06,388][SEVERE][sys-stripe-3-#4][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=SQL_PUBLIC_MILLION, topVer=AffinityTopologyVersion [topVer=82, minorTopVer=0], history=[AffinityTopologyVersion [topVer=84, minorTopVer=0], AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion [topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, minorTopVer=0], AffinityTopologyVersion [topVer=88, minorTopVer=0], AffinityTopologyVersion [topVer=89, minorTopVer=0], AffinityTopologyVersion [topVer=90, minorTopVer=0], AffinityTopologyVersion [topVer=91, minorTopVer=0], AffinityTopologyVersion [topVer=92, minorTopVer=0], AffinityTopologyVersion [topVer=93, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0]], locNode=TcpDiscoveryNode [id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=4, intOrder=3, lastExchangeTime=1594054666300, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=SQL_PUBLIC_MILLION, topVer=AffinityTopologyVersion [topVer=82, minorTopVer=0], history=[AffinityTopologyVersion [topVer=84, minorTopVer=0], AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion [topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, minorTopVer=0], AffinityTopologyVersion [topVer=88, minorTopVer=0], AffinityTopologyVersion [topVer=89, minorTopVer=0], AffinityTopologyVersion [topVer=90, minorTopVer=0], AffinityTopologyVersion [topVer=91, minorTopVer=0], AffinityTopologyVersion [topVer=92, minorTopVer=0], AffinityTopologyVersion [topVer=93, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0]], locNode=TcpDiscoveryNode [id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=4, intOrder=3, lastExchangeTime=1594054666300, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.lambda$null$0(IgniteTxHandler.java:645)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[18:44:06,404][SEVERE][sys-stripe-14-#15][GridCacheIoManager] Failed processing message [senderId=35c37b55-ec30-4457-b861-403bcfc20c12, msg=GridNearTxPrepareRequest [futId=62e33fa2371-a9969979-f2fb-410a-8d90-73efb0e13b0c, miniId=1, topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0], subjId=35c37b55-ec30-4457-b861-403bcfc20c12, taskNameHash=0, txLbl=null, flags=, super=GridDistributedTxPrepareRequest [threadId=4722, concurrency=PESSIMISTIC, isolation=REPEATABLE_READ, writeVer=GridCacheVersion [topVer=205532486, order=1594223109704, nodeOrder=51], timeout=0, reads=ArrayList [], writes=ArrayList [IgniteTxEntry [txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl [name=lt_642138, grpName=default-volatile-ds-group], hasValBytes=true], cacheId=1968300681], val=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false, entry=null, prepared=0, locked=false, nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=2, txState=IgniteTxStateImpl [activeCacheIds=[1968300681], recovery=false, mvccEnabled=false, mvccCachingCacheIds=[], txMap=ArrayList [IgniteTxEntry [txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl [name=lt_642138, grpName=default-volatile-ds-group], hasValBytes=true], cacheId=1968300681], val=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false, entry=GridDhtCacheEntry [rdrs=ReaderId[] [], part=72, super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl [name=lt_642138, grpName=default-volatile-ds-group], hasValBytes=true], val=CacheObjectImpl [val=null, hasValBytes=true], ver=GridCacheVersion [topVer=205532475, order=1594223109257, nodeOrder=4], hash=-1064371155, extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc [locs=LinkedList [GridCacheMvccCandidate [nodeId=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, ver=GridCacheVersion [topVer=205532486, order=1594223109711, nodeOrder=4], threadId=4722, id=56494472, topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], reentry=null, otherNodeId=35c37b55-ec30-4457-b861-403bcfc20c12, otherVer=GridCacheVersion [topVer=205532486, order=1594223109704, nodeOrder=51], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl [name=lt_642138, grpName=default-volatile-ds-group], hasValBytes=true], masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0, prevVer=null, nextVer=null]], rmts=null]], flags=2]]], prepared=0, locked=false, nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=205532486, order=1594223109711, nodeOrder=4]]]], flags=onePhase|last|sys, super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=205532486, order=1594223109704, nodeOrder=51], committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage [cacheId=0, super=GridCacheMessage [msgId=11018564, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=81, minorTopVer=44], err=null, skipPrepare=false]]]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=N/A, topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], history=[AffinityTopologyVersion [topVer=84, minorTopVer=0], AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion [topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, minorTopVer=0], AffinityTopologyVersion [topVer=88, minorTopVer=0], AffinityTopologyVersion [topVer=89, minorTopVer=0], AffinityTopologyVersion [topVer=90, minorTopVer=0], AffinityTopologyVersion [topVer=91, minorTopVer=0], AffinityTopologyVersion [topVer=92, minorTopVer=0], AffinityTopologyVersion [topVer=93, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0]], locNode=TcpDiscoveryNode [id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=4, intOrder=3, lastExchangeTime=1594054666300, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.node(GridDiscoveryManager.java:1821)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare(GridDhtTxPrepareFuture.java:1056)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.prepareAsync(GridDhtTxLocal.java:411)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:577)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:183)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[18:44:06,665][WARNING][sys-stripe-3-#4][CacheDiagnosticManager] Page locks dump:

Thread=[name=auth-#69, id=119], state=WAITING
Locked pages = []
Locked pages log: name=auth-#69 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=checkpoint-runner-#113, id=167], state=RUNNABLE
Locked pages = []
Locked pages log: name=checkpoint-runner-#113 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=checkpoint-runner-#114, id=168], state=RUNNABLE
Locked pages = []
Locked pages log: name=checkpoint-runner-#114 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=checkpoint-runner-#115, id=169], state=RUNNABLE
Locked pages = []
Locked pages log: name=checkpoint-runner-#115 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=checkpoint-runner-#116, id=170], state=RUNNABLE
Locked pages = []
Locked pages log: name=checkpoint-runner-#116 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6594, id=6902], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6594 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6595, id=6903], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6595 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6598, id=6906], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6598 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6599, id=6907], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6599 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6602, id=6911], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6602 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6603, id=6912], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6603 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6606, id=6915], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6606 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6607, id=6916], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6607 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=db-checkpoint-thread-#112, id=166], state=WAITING
Locked pages = []
Locked pages log: name=db-checkpoint-thread-#112 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=dms-writer-thread-#76, id=128], state=WAITING
Locked pages = []
Locked pages log: name=dms-writer-thread-#76 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=exchange-worker-#70, id=122], state=RUNNABLE
Locked pages = []
Locked pages log: name=exchange-worker-#70 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=main, id=1], state=WAITING
Locked pages = []
Locked pages log: name=main time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-0-#1, id=20], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-0-#1 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-1-#2, id=21], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-1-#2 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-10-#11, id=30], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-10-#11 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-11-#12, id=31], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-11-#12 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-12-#13, id=32], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-12-#13 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-13-#14, id=33], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-13-#14 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-14-#15, id=34], state=RUNNABLE
Locked pages = []
Locked pages log: name=sys-stripe-14-#15 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-15-#16, id=35], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-15-#16 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-16-#17, id=36], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-16-#17 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-17-#18, id=37], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-17-#18 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-18-#19, id=38], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-18-#19 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-19-#20, id=39], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-19-#20 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-2-#3, id=22], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-2-#3 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-20-#21, id=40], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-20-#21 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-21-#22, id=41], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-21-#22 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-22-#23, id=42], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-22-#23 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-23-#24, id=43], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-23-#24 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-24-#25, id=44], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-24-#25 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-25-#26, id=45], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-25-#26 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-26-#27, id=46], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-26-#27 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-27-#28, id=47], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-27-#28 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-28-#29, id=48], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-28-#29 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-29-#30, id=49], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-29-#30 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-3-#4, id=23], state=RUNNABLE
Locked pages = []
Locked pages log: name=sys-stripe-3-#4 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-30-#31, id=50], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-30-#31 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-31-#32, id=51], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-31-#32 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-4-#5, id=24], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-4-#5 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-5-#6, id=25], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-5-#6 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-6-#7, id=26], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-6-#7 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-7-#8, id=27], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-7-#8 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-8-#9, id=28], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-8-#9 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-9-#10, id=29], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-9-#10 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=ttl-cleanup-worker-#108, id=160], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=ttl-cleanup-worker-#108 time=(1594233846390, 2020-07-08 18:44:06.390)



[18:44:06,666][SEVERE][sys-stripe-3-#4][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=SQL_PUBLIC_MILLION, topVer=AffinityTopologyVersion [topVer=82, minorTopVer=0], history=[AffinityTopologyVersion [topVer=84, minorTopVer=0], AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion [topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, minorTopVer=0], AffinityTopologyVersion [topVer=88, minorTopVer=0], AffinityTopologyVersion [topVer=89, minorTopVer=0], AffinityTopologyVersion [topVer=90, minorTopVer=0], AffinityTopologyVersion [topVer=91, minorTopVer=0], AffinityTopologyVersion [topVer=92, minorTopVer=0], AffinityTopologyVersion [topVer=93, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0]], locNode=TcpDiscoveryNode [id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=4, intOrder=3, lastExchangeTime=1594054666300, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: ignite 2.8.1: Failed to resolve node topology

Hello!

It seems that the node fell behind in switching to new topology. Can you provide full log of this node? I guess that something was blocking PME.

Regards,
--
Ilya Kasnacheev


чт, 9 июл. 2020 г. в 09:44, Mahesh Renduchintala <[hidden email]>:
 Hi, 

We have a crash in our environment with the below error. 
Any insight into what might have gone wrong?

regards
Mahesh


^-- Heap [used=17211MB, free=64.98%, comm=49152MB]
    ^-- Off-heap [used=48448MB, free=26.41%, comm=65736MB]
    ^--   sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
    ^--   default region [used=48447MB, free=26.07%, comm=65536MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.21%, comm=0MB]
    ^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=48647MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=48646MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=4, qSize=0]
    ^-- System thread pool [active=11, idle=12, qSize=0]
[18:44:06,313][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], err=null, rebalanced=true, wasRebalanced=true]
[18:44:06,316][WARNING][sys-stripe-21-#22][finish] Received finish request for completed transaction (the message may be too late) [txId=GridCacheVersion [topVer=205532475, order=1594223109470, nodeOrder=51], dhtTxId=null, node=35c37b55-ec30-4457-b861-403bcfc20c12, commit=false]
[18:44:06,317][WARNING][sys-stripe-18-#19][finish] Received finish request for completed transaction (the message may be too late) [txId=GridCacheVersion [topVer=205532475, order=1594223109273, nodeOrder=51], dhtTxId=null, node=35c37b55-ec30-4457-b861-403bcfc20c12, commit=false]
[18:44:06,317][WARNING][sys-stripe-23-#24][finish] Received finish request for completed transaction (the message may be too late) [txId=GridCacheVersion [topVer=205532475, order=1594223109436, nodeOrder=51], dhtTxId=null, node=35c37b55-ec30-4457-b861-403bcfc20c12, commit=false]
[18:44:06,321][WARNING][sys-stripe-21-#22][finish] Received finish request for completed transaction (the message may be too late) [txId=GridCacheVersion [topVer=205532475, order=1594223109502, nodeOrder=51], dhtTxId=null, node=35c37b55-ec30-4457-b861-403bcfc20c12, commit=false]
[18:44:06,327][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=c19d4735-2b52-487f-9d6d-0574b8f15858, consistentId=L1APISERVICE_7aa60a92371, addrs=ArrayList [10.244.0.93, 127.0.0.1], sockAddrs=HashSet [/10.244.0.93:0, /127.0.0.1:0], discPort=0, order=68, intOrder=39, lastExchangeTime=1594131247817, loc=false, ver=2.8.1#20200521-sha1:86422096, isClient=true], done=true, newCrdFut=null], topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0]]
[18:44:06,327][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture] Exchange timings [startVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], stage="Waiting in exchange queue" (0 ms), stage="Exchange parameters initialization" (0 ms), stage="Determine exchange type" (16 ms), stage="Exchange done" (71053 ms), stage="Total time" (71069 ms)]
[18:44:06,327][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture] Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=83, minorTopVer=0]]
[18:44:06,327][INFO][exchange-worker-#70][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], crd=false]
[18:44:06,345][INFO][db-checkpoint-thread-#112][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=92718d25-2db4-4691-886c-ec26b8b6ecba, startPtr=FileWALPointer [idx=389, fileOff=2151853, len=17162371], checkpointBeforeLockTime=869ms, checkpointLockWait=86163ms, checkpointListenersExecuteTime=784ms, checkpointLockHoldTime=859ms, walCpRecordFsyncDuration=20ms, writeCheckpointEntryDuration=1ms, splitAndSortCpPagesDuration=24ms,  pages=28028, reason='timeout']
[18:44:06,388][SEVERE][sys-stripe-3-#4][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=SQL_PUBLIC_MILLION, topVer=AffinityTopologyVersion [topVer=82, minorTopVer=0], history=[AffinityTopologyVersion [topVer=84, minorTopVer=0], AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion [topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, minorTopVer=0], AffinityTopologyVersion [topVer=88, minorTopVer=0], AffinityTopologyVersion [topVer=89, minorTopVer=0], AffinityTopologyVersion [topVer=90, minorTopVer=0], AffinityTopologyVersion [topVer=91, minorTopVer=0], AffinityTopologyVersion [topVer=92, minorTopVer=0], AffinityTopologyVersion [topVer=93, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0]], locNode=TcpDiscoveryNode [id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=4, intOrder=3, lastExchangeTime=1594054666300, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=SQL_PUBLIC_MILLION, topVer=AffinityTopologyVersion [topVer=82, minorTopVer=0], history=[AffinityTopologyVersion [topVer=84, minorTopVer=0], AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion [topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, minorTopVer=0], AffinityTopologyVersion [topVer=88, minorTopVer=0], AffinityTopologyVersion [topVer=89, minorTopVer=0], AffinityTopologyVersion [topVer=90, minorTopVer=0], AffinityTopologyVersion [topVer=91, minorTopVer=0], AffinityTopologyVersion [topVer=92, minorTopVer=0], AffinityTopologyVersion [topVer=93, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0]], locNode=TcpDiscoveryNode [id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=4, intOrder=3, lastExchangeTime=1594054666300, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.lambda$null$0(IgniteTxHandler.java:645)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[18:44:06,404][SEVERE][sys-stripe-14-#15][GridCacheIoManager] Failed processing message [senderId=35c37b55-ec30-4457-b861-403bcfc20c12, msg=GridNearTxPrepareRequest [futId=62e33fa2371-a9969979-f2fb-410a-8d90-73efb0e13b0c, miniId=1, topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0], subjId=35c37b55-ec30-4457-b861-403bcfc20c12, taskNameHash=0, txLbl=null, flags=, super=GridDistributedTxPrepareRequest [threadId=4722, concurrency=PESSIMISTIC, isolation=REPEATABLE_READ, writeVer=GridCacheVersion [topVer=205532486, order=1594223109704, nodeOrder=51], timeout=0, reads=ArrayList [], writes=ArrayList [IgniteTxEntry [txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl [name=lt_642138, grpName=default-volatile-ds-group], hasValBytes=true], cacheId=1968300681], val=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false, entry=null, prepared=0, locked=false, nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=2, txState=IgniteTxStateImpl [activeCacheIds=[1968300681], recovery=false, mvccEnabled=false, mvccCachingCacheIds=[], txMap=ArrayList [IgniteTxEntry [txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl [name=lt_642138, grpName=default-volatile-ds-group], hasValBytes=true], cacheId=1968300681], val=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false, entry=GridDhtCacheEntry [rdrs=ReaderId[] [], part=72, super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl [name=lt_642138, grpName=default-volatile-ds-group], hasValBytes=true], val=CacheObjectImpl [val=null, hasValBytes=true], ver=GridCacheVersion [topVer=205532475, order=1594223109257, nodeOrder=4], hash=-1064371155, extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc [locs=LinkedList [GridCacheMvccCandidate [nodeId=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, ver=GridCacheVersion [topVer=205532486, order=1594223109711, nodeOrder=4], threadId=4722, id=56494472, topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], reentry=null, otherNodeId=35c37b55-ec30-4457-b861-403bcfc20c12, otherVer=GridCacheVersion [topVer=205532486, order=1594223109704, nodeOrder=51], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl [name=lt_642138, grpName=default-volatile-ds-group], hasValBytes=true], masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0, prevVer=null, nextVer=null]], rmts=null]], flags=2]]], prepared=0, locked=false, nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=205532486, order=1594223109711, nodeOrder=4]]]], flags=onePhase|last|sys, super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=205532486, order=1594223109704, nodeOrder=51], committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage [cacheId=0, super=GridCacheMessage [msgId=11018564, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=81, minorTopVer=44], err=null, skipPrepare=false]]]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=N/A, topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], history=[AffinityTopologyVersion [topVer=84, minorTopVer=0], AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion [topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, minorTopVer=0], AffinityTopologyVersion [topVer=88, minorTopVer=0], AffinityTopologyVersion [topVer=89, minorTopVer=0], AffinityTopologyVersion [topVer=90, minorTopVer=0], AffinityTopologyVersion [topVer=91, minorTopVer=0], AffinityTopologyVersion [topVer=92, minorTopVer=0], AffinityTopologyVersion [topVer=93, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0]], locNode=TcpDiscoveryNode [id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=4, intOrder=3, lastExchangeTime=1594054666300, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.node(GridDiscoveryManager.java:1821)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare(GridDhtTxPrepareFuture.java:1056)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.prepareAsync(GridDhtTxLocal.java:411)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:577)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:183)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[18:44:06,665][WARNING][sys-stripe-3-#4][CacheDiagnosticManager] Page locks dump:

Thread=[name=auth-#69, id=119], state=WAITING
Locked pages = []
Locked pages log: name=auth-#69 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=checkpoint-runner-#113, id=167], state=RUNNABLE
Locked pages = []
Locked pages log: name=checkpoint-runner-#113 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=checkpoint-runner-#114, id=168], state=RUNNABLE
Locked pages = []
Locked pages log: name=checkpoint-runner-#114 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=checkpoint-runner-#115, id=169], state=RUNNABLE
Locked pages = []
Locked pages log: name=checkpoint-runner-#115 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=checkpoint-runner-#116, id=170], state=RUNNABLE
Locked pages = []
Locked pages log: name=checkpoint-runner-#116 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6594, id=6902], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6594 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6595, id=6903], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6595 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6598, id=6906], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6598 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6599, id=6907], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6599 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6602, id=6911], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6602 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6603, id=6912], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6603 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6606, id=6915], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6606 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=client-connector-#6607, id=6916], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6607 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=db-checkpoint-thread-#112, id=166], state=WAITING
Locked pages = []
Locked pages log: name=db-checkpoint-thread-#112 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=dms-writer-thread-#76, id=128], state=WAITING
Locked pages = []
Locked pages log: name=dms-writer-thread-#76 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=exchange-worker-#70, id=122], state=RUNNABLE
Locked pages = []
Locked pages log: name=exchange-worker-#70 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=main, id=1], state=WAITING
Locked pages = []
Locked pages log: name=main time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-0-#1, id=20], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-0-#1 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-1-#2, id=21], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-1-#2 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-10-#11, id=30], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-10-#11 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-11-#12, id=31], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-11-#12 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-12-#13, id=32], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-12-#13 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-13-#14, id=33], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-13-#14 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-14-#15, id=34], state=RUNNABLE
Locked pages = []
Locked pages log: name=sys-stripe-14-#15 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-15-#16, id=35], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-15-#16 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-16-#17, id=36], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-16-#17 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-17-#18, id=37], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-17-#18 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-18-#19, id=38], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-18-#19 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-19-#20, id=39], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-19-#20 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-2-#3, id=22], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-2-#3 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-20-#21, id=40], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-20-#21 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-21-#22, id=41], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-21-#22 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-22-#23, id=42], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-22-#23 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-23-#24, id=43], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-23-#24 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-24-#25, id=44], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-24-#25 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-25-#26, id=45], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-25-#26 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-26-#27, id=46], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-26-#27 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-27-#28, id=47], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-27-#28 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-28-#29, id=48], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-28-#29 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-29-#30, id=49], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-29-#30 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-3-#4, id=23], state=RUNNABLE
Locked pages = []
Locked pages log: name=sys-stripe-3-#4 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-30-#31, id=50], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-30-#31 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-31-#32, id=51], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-31-#32 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-4-#5, id=24], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-4-#5 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-5-#6, id=25], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-5-#6 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-6-#7, id=26], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-6-#7 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-7-#8, id=27], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-7-#8 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-8-#9, id=28], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-8-#9 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=sys-stripe-9-#10, id=29], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-9-#10 time=(1594233846390, 2020-07-08 18:44:06.390)


Thread=[name=ttl-cleanup-worker-#108, id=160], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=ttl-cleanup-worker-#108 time=(1594233846390, 2020-07-08 18:44:06.390)



[18:44:06,666][SEVERE][sys-stripe-3-#4][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=SQL_PUBLIC_MILLION, topVer=AffinityTopologyVersion [topVer=82, minorTopVer=0], history=[AffinityTopologyVersion [topVer=84, minorTopVer=0], AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion [topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, minorTopVer=0], AffinityTopologyVersion [topVer=88, minorTopVer=0], AffinityTopologyVersion [topVer=89, minorTopVer=0], AffinityTopologyVersion [topVer=90, minorTopVer=0], AffinityTopologyVersion [topVer=91, minorTopVer=0], AffinityTopologyVersion [topVer=92, minorTopVer=0], AffinityTopologyVersion [topVer=93, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0]], locNode=TcpDiscoveryNode [id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=4, intOrder=3, lastExchangeTime=1594054666300, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]