Failed to Resolve NodeTopology - ignite 2.8.1

classic Classic list List threaded Threaded
4 messages Options
Mahesh Renduchintala Mahesh Renduchintala
Reply | Threaded
Open this post in threaded view
|

Failed to Resolve NodeTopology - ignite 2.8.1

Hi, 

We saw all ignite nodes crash today morning. Below are the error logs?
why would "Failed to Resolve Node Topology" occur?
what would cause this?
If there is a network disturbance, should I not get some sort of segmentation error?




[14:30:10,095][INFO][grid-timeout-worker-#43][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=1ec1e3f9, uptime=1 day, 03:54:03.143]
    ^-- H/N/C [hosts=11, nodes=11, CPUs=46]
    ^-- CPU [cur=0.93%, avg=7.03%, GC=0%]
    ^-- PageMemory [pages=11582153]
    ^-- Heap [used=25392MB, free=48.34%, comm=49152MB]
    ^-- Off-heap [used=45772MB, free=30.47%, comm=65736MB]
    ^--   sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
    ^--   default region [used=45771MB, free=30.16%, comm=65536MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.03%, comm=0MB]
    ^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=53690MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=53689MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=2, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
[14:30:10,976][SEVERE][sys-stripe-17-#18][GridCacheIoManager] Failed processing message [senderId=8f0d3c00-7b18-456c-9066-3852abca7254, msg=GridNearTxPrepareRequest [futId=196a4f48571-8d97cb24-fd81-45b2-a60f-60c26db22c90, miniId=1, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], subjId=8f0d3c00-7b18-456c-9066-3852abca7254, taskNameHash=0, txLbl=null, flags=[firstClientReq][implicitSingle], super=GridDistributedTxPrepareRequest [threadId=2651, concurrency=OPTIMISTIC, isolation=READ_COMMITTED, writeVer=GridCacheVersion [topVer=215702708, order=1604374451397, nodeOrder=21], timeout=420000, reads=null, writes=ArrayList [IgniteTxEntry [txKey=null, val=CacheObjectImpl [val=null, hasValBytes=true][op=CREATE, val=], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false, entry=null, prepared=0, locked=false, nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=2, txState=null, flags=onePhase|last, super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=215702708, order=1604374451397, nodeOrder=21], committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage [cacheId=0, super=GridCacheMessage [msgId=2767960, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], err=null, skipPrepare=false]]]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[14:30:10,976][SEVERE][sys-stripe-17-#18][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[14:30:11,078][WARNING][sys-stripe-17-#18][CacheDiagnosticManager] Page locks dump:

Thread=[name=auth-#69, id=119], state=WAITING
Locked pages = []
Locked pages log: name=auth-#69 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=checkpoint-runner-#113, id=167], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#113 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=checkpoint-runner-#114, id=168], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#114 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=checkpoint-runner-#115, id=169], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#115 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=checkpoint-runner-#116, id=170], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#116 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#1396, id=1594], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#1396 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#1397, id=1595], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#1397 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#239, id=370], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#239 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#240, id=371], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#240 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#241, id=372], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#241 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#242, id=374], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#242 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#435, id=620], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#435 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#437, id=622], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#437 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#439, id=624], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#439 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#440, id=625], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#440 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#441, id=626], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#441 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#442, id=627], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#442 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#443, id=628], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#443 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#444, id=629], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#444 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#445, id=630], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#445 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#446, id=631], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#446 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=db-checkpoint-thread-#112, id=166], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=db-checkpoint-thread-#112 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=dms-writer-thread-#78, id=130], state=WAITING
Locked pages = []
Locked pages log: name=dms-writer-thread-#78 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=exchange-worker-#70, id=122], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=exchange-worker-#70 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=main, id=1], state=WAITING
Locked pages = []
Locked pages log: name=main time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-0-#1, id=20], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-0-#1 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-1-#2, id=21], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-1-#2 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-10-#11, id=30], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-10-#11 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-11-#12, id=31], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-11-#12 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-12-#13, id=32], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-12-#13 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-13-#14, id=33], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-13-#14 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-14-#15, id=34], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-14-#15 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-15-#16, id=35], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-15-#16 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-16-#17, id=36], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-16-#17 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-17-#18, id=37], state=RUNNABLE
Locked pages = []
Locked pages log: name=sys-stripe-17-#18 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-18-#19, id=38], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-18-#19 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-19-#20, id=39], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-19-#20 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-2-#3, id=22], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-2-#3 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-20-#21, id=40], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-20-#21 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-21-#22, id=41], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-21-#22 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-22-#23, id=42], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-22-#23 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-23-#24, id=43], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-23-#24 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-24-#25, id=44], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-24-#25 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-25-#26, id=45], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-25-#26 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-26-#27, id=46], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-26-#27 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-27-#28, id=47], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-27-#28 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-28-#29, id=48], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-28-#29 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-29-#30, id=49], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-29-#30 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-3-#4, id=23], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-3-#4 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-30-#31, id=50], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-30-#31 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-31-#32, id=51], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-31-#32 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-4-#5, id=24], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-4-#5 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-5-#6, id=25], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-5-#6 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-6-#7, id=26], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-6-#7 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-7-#8, id=27], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-7-#8 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-8-#9, id=28], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-8-#9 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-9-#10, id=29], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-9-#10 time=(1604327410980, 2020-11-02 14:30:10.980)



[14:30:11,079][SEVERE][sys-stripe-17-#18][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]


ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Failed to Resolve NodeTopology - ignite 2.8.1

Hello!

You seemed to have a very old transaction which tried to access a topology version which was not in the history. Do you have IGNITE_EXCHANGE_HISTORY_SIZE set?

I think this is a bug that it causes node failure. I would expect that transaction is killed, that's all. Can you please file a ticket about this issue against Apache Ignite JIRA?


Regards,
--
Ilya Kasnacheev


вт, 3 нояб. 2020 г. в 05:59, Mahesh Renduchintala <[hidden email]>:
Hi, 

We saw all ignite nodes crash today morning. Below are the error logs?
why would "Failed to Resolve Node Topology" occur?
what would cause this?
If there is a network disturbance, should I not get some sort of segmentation error?




[14:30:10,095][INFO][grid-timeout-worker-#43][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=1ec1e3f9, uptime=1 day, 03:54:03.143]
    ^-- H/N/C [hosts=11, nodes=11, CPUs=46]
    ^-- CPU [cur=0.93%, avg=7.03%, GC=0%]
    ^-- PageMemory [pages=11582153]
    ^-- Heap [used=25392MB, free=48.34%, comm=49152MB]
    ^-- Off-heap [used=45772MB, free=30.47%, comm=65736MB]
    ^--   sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
    ^--   default region [used=45771MB, free=30.16%, comm=65536MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.03%, comm=0MB]
    ^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=53690MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=53689MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=2, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
[14:30:10,976][SEVERE][sys-stripe-17-#18][GridCacheIoManager] Failed processing message [senderId=8f0d3c00-7b18-456c-9066-3852abca7254, msg=GridNearTxPrepareRequest [futId=196a4f48571-8d97cb24-fd81-45b2-a60f-60c26db22c90, miniId=1, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], subjId=8f0d3c00-7b18-456c-9066-3852abca7254, taskNameHash=0, txLbl=null, flags=[firstClientReq][implicitSingle], super=GridDistributedTxPrepareRequest [threadId=2651, concurrency=OPTIMISTIC, isolation=READ_COMMITTED, writeVer=GridCacheVersion [topVer=215702708, order=1604374451397, nodeOrder=21], timeout=420000, reads=null, writes=ArrayList [IgniteTxEntry [txKey=null, val=CacheObjectImpl [val=null, hasValBytes=true][op=CREATE, val=], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false, entry=null, prepared=0, locked=false, nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=2, txState=null, flags=onePhase|last, super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=215702708, order=1604374451397, nodeOrder=21], committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage [cacheId=0, super=GridCacheMessage [msgId=2767960, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], err=null, skipPrepare=false]]]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[14:30:10,976][SEVERE][sys-stripe-17-#18][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[14:30:11,078][WARNING][sys-stripe-17-#18][CacheDiagnosticManager] Page locks dump:

Thread=[name=auth-#69, id=119], state=WAITING
Locked pages = []
Locked pages log: name=auth-#69 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=checkpoint-runner-#113, id=167], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#113 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=checkpoint-runner-#114, id=168], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#114 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=checkpoint-runner-#115, id=169], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#115 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=checkpoint-runner-#116, id=170], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#116 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#1396, id=1594], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#1396 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#1397, id=1595], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#1397 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#239, id=370], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#239 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#240, id=371], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#240 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#241, id=372], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#241 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#242, id=374], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#242 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#435, id=620], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#435 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#437, id=622], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#437 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#439, id=624], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#439 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#440, id=625], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#440 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#441, id=626], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#441 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#442, id=627], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#442 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#443, id=628], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#443 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#444, id=629], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#444 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#445, id=630], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#445 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#446, id=631], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#446 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=db-checkpoint-thread-#112, id=166], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=db-checkpoint-thread-#112 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=dms-writer-thread-#78, id=130], state=WAITING
Locked pages = []
Locked pages log: name=dms-writer-thread-#78 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=exchange-worker-#70, id=122], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=exchange-worker-#70 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=main, id=1], state=WAITING
Locked pages = []
Locked pages log: name=main time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-0-#1, id=20], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-0-#1 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-1-#2, id=21], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-1-#2 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-10-#11, id=30], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-10-#11 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-11-#12, id=31], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-11-#12 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-12-#13, id=32], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-12-#13 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-13-#14, id=33], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-13-#14 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-14-#15, id=34], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-14-#15 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-15-#16, id=35], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-15-#16 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-16-#17, id=36], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-16-#17 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-17-#18, id=37], state=RUNNABLE
Locked pages = []
Locked pages log: name=sys-stripe-17-#18 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-18-#19, id=38], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-18-#19 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-19-#20, id=39], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-19-#20 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-2-#3, id=22], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-2-#3 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-20-#21, id=40], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-20-#21 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-21-#22, id=41], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-21-#22 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-22-#23, id=42], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-22-#23 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-23-#24, id=43], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-23-#24 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-24-#25, id=44], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-24-#25 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-25-#26, id=45], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-25-#26 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-26-#27, id=46], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-26-#27 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-27-#28, id=47], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-27-#28 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-28-#29, id=48], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-28-#29 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-29-#30, id=49], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-29-#30 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-3-#4, id=23], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-3-#4 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-30-#31, id=50], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-30-#31 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-31-#32, id=51], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-31-#32 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-4-#5, id=24], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-4-#5 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-5-#6, id=25], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-5-#6 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-6-#7, id=26], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-6-#7 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-7-#8, id=27], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-7-#8 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-8-#9, id=28], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-8-#9 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-9-#10, id=29], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-9-#10 time=(1604327410980, 2020-11-02 14:30:10.980)



[14:30:11,079][SEVERE][sys-stripe-17-#18][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]


Mahesh Renduchintala Mahesh Renduchintala
Reply | Threaded
Open this post in threaded view
|

Re: Failed to Resolve NodeTopology - ignite 2.8.1

Yes. I have IGNITE_EXCHANGE_HISTORY_SIZE set to 10. 

should I set IGNITE_EXCHANGE_HISTORY_SIZE = 0 ???

will file a bug shortly




From: Ilya Kasnacheev <[hidden email]>
Sent: Tuesday, November 3, 2020 8:57 PM
To: [hidden email] <[hidden email]>
Subject: Re: Failed to Resolve NodeTopology - ignite 2.8.1
 
Hello!

You seemed to have a very old transaction which tried to access a topology version which was not in the history. Do you have IGNITE_EXCHANGE_HISTORY_SIZE set?

I think this is a bug that it causes node failure. I would expect that transaction is killed, that's all. Can you please file a ticket about this issue against Apache Ignite JIRA?


Regards,
--
Ilya Kasnacheev


вт, 3 нояб. 2020 г. в 05:59, Mahesh Renduchintala <[hidden email]>:
Hi, 

We saw all ignite nodes crash today morning. Below are the error logs?
why would "Failed to Resolve Node Topology" occur?
what would cause this?
If there is a network disturbance, should I not get some sort of segmentation error?




[14:30:10,095][INFO][grid-timeout-worker-#43][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=1ec1e3f9, uptime=1 day, 03:54:03.143]
    ^-- H/N/C [hosts=11, nodes=11, CPUs=46]
    ^-- CPU [cur=0.93%, avg=7.03%, GC=0%]
    ^-- PageMemory [pages=11582153]
    ^-- Heap [used=25392MB, free=48.34%, comm=49152MB]
    ^-- Off-heap [used=45772MB, free=30.47%, comm=65736MB]
    ^--   sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
    ^--   default region [used=45771MB, free=30.16%, comm=65536MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.03%, comm=0MB]
    ^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=53690MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=53689MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=2, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
[14:30:10,976][SEVERE][sys-stripe-17-#18][GridCacheIoManager] Failed processing message [senderId=8f0d3c00-7b18-456c-9066-3852abca7254, msg=GridNearTxPrepareRequest [futId=196a4f48571-8d97cb24-fd81-45b2-a60f-60c26db22c90, miniId=1, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], subjId=8f0d3c00-7b18-456c-9066-3852abca7254, taskNameHash=0, txLbl=null, flags=[firstClientReq][implicitSingle], super=GridDistributedTxPrepareRequest [threadId=2651, concurrency=OPTIMISTIC, isolation=READ_COMMITTED, writeVer=GridCacheVersion [topVer=215702708, order=1604374451397, nodeOrder=21], timeout=420000, reads=null, writes=ArrayList [IgniteTxEntry [txKey=null, val=CacheObjectImpl [val=null, hasValBytes=true][op=CREATE, val=], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false, entry=null, prepared=0, locked=false, nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=2, txState=null, flags=onePhase|last, super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=215702708, order=1604374451397, nodeOrder=21], committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage [cacheId=0, super=GridCacheMessage [msgId=2767960, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], err=null, skipPrepare=false]]]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[14:30:10,976][SEVERE][sys-stripe-17-#18][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[14:30:11,078][WARNING][sys-stripe-17-#18][CacheDiagnosticManager] Page locks dump:

Thread=[name=auth-#69, id=119], state=WAITING
Locked pages = []
Locked pages log: name=auth-#69 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=checkpoint-runner-#113, id=167], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#113 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=checkpoint-runner-#114, id=168], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#114 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=checkpoint-runner-#115, id=169], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#115 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=checkpoint-runner-#116, id=170], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#116 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#1396, id=1594], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#1396 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#1397, id=1595], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#1397 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#239, id=370], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#239 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#240, id=371], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#240 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#241, id=372], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#241 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#242, id=374], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#242 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#435, id=620], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#435 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#437, id=622], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#437 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#439, id=624], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#439 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#440, id=625], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#440 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#441, id=626], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#441 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#442, id=627], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#442 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#443, id=628], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#443 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#444, id=629], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#444 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#445, id=630], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#445 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#446, id=631], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#446 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=db-checkpoint-thread-#112, id=166], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=db-checkpoint-thread-#112 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=dms-writer-thread-#78, id=130], state=WAITING
Locked pages = []
Locked pages log: name=dms-writer-thread-#78 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=exchange-worker-#70, id=122], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=exchange-worker-#70 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=main, id=1], state=WAITING
Locked pages = []
Locked pages log: name=main time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-0-#1, id=20], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-0-#1 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-1-#2, id=21], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-1-#2 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-10-#11, id=30], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-10-#11 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-11-#12, id=31], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-11-#12 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-12-#13, id=32], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-12-#13 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-13-#14, id=33], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-13-#14 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-14-#15, id=34], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-14-#15 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-15-#16, id=35], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-15-#16 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-16-#17, id=36], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-16-#17 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-17-#18, id=37], state=RUNNABLE
Locked pages = []
Locked pages log: name=sys-stripe-17-#18 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-18-#19, id=38], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-18-#19 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-19-#20, id=39], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-19-#20 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-2-#3, id=22], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-2-#3 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-20-#21, id=40], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-20-#21 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-21-#22, id=41], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-21-#22 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-22-#23, id=42], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-22-#23 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-23-#24, id=43], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-23-#24 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-24-#25, id=44], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-24-#25 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-25-#26, id=45], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-25-#26 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-26-#27, id=46], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-26-#27 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-27-#28, id=47], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-27-#28 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-28-#29, id=48], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-28-#29 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-29-#30, id=49], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-29-#30 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-3-#4, id=23], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-3-#4 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-30-#31, id=50], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-30-#31 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-31-#32, id=51], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-31-#32 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-4-#5, id=24], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-4-#5 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-5-#6, id=25], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-5-#6 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-6-#7, id=26], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-6-#7 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-7-#8, id=27], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-7-#8 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-8-#9, id=28], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-8-#9 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-9-#10, id=29], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-9-#10 time=(1604327410980, 2020-11-02 14:30:10.980)



[14:30:11,079][SEVERE][sys-stripe-17-#18][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]


ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Failed to Resolve NodeTopology - ignite 2.8.1

Hello!

This is such a small value for volatile topologies such as yours. I recommend switching to 2.9.0 and then changing this number to 200.

Regards,
--
Ilya Kasnacheev


вт, 3 нояб. 2020 г. в 21:40, Mahesh Renduchintala <[hidden email]>:
Yes. I have IGNITE_EXCHANGE_HISTORY_SIZE set to 10. 

should I set IGNITE_EXCHANGE_HISTORY_SIZE = 0 ???

will file a bug shortly




From: Ilya Kasnacheev <[hidden email]>
Sent: Tuesday, November 3, 2020 8:57 PM
To: [hidden email] <[hidden email]>
Subject: Re: Failed to Resolve NodeTopology - ignite 2.8.1
 
Hello!

You seemed to have a very old transaction which tried to access a topology version which was not in the history. Do you have IGNITE_EXCHANGE_HISTORY_SIZE set?

I think this is a bug that it causes node failure. I would expect that transaction is killed, that's all. Can you please file a ticket about this issue against Apache Ignite JIRA?


Regards,
--
Ilya Kasnacheev


вт, 3 нояб. 2020 г. в 05:59, Mahesh Renduchintala <[hidden email]>:
Hi, 

We saw all ignite nodes crash today morning. Below are the error logs?
why would "Failed to Resolve Node Topology" occur?
what would cause this?
If there is a network disturbance, should I not get some sort of segmentation error?




[14:30:10,095][INFO][grid-timeout-worker-#43][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=1ec1e3f9, uptime=1 day, 03:54:03.143]
    ^-- H/N/C [hosts=11, nodes=11, CPUs=46]
    ^-- CPU [cur=0.93%, avg=7.03%, GC=0%]
    ^-- PageMemory [pages=11582153]
    ^-- Heap [used=25392MB, free=48.34%, comm=49152MB]
    ^-- Off-heap [used=45772MB, free=30.47%, comm=65736MB]
    ^--   sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
    ^--   default region [used=45771MB, free=30.16%, comm=65536MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.03%, comm=0MB]
    ^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=53690MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=53689MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=2, qSize=0]
    ^-- System thread pool [active=0, idle=32, qSize=0]
[14:30:10,976][SEVERE][sys-stripe-17-#18][GridCacheIoManager] Failed processing message [senderId=8f0d3c00-7b18-456c-9066-3852abca7254, msg=GridNearTxPrepareRequest [futId=196a4f48571-8d97cb24-fd81-45b2-a60f-60c26db22c90, miniId=1, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], subjId=8f0d3c00-7b18-456c-9066-3852abca7254, taskNameHash=0, txLbl=null, flags=[firstClientReq][implicitSingle], super=GridDistributedTxPrepareRequest [threadId=2651, concurrency=OPTIMISTIC, isolation=READ_COMMITTED, writeVer=GridCacheVersion [topVer=215702708, order=1604374451397, nodeOrder=21], timeout=420000, reads=null, writes=ArrayList [IgniteTxEntry [txKey=null, val=CacheObjectImpl [val=null, hasValBytes=true][op=CREATE, val=], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false, entry=null, prepared=0, locked=false, nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=2, txState=null, flags=onePhase|last, super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=215702708, order=1604374451397, nodeOrder=21], committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage [cacheId=0, super=GridCacheMessage [msgId=2767960, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], err=null, skipPrepare=false]]]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[14:30:10,976][SEVERE][sys-stripe-17-#18][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[14:30:11,078][WARNING][sys-stripe-17-#18][CacheDiagnosticManager] Page locks dump:

Thread=[name=auth-#69, id=119], state=WAITING
Locked pages = []
Locked pages log: name=auth-#69 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=checkpoint-runner-#113, id=167], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#113 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=checkpoint-runner-#114, id=168], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#114 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=checkpoint-runner-#115, id=169], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#115 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=checkpoint-runner-#116, id=170], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#116 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#1396, id=1594], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#1396 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#1397, id=1595], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#1397 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#239, id=370], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#239 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#240, id=371], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#240 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#241, id=372], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#241 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#242, id=374], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#242 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#435, id=620], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#435 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#437, id=622], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#437 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#439, id=624], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#439 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#440, id=625], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#440 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#441, id=626], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#441 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#442, id=627], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#442 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=client-connector-#443, id=628], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#443 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#444, id=629], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#444 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#445, id=630], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#445 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=client-connector-#446, id=631], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#446 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=db-checkpoint-thread-#112, id=166], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=db-checkpoint-thread-#112 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=dms-writer-thread-#78, id=130], state=WAITING
Locked pages = []
Locked pages log: name=dms-writer-thread-#78 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=exchange-worker-#70, id=122], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=exchange-worker-#70 time=(1604327410981, 2020-11-02 14:30:10.981)


Thread=[name=main, id=1], state=WAITING
Locked pages = []
Locked pages log: name=main time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-0-#1, id=20], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-0-#1 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-1-#2, id=21], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-1-#2 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-10-#11, id=30], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-10-#11 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-11-#12, id=31], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-11-#12 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-12-#13, id=32], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-12-#13 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-13-#14, id=33], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-13-#14 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-14-#15, id=34], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-14-#15 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-15-#16, id=35], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-15-#16 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-16-#17, id=36], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-16-#17 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-17-#18, id=37], state=RUNNABLE
Locked pages = []
Locked pages log: name=sys-stripe-17-#18 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-18-#19, id=38], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-18-#19 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-19-#20, id=39], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-19-#20 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-2-#3, id=22], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-2-#3 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-20-#21, id=40], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-20-#21 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-21-#22, id=41], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-21-#22 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-22-#23, id=42], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-22-#23 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-23-#24, id=43], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-23-#24 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-24-#25, id=44], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-24-#25 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-25-#26, id=45], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-25-#26 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-26-#27, id=46], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-26-#27 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-27-#28, id=47], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-27-#28 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-28-#29, id=48], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-28-#29 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-29-#30, id=49], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-29-#30 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-3-#4, id=23], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-3-#4 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-30-#31, id=50], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-30-#31 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-31-#32, id=51], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-31-#32 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-4-#5, id=24], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-4-#5 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-5-#6, id=25], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-5-#6 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-6-#7, id=26], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-6-#7 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-7-#8, id=27], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-7-#8 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-8-#9, id=28], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-8-#9 time=(1604327410980, 2020-11-02 14:30:10.980)


Thread=[name=sys-stripe-9-#10, id=29], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-9-#10 time=(1604327410980, 2020-11-02 14:30:10.980)



[14:30:11,079][SEVERE][sys-stripe-17-#18][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]