Partition eviction failed (current node is stopping)

classic Classic list List threaded Threaded
2 messages Options
Naveen Naveen
Reply | Threaded
Open this post in threaded view

Partition eviction failed (current node is stopping)


I am usng 2.9.0, node is joining the cluster and immediately getting removed
from the cluster
When do we get these errors and how do we resolve them

2020-12-17 16:54:18,948][ERROR][tcp-disco-msg-worker-[70cbe8b9]-#2][G] Blocked system-critical thread has been detected.
This can lead to cluster-wide undefined behaviour
threadName=grid-nio-worker-tcp-comm-1-#25, blockedFor=20s]
[2020-12-17 16:54:18,948][WARN ][tcp-disco-msg-worker-[70cbe8b9]-#2][G] Thread [name="grid-nio-worker-tcp-comm-1-#25",
id=58, state=RUNNABLE, blockCnt=0, waitCnt=0]

[2020-12-17 16:54:18,949][WARN ][tcp-disco-msg-worker-[70cbe8b9]-#2][root] Possible failure suppressed accordingly to a
configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false,
timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-1,
igniteInstanceName=null, finished=false, heartbeatTs=1608209658947]]]

Seeing these messages for all the caches. why does it evict when we have
still enough RAM to load,

[2020-12-17 14:11:29,537][INFO ][sys-#62][PartitionsEvictManager] Eviction
in progress [permits=1, threads=2, groups=6, remainingPartsToEvict=331]

Any pointers would be great help


Sent from:
akurbanov akurbanov
Reply | Threaded
Open this post in threaded view

Re: Partition eviction failed (current node is stopping)

Hello Naveen,

Could you please post the full log?

I suspect that the node where you see this messages was offline for some
time and now full rebalance is currently in progress, which evicts
partitions at the first stage (based on the "remainingPartsToEvict=331"

Best regards,

Sent from: