Ignite yarn job's resources (number of executors and cores) increase automatically after running for 7 days

classic Classic list List threaded Threaded
2 messages Options
ChandanS ChandanS
Reply | Threaded
Open this post in threaded view
|

Ignite yarn job's resources (number of executors and cores) increase automatically after running for 7 days

Hi,

I had submitted a spark job for ignite-yarn application on last 12th March,
2020 with 40 executors and 2 cores (running containers 41 and allocated CPU
81 on spark web UI page) and after running it for 7 days, Today on 19th
March I have seen on spark web UI page that resources were went up by
"running containers 247 and allocated CPU 493" and it's keep on increasing
every one hour and so. How can I maintain the ignite-yarn application not to
increase the resources of its own? Otherwise, it will a concern for the
cluster if it keep on increasing. Below is the part of yarn log for my
ignite-yarn job.

log4j:WARN No appenders could be found for logger
(org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
[08:58:25,564][SEVERE][grid-timeout-worker-#103%titan-resourceUP%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=tcp-comm-worker,
blockedFor=17s]
[08:58:25,565][SEVERE][grid-timeout-worker-#103%titan-resourceUP%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]]]
class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
        at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
        at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
        at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:58:34,698][SEVERE][grid-timeout-worker-#103%titan-resourceUP%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=tcp-comm-worker,
blockedFor=26s]
[08:58:34,699][SEVERE][grid-timeout-worker-#103%titan-resourceUP%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]]]
class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
        at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
        at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
        at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:58:43,805][SEVERE][nio-acceptor-tcp-comm-#128%titan-resourceUP%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=tcp-comm-worker,
blockedFor=35s]
[08:58:43,805][SEVERE][nio-acceptor-tcp-comm-#128%titan-resourceUP%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]]]
class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
        at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
        at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2938)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:58:56,406][SEVERE][ttl-cleanup-worker-#171%titan-resourceUP%][G] Blocked
system-critical thread has been detected. This can lead to cluster-wide
undefined behaviour [threadName=tcp-comm-worker, blockedFor=47s]
[08:58:56,407][SEVERE][ttl-cleanup-worker-#171%titan-resourceUP%][] Critical
system error detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]]]
class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
        at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
        at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
        at
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:151)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:05,506][SEVERE][nio-acceptor-tcp-comm-#128%titan-resourceUP%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=tcp-comm-worker,
blockedFor=56s]
[08:59:05,506][SEVERE][nio-acceptor-tcp-comm-#128%titan-resourceUP%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]]]
class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
        at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
        at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2938)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,591][SEVERE][grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=9,
bytesRcvd=4475652, bytesSent=1731122, bytesRcvd0=18, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-9,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549587, hashCode=344426864, interrupted=false,
runner=grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=166400 cap=166400],
readBuf=java.nio.DirectByteBuffer[pos=18 lim=18 cap=117552],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/10.204.138.138:39970,
rmtAddr=c250xmv.intqa.bigdata.int.thomsonreuters.com/10.204.138.66:45400,
createTime=1584003549587, closeTime=0, bytesSent=0, bytesRcvd=18,
bytesSent0=0, bytesRcvd0=18, sndSchedTime=1584003549587,
lastSndTime=1584003549587, lastRcvTime=1584003549587, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false, markedForClose=false]]]
java.lang.NullPointerException
        at
org.apache.ignite.internal.util.nio.GridNioServer.cancelConnect(GridNioServer.java:898)
        at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.cancel(TcpCommunicationConnectionCheckFuture.java:338)
        at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture.cancelFutures(TcpCommunicationConnectionCheckFuture.java:475)
        at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture.receivedAddressStatus(TcpCommunicationConnectionCheckFuture.java:494)
        at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture$1.onStatusReceived(TcpCommunicationConnectionCheckFuture.java:433)
        at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.finish(TcpCommunicationConnectionCheckFuture.java:362)
        at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.onConnected(TcpCommunicationConnectionCheckFuture.java:348)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:779)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:389)
        at
org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
        at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
        at
org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:117)
        at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
        at
org.apache.ignite.internal.util.nio.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:88)
        at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
        at
org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3553)
        at
org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1303)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,591][SEVERE][grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception.
class org.apache.ignite.internal.util.nio.GridNioException: null
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2415)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
        at
org.apache.ignite.internal.util.nio.GridNioServer.cancelConnect(GridNioServer.java:898)
        at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.cancel(TcpCommunicationConnectionCheckFuture.java:338)
        at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture.cancelFutures(TcpCommunicationConnectionCheckFuture.java:475)
        at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture.receivedAddressStatus(TcpCommunicationConnectionCheckFuture.java:494)
        at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture$1.onStatusReceived(TcpCommunicationConnectionCheckFuture.java:433)
        at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.finish(TcpCommunicationConnectionCheckFuture.java:362)
        at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.onConnected(TcpCommunicationConnectionCheckFuture.java:348)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:779)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:389)
        at
org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
        at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
        at
org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:117)
        at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
        at
org.apache.ignite.internal.util.nio.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:88)
        at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
        at
org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3553)
        at
org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1303)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        ... 4 more
[08:59:10,593][SEVERE][grid-nio-worker-tcp-comm-19-#123%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=19,
bytesRcvd=4112614, bytesSent=5001528, bytesRcvd0=0, bytesSent0=18,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-19,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549587, hashCode=1777632028, interrupted=false,
runner=grid-nio-worker-tcp-comm-19-#123%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40456,
createTime=1584003549587, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549587,
lastSndTime=1584003549587, lastRcvTime=1584003549587, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,597][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=2,
bytesRcvd=6049119, bytesSent=6778991, bytesRcvd0=18, bytesSent0=36,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-2,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549587, hashCode=813216397, interrupted=false,
runner=grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40512,
createTime=1584003549587, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549587,
lastSndTime=1584003549587, lastRcvTime=1584003549587, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,597][SEVERE][grid-nio-worker-tcp-comm-7-#111%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,599][SEVERE][grid-nio-worker-tcp-comm-6-#110%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=6,
bytesRcvd=5589344, bytesSent=3069900, bytesRcvd0=54, bytesSent0=18,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-6,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549597, hashCode=293379764, interrupted=false,
runner=grid-nio-worker-tcp-comm-6-#110%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40540,
createTime=1584003549597, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549597,
lastSndTime=1584003549597, lastRcvTime=1584003549597, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-21-#125%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-3-#107%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-18-#122%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=18,
bytesRcvd=3987619, bytesSent=1696258, bytesRcvd0=36, bytesSent0=54,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-18,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549597, hashCode=1083458718, interrupted=false,
runner=grid-nio-worker-tcp-comm-18-#122%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40552,
createTime=1584003549597, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549597,
lastSndTime=1584003549597, lastRcvTime=1584003549597, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-11-#115%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-13-#117%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,601][SEVERE][grid-nio-worker-tcp-comm-15-#119%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,601][SEVERE][grid-nio-worker-tcp-comm-17-#121%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,601][SEVERE][grid-nio-worker-tcp-comm-23-#127%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,602][SEVERE][grid-nio-worker-tcp-comm-1-#105%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,602][SEVERE][grid-nio-worker-tcp-comm-4-#108%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,602][SEVERE][grid-nio-worker-tcp-comm-8-#112%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,603][SEVERE][grid-nio-worker-tcp-comm-20-#124%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,608][SEVERE][grid-nio-worker-tcp-comm-0-#104%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:10,609][SEVERE][grid-nio-worker-tcp-comm-10-#114%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,592][SEVERE][grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,595][SEVERE][grid-nio-worker-tcp-comm-19-#123%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,598][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,598][SEVERE][grid-nio-worker-tcp-comm-7-#111%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,601][SEVERE][grid-nio-worker-tcp-comm-14-#118%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=14,
bytesRcvd=3350068, bytesSent=8196, bytesRcvd0=72, bytesSent0=126,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-14,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549617, hashCode=907545167, interrupted=false,
runner=grid-nio-worker-tcp-comm-14-#118%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40716,
createTime=1584003549607, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549607,
lastSndTime=1584003549607, lastRcvTime=1584003549607, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,601][SEVERE][grid-nio-worker-tcp-comm-12-#116%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=12,
bytesRcvd=3969821, bytesSent=3288859, bytesRcvd0=108, bytesSent0=108,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-12,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549617, hashCode=1020912454, interrupted=false,
runner=grid-nio-worker-tcp-comm-12-#116%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40748,
createTime=1584003549607, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549607,
lastSndTime=1584003549607, lastRcvTime=1584003549607, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,602][SEVERE][grid-nio-worker-tcp-comm-17-#121%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,602][SEVERE][grid-nio-worker-tcp-comm-11-#115%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,603][SEVERE][grid-nio-worker-tcp-comm-8-#112%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,604][SEVERE][grid-nio-worker-tcp-comm-5-#109%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=5,
bytesRcvd=4011044, bytesSent=7399489, bytesRcvd0=72, bytesSent0=108,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-5,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549607, hashCode=606372690, interrupted=false,
runner=grid-nio-worker-tcp-comm-5-#109%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40648,
createTime=1584003549597, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549597,
lastSndTime=1584003549597, lastRcvTime=1584003549597, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,609][SEVERE][grid-nio-worker-tcp-comm-1-#105%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=1,
bytesRcvd=2726130, bytesSent=6003920, bytesRcvd0=90, bytesSent0=72,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003550603, hashCode=1883663272, interrupted=false,
runner=grid-nio-worker-tcp-comm-1-#105%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40664,
createTime=1584003550593, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003550593,
lastSndTime=1584003550593, lastRcvTime=1584003550593, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,609][SEVERE][grid-nio-worker-tcp-comm-0-#104%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0,
bytesRcvd=4890192, bytesSent=12608476, bytesRcvd0=54, bytesSent0=595,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003550603, hashCode=1994247289, interrupted=false,
runner=grid-nio-worker-tcp-comm-0-#104%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40640,
createTime=1584003549597, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549597,
lastSndTime=1584003549597, lastRcvTime=1584003549597, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:11,610][SEVERE][grid-nio-worker-tcp-comm-10-#114%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=10,
bytesRcvd=4013857, bytesSent=3362419, bytesRcvd0=90, bytesSent0=72,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-10,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003550603, hashCode=1302629280, interrupted=false,
runner=grid-nio-worker-tcp-comm-10-#114%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40496,
createTime=1584003549587, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549587,
lastSndTime=1584003549587, lastRcvTime=1584003549587, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:12,593][SEVERE][grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:12,598][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:12,599][SEVERE][grid-nio-worker-tcp-comm-7-#111%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:12,602][SEVERE][grid-nio-worker-tcp-comm-15-#119%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=15,
bytesRcvd=5571714, bytesSent=1717776, bytesRcvd0=0, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-15,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003550623, hashCode=1293929915, interrupted=false,
runner=grid-nio-worker-tcp-comm-15-#119%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40684,
createTime=1584003550593, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1584003550593,
lastSndTime=1584003550593, lastRcvTime=1584003550593, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:12,602][SEVERE][grid-nio-worker-tcp-comm-18-#122%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=18,
bytesRcvd=3987655, bytesSent=1696348, bytesRcvd0=0, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-18,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003551589, hashCode=1083458718, interrupted=false,
runner=grid-nio-worker-tcp-comm-18-#122%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40720,
createTime=1584003550593, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1584003550593,
lastSndTime=1584003550593, lastRcvTime=1584003550593, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:13,598][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
[08:59:41,932][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi]
Failed to read data from remote connection (will wait for 2000ms).
class org.apache.ignite.IgniteCheckedException: Failed to select events on
selector.
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2185)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
        at
java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:197)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2000)
        ... 3 more
[13:49:50,352][SEVERE][grid-nio-worker-tcp-comm-0-#104%titan-resourceUP%][TcpCommunicationSpi]
Failed to read data from remote connection (will wait for 2000ms).
class org.apache.ignite.IgniteCheckedException: Failed to select events on
selector.
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2185)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
        at
java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:197)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2000)
        ... 3 more
[14:45:35,550][SEVERE][grid-nio-worker-tcp-comm-1-#105%titan-resourceUP%][TcpCommunicationSpi]
Failed to read data from remote connection (will wait for 2000ms).
class org.apache.ignite.IgniteCheckedException: Failed to select events on
selector.
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2185)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
        at
java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:197)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2000)
        ... 3 more

LogType:stdout
Log Upload Time:Sat Mar 14 08:29:54 +0000 2020
LogLength:19390
Log Contents:
[08:57:21]    __________  ________________
[08:57:21]   /  _/ ___/ |/ /  _/_  __/ __/
[08:57:21]  _/ // (7 7    // /  / / / _/  
[08:57:21] /___/\___/_/|_/___/ /_/ /___/  
[08:57:21]
[08:57:21] ver. 2.7.0#20181130-sha1:256ae401
[08:57:21] 2018 Copyright(C) Apache Software Foundation
[08:57:21]
[08:57:21] Ignite documentation: http://ignite.apache.org
[08:57:21]
[08:57:21] Quiet mode.
[08:57:21]   ^-- Logging to file
'/data/12/yarn/nm/usercache/bigdata-app-ecpdevbermuda-uncomparator/appcache/application_1564355610025_561993/container_e192_1564355610025_561993_01_000044/ignite/apache-ignite-2.7.0-bin/work/log/ignite-2ee49457.0.log'
[08:57:21]   ^-- Logging by 'JavaLogger [quiet=true, config=null]'
[08:57:21]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[08:57:21]
[08:57:21] OS: Linux 3.10.0-1062.9.1.el7.x86_64 amd64
[08:57:21] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13
[08:57:21] Please set system property '-Djava.net.preferIPv4Stack=true' to
avoid possible problems in mixed environments.
[08:57:21] Configured plugins:
[08:57:21]   ^-- None
[08:57:21]
[08:57:21] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]]]
[08:57:21] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
[08:57:22] Security status [authentication=off, tls/ssl=off]
[08:57:22] Topology snapshot [ver=12, locNode=2ee49457, servers=12,
clients=0, state=ACTIVE, CPUs=576, offheap=910.0GB, heap=180.0GB]
[08:57:22] Topology snapshot [ver=13, locNode=2ee49457, servers=13,
clients=0, state=ACTIVE, CPUs=624, offheap=980.0GB, heap=190.0GB]
[08:57:22] Topology snapshot [ver=14, locNode=2ee49457, servers=14,
clients=0, state=ACTIVE, CPUs=672, offheap=1100.0GB, heap=210.0GB]
[08:57:23] Performance suggestions for grid 'titan-resourceUP' (fix if
possible)
[08:57:23] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[08:57:23]   ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM
options)
[08:57:23]   ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[08:57:23]   ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
[08:57:23]   ^-- Speed up flushing of dirty pages by OS (alter
vm.dirty_expire_centisecs parameter by setting to 500)
[08:57:23] Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[08:57:23]
[08:57:23] To start Console Management & Monitoring run
ignitevisorcmd.{sh|bat}
[08:57:23] Data Regions Configured:
[08:57:23]   ^-- default [initSize=256.0 MiB, maxSize=75.5 GiB,
persistence=false]
[08:57:23]
[08:57:23] Ignite node started OK (id=2ee49457, instance
name=titan-resourceUP)
[08:57:23] Topology snapshot [ver=11, locNode=2ee49457, servers=11,
clients=0, state=ACTIVE, CPUs=528, offheap=830.0GB, heap=160.0GB]
[08:57:23] Topology snapshot [ver=15, locNode=2ee49457, servers=15,
clients=0, state=ACTIVE, CPUs=712, offheap=1100.0GB, heap=220.0GB]
[08:57:23] Topology snapshot [ver=16, locNode=2ee49457, servers=16,
clients=0, state=ACTIVE, CPUs=752, offheap=1200.0GB, heap=230.0GB]
[08:57:23] Topology snapshot [ver=17, locNode=2ee49457, servers=17,
clients=0, state=ACTIVE, CPUs=792, offheap=1200.0GB, heap=250.0GB]
[08:57:23] Topology snapshot [ver=18, locNode=2ee49457, servers=18,
clients=0, state=ACTIVE, CPUs=832, offheap=1300.0GB, heap=260.0GB]
[08:57:23] Topology snapshot [ver=19, locNode=2ee49457, servers=19,
clients=0, state=ACTIVE, CPUs=872, offheap=1300.0GB, heap=280.0GB]
[08:57:23] Topology snapshot [ver=20, locNode=2ee49457, servers=20,
clients=0, state=ACTIVE, CPUs=912, offheap=1400.0GB, heap=290.0GB]
[08:57:23] Topology snapshot [ver=21, locNode=2ee49457, servers=21,
clients=0, state=ACTIVE, CPUs=952, offheap=1400.0GB, heap=310.0GB]
[08:57:23] Topology snapshot [ver=22, locNode=2ee49457, servers=22,
clients=0, state=ACTIVE, CPUs=992, offheap=1500.0GB, heap=320.0GB]
[08:57:23] Topology snapshot [ver=23, locNode=2ee49457, servers=23,
clients=0, state=ACTIVE, CPUs=1032, offheap=1500.0GB, heap=340.0GB]
[08:57:23] Topology snapshot [ver=24, locNode=2ee49457, servers=24,
clients=0, state=ACTIVE, CPUs=1072, offheap=1600.0GB, heap=350.0GB]
[08:57:23] Topology snapshot [ver=25, locNode=2ee49457, servers=25,
clients=0, state=ACTIVE, CPUs=1112, offheap=1600.0GB, heap=370.0GB]
[08:57:24] Topology snapshot [ver=26, locNode=2ee49457, servers=26,
clients=0, state=ACTIVE, CPUs=1152, offheap=1700.0GB, heap=380.0GB]
[08:57:29] Topology snapshot [ver=27, locNode=2ee49457, servers=27,
clients=0, state=ACTIVE, CPUs=1200, offheap=1700.0GB, heap=400.0GB]
[08:57:30] Topology snapshot [ver=28, locNode=2ee49457, servers=28,
clients=0, state=ACTIVE, CPUs=1248, offheap=1800.0GB, heap=410.0GB]
[08:57:31] Topology snapshot [ver=29, locNode=2ee49457, servers=29,
clients=0, state=ACTIVE, CPUs=1296, offheap=1900.0GB, heap=420.0GB]
[08:57:31] Topology snapshot [ver=30, locNode=2ee49457, servers=29,
clients=1, state=ACTIVE, CPUs=1296, offheap=1900.0GB, heap=460.0GB]
[08:57:36] Topology snapshot [ver=31, locNode=2ee49457, servers=29,
clients=2, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=480.0GB]
[08:57:37] Topology snapshot [ver=32, locNode=2ee49457, servers=29,
clients=3, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=490.0GB]
[08:57:37] Topology snapshot [ver=33, locNode=2ee49457, servers=29,
clients=4, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=510.0GB]
[08:57:37] Topology snapshot [ver=34, locNode=2ee49457, servers=29,
clients=5, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=520.0GB]
[08:57:37] Topology snapshot [ver=35, locNode=2ee49457, servers=29,
clients=6, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=540.0GB]
[08:57:37] Topology snapshot [ver=36, locNode=2ee49457, servers=29,
clients=7, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=550.0GB]
[08:57:37] Topology snapshot [ver=37, locNode=2ee49457, servers=29,
clients=8, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=570.0GB]
[08:57:37] Topology snapshot [ver=38, locNode=2ee49457, servers=29,
clients=9, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=580.0GB]
[08:57:37] Topology snapshot [ver=39, locNode=2ee49457, servers=29,
clients=10, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=600.0GB]
[08:57:37] Topology snapshot [ver=40, locNode=2ee49457, servers=29,
clients=11, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=610.0GB]
[08:57:37] Topology snapshot [ver=41, locNode=2ee49457, servers=29,
clients=12, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=630.0GB]
[08:57:37] Topology snapshot [ver=42, locNode=2ee49457, servers=29,
clients=13, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=640.0GB]
[08:57:37] Topology snapshot [ver=43, locNode=2ee49457, servers=29,
clients=14, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=660.0GB]
[08:57:37] Topology snapshot [ver=44, locNode=2ee49457, servers=29,
clients=15, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=670.0GB]
[08:57:37] Topology snapshot [ver=45, locNode=2ee49457, servers=29,
clients=16, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=690.0GB]
[08:57:37] Topology snapshot [ver=46, locNode=2ee49457, servers=29,
clients=17, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=700.0GB]
[08:57:37] Topology snapshot [ver=47, locNode=2ee49457, servers=29,
clients=18, state=ACTIVE, CPUs=1384, offheap=1900.0GB, heap=720.0GB]
[08:57:37] Topology snapshot [ver=48, locNode=2ee49457, servers=29,
clients=19, state=ACTIVE, CPUs=1384, offheap=1900.0GB, heap=730.0GB]
[08:57:37] Topology snapshot [ver=49, locNode=2ee49457, servers=29,
clients=20, state=ACTIVE, CPUs=1424, offheap=1900.0GB, heap=750.0GB]
[08:57:37] Topology snapshot [ver=50, locNode=2ee49457, servers=29,
clients=21, state=ACTIVE, CPUs=1424, offheap=1900.0GB, heap=760.0GB]
[08:57:37] Topology snapshot [ver=51, locNode=2ee49457, servers=29,
clients=22, state=ACTIVE, CPUs=1464, offheap=1900.0GB, heap=780.0GB]
[08:57:37] Topology snapshot [ver=52, locNode=2ee49457, servers=29,
clients=23, state=ACTIVE, CPUs=1464, offheap=1900.0GB, heap=790.0GB]
[08:57:37] Topology snapshot [ver=53, locNode=2ee49457, servers=29,
clients=24, state=ACTIVE, CPUs=1504, offheap=1900.0GB, heap=810.0GB]
[08:57:37] Topology snapshot [ver=54, locNode=2ee49457, servers=29,
clients=25, state=ACTIVE, CPUs=1504, offheap=1900.0GB, heap=820.0GB]
[08:57:37] Topology snapshot [ver=55, locNode=2ee49457, servers=29,
clients=26, state=ACTIVE, CPUs=1544, offheap=1900.0GB, heap=840.0GB]
[08:57:37] Topology snapshot [ver=56, locNode=2ee49457, servers=29,
clients=27, state=ACTIVE, CPUs=1584, offheap=1900.0GB, heap=850.0GB]
[08:57:37] Topology snapshot [ver=57, locNode=2ee49457, servers=29,
clients=28, state=ACTIVE, CPUs=1584, offheap=1900.0GB, heap=870.0GB]
[08:57:38] Topology snapshot [ver=58, locNode=2ee49457, servers=29,
clients=29, state=ACTIVE, CPUs=1584, offheap=1900.0GB, heap=880.0GB]
[08:57:38] Topology snapshot [ver=59, locNode=2ee49457, servers=29,
clients=30, state=ACTIVE, CPUs=1584, offheap=1900.0GB, heap=900.0GB]
[08:57:38] Topology snapshot [ver=60, locNode=2ee49457, servers=29,
clients=31, state=ACTIVE, CPUs=1616, offheap=1900.0GB, heap=910.0GB]
[08:57:38] Topology snapshot [ver=61, locNode=2ee49457, servers=29,
clients=32, state=ACTIVE, CPUs=1656, offheap=1900.0GB, heap=930.0GB]
[08:57:38] Topology snapshot [ver=62, locNode=2ee49457, servers=29,
clients=33, state=ACTIVE, CPUs=1696, offheap=1900.0GB, heap=940.0GB]
[08:57:38] Topology snapshot [ver=63, locNode=2ee49457, servers=29,
clients=34, state=ACTIVE, CPUs=1696, offheap=1900.0GB, heap=960.0GB]
[08:57:38] Topology snapshot [ver=64, locNode=2ee49457, servers=29,
clients=35, state=ACTIVE, CPUs=1696, offheap=1900.0GB, heap=970.0GB]
[08:57:38] Topology snapshot [ver=65, locNode=2ee49457, servers=29,
clients=36, state=ACTIVE, CPUs=1696, offheap=1900.0GB, heap=990.0GB]
[08:57:38] Topology snapshot [ver=66, locNode=2ee49457, servers=29,
clients=37, state=ACTIVE, CPUs=1736, offheap=1900.0GB, heap=1000.0GB]
[08:57:38] Topology snapshot [ver=67, locNode=2ee49457, servers=29,
clients=38, state=ACTIVE, CPUs=1736, offheap=1900.0GB, heap=1000.0GB]
[08:57:38] Topology snapshot [ver=68, locNode=2ee49457, servers=29,
clients=39, state=ACTIVE, CPUs=1776, offheap=1900.0GB, heap=1000.0GB]
[08:57:38] Topology snapshot [ver=69, locNode=2ee49457, servers=29,
clients=40, state=ACTIVE, CPUs=1808, offheap=1900.0GB, heap=1000.0GB]
[08:57:38] Topology snapshot [ver=70, locNode=2ee49457, servers=29,
clients=41, state=ACTIVE, CPUs=1840, offheap=1900.0GB, heap=1100.0GB]
[08:58:01] Topology snapshot [ver=71, locNode=2ee49457, servers=30,
clients=41, state=ACTIVE, CPUs=1840, offheap=1900.0GB, heap=1100.0GB]
[08:58:02] Topology snapshot [ver=72, locNode=2ee49457, servers=31,
clients=41, state=ACTIVE, CPUs=1840, offheap=2000.0GB, heap=1100.0GB]
[08:58:02] Topology snapshot [ver=73, locNode=2ee49457, servers=32,
clients=41, state=ACTIVE, CPUs=1840, offheap=2000.0GB, heap=1100.0GB]
[08:58:02] Topology snapshot [ver=74, locNode=2ee49457, servers=33,
clients=41, state=ACTIVE, CPUs=1840, offheap=2100.0GB, heap=1100.0GB]
[08:58:02] Topology snapshot [ver=75, locNode=2ee49457, servers=34,
clients=41, state=ACTIVE, CPUs=1840, offheap=2100.0GB, heap=1100.0GB]
[08:58:02] Topology snapshot [ver=76, locNode=2ee49457, servers=35,
clients=41, state=ACTIVE, CPUs=1880, offheap=2200.0GB, heap=1100.0GB]
[08:58:02] Topology snapshot [ver=77, locNode=2ee49457, servers=36,
clients=41, state=ACTIVE, CPUs=1880, offheap=2200.0GB, heap=1200.0GB]
[08:58:02] Topology snapshot [ver=78, locNode=2ee49457, servers=37,
clients=41, state=ACTIVE, CPUs=1920, offheap=2300.0GB, heap=1200.0GB]
[08:58:03] Topology snapshot [ver=79, locNode=2ee49457, servers=38,
clients=41, state=ACTIVE, CPUs=1960, offheap=2300.0GB, heap=1200.0GB]
[08:58:05] Topology snapshot [ver=80, locNode=2ee49457, servers=39,
clients=41, state=ACTIVE, CPUs=2000, offheap=2400.0GB, heap=1200.0GB]
[08:59:10] Topology snapshot [ver=81, locNode=2ee49457, servers=38,
clients=41, state=ACTIVE, CPUs=1960, offheap=2300.0GB, heap=1200.0GB]
[08:59:16] Topology snapshot [ver=82, locNode=2ee49457, servers=38,
clients=40, state=ACTIVE, CPUs=1960, offheap=2300.0GB, heap=1200.0GB]
[08:59:17] Topology snapshot [ver=83, locNode=2ee49457, servers=38,
clients=39, state=ACTIVE, CPUs=1960, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=84, locNode=2ee49457, servers=38,
clients=38, state=ACTIVE, CPUs=1912, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=85, locNode=2ee49457, servers=38,
clients=37, state=ACTIVE, CPUs=1912, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=86, locNode=2ee49457, servers=38,
clients=36, state=ACTIVE, CPUs=1912, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=87, locNode=2ee49457, servers=38,
clients=35, state=ACTIVE, CPUs=1872, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=88, locNode=2ee49457, servers=38,
clients=34, state=ACTIVE, CPUs=1832, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=89, locNode=2ee49457, servers=38,
clients=33, state=ACTIVE, CPUs=1832, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=90, locNode=2ee49457, servers=38,
clients=32, state=ACTIVE, CPUs=1800, offheap=2300.0GB, heap=1000.0GB]
[08:59:17] Topology snapshot [ver=91, locNode=2ee49457, servers=38,
clients=31, state=ACTIVE, CPUs=1800, offheap=2300.0GB, heap=1000.0GB]
[08:59:17] Topology snapshot [ver=92, locNode=2ee49457, servers=38,
clients=30, state=ACTIVE, CPUs=1800, offheap=2300.0GB, heap=1000.0GB]
[08:59:17] Topology snapshot [ver=93, locNode=2ee49457, servers=38,
clients=29, state=ACTIVE, CPUs=1800, offheap=2300.0GB, heap=990.0GB]
[08:59:17] Topology snapshot [ver=94, locNode=2ee49457, servers=38,
clients=28, state=ACTIVE, CPUs=1760, offheap=2300.0GB, heap=980.0GB]
[08:59:17] Topology snapshot [ver=95, locNode=2ee49457, servers=38,
clients=27, state=ACTIVE, CPUs=1760, offheap=2300.0GB, heap=960.0GB]
[08:59:17] Topology snapshot [ver=96, locNode=2ee49457, servers=38,
clients=26, state=ACTIVE, CPUs=1760, offheap=2300.0GB, heap=950.0GB]
[08:59:18] Topology snapshot [ver=97, locNode=2ee49457, servers=38,
clients=25, state=ACTIVE, CPUs=1760, offheap=2300.0GB, heap=930.0GB]
[08:59:18] Topology snapshot [ver=98, locNode=2ee49457, servers=38,
clients=24, state=ACTIVE, CPUs=1720, offheap=2300.0GB, heap=920.0GB]
[08:59:18] Topology snapshot [ver=99, locNode=2ee49457, servers=38,
clients=23, state=ACTIVE, CPUs=1720, offheap=2300.0GB, heap=900.0GB]
[08:59:18] Topology snapshot [ver=100, locNode=2ee49457, servers=38,
clients=22, state=ACTIVE, CPUs=1720, offheap=2300.0GB, heap=890.0GB]
[08:59:18] Topology snapshot [ver=101, locNode=2ee49457, servers=38,
clients=21, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=870.0GB]
[08:59:18] Topology snapshot [ver=102, locNode=2ee49457, servers=38,
clients=20, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=860.0GB]
[08:59:18] Topology snapshot [ver=103, locNode=2ee49457, servers=38,
clients=19, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=840.0GB]
[08:59:18] Topology snapshot [ver=104, locNode=2ee49457, servers=38,
clients=18, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=830.0GB]
[08:59:18] Topology snapshot [ver=105, locNode=2ee49457, servers=38,
clients=17, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=810.0GB]
[08:59:18] Topology snapshot [ver=106, locNode=2ee49457, servers=38,
clients=16, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=800.0GB]
[08:59:18] Topology snapshot [ver=107, locNode=2ee49457, servers=38,
clients=15, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=780.0GB]
[08:59:18] Topology snapshot [ver=108, locNode=2ee49457, servers=38,
clients=14, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=770.0GB]
[08:59:18] Topology snapshot [ver=109, locNode=2ee49457, servers=38,
clients=13, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=750.0GB]
[08:59:18] Topology snapshot [ver=110, locNode=2ee49457, servers=38,
clients=12, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=740.0GB]
[08:59:18] Topology snapshot [ver=111, locNode=2ee49457, servers=38,
clients=11, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=720.0GB]
[08:59:18] Topology snapshot [ver=112, locNode=2ee49457, servers=38,
clients=10, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=710.0GB]
[08:59:18] Topology snapshot [ver=113, locNode=2ee49457, servers=38,
clients=9, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=690.0GB]
[08:59:18] Topology snapshot [ver=114, locNode=2ee49457, servers=38,
clients=8, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=680.0GB]
[08:59:18] Topology snapshot [ver=115, locNode=2ee49457, servers=38,
clients=7, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=660.0GB]
[08:59:18] Topology snapshot [ver=116, locNode=2ee49457, servers=38,
clients=6, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=650.0GB]
[08:59:18] Topology snapshot [ver=117, locNode=2ee49457, servers=38,
clients=5, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=630.0GB]
[08:59:18] Topology snapshot [ver=118, locNode=2ee49457, servers=38,
clients=4, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=620.0GB]
[08:59:18] Topology snapshot [ver=119, locNode=2ee49457, servers=38,
clients=3, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=600.0GB]
[08:59:18] Topology snapshot [ver=120, locNode=2ee49457, servers=38,
clients=2, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=590.0GB]
[08:59:18] Topology snapshot [ver=121, locNode=2ee49457, servers=38,
clients=1, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=570.0GB]
[08:59:18] Topology snapshot [ver=122, locNode=2ee49457, servers=38,
clients=0, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=560.0GB]
[08:59:19] Topology snapshot [ver=123, locNode=2ee49457, servers=37,
clients=0, state=ACTIVE, CPUs=1616, offheap=2300.0GB, heap=540.0GB]
[08:59:20] Topology snapshot [ver=124, locNode=2ee49457, servers=38,
clients=0, state=ACTIVE, CPUs=1616, offheap=2300.0GB, heap=560.0GB]
[08:59:35] Topology snapshot [ver=125, locNode=2ee49457, servers=39,
clients=0, state=ACTIVE, CPUs=1664, offheap=2400.0GB, heap=570.0GB]
[08:59:52] Topology snapshot [ver=126, locNode=2ee49457, servers=40,
clients=0, state=ACTIVE, CPUs=1704, offheap=2500.0GB, heap=590.0GB]
[13:32:21] Topology snapshot [ver=127, locNode=2ee49457, servers=39,
clients=0, state=ACTIVE, CPUs=1656, offheap=2400.0GB, heap=570.0GB]
[13:32:25] Topology snapshot [ver=128, locNode=2ee49457, servers=40,
clients=0, state=ACTIVE, CPUs=1656, offheap=2400.0GB, heap=590.0GB]
[13:49:48] Topology snapshot [ver=129, locNode=2ee49457, servers=39,
clients=0, state=ACTIVE, CPUs=1616, offheap=2400.0GB, heap=570.0GB]
[13:50:03] Topology snapshot [ver=130, locNode=2ee49457, servers=40,
clients=0, state=ACTIVE, CPUs=1664, offheap=2500.0GB, heap=590.0GB]
[14:45:29] Topology snapshot [ver=131, locNode=2ee49457, servers=39,
clients=0, state=ACTIVE, CPUs=1664, offheap=2400.0GB, heap=570.0GB]
[14:45:46] Topology snapshot [ver=132, locNode=2ee49457, servers=40,
clients=0, state=ACTIVE, CPUs=1704, offheap=2500.0GB, heap=590.0GB]



Container: container_e192_1564355610025_561993_01_008768 on
c010gst.intqa.bigdata.int.thomsonreuters.com_8041
===============================================================================================================
LogType:stderr
Log Upload Time:Thu Mar 19 13:32:36 +0000 2020
LogLength:11894
Log Contents:
log4j:WARN No appenders could be found for logger
(org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
[08:32:34,822][SEVERE][sys-#148%titan-resourceUP%][TcpCommunicationSpi]
Failed to send message to remote node [node=ZookeeperClusterNode
[id=00d6c796-5bce-404e-867f-3dd67e7c88f6, addrs=[10.204.138.136, 127.0.0.1],
order=258, loc=false, client=false], msg=GridIoMessage [plc=2, topic=T6
[topic=TOPIC_CACHE, id1=83e8ca36-2305-3266-8e65-1463be879baa, id2=0],
topicOrd=-1, ordered=true, timeout=10000, skipOnTimeout=false,
msg=GridDhtPartitionDemandMessage [rebalanceId=41922,
parts=IgniteDhtDemandedPartitionsMap [historical=null], timeout=10000,
workerId=-1, topVer=AffinityTopologyVersion [topVer=323, minorTopVer=0],
partCnt=0, super=GridCacheGroupIdMessage [grpId=-1569773056]]]]
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
Failed to send message (node left topology): ZookeeperClusterNode
[id=00d6c796-5bce-404e-867f-3dd67e7c88f6, addrs=[10.204.138.136, 127.0.0.1],
order=258, loc=false, client=false]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3270)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2713)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2672)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1656)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1766)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendOrderedMessage(GridCacheIoManager.java:1231)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.handleSupplyMessage(GridDhtPartitionDemander.java:845)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.handleSupplyMessage(GridDhtPreloader.java:387)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:418)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:408)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$700(GridCacheIoManager.java:101)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1613)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4100(GridIoManager.java:127)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2768)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1529)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4400(GridIoManager.java:127)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$10.run(GridIoManager.java:1498)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[08:32:35,003][SEVERE][exchange-worker-#123%titan-resourceUP%][TcpCommunicationSpi]
Failed to send message to remote node [node=ZookeeperClusterNode
[id=f8377bf6-7e5e-4efd-bd31-95af11edc52a, addrs=[10.204.138.219, 127.0.0.1],
order=130, loc=false, client=false], msg=GridIoMessage [plc=2,
topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
skipOnTimeout=false, msg=GridDhtPartitionsSingleMessage
[parts={-2100569601=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=326, minorTopVer=0], updateSeq=440,
size=100], -2085876587=GridDhtPartitionMap [moving=5,
top=AffinityTopologyVersion [topVer=326, minorTopVer=0], updateSeq=495,
size=75], -1569773056=GridDhtPartitionMap [moving=4,
top=AffinityTopologyVersion [topVer=326, minorTopVer=0], updateSeq=501,
size=75]}, partCntrs={-2100569601=CachePartitionPartialCountersMap {},
-2085876587=CachePartitionPartialCountersMap {7=(0,5825), 53=(0,5709),
60=(0,5911), 79=(0,5806), 96=(0,5754), 100=(0,5678), 102=(0,5803),
114=(0,5856), 127=(0,5695), 167=(0,5766), 179=(0,5800), 204=(0,5869),
205=(0,5761), 208=(0,5792), 212=(0,5763), 221=(0,5778), 228=(0,5797),
235=(0,5818), 239=(0,5728), 245=(0,5726), 249=(0,5754), 250=(0,5819),
253=(0,5745), 262=(0,5719), 286=(0,5728), 287=(0,5783), 294=(0,5713),
307=(0,5760), 312=(0,5702), 318=(0,5738), 323=(0,5782), 335=(0,5824),
351=(0,5826), 353=(0,5725), 367=(0,5725), 379=(0,5856), 385=(0,5801),
391=(0,5805), 402=(0,5886), 411=(0,5770), 417=(0,5764), 432=(0,5792),
442=(0,5749), 449=(0,5679), 452=(0,5722), 456=(0,5828), 470=(0,5689),
485=(0,5732), 494=(0,5837), 504=(0,5809), 505=(0,5679), 516=(0,5786),
528=(0,5793), 530=(0,5699), 557=(0,5746), 580=(0,5680), 594=(0,5830),
603=(0,5758), 608=(0,5876), 624=(0,5715), 626=(0,5743), 636=(0,5795),
674=(0,5751), 676=(0,5875), 725=(0,5780), 729=(0,5795), 741=(0,5782),
759=(0,5765), 762=(0,5728), 767=(0,5697), 777=(0,5762), 798=(0,5774),
829=(0,5705), 849=(0,5876), 853=(0,5854), 854=(0,5834), 857=(0,5660),
863=(0,5742), 864=(0,5824), 867=(0,5757), 877=(0,5680), 883=(0,5758),
905=(0,5911), 917=(0,5630), 920=(0,5832), 924=(0,5580), 939=(0,5805),
940=(0,5644), 951=(0,5726), 955=(0,5750), 963=(0,5699), 965=(0,5731),
971=(0,5707), 985=(0,5735), 1017=(0,5729)},
-1569773056=CachePartitionPartialCountersMap {7=(0,6915), 53=(0,6912),
60=(0,6825), 79=(0,6962), 96=(0,7046), 100=(0,6926), 102=(0,6813),
114=(0,6850), 127=(0,6926), 167=(0,6816), 179=(0,6795), 204=(0,6894),
205=(0,6950), 208=(0,6742), 212=(0,6939), 221=(0,6933), 228=(0,6932),
235=(0,6903), 239=(0,6816), 245=(0,6928), 249=(0,6895), 250=(0,6892),
253=(0,6934), 262=(0,6927), 286=(0,6906), 287=(0,7015), 294=(0,6781),
307=(0,6899), 312=(0,6773), 318=(0,6944), 323=(0,6999), 335=(0,6925),
351=(0,6885), 353=(0,6797), 367=(0,7074), 379=(0,7041), 385=(0,6921),
391=(0,7101), 402=(0,6858), 411=(0,7057), 417=(0,6862), 432=(0,6752),
442=(0,6911), 449=(0,6939), 452=(0,6834), 456=(0,6792), 470=(0,6936),
485=(0,6903), 494=(0,6955), 504=(0,6943), 505=(0,6893), 516=(0,6838),
528=(0,6922), 530=(0,6893), 557=(0,6849), 580=(0,6822), 594=(0,6994),
603=(0,6872), 608=(0,6862), 624=(0,6987), 626=(0,6913), 636=(0,7033),
674=(0,6921), 676=(0,6967), 725=(0,6831), 729=(0,6854), 741=(0,6853),
759=(0,6871), 762=(0,6873), 767=(0,6700), 777=(0,6765), 798=(0,6712),
829=(0,6933), 849=(0,6936), 853=(0,6967), 854=(0,6995), 857=(0,6908),
863=(0,6987), 864=(0,6872), 867=(0,6847), 877=(0,7032), 883=(0,6830),
905=(0,6790), 917=(0,6953), 920=(0,6941), 924=(0,6819), 939=(0,6859),
940=(0,6814), 951=(0,6879), 955=(0,6915), 963=(0,7002), 965=(0,6858),
971=(0,6919), 985=(0,6746), 1017=(0,6700)}},
partsSizes={-2085876587={385=5801, 262=5719, 7=5825, 391=5805, 777=5762,
905=5911, 528=5793, 402=5886, 530=5699, 920=5832, 286=5728, 798=5774,
287=5783, 294=5713, 940=5644, 557=5746, 432=5792, 179=5800, 53=5709,
312=5702, 442=5749, 955=5750, 60=5911, 318=5738, 449=5679, 323=5782,
580=5680, 456=5828, 971=5707, 205=5761, 79=5806, 335=5824, 208=5792,
594=5830, 725=5780, 853=5854, 729=5795, 857=5660, 985=5735, 603=5758,
221=5778, 351=5826, 863=5742, 96=5754, 864=5824, 353=5725, 867=5757,
100=5678, 228=5797, 485=5732, 102=5803, 235=5818, 877=5680, 494=5837,
367=5725, 624=5715, 883=5758, 245=5726, 759=5765, 504=5809, 249=5754,
505=5679, 379=5856, 636=5795, 253=5745, 127=5695, 767=5697},
-1569773056={385=6921, 262=6927, 7=6915, 391=7101, 777=6765, 905=6790,
528=6922, 402=6858, 530=6893, 920=6941, 286=6906, 798=6712, 287=7015,
294=6781, 167=6816, 940=6814, 557=6849, 432=6752, 179=6795, 53=6912,
312=6773, 442=6911, 955=6915, 60=6825, 318=6944, 449=6939, 323=6999,
580=6822, 456=6792, 971=6919, 205=6950, 79=6962, 335=6925, 208=6742,
594=6994, 725=6831, 853=6967, 854=5329, 729=6854, 857=6908, 985=6746,
603=6872, 221=6933, 351=6885, 863=6987, 96=7046, 864=6872, 353=6797,
867=6847, 100=6926, 228=6932, 485=6903, 102=6813, 235=6903, 877=7032,
494=6955, 367=7074, 624=6987, 883=6830, 245=6928, 759=6871, 504=6943,
249=6895, 505=6893, 379=7041, 636=7033, 253=6934, 127=6926, 767=6700}},
partHistCntrs=null, err=null, client=false, compress=true, finishMsg=null,
activeQryTrackers=null, super=GridDhtPartitionsAbstractMessage
[exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=327, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=00d6c796-5bce-404e-867f-3dd67e7c88f6,
addrs=[10.204.138.136, 127.0.0.1], order=258, loc=false, client=false],
topVer=327, nodeId8=654c8cd2, msg=Node failed: ZookeeperClusterNode
[id=00d6c796-5bce-404e-867f-3dd67e7c88f6, addrs=[10.204.138.136, 127.0.0.1],
order=258, loc=false, client=false], type=NODE_FAILED,
tstamp=1584624754811], nodeId=00d6c796, evt=NODE_FAILED],
lastVer=GridCacheVersion [topVer=195483760, order=1584599672955,
nodeOrder=274], super=GridCacheMessage [msgId=1510, depInfo=null, err=null,
skipPrepare=false]]]]]
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
Failed to send message (node left topology): ZookeeperClusterNode
[id=f8377bf6-7e5e-4efd-bd31-95af11edc52a, addrs=[10.204.138.219, 127.0.0.1],
order=130, loc=false, client=false]
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3270)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2713)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2672)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1656)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1731)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1170)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendLocalPartitions(GridDhtPartitionsExchangeFuture.java:1880)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendPartitions(GridDhtPartitionsExchangeFuture.java:2011)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1501)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:806)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2667)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2539)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)


How to keep the ignite-yarn application steady without increasing the
resources of its own?


Thanks,
Chandan




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite yarn job's resources (number of executors and cores) increase automatically after running for 7 days

Hi,

Please check that you don't have the following property:
spark.dynamicAllocation.enabled=true

The current option provides your Spark job use as many executors as possible if it will be required.

In case if you are going to use this option that please limit the number of executors as follows:

spark.dynamicAllocation.maxExecutors=...

However, it looks like your problem related to configuration of Spark. Please review what you have there.

BR,
Andrei

3/19/2020 10:01 PM, ChandanS пишет:
Hi,

I had submitted a spark job for ignite-yarn application on last 12th March,
2020 with 40 executors and 2 cores (running containers 41 and allocated CPU
81 on spark web UI page) and after running it for 7 days, Today on 19th
March I have seen on spark web UI page that resources were went up by
"running containers 247 and allocated CPU 493" and it's keep on increasing
every one hour and so. How can I maintain the ignite-yarn application not to
increase the resources of its own? Otherwise, it will a concern for the
cluster if it keep on increasing. Below is the part of yarn log for my
ignite-yarn job. 

log4j:WARN No appenders could be found for logger
(org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
[08:58:25,564][SEVERE][grid-timeout-worker-#103%titan-resourceUP%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=tcp-comm-worker,
blockedFor=17s]
[08:58:25,565][SEVERE][grid-timeout-worker-#103%titan-resourceUP%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]]]
class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]
	at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
	at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
	at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
	at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
	at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:58:34,698][SEVERE][grid-timeout-worker-#103%titan-resourceUP%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=tcp-comm-worker,
blockedFor=26s]
[08:58:34,699][SEVERE][grid-timeout-worker-#103%titan-resourceUP%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]]]
class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]
	at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
	at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
	at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
	at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
	at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:58:43,805][SEVERE][nio-acceptor-tcp-comm-#128%titan-resourceUP%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=tcp-comm-worker,
blockedFor=35s]
[08:58:43,805][SEVERE][nio-acceptor-tcp-comm-#128%titan-resourceUP%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]]]
class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]
	at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
	at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
	at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
	at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
	at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2938)
	at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:58:56,406][SEVERE][ttl-cleanup-worker-#171%titan-resourceUP%][G] Blocked
system-critical thread has been detected. This can lead to cluster-wide
undefined behaviour [threadName=tcp-comm-worker, blockedFor=47s]
[08:58:56,407][SEVERE][ttl-cleanup-worker-#171%titan-resourceUP%][] Critical
system error detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]]]
class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]
	at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
	at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
	at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
	at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
	at
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:151)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:05,506][SEVERE][nio-acceptor-tcp-comm-#128%titan-resourceUP%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=tcp-comm-worker,
blockedFor=56s]
[08:59:05,506][SEVERE][nio-acceptor-tcp-comm-#128%titan-resourceUP%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]]]
class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003488512]
	at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
	at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
	at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
	at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
	at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2938)
	at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,591][SEVERE][grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=9,
bytesRcvd=4475652, bytesSent=1731122, bytesRcvd0=18, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-9,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549587, hashCode=344426864, interrupted=false,
runner=grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=166400 cap=166400],
readBuf=java.nio.DirectByteBuffer[pos=18 lim=18 cap=117552],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/10.204.138.138:39970,
rmtAddr=c250xmv.intqa.bigdata.int.thomsonreuters.com/10.204.138.66:45400,
createTime=1584003549587, closeTime=0, bytesSent=0, bytesRcvd=18,
bytesSent0=0, bytesRcvd0=18, sndSchedTime=1584003549587,
lastSndTime=1584003549587, lastRcvTime=1584003549587, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false, markedForClose=false]]]
java.lang.NullPointerException
	at
org.apache.ignite.internal.util.nio.GridNioServer.cancelConnect(GridNioServer.java:898)
	at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.cancel(TcpCommunicationConnectionCheckFuture.java:338)
	at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture.cancelFutures(TcpCommunicationConnectionCheckFuture.java:475)
	at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture.receivedAddressStatus(TcpCommunicationConnectionCheckFuture.java:494)
	at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture$1.onStatusReceived(TcpCommunicationConnectionCheckFuture.java:433)
	at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.finish(TcpCommunicationConnectionCheckFuture.java:362)
	at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.onConnected(TcpCommunicationConnectionCheckFuture.java:348)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:779)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:389)
	at
org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
	at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
	at
org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:117)
	at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
	at
org.apache.ignite.internal.util.nio.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:88)
	at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
	at
org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3553)
	at
org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1303)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,591][SEVERE][grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception.
class org.apache.ignite.internal.util.nio.GridNioException: null
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2415)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at
org.apache.ignite.internal.util.nio.GridNioServer.cancelConnect(GridNioServer.java:898)
	at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.cancel(TcpCommunicationConnectionCheckFuture.java:338)
	at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture.cancelFutures(TcpCommunicationConnectionCheckFuture.java:475)
	at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture.receivedAddressStatus(TcpCommunicationConnectionCheckFuture.java:494)
	at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture$1.onStatusReceived(TcpCommunicationConnectionCheckFuture.java:433)
	at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.finish(TcpCommunicationConnectionCheckFuture.java:362)
	at
org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.onConnected(TcpCommunicationConnectionCheckFuture.java:348)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:779)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:389)
	at
org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
	at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
	at
org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:117)
	at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
	at
org.apache.ignite.internal.util.nio.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:88)
	at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
	at
org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3553)
	at
org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1303)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	... 4 more
[08:59:10,593][SEVERE][grid-nio-worker-tcp-comm-19-#123%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=19,
bytesRcvd=4112614, bytesSent=5001528, bytesRcvd0=0, bytesSent0=18,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-19,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549587, hashCode=1777632028, interrupted=false,
runner=grid-nio-worker-tcp-comm-19-#123%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40456,
createTime=1584003549587, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549587,
lastSndTime=1584003549587, lastRcvTime=1584003549587, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,597][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=2,
bytesRcvd=6049119, bytesSent=6778991, bytesRcvd0=18, bytesSent0=36,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-2,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549587, hashCode=813216397, interrupted=false,
runner=grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40512,
createTime=1584003549587, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549587,
lastSndTime=1584003549587, lastRcvTime=1584003549587, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,597][SEVERE][grid-nio-worker-tcp-comm-7-#111%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,599][SEVERE][grid-nio-worker-tcp-comm-6-#110%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=6,
bytesRcvd=5589344, bytesSent=3069900, bytesRcvd0=54, bytesSent0=18,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-6,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549597, hashCode=293379764, interrupted=false,
runner=grid-nio-worker-tcp-comm-6-#110%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40540,
createTime=1584003549597, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549597,
lastSndTime=1584003549597, lastRcvTime=1584003549597, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-21-#125%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-3-#107%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-18-#122%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=18,
bytesRcvd=3987619, bytesSent=1696258, bytesRcvd0=36, bytesSent0=54,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-18,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549597, hashCode=1083458718, interrupted=false,
runner=grid-nio-worker-tcp-comm-18-#122%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40552,
createTime=1584003549597, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549597,
lastSndTime=1584003549597, lastRcvTime=1584003549597, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-11-#115%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-13-#117%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,601][SEVERE][grid-nio-worker-tcp-comm-15-#119%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,601][SEVERE][grid-nio-worker-tcp-comm-17-#121%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,601][SEVERE][grid-nio-worker-tcp-comm-23-#127%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,602][SEVERE][grid-nio-worker-tcp-comm-1-#105%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,602][SEVERE][grid-nio-worker-tcp-comm-4-#108%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,602][SEVERE][grid-nio-worker-tcp-comm-8-#112%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,603][SEVERE][grid-nio-worker-tcp-comm-20-#124%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,608][SEVERE][grid-nio-worker-tcp-comm-0-#104%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:10,609][SEVERE][grid-nio-worker-tcp-comm-10-#114%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,592][SEVERE][grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,595][SEVERE][grid-nio-worker-tcp-comm-19-#123%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,598][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,598][SEVERE][grid-nio-worker-tcp-comm-7-#111%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,601][SEVERE][grid-nio-worker-tcp-comm-14-#118%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=14,
bytesRcvd=3350068, bytesSent=8196, bytesRcvd0=72, bytesSent0=126,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-14,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549617, hashCode=907545167, interrupted=false,
runner=grid-nio-worker-tcp-comm-14-#118%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40716,
createTime=1584003549607, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549607,
lastSndTime=1584003549607, lastRcvTime=1584003549607, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,601][SEVERE][grid-nio-worker-tcp-comm-12-#116%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=12,
bytesRcvd=3969821, bytesSent=3288859, bytesRcvd0=108, bytesSent0=108,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-12,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549617, hashCode=1020912454, interrupted=false,
runner=grid-nio-worker-tcp-comm-12-#116%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40748,
createTime=1584003549607, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549607,
lastSndTime=1584003549607, lastRcvTime=1584003549607, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,602][SEVERE][grid-nio-worker-tcp-comm-17-#121%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,602][SEVERE][grid-nio-worker-tcp-comm-11-#115%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,603][SEVERE][grid-nio-worker-tcp-comm-8-#112%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,604][SEVERE][grid-nio-worker-tcp-comm-5-#109%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=5,
bytesRcvd=4011044, bytesSent=7399489, bytesRcvd0=72, bytesSent0=108,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-5,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003549607, hashCode=606372690, interrupted=false,
runner=grid-nio-worker-tcp-comm-5-#109%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40648,
createTime=1584003549597, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549597,
lastSndTime=1584003549597, lastRcvTime=1584003549597, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,609][SEVERE][grid-nio-worker-tcp-comm-1-#105%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=1,
bytesRcvd=2726130, bytesSent=6003920, bytesRcvd0=90, bytesSent0=72,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003550603, hashCode=1883663272, interrupted=false,
runner=grid-nio-worker-tcp-comm-1-#105%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40664,
createTime=1584003550593, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003550593,
lastSndTime=1584003550593, lastRcvTime=1584003550593, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,609][SEVERE][grid-nio-worker-tcp-comm-0-#104%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0,
bytesRcvd=4890192, bytesSent=12608476, bytesRcvd0=54, bytesSent0=595,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003550603, hashCode=1994247289, interrupted=false,
runner=grid-nio-worker-tcp-comm-0-#104%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40640,
createTime=1584003549597, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549597,
lastSndTime=1584003549597, lastRcvTime=1584003549597, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:11,610][SEVERE][grid-nio-worker-tcp-comm-10-#114%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=10,
bytesRcvd=4013857, bytesSent=3362419, bytesRcvd0=90, bytesSent0=72,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-10,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003550603, hashCode=1302629280, interrupted=false,
runner=grid-nio-worker-tcp-comm-10-#114%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40496,
createTime=1584003549587, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549587,
lastSndTime=1584003549587, lastRcvTime=1584003549587, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:12,593][SEVERE][grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:12,598][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:12,599][SEVERE][grid-nio-worker-tcp-comm-7-#111%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:12,602][SEVERE][grid-nio-worker-tcp-comm-15-#119%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=15,
bytesRcvd=5571714, bytesSent=1717776, bytesRcvd0=0, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-15,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003550623, hashCode=1293929915, interrupted=false,
runner=grid-nio-worker-tcp-comm-15-#119%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40684,
createTime=1584003550593, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1584003550593,
lastSndTime=1584003550593, lastRcvTime=1584003550593, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:12,602][SEVERE][grid-nio-worker-tcp-comm-18-#122%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=18,
bytesRcvd=3987655, bytesSent=1696348, bytesRcvd0=0, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-18,
igniteInstanceName=titan-resourceUP, finished=false,
heartbeatTs=1584003551589, hashCode=1083458718, interrupted=false,
runner=grid-nio-worker-tcp-comm-18-#122%titan-resourceUP%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40720,
createTime=1584003550593, closeTime=0, bytesSent=18, bytesRcvd=0,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1584003550593,
lastSndTime=1584003550593, lastRcvTime=1584003550593, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:13,598][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi]
Failed to process selector key [ses=null]
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
[08:59:41,932][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi]
Failed to read data from remote connection (will wait for 2000ms).
class org.apache.ignite.IgniteCheckedException: Failed to select events on
selector.
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2185)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
	at
java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:197)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2000)
	... 3 more
[13:49:50,352][SEVERE][grid-nio-worker-tcp-comm-0-#104%titan-resourceUP%][TcpCommunicationSpi]
Failed to read data from remote connection (will wait for 2000ms).
class org.apache.ignite.IgniteCheckedException: Failed to select events on
selector.
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2185)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
	at
java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:197)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2000)
	... 3 more
[14:45:35,550][SEVERE][grid-nio-worker-tcp-comm-1-#105%titan-resourceUP%][TcpCommunicationSpi]
Failed to read data from remote connection (will wait for 2000ms).
class org.apache.ignite.IgniteCheckedException: Failed to select events on
selector.
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2185)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
	at
java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:197)
	at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2000)
	... 3 more

LogType:stdout
Log Upload Time:Sat Mar 14 08:29:54 +0000 2020
LogLength:19390
Log Contents:
[08:57:21]    __________  ________________ 
[08:57:21]   /  _/ ___/ |/ /  _/_  __/ __/ 
[08:57:21]  _/ // (7 7    // /  / / / _/   
[08:57:21] /___/\___/_/|_/___/ /_/ /___/  
[08:57:21] 
[08:57:21] ver. 2.7.0#20181130-sha1:256ae401
[08:57:21] 2018 Copyright(C) Apache Software Foundation
[08:57:21] 
[08:57:21] Ignite documentation: http://ignite.apache.org
[08:57:21] 
[08:57:21] Quiet mode.
[08:57:21]   ^-- Logging to file
'/data/12/yarn/nm/usercache/bigdata-app-ecpdevbermuda-uncomparator/appcache/application_1564355610025_561993/container_e192_1564355610025_561993_01_000044/ignite/apache-ignite-2.7.0-bin/work/log/ignite-2ee49457.0.log'
[08:57:21]   ^-- Logging by 'JavaLogger [quiet=true, config=null]'
[08:57:21]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[08:57:21] 
[08:57:21] OS: Linux 3.10.0-1062.9.1.el7.x86_64 amd64
[08:57:21] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13
[08:57:21] Please set system property '-Djava.net.preferIPv4Stack=true' to
avoid possible problems in mixed environments.
[08:57:21] Configured plugins:
[08:57:21]   ^-- None
[08:57:21] 
[08:57:21] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]]]
[08:57:21] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
[08:57:22] Security status [authentication=off, tls/ssl=off]
[08:57:22] Topology snapshot [ver=12, locNode=2ee49457, servers=12,
clients=0, state=ACTIVE, CPUs=576, offheap=910.0GB, heap=180.0GB]
[08:57:22] Topology snapshot [ver=13, locNode=2ee49457, servers=13,
clients=0, state=ACTIVE, CPUs=624, offheap=980.0GB, heap=190.0GB]
[08:57:22] Topology snapshot [ver=14, locNode=2ee49457, servers=14,
clients=0, state=ACTIVE, CPUs=672, offheap=1100.0GB, heap=210.0GB]
[08:57:23] Performance suggestions for grid 'titan-resourceUP' (fix if
possible)
[08:57:23] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[08:57:23]   ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM
options)
[08:57:23]   ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[08:57:23]   ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
[08:57:23]   ^-- Speed up flushing of dirty pages by OS (alter
vm.dirty_expire_centisecs parameter by setting to 500)
[08:57:23] Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[08:57:23] 
[08:57:23] To start Console Management & Monitoring run
ignitevisorcmd.{sh|bat}
[08:57:23] Data Regions Configured:
[08:57:23]   ^-- default [initSize=256.0 MiB, maxSize=75.5 GiB,
persistence=false]
[08:57:23] 
[08:57:23] Ignite node started OK (id=2ee49457, instance
name=titan-resourceUP)
[08:57:23] Topology snapshot [ver=11, locNode=2ee49457, servers=11,
clients=0, state=ACTIVE, CPUs=528, offheap=830.0GB, heap=160.0GB]
[08:57:23] Topology snapshot [ver=15, locNode=2ee49457, servers=15,
clients=0, state=ACTIVE, CPUs=712, offheap=1100.0GB, heap=220.0GB]
[08:57:23] Topology snapshot [ver=16, locNode=2ee49457, servers=16,
clients=0, state=ACTIVE, CPUs=752, offheap=1200.0GB, heap=230.0GB]
[08:57:23] Topology snapshot [ver=17, locNode=2ee49457, servers=17,
clients=0, state=ACTIVE, CPUs=792, offheap=1200.0GB, heap=250.0GB]
[08:57:23] Topology snapshot [ver=18, locNode=2ee49457, servers=18,
clients=0, state=ACTIVE, CPUs=832, offheap=1300.0GB, heap=260.0GB]
[08:57:23] Topology snapshot [ver=19, locNode=2ee49457, servers=19,
clients=0, state=ACTIVE, CPUs=872, offheap=1300.0GB, heap=280.0GB]
[08:57:23] Topology snapshot [ver=20, locNode=2ee49457, servers=20,
clients=0, state=ACTIVE, CPUs=912, offheap=1400.0GB, heap=290.0GB]
[08:57:23] Topology snapshot [ver=21, locNode=2ee49457, servers=21,
clients=0, state=ACTIVE, CPUs=952, offheap=1400.0GB, heap=310.0GB]
[08:57:23] Topology snapshot [ver=22, locNode=2ee49457, servers=22,
clients=0, state=ACTIVE, CPUs=992, offheap=1500.0GB, heap=320.0GB]
[08:57:23] Topology snapshot [ver=23, locNode=2ee49457, servers=23,
clients=0, state=ACTIVE, CPUs=1032, offheap=1500.0GB, heap=340.0GB]
[08:57:23] Topology snapshot [ver=24, locNode=2ee49457, servers=24,
clients=0, state=ACTIVE, CPUs=1072, offheap=1600.0GB, heap=350.0GB]
[08:57:23] Topology snapshot [ver=25, locNode=2ee49457, servers=25,
clients=0, state=ACTIVE, CPUs=1112, offheap=1600.0GB, heap=370.0GB]
[08:57:24] Topology snapshot [ver=26, locNode=2ee49457, servers=26,
clients=0, state=ACTIVE, CPUs=1152, offheap=1700.0GB, heap=380.0GB]
[08:57:29] Topology snapshot [ver=27, locNode=2ee49457, servers=27,
clients=0, state=ACTIVE, CPUs=1200, offheap=1700.0GB, heap=400.0GB]
[08:57:30] Topology snapshot [ver=28, locNode=2ee49457, servers=28,
clients=0, state=ACTIVE, CPUs=1248, offheap=1800.0GB, heap=410.0GB]
[08:57:31] Topology snapshot [ver=29, locNode=2ee49457, servers=29,
clients=0, state=ACTIVE, CPUs=1296, offheap=1900.0GB, heap=420.0GB]
[08:57:31] Topology snapshot [ver=30, locNode=2ee49457, servers=29,
clients=1, state=ACTIVE, CPUs=1296, offheap=1900.0GB, heap=460.0GB]
[08:57:36] Topology snapshot [ver=31, locNode=2ee49457, servers=29,
clients=2, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=480.0GB]
[08:57:37] Topology snapshot [ver=32, locNode=2ee49457, servers=29,
clients=3, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=490.0GB]
[08:57:37] Topology snapshot [ver=33, locNode=2ee49457, servers=29,
clients=4, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=510.0GB]
[08:57:37] Topology snapshot [ver=34, locNode=2ee49457, servers=29,
clients=5, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=520.0GB]
[08:57:37] Topology snapshot [ver=35, locNode=2ee49457, servers=29,
clients=6, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=540.0GB]
[08:57:37] Topology snapshot [ver=36, locNode=2ee49457, servers=29,
clients=7, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=550.0GB]
[08:57:37] Topology snapshot [ver=37, locNode=2ee49457, servers=29,
clients=8, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=570.0GB]
[08:57:37] Topology snapshot [ver=38, locNode=2ee49457, servers=29,
clients=9, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=580.0GB]
[08:57:37] Topology snapshot [ver=39, locNode=2ee49457, servers=29,
clients=10, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=600.0GB]
[08:57:37] Topology snapshot [ver=40, locNode=2ee49457, servers=29,
clients=11, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=610.0GB]
[08:57:37] Topology snapshot [ver=41, locNode=2ee49457, servers=29,
clients=12, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=630.0GB]
[08:57:37] Topology snapshot [ver=42, locNode=2ee49457, servers=29,
clients=13, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=640.0GB]
[08:57:37] Topology snapshot [ver=43, locNode=2ee49457, servers=29,
clients=14, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=660.0GB]
[08:57:37] Topology snapshot [ver=44, locNode=2ee49457, servers=29,
clients=15, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=670.0GB]
[08:57:37] Topology snapshot [ver=45, locNode=2ee49457, servers=29,
clients=16, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=690.0GB]
[08:57:37] Topology snapshot [ver=46, locNode=2ee49457, servers=29,
clients=17, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=700.0GB]
[08:57:37] Topology snapshot [ver=47, locNode=2ee49457, servers=29,
clients=18, state=ACTIVE, CPUs=1384, offheap=1900.0GB, heap=720.0GB]
[08:57:37] Topology snapshot [ver=48, locNode=2ee49457, servers=29,
clients=19, state=ACTIVE, CPUs=1384, offheap=1900.0GB, heap=730.0GB]
[08:57:37] Topology snapshot [ver=49, locNode=2ee49457, servers=29,
clients=20, state=ACTIVE, CPUs=1424, offheap=1900.0GB, heap=750.0GB]
[08:57:37] Topology snapshot [ver=50, locNode=2ee49457, servers=29,
clients=21, state=ACTIVE, CPUs=1424, offheap=1900.0GB, heap=760.0GB]
[08:57:37] Topology snapshot [ver=51, locNode=2ee49457, servers=29,
clients=22, state=ACTIVE, CPUs=1464, offheap=1900.0GB, heap=780.0GB]
[08:57:37] Topology snapshot [ver=52, locNode=2ee49457, servers=29,
clients=23, state=ACTIVE, CPUs=1464, offheap=1900.0GB, heap=790.0GB]
[08:57:37] Topology snapshot [ver=53, locNode=2ee49457, servers=29,
clients=24, state=ACTIVE, CPUs=1504, offheap=1900.0GB, heap=810.0GB]
[08:57:37] Topology snapshot [ver=54, locNode=2ee49457, servers=29,
clients=25, state=ACTIVE, CPUs=1504, offheap=1900.0GB, heap=820.0GB]
[08:57:37] Topology snapshot [ver=55, locNode=2ee49457, servers=29,
clients=26, state=ACTIVE, CPUs=1544, offheap=1900.0GB, heap=840.0GB]
[08:57:37] Topology snapshot [ver=56, locNode=2ee49457, servers=29,
clients=27, state=ACTIVE, CPUs=1584, offheap=1900.0GB, heap=850.0GB]
[08:57:37] Topology snapshot [ver=57, locNode=2ee49457, servers=29,
clients=28, state=ACTIVE, CPUs=1584, offheap=1900.0GB, heap=870.0GB]
[08:57:38] Topology snapshot [ver=58, locNode=2ee49457, servers=29,
clients=29, state=ACTIVE, CPUs=1584, offheap=1900.0GB, heap=880.0GB]
[08:57:38] Topology snapshot [ver=59, locNode=2ee49457, servers=29,
clients=30, state=ACTIVE, CPUs=1584, offheap=1900.0GB, heap=900.0GB]
[08:57:38] Topology snapshot [ver=60, locNode=2ee49457, servers=29,
clients=31, state=ACTIVE, CPUs=1616, offheap=1900.0GB, heap=910.0GB]
[08:57:38] Topology snapshot [ver=61, locNode=2ee49457, servers=29,
clients=32, state=ACTIVE, CPUs=1656, offheap=1900.0GB, heap=930.0GB]
[08:57:38] Topology snapshot [ver=62, locNode=2ee49457, servers=29,
clients=33, state=ACTIVE, CPUs=1696, offheap=1900.0GB, heap=940.0GB]
[08:57:38] Topology snapshot [ver=63, locNode=2ee49457, servers=29,
clients=34, state=ACTIVE, CPUs=1696, offheap=1900.0GB, heap=960.0GB]
[08:57:38] Topology snapshot [ver=64, locNode=2ee49457, servers=29,
clients=35, state=ACTIVE, CPUs=1696, offheap=1900.0GB, heap=970.0GB]
[08:57:38] Topology snapshot [ver=65, locNode=2ee49457, servers=29,
clients=36, state=ACTIVE, CPUs=1696, offheap=1900.0GB, heap=990.0GB]
[08:57:38] Topology snapshot [ver=66, locNode=2ee49457, servers=29,
clients=37, state=ACTIVE, CPUs=1736, offheap=1900.0GB, heap=1000.0GB]
[08:57:38] Topology snapshot [ver=67, locNode=2ee49457, servers=29,
clients=38, state=ACTIVE, CPUs=1736, offheap=1900.0GB, heap=1000.0GB]
[08:57:38] Topology snapshot [ver=68, locNode=2ee49457, servers=29,
clients=39, state=ACTIVE, CPUs=1776, offheap=1900.0GB, heap=1000.0GB]
[08:57:38] Topology snapshot [ver=69, locNode=2ee49457, servers=29,
clients=40, state=ACTIVE, CPUs=1808, offheap=1900.0GB, heap=1000.0GB]
[08:57:38] Topology snapshot [ver=70, locNode=2ee49457, servers=29,
clients=41, state=ACTIVE, CPUs=1840, offheap=1900.0GB, heap=1100.0GB]
[08:58:01] Topology snapshot [ver=71, locNode=2ee49457, servers=30,
clients=41, state=ACTIVE, CPUs=1840, offheap=1900.0GB, heap=1100.0GB]
[08:58:02] Topology snapshot [ver=72, locNode=2ee49457, servers=31,
clients=41, state=ACTIVE, CPUs=1840, offheap=2000.0GB, heap=1100.0GB]
[08:58:02] Topology snapshot [ver=73, locNode=2ee49457, servers=32,
clients=41, state=ACTIVE, CPUs=1840, offheap=2000.0GB, heap=1100.0GB]
[08:58:02] Topology snapshot [ver=74, locNode=2ee49457, servers=33,
clients=41, state=ACTIVE, CPUs=1840, offheap=2100.0GB, heap=1100.0GB]
[08:58:02] Topology snapshot [ver=75, locNode=2ee49457, servers=34,
clients=41, state=ACTIVE, CPUs=1840, offheap=2100.0GB, heap=1100.0GB]
[08:58:02] Topology snapshot [ver=76, locNode=2ee49457, servers=35,
clients=41, state=ACTIVE, CPUs=1880, offheap=2200.0GB, heap=1100.0GB]
[08:58:02] Topology snapshot [ver=77, locNode=2ee49457, servers=36,
clients=41, state=ACTIVE, CPUs=1880, offheap=2200.0GB, heap=1200.0GB]
[08:58:02] Topology snapshot [ver=78, locNode=2ee49457, servers=37,
clients=41, state=ACTIVE, CPUs=1920, offheap=2300.0GB, heap=1200.0GB]
[08:58:03] Topology snapshot [ver=79, locNode=2ee49457, servers=38,
clients=41, state=ACTIVE, CPUs=1960, offheap=2300.0GB, heap=1200.0GB]
[08:58:05] Topology snapshot [ver=80, locNode=2ee49457, servers=39,
clients=41, state=ACTIVE, CPUs=2000, offheap=2400.0GB, heap=1200.0GB]
[08:59:10] Topology snapshot [ver=81, locNode=2ee49457, servers=38,
clients=41, state=ACTIVE, CPUs=1960, offheap=2300.0GB, heap=1200.0GB]
[08:59:16] Topology snapshot [ver=82, locNode=2ee49457, servers=38,
clients=40, state=ACTIVE, CPUs=1960, offheap=2300.0GB, heap=1200.0GB]
[08:59:17] Topology snapshot [ver=83, locNode=2ee49457, servers=38,
clients=39, state=ACTIVE, CPUs=1960, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=84, locNode=2ee49457, servers=38,
clients=38, state=ACTIVE, CPUs=1912, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=85, locNode=2ee49457, servers=38,
clients=37, state=ACTIVE, CPUs=1912, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=86, locNode=2ee49457, servers=38,
clients=36, state=ACTIVE, CPUs=1912, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=87, locNode=2ee49457, servers=38,
clients=35, state=ACTIVE, CPUs=1872, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=88, locNode=2ee49457, servers=38,
clients=34, state=ACTIVE, CPUs=1832, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=89, locNode=2ee49457, servers=38,
clients=33, state=ACTIVE, CPUs=1832, offheap=2300.0GB, heap=1100.0GB]
[08:59:17] Topology snapshot [ver=90, locNode=2ee49457, servers=38,
clients=32, state=ACTIVE, CPUs=1800, offheap=2300.0GB, heap=1000.0GB]
[08:59:17] Topology snapshot [ver=91, locNode=2ee49457, servers=38,
clients=31, state=ACTIVE, CPUs=1800, offheap=2300.0GB, heap=1000.0GB]
[08:59:17] Topology snapshot [ver=92, locNode=2ee49457, servers=38,
clients=30, state=ACTIVE, CPUs=1800, offheap=2300.0GB, heap=1000.0GB]
[08:59:17] Topology snapshot [ver=93, locNode=2ee49457, servers=38,
clients=29, state=ACTIVE, CPUs=1800, offheap=2300.0GB, heap=990.0GB]
[08:59:17] Topology snapshot [ver=94, locNode=2ee49457, servers=38,
clients=28, state=ACTIVE, CPUs=1760, offheap=2300.0GB, heap=980.0GB]
[08:59:17] Topology snapshot [ver=95, locNode=2ee49457, servers=38,
clients=27, state=ACTIVE, CPUs=1760, offheap=2300.0GB, heap=960.0GB]
[08:59:17] Topology snapshot [ver=96, locNode=2ee49457, servers=38,
clients=26, state=ACTIVE, CPUs=1760, offheap=2300.0GB, heap=950.0GB]
[08:59:18] Topology snapshot [ver=97, locNode=2ee49457, servers=38,
clients=25, state=ACTIVE, CPUs=1760, offheap=2300.0GB, heap=930.0GB]
[08:59:18] Topology snapshot [ver=98, locNode=2ee49457, servers=38,
clients=24, state=ACTIVE, CPUs=1720, offheap=2300.0GB, heap=920.0GB]
[08:59:18] Topology snapshot [ver=99, locNode=2ee49457, servers=38,
clients=23, state=ACTIVE, CPUs=1720, offheap=2300.0GB, heap=900.0GB]
[08:59:18] Topology snapshot [ver=100, locNode=2ee49457, servers=38,
clients=22, state=ACTIVE, CPUs=1720, offheap=2300.0GB, heap=890.0GB]
[08:59:18] Topology snapshot [ver=101, locNode=2ee49457, servers=38,
clients=21, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=870.0GB]
[08:59:18] Topology snapshot [ver=102, locNode=2ee49457, servers=38,
clients=20, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=860.0GB]
[08:59:18] Topology snapshot [ver=103, locNode=2ee49457, servers=38,
clients=19, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=840.0GB]
[08:59:18] Topology snapshot [ver=104, locNode=2ee49457, servers=38,
clients=18, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=830.0GB]
[08:59:18] Topology snapshot [ver=105, locNode=2ee49457, servers=38,
clients=17, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=810.0GB]
[08:59:18] Topology snapshot [ver=106, locNode=2ee49457, servers=38,
clients=16, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=800.0GB]
[08:59:18] Topology snapshot [ver=107, locNode=2ee49457, servers=38,
clients=15, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=780.0GB]
[08:59:18] Topology snapshot [ver=108, locNode=2ee49457, servers=38,
clients=14, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=770.0GB]
[08:59:18] Topology snapshot [ver=109, locNode=2ee49457, servers=38,
clients=13, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=750.0GB]
[08:59:18] Topology snapshot [ver=110, locNode=2ee49457, servers=38,
clients=12, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=740.0GB]
[08:59:18] Topology snapshot [ver=111, locNode=2ee49457, servers=38,
clients=11, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=720.0GB]
[08:59:18] Topology snapshot [ver=112, locNode=2ee49457, servers=38,
clients=10, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=710.0GB]
[08:59:18] Topology snapshot [ver=113, locNode=2ee49457, servers=38,
clients=9, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=690.0GB]
[08:59:18] Topology snapshot [ver=114, locNode=2ee49457, servers=38,
clients=8, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=680.0GB]
[08:59:18] Topology snapshot [ver=115, locNode=2ee49457, servers=38,
clients=7, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=660.0GB]
[08:59:18] Topology snapshot [ver=116, locNode=2ee49457, servers=38,
clients=6, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=650.0GB]
[08:59:18] Topology snapshot [ver=117, locNode=2ee49457, servers=38,
clients=5, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=630.0GB]
[08:59:18] Topology snapshot [ver=118, locNode=2ee49457, servers=38,
clients=4, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=620.0GB]
[08:59:18] Topology snapshot [ver=119, locNode=2ee49457, servers=38,
clients=3, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=600.0GB]
[08:59:18] Topology snapshot [ver=120, locNode=2ee49457, servers=38,
clients=2, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=590.0GB]
[08:59:18] Topology snapshot [ver=121, locNode=2ee49457, servers=38,
clients=1, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=570.0GB]
[08:59:18] Topology snapshot [ver=122, locNode=2ee49457, servers=38,
clients=0, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=560.0GB]
[08:59:19] Topology snapshot [ver=123, locNode=2ee49457, servers=37,
clients=0, state=ACTIVE, CPUs=1616, offheap=2300.0GB, heap=540.0GB]
[08:59:20] Topology snapshot [ver=124, locNode=2ee49457, servers=38,
clients=0, state=ACTIVE, CPUs=1616, offheap=2300.0GB, heap=560.0GB]
[08:59:35] Topology snapshot [ver=125, locNode=2ee49457, servers=39,
clients=0, state=ACTIVE, CPUs=1664, offheap=2400.0GB, heap=570.0GB]
[08:59:52] Topology snapshot [ver=126, locNode=2ee49457, servers=40,
clients=0, state=ACTIVE, CPUs=1704, offheap=2500.0GB, heap=590.0GB]
[13:32:21] Topology snapshot [ver=127, locNode=2ee49457, servers=39,
clients=0, state=ACTIVE, CPUs=1656, offheap=2400.0GB, heap=570.0GB]
[13:32:25] Topology snapshot [ver=128, locNode=2ee49457, servers=40,
clients=0, state=ACTIVE, CPUs=1656, offheap=2400.0GB, heap=590.0GB]
[13:49:48] Topology snapshot [ver=129, locNode=2ee49457, servers=39,
clients=0, state=ACTIVE, CPUs=1616, offheap=2400.0GB, heap=570.0GB]
[13:50:03] Topology snapshot [ver=130, locNode=2ee49457, servers=40,
clients=0, state=ACTIVE, CPUs=1664, offheap=2500.0GB, heap=590.0GB]
[14:45:29] Topology snapshot [ver=131, locNode=2ee49457, servers=39,
clients=0, state=ACTIVE, CPUs=1664, offheap=2400.0GB, heap=570.0GB]
[14:45:46] Topology snapshot [ver=132, locNode=2ee49457, servers=40,
clients=0, state=ACTIVE, CPUs=1704, offheap=2500.0GB, heap=590.0GB]



Container: container_e192_1564355610025_561993_01_008768 on
c010gst.intqa.bigdata.int.thomsonreuters.com_8041
===============================================================================================================
LogType:stderr
Log Upload Time:Thu Mar 19 13:32:36 +0000 2020
LogLength:11894
Log Contents:
log4j:WARN No appenders could be found for logger
(org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
[08:32:34,822][SEVERE][sys-#148%titan-resourceUP%][TcpCommunicationSpi]
Failed to send message to remote node [node=ZookeeperClusterNode
[id=00d6c796-5bce-404e-867f-3dd67e7c88f6, addrs=[10.204.138.136, 127.0.0.1],
order=258, loc=false, client=false], msg=GridIoMessage [plc=2, topic=T6
[topic=TOPIC_CACHE, id1=83e8ca36-2305-3266-8e65-1463be879baa, id2=0],
topicOrd=-1, ordered=true, timeout=10000, skipOnTimeout=false,
msg=GridDhtPartitionDemandMessage [rebalanceId=41922,
parts=IgniteDhtDemandedPartitionsMap [historical=null], timeout=10000,
workerId=-1, topVer=AffinityTopologyVersion [topVer=323, minorTopVer=0],
partCnt=0, super=GridCacheGroupIdMessage [grpId=-1569773056]]]]
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
Failed to send message (node left topology): ZookeeperClusterNode
[id=00d6c796-5bce-404e-867f-3dd67e7c88f6, addrs=[10.204.138.136, 127.0.0.1],
order=258, loc=false, client=false]
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3270)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2713)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2672)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1656)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1766)
	at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendOrderedMessage(GridCacheIoManager.java:1231)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.handleSupplyMessage(GridDhtPartitionDemander.java:845)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.handleSupplyMessage(GridDhtPreloader.java:387)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:418)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:408)
	at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
	at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
	at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$700(GridCacheIoManager.java:101)
	at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1613)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4100(GridIoManager.java:127)
	at
org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2768)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1529)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4400(GridIoManager.java:127)
	at
org.apache.ignite.internal.managers.communication.GridIoManager$10.run(GridIoManager.java:1498)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[08:32:35,003][SEVERE][exchange-worker-#123%titan-resourceUP%][TcpCommunicationSpi]
Failed to send message to remote node [node=ZookeeperClusterNode
[id=f8377bf6-7e5e-4efd-bd31-95af11edc52a, addrs=[10.204.138.219, 127.0.0.1],
order=130, loc=false, client=false], msg=GridIoMessage [plc=2,
topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
skipOnTimeout=false, msg=GridDhtPartitionsSingleMessage
[parts={-2100569601=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=326, minorTopVer=0], updateSeq=440,
size=100], -2085876587=GridDhtPartitionMap [moving=5,
top=AffinityTopologyVersion [topVer=326, minorTopVer=0], updateSeq=495,
size=75], -1569773056=GridDhtPartitionMap [moving=4,
top=AffinityTopologyVersion [topVer=326, minorTopVer=0], updateSeq=501,
size=75]}, partCntrs={-2100569601=CachePartitionPartialCountersMap {},
-2085876587=CachePartitionPartialCountersMap {7=(0,5825), 53=(0,5709),
60=(0,5911), 79=(0,5806), 96=(0,5754), 100=(0,5678), 102=(0,5803),
114=(0,5856), 127=(0,5695), 167=(0,5766), 179=(0,5800), 204=(0,5869),
205=(0,5761), 208=(0,5792), 212=(0,5763), 221=(0,5778), 228=(0,5797),
235=(0,5818), 239=(0,5728), 245=(0,5726), 249=(0,5754), 250=(0,5819),
253=(0,5745), 262=(0,5719), 286=(0,5728), 287=(0,5783), 294=(0,5713),
307=(0,5760), 312=(0,5702), 318=(0,5738), 323=(0,5782), 335=(0,5824),
351=(0,5826), 353=(0,5725), 367=(0,5725), 379=(0,5856), 385=(0,5801),
391=(0,5805), 402=(0,5886), 411=(0,5770), 417=(0,5764), 432=(0,5792),
442=(0,5749), 449=(0,5679), 452=(0,5722), 456=(0,5828), 470=(0,5689),
485=(0,5732), 494=(0,5837), 504=(0,5809), 505=(0,5679), 516=(0,5786),
528=(0,5793), 530=(0,5699), 557=(0,5746), 580=(0,5680), 594=(0,5830),
603=(0,5758), 608=(0,5876), 624=(0,5715), 626=(0,5743), 636=(0,5795),
674=(0,5751), 676=(0,5875), 725=(0,5780), 729=(0,5795), 741=(0,5782),
759=(0,5765), 762=(0,5728), 767=(0,5697), 777=(0,5762), 798=(0,5774),
829=(0,5705), 849=(0,5876), 853=(0,5854), 854=(0,5834), 857=(0,5660),
863=(0,5742), 864=(0,5824), 867=(0,5757), 877=(0,5680), 883=(0,5758),
905=(0,5911), 917=(0,5630), 920=(0,5832), 924=(0,5580), 939=(0,5805),
940=(0,5644), 951=(0,5726), 955=(0,5750), 963=(0,5699), 965=(0,5731),
971=(0,5707), 985=(0,5735), 1017=(0,5729)},
-1569773056=CachePartitionPartialCountersMap {7=(0,6915), 53=(0,6912),
60=(0,6825), 79=(0,6962), 96=(0,7046), 100=(0,6926), 102=(0,6813),
114=(0,6850), 127=(0,6926), 167=(0,6816), 179=(0,6795), 204=(0,6894),
205=(0,6950), 208=(0,6742), 212=(0,6939), 221=(0,6933), 228=(0,6932),
235=(0,6903), 239=(0,6816), 245=(0,6928), 249=(0,6895), 250=(0,6892),
253=(0,6934), 262=(0,6927), 286=(0,6906), 287=(0,7015), 294=(0,6781),
307=(0,6899), 312=(0,6773), 318=(0,6944), 323=(0,6999), 335=(0,6925),
351=(0,6885), 353=(0,6797), 367=(0,7074), 379=(0,7041), 385=(0,6921),
391=(0,7101), 402=(0,6858), 411=(0,7057), 417=(0,6862), 432=(0,6752),
442=(0,6911), 449=(0,6939), 452=(0,6834), 456=(0,6792), 470=(0,6936),
485=(0,6903), 494=(0,6955), 504=(0,6943), 505=(0,6893), 516=(0,6838),
528=(0,6922), 530=(0,6893), 557=(0,6849), 580=(0,6822), 594=(0,6994),
603=(0,6872), 608=(0,6862), 624=(0,6987), 626=(0,6913), 636=(0,7033),
674=(0,6921), 676=(0,6967), 725=(0,6831), 729=(0,6854), 741=(0,6853),
759=(0,6871), 762=(0,6873), 767=(0,6700), 777=(0,6765), 798=(0,6712),
829=(0,6933), 849=(0,6936), 853=(0,6967), 854=(0,6995), 857=(0,6908),
863=(0,6987), 864=(0,6872), 867=(0,6847), 877=(0,7032), 883=(0,6830),
905=(0,6790), 917=(0,6953), 920=(0,6941), 924=(0,6819), 939=(0,6859),
940=(0,6814), 951=(0,6879), 955=(0,6915), 963=(0,7002), 965=(0,6858),
971=(0,6919), 985=(0,6746), 1017=(0,6700)}},
partsSizes={-2085876587={385=5801, 262=5719, 7=5825, 391=5805, 777=5762,
905=5911, 528=5793, 402=5886, 530=5699, 920=5832, 286=5728, 798=5774,
287=5783, 294=5713, 940=5644, 557=5746, 432=5792, 179=5800, 53=5709,
312=5702, 442=5749, 955=5750, 60=5911, 318=5738, 449=5679, 323=5782,
580=5680, 456=5828, 971=5707, 205=5761, 79=5806, 335=5824, 208=5792,
594=5830, 725=5780, 853=5854, 729=5795, 857=5660, 985=5735, 603=5758,
221=5778, 351=5826, 863=5742, 96=5754, 864=5824, 353=5725, 867=5757,
100=5678, 228=5797, 485=5732, 102=5803, 235=5818, 877=5680, 494=5837,
367=5725, 624=5715, 883=5758, 245=5726, 759=5765, 504=5809, 249=5754,
505=5679, 379=5856, 636=5795, 253=5745, 127=5695, 767=5697},
-1569773056={385=6921, 262=6927, 7=6915, 391=7101, 777=6765, 905=6790,
528=6922, 402=6858, 530=6893, 920=6941, 286=6906, 798=6712, 287=7015,
294=6781, 167=6816, 940=6814, 557=6849, 432=6752, 179=6795, 53=6912,
312=6773, 442=6911, 955=6915, 60=6825, 318=6944, 449=6939, 323=6999,
580=6822, 456=6792, 971=6919, 205=6950, 79=6962, 335=6925, 208=6742,
594=6994, 725=6831, 853=6967, 854=5329, 729=6854, 857=6908, 985=6746,
603=6872, 221=6933, 351=6885, 863=6987, 96=7046, 864=6872, 353=6797,
867=6847, 100=6926, 228=6932, 485=6903, 102=6813, 235=6903, 877=7032,
494=6955, 367=7074, 624=6987, 883=6830, 245=6928, 759=6871, 504=6943,
249=6895, 505=6893, 379=7041, 636=7033, 253=6934, 127=6926, 767=6700}},
partHistCntrs=null, err=null, client=false, compress=true, finishMsg=null,
activeQryTrackers=null, super=GridDhtPartitionsAbstractMessage
[exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=327, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=00d6c796-5bce-404e-867f-3dd67e7c88f6,
addrs=[10.204.138.136, 127.0.0.1], order=258, loc=false, client=false],
topVer=327, nodeId8=654c8cd2, msg=Node failed: ZookeeperClusterNode
[id=00d6c796-5bce-404e-867f-3dd67e7c88f6, addrs=[10.204.138.136, 127.0.0.1],
order=258, loc=false, client=false], type=NODE_FAILED,
tstamp=1584624754811], nodeId=00d6c796, evt=NODE_FAILED],
lastVer=GridCacheVersion [topVer=195483760, order=1584599672955,
nodeOrder=274], super=GridCacheMessage [msgId=1510, depInfo=null, err=null,
skipPrepare=false]]]]]
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
Failed to send message (node left topology): ZookeeperClusterNode
[id=f8377bf6-7e5e-4efd-bd31-95af11edc52a, addrs=[10.204.138.219, 127.0.0.1],
order=130, loc=false, client=false]
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3270)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2713)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2672)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1656)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1731)
	at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1170)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendLocalPartitions(GridDhtPartitionsExchangeFuture.java:1880)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendPartitions(GridDhtPartitionsExchangeFuture.java:2011)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1501)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:806)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2667)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2539)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at java.lang.Thread.run(Thread.java:748)


How to keep the ignite-yarn application steady without increasing the
resources of its own?


Thanks,
Chandan




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/