TCP Discovery for a long time

classic Classic list List threaded Threaded
2 messages Options
38797715 38797715
Reply | Threaded
Open this post in threaded view
|

TCP Discovery for a long time

Hi,
We have five nodes, which take a long time to start up, mainly in the TCP discovery part.
We have closed the firewall.
discoverySpi config are as follows:
<property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                        <property name="addresses">
                            <list>
                                <value>10.76.236.20:47500</value>
                                <value>10.76.236.21:47500</value>
                                <value>10.76.236.22:47500</value>
                                <value>10.76.236.24:47500</value>
<value>10.76.236.71:47500</value>
<value>10.76.236.74:47500</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>
Node discovery took more than ten minutes. What is the possible cause of this problem?
the log are as follows:
[2020-05-22T21:43:44,964][INFO ][main][TcpDiscoverySpi] Successfully bound to TCP port [port=47500, localHost=0.0.0.0/0.0.0.0, locNodeId=ff7f623a-0d52-414a-994e-942bc4debc52]
[2020-05-22T21:44:58,368][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.22, rmtPort=50784]
[2020-05-22T21:44:58,377][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.22, rmtPort=50784]
[2020-05-22T21:44:58,378][INFO ][tcp-disco-sock-reader-[]-#4][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.22:50784, rmtPort=50784]
[2020-05-22T21:44:58,380][INFO ][tcp-disco-sock-reader-[917f689f 10.76.236.22:50784]-#4][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, rmtAddr=/10.76.236.22:50784]
[2020-05-22T21:44:58,391][INFO ][tcp-disco-sock-reader-[917f689f 10.76.236.22:50784]-#4][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.22:50784, rmtPort=50784
[2020-05-22T21:46:10,625][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.20, rmtPort=56571]
[2020-05-22T21:46:10,625][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.20, rmtPort=56571]
[2020-05-22T21:46:10,626][INFO ][tcp-disco-sock-reader-[]-#5][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.20:56571, rmtPort=56571]
[2020-05-22T21:46:10,627][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:56571]-#5][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=2a6a1afd-db78-4bc0-beda-37f4d1110b13, rmtAddr=/10.76.236.20:56571]
[2020-05-22T21:46:10,632][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:56571]-#5][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.20:56571, rmtPort=56571
[2020-05-22T21:46:16,843][WARN ][main][TcpDiscoverySpi] Timed out waiting for message to be read (most probably, the reason is long GC pauses on remote node) [curTimeout=120000, rmtAddr=/10.76.236.20:47500, rmtPort=47500]
[2020-05-22T21:48:11,614][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.74, rmtPort=33010]
[2020-05-22T21:48:11,614][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.74, rmtPort=33010]
[2020-05-22T21:48:11,615][INFO ][tcp-disco-sock-reader-[]-#6][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.74:33010, rmtPort=33010]
[2020-05-22T21:48:11,618][INFO ][tcp-disco-sock-reader-[e01c5114 10.76.236.74:33010]-#6][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=e01c5114-3485-41b3-b6fc-86de1d1d60c2, rmtAddr=/10.76.236.74:33010]
[2020-05-22T21:48:11,626][INFO ][tcp-disco-sock-reader-[e01c5114 10.76.236.74:33010]-#6][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.74:33010, rmtPort=33010
[2020-05-22T21:48:18,953][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=54688]
[2020-05-22T21:48:18,953][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=54688]
[2020-05-22T21:48:18,954][INFO ][tcp-disco-sock-reader-[]-#7][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:54688, rmtPort=54688]
[2020-05-22T21:48:18,955][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:54688]-#7][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:54688]
[2020-05-22T21:48:18,962][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:54688]-#7][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:54688, rmtPort=54688
[2020-05-22T21:48:20,865][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.20, rmtPort=49284]
[2020-05-22T21:48:20,865][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.20, rmtPort=49284]
[2020-05-22T21:48:20,866][INFO ][tcp-disco-sock-reader-[]-#8][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.20:49284, rmtPort=49284]
[2020-05-22T21:48:20,866][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:49284]-#8][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=2a6a1afd-db78-4bc0-beda-37f4d1110b13, rmtAddr=/10.76.236.20:49284]
[2020-05-22T21:48:20,871][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:49284]-#8][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.20:49284, rmtPort=49284
[2020-05-22T21:48:31,141][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.22, rmtPort=44604]
[2020-05-22T21:48:31,142][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.22, rmtPort=44604]
[2020-05-22T21:48:31,142][INFO ][tcp-disco-sock-reader-[]-#9][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.22:44604, rmtPort=44604]
[2020-05-22T21:48:31,143][INFO ][tcp-disco-sock-reader-[917f689f 10.76.236.22:44604]-#9][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, rmtAddr=/10.76.236.22:44604]
[2020-05-22T21:48:31,146][INFO ][tcp-disco-sock-reader-[917f689f 10.76.236.22:44604]-#9][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.22:44604, rmtPort=44604
[2020-05-22T21:48:31,148][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=47388]
[2020-05-22T21:48:31,148][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=47388]
[2020-05-22T21:48:31,149][INFO ][tcp-disco-sock-reader-[]-#10][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:47388, rmtPort=47388]
[2020-05-22T21:48:31,149][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:47388]-#10][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:47388]
[2020-05-22T21:48:31,152][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:47388]-#10][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:47388, rmtPort=47388
[2020-05-22T21:48:33,147][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.22, rmtPort=42663]
[2020-05-22T21:48:33,148][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.22, rmtPort=42663]
[2020-05-22T21:48:33,148][INFO ][tcp-disco-sock-reader-[]-#11][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.22:42663, rmtPort=42663]
[2020-05-22T21:48:33,149][INFO ][tcp-disco-sock-reader-[917f689f 10.76.236.22:42663]-#11][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, rmtAddr=/10.76.236.22:42663]
[2020-05-22T21:48:33,151][INFO ][tcp-disco-sock-reader-[917f689f 10.76.236.22:42663]-#11][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.22:42663, rmtPort=42663
[2020-05-22T21:48:36,188][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.20, rmtPort=44920]
[2020-05-22T21:48:36,189][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.20, rmtPort=44920]
[2020-05-22T21:48:36,189][INFO ][tcp-disco-sock-reader-[]-#12][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.20:44920, rmtPort=44920]
[2020-05-22T21:48:36,190][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:44920]-#12][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=2a6a1afd-db78-4bc0-beda-37f4d1110b13, rmtAddr=/10.76.236.20:44920]
[2020-05-22T21:48:36,194][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:44920]-#12][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.20:44920, rmtPort=44920
[2020-05-22T21:48:36,199][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.74, rmtPort=36247]
[2020-05-22T21:48:36,199][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.74, rmtPort=36247]
[2020-05-22T21:48:36,200][INFO ][tcp-disco-sock-reader-[]-#13][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.74:36247, rmtPort=36247]
[2020-05-22T21:48:36,200][INFO ][tcp-disco-sock-reader-[e01c5114 10.76.236.74:36247]-#13][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=e01c5114-3485-41b3-b6fc-86de1d1d60c2, rmtAddr=/10.76.236.74:36247]
[2020-05-22T21:48:36,203][INFO ][tcp-disco-sock-reader-[e01c5114 10.76.236.74:36247]-#13][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.74:36247, rmtPort=36247
[2020-05-22T21:48:37,527][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=49170]
[2020-05-22T21:48:37,527][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=49170]
[2020-05-22T21:48:37,528][INFO ][tcp-disco-sock-reader-[]-#14][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:49170, rmtPort=49170]
[2020-05-22T21:48:37,528][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:49170]-#14][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:49170]
[2020-05-22T21:48:37,531][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:49170]-#14][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:49170, rmtPort=49170
[2020-05-22T21:48:39,539][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=56933]
[2020-05-22T21:48:39,539][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=56933]
[2020-05-22T21:48:39,539][INFO ][tcp-disco-sock-reader-[]-#15][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:56933, rmtPort=56933]
[2020-05-22T21:48:39,540][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:56933]-#15][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:56933]
[2020-05-22T21:48:39,543][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:56933]-#15][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:56933, rmtPort=56933
[2020-05-22T21:48:42,195][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=60682]
[2020-05-22T21:48:42,196][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=60682]
[2020-05-22T21:48:42,196][INFO ][tcp-disco-sock-reader-[]-#16][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:60682, rmtPort=60682]
[2020-05-22T21:48:42,197][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:60682]-#16][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:60682]
[2020-05-22T21:48:42,200][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:60682]-#16][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:60682, rmtPort=60682
[2020-05-22T21:48:43,179][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.22, rmtPort=44497]
[2020-05-22T21:48:43,179][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.22, rmtPort=44497]
[2020-05-22T21:48:43,180][INFO ][tcp-disco-sock-reader-[]-#17][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.22:44497, rmtPort=44497]
[2020-05-22T21:48:43,181][INFO ][tcp-disco-sock-reader-[917f689f 10.76.236.22:44497]-#17][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, rmtAddr=/10.76.236.22:44497]
[2020-05-22T21:48:43,183][INFO ][tcp-disco-sock-reader-[917f689f 10.76.236.22:44497]-#17][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.22:44497, rmtPort=44497
[2020-05-22T21:48:46,269][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.74, rmtPort=56524]
[2020-05-22T21:48:46,269][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.74, rmtPort=56524]
[2020-05-22T21:48:46,270][INFO ][tcp-disco-sock-reader-[]-#18][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.74:56524, rmtPort=56524]
[2020-05-22T21:48:46,270][INFO ][tcp-disco-sock-reader-[e01c5114 10.76.236.74:56524]-#18][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=e01c5114-3485-41b3-b6fc-86de1d1d60c2, rmtAddr=/10.76.236.74:56524]
[2020-05-22T21:48:46,272][INFO ][tcp-disco-sock-reader-[e01c5114 10.76.236.74:56524]-#18][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.74:56524, rmtPort=56524
[2020-05-22T21:48:47,209][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=41600]
[2020-05-22T21:48:47,209][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=41600]
[2020-05-22T21:48:47,209][INFO ][tcp-disco-sock-reader-[]-#19][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:41600, rmtPort=41600]
[2020-05-22T21:48:47,210][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:41600]-#19][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:41600]
[2020-05-22T21:48:47,213][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:41600]-#19][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:41600, rmtPort=41600
[2020-05-22T21:48:51,489][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.20, rmtPort=58689]
[2020-05-22T21:48:51,490][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.20, rmtPort=58689]
[2020-05-22T21:48:51,490][INFO ][tcp-disco-sock-reader-[]-#20][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.20:58689, rmtPort=58689]
[2020-05-22T21:48:51,491][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:58689]-#20][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=2a6a1afd-db78-4bc0-beda-37f4d1110b13, rmtAddr=/10.76.236.20:58689]
[2020-05-22T21:48:51,495][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:58689]-#20][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.20:58689, rmtPort=58689
[2020-05-22T21:48:52,233][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=41523]
[2020-05-22T21:48:52,233][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=41523]
[2020-05-22T21:48:52,233][INFO ][tcp-disco-sock-reader-[]-#21][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:41523, rmtPort=41523]
[2020-05-22T21:48:52,234][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:41523]-#21][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:41523]
[2020-05-22T21:48:52,237][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:41523]-#21][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:41523, rmtPort=41523
[2020-05-22T21:49:08,597][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=56637]
[2020-05-22T21:49:08,598][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=56637]
[2020-05-22T21:49:08,598][INFO ][tcp-disco-sock-reader-[]-#22][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:56637, rmtPort=56637]
[2020-05-22T21:49:08,599][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:56637]-#22][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:56637]
[2020-05-22T21:49:08,602][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:56637]-#22][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:56637, rmtPort=56637
[2020-05-22T21:49:13,638][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=37755]
[2020-05-22T21:49:13,638][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=37755]
[2020-05-22T21:49:13,639][INFO ][tcp-disco-sock-reader-[]-#23][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:37755, rmtPort=37755]
[2020-05-22T21:49:13,639][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:37755]-#23][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:37755]
[2020-05-22T21:49:13,641][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:37755]-#23][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:37755, rmtPort=37755
[2020-05-22T21:49:18,657][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=40749]
[2020-05-22T21:49:18,658][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=40749]
[2020-05-22T21:49:18,658][INFO ][tcp-disco-sock-reader-[]-#24][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:40749, rmtPort=40749]
[2020-05-22T21:49:18,659][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:40749]-#24][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:40749]
[2020-05-22T21:49:18,661][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:40749]-#24][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:40749, rmtPort=40749
[2020-05-22T21:49:21,639][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.20, rmtPort=36959]
[2020-05-22T21:49:21,639][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.20, rmtPort=36959]
[2020-05-22T21:49:21,640][INFO ][tcp-disco-sock-reader-[]-#25][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.20:36959, rmtPort=36959]
[2020-05-22T21:49:21,640][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:36959]-#25][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=2a6a1afd-db78-4bc0-beda-37f4d1110b13, rmtAddr=/10.76.236.20:36959]
[2020-05-22T21:49:21,644][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:36959]-#25][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.20:36959, rmtPort=36959
[2020-05-22T21:49:37,759][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=49093]
[2020-05-22T21:49:37,759][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=49093]
[2020-05-22T21:49:37,759][INFO ][tcp-disco-sock-reader-[]-#26][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:49093, rmtPort=49093]
[2020-05-22T21:49:37,760][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:49093]-#26][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:49093]
[2020-05-22T21:49:37,763][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:49093]-#26][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:49093, rmtPort=49093
[2020-05-22T21:49:59,158][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.20, rmtPort=60475]
[2020-05-22T21:49:59,159][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.20, rmtPort=60475]
[2020-05-22T21:49:59,159][INFO ][tcp-disco-sock-reader-[]-#27][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.20:60475, rmtPort=60475]
[2020-05-22T21:49:59,160][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:60475]-#27][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=2a6a1afd-db78-4bc0-beda-37f4d1110b13, rmtAddr=/10.76.236.20:60475]
[2020-05-22T21:49:59,164][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:60475]-#27][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.20:60475, rmtPort=60475
[2020-05-22T21:49:59,415][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=46407]
[2020-05-22T21:49:59,415][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=46407]
[2020-05-22T21:49:59,416][INFO ][tcp-disco-sock-reader-[]-#28][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:46407, rmtPort=46407]
[2020-05-22T21:49:59,416][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:46407]-#28][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:46407]
[2020-05-22T21:49:59,419][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:46407]-#28][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:46407, rmtPort=46407
[2020-05-22T21:50:10,444][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=35759]
[2020-05-22T21:50:10,445][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=35759]
[2020-05-22T21:50:10,445][INFO ][tcp-disco-sock-reader-[]-#29][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:35759, rmtPort=35759]
[2020-05-22T21:50:10,446][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:35759]-#29][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:35759]
[2020-05-22T21:50:10,450][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:35759]-#29][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:35759, rmtPort=35759
[2020-05-22T21:50:14,250][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.20, rmtPort=58320]
[2020-05-22T21:50:14,250][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.20, rmtPort=58320]
[2020-05-22T21:50:14,251][INFO ][tcp-disco-sock-reader-[]-#30][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.20:58320, rmtPort=58320]
[2020-05-22T21:50:14,251][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:58320]-#30][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=2a6a1afd-db78-4bc0-beda-37f4d1110b13, rmtAddr=/10.76.236.20:58320]
[2020-05-22T21:50:14,255][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:58320]-#30][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.20:58320, rmtPort=58320
[2020-05-22T21:50:24,556][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=56397]
[2020-05-22T21:50:24,556][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=56397]
[2020-05-22T21:50:24,557][INFO ][tcp-disco-sock-reader-[]-#31][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:56397, rmtPort=56397]
[2020-05-22T21:50:24,557][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:56397]-#31][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:56397]
[2020-05-22T21:50:24,560][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:56397]-#31][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:56397, rmtPort=56397
[2020-05-22T21:50:29,340][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.20, rmtPort=57054]
[2020-05-22T21:50:29,341][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.20, rmtPort=57054]
[2020-05-22T21:50:29,341][INFO ][tcp-disco-sock-reader-[]-#32][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.20:57054, rmtPort=57054]
[2020-05-22T21:50:29,342][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:57054]-#32][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=2a6a1afd-db78-4bc0-beda-37f4d1110b13, rmtAddr=/10.76.236.20:57054]
[2020-05-22T21:50:29,346][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:57054]-#32][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.20:57054, rmtPort=57054
[2020-05-22T21:50:37,624][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=56382]
[2020-05-22T21:50:37,624][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=56382]
[2020-05-22T21:50:37,625][INFO ][tcp-disco-sock-reader-[]-#33][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:56382, rmtPort=56382]
[2020-05-22T21:50:37,626][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:56382]-#33][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:56382]
[2020-05-22T21:50:37,628][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:56382]-#33][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:56382, rmtPort=56382
[2020-05-22T21:50:39,386][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.20, rmtPort=57065]
[2020-05-22T21:50:39,386][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.20, rmtPort=57065]
[2020-05-22T21:50:39,387][INFO ][tcp-disco-sock-reader-[]-#34][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.20:57065, rmtPort=57065]
[2020-05-22T21:50:39,388][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:57065]-#34][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=2a6a1afd-db78-4bc0-beda-37f4d1110b13, rmtAddr=/10.76.236.20:57065]
[2020-05-22T21:50:39,393][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:57065]-#34][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.20:57065, rmtPort=57065
[2020-05-22T21:50:44,439][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.20, rmtPort=48951]
[2020-05-22T21:50:44,440][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.20, rmtPort=48951]
[2020-05-22T21:50:44,440][INFO ][tcp-disco-sock-reader-[]-#35][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.20:48951, rmtPort=48951]
[2020-05-22T21:50:44,441][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:48951]-#35][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=2a6a1afd-db78-4bc0-beda-37f4d1110b13, rmtAddr=/10.76.236.20:48951]
[2020-05-22T21:50:44,445][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:48951]-#35][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.20:48951, rmtPort=48951
[2020-05-22T21:50:49,384][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.22, rmtPort=50869]
[2020-05-22T21:50:49,385][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.22, rmtPort=50869]
[2020-05-22T21:50:49,385][INFO ][tcp-disco-sock-reader-[]-#36][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.22:50869, rmtPort=50869]
[2020-05-22T21:50:49,386][INFO ][tcp-disco-sock-reader-[]-#36][TcpDiscoverySpi] Received ping request from the remote node [rmtNodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, rmtAddr=/10.76.236.22:50869, rmtPort=50869]
[2020-05-22T21:50:49,388][INFO ][tcp-disco-sock-reader-[]-#36][TcpDiscoverySpi] Finished writing ping response [rmtNodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, rmtAddr=/10.76.236.22:50869, rmtPort=50869]
[2020-05-22T21:50:49,389][INFO ][tcp-disco-sock-reader-[]-#36][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.22:50869, rmtPort=50869
[2020-05-22T21:50:54,389][WARN ][main][TcpDiscoverySpi] Node has not been connected to topology and will repeat join process. Check remote nodes logs for possible error messages. Note that large topology may require significant time to start. Increase 'TcpDiscoverySpi.networkTimeout' configuration property if getting this message on the starting nodes [networkTimeout=5000]
[2020-05-22T21:50:55,485][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.20, rmtPort=59615]
[2020-05-22T21:50:55,485][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.20, rmtPort=59615]
[2020-05-22T21:50:55,485][INFO ][tcp-disco-sock-reader-[]-#37][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.20:59615, rmtPort=59615]
[2020-05-22T21:50:55,486][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:59615]-#37][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=2a6a1afd-db78-4bc0-beda-37f4d1110b13, rmtAddr=/10.76.236.20:59615]
[2020-05-22T21:50:55,490][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:59615]-#37][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.20:59615, rmtPort=59615
[2020-05-22T21:50:55,705][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=57279]
[2020-05-22T21:50:55,705][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=57279]
[2020-05-22T21:50:55,706][INFO ][tcp-disco-sock-reader-[]-#38][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:57279, rmtPort=57279]
[2020-05-22T21:50:55,706][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:57279]-#38][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:57279]
[2020-05-22T21:50:55,709][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:57279]-#38][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:57279, rmtPort=57279
[2020-05-22T21:51:00,501][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.20, rmtPort=37206]
[2020-05-22T21:51:00,502][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.20, rmtPort=37206]
[2020-05-22T21:51:00,502][INFO ][tcp-disco-sock-reader-[]-#39][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.20:37206, rmtPort=37206]
[2020-05-22T21:51:00,503][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:37206]-#39][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=2a6a1afd-db78-4bc0-beda-37f4d1110b13, rmtAddr=/10.76.236.20:37206]
[2020-05-22T21:51:00,507][INFO ][tcp-disco-sock-reader-[2a6a1afd 10.76.236.20:37206]-#39][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.20:37206, rmtPort=37206
[2020-05-22T21:51:08,773][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=54462]
[2020-05-22T21:51:08,773][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=54462]
[2020-05-22T21:51:08,773][INFO ][tcp-disco-sock-reader-[]-#40][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:54462, rmtPort=54462]
[2020-05-22T21:51:08,774][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:54462]-#40][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:54462]
[2020-05-22T21:51:08,777][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:54462]-#40][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:54462, rmtPort=54462
[2020-05-22T21:51:42,905][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=37570]
[2020-05-22T21:51:42,905][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=37570]
[2020-05-22T21:51:42,906][INFO ][tcp-disco-sock-reader-[]-#41][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:37570, rmtPort=37570]
[2020-05-22T21:51:42,906][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:37570]-#41][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:37570]
[2020-05-22T21:51:42,909][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:37570]-#41][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:37570, rmtPort=37570
[2020-05-22T21:51:50,955][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=60477]
[2020-05-22T21:51:50,956][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=60477]
[2020-05-22T21:51:50,956][INFO ][tcp-disco-sock-reader-[]-#42][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:60477, rmtPort=60477]
[2020-05-22T21:51:50,957][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:60477]-#42][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:60477]
[2020-05-22T21:51:50,959][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:60477]-#42][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:60477, rmtPort=60477
[2020-05-22T21:51:55,970][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=51585]
[2020-05-22T21:51:55,970][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=51585]
[2020-05-22T21:51:55,970][INFO ][tcp-disco-sock-reader-[]-#43][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:51585, rmtPort=51585]
[2020-05-22T21:51:55,971][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:51585]-#43][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:51585]
[2020-05-22T21:51:55,974][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:51585]-#43][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:51585, rmtPort=51585
[2020-05-22T21:52:07,027][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=59986]
[2020-05-22T21:52:07,028][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=59986]
[2020-05-22T21:52:07,028][INFO ][tcp-disco-sock-reader-[]-#44][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:59986, rmtPort=59986]
[2020-05-22T21:52:07,029][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:59986]-#44][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:59986]
[2020-05-22T21:52:07,031][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:59986]-#44][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:59986, rmtPort=59986
[2020-05-22T21:52:12,046][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=54815]
[2020-05-22T21:52:12,046][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=54815]
[2020-05-22T21:52:12,046][INFO ][tcp-disco-sock-reader-[]-#45][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:54815, rmtPort=54815]
[2020-05-22T21:52:12,047][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:54815]-#45][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:54815]
[2020-05-22T21:52:12,049][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:54815]-#45][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:54815, rmtPort=54815
[2020-05-22T21:52:20,106][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=41882]
[2020-05-22T21:52:20,106][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=41882]
[2020-05-22T21:52:20,107][INFO ][tcp-disco-sock-reader-[]-#46][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:41882, rmtPort=41882]
[2020-05-22T21:52:20,108][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:41882]-#46][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:41882]
[2020-05-22T21:52:20,110][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:41882]-#46][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:41882, rmtPort=41882
[2020-05-22T21:52:40,199][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=49865]
[2020-05-22T21:52:40,199][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=49865]
[2020-05-22T21:52:40,199][INFO ][tcp-disco-sock-reader-[]-#47][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:49865, rmtPort=49865]
[2020-05-22T21:52:40,200][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:49865]-#47][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:49865]
[2020-05-22T21:52:40,202][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:49865]-#47][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:49865, rmtPort=49865
[2020-05-22T21:52:51,232][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=45084]
[2020-05-22T21:52:51,232][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=45084]
[2020-05-22T21:52:51,233][INFO ][tcp-disco-sock-reader-[]-#48][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:45084, rmtPort=45084]
[2020-05-22T21:52:51,233][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:45084]-#48][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:45084]
[2020-05-22T21:52:51,235][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:45084]-#48][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:45084, rmtPort=45084
[2020-05-22T21:52:57,411][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.22, rmtPort=60304]
[2020-05-22T21:52:57,412][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.22, rmtPort=60304]
[2020-05-22T21:52:57,412][INFO ][tcp-disco-sock-reader-[]-#49][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.22:60304, rmtPort=60304]
[2020-05-22T21:52:57,412][WARN ][main][TcpDiscoverySpi] Timed out waiting for message delivery receipt (most probably, the reason is in long GC pauses on remote node; consider tuning GC and increasing 'ackTimeout' configuration property). Will retry to send message with increased timeout [currentTimeout=119998, rmtAddr=/10.76.236.22:47500, rmtPort=47500]
[2020-05-22T21:52:57,413][INFO ][tcp-disco-sock-reader-[]-#49][TcpDiscoverySpi] Received ping request from the remote node [rmtNodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, rmtAddr=/10.76.236.22:60304, rmtPort=60304]
[2020-05-22T21:52:57,413][INFO ][tcp-disco-sock-reader-[]-#49][TcpDiscoverySpi] Finished writing ping response [rmtNodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, rmtAddr=/10.76.236.22:60304, rmtPort=60304]
[2020-05-22T21:52:57,413][INFO ][tcp-disco-sock-reader-[]-#49][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.22:60304, rmtPort=60304
[2020-05-22T21:53:02,416][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.74, rmtPort=33813]
[2020-05-22T21:53:02,416][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.74, rmtPort=33813]
[2020-05-22T21:53:02,417][INFO ][tcp-disco-sock-reader-[]-#50][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.74:33813, rmtPort=33813]
[2020-05-22T21:53:02,418][INFO ][tcp-disco-sock-reader-[e01c5114 10.76.236.74:33813]-#50][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=e01c5114-3485-41b3-b6fc-86de1d1d60c2, rmtAddr=/10.76.236.74:33813]
[2020-05-22T21:53:02,420][INFO ][tcp-disco-sock-reader-[e01c5114 10.76.236.74:33813]-#50][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.74:33813, rmtPort=33813
[2020-05-22T21:53:14,309][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=60188]
[2020-05-22T21:53:14,309][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=60188]
[2020-05-22T21:53:14,310][INFO ][tcp-disco-sock-reader-[]-#51][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:60188, rmtPort=60188]
[2020-05-22T21:53:14,311][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:60188]-#51][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:60188]
[2020-05-22T21:53:14,313][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:60188]-#51][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:60188, rmtPort=60188
[2020-05-22T21:53:25,369][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=43318]
[2020-05-22T21:53:25,369][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=43318]
[2020-05-22T21:53:25,369][INFO ][tcp-disco-sock-reader-[]-#52][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:43318, rmtPort=43318]
[2020-05-22T21:53:25,370][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:43318]-#52][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:43318]
[2020-05-22T21:53:25,372][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:43318]-#52][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:43318, rmtPort=43318
[2020-05-22T21:53:35,421][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=45593]
[2020-05-22T21:53:35,422][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=45593]
[2020-05-22T21:53:35,422][INFO ][tcp-disco-sock-reader-[]-#53][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:45593, rmtPort=45593]
[2020-05-22T21:53:35,423][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:45593]-#53][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:45593]
[2020-05-22T21:53:35,425][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:45593]-#53][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:45593, rmtPort=45593
[2020-05-22T21:53:46,455][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=33125]
[2020-05-22T21:53:46,455][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=33125]
[2020-05-22T21:53:46,456][INFO ][tcp-disco-sock-reader-[]-#54][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:33125, rmtPort=33125]
[2020-05-22T21:53:46,456][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:33125]-#54][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:33125]
[2020-05-22T21:53:46,459][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:33125]-#54][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:33125, rmtPort=33125
[2020-05-22T21:53:51,489][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=40056]
[2020-05-22T21:53:51,490][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=40056]
[2020-05-22T21:53:51,490][INFO ][tcp-disco-sock-reader-[]-#55][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:40056, rmtPort=40056]
[2020-05-22T21:53:51,491][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:40056]-#55][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:40056]
[2020-05-22T21:53:51,493][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:40056]-#55][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:40056, rmtPort=40056
[2020-05-22T21:53:56,510][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=53217]
[2020-05-22T21:53:56,511][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=53217]
[2020-05-22T21:53:56,511][INFO ][tcp-disco-sock-reader-[]-#56][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:53217, rmtPort=53217]
[2020-05-22T21:53:56,512][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:53217]-#56][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:53217]
[2020-05-22T21:53:56,514][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:53217]-#56][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:53217, rmtPort=53217
[2020-05-22T21:54:09,568][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=42237]
[2020-05-22T21:54:09,568][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=42237]
[2020-05-22T21:54:09,569][INFO ][tcp-disco-sock-reader-[]-#57][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:42237, rmtPort=42237]
[2020-05-22T21:54:09,570][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:42237]-#57][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:42237]
[2020-05-22T21:54:09,572][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:42237]-#57][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:42237, rmtPort=42237
[2020-05-22T21:54:17,626][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=48288]
[2020-05-22T21:54:17,626][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=48288]
[2020-05-22T21:54:17,627][INFO ][tcp-disco-sock-reader-[]-#58][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:48288, rmtPort=48288]
[2020-05-22T21:54:17,627][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:48288]-#58][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:48288]
[2020-05-22T21:54:17,629][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:48288]-#58][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:48288, rmtPort=48288
[2020-05-22T21:54:45,739][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=38752]
[2020-05-22T21:54:45,739][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=38752]
[2020-05-22T21:54:45,739][INFO ][tcp-disco-sock-reader-[]-#59][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:38752, rmtPort=38752]
[2020-05-22T21:54:45,740][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:38752]-#59][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:38752]
[2020-05-22T21:54:45,742][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:38752]-#59][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:38752, rmtPort=38752
[2020-05-22T21:55:01,808][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=51975]
[2020-05-22T21:55:01,808][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=51975]
[2020-05-22T21:55:01,808][INFO ][tcp-disco-sock-reader-[]-#60][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:51975, rmtPort=51975]
[2020-05-22T21:55:01,809][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:51975]-#60][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:51975]
[2020-05-22T21:55:01,811][INFO ][tcp-disco-sock-reader-[3465ccdd 10.76.236.24:51975]-#60][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:51975, rmtPort=51975
[2020-05-22T21:55:05,514][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.22, rmtPort=38206]
[2020-05-22T21:55:05,515][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.22, rmtPort=38206]
[2020-05-22T21:55:05,515][INFO ][tcp-disco-sock-reader-[]-#61][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.22:38206, rmtPort=38206]
[2020-05-22T21:55:05,515][INFO ][tcp-disco-sock-reader-[]-#61][TcpDiscoverySpi] Received ping request from the remote node [rmtNodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, rmtAddr=/10.76.236.22:38206, rmtPort=38206]
[2020-05-22T21:55:05,516][INFO ][tcp-disco-sock-reader-[]-#61][TcpDiscoverySpi] Finished writing ping response [rmtNodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, rmtAddr=/10.76.236.22:38206, rmtPort=38206]
[2020-05-22T21:55:05,516][INFO ][tcp-disco-sock-reader-[]-#61][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.22:38206, rmtPort=38206
[2020-05-22T21:57:02,140][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.74, rmtPort=47749]
[2020-05-22T21:57:02,140][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.74, rmtPort=47749]
[2020-05-22T21:57:02,140][INFO ][tcp-disco-sock-reader-[]-#62][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.74:47749, rmtPort=47749]
[2020-05-22T21:57:02,141][INFO ][tcp-disco-sock-reader-[e01c5114 10.76.236.74:47749]-#62][TcpDiscoverySpi] Initialized connection with remote server node [nodeId=e01c5114-3485-41b3-b6fc-86de1d1d60c2, rmtAddr=/10.76.236.74:47749]
[2020-05-22T21:57:10,620][WARN ][main][TcpDiscoverySpi] Timed out waiting for message delivery receipt (most probably, the reason is in long GC pauses on remote node; consider tuning GC and increasing 'ackTimeout' configuration property). Will retry to send message with increased timeout [currentTimeout=119998, rmtAddr=/10.76.236.24:47500, rmtPort=47500]
[2020-05-22T21:57:15,603][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.24, rmtPort=57287]
[2020-05-22T21:57:15,603][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.24, rmtPort=57287]
[2020-05-22T21:57:15,604][INFO ][tcp-disco-sock-reader-[]-#63][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.24:57287, rmtPort=57287]
[2020-05-22T21:57:15,604][INFO ][tcp-disco-sock-reader-[]-#63][TcpDiscoverySpi] Received ping request from the remote node [rmtNodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:57287, rmtPort=57287]
[2020-05-22T21:57:15,604][INFO ][tcp-disco-sock-reader-[]-#63][TcpDiscoverySpi] Finished writing ping response [rmtNodeId=3465ccdd-52da-49dc-9d35-7bd7d1f010ed, rmtAddr=/10.76.236.24:57287, rmtPort=57287]
[2020-05-22T21:57:15,605][INFO ][tcp-disco-sock-reader-[]-#63][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.24:57287, rmtPort=57287
[2020-05-22T21:57:15,660][WARN ][main][TcpDiscoverySpi] Node has not been connected to topology and will repeat join process. Check remote nodes logs for possible error messages. Note that large topology may require significant time to start. Increase 'TcpDiscoverySpi.networkTimeout' configuration property if getting this message on the starting nodes [networkTimeout=5000]
[2020-05-22T21:59:02,373][INFO ][disco-notifier-worker-#65][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], nodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, ver=1590155438679, local=false, initialized=false]]
[2020-05-22T21:59:02,379][INFO ][disco-notifier-worker-#65][GridClusterStateProcessor] Sending activate request with BaselineTopology BaselineTopology [id=0, branchingHash=8311, branchingType='Cluster activation', baselineNodes=[2224717420]]
[2020-05-22T21:59:02,388][INFO ][sys-#66][ClusterProcessor] Writing cluster ID and tag to metastorage on ready for write ClusterIdAndTag [id=30f98a7b-77c6-409b-9672-d58920f7990d, tag=adoring_turing]
[2020-05-22T21:59:15,740][WARN ][main][TcpDiscoverySpi] Timed out waiting for message delivery receipt (most probably, the reason is in long GC pauses on remote node; consider tuning GC and increasing 'ackTimeout' configuration property). Will retry to send message with increased timeout [currentTimeout=119998, rmtAddr=/10.76.236.22:47500, rmtPort=47500]
[2020-05-22T21:59:15,781][WARN ][main][IgniteKernal] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=89188MB, available=110597MB]
[2020-05-22T21:59:15,789][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.76.236.22, rmtPort=42218]
[2020-05-22T21:59:15,790][INFO ][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.76.236.22, rmtPort=42218]
[2020-05-22T21:59:15,790][INFO ][tcp-disco-sock-reader-[]-#64][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.76.236.22:42218, rmtPort=42218]
[2020-05-22T21:59:15,790][INFO ][tcp-disco-sock-reader-[]-#64][TcpDiscoverySpi] Received ping request from the remote node [rmtNodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, rmtAddr=/10.76.236.22:42218, rmtPort=42218]
[2020-05-22T21:59:15,790][INFO ][tcp-disco-sock-reader-[]-#64][TcpDiscoverySpi] Finished writing ping response [rmtNodeId=917f689f-753b-4715-bbe8-ed5e9e9024b9, rmtAddr=/10.76.236.22:42218, rmtPort=42218]
[2020-05-22T21:59:15,791][INFO ][tcp-disco-sock-reader-[]-#64][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.76.236.22:42218, rmtPort=42218


akorensh akorensh
Reply | Threaded
Open this post in threaded view
|

Re: TCP Discovery for a long time

Hi,
   Looks like node 10.76.236.24 is constantly re-connecting. (see below)
   Leave only one address in the discovery section on both nodes and try to
connect.
   Then add another address discovery address and retry. You will find the
node that is causing the issue.

    Make sure your ports are open and that there is no load-balancing turned
on.
Thanks, Alex


  [2020-05-22T21:57:15,603][INFO
][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted
incoming connection [rmtAddr=/10.76.236.24, rmtPort=57287]
[2020-05-22T21:57:15,603][INFO
][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new
thread for connection [rmtAddr=/10.76.236.24, rmtPort=57287]





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/