Ignite 2.7.0 : thick client has all system threads busy indefinitely

classic Classic list List threaded Threaded
4 messages Options
Mahesh Renduchintala Mahesh Renduchintala
Reply | Threaded
Open this post in threaded view
|

Ignite 2.7.0 : thick client has all system threads busy indefinitely

Hi, 


Please see the attached log. 


A ignite client (think client) completely goes 


FROM 

>>System thread pool [active=0, idle=0, qSize=0]


TO

>>^-- System thread pool [active=8, idle=0, qSize=77]


When I checked the ignite cluster, we see that this node is continuously connecting to ignite with these logs.


[06:20:16,203][INFO][grid-nio-worker-tcp-comm-7-#47][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/192.168.1.6:47100, rmtAddr=/192.168.1.22:32935]
[06:20:16,203][INFO][grid-nio-worker-tcp-comm-7-#47][TcpCommunicationSpi] Received incoming connection from remote node while connecting to this node, rejecting [locNode=527c8173-b079-459a-8bb5-260f62d47797, locNodeOrder=1, rmtNode=a89f3605-fe0b-4abf-8d98-245d75205743, rmtNodeOrder=587]
[06:20:16,405][INFO][grid-nio-worker-tcp-comm-0-#40][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/192.168.1.6:47100, rmtAddr=/192.168.1.22:38152]
[06:20:16,405][INFO][grid-nio-worker-tcp-comm-0-#40][TcpCommunicationSpi] Received incoming connection from remote node while connecting to this node, rejecting [locNode=527c8173-b079-459a-8bb5-260f62d47797, locNodeOrder=1, rmtNode=a89f3605-fe0b-4abf-8d98-245d75205743, rmtNodeOrder=587]
[06:20:16,607][INFO][grid-nio-worker-tcp-comm-1-#41][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/192.168.1.6:47100, rmtAddr=/192.168.1.22:49095]
[06:20:16,607][INFO][grid-nio-worker-tcp-comm-1-#41][TcpCommunicationSpi] Received incoming connection from remote node while connecting to this node, rejecting [locNode=527c8173-b079-459a-8bb5-260f62d47797, locNodeOrder=1, rmtNode=a89f3605-fe0b-4abf-8d98-245d75205743, rmtNodeOrder=587]

is there any known issue around this?

regards
Mahesh





poolthreadstarvation.log (343K) Download Attachment
Vladimir Pligin Vladimir Pligin
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.7.0 : thick client has all system threads busy indefinitely

Hi, could you please share your configurations of both server and client?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Mahesh Renduchintala Mahesh Renduchintala
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.7.0 : thick client has all system threads busy indefinitely

please see client and server configs. 




config.zip (4K) Download Attachment
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.7.0 : thick client has all system threads busy indefinitely

Hello!

I can see you have large network timeouts. Sometimes it makes sense, but it may also cause nodes to have half open connections (one side thinks it's OK but other already disconnected), which would cause errors like this one.

You should also investigate why connection was closed in the first place.

Regards,
--
Ilya Kasnacheev


пн, 22 июл. 2019 г. в 20:51, Mahesh Renduchintala <[hidden email]>:

please see client and server configs.