tcp-comm-worker-#1 attempt to connect to unrecognized IP addresses

classic Classic list List threaded Threaded
2 messages Options
Conrad Mukai (cmukai) Conrad Mukai (cmukai)
Reply | Threaded
Open this post in threaded view
|

tcp-comm-worker-#1 attempt to connect to unrecognized IP addresses

I am running a 4 node ignite cluster in AWS. We are running ignite inside of Docker containers (the image is apacheignite/ignite:2.7.6 from Dockerhub). The container is setup to use the host network.

 

I noticed a lot of warning from the tcp-comm-worker-#1 thread in which it is attempting to connect to unrecognized IP addresses:

 

[2019-11-08 22:38:24,205][WARN ][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=ip-172-17-0-1.ec2.internal/172.17.0.1:47104, failureDetectionTimeout=300000]

[2019-11-08 22:40:31,501][WARN ][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/9.0.1.1:47104, failureDetectionTimeout=300000]

[2019-11-08 22:44:40,973][WARN ][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/198.51.100.3:47105, failureDetectionTimeout=300000]

[2019-11-08 22:45:03,931][WARN ][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=ip-172-17-0-1.ec2.internal/172.17.0.1:47104, failureDetectionTimeout=300000]

[2019-11-08 22:47:11,245][WARN ][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=ip-172-31-254-1.ec2.internal/172.31.254.1:47104, failureDetectionTimeout=300000]

[2019-11-08 22:50:00,333][WARN ][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/198.51.100.1:47109, failureDetectionTimeout=300000]

 

These IP addresses were not configured in the discoverySpi. I recognize 172.17.0.1 as part of the default Docker bridge network, but as I mentioned earlier we are using the host network. Also 172.31.0.0/16 is the default VPC in AWS, but we are not using that VPC. I have no idea why these IP addresses are being accessed. Can someone explain what is going on?

 

Thanks in advance,

Conrad

 

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: tcp-comm-worker-#1 attempt to connect to unrecognized IP addresses

Hello!

There is a known issue with Docker gateway - it is present on all machines, leading Ignite nodes to believe they can reach each other via this interface.

Perhaps the same reasoning for other addresses.

You can set IgniteConfiguration.localHost or TcpCommunicationSpi.localAddress to the desired address to mitigate this. The caveat is, you have to know specific local address to supply it in configuration.

Regards,
--
Ilya Kasnacheev


сб, 9 нояб. 2019 г. в 03:11, Conrad Mukai (cmukai) <[hidden email]>:

I am running a 4 node ignite cluster in AWS. We are running ignite inside of Docker containers (the image is apacheignite/ignite:2.7.6 from Dockerhub). The container is setup to use the host network.

 

I noticed a lot of warning from the tcp-comm-worker-#1 thread in which it is attempting to connect to unrecognized IP addresses:

 

[2019-11-08 22:38:24,205][WARN ][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=ip-172-17-0-1.ec2.internal/172.17.0.1:47104, failureDetectionTimeout=300000]

[2019-11-08 22:40:31,501][WARN ][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/9.0.1.1:47104, failureDetectionTimeout=300000]

[2019-11-08 22:44:40,973][WARN ][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/198.51.100.3:47105, failureDetectionTimeout=300000]

[2019-11-08 22:45:03,931][WARN ][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=ip-172-17-0-1.ec2.internal/172.17.0.1:47104, failureDetectionTimeout=300000]

[2019-11-08 22:47:11,245][WARN ][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=ip-172-31-254-1.ec2.internal/172.31.254.1:47104, failureDetectionTimeout=300000]

[2019-11-08 22:50:00,333][WARN ][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/198.51.100.1:47109, failureDetectionTimeout=300000]

 

These IP addresses were not configured in the discoverySpi. I recognize 172.17.0.1 as part of the default Docker bridge network, but as I mentioned earlier we are using the host network. Also 172.31.0.0/16 is the default VPC in AWS, but we are not using that VPC. I have no idea why these IP addresses are being accessed. Can someone explain what is going on?

 

Thanks in advance,

Conrad