Ignite 2.9.1 - An existing connection was forcibly closed by the remote host

[Charlin]

Hi,

i'm running an ASP.Net application with ignite 2.9.1 and seeing below error details in ignite client log and my web site stopped working. I have faced the same issue  with ignite 2.8.1 and I have upgraded 2.9.1 just before 1 day, since ignite 2.9.0 release notes saying below points:

1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a node fails to send a message or ping, now it drops the current connection strictly within this timeout and begins establishing a new connection much faster.

2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a node loses connection, now it strictly obtains a new connection to the ring of gets segmented within this timeout.

It's back to normal after restarting the application pool.

Please any suggestions? to avoid such issues.

Log files attached here for your reference.

Thanks & Regards,

Charlin

[ilya.kasnacheev]

Hello!

I think I've just answered the same question for 2.8.1:

https://lists.apache.org/thread.html/r102cf85775f7759e6772ceeed6d3a6843ea736b20647020fb8374833%40%3Cuser.ignite.apache.org%3E

I'm not sure why your node does not recognize itself as SEGMENTED.

[15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi] Failed to send message: null
java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Maybe there's something in your network configuration which prevents relaying of "connection closed" packets from client to server?

Can you collect thread dump from such failing client when it has already started to exhibit that behavior?

Regards,

пн, 15 февр. 2021 г. в 11:00, Charlin S <[hidden email]>:

Hi,

i'm running an ASP.Net application with ignite 2.9.1 and seeing below error details in ignite client log and my web site stopped working. I have faced the same issue  with ignite 2.8.1 and I have upgraded 2.9.1 just before 1 day, since ignite 2.9.0 release notes saying below points:

1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a node fails to send a message or ping, now it drops the current connection strictly within this timeout and begins establishing a new connection much faster.

2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a node loses connection, now it strictly obtains a new connection to the ring of gets segmented within this timeout.

It's back to normal after restarting the application pool.

Please any suggestions? to avoid such issues.

Log files attached here for your reference.

Thanks & Regards,

Charlin

[Charlin]

Hello,

Thanks for your help,

As per our network team, there were no packet drops between client and server.

Do I need to collect either jvm thread dump or IIS worker process thread dump ?

Thanks & Regards,

Charlin

On Mon, 15 Feb 2021 at 16:40, Ilya Kasnacheev <[hidden email]> wrote:

Hello!

I think I've just answered the same question for 2.8.1:

https://lists.apache.org/thread.html/r102cf85775f7759e6772ceeed6d3a6843ea736b20647020fb8374833%40%3Cuser.ignite.apache.org%3E

I'm not sure why your node does not recognize itself as SEGMENTED.

[15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi] Failed to send message: null
java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Maybe there's something in your network configuration which prevents relaying of "connection closed" packets from client to server?

Can you collect thread dump from such failing client when it has already started to exhibit that behavior?

Regards,

--

Ilya Kasnacheev

пн, 15 февр. 2021 г. в 11:00, Charlin S <[hidden email]>:

Hi,

i'm running an ASP.Net application with ignite 2.9.1 and seeing below error details in ignite client log and my web site stopped working. I have faced the same issue  with ignite 2.8.1 and I have upgraded 2.9.1 just before 1 day, since ignite 2.9.0 release notes saying below points:

1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a node fails to send a message or ping, now it drops the current connection strictly within this timeout and begins establishing a new connection much faster.

2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a node loses connection, now it strictly obtains a new connection to the ring of gets segmented within this timeout.

It's back to normal after restarting the application pool.

Please any suggestions? to avoid such issues.

Log files attached here for your reference.

Thanks & Regards,

Charlin

[ilya.kasnacheev]

Hello!

It should be JVM thread dump.

Packet drops is one thing, network misconfiguration is another.

Regards,

пн, 15 февр. 2021 г. в 15:12, Charlin S <[hidden email]>:

Hello,

Thanks for your help,

As per our network team, there were no packet drops between client and server.

Do I need to collect either jvm thread dump or IIS worker process thread dump ?

Thanks & Regards,

Charlin

On Mon, 15 Feb 2021 at 16:40, Ilya Kasnacheev <[hidden email]> wrote:

Hello!

I think I've just answered the same question for 2.8.1:

https://lists.apache.org/thread.html/r102cf85775f7759e6772ceeed6d3a6843ea736b20647020fb8374833%40%3Cuser.ignite.apache.org%3E

I'm not sure why your node does not recognize itself as SEGMENTED.

[15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi] Failed to send message: null
java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Maybe there's something in your network configuration which prevents relaying of "connection closed" packets from client to server?

Can you collect thread dump from such failing client when it has already started to exhibit that behavior?

Regards,

--

Ilya Kasnacheev

пн, 15 февр. 2021 г. в 11:00, Charlin S <[hidden email]>:

Hi,

i'm running an ASP.Net application with ignite 2.9.1 and seeing below error details in ignite client log and my web site stopped working. I have faced the same issue  with ignite 2.8.1 and I have upgraded 2.9.1 just before 1 day, since ignite 2.9.0 release notes saying below points:

1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a node fails to send a message or ping, now it drops the current connection strictly within this timeout and begins establishing a new connection much faster.

2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a node loses connection, now it strictly obtains a new connection to the ring of gets segmented within this timeout.

It's back to normal after restarting the application pool.

Please any suggestions? to avoid such issues.

Log files attached here for your reference.

Thanks & Regards,

Charlin

[Charlin]

Hello,

Thanks for your help,

I am not sure where to check the network misconfiguration either in System or in Application configuration (ignite ean file).

I am thinking to reduce interval to 1 min and retry count 5 and currently its 2 minutes and 10 times.

My current configuration is as below and please let me know if any changes are required.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
                           http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/util
                           http://www.springframework.org/schema/util/spring-util.xsd">
  <bean class="org.apache.ignite.configuration.IgniteConfiguration">
     <property name="clientMode" value="true"/>
<property name="localHost" value="10.212.120.180"/>
     <property name="clientFailureDetectionTimeout" value="60000"/>
     <property name="igniteInstanceName" value="hh_StaticGrid"/>
     <property name="longQueryWarningTimeout" value="4000"/>
     <property name="failureDetectionTimeout" value="60000"/>
     <property name="segmentationPolicy" value="RESTART_JVM"/>
     <property name="segmentCheckFrequency" value="20000"/>
     <property name="segmentationResolveAttempts" value="5"/>
     <property name="discoverySpi">
      <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localPort" value="51500"/>
        <property name="ipFinder">
          <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
            <property name="addresses">
              <list>
<value>10.212.120.66:51500</value>
<value>10.212.120.67:51500</value>
              </list>
            </property>
          </bean>
        </property>
      </bean>
    </property>
<property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
   <property name="sharedMemoryPort" value="-1"/>
            <property name="localPort" value="51605"/>
        </bean>
    </property>
  </bean>
</beans>

Thanks & Regards

Charlin

On Tue, 16 Feb 2021 at 15:00, Ilya Kasnacheev <[hidden email]> wrote:

Hello!

It should be JVM thread dump.

Packet drops is one thing, network misconfiguration is another.

Regards,

--

Ilya Kasnacheev

пн, 15 февр. 2021 г. в 15:12, Charlin S <[hidden email]>:

Hello,

Thanks for your help,

As per our network team, there were no packet drops between client and server.

Do I need to collect either jvm thread dump or IIS worker process thread dump ?

Thanks & Regards,

Charlin

On Mon, 15 Feb 2021 at 16:40, Ilya Kasnacheev <[hidden email]> wrote:

Hello!

I think I've just answered the same question for 2.8.1:

https://lists.apache.org/thread.html/r102cf85775f7759e6772ceeed6d3a6843ea736b20647020fb8374833%40%3Cuser.ignite.apache.org%3E

I'm not sure why your node does not recognize itself as SEGMENTED.

[15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi] Failed to send message: null
java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Maybe there's something in your network configuration which prevents relaying of "connection closed" packets from client to server?

Can you collect thread dump from such failing client when it has already started to exhibit that behavior?

Regards,

--

Ilya Kasnacheev

пн, 15 февр. 2021 г. в 11:00, Charlin S <[hidden email]>:

Hi,

i'm running an ASP.Net application with ignite 2.9.1 and seeing below error details in ignite client log and my web site stopped working. I have faced the same issue  with ignite 2.8.1 and I have upgraded 2.9.1 just before 1 day, since ignite 2.9.0 release notes saying below points:

1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a node fails to send a message or ping, now it drops the current connection strictly within this timeout and begins establishing a new connection much faster.

2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a node loses connection, now it strictly obtains a new connection to the ring of gets segmented within this timeout.

It's back to normal after restarting the application pool.

Please any suggestions? to avoid such issues.

Log files attached here for your reference.

Thanks & Regards,

Charlin

[ilya.kasnacheev]

Hello!

Have you gotten around collecting thread dump from the client?

Regards,

вт, 16 февр. 2021 г. в 16:59, Charlin S <[hidden email]>:

Hello,

Thanks for your help,

I am not sure where to check the network misconfiguration either in System or in Application configuration (ignite ean file).

I am thinking to reduce interval to 1 min and retry count 5 and currently its 2 minutes and 10 times.

My current configuration is as below and please let me know if any changes are required.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
                           http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/util
                           http://www.springframework.org/schema/util/spring-util.xsd">
  <bean class="org.apache.ignite.configuration.IgniteConfiguration">
     <property name="clientMode" value="true"/>
<property name="localHost" value="10.212.120.180"/>
     <property name="clientFailureDetectionTimeout" value="60000"/>
     <property name="igniteInstanceName" value="hh_StaticGrid"/>
     <property name="longQueryWarningTimeout" value="4000"/>
     <property name="failureDetectionTimeout" value="60000"/>
     <property name="segmentationPolicy" value="RESTART_JVM"/>
     <property name="segmentCheckFrequency" value="20000"/>
     <property name="segmentationResolveAttempts" value="5"/>
     <property name="discoverySpi">
      <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localPort" value="51500"/>
        <property name="ipFinder">
          <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
            <property name="addresses">
              <list>
<value>10.212.120.66:51500</value>
<value>10.212.120.67:51500</value>
              </list>
            </property>
          </bean>
        </property>
      </bean>
    </property>
<property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
   <property name="sharedMemoryPort" value="-1"/>
            <property name="localPort" value="51605"/>
        </bean>
    </property>
  </bean>
</beans>

Thanks & Regards

Charlin

On Tue, 16 Feb 2021 at 15:00, Ilya Kasnacheev <[hidden email]> wrote:

Hello!

It should be JVM thread dump.

Packet drops is one thing, network misconfiguration is another.

Regards,

--

Ilya Kasnacheev

пн, 15 февр. 2021 г. в 15:12, Charlin S <[hidden email]>:

Hello,

Thanks for your help,

As per our network team, there were no packet drops between client and server.

Do I need to collect either jvm thread dump or IIS worker process thread dump ?

Thanks & Regards,

Charlin

On Mon, 15 Feb 2021 at 16:40, Ilya Kasnacheev <[hidden email]> wrote:

Hello!

I think I've just answered the same question for 2.8.1:

https://lists.apache.org/thread.html/r102cf85775f7759e6772ceeed6d3a6843ea736b20647020fb8374833%40%3Cuser.ignite.apache.org%3E

I'm not sure why your node does not recognize itself as SEGMENTED.

[15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi] Failed to send message: null
java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Maybe there's something in your network configuration which prevents relaying of "connection closed" packets from client to server?

Can you collect thread dump from such failing client when it has already started to exhibit that behavior?

Regards,

--

Ilya Kasnacheev

пн, 15 февр. 2021 г. в 11:00, Charlin S <[hidden email]>:

Hi,

i'm running an ASP.Net application with ignite 2.9.1 and seeing below error details in ignite client log and my web site stopped working. I have faced the same issue  with ignite 2.8.1 and I have upgraded 2.9.1 just before 1 day, since ignite 2.9.0 release notes saying below points:

1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a node fails to send a message or ping, now it drops the current connection strictly within this timeout and begins establishing a new connection much faster.

2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a node loses connection, now it strictly obtains a new connection to the ring of gets segmented within this timeout.

It's back to normal after restarting the application pool.

Please any suggestions? to avoid such issues.

Log files attached here for your reference.

Thanks & Regards,

Charlin