Ignite 2.9.1 - An existing connection was forcibly closed by the remote host

classic Classic list List threaded Threaded
6 messages Options
Charlin Charlin
Reply | Threaded
Open this post in threaded view
|

Ignite 2.9.1 - An existing connection was forcibly closed by the remote host

Hi,

i'm running an ASP.Net application with ignite 2.9.1 and seeing below error details in ignite client log and my web site stopped working. I have faced the same issue  with ignite 2.8.1 and I have upgraded 2.9.1 just before 1 day, since ignite 2.9.0 release notes saying below points:
1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a node fails to send a message or ping, now it drops the current connection strictly within this timeout and begins establishing a new connection much faster.
2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a node loses connection, now it strictly obtains a new connection to the ring of gets segmented within this timeout.

It's back to normal after restarting the application pool.
Please any suggestions? to avoid such issues.
Log files attached here for your reference.

Thanks & Regards,
Charlin


14Feb2021ErrorClientNodeStoppedWorking.zip (1M) Download Attachment
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.9.1 - An existing connection was forcibly closed by the remote host

Hello!

I think I've just answered the same question for 2.8.1:

I'm not sure why your node does not recognize itself as SEGMENTED.
[15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi] Failed to send message: null
java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Maybe there's something in your network configuration which prevents relaying of "connection closed" packets from client to server?
Can you collect thread dump from such failing client when it has already started to exhibit that behavior?

Regards,
--
Ilya Kasnacheev


пн, 15 февр. 2021 г. в 11:00, Charlin S <[hidden email]>:
Hi,

i'm running an ASP.Net application with ignite 2.9.1 and seeing below error details in ignite client log and my web site stopped working. I have faced the same issue  with ignite 2.8.1 and I have upgraded 2.9.1 just before 1 day, since ignite 2.9.0 release notes saying below points:
1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a node fails to send a message or ping, now it drops the current connection strictly within this timeout and begins establishing a new connection much faster.
2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a node loses connection, now it strictly obtains a new connection to the ring of gets segmented within this timeout.

It's back to normal after restarting the application pool.
Please any suggestions? to avoid such issues.
Log files attached here for your reference.

Thanks & Regards,
Charlin

Charlin Charlin
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.9.1 - An existing connection was forcibly closed by the remote host

Hello,
Thanks for your help,
As per our network team, there were no packet drops between client and server.
Do I need to collect either jvm thread dump or IIS worker process thread dump ?

Thanks & Regards,
Charlin



On Mon, 15 Feb 2021 at 16:40, Ilya Kasnacheev <[hidden email]> wrote:
Hello!

I think I've just answered the same question for 2.8.1:

I'm not sure why your node does not recognize itself as SEGMENTED.
[15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi] Failed to send message: null
java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Maybe there's something in your network configuration which prevents relaying of "connection closed" packets from client to server?
Can you collect thread dump from such failing client when it has already started to exhibit that behavior?

Regards,
--
Ilya Kasnacheev


пн, 15 февр. 2021 г. в 11:00, Charlin S <[hidden email]>:
Hi,

i'm running an ASP.Net application with ignite 2.9.1 and seeing below error details in ignite client log and my web site stopped working. I have faced the same issue  with ignite 2.8.1 and I have upgraded 2.9.1 just before 1 day, since ignite 2.9.0 release notes saying below points:
1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a node fails to send a message or ping, now it drops the current connection strictly within this timeout and begins establishing a new connection much faster.
2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a node loses connection, now it strictly obtains a new connection to the ring of gets segmented within this timeout.

It's back to normal after restarting the application pool.
Please any suggestions? to avoid such issues.
Log files attached here for your reference.

Thanks & Regards,
Charlin

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.9.1 - An existing connection was forcibly closed by the remote host

Hello!

It should be JVM thread dump.

Packet drops is one thing, network misconfiguration is another.

Regards,
--
Ilya Kasnacheev


пн, 15 февр. 2021 г. в 15:12, Charlin S <[hidden email]>:
Hello,
Thanks for your help,
As per our network team, there were no packet drops between client and server.
Do I need to collect either jvm thread dump or IIS worker process thread dump ?

Thanks & Regards,
Charlin



On Mon, 15 Feb 2021 at 16:40, Ilya Kasnacheev <[hidden email]> wrote:
Hello!

I think I've just answered the same question for 2.8.1:

I'm not sure why your node does not recognize itself as SEGMENTED.
[15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi] Failed to send message: null
java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Maybe there's something in your network configuration which prevents relaying of "connection closed" packets from client to server?
Can you collect thread dump from such failing client when it has already started to exhibit that behavior?

Regards,
--
Ilya Kasnacheev


пн, 15 февр. 2021 г. в 11:00, Charlin S <[hidden email]>:
Hi,

i'm running an ASP.Net application with ignite 2.9.1 and seeing below error details in ignite client log and my web site stopped working. I have faced the same issue  with ignite 2.8.1 and I have upgraded 2.9.1 just before 1 day, since ignite 2.9.0 release notes saying below points:
1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a node fails to send a message or ping, now it drops the current connection strictly within this timeout and begins establishing a new connection much faster.
2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a node loses connection, now it strictly obtains a new connection to the ring of gets segmented within this timeout.

It's back to normal after restarting the application pool.
Please any suggestions? to avoid such issues.
Log files attached here for your reference.

Thanks & Regards,
Charlin

Charlin Charlin
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.9.1 - An existing connection was forcibly closed by the remote host

Hello,
Thanks for your help,
I am not sure where to check the network misconfiguration either in System or in Application configuration (ignite ean file).
I am thinking to reduce interval to 1 min and retry count 5 and currently its 2 minutes and 10 times.
My current configuration is as below and please let me know if any changes are required.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
                           http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/util
                           http://www.springframework.org/schema/util/spring-util.xsd">
  <bean class="org.apache.ignite.configuration.IgniteConfiguration">
     <property name="clientMode" value="true"/>
<property name="localHost" value="10.212.120.180"/>
     <property name="clientFailureDetectionTimeout" value="60000"/>
     <property name="igniteInstanceName" value="hh_StaticGrid"/>
     <property name="longQueryWarningTimeout" value="4000"/>
     <property name="failureDetectionTimeout" value="60000"/>
     <property name="segmentationPolicy" value="RESTART_JVM"/>
     <property name="segmentCheckFrequency" value="20000"/>
     <property name="segmentationResolveAttempts" value="5"/>
     <property name="discoverySpi">
      <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localPort" value="51500"/>
        <property name="ipFinder">
          <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
            <property name="addresses">
              <list>
<value>10.212.120.66:51500</value>
<value>10.212.120.67:51500</value>
              </list>
            </property>
          </bean>
        </property>
      </bean>
    </property>
<property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
   <property name="sharedMemoryPort" value="-1"/>
            <property name="localPort" value="51605"/>
        </bean>
    </property>
  </bean>
</beans>

Thanks & Regards
Charlin



On Tue, 16 Feb 2021 at 15:00, Ilya Kasnacheev <[hidden email]> wrote:
Hello!

It should be JVM thread dump.

Packet drops is one thing, network misconfiguration is another.

Regards,
--
Ilya Kasnacheev


пн, 15 февр. 2021 г. в 15:12, Charlin S <[hidden email]>:
Hello,
Thanks for your help,
As per our network team, there were no packet drops between client and server.
Do I need to collect either jvm thread dump or IIS worker process thread dump ?

Thanks & Regards,
Charlin



On Mon, 15 Feb 2021 at 16:40, Ilya Kasnacheev <[hidden email]> wrote:
Hello!

I think I've just answered the same question for 2.8.1:

I'm not sure why your node does not recognize itself as SEGMENTED.
[15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi] Failed to send message: null
java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Maybe there's something in your network configuration which prevents relaying of "connection closed" packets from client to server?
Can you collect thread dump from such failing client when it has already started to exhibit that behavior?

Regards,
--
Ilya Kasnacheev


пн, 15 февр. 2021 г. в 11:00, Charlin S <[hidden email]>:
Hi,

i'm running an ASP.Net application with ignite 2.9.1 and seeing below error details in ignite client log and my web site stopped working. I have faced the same issue  with ignite 2.8.1 and I have upgraded 2.9.1 just before 1 day, since ignite 2.9.0 release notes saying below points:
1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a node fails to send a message or ping, now it drops the current connection strictly within this timeout and begins establishing a new connection much faster.
2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a node loses connection, now it strictly obtains a new connection to the ring of gets segmented within this timeout.

It's back to normal after restarting the application pool.
Please any suggestions? to avoid such issues.
Log files attached here for your reference.

Thanks & Regards,
Charlin

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite 2.9.1 - An existing connection was forcibly closed by the remote host

Hello!

Have you gotten around collecting thread dump from the client?

Regards,
--
Ilya Kasnacheev


вт, 16 февр. 2021 г. в 16:59, Charlin S <[hidden email]>:
Hello,
Thanks for your help,
I am not sure where to check the network misconfiguration either in System or in Application configuration (ignite ean file).
I am thinking to reduce interval to 1 min and retry count 5 and currently its 2 minutes and 10 times.
My current configuration is as below and please let me know if any changes are required.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
                           http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/util
                           http://www.springframework.org/schema/util/spring-util.xsd">
  <bean class="org.apache.ignite.configuration.IgniteConfiguration">
     <property name="clientMode" value="true"/>
<property name="localHost" value="10.212.120.180"/>
     <property name="clientFailureDetectionTimeout" value="60000"/>
     <property name="igniteInstanceName" value="hh_StaticGrid"/>
     <property name="longQueryWarningTimeout" value="4000"/>
     <property name="failureDetectionTimeout" value="60000"/>
     <property name="segmentationPolicy" value="RESTART_JVM"/>
     <property name="segmentCheckFrequency" value="20000"/>
     <property name="segmentationResolveAttempts" value="5"/>
     <property name="discoverySpi">
      <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localPort" value="51500"/>
        <property name="ipFinder">
          <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
            <property name="addresses">
              <list>
<value>10.212.120.66:51500</value>
<value>10.212.120.67:51500</value>
              </list>
            </property>
          </bean>
        </property>
      </bean>
    </property>
<property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
   <property name="sharedMemoryPort" value="-1"/>
            <property name="localPort" value="51605"/>
        </bean>
    </property>
  </bean>
</beans>

Thanks & Regards
Charlin



On Tue, 16 Feb 2021 at 15:00, Ilya Kasnacheev <[hidden email]> wrote:
Hello!

It should be JVM thread dump.

Packet drops is one thing, network misconfiguration is another.

Regards,
--
Ilya Kasnacheev


пн, 15 февр. 2021 г. в 15:12, Charlin S <[hidden email]>:
Hello,
Thanks for your help,
As per our network team, there were no packet drops between client and server.
Do I need to collect either jvm thread dump or IIS worker process thread dump ?

Thanks & Regards,
Charlin



On Mon, 15 Feb 2021 at 16:40, Ilya Kasnacheev <[hidden email]> wrote:
Hello!

I think I've just answered the same question for 2.8.1:

I'm not sure why your node does not recognize itself as SEGMENTED.
[15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi] Failed to send message: null
java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Maybe there's something in your network configuration which prevents relaying of "connection closed" packets from client to server?
Can you collect thread dump from such failing client when it has already started to exhibit that behavior?

Regards,
--
Ilya Kasnacheev


пн, 15 февр. 2021 г. в 11:00, Charlin S <[hidden email]>:
Hi,

i'm running an ASP.Net application with ignite 2.9.1 and seeing below error details in ignite client log and my web site stopped working. I have faced the same issue  with ignite 2.8.1 and I have upgraded 2.9.1 just before 1 day, since ignite 2.9.0 release notes saying below points:
1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a node fails to send a message or ping, now it drops the current connection strictly within this timeout and begins establishing a new connection much faster.
2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a node loses connection, now it strictly obtains a new connection to the ring of gets segmented within this timeout.

It's back to normal after restarting the application pool.
Please any suggestions? to avoid such issues.
Log files attached here for your reference.

Thanks & Regards,
Charlin