An existing connection was forcibly closed by the remote host

classic Classic list List threaded Threaded
8 messages Options
siva siva
Reply | Threaded
Open this post in threaded view
|

An existing connection was forcibly closed by the remote host

This post was updated on .
Hi,
I have .NetCore(v2.2.103) client and Server Ignite(v2.7.5) Application.And I
am using third party thin client
making request to Server to Put And Read data to/from Server.

Normal application request read and put operation is happening if no of
connection is less.

But onces sending continously data for put and no of connection increased
after around more than 3000 getting continously Exception

"An existing connection was forcibly closed by the remote host".

*Here is the Exception:*
[07:16:22,681][SEVERE][grid-nio-worker-client-listener-3-#52%ServerNode%][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=3, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-3, igniteInstanceName=ServerNode, finished=false, heartbeatTs=1562569764447, hashCode=117867960, interrupted=false, runner=grid-nio-worker-client-listener-3-#52%ServerNode%]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/173.16.4.13:10800, rmtAddr=/173.16.4.8:54238, createTime=1562568311896, closeTime=0, bytesSent=112344, bytesRcvd=77846, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1562568800561, lastSndTime=1562568800561, lastRcvTime=1562568798463, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true, markedForClose=false]]]
java.io.IOException: An existing connection was forcibly closed by the remote host
        at sun.nio.ch.SocketDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1120)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2386)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2153)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)

*Put Request:*
----------------//method calling rate 10 thousand times in a second.

  public async Task CreateOrUpdateRecordAsync(string cacheName, ICustomCacheStore data, IKeyModel key)
        {
            //get client configuration
            try
            {
               
                using (IIgniteClient client = Ignition.StartClient(this._igniteClientConfiguration))
                {
                    //get cache configuraation
                    var cache = client.GetCache<string, ICustomCacheStore>(cacheName);
                    string json = JsonConvert.SerializeObject(key,Formatting.None);
                    string base64EncodedKey = Convert.ToBase64String(Encoding.UTF8.GetBytes(json));
                    await cache.PutAsync(base64EncodedKey, data);

                }
            }
            catch (Exception ex)
            {
                throw;
            }
        }

Please help,how to solve above exception.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: An existing connection was forcibly closed by the remote host

Hello!

Unfortunately your letter does not seem to contain any error messages. Can you paste them in as text?

Regards,
--
Ilya Kasnacheev


пн, 8 июл. 2019 г. в 15:33, siva <[hidden email]>:
Hi,
I have .NetCore(v2.2.103) client and Server Ignite(v2.7.5) Application.And I
am using third party thin client
making request to Server to Put And Read data to/from Server.

Normal application request read and put operation is happening if no of
connection is less.

But onces sending continously data for put and no of connection increased
after around more than 3000 getting continously Exception

"An existing connection was forcibly closed by the remote host".

*Here is the Exception:*


*Put Request:*
----------------//method calling rate 10 thousand times in a second.



Please help,how to solve above exception.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
siva siva
Reply | Threaded
Open this post in threaded view
|

Re: An existing connection was forcibly closed by the remote host

This post was updated on .
Sorry,i have pasted but it's didn't attached with message.
*Exception:*
[07:16:22,681][SEVERE][grid-nio-worker-client-listener-3-#52%ServerNode%][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=3, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-3, igniteInstanceName=ServerNode,
finished=false, heartbeatTs=1562569764447, hashCode=117867960,
interrupted=false,
runner=grid-nio-worker-client-listener-3-#52%ServerNode%]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/173.16.4.13:10800, rmtAddr=/173.16.4.8:54238,
createTime=1562568311896, closeTime=0, bytesSent=112344, bytesRcvd=77846,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1562568800561,
lastSndTime=1562568800561, lastRcvTime=1562568798463, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true, markedForClose=false]]]
java.io.IOException: An existing connection was forcibly closed by the
remote host
        at sun.nio.ch.SocketDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1120)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2386)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2153)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: An existing connection was forcibly closed by the remote host

Hello!

I have seen such messages under heavy load on a large cluster.

My recommendation is to increase socketWriteTimeout to 5s on TcpCommunicationSpi. The default of 2s is too small, even smaller than TCP retransmit.

Regards,
--
Ilya Kasnacheev


пн, 8 июл. 2019 г. в 15:38, siva <[hidden email]>:
Sorry,don't i have pasted but it's did attached with message.
*Exception:*
[07:16:22,681][SEVERE][grid-nio-worker-client-listener-3-#52%ServerNode%][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=3, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-3, igniteInstanceName=ServerNode,
finished=false, heartbeatTs=1562569764447, hashCode=117867960,
interrupted=false,
runner=grid-nio-worker-client-listener-3-#52%ServerNode%]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/173.16.4.13:10800, rmtAddr=/173.16.4.8:54238,
createTime=1562568311896, closeTime=0, bytesSent=112344, bytesRcvd=77846,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1562568800561,
lastSndTime=1562568800561, lastRcvTime=1562568798463, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true, markedForClose=false]]]
java.io.IOException: An existing connection was forcibly closed by the
remote host
        at sun.nio.ch.SocketDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1120)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2386)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2153)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
siva siva
Reply | Threaded
Open this post in threaded view
|

Re: An existing connection was forcibly closed by the remote host

Hi,
Above configuration is that related to thinclient? already i am using
socketTimeout on thinclient.

 like below:

   _igniteClientConfiguration = new IgniteClientConfiguration
            {
                Endpoints = new string[] { endPoint },
                SocketTimeout = TimeSpan.FromSeconds(30)
        }
      ....
      ....
 
   public async Task CreateOrUpdateRecordAsync(string cacheName,
ICustomCacheStore data, IKeyModel key)
        {
            //get client configuration
            try
            {
                //
                using (IIgniteClient client =
Ignition.StartClient(this._igniteClientConfiguration))
                {
                    //get cache configuraation
                    var cache = client.GetCache<string,
ICustomCacheStore>(cacheName);
                    string json =
JsonConvert.SerializeObject(key,Formatting.None);
                    string base64EncodedKey =
Convert.ToBase64String(Encoding.UTF8.GetBytes(json));
                    await cache.PutAsync(base64EncodedKey, data);

                }
            }
            catch (Exception ex)
            {
                throw;
            }
        }

with above timeout getting the same exception.


Actually where i need to set is that need to set on server configuration in
xml file?
I thing some where i am missing docs ...i didn't find in  documentation
<https://apacheignite.readme.io/docs/network-config>   property
"socketWriteTimeout".

<bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
        ...
        ...
        <property name="communicationSpi">
      <bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
        <property name="socketWriteTimeout" value="5000" />
      </bean>
    </property>
       
</bean>









--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: An existing connection was forcibly closed by the remote host

Hello!

If it's thin client than you should be able to simply re-open your client when it turns bad.

Then maybe it's client that closing this connection, maybe it's network stack (inactivity, etc).

It is recommended to close Thin Client when not using it, rather than let it hang around.

Regards,
--
Ilya Kasnacheev


пн, 8 июл. 2019 г. в 19:26, siva <[hidden email]>:
Hi,
Above configuration is that related to thinclient? already i am using
socketTimeout on thinclient.

 like below:

   _igniteClientConfiguration = new IgniteClientConfiguration
            {
                Endpoints = new string[] { endPoint },
                SocketTimeout = TimeSpan.FromSeconds(30)
        }
      ....
      ....

   public async Task CreateOrUpdateRecordAsync(string cacheName,
ICustomCacheStore data, IKeyModel key)
        {
            //get client configuration
            try
            {
                //
                using (IIgniteClient client =
Ignition.StartClient(this._igniteClientConfiguration))
                {
                    //get cache configuraation
                    var cache = client.GetCache<string,
ICustomCacheStore>(cacheName);
                    string json =
JsonConvert.SerializeObject(key,Formatting.None);
                    string base64EncodedKey =
Convert.ToBase64String(Encoding.UTF8.GetBytes(json));
                    await cache.PutAsync(base64EncodedKey, data);

                }
            }
            catch (Exception ex)
            {
                throw;
            }
        }

with above timeout getting the same exception.


Actually where i need to set is that need to set on server configuration in
xml file?
I thing some where i am missing docs ...i didn't find in  documentation
<https://apacheignite.readme.io/docs/network-config>   property
"socketWriteTimeout".

<bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
        ...
        ...
        <property name="communicationSpi">
      <bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
        <property name="socketWriteTimeout" value="5000" />
      </bean>
    </property>

</bean>









--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
siva siva
Reply | Threaded
Open this post in threaded view
|

Re: An existing connection was forcibly closed by the remote host

This post was updated on .
ok.
One more question on this,
Our dot net services(lets say micro service 15) is processing 40K to 60K
messages parallelly.  In each service, I am opening and closing connection for each
message processing and inserting/reading into ignite.
Generally, Is it good practice to do..?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: An existing connection was forcibly closed by the remote host

Hello!

I think that it will make sense to have a single client per handling thread. Close only when idle.

Regards.
--
Ilya Kasnacheev


пн, 8 июл. 2019 г. в 20:07, siva <[hidden email]>:
ok.
One more question on this,
Our dot net services(lets say micro service 15) is processing 40K to 60K
messages.  In each service, I am opening and closing connection for each
message processing and inserting/reading into ignite.
Generally, Is it good practice to do..?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/