2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

classic Classic list List threaded Threaded
16 messages Options
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

Hello,

When do EVT_NODE_SEGMENTED event get generated on the client side in 2.8.1 ?

I noticed that in 2.7.6 , when all the servers are restarted and the client
reconnects again with a new node id - it used to receive a
EVT_NODE_SEGMENTED . In 2.8.1 it receives EVT_NODE_RECONNECTED.
Kindly confirm my understanding.

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
akurbanov akurbanov
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

Hello Veena,

Could you please double-check if the situation is the same? Reconnected
event should be fired if node reconnects with the same id, let me go through
the changes that could possibly affect that.

Best regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

This post was updated on .
HI Anton,

Checked again -In 2.8.1 it receives the EVT_NODE_RECONNECTED with a new node
id .  It receives Eventtype 17 ( EVT_NODE_RECONNECTED  ) and not Eventtype
14 ( EVT_NODE_SEGMENTED ) .
Also it connects with a new node id. Please find the log with relevant parts
highlighted.


2020-07-31 06:32:17,150
[tcp-client-disco-msg-worker-#4%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi [] - Client node
disconnected from cluster, will try to reconnect with new id

[newId=bcbaa5ad-07aa-4118-802e-433dcfcb83b4,
prevId=bca9b2da-bca5-4eac-817b-c989a072b50a, locNode=TcpDiscoveryNode
[id=bca9b2da-bca5-4eac-817b-c989a072b50a,
consistentId=bca9b2da-bca5-4eac-817b-c989a072b50a, addrs=ArrayList
[0:0:0:0:0:0:0:1, a.b.c.d, 127.0.0.1], sockAddrs=HashSet
[machinename/a.b.c.d:0, /0:0:0:0:0:0:0:1:0, /127.0.0.1:0], discPort=0,
order=4, intOrder=0, lastExchangeTime=1596173504995, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=true]]

2020-07-31 06:32:46,197
[tcp-client-disco-msg-worker-#4%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi [] - Router node:
TcpDiscoveryNode [id=4ca9ed4f-416b-43f2-ada9-d5569e7fe8ce,
consistentId=d46c9ddd-d97c-4d07-b857-fc4281d9f495, addrs=ArrayList
[0:0:0:0:0:0:0:1, a.b.c.d, 127.0.0.1], sockAddrs=HashSet
[/0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501, machinename/a.b.c.d:47501],
discPort=47501, order=3, intOrder=3, lastExchangeTime=1596173566191,
loc=false, ver=2.8.1#20200521-sha1:86422096, isClient=false]

2020-07-31 06:32:46,232
[disco-notifier-worker-#96%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl [] -
Assigned mvcc coordinator [crd=MvccCoordinator
[topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0],
nodeId=d0b9191e-18ea-42fe-8148-b7d28a75d9f8, ver=1596173562700, local=false,
initialized=false]]
2020-07-31 06:32:46,233
[exchange-worker-#136%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.internal.processors.cache.GridCacheProcessor [] - Stopped
cache [cacheName=ignite-sys-cache]
2020-07-31 06:32:46,233
[tcp-client-disco-msg-worker-#4%client-null-igniteclient-SINGLE%] WARN
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi [] - Client node was
reconnected after it was already considered failed by the server topology
(this could happen after all servers restarted or due to a long network
outage between the client and servers). All continuous queries and remote
event listeners created by this client will be unsubscribed, consider
listening to EVT_CLIENT_NODE_RECONNECTED event to restore them
.
[06:32:46] Client node was reconnected after it was already considered
failed by the server topology (this could happen after all servers restarted
or due to a long network outage between the client and servers). All
continuous queries and remote event listeners created by this client will be
unsubscribed, consider listening to EVT_CLIENT_NODE_RECONNECTED event to
restore them.
2020-07-31 06:32:46,237
[disco-event-worker-#98%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager [] -
Client node reconnected to topology: TcpDiscoveryNode
[id=bcbaa5ad-07aa-4118-802e-433dcfcb83b4,
consistentId=bca9b2da-bca5-4eac-817b-c989a072b50a, addrs=ArrayList
[0:0:0:0:0:0:0:1, a.b.c.d, 127.0.0.1], sockAddrs=HashSet
[machinename/a.b.c.d:0, /0:0:0:0:0:0:0:1:0, /127.0.0.1:0], discPort=0,
order=4, intOrder=0, lastExchangeTime=1596173504995, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=true]

[06:32:46] Topology snapshot [ver=4, locNode=bcbaa5ad, servers=3, clients=1,
state=INACTIVE, CPUs=32, offheap=12.0GB, heap=28.0GB]
[06:32:46]   ^-- Baseline [id=0, size=3, online=3, offline=0]
2020-07-31 06:32:46,238
[disco-event-worker-#98%pricingClient-null-igniteclient-SINGLE%] INFO
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager [] -
Topology snapshot [ver=4, locNode=bcbaa5ad, servers=3, clients=1,
state=INACTIVE, CPUs=32, offheap=12.0GB, heap=28.0GB]
2020-07-31 06:32:46,238
[disco-event-worker-#98%pricingClient-null-igniteclient-SINGLE%] INFO
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager [] -  
^-- Baseline [id=0, size=3, online=3, offline=0]

2020-07-31 06:32:46,238
[disco-event-worker-#98%pricingClient-null-igniteclient-SINGLE%] DEBUG
SegmentationEventListener [] - Received Event 17 Event Node Id
bcbaa5ad-07aa-4118-802e-433dcfcb83b4 for Node :
bcbaa5ad-07aa-4118-802e-433dcfcb83b4


regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
akurbanov akurbanov
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

Hello,

This case seems to be correct, it logs the event of client state updated
from DISCONNECTED to RECONNECTED because the node succeeded to join the
topology back within some time, the node was not segmented from the
topology. What timeouts do you use in your nodes configuration?

I would suggest to perform a test with different timeouts and check that if
node stays offline for too long, the node gets segmented from the topology.

Best regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

Hi Anton,

There are a few thoughts on this :

1. I found this issue after I upgraded from 2.7.6 to 2.8.1 ( No timeout
values etc has been changed in our application ). So for the same scenario
and the same application configuration
a. in 2.7.6 , the client receives a evt_node_segmented
b. in 2.8.1 , the client receives a evt_node_reconnected.

2. >>>>it logs the event of client state updated
from DISCONNECTED to RECONNECTED because the node succeeded to join the
topology back within some time, the node was not segmented from the
topology

From the logs I posted, it looks like client tries to reconnect using a new
client id
>>[tcp-client-disco-msg-worker-#4%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi [] - Client node
disconnected from cluster, will try to reconnect with new id

3. The logs also say
>>Client node was
reconnected after it was already considered failed by the server topology
(this could happen after all servers restarted or due to a long network
outage between the client and servers). All continuous queries and remote
event listeners created by this client will be unsubscribed, consider
listening to EVT_CLIENT_NODE_RECONNECTED event to restore them.

If a client node reconnects after it was considered failed by server,
shouldnt it receive a EVT_NODE_SEGMENTED ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

Hi Team,

Kindly awaiting a reply on this,

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

This post was updated on .
I rechecked this. My cluster setup is :
servers : 3
client : 1

Bring down all server nodes in the cluster for about 2 mins (
failuredetectiontimeout is 10 seconds ) and restart. The client receives
EVT_NODE_RECONNECTED and does not receive EVT_NODE_SEGMENTED.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

Hi ,

Kindly awaiting a reply on this ,

regards,
Veena



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
akurbanov akurbanov
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

Hi Veena,

I can confirm that there is no SEGMENTED event thrown on 2.8.1.

I cannot reproduce catching the SEGMENTED event on 2.7.6. Could you help to
reproduce the behaviour or share the log where this event might be noticed?

Best regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

Hi Anton,

Thank you for the reply .

>>I can confirm that there is no SEGMENTED event thrown on 2.8.1.

I guess you meant that there is no SEGMENTED event thrown  in 2.8.1 client
when the servers are brought down and started again. Because SEGMENTED event
could be thrown in 2.8.1 in other scenarios as well I guess, like a long GC
pause .

I have captured the logs for 2.7.6 client , where it receives a SEGMENTED
event. On closer look, it looks like the client's attempt to reconnect
failed in this case with 'Authentication Failure' and this has caused the
SEGMENTED event.

Check this part of the log in the attached log :
2020-10-08 09:59:33,327
[tcp-client-disco-msg-worker-#4%ProjectClient-null-igniteclient-SINGLE%]
ERROR
com.company.projectname.project.common.plugin.discoveryspi.CustomTcpDiscoverySpi
[] - Failed to reconnect, segment local node.
org.apache.ignite.spi.IgniteSpiException: Authentication failed
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1946)
~[ignite-core-2.7.6.jar:2.7.6]
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1888)
~[ignite-core-2.7.6.jar:2.7.6]
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.7.6.jar:2.7.6]
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:304)
~[ignite-core-2.7.6.jar:2.7.6]
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
~[ignite-core-2.7.6.jar:2.7.6]
2020-10-08 09:59:33,328
[tcp-client-disco-msg-worker-#4%ProjectClient-null-igniteclient-SINGLE%]
DEBUG

In 2.8.1, we have changed the security plugin , hence looks like the client
is able to reconnect successfully.

In terms of the client, do we need to have the below 2 EVENT handlers? :
1. EVT_NODE_SEGMENTED - to restart the node and deploy and continuous
queries if needed.
2. EVT_CLIENT_NODE_RECONNECTED - to just deploy the continuous queries ( no
restart of node needed, since the node is already connected )

2.txt <http://apache-ignite-users.70518.x6.nabble.com/file/t2757/2.txt>  


regards,
Veena






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

In reply to this post by akurbanov
Hi Anton,

Thank you for the reply .

>>I can confirm that there is no SEGMENTED event thrown on 2.8.1.

I guess you meant that there is no SEGMENTED event thrown  in 2.8.1 client
when the servers are brought down and started again. Because SEGMENTED event
could be thrown in 2.8.1 in other scenarios as well I guess, like a long GC
pause .

I have captured the logs for 2.7.6 client , where it receives a SEGMENTED
event. On closer look, it looks like the client's attempt to reconnect
failed in this case with 'Authentication Failure' and this has caused the
SEGMENTED event.

Check this part of the log in the attached log :
2020-10-08 09:59:33,327
[tcp-client-disco-msg-worker-#4%ProjectClient-null-igniteclient-SINGLE%]
ERROR
com.company.projectname.project.common.plugin.discoveryspi.CustomTcpDiscoverySpi
[] - Failed to reconnect, segment local node.
org.apache.ignite.spi.IgniteSpiException: Authentication failed
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1946)
~[ignite-core-2.7.6.jar:2.7.6]
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1888)
~[ignite-core-2.7.6.jar:2.7.6]
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.7.6.jar:2.7.6]
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:304)
~[ignite-core-2.7.6.jar:2.7.6]
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
~[ignite-core-2.7.6.jar:2.7.6]
2020-10-08 09:59:33,328
[tcp-client-disco-msg-worker-#4%ProjectClient-null-igniteclient-SINGLE%]
DEBUG

In 2.8.1, we have changed the security plugin , hence looks like the client
is able to reconnect successfully.

In terms of the client, do we need to have the below 2 EVENT handlers? :
1. EVT_NODE_SEGMENTED - to restart the node and deploy and continuous
queries if needed.
2. EVT_CLIENT_NODE_RECONNECTED - to just deploy the continuous queries ( no
restart of node needed, since the node is already connected )

SegmentedClientLogs.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t2757/SegmentedClientLogs.txt>  

regards,
Veena






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

Hi Team,


In terms of the client, do we need to have the below 2 EVENT handlers? :
1. EVT_NODE_SEGMENTED - to restart the node and deploy and continuous
queries if needed.
2. EVT_CLIENT_NODE_RECONNECTED - to just deploy the continuous queries ( no
restart of ignite needed, since the node is already connected )

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

Hello!

I don't think so. Node can reconnect to a *different* cluster, or it may be disconnected for a while and return to the same cluster. For client node, segmented/disconnected difference is not relevant. Neither it is for server node, in fact.

Regards,
--
Ilya Kasnacheev


вт, 13 окт. 2020 г. в 10:18, VeenaMithare <[hidden email]>:
Hi Team,


In terms of the client, do we need to have the below 2 EVENT handlers? :
1. EVT_NODE_SEGMENTED - to restart the node and deploy and continuous
queries if needed.
2. EVT_CLIENT_NODE_RECONNECTED - to just deploy the continuous queries ( no
restart of ignite needed, since the node is already connected )

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

Thanks Ilya,

>>I don't think so. Node can reconnect to a *different* cluster, or it may
be disconnected for a while and return to the same cluster. For client node,
segmented/disconnected difference is not relevant. Neither it is for server
node, in fact.

In  that case, we will have the same handler for segmented and reconnected
events. The handler will restart ignite and deploy any continuous queries as
needed.
Thank you for this guidance.

Does the server or the client need to have a handler for any other 'cluster
specific' events ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VeenaMithare VeenaMithare
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

In reply to this post by ilya.kasnacheev
Thanks Ilya,

>>I don't think so. Node can reconnect to a *different* cluster, or it may
be disconnected for a while and return to the same cluster. For client node,
segmented/disconnected difference is not relevant. Neither it is for server
node, in fact.

In  that case, we will have the same handler for segmented and reconnected
events. The handler will restart ignite and deploy any continuous queries as
needed.
Thank you for this guidance.

Does the server or the client need to have a handler for any other 'cluster
specific' events ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

In reply to this post by VeenaMithare
Hello!

You can just set clientReconnectEnabled to false. Then, if it is out of topology, just start a new one to make sure you have a fresh start.

Not sure about other events. Maybe handle ACTIVATED/DEACTIVATED?

Regards,
--
Ilya Kasnacheev


вт, 13 окт. 2020 г. в 14:49, VeenaMithare <[hidden email]>:
Thanks Ilya,

>>I don't think so. Node can reconnect to a *different* cluster, or it may
be disconnected for a while and return to the same cluster. For client node,
segmented/disconnected difference is not relevant. Neither it is for server
node, in fact.

In  that case, we will have the same handler for segmented and reconnected
events. The handler will restart ignite and deploy any continuous queries as
needed.
Thank you for this guidance.

Does the server or the client need to have a handler for any other 'cluster
specific' events ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/