Native persistence: ignite server failed to join when client has been started

classic Classic list List threaded Threaded
5 messages Options
radha radha
Reply | Threaded
Open this post in threaded view
|

Native persistence: ignite server failed to join when client has been started

This post was updated on .
Hi

I started two ignite servers pods by enabling the native persistence and authentication on a k8s cluster.
I checked in the baseline topology two servers were there, after activating the cluster.
Then i started the client. suddenly one of the server pod went down.
I checked topology in the visor , only one server and one client was there.
Then i restarted the  ignite server pod which was down, still it is not able to join the cluster.
I am using ignite 2.6
Below is  the error message:

{"type":"log","host":"ignite-cluster-kru-ign-1","level":"ERROR","systemid":"bcfa2b5e","system":"ignite","time":"2019-01-07
04:32:39,135","logger":"TcpDiscoverySpi","timezone":"UTC","marker""","log":"TcpDiscoverSpi's
message worker thread failed abnormally. Stopping the node in order to
prevent cluster wide instability. class org.apache.ignite.IgniteException:
Node with BaselineTopology cannot join mixed cluster running in
compatibility mode
        at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onGridDataReceived(GridClusterStateProcessor.java:714)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onExchange(GridDiscoveryManager.java:883)
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:1939)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddedMessage(ServerImpl.java:4354)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2744)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2536)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6775)
        at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2621)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
"}

Regards
radha
Roman Guseinov Roman Guseinov
Reply | Threaded
Open this post in threaded view
|

Re: Native persistence: ignite server failed to join when client has been started

Hi radha,

It looks like a known issue which is already fixed in 2.7
https://issues.apache.org/jira/browse/IGNITE-8774. Could you try to
reproduce this using Ignite 2.7?

Best Regards,
Roman



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
radha radha
Reply | Threaded
Open this post in threaded view
|

Re: Native persistence: ignite server failed to join when client has been started

thanks for the reply.
But currentlly i am working on 2.6.0. It takes some time to move to 2.7.0.
Is there a workaround.
This issue we face when persistence is enabled?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
radha radha
Reply | Threaded
Open this post in threaded view
|

Re: Native persistence: ignite server failed to join when client has been started

In reply to this post by Roman Guseinov
Hi roman,
     I am facing one more issue now , servers are going down with below error
     When client connects to server, after some time one server goes down. then later another server.
     {"type":"log","host":"ignite-cluster-harness-server-0","level":"ERROR","systemid":"1dcfe238","system":"CIGN","time":"2019-01-08 13:24:47,330","logger":"ClientListenerProcessor","timezone":"UTC","marker""","log":"Closing NIO session because of unhandled exception. class org.apache.ignite.IgniteCheckedException: Invalid handshake message
        at org.apache.ignite.internal.processors.odbc.ClientListenerNioServerBuffer.read(ClientListenerNioServerBuffer.java:115)
        at org.apache.ignite.internal.processors.odbc.ClientListenerBufferedParser.decode(ClientListenerBufferedParser.java:60)
        at org.apache.ignite.internal.processors.odbc.ClientListenerBufferedParser.decode(ClientListenerBufferedParser.java:40)
        at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:114)
        at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
        at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3490)
        at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
        at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1113)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)
"}

Regards
radha

On Mon, 7 Jan 2019 at 12:12, Roman Guseinov <[hidden email]> wrote:
Hi radha,

It looks like a known issue which is already fixed in 2.7
https://issues.apache.org/jira/browse/IGNITE-8774. Could you try to
reproduce this using Ignite 2.7?

Best Regards,
Roman



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Native persistence: ignite server failed to join when client has been started

In reply to this post by radha
Hello!

I'm afraid you will have to avoid using visor, use web console instead. The issue should only arise if there are daemon nodes, such as Visor.

Otherwise, if you need node to rejoin cluster & do not mind data, you can remove Ignite work dir in order to start anew.

Regards,
--
Ilya Kasnacheev


вт, 8 янв. 2019 г. в 16:38, radha <[hidden email]>:
thanks for the reply.
But currentlly i am working on 2.6.0. It takes some time to move to 2.7.0.
Is there a workaround.
This issue we face when persistence is enabled?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/