Correctly stopping/deactivating server node

classic Classic list List threaded Threaded
10 messages Options
DonTequila DonTequila
Reply | Threaded
Open this post in threaded view
|

Correctly stopping/deactivating server node

Hi,

when running Ignite inside a docker container, and the container is
being stopped, what is the safest way to handle shutdown event?

Currently I'm using a ShutdownHook to the Java runtime and try to
quickly deactivate and disconnect the cluster. But I get an exception
like this:

class org.apache.ignite.IgniteException: Failed to execute deactivate
request, node is stopping.
         at
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1067)
         at
org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:317)
         at
mediafan.net.ignite.IgniteHelper.setClusterActiveStatus(IgniteHelper.java:238)
         at mediafan.client.helper.ConsoleApp.setActive(ConsoleApp.java:370)
         at
mediafan.client.helper.ConsoleApp.disconnect(ConsoleApp.java:342)
         at mediafan.client.helper.ConsoleApp$1.run(ConsoleApp.java:70)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
execute deactivate request, node is stopping.
         at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState0(GridClusterStateProcessor.java:1007)
         at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:832)
         at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:820)
         at
org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:314)
         ... 4 more

If I only do a disconnect without deactivate then it works well. But my
understanding is the last server node should also deactivate the
cluster? How can this be achieved in such Docker environment?

Thanks,
Thomas.


DonTequila DonTequila
Reply | Threaded
Open this post in threaded view
|

Re: Correctly stopping/deactivating server node

Has anyone been able to look at this and give me some advise? Thanks!


DonTequila wrote

> Hi,
>
> when running Ignite inside a docker container, and the container is
> being stopped, what is the safest way to handle shutdown event?
>
> Currently I'm using a ShutdownHook to the Java runtime and try to
> quickly deactivate and disconnect the cluster. But I get an exception
> like this:
>
> class org.apache.ignite.IgniteException: Failed to execute deactivate
> request, node is stopping.
>          at
> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1067)
>          at
> org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:317)
>          at
> mediafan.net.ignite.IgniteHelper.setClusterActiveStatus(IgniteHelper.java:238)
>          at
> mediafan.client.helper.ConsoleApp.setActive(ConsoleApp.java:370)
>          at
> mediafan.client.helper.ConsoleApp.disconnect(ConsoleApp.java:342)
>          at mediafan.client.helper.ConsoleApp$1.run(ConsoleApp.java:70)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
> execute deactivate request, node is stopping.
>          at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState0(GridClusterStateProcessor.java:1007)
>          at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:832)
>          at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:820)
>          at
> org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:314)
>          ... 4 more
>
> If I only do a disconnect without deactivate then it works well. But my
> understanding is the last server node should also deactivate the
> cluster? How can this be achieved in such Docker environment?
>
> Thanks,
> Thomas.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Correctly stopping/deactivating server node

Hello!

I guess that it's already too late to deactivate if the node is shutting down.

What are scenarios for which you are trying to prepare here?

Regards,
--
Ilya Kasnacheev


ср, 31 мар. 2021 г. в 22:13, DonTequila <[hidden email]>:
Has anyone been able to look at this and give me some advise? Thanks!


DonTequila wrote
> Hi,
>
> when running Ignite inside a docker container, and the container is
> being stopped, what is the safest way to handle shutdown event?
>
> Currently I'm using a ShutdownHook to the Java runtime and try to
> quickly deactivate and disconnect the cluster. But I get an exception
> like this:
>
> class org.apache.ignite.IgniteException: Failed to execute deactivate
> request, node is stopping.
>          at
> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1067)
>          at
> org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:317)
>          at
> mediafan.net.ignite.IgniteHelper.setClusterActiveStatus(IgniteHelper.java:238)
>          at
> mediafan.client.helper.ConsoleApp.setActive(ConsoleApp.java:370)
>          at
> mediafan.client.helper.ConsoleApp.disconnect(ConsoleApp.java:342)
>          at mediafan.client.helper.ConsoleApp$1.run(ConsoleApp.java:70)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
> execute deactivate request, node is stopping.
>          at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState0(GridClusterStateProcessor.java:1007)
>          at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:832)
>          at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:820)
>          at
> org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:314)
>          ... 4 more
>
> If I only do a disconnect without deactivate then it works well. But my
> understanding is the last server node should also deactivate the
> cluster? How can this be achieved in such Docker environment?
>
> Thanks,
> Thomas.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Shubham Agrawal Shubham Agrawal
Reply | Threaded
Open this post in threaded view
|

Re: Correctly stopping/deactivating server node

We need to see how the docker container is shutting down. Like what is the order of the services to shut down.

If you are shutting down the ignite service before shutting down the Java process, we might encounter the above error you mentioned.

Vice Versa, this problem should not come. You can try this and check out if this works.

Regards,
Shubham Agrawal


On Thu, Apr 1, 2021 at 10:58 AM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

I guess that it's already too late to deactivate if the node is shutting down.

What are scenarios for which you are trying to prepare here?

Regards,
--
Ilya Kasnacheev


ср, 31 мар. 2021 г. в 22:13, DonTequila <[hidden email]>:
Has anyone been able to look at this and give me some advise? Thanks!


DonTequila wrote
> Hi,
>
> when running Ignite inside a docker container, and the container is
> being stopped, what is the safest way to handle shutdown event?
>
> Currently I'm using a ShutdownHook to the Java runtime and try to
> quickly deactivate and disconnect the cluster. But I get an exception
> like this:
>
> class org.apache.ignite.IgniteException: Failed to execute deactivate
> request, node is stopping.
>          at
> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1067)
>          at
> org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:317)
>          at
> mediafan.net.ignite.IgniteHelper.setClusterActiveStatus(IgniteHelper.java:238)
>          at
> mediafan.client.helper.ConsoleApp.setActive(ConsoleApp.java:370)
>          at
> mediafan.client.helper.ConsoleApp.disconnect(ConsoleApp.java:342)
>          at mediafan.client.helper.ConsoleApp$1.run(ConsoleApp.java:70)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
> execute deactivate request, node is stopping.
>          at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState0(GridClusterStateProcessor.java:1007)
>          at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:832)
>          at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:820)
>          at
> org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:314)
>          ... 4 more
>
> If I only do a disconnect without deactivate then it works well. But my
> understanding is the last server node should also deactivate the
> cluster? How can this be achieved in such Docker environment?
>
> Thanks,
> Thomas.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
DonTequila DonTequila
Reply | Threaded
Open this post in threaded view
|

Re: Correctly stopping/deactivating server node

My standalone Java application is executed as process 1 in the Docker
container. At start it activates the cluster which has persistence enabled.
I'm not using the Ignite script bin/ignite.sh.

"docker stop" sends a SIGTERM to my java application. My application
shutdown hook gets called where I try to deactivate and then disconnect the
Ignite instance. This works well when only disconnecting, but doesn't work
when deactivating the cluster first. Exception says "Failed to execute
deactivate request, node is stopping." Full stacktrace in my original post.

Docker waits 10 seconds for a graceful stop of the process before it sends a
SIGKILL.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Shubham Agrawal Shubham Agrawal
Reply | Threaded
Open this post in threaded view
|

Re: Correctly stopping/deactivating server node

I guess, basically you are trying to see, which java process is stopping at last. If this is the last process, you have to deactivate the cluster, or else just disconnect. You might be getting the above error as one node is trying to disconnect and another node is trying to deactivate.

In order to resolve this, you can try out a centralised approach to check how many processes are running. For instance, zookeeper. If a new node is spawning, we ideally add a Z node in the zookeeper and remove it if it gets deleted.  If you don't want to use a zookeeper, you can have another approach like files for different processes in a centralised location. You can add a file on spawning and delete when getting killed. In the shutdown hook, every process checks how many files are there in the centralised location or how many nodes are there in the zookeeper. If the value is 1, deactivate the cluster, else only disconnect. There could be other approaches, but this is something you could try.

On Sat, Apr 3, 2021 at 12:01 PM DonTequila <[hidden email]> wrote:
My standalone Java application is executed as process 1 in the Docker
container. At start it activates the cluster which has persistence enabled.
I'm not using the Ignite script bin/ignite.sh.

"docker stop" sends a SIGTERM to my java application. My application
shutdown hook gets called where I try to deactivate and then disconnect the
Ignite instance. This works well when only disconnecting, but doesn't work
when deactivating the cluster first. Exception says "Failed to execute
deactivate request, node is stopping." Full stacktrace in my original post.

Docker waits 10 seconds for a graceful stop of the process before it sends a
SIGKILL.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
DonTequila DonTequila
Reply | Threaded
Open this post in threaded view
|

Re: Correctly stopping/deactivating server node

No, this is happening for the last server node with no other nodes connected.

To me it looks like Ignite itself is already handling its own shutdown hook
when SIGTERM is sent and then internally starts disconnect. So Ignite
doesn't allow deactivate any more. Is this possible true?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
DonTequila DonTequila
Reply | Threaded
Open this post in threaded view
|

Re: Correctly stopping/deactivating server node

Actually I found that this is indeed true. Ignite supports the JRE option:

-DIGNITE_NO_SHUTDOWN_HOOK=true

When I set this my app shutdown hook has enough time to call deactive the
cluster.

I wonder if this is safe to do?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Correctly stopping/deactivating server node

Hello!

Yes it should be safe, provided that your own hook works correctly.

Regards,
--
Ilya Kasnacheev


вс, 4 апр. 2021 г. в 23:18, DonTequila <[hidden email]>:
Actually I found that this is indeed true. Ignite supports the JRE option:

-DIGNITE_NO_SHUTDOWN_HOOK=true

When I set this my app shutdown hook has enough time to call deactive the
cluster.

I wonder if this is safe to do?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
DonTequila DonTequila
Reply | Threaded
Open this post in threaded view
|

Re: Re: Correctly stopping/deactivating server node

Yes it works well indeed! Thanks!



On 05.04.21 at 12:56, Ilya Kasnacheev wrote:

From: "Ilya Kasnacheev" <[hidden email]>
Date: 5. April 2021
To: [hidden email]
Cc:
Subject: Re: Correctly stopping/deactivating server node
Hello!

Yes it should be safe, provided that your own hook works correctly.

Regards,
--
Ilya Kasnacheev


вс, 4 апр. 2021 г. в 23:18, DonTequila <[hidden email]>:
Actually I found that this is indeed true. Ignite supports the JRE option:

-DIGNITE_NO_SHUTDOWN_HOOK=true

When I set this my app shutdown hook has enough time to call deactive the
cluster.

I wonder if this is safe to do?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/