Graceful shutdown

classic Classic list List threaded Threaded
3 messages Options
djm132 djm132
Reply | Threaded
Open this post in threaded view
|

Graceful shutdown

Hi,

I've implemented grid task workers and what to shutdown them gracefully (after finishing all active tasks). I see there is ignite.close() method which can be broadcasted to remote nodes.

One of solution I've got from another thread looks now like this:

    static class PoisonPill implements IgniteRunnable {
        @IgniteInstanceResource
        private transient Ignite ignite;

        @Override
        public void run() {
            new Thread() {
                @Override public void run() {
                    ignite.close();
                }
            }.start();
        }
    }

    // Called from http API
    public static void shutdownCluster() {
        // Stop remote servers if connected
        ClusterGroup remotes = instance.ignite.cluster().forRemotes();
        if (remotes.nodes().size() > 0) {
            instance.ignite.compute(remotes).broadcast(new PoisonPill());
        }

        // Close this instance
        instance.ignite.close();
    }

This code works fine, but I only if there is no active tasks on node. If there is some task executing, I am receiving this:

2016/05/12 20:45:35.959 [http-2] WARN Will cancel unfinished tasks due to stopping of the grid [cnt=1]
[20:45:35] (err) Failed to notify listener: dmp.grid.Engine$$Lambda$3/31794917@347cb6e2class org.apache.ignite.compute.ComputeTaskCancelledException: Task cancelled due to stopping of the grid....

May be there is a way to correctly write tasks and checking for termination ?

Thanks.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Graceful shutdown

Hi,

Yes, ignite.close() cancels all currently running tasks. You can replace this call with the following:

Ignition.stop(ignite.name(), false)

The second parameter means that the stopping node should not cancel tasks and will wait for their completion.

Hope this helps.

-Val
djm132 djm132
Reply | Threaded
Open this post in threaded view
|

Re: Graceful shutdown

Thanks, now it waits for task completion. I've also implemented shutdown check during the tasks to allow correct termination.