Can't stop Ignite instance using Ignition.stopAll() or Ignition.kill() until writeBehindFlushFrequency passed

classic Classic list List threaded Threaded
10 messages Options
zshamrock zshamrock
Reply | Threaded
Open this post in threaded view
|

Can't stop Ignite instance using Ignition.stopAll() or Ignition.kill() until writeBehindFlushFrequency passed

This post was updated on .
I do some integration testing, and actually all of the IT cases I wrote works fine (ignite server is starting, client node connects, and in the end of the IT case both are closed). Everything works fine.

Although, one test never ends. It doesn't much differ from the rest of the IT cases I have, but apparently something is different, not in setup, but something related to Ignite.

So, I have the following tear down method (using Spock here):

def cleanupSpec() {

        def grids = Ignition.allGrids()
        // stop all clients first
        grids.findAll { it.configuration().isClientMode() } forEach { it.close() }
        // stop all the rest
        Ignition.stopAll(true)
}

A few things here to mention:
- Ignition.allGrids() returns 2 (where one instance is client, another is server, both were running from the same JVM process)
- I try to stop all the clients (well, just one) first
- Ignition.allGrids() now returns 1
- Ignition.stopAll(true) triggers application CacheStore implementation (there are 2 standing entries in the cache), they are saved are correctly
- And this is it, the process never stops, so the test is hanging forever

Instead of Ignition.stopAll(true), I've tried Ignition.kill(true), same result, the process (IT case) keeps running, and never stops.

UPDATE:

Eventually it has stopped:
>>> +---------------------------------------------------------------------------------------+
>>> Ignite ver. 1.5.0-final#20151229-sha1:f1f8cda2f3f62231f42a59951bf34c39577c1bec stopped OK
>>> +---------------------------------------------------------------------------------------+
>>> Grid name: server.ignite
>>> Grid uptime: 00:05:18:285

Grip uptime looks very close to writeBehindFlushFrequency, which I set to 5 minutes. Could it be the reason?

Does Ignition not stop the server (and even kill) until writeBehindFlushFrequency has elapsed?

Any ideas what could be the reason, and how I can tweak the behavior?

UPDATE:

So, yes, actually changed writeBehindFlushFrequency to 1 second, and everything finished just instantly.

So, still quite a strange behavior to me. Is not it?
vdpyatkov vdpyatkov
Reply | Threaded
Open this post in threaded view
|

Re: Can't stop Ignite instance using Ignition.stopAll() or Ignition.kill() until writeBehindFlushFrequency passed

Hello,
I can not reproduce the issue.
In my test case all data entries save immediately after call Ignition.stopAll(true).
If you had have example, which demonstrate the behavior, can you please provide source code?
zshamrock zshamrock
Reply | Threaded
Open this post in threaded view
|

Re: Can't stop Ignite instance using Ignition.stopAll() or Ignition.kill() until writeBehindFlushFrequency passed

They do indeed are saved immediately, the issue is that the ignite process is not stopping until writeBehindFlushFrequency elapsed after `kill` or `stopAll` call. So, in my case for that specific IT case I have to wait 5 minutes before it completes, unless I overwrite the value of writeBehindFlushFrequency.
vdpyatkov vdpyatkov
Reply | Threaded
Open this post in threaded view
|

Re: Can't stop Ignite instance using Ignition.stopAll() or Ignition.kill() until writeBehindFlushFrequency passed

I do exactly what you say, but can not see any delay.
I try to reproduce in last Ignite version (1.5.0.final). Try this version, if you use another.
Could you please provide example or configuration xml at least?

On Thu, May 12, 2016 at 5:37 PM, zshamrock <[hidden email]> wrote:
They do indeed are saved immediately, the issue is that the ignite process is
not stopping until writeBehindFlushFrequency elapsed after `kill` or
`stopAll` call. So, in my case for that specific IT case I have to wait 5
minutes before it completes, unless I overwrite the value of
writeBehindFlushFrequency.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-t-stop-Ignite-instance-using-Ignition-stopAll-or-Ignition-kill-until-writeBehindFlushFrequency-pd-tp4837p4898.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--
Vladislav Pyatkov
alexey.goncharuk alexey.goncharuk
Reply | Threaded
Open this post in threaded view
|

Re: Can't stop Ignite instance using Ignition.stopAll() or Ignition.kill() until writeBehindFlushFrequency passed

Hi,

Can you take a thread dump of the process in this 'hanging' state and attach in to the thread?
zshamrock zshamrock
Reply | Threaded
Open this post in threaded view
|

Re: Can't stop Ignite instance using Ignition.stopAll() or Ignition.kill() until writeBehindFlushFrequency passed

This post was updated on .
Ignite Hanging

And then it is hanging for about 5 minutes after each IT case.
Does the picture above help somehow?

How I can do a thread dump? I've tried to use jstack, although could not find anything meaningful in the report it has produced.

And again setting -Dcache.stats.write.behind.flushFrequency.ms=1000 during the run helps. But using the default one (5 minutes, in my case) triggers IT case to wait 5 minutes before being complete.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Can't stop Ignite instance using Ignition.stopAll() or Ignition.kill() until writeBehindFlushFrequency passed

What is 'cache.stats.write.behind.flushFrequency.ms' system property? This property is not defined in Ignite, do you have your own logic that uses it?

-Val
zshamrock zshamrock
Reply | Threaded
Open this post in threaded view
|

Re: Can't stop Ignite instance using Ignition.stopAll() or Ignition.kill() until writeBehindFlushFrequency passed

It is just an application specific property, I pass it to configure the application behavior at runtime, it just sets the write behind flush frequency for the stats cache.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Can't stop Ignite instance using Ignition.stopAll() or Ignition.kill() until writeBehindFlushFrequency passed

I can't reproduce this either. GridCacheWriteBehindStore flushes all the data immediately after stop process is initiated. Is it possible that it just takes more time to flush the whole queue? With bigger timeout it's possible that you collect more data.

It would be also great if you provide a small example (e.g., a GitHub project) that reproduces the problem. Someone will take a look.

-Val
zshamrock zshamrock
Reply | Threaded
Open this post in threaded view
|

Re: Can't stop Ignite instance using Ignition.stopAll() or Ignition.kill() until writeBehindFlushFrequency passed

I will try to come with a simple example. But I am afraid that it could be due to the dependencies and the setup for IT tests I use. Will share the github example when it is ready with you.