How to speedup activation on a node with persistence

classic Classic list List threaded Threaded
4 messages Options
Kamil Misuth Kamil Misuth
Reply | Threaded
Open this post in threaded view
|

How to speedup activation on a node with persistence

Greetings,

I have a Spring Boot app which contains embedded Ignite server node and
acts like "a single node cluster". Persistence is enabled. The embedded
server node is activated on ContextRefresh event.
Sometimes the app starts in seconds (6 to 9 seconds), other times it
hangs for 10 minutes waiting for node activation to finish. The app is
the only Ignite node on the network.
Is the 10 minute long activation normal or could it be faster after some
tweaking?

This is what I see in the log when stuck:

[ INFO] [15.12.2017 13:05:36.589] [] [xchange-worker-#42]
[ridDhtPartitionsExchangeFuture]: Finished waiting for partition release
future [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1],
waitTime=0ms, futInfo=NA]
[ INFO] [15.12.2017 13:06:38.462] [] [ate-notifier-timer]
[o.a.i.i.p.c.GridUpdateNotifier]: Update status is not available.
[ INFO] [15.12.2017 13:14:18.818] [] [xchange-worker-#42]
[GridCacheDatabaseSharedManager]: Read checkpoint status
[startMarker=/tmp/ignite/work/myapp/persistent_store/node00-68e9e45a-3193-49f2-8f8b-93dfa5b08a6a/cp/1513339132540-b562bb70-f708-441a-9dab-48cce4f4ea17-START.bin,
endMarker=/tmp/ignite/work/myapp/persistent_store/node00-68e9e45a-3193-49f2-8f8b-93dfa5b08a6a/cp/1513339132540-b562bb70-f708-441a-9dab-48cce4f4ea17-END.bin]
[ INFO] [15.12.2017 13:14:18.818] [] [xchange-worker-#42]
[GridCacheDatabaseSharedManager]: Applying lost cache updates since last
checkpoint record

Main thread booting the app is suspended and waiting for activation to
finish:

"main" - Thread t@1
    java.lang.Thread.State: WAITING
         at sun.misc.Unsafe.park(Native Method)
         at java.util.concurrent.locks.LockSupport.park(Unknown Source)
         at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
         at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
         at
org.apache.ignite.internal.IgniteKernal.active(IgniteKernal.java:3510)
         at my.ignite.IgniteActivator.start(IgniteActivator.java:40)
         at
org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:175)
         at
org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:50)
         at
org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:348)
         at
org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:151)
         at
org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:114)
         at
org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:880)
         at
org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144)
         at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:546)
         - locked <35a3518d> (a java.lang.Object)
         at
org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
         at
org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693)
         at
org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360)
         at
org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
         at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1118)
         at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1107)
         at my.MyApplication.main(MyApplication.java:25)

Could somebody give me any pointers as to why this is happening and what
may be wrong with my configuration?

Thank you!

Kamil
mcherkasov mcherkasov
Reply | Threaded
Open this post in threaded view
|

Re: How to speedup activation on a node with persistence

 Hi Kamil,

if you have magnetic tape storage and store terabytes of data, then 10
minutes might be ok for startup, but I don't think that it's your case.

Could you please share a full log of Ignite for a slow and fast startup?

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Fwd: How to speedup activation on a node with persistence

Hi Kamil, 

Have you been able to resolve your issue? If yes, it would be great if you could share it with the community.

Thanks,
D.

---------- Forwarded message ----------
From: mcherkasov <[hidden email]>
Date: Mon, Dec 18, 2017 at 9:27 AM
Subject: Re: How to speedup activation on a node with persistence
To: [hidden email]


 Hi Kamil,

if you have magnetic tape storage and store terabytes of data, then 10
minutes might be ok for startup, but I don't think that it's your case.

Could you please share a full log of Ignite for a slow and fast startup?

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Kamil Misuth Kamil Misuth
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: How to speedup activation on a node with persistence

Hi,

there were multiple issues, one of them being shared and overloaded
disks on a hypervisor. That being said, I am not sure if the problem
disappeared completely. The node was up since then. The dataset is some
40 GB "bootstrapped" from Kafka. I would need to play with it more, but
don't have the time right now.
I will try to report back some more info when I get time.

Kamil

On 2018-01-14 03:41, Dmitriy Setrakyan wrote:

> Hi Kamil,
>
> Have you been able to resolve your issue? If yes, it would be great if
> you could share it with the community.
>
> Thanks,
> D.
>
> ---------- Forwarded message ----------
> From: MCHERKASOV <[hidden email]>
> Date: Mon, Dec 18, 2017 at 9:27 AM
> Subject: Re: How to speedup activation on a node with persistence
> To: [hidden email]
>
>  Hi Kamil,
>
> if you have magnetic tape storage and store terabytes of data, then 10
> minutes might be ok for startup, but I don't think that it's your
> case.
>
> Could you please share a full log of Ignite for a slow and fast
> startup?
>
> Thanks,
> Mike.
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ [1]
>
>
>
> Links:
> ------
> [1] http://apache-ignite-users.70518.x6.nabble.com/