Slow startup of server node and query

[DonTequila]

Hi,

I'm seeing a pretty slow Ignite server node startup time, of approx
20mins, with persistence enabled for most caches. Of course I can see
startup-time highly depends on disk speed, but I wonder if there are
settings that could improve this?

I've attached a server log of the time when it starts. Also the Ignite
configs attached.

Also, when first accessed from a client with SQL there's a long running
query which shouldn't be the case in my opinion for binary storage and
indexes?

Can you spot anything that is wrong with my system startup time?

Thanks,
Thomas.

[DonTequila]

Has anyone been able to look at this and give me some advise? Thanks!


DonTequila wrote

> Hi,
>
> I'm seeing a pretty slow Ignite server node startup time, of approx
> 20mins, with persistence enabled for most caches. Of course I can see
> startup-time highly depends on disk speed, but I wonder if there are
> settings that could improve this?
>
> I've attached a server log of the time when it starts. Also the Ignite
> configs attached.
>
> Also, when first accessed from a client with SQL there's a long running
> query which shouldn't be the case in my opinion for binary storage and
> indexes?
>
> Can you spot anything that is wrong with my system startup time?
>
> Thanks,
> Thomas.
>
>
>
>
> server.log (102K)
> <http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/0/server.log>
> ignite-caches-config.xml (39K)
> <http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/1/ignite-caches-config.xml>
> ignite-caches-dataregions.xml (2K)
> <http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/2/ignite-caches-dataregions.xml>
> ignite-mediafan-db.xml (4K)
> <http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/3/ignite-mediafan-db.xml>

--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[Gianluca Bonetti]

Hello

It happened to me on a few occasions.
I have a couple of suggestions which may, or may not, solve your problem.

Please try :)

First, please ensure you use /dev/urandom as an entropy source.

/dev/random does a better job, but may cause long pauses (10 minutes or longer may happen easier than expected) and /dev/urandom is good enough for most cases.

Second, please ensure not to use -XX:+AlwaysPreTouch in your JVM flags as zeroing large heaps on initialization may take a very long time.

Cheers

Gianluca

Il giorno mer 31 mar 2021 alle ore 21:12 DonTequila <[hidden email]> ha scritto:

Has anyone been able to look at this and give me some advise? Thanks!


DonTequila wrote
> Hi,
>
> I'm seeing a pretty slow Ignite server node startup time, of approx
> 20mins, with persistence enabled for most caches. Of course I can see
> startup-time highly depends on disk speed, but I wonder if there are
> settings that could improve this?
>
> I've attached a server log of the time when it starts. Also the Ignite
> configs attached.
>
> Also, when first accessed from a client with SQL there's a long running
> query which shouldn't be the case in my opinion for binary storage and
> indexes?
>
> Can you spot anything that is wrong with my system startup time?
>
> Thanks,
> Thomas.
>
>
>
>
> server.log (102K)
> &lt;http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/0/server.log&gt;
> ignite-caches-config.xml (39K)
> &lt;http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/1/ignite-caches-config.xml&gt;
> ignite-caches-dataregions.xml (2K)
> &lt;http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/2/ignite-caches-dataregions.xml&gt;
> ignite-mediafan-db.xml (4K)
> &lt;http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/3/ignite-mediafan-db.xml&gt;





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[DonTequila]

Hi,

thanks for your suggestions. Unfortunately this didn't help improving the
startup time.

Looking at the log it takes most of the time (about 9 minutes) here:

23-03-2021 14:34:38.424 [     1] INFO  GridCacheProcessor                      
Started cache in recovery mode [name=MediaFan_Jobs, id=-2084285946,
dataRegionName=default, mode=PARTITIONED, atomicity=TRANSACTIONAL,
backups=1, mvcc=false]
23-03-2021 14:37:43.080 [     1] INFO  GridCacheProcessor                      
Started cache in recovery mode [name=MediaFan_MediaSetStatusHistory,
id=-852007084, dataRegionName=default, mode=PARTITIONED,
atomicity=TRANSACTIONAL, backups=1, mvcc=false]
23-03-2021 14:43:51.774 [     1] INFO  GridCacheProcessor                      
Started cache in recovery mode [name=MediaFan_BladeServices, id=492854502,
dataRegionName=default, mode=PARTITIONED, atomicity=TRANSACTIONAL,
backups=1, mvcc=false]

What does it mean "started cache in recovery mode"? Why are the caches in
recovery mode?

Thanks,
Thomas.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[Gianluca Bonetti]

Hi there

Il giorno dom 4 apr 2021 alle ore 21:39 DonTequila <[hidden email]> ha scritto:

Hi,

thanks for your suggestions. Unfortunately this didn't help improving the
startup time.

I only had such slow startup times because of the two problems I described before.

I had a look at your logs and there's no glaring problem, at least to my eyes.

Only a couple of things I spotted...

You're using JVM 11 but you didn't add the required VM arguments.

https://ignite.apache.org/docs/latest/quick-start/java#running-ignite-with-java-11-or-later

You're also running an old Ignite version, please consider updating to the latest.

Just a side note, how much data is stored in your caches?

Especially MediaFan_Jobs and MediaFan_MediaSetStatusHistory

Cheers

Gianluca