Slow startup of server node and query

classic Classic list List threaded Threaded
5 messages Options
DonTequila DonTequila
Reply | Threaded
Open this post in threaded view
|

Slow startup of server node and query

Hi,

I'm seeing a pretty slow Ignite server node startup time, of approx
20mins, with persistence enabled for most caches. Of course I can see
startup-time highly depends on disk speed, but I wonder if there are
settings that could improve this?

I've attached a server log of the time when it starts. Also the Ignite
configs attached.

Also, when first accessed from a client with SQL there's a long running
query which shouldn't be the case in my opinion for binary storage and
indexes?

Can you spot anything that is wrong with my system startup time?

Thanks,
Thomas.



server.log (102K) Download Attachment
ignite-caches-config.xml (39K) Download Attachment
ignite-caches-dataregions.xml (2K) Download Attachment
ignite-mediafan-db.xml (4K) Download Attachment
DonTequila DonTequila
Reply | Threaded
Open this post in threaded view
|

Re: Slow startup of server node and query

Has anyone been able to look at this and give me some advise? Thanks!


DonTequila wrote

> Hi,
>
> I'm seeing a pretty slow Ignite server node startup time, of approx
> 20mins, with persistence enabled for most caches. Of course I can see
> startup-time highly depends on disk speed, but I wonder if there are
> settings that could improve this?
>
> I've attached a server log of the time when it starts. Also the Ignite
> configs attached.
>
> Also, when first accessed from a client with SQL there's a long running
> query which shouldn't be the case in my opinion for binary storage and
> indexes?
>
> Can you spot anything that is wrong with my system startup time?
>
> Thanks,
> Thomas.
>
>
>
>
> server.log (102K)
> <http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/0/server.log>
> ignite-caches-config.xml (39K)
> <http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/1/ignite-caches-config.xml>
> ignite-caches-dataregions.xml (2K)
> <http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/2/ignite-caches-dataregions.xml>
> ignite-mediafan-db.xml (4K)
> <http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/3/ignite-mediafan-db.xml>





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Gianluca Bonetti Gianluca Bonetti
Reply | Threaded
Open this post in threaded view
|

Re: Slow startup of server node and query

Hello

It happened to me on a few occasions.
I have a couple of suggestions which may, or may not, solve your problem.
Please try :)

First, please ensure you use /dev/urandom as an entropy source.
/dev/random does a better job, but may cause long pauses (10 minutes or longer may happen easier than expected) and /dev/urandom is good enough for most cases.

Second, please ensure not to use -XX:+AlwaysPreTouch in your JVM flags as zeroing large heaps on initialization may take a very long time.

Cheers
Gianluca

Il giorno mer 31 mar 2021 alle ore 21:12 DonTequila <[hidden email]> ha scritto:
Has anyone been able to look at this and give me some advise? Thanks!


DonTequila wrote
> Hi,
>
> I'm seeing a pretty slow Ignite server node startup time, of approx
> 20mins, with persistence enabled for most caches. Of course I can see
> startup-time highly depends on disk speed, but I wonder if there are
> settings that could improve this?
>
> I've attached a server log of the time when it starts. Also the Ignite
> configs attached.
>
> Also, when first accessed from a client with SQL there's a long running
> query which shouldn't be the case in my opinion for binary storage and
> indexes?
>
> Can you spot anything that is wrong with my system startup time?
>
> Thanks,
> Thomas.
>
>
>
>
> server.log (102K)
> &lt;http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/0/server.log&gt;
> ignite-caches-config.xml (39K)
> &lt;http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/1/ignite-caches-config.xml&gt;
> ignite-caches-dataregions.xml (2K)
> &lt;http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/2/ignite-caches-dataregions.xml&gt;
> ignite-mediafan-db.xml (4K)
> &lt;http://apache-ignite-users.70518.x6.nabble.com/attachment/35676/3/ignite-mediafan-db.xml&gt;





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
DonTequila DonTequila
Reply | Threaded
Open this post in threaded view
|

Re: Slow startup of server node and query

Hi,

thanks for your suggestions. Unfortunately this didn't help improving the
startup time.

Looking at the log it takes most of the time (about 9 minutes) here:

23-03-2021 14:34:38.424 [     1] INFO  GridCacheProcessor                      
Started cache in recovery mode [name=MediaFan_Jobs, id=-2084285946,
dataRegionName=default, mode=PARTITIONED, atomicity=TRANSACTIONAL,
backups=1, mvcc=false]
23-03-2021 14:37:43.080 [     1] INFO  GridCacheProcessor                      
Started cache in recovery mode [name=MediaFan_MediaSetStatusHistory,
id=-852007084, dataRegionName=default, mode=PARTITIONED,
atomicity=TRANSACTIONAL, backups=1, mvcc=false]
23-03-2021 14:43:51.774 [     1] INFO  GridCacheProcessor                      
Started cache in recovery mode [name=MediaFan_BladeServices, id=492854502,
dataRegionName=default, mode=PARTITIONED, atomicity=TRANSACTIONAL,
backups=1, mvcc=false]

What does it mean "started cache in recovery mode"? Why are the caches in
recovery mode?

Thanks,
Thomas.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Gianluca Bonetti Gianluca Bonetti
Reply | Threaded
Open this post in threaded view
|

Re: Slow startup of server node and query

Hi there

Il giorno dom 4 apr 2021 alle ore 21:39 DonTequila <[hidden email]> ha scritto:
Hi,

thanks for your suggestions. Unfortunately this didn't help improving the
startup time.

I only had such slow startup times because of the two problems I described before.

I had a look at your logs and there's no glaring problem, at least to my eyes.
Only a couple of things I spotted...

You're using JVM 11 but you didn't add the required VM arguments.

You're also running an old Ignite version, please consider updating to the latest.

Just a side note, how much data is stored in your caches?
Especially MediaFan_Jobs and MediaFan_MediaSetStatusHistory

Cheers
Gianluca