Ignite startu is very slow

classic Classic list List threaded Threaded
22 messages Options
12
kvenkatramtreddy kvenkatramtreddy
Reply | Threaded
Open this post in threaded view
|

Ignite startu is very slow

Hi Team,

I am using Snapshot version of 2.7(Ignite ver.
2.7.0.20180806#19700101-sha1:DEV). Startup time is having between 8-30 mins.

My complete data set also very small, it is around 3 GB to 4 GB on Disk.



Please find attached logs and configuration in it.

example-ignite.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1700/example-ignite.xml>  
ignite.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t1700/ignite.log>  

FYI: I am using this snapshot because 2.6 was giving page corrupted or
Upperboud errors. It is stopped as soon as I have updated it to 2.7
Developer snapshot build.

Thanks & Regards,
Venkat




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Mikael Mikael
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

Hi!

This one looks a bit fishy to me:

"Failed to wait for initial partition map exchange. Possible reasons are:..."

Do you use transactions ?

Mikael
Den 2018-11-16 kl. 07:09, skrev kvenkatramtreddy:
Hi Team,

I am using Snapshot version of 2.7(Ignite ver.
2.7.0.20180806#19700101-sha1:DEV). Startup time is having between 8-30 mins.

My complete data set also very small, it is around 3 GB to 4 GB on Disk. 



Please find attached logs and configuration in it.

example-ignite.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1700/example-ignite.xml>  
ignite.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t1700/ignite.log>  

FYI: I am using this snapshot because 2.6 was giving page corrupted or
Upperboud errors. It is stopped as soon as I have updated it to 2.7
Developer snapshot build.

Thanks & Regards,
Venkat




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


kvenkatramtreddy kvenkatramtreddy
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

No, As you can see in the configuration it is ATOMIC mode. I did not write
any transactional code, just get and put and scanQuery.

Native persistence is enabled.

Most of the writes done by one node and other 2 nodes mostly read and less
write.

It is got only 3 nodes.

each node is 4 CPU and 8 GB RAM





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
kvenkatramtreddy kvenkatramtreddy
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startup is very slow

Hi Team,

Please could any body help me or guide me to solve this issues.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
kvenkatramtreddy kvenkatramtreddy
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startup is very slow

Hi Team,

any update on above ticket. any hints to investigate further. As per me, it
seems to be Partition Map exchange. But I am unable to understand why it is
taking time on startup. It is got only 3 nodes.

I had the ran the SAR command as well, it was showing iowait as 6-10%.



Even it is taking time on first node start up as well.

Can we do anything like, background thread to re-balance Partition map
exchange when cluster is online.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
kvenkatramtreddy kvenkatramtreddy
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

In reply to this post by kvenkatramtreddy
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

Hi,

How many heap do you have?

From logs I see, that before this, cluster was not stopped properly and the checkpoint was not saved to the disk. Because of that, after the start, nodes started to applying WAL changes(https://apacheignite.readme.io/docs/write-ahead-log). It took 1491578ms, probably because you have a very small heap size. Also, usually messages like "Possible too long JVM pause: 2710 milliseconds." is a symptom of a long GC pauses.

Evgenii

пт, 23 нояб. 2018 г. в 08:04, kvenkatramtreddy <[hidden email]>:
Please help.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
kvenkatramtreddy kvenkatramtreddy
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

Heap size is 3 GB.
And MaxDirectMemorySize is 1.5 GB.

My cache size is around 2 GB and on Disk around 5-6 GB maximum. My total
cache elements are around 50-60 thousands.

I will receive around 700-800 thousand messages for a day and I will update
that in cache.

So please could you let me know recommended Heap and Offheap settings for my
requirement so that it can start and faster.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

For this particular case, it's just enough to stop cluster gracefully before starting it again. Just deactivate cluster before stopping. In this case, when you will start the nodes again, they won't need to apply WAL changes.

Evgenii

вс, 25 нояб. 2018 г. в 04:54, kvenkatramtreddy <[hidden email]>:
Heap size is 3 GB.
And MaxDirectMemorySize is 1.5 GB.

My cache size is around 2 GB and on Disk around 5-6 GB maximum. My total
cache elements are around 50-60 thousands.

I will receive around 700-800 thousand messages for a day and I will update
that in cache.

So please could you let me know recommended Heap and Offheap settings for my
requirement so that it can start and faster.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
kvenkatramtreddy kvenkatramtreddy
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

Hi,

I have deActivated the cluster and closed, but still same issue persists.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

Can you share logs from a startup after deactivation?

Evgenii

пн, 26 нояб. 2018 г. в 20:58, kvenkatramtreddy <[hidden email]>:
Hi,

I have deActivated the cluster and closed, but still same issue persists.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
kvenkatramtreddy kvenkatramtreddy
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

Hi,
Please find attached log.
ignite_deactivate.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t1700/ignite_deactivate.log>  

whole cluster is de-activated and it is going in hung state.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Magda-2 Denis Magda-2
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startup is very slow

In reply to this post by kvenkatramtreddy
What do you see in the logs? Please share them. It's hard to guess what's going on in your environment.

--
Denis

On Sun, Nov 18, 2018 at 7:32 PM kvenkatramtreddy <[hidden email]> wrote:
Hi Team,

any update on above ticket. any hints to investigate further. As per me, it
seems to be Partition Map exchange. But I am unable to understand why it is
taking time on startup. It is got only 3 nodes.

I had the ran the SAR command as well, it was showing iowait as 6-10%.



Even it is taking time on first node start up as well.

Can we do anything like, background thread to re-balance Partition map
exchange when cluster is online.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

In reply to this post by kvenkatramtreddy
I've asked for the logs of node startup, not the log for stopping the cluster.

Evgenii

вт, 27 нояб. 2018 г. в 23:19, kvenkatramtreddy <[hidden email]>:
Hi,
Please find attached log.
ignite_deactivate.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t1700/ignite_deactivate.log

whole cluster is de-activated and it is going in hung state.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
kvenkatramtreddy kvenkatramtreddy
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

Hi,

This time it is got hung. so I have killed the nodes and restarted again.
Please find attached logs on startup.

node1.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t1700/node1.log>  
node2.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t1700/node2.log>  

As you mentioned in your earlier reply, it seems to be an issue WAL
applyLastUpdate changes. it is taking time there.

Is there anyway, can we write into disk as much as possible while cluster
Online.

fyi: second time restart I was unable to catch the logs, I have started the
nodes and left it and logs are overridden.

I was able to find the following log statement

[11/29/18 5:48:23:255 UTC] 00000090
.processors.cache.persistence.GridCacheDatabaseSharedManager I Finished
applying WAL changes [updatesApplied=0, time=1984500ms]


Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
kvenkatramtreddy kvenkatramtreddy
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

Hi ezhuravlev,

I have waited for more than 2 hours and captured the logs. please find
attached log in the post.

nodeStartup.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t1700/nodeStartup.log>
.

Please could you provide me the guidance to fix it.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
kvenkatramtreddy kvenkatramtreddy
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

hi Team,

any update or clue on above issue.

Thanks & Regards,
Venkat




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

Hi,

You have a pretty small amount of heap and 30 caches. Each cache creates some overhead for node startup, because node should read states for each partition of each cache(and each partition is a separate file). You can reduce this overhead by configuring the same cache group:https://apacheignite.readme.io/docs/cache-groups. Also, if you have only 3 nodes, it makes sense to reduce the number of partitions for each cache, by configuring affinity function.

Evgenii

пн, 3 дек. 2018 г. в 20:39, kvenkatramtreddy <[hidden email]>:
hi Team,

any update or clue on above issue.

Thanks & Regards,
Venkat




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
kvenkatramtreddy kvenkatramtreddy
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

Hi,

Please could you suggest the number of partitions for my caches, My Cache
size currently on Disk is 700 MB and smallest 10 MB.

only 5-6 caches are 700 MB rest of them are all around 100 MB.

Current I configured RendezvousAffinityFunction, let me know do you have
recommendations here.

    <bean
class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction"
id="rendezvousAffinityFunction" scope="prototype">
        <property name="excludeNeighbors" value="true"/>
    </bean>

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite startu is very slow

Hi,

I'd say that 128 partitions should be more than enough for 3 nodes, just try it. 

Note: you will need to reload all data in caches, because they will have different affinity.

Best Regards,
Evgenii

вт, 4 дек. 2018 г. в 02:36, kvenkatramtreddy <[hidden email]>:
Hi,

Please could you suggest the number of partitions for my caches, My Cache
size currently on Disk is 700 MB and smallest 10 MB.

only 5-6 caches are 700 MB rest of them are all around 100 MB.

Current I configured RendezvousAffinityFunction, let me know do you have
recommendations here.

    <bean
class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction"
id="rendezvousAffinityFunction" scope="prototype">
        <property name="excludeNeighbors" value="true"/>
    </bean>

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
12