Long activation times with Ignite persistence enabled

classic Classic list List threaded Threaded
18 messages Options
avk avk
Reply | Threaded
Open this post in threaded view
|

Long activation times with Ignite persistence enabled

Hello,

I'm wondering if there is a way to improve the startup time of Ignite node when the persistence is enabled?

It seems the time is proportional to the size (and number) of the partition files. This is somewhat surprising as I expected the startup time be the same (plus-minus some constant factor) regardless of the amount of data persisted. 

The delay looks to be due to Ignite loading *all* partition files for *all* persistence-enabled caches as part of a node's join. Here's an example of the startup log output:

2018-01-18 14:00:40,230 INFO  [exchange-worker-#42%ignite-1%] GridCacheDatabaseSharedManager - Read checkpoint status [startMarker=/tmp/storage/data/1/cp/1516311778910-d56f8ceb-2205-4bef-9ed3-a7446e34aa06-START.bin, endMarker=/tmp/storage/data/1/cp/1516311778910-d56f8ceb-2205-4bef-9ed3-a7446e34aa06-END.bin]
2018-01-18 14:00:40,230 INFO  [exchange-worker-#42%ignite-1%] GridCacheDatabaseSharedManager - Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=1693, fileOff=7970054, len=60339], lastCheckpointId=d56f8ceb-2205-4bef-9ed3-a7446e34aa06]
2018-01-18 14:00:57,114 WARN  [exchange-worker-#42%ignite-1%] PageMemoryImpl - Page evictions started, this will affect storage performance (consider increasing DataRegionConfiguration#setMaxSize).
2018-01-18 14:02:05,469 INFO  [exchange-worker-#42%ignite-1%] GridCacheDatabaseSharedManager - Finished applying WAL changes [updatesApplied=0, time=85234ms]

It took ≈1.5 minute to activate a node. To add insult to injury, the eviction kicked in and most of the loaded pages got evicted (in this test, I had the caches sharing a 1GB memory region loading about 10GB of data and index). In general, I think it's not unreasonable to expect 1-to-10 ratio of the data region size to the total persisted data size.

Why load all that data in the first place? It seems like a huge waste of time. Can the data partitions be loaded lazily on demand while the index partition can still be loaded during node startup?

Thanks
Andrey

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

Hello!

As far as my understanding goes, there's no distinction in Ignite between
"data pages" and "index pages"
All pages in durable memory are parts of "B+ tree" and are holding both data
and references to downward branches in other pages.

I'm not sure it had to load all the pages. Maybe only ones that needed
applying updates from WAL? Which, in case of infrequent checkpoints, were
plentiful?

I expect most persistent databases would have non-trivial startup times when
full with data.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
avk avk
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

Hey Ilya!

In fact I was talking about pre-loading the index *partitions* not pages. But in any case, it's besides the point, just a potential enhancement.

What concerns me the most is how much time the activation takes and its apparent linear growth with the size of the on-disk data. In my case, I had only 10G of data on disk and that took 1.5 minutes from node start to the active state. Now, in production I'm looking at more than 100G of data per node (primary + backup partitions). Should I expect the activation to take 15 minutes? If this is indeed the case, then Ignite Persistence looses a lot of its initial appeal - "Instantaneous Cluster Restarts" - as touted here https://ignite.apache.org/features/durablememory.html. For comparison, on a 10Gpbs network I can transfer 100G in about 90 seconds (optimistically).

Next, to your point about "maybe applying updates from the WAL", I don't think that was the case (as indicated by "Finished applying WAL changes [updatesApplied=0, time=85234ms]" in the log output). Except for the initial load, the data has never changed between multiple restarts of the cluster (as well as of individual nodes). I even tried deleting all WAL files and the WAL archive to see if it makes any difference at all. In every instance the behavior was the same: 1.5 minutes of activation.

So, the question remains: are these activation times expected and the current state of the implementation is as good as it gets, or there is something (configuration-wise or in Ignite code) that can be done to improve the startup time?

Thanks!
Andrey


From: ilya.kasnacheev <[hidden email]>
Sent: Friday, January 19, 2018 4:29 AM
To: [hidden email]
Subject: Re: Long activation times with Ignite persistence enabled
 
Hello!

As far as my understanding goes, there's no distinction in Ignite between
"data pages" and "index pages"
All pages in durable memory are parts of "B+ tree" and are holding both data
and references to downward branches in other pages.

I'm not sure it had to load all the pages. Maybe only ones that needed
applying updates from WAL? Which, in case of infrequent checkpoints, were
plentiful?

I expect most persistent databases would have non-trivial startup times when
full with data.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
alexey.goncharuk alexey.goncharuk
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

In reply to this post by avk
Andrey,

Can you please describe in greater detail the configuration of your nodes (specifically, number of caches and number of partitions). Ignite would not load all the partitions into memory on startup simply because there is no such logic. What it does, however, is loading meta pages for each partition in each cache group to determine the correct cluster state and schedule rebalancing, if needed. If the number of caches x number of partitions is high, this may take a while.
If this is the case, you can either reduce the number of partitions or group logical caches with the same affinity into physical cache group, so that those caches will share the same partition file. See CacheConfiguration#setGroupName(String) for more detail.

Last but not least, it looks very suspicious that with 0 pending updates it took almost 90 seconds to read WAL. From the code, I see that this again may be related to partition state recovery, I will need to re-check this and get back to you later.

Thanks,
AG

2018-01-19 2:51 GMT+03:00 Andrey Kornev <[hidden email]>:
Hello,

I'm wondering if there is a way to improve the startup time of Ignite node when the persistence is enabled?

It seems the time is proportional to the size (and number) of the partition files. This is somewhat surprising as I expected the startup time be the same (plus-minus some constant factor) regardless of the amount of data persisted. 

The delay looks to be due to Ignite loading *all* partition files for *all* persistence-enabled caches as part of a node's join. Here's an example of the startup log output:

2018-01-18 14:00:40,230 INFO  [exchange-worker-#42%ignite-1%] GridCacheDatabaseSharedManager - Read checkpoint status [startMarker=/tmp/storage/data/1/cp/1516311778910-d56f8ceb-2205-4bef-9ed3-a7446e34aa06-START.bin, endMarker=/tmp/storage/data/1/cp/1516311778910-d56f8ceb-2205-4bef-9ed3-a7446e34aa06-END.bin]
2018-01-18 14:00:40,230 INFO  [exchange-worker-#42%ignite-1%] GridCacheDatabaseSharedManager - Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=1693, fileOff=7970054, len=60339], lastCheckpointId=d56f8ceb-2205-4bef-9ed3-a7446e34aa06]
2018-01-18 14:00:57,114 WARN  [exchange-worker-#42%ignite-1%] PageMemoryImpl - Page evictions started, this will affect storage performance (consider increasing DataRegionConfiguration#setMaxSize).
2018-01-18 14:02:05,469 INFO  [exchange-worker-#42%ignite-1%] GridCacheDatabaseSharedManager - Finished applying WAL changes [updatesApplied=0, time=85234ms]

It took ≈1.5 minute to activate a node. To add insult to injury, the eviction kicked in and most of the loaded pages got evicted (in this test, I had the caches sharing a 1GB memory region loading about 10GB of data and index). In general, I think it's not unreasonable to expect 1-to-10 ratio of the data region size to the total persisted data size.

Why load all that data in the first place? It seems like a huge waste of time. Can the data partitions be loaded lazily on demand while the index partition can still be loaded during node startup?

Thanks
Andrey


avk avk
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

Alexey,

Thanks a lot for looking into this!

My configuration is very basic: 3 caches all using standard 1024 partitions, sharing a 1GB persistent memory region.

Please find below the stack trace of the exchange worker thread captured while the node's activation is in progress (2.4 Ignite branch). 

Hope it helps!

Thanks!
Andrey

"exchange-worker-#42%ignite-2%" #82 prio=5 os_prio=31 tid=0x00007ffe8bf1c000 nid=0xc403 waiting on condition [0x000070000ed43000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
at org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:145)
at org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.read(AsyncFileIO.java:95)
at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:324)
at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:306)
at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:291)
at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:656)
at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:576)
at org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:130)
at org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.init(PagesList.java:212)
at org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.<init>(AbstractFreeList.java:367)
at org.apache.ignite.internal.processors.cache.persistence.freelist.CacheFreeListImpl.<init>(CacheFreeListImpl.java:47)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore$1.<init>(GridCacheOffheapManager.java:1041)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.init0(GridCacheOffheapManager.java:1041)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.updateCounter(GridCacheOffheapManager.java:1247)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.updateCounter(GridDhtLocalPartition.java:835)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.casState(GridDhtLocalPartition.java:523)
- locked <0x000000077a3d1120> (a org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.<init>(GridDhtLocalPartition.java:218)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.forceCreatePartition(GridDhtPartitionTopologyImpl.java:804)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restorePartitionState(GridCacheDatabaseSharedManager.java:2196)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:2155)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:1322)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:1113)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1063)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:661)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2329)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)

2018-01-22 11:30:01,049 INFO  [exchange-worker-#42%ContentStore-2%] GridCacheDatabaseSharedManager - Finished applying WAL changes [updatesApplied=0, time=68435ms]
2018-01-22 11:30:01,789 INFO  [main] GridDiscoveryManager - Topology snapshot [ver=4, servers=2, clients=0, CPUs=8, offheap=26.0GB, heap=4.0GB]
2018-01-22 11:30:01,789 INFO  [main] GridDiscoveryManager - Data Regions Configured:
2018-01-22 11:30:01,789 INFO  [main] GridDiscoveryManager -   ^-- default [initSize=256.0 MiB, maxSize=12.0 GiB, persistenceEnabled=false]
2018-01-22 11:30:01,789 INFO  [main] GridDiscoveryManager -   ^-- durable [initSize=256.0 MiB, maxSize=1.0 GiB, persistenceEnabled=true]




From: Alexey Goncharuk <[hidden email]>
Sent: Monday, January 22, 2018 12:30 AM
To: [hidden email]
Subject: Re: Long activation times with Ignite persistence enabled
 
Andrey,

Can you please describe in greater detail the configuration of your nodes (specifically, number of caches and number of partitions). Ignite would not load all the partitions into memory on startup simply because there is no such logic. What it does, however, is loading meta pages for each partition in each cache group to determine the correct cluster state and schedule rebalancing, if needed. If the number of caches x number of partitions is high, this may take a while.
If this is the case, you can either reduce the number of partitions or group logical caches with the same affinity into physical cache group, so that those caches will share the same partition file. See CacheConfiguration#setGroupName(String) for more detail.

Last but not least, it looks very suspicious that with 0 pending updates it took almost 90 seconds to read WAL. From the code, I see that this again may be related to partition state recovery, I will need to re-check this and get back to you later.

Thanks,
AG

2018-01-19 2:51 GMT+03:00 Andrey Kornev <[hidden email]>:
Hello,

I'm wondering if there is a way to improve the startup time of Ignite node when the persistence is enabled?

It seems the time is proportional to the size (and number) of the partition files. This is somewhat surprising as I expected the startup time be the same (plus-minus some constant factor) regardless of the amount of data persisted. 

The delay looks to be due to Ignite loading *all* partition files for *all* persistence-enabled caches as part of a node's join. Here's an example of the startup log output:

2018-01-18 14:00:40,230 INFO  [exchange-worker-#42%ignite-1%] GridCacheDatabaseSharedManager - Read checkpoint status [startMarker=/tmp/storage/data/1/cp/1516311778910-d56f8ceb-2205-4bef-9ed3-a7446e34aa06-START.bin, endMarker=/tmp/storage/data/1/cp/1516311778910-d56f8ceb-2205-4bef-9ed3-a7446e34aa06-END.bin]
2018-01-18 14:00:40,230 INFO  [exchange-worker-#42%ignite-1%] GridCacheDatabaseSharedManager - Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=1693, fileOff=7970054, len=60339], lastCheckpointId=d56f8ceb-2205-4bef-9ed3-a7446e34aa06]
2018-01-18 14:00:57,114 WARN  [exchange-worker-#42%ignite-1%] PageMemoryImpl - Page evictions started, this will affect storage performance (consider increasing DataRegionConfiguration#setMaxSize).
2018-01-18 14:02:05,469 INFO  [exchange-worker-#42%ignite-1%] GridCacheDatabaseSharedManager - Finished applying WAL changes [updatesApplied=0, time=85234ms]

It took ≈1.5 minute to activate a node. To add insult to injury, the eviction kicked in and most of the loaded pages got evicted (in this test, I had the caches sharing a 1GB memory region loading about 10GB of data and index). In general, I think it's not unreasonable to expect 1-to-10 ratio of the data region size to the total persisted data size.

Why load all that data in the first place? It seems like a huge waste of time. Can the data partitions be loaded lazily on demand while the index partition can still be loaded during node startup?

Thanks
Andrey


avk avk
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

Alexey,

I'm wondering you had a chance to look into this? I'd like to understand what to expect in terms of node activation time and how it's related to the data volume.

Thanks!
Andrey


From: Andrey Kornev <[hidden email]>
Sent: Monday, January 22, 2018 11:36 AM
To: Alexey Goncharuk; [hidden email]
Subject: Re: Long activation times with Ignite persistence enabled
 
Alexey,

Thanks a lot for looking into this!

My configuration is very basic: 3 caches all using standard 1024 partitions, sharing a 1GB persistent memory region.

Please find below the stack trace of the exchange worker thread captured while the node's activation is in progress (2.4 Ignite branch). 

Hope it helps!

Thanks!
Andrey

"exchange-worker-#42%ignite-2%" #82 prio=5 os_prio=31 tid=0x00007ffe8bf1c000 nid=0xc403 waiting on condition [0x000070000ed43000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
at org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:145)
at org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.read(AsyncFileIO.java:95)
at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:324)
at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:306)
at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:291)
at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:656)
at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:576)
at org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:130)
at org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.init(PagesList.java:212)
at org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.<init>(AbstractFreeList.java:367)
at org.apache.ignite.internal.processors.cache.persistence.freelist.CacheFreeListImpl.<init>(CacheFreeListImpl.java:47)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore$1.<init>(GridCacheOffheapManager.java:1041)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.init0(GridCacheOffheapManager.java:1041)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.updateCounter(GridCacheOffheapManager.java:1247)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.updateCounter(GridDhtLocalPartition.java:835)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.casState(GridDhtLocalPartition.java:523)
- locked <0x000000077a3d1120> (a org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.<init>(GridDhtLocalPartition.java:218)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.forceCreatePartition(GridDhtPartitionTopologyImpl.java:804)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restorePartitionState(GridCacheDatabaseSharedManager.java:2196)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:2155)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:1322)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:1113)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1063)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:661)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2329)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)

2018-01-22 11:30:01,049 INFO  [exchange-worker-#42%ContentStore-2%] GridCacheDatabaseSharedManager - Finished applying WAL changes [updatesApplied=0, time=68435ms]
2018-01-22 11:30:01,789 INFO  [main] GridDiscoveryManager - Topology snapshot [ver=4, servers=2, clients=0, CPUs=8, offheap=26.0GB, heap=4.0GB]
2018-01-22 11:30:01,789 INFO  [main] GridDiscoveryManager - Data Regions Configured:
2018-01-22 11:30:01,789 INFO  [main] GridDiscoveryManager -   ^-- default [initSize=256.0 MiB, maxSize=12.0 GiB, persistenceEnabled=false]
2018-01-22 11:30:01,789 INFO  [main] GridDiscoveryManager -   ^-- durable [initSize=256.0 MiB, maxSize=1.0 GiB, persistenceEnabled=true]


ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

Hello!

Can you please also provide full cache configuration and your hardware specs
(especially for storage)?
After 2.4 release is done, I hope your case will get attention of PDS
developers.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Naveen Naveen
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

Do we have any  update long activation times ?

I too face the same issue, am using 2.6.

Cluster with 100 GB of disk size, got activated in 5 minutes, and when I
tried with a cluster which has 3 TB is taking close to an hour.

Is it the expected behavior OR some configuration I am missing here

Thanks
 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

Naveen,

How many caches do you have?
As Alexey mentioned, usage of cache groups could reduce the number of created partitions and improve the startup time.

Denis

сб, 27 окт. 2018 г. в 11:12, Naveen <[hidden email]>:
Do we have any  update long activation times ?

I too face the same issue, am using 2.6.

Cluster with 100 GB of disk size, got activated in 5 minutes, and when I
tried with a cluster which has 3 TB is taking close to an hour.

Is it the expected behavior OR some configuration I am missing here

Thanks




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Naveen Naveen
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

Hi Denis

We have only 40 caches in our cluster.
If we introduce grouping of caches, guess we need to reload the data from
scratch, right ??

We do have very powerful machines as part of cluster, they are 128 CPU very
high end boxes and huge resources available, by increasing any of the below
thread pools, can we reduce the cluster activation time.

System Pool
Public Pool
Striped Pool
Custom Thread Pools

Thanks
Naveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Gianluca Bonetti Gianluca Bonetti
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

Hello

In my case of slow startup, as suggested from a member of this mailing list, I deleted the -XX:+AlwaysPreTouch command line option from JVM launch, and the cluster got back to very fast startup.
Don't know if you are using this option, hope it helps.

Cheers
Gianluca

Il giorno lun 5 nov 2018 alle ore 14:10 Naveen <[hidden email]> ha scritto:
Hi Denis

We have only 40 caches in our cluster.
If we introduce grouping of caches, guess we need to reload the data from
scratch, right ??

We do have very powerful machines as part of cluster, they are 128 CPU very
high end boxes and huge resources available, by increasing any of the below
thread pools, can we reduce the cluster activation time.

System Pool
Public Pool
Striped Pool
Custom Thread Pools

Thanks
Naveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

In reply to this post by Naveen
Naveen,

40 caches is quite a lot. It means, that Ignite needs to handle 40 * (number of partitions) files.
By default each cache has 1024 partitions.
This is quite a lot, and a disk is the bottleneck here. Changing of thread pool sizes won't save you.
If you divide your caches into cache groups, then they will share the same partitions, thus number of files will be reduced.
You can also try reducing the number of partitions, but it may lead to uneven distribution of data between nodes.
Any of these changes will require reloading of the data.

You can record a dstat on the host machine to make sure, that disk is the weak place.
If its utilization is high, while CPU is not used, then it means, that you need a faster disk.

Denis


пн, 5 нояб. 2018 г. в 17:10, Naveen <[hidden email]>:
Hi Denis

We have only 40 caches in our cluster.
If we introduce grouping of caches, guess we need to reload the data from
scratch, right ??

We do have very powerful machines as part of cluster, they are 128 CPU very
high end boxes and huge resources available, by increasing any of the below
thread pools, can we reduce the cluster activation time.

System Pool
Public Pool
Striped Pool
Custom Thread Pools

Thanks
Naveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Naveen Naveen
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

In reply to this post by Gianluca Bonetti
HI  Gianluca

Removing this  -XX:+AlwaysPreTouch did not help us at all, it took the same
time with or without this.
But I observed without this was, OS has not allocated entire heap memory we
have given as JVM options, we were giving 200G as heap for ignite node, so
when we execute top command, Ignite node used to use 250GB odd.
Now after removing AlwaysPreTouch, ignite node use only 50G (around).

Other than, activation time remains same with or without AlwaysPreTouch.

Thanks
Naveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Naveen Naveen
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

In reply to this post by Denis Mekhanikov
Hi Denis

We have already reduced the partition to 128, after which activation time
has come down a bit.

You were saying that, by reducing the partitions, it may lead to uneven
distribution of data between nodes. Isn't it the same when we go for cache
groups, group of caches will use the same resources /partitions, so here
also resource contention may be there right ?? here also same set of
partitions used by group of caches ?
If we use cache group, partition size may grow very high since all the
caches belong to that group will use the same set of partitions, does it
have any negative effect on the cluster performance ??



Thanks
Naveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Pavel Kovalenko Pavel Kovalenko
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

Hi Naveen and Andrey,

We've recently done major optimization https://issues.apache.org/jira/browse/IGNITE-9420 that will speed-up activation time in your case.
Iteration over WAL now happens only on a node start-up, so it will not affect activation anymore.
Partitions state restoring (which is the slowest part of the activation phase as I see in the first message in the thread) was also optimized. 
Now it is performed in parallel for each of available cache groups. Parallelism level of that operation is controlled by System Pool size. 
If you have enough CPU cores on your machines (more than the number of configured cache groups) you can adjust System pool size and your activation time will be significantly improved.

вт, 6 нояб. 2018 г. в 17:23, Naveen <[hidden email]>:
Hi Denis

We have already reduced the partition to 128, after which activation time
has come down a bit.

You were saying that, by reducing the partitions, it may lead to uneven
distribution of data between nodes. Isn't it the same when we go for cache
groups, group of caches will use the same resources /partitions, so here
also resource contention may be there right ?? here also same set of
partitions used by group of caches ?
If we use cache group, partition size may grow very high since all the
caches belong to that group will use the same set of partitions, does it
have any negative effect on the cluster performance ??



Thanks
Naveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Naveen Naveen
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

Hi Pavel

We are using Ignite 2.6
You were saying, usage of cache groups is definitely needed to improve to
the cluster activation time ?

I could see below documentation on the usage of cache groups.
 
*Should the grouping be used all the times?
With all the benefits the cache groups have, they might impact the
performance of read operations and indexes lookups. This is caused by the
fact that all the data and indexes get mixed in shared data structures
(partition maps, B+trees) and it will take more time querying over them.

Thus, consider using the cache groups if you have a cluster of dozens and
hundreds of nodes and caches, and you spot increased Java heap usage by
internal structures, checkpointing performance drop, slow node connectivity
to the cluster.
*

In our case, we do have around 50 caches, and at the max we may have 10
nodes. do you still recommend cache groups for our use case.

And, ours upserts TPS is very low, may be 1 k per sec, but query or read TPS
is quite high, close to 10 K TPS. So as per the above lines, it says
performance of read operations is impacted since all the caches are gong to
use the shared structures.
We are looking for a design which can improve the cluster activation time,
but not at the expense of compromising on the query performance, since our
solution being a read intensive, we cant afford to reduce query performance.
In worst scenario, we can live with poor cluster activation process as well,
since it only affects us at the time cluster restart which is performed only
in case of cluster crash or planned maintenance.

One more thing, if at all we need to change the system pool, below is the
command to change the system pool ??
IgniteConfiguration.setSystemThreadPoolSize(...)
We have 128 CPU machines, what would be the ideal system thread pool size ??
of course, it should be tried and tested but still some number..

Regarding the cache groups design, anything I should consider when we are
grouping the cache

1. We have around 40 caches, no indexes, we only have lookup on primary key,
some of them are simple keys and some of them are having complex primary
key. Some of the caches are queries together, does it help if we group them
into a cache group ?
2. What if we are trying query the caches which are part of different cache
groups ?
3. We are going to have close to half a billion records in each cache, so
how do we group them
4. Some of the caches are independent , does not have any relation with
other caches

So in case, I am going with cache group, shall I change the partition to 128
or keep the default ?


Thanks
Naveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Naveen Naveen
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

In reply to this post by Pavel Kovalenko
Hi Pavel

We are using Ignite 2.6
You were saying, usage of cache groups is definitely needed to improve to
the cluster activation time ?

I could see below documentation on the usage of cache groups.
 
Should the grouping be used all the times?
With all the benefits the cache groups have, they might impact the
performance of read operations and indexes lookups. This is caused by the
fact that all the data and indexes get mixed in shared data structures
(partition maps, B+trees) and it will take more time querying over them.

Thus, consider using the cache groups if you have a cluster of dozens and
hundreds of nodes and caches, and you spot increased Java heap usage by
internal structures, checkpointing performance drop, slow node connectivity
to the cluster.


In our case, we do have around 50 caches, and at the max we may have 10
nodes. do you still recommend cache groups for our use case.

And, ours upserts TPS is very low, may be 1 k per sec, but query or read TPS
is quite high, close to 10 K TPS. So as per the above lines, it says
performance of read operations is impacted since all the caches are gong to
use the shared structures.
We are looking for a design which can improve the cluster activation time,
but not at the expense of compromising on the query performance, since our
solution being a read intensive, we cant afford to reduce query performance.
In worst scenario, we can live with poor cluster activation process as well,
since it only affects us at the time cluster restart which is performed only
in case of cluster crash or planned maintenance.

One more thing, if at all we need to change the system pool, below is the
command to change the system pool ??
IgniteConfiguration.setSystemThreadPoolSize(...)
We have 128 CPU machines, what would be the ideal system thread pool size ??
of course, it should be tried and tested but still some number..

Regarding the cache groups design, anything I should consider when we are
grouping the cache

1. We have around 40 caches, no indexes, we only have lookup on primary key,
some of them are simple keys and some of them are having complex primary
key. Some of the caches are queries together, does it help if we group them
into a cache group ?
2. What if we are trying query the caches which are part of different cache
groups ?
3. We are going to have close to half a billion records in each cache, so
how do we group them
4. Some of the caches are independent , does not have any relation with
other caches

So in case, I am going with cache group, shall I change the partition to 128
or keep the default ?


Thanks
Naveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Naveen Naveen
Reply | Threaded
Open this post in threaded view
|

Re: Long activation times with Ignite persistence enabled

Do we have any update on this

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/