Ever increasing startup times as data grow in persistent storage

classic Classic list List threaded Threaded
18 messages Options
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

Ever increasing startup times as data grow in persistent storage

We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.

This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.

After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).

We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.

We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.

We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.

Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 

Can anyone confirm this is expected behaviour and recommendations for resolving it?

Will reducing checking pointing intervals help?
Is the entire content of a partition read while applying WAL changes?
Does anyone else have this issue?

Thanks,
Raymond.


--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
[hidden email]
Naveen Naveen
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage

Hi Raymond

Did you try checkpointPageBufferSize instead of time interval, we have used
24MB as checkpointPageBufferSize , working fine for us, we also have close
to 12 TB of data and does take good 6 to 10 mts to bring up the node and
become cluster active
Regarding the no of partitions also, 128 partitions should do and its doing
good for us

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Zhenya Stanilovsky Zhenya Stanilovsky
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage

In reply to this post by Raymond Wilson



 
We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.
 
This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.
 
After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).
 
We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.
 
We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.
 
We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.
 
Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 
 
Can anyone confirm this is expected behaviour and recommendations for resolving it?
 
Will reducing checking pointing intervals help?
 
yes, it will help. Check https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
Is the entire content of a partition read while applying WAL changes?
 
don`t think so, may be someone else suggest here?
Does anyone else have this issue?
 
Thanks,
Raymond.
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
Kirill Tkalenko Kirill Tkalenko
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage

In reply to this post by Raymond Wilson
Hello, Raymond! What version are you using?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage

In reply to this post by Naveen
Hi Naveen,

We currently have two data regions. A small one (for ingest), set to 128 Mb, and a larger one for requests (4Gb). We leave the checkpoint page buffer size at the default value, so this will be 1Gb for the larger region, and possibly 128Mb for the smaller region (if I recall the rules correctly). I'm planning to combine them to improve checkpointing behaviour.

I guess what you're saying is by setting the buffer size smaller, you cap the volume of WAL updates that need to be applied when restarting your nodes?

Sounds like something worth trying...

I guess the risk here is that if the number of dirty pages hits that limit during a checkpoint then Ignite will block writes until that check point is complete, and schedule another checkpoint immediately after the first has completed. Do you see that occurring in your system?

Thanks,
Raymond

On Wed, Jan 13, 2021 at 7:52 PM Naveen <[hidden email]> wrote:
Hi Raymond

Did you try checkpointPageBufferSize instead of time interval, we have used
24MB as checkpointPageBufferSize , working fine for us, we also have close
to 12 TB of data and does take good 6 to 10 mts to bring up the node and
become cluster active
Regarding the no of partitions also, 128 partitions should do and its doing
good for us

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
[hidden email]
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage

In reply to this post by Kirill Tkalenko
We are currently using AI 2.8.1 with the c# client.

On Wed, Jan 13, 2021 at 8:12 PM Kirill Tkalenko <[hidden email]> wrote:
Hello, Raymond! What version are you using?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
[hidden email]
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage

In reply to this post by Zhenya Stanilovsky
Hi Zhenya,

Thanks for confirming performing checkpoints more often will help here.

I have established this configuration so will experiment with settings little.

On a related note, is there any way to automatically trigger a checkpoint, for instance as a pre-shutdown activity? Checkpoints seem to be much faster than the process of applying WAL updates.

Raymond.

On Wed, Jan 13, 2021 at 8:07 PM Zhenya Stanilovsky <[hidden email]> wrote:



 
We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.
 
This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.
 
After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).
 
We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.
 
We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.
 
We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.
 
Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 
 
Can anyone confirm this is expected behaviour and recommendations for resolving it?
 
Will reducing checking pointing intervals help?
 
Is the entire content of a partition read while applying WAL changes?
 
don`t think so, may be someone else suggest here?
Does anyone else have this issue?
 
Thanks,
Raymond.
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 


--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
[hidden email]
Zhenya Stanilovsky Zhenya Stanilovsky
Reply | Threaded
Open this post in threaded view
|

Re[2]: Ever increasing startup times as data grow in persistent storage



 
Hi Zhenya,
 
Thanks for confirming performing checkpoints more often will help here.
Hi Raymond !
 
I have established this configuration so will experiment with settings little.
 
On a related note, is there any way to automatically trigger a checkpoint, for instance as a pre-shutdown activity?
 
If you shutdown your cluster gracefully = with deactivation [1] further start will not trigger wal readings.
 
[1] https://www.gridgain.com/docs/latest/administrators-guide/control-script#deactivating-cluster
 
Checkpoints seem to be much faster than the process of applying WAL updates.
 
Raymond.
 
On Wed, Jan 13, 2021 at 8:07 PM Zhenya Stanilovsky <arzamas123@...> wrote:



 
We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.
 
This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.
 
After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).
 
We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.
 
We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.
 
We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.
 
Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 
 
Can anyone confirm this is expected behaviour and recommendations for resolving it?
 
Will reducing checking pointing intervals help?
 
Is the entire content of a partition read while applying WAL changes?
 
don`t think so, may be someone else suggest here?
Does anyone else have this issue?
 
Thanks,
Raymond.
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

Re: Re[2]: Ever increasing startup times as data grow in persistent storage

Thank Zhenya.

Currently we call Ignition.Stop() with the flag to allow jobs to complete. I assume when using deactivate we don;t need to call that, or is it still a good idea as a belt and braces shut down for the grid?

Raymond

On Wed, Jan 13, 2021 at 8:28 PM Zhenya Stanilovsky <[hidden email]> wrote:


 
Hi Zhenya,
 
Thanks for confirming performing checkpoints more often will help here.
Hi Raymond !
 
I have established this configuration so will experiment with settings little.
 
On a related note, is there any way to automatically trigger a checkpoint, for instance as a pre-shutdown activity?
 
If you shutdown your cluster gracefully = with deactivation [1] further start will not trigger wal readings.
 
 
Checkpoints seem to be much faster than the process of applying WAL updates.
 
Raymond.
 
On Wed, Jan 13, 2021 at 8:07 PM Zhenya Stanilovsky <arzamas123@...> wrote:



 
We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.
 
This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.
 
After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).
 
We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.
 
We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.
 
We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.
 
Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 
 
Can anyone confirm this is expected behaviour and recommendations for resolving it?
 
Will reducing checking pointing intervals help?
 
Is the entire content of a partition read while applying WAL changes?
 
don`t think so, may be someone else suggest here?
Does anyone else have this issue?
 
Thanks,
Raymond.
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 


--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
[hidden email]
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

Re: Re[2]: Ever increasing startup times as data grow in persistent storage

In reply to this post by Zhenya Stanilovsky
Is there an API version of the cluster deactivation?

On Wed, Jan 13, 2021 at 8:28 PM Zhenya Stanilovsky <[hidden email]> wrote:


 
Hi Zhenya,
 
Thanks for confirming performing checkpoints more often will help here.
Hi Raymond !
 
I have established this configuration so will experiment with settings little.
 
On a related note, is there any way to automatically trigger a checkpoint, for instance as a pre-shutdown activity?
 
If you shutdown your cluster gracefully = with deactivation [1] further start will not trigger wal readings.
 
 
Checkpoints seem to be much faster than the process of applying WAL updates.
 
Raymond.
 
On Wed, Jan 13, 2021 at 8:07 PM Zhenya Stanilovsky <arzamas123@...> wrote:



 
We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.
 
This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.
 
After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).
 
We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.
 
We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.
 
We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.
 
Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 
 
Can anyone confirm this is expected behaviour and recommendations for resolving it?
 
Will reducing checking pointing intervals help?
 
Is the entire content of a partition read while applying WAL changes?
 
don`t think so, may be someone else suggest here?
Does anyone else have this issue?
 
Thanks,
Raymond.
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 


--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
[hidden email]
Zhenya Stanilovsky Zhenya Stanilovsky
Reply | Threaded
Open this post in threaded view
|

Re[4]: Ever increasing startup times as data grow in persistent storage




 
Is there an API version of the cluster deactivation?
 
 
On Wed, Jan 13, 2021 at 8:28 PM Zhenya Stanilovsky <arzamas123@...> wrote:


 
Hi Zhenya,
 
Thanks for confirming performing checkpoints more often will help here.
Hi Raymond !
 
I have established this configuration so will experiment with settings little.
 
On a related note, is there any way to automatically trigger a checkpoint, for instance as a pre-shutdown activity?
 
If you shutdown your cluster gracefully = with deactivation [1] further start will not trigger wal readings.
 
 
Checkpoints seem to be much faster than the process of applying WAL updates.
 
Raymond.
 
On Wed, Jan 13, 2021 at 8:07 PM Zhenya Stanilovsky <arzamas123@...> wrote:



 
We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.
 
This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.
 
After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).
 
We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.
 
We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.
 
We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.
 
Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 
 
Can anyone confirm this is expected behaviour and recommendations for resolving it?
 
Will reducing checking pointing intervals help?
 
Is the entire content of a partition read while applying WAL changes?
 
don`t think so, may be someone else suggest here?
Does anyone else have this issue?
 
Thanks,
Raymond.
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage

Of course. Obvious! :)

Sent from my iPhone

On 13/01/2021, at 9:15 PM, Zhenya Stanilovsky <[hidden email]> wrote:





 
Is there an API version of the cluster deactivation?
 
 
On Wed, Jan 13, 2021 at 8:28 PM Zhenya Stanilovsky <arzamas123@...> wrote:


 
Hi Zhenya,
 
Thanks for confirming performing checkpoints more often will help here.
Hi Raymond !
 
I have established this configuration so will experiment with settings little.
 
On a related note, is there any way to automatically trigger a checkpoint, for instance as a pre-shutdown activity?
 
If you shutdown your cluster gracefully = with deactivation [1] further start will not trigger wal readings.
 
 
Checkpoints seem to be much faster than the process of applying WAL updates.
 
Raymond.
 
On Wed, Jan 13, 2021 at 8:07 PM Zhenya Stanilovsky <arzamas123@...> wrote:



 
We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.
 
This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.
 
After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).
 
We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.
 
We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.
 
We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.
 
Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 
 
Can anyone confirm this is expected behaviour and recommendations for resolving it?
 
Will reducing checking pointing intervals help?
 
Is the entire content of a partition read while applying WAL changes?
 
don`t think so, may be someone else suggest here?
Does anyone else have this issue?
 
Thanks,
Raymond.
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage


On Wed, Jan 13, 2021 at 11:54 AM Raymond Wilson <[hidden email]> wrote:
Of course. Obvious! :)

Sent from my iPhone

On 13/01/2021, at 9:15 PM, Zhenya Stanilovsky <[hidden email]> wrote:





 
Is there an API version of the cluster deactivation?
 
 
On Wed, Jan 13, 2021 at 8:28 PM Zhenya Stanilovsky <arzamas123@...> wrote:


 
Hi Zhenya,
 
Thanks for confirming performing checkpoints more often will help here.
Hi Raymond !
 
I have established this configuration so will experiment with settings little.
 
On a related note, is there any way to automatically trigger a checkpoint, for instance as a pre-shutdown activity?
 
If you shutdown your cluster gracefully = with deactivation [1] further start will not trigger wal readings.
 
 
Checkpoints seem to be much faster than the process of applying WAL updates.
 
Raymond.
 
On Wed, Jan 13, 2021 at 8:07 PM Zhenya Stanilovsky <arzamas123@...> wrote:



 
We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.
 
This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.
 
After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).
 
We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.
 
We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.
 
We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.
 
Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 
 
Can anyone confirm this is expected behaviour and recommendations for resolving it?
 
Will reducing checking pointing intervals help?
 
Is the entire content of a partition read while applying WAL changes?
 
don`t think so, may be someone else suggest here?
Does anyone else have this issue?
 
Thanks,
Raymond.
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
Naveen Naveen
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage

In reply to this post by Raymond Wilson
Hi Raymond

It does block writes until the checkpoint is complete, but this only happens
when we restart our nodes, that time all the piled up requests (during the
shutdown) gets processed, thats when bulk data ingestion happens, otherwise
for normal day to day real time operations it does not really hurt us since
we do not have any bulk writes etc.

Thanks




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage

In reply to this post by ptupitsyn
We have been experimenting with using deactivation to shutdown the grid to reduce the time for the grid to start up again.

It appears there is a downside to this: once deactivated the grid does not appear to auto-activate once baseline topology is achieved, which means we will need to run through the bootstrapping protocol of ensuring the grid has restarted correctly before activating it once again.

The baseline topology documentation at https://ignite.apache.org/docs/latest/clustering/baseline-topology does not cover this condition.

Is this expected?

Thanks,
Raymond.


On Wed, Jan 13, 2021 at 11:49 PM Pavel Tupitsyn <[hidden email]> wrote:

On Wed, Jan 13, 2021 at 11:54 AM Raymond Wilson <[hidden email]> wrote:
Of course. Obvious! :)

Sent from my iPhone

On 13/01/2021, at 9:15 PM, Zhenya Stanilovsky <[hidden email]> wrote:





 
Is there an API version of the cluster deactivation?
 
 
On Wed, Jan 13, 2021 at 8:28 PM Zhenya Stanilovsky <arzamas123@...> wrote:


 
Hi Zhenya,
 
Thanks for confirming performing checkpoints more often will help here.
Hi Raymond !
 
I have established this configuration so will experiment with settings little.
 
On a related note, is there any way to automatically trigger a checkpoint, for instance as a pre-shutdown activity?
 
If you shutdown your cluster gracefully = with deactivation [1] further start will not trigger wal readings.
 
 
Checkpoints seem to be much faster than the process of applying WAL updates.
 
Raymond.
 
On Wed, Jan 13, 2021 at 8:07 PM Zhenya Stanilovsky <arzamas123@...> wrote:



 
We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.
 
This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.
 
After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).
 
We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.
 
We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.
 
We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.
 
Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 
 
Can anyone confirm this is expected behaviour and recommendations for resolving it?
 
Will reducing checking pointing intervals help?
 
Is the entire content of a partition read while applying WAL changes?
 
don`t think so, may be someone else suggest here?
Does anyone else have this issue?
 
Thanks,
Raymond.
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
     
 
 
 
 
 


--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
[hidden email]
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage

Hi,

Yes, that was to be expected. The main autoactivation scenario is cluster restart. If you are using manual deactivation, you should also manually activate your cluster.

BR,
Andrei

1/20/2021 5:50 AM, Raymond Wilson пишет:
We have been experimenting with using deactivation to shutdown the grid to reduce the time for the grid to start up again.

It appears there is a downside to this: once deactivated the grid does not appear to auto-activate once baseline topology is achieved, which means we will need to run through the bootstrapping protocol of ensuring the grid has restarted correctly before activating it once again.

The baseline topology documentation at https://ignite.apache.org/docs/latest/clustering/baseline-topology does not cover this condition.

Is this expected?

Thanks,
Raymond.


On Wed, Jan 13, 2021 at 11:49 PM Pavel Tupitsyn <[hidden email]> wrote:

On Wed, Jan 13, 2021 at 11:54 AM Raymond Wilson <[hidden email]> wrote:
Of course. Obvious! :)

Sent from my iPhone

On 13/01/2021, at 9:15 PM, Zhenya Stanilovsky <[hidden email]> wrote:





 
Is there an API version of the cluster deactivation?
 
 
On Wed, Jan 13, 2021 at 8:28 PM Zhenya Stanilovsky <arzamas123@...> wrote:


 
Hi Zhenya,
 
Thanks for confirming performing checkpoints more often will help here.
Hi Raymond !
 
I have established this configuration so will experiment with settings little.
 
On a related note, is there any way to automatically trigger a checkpoint, for instance as a pre-shutdown activity?
 
If you shutdown your cluster gracefully = with deactivation [1] further start will not trigger wal readings.
 
 
Checkpoints seem to be much faster than the process of applying WAL updates.
 
Raymond.
 
On Wed, Jan 13, 2021 at 8:07 PM Zhenya Stanilovsky <arzamas123@...> wrote:



 
We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.
 
This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.
 
After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).
 
We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.
 
We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.
 
We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.
 
Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 
 
Can anyone confirm this is expected behaviour and recommendations for resolving it?
 
Will reducing checking pointing intervals help?
 
Is the entire content of a partition read while applying WAL changes?
 
don`t think so, may be someone else suggest here?
Does anyone else have this issue?
 
Thanks,
Raymond.
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
         
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
         
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
         
 
 
 
 
 


--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
[hidden email]






Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage

Hi Andre,

I would like to see Ignite support a graceful shutdown scenario you get with deactivation, but which does not need to be manually reactivated.

We run a pretty agile process and it is not uncommon to have multiple deploys to production throughout a week. This is a pretty automated affair (essentially push-button) and it works well, except for the WAL rescan on startup.

Today there are two approaches we can take for a deployment:

1. Stop the nodes (which is what we currently do), leaving the WAL and persistent store inconsistent. This requires a rescan of the WAL before the grid is auto re-activated on startup. The time to do this is increasing with the size of the persistent store - it does not appear to be related to the size of the WAL.
2. Deactivate the grid, which leaves the WAL and persistent store in a consistent state. This requires manual re-activation on restart, but does not incur the increasing WAL restart cost.

Is an option like the one below possible?:

3. Suspend the grid, which performs the same steps deactivation does to make the WAL and persistent store consistent, but which leaves the grid activated so the manual activation process is not required on restart.

Thanks,
Raymond.
 

On Thu, Jan 21, 2021 at 4:02 AM andrei <[hidden email]> wrote:

Hi,

Yes, that was to be expected. The main autoactivation scenario is cluster restart. If you are using manual deactivation, you should also manually activate your cluster.

BR,
Andrei

1/20/2021 5:50 AM, Raymond Wilson пишет:
We have been experimenting with using deactivation to shutdown the grid to reduce the time for the grid to start up again.

It appears there is a downside to this: once deactivated the grid does not appear to auto-activate once baseline topology is achieved, which means we will need to run through the bootstrapping protocol of ensuring the grid has restarted correctly before activating it once again.

The baseline topology documentation at https://ignite.apache.org/docs/latest/clustering/baseline-topology does not cover this condition.

Is this expected?

Thanks,
Raymond.


On Wed, Jan 13, 2021 at 11:49 PM Pavel Tupitsyn <[hidden email]> wrote:

On Wed, Jan 13, 2021 at 11:54 AM Raymond Wilson <[hidden email]> wrote:
Of course. Obvious! :)

Sent from my iPhone

On 13/01/2021, at 9:15 PM, Zhenya Stanilovsky <[hidden email]> wrote:





 
Is there an API version of the cluster deactivation?
 
 
On Wed, Jan 13, 2021 at 8:28 PM Zhenya Stanilovsky <arzamas123@...> wrote:


 
Hi Zhenya,
 
Thanks for confirming performing checkpoints more often will help here.
Hi Raymond !
 
I have established this configuration so will experiment with settings little.
 
On a related note, is there any way to automatically trigger a checkpoint, for instance as a pre-shutdown activity?
 
If you shutdown your cluster gracefully = with deactivation [1] further start will not trigger wal readings.
 
 
Checkpoints seem to be much faster than the process of applying WAL updates.
 
Raymond.
 
On Wed, Jan 13, 2021 at 8:07 PM Zhenya Stanilovsky <arzamas123@...> wrote:



 
We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.
 
This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.
 
After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).
 
We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.
 
We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.
 
We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.
 
Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 
 
Can anyone confirm this is expected behaviour and recommendations for resolving it?
 
Will reducing checking pointing intervals help?
 
Is the entire content of a partition read while applying WAL changes?
 
don`t think so, may be someone else suggest here?
Does anyone else have this issue?
 
Thanks,
Raymond.
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
         
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
         
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
         
 
 
 
 
 


--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
[hidden email]








--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
[hidden email]
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Ever increasing startup times as data grow in persistent storage

Hi,

I don't think there are any other options at the moment other than the ones you mentioned.

However, you can also create your own application that will check the topology and activate it when all nodes from the baseline are online. For example, additional java code when starting a server node.

In case you require any changes to the current Ignite implementation, you can create a thread in the Ignite developer list:

http://apache-ignite-developers.2346864.n4.nabble.com/

BR,
Andrei


1/20/2021 9:16 PM, Raymond Wilson пишет:
Hi Andre,

I would like to see Ignite support a graceful shutdown scenario you get with deactivation, but which does not need to be manually reactivated.

We run a pretty agile process and it is not uncommon to have multiple deploys to production throughout a week. This is a pretty automated affair (essentially push-button) and it works well, except for the WAL rescan on startup.

Today there are two approaches we can take for a deployment:

1. Stop the nodes (which is what we currently do), leaving the WAL and persistent store inconsistent. This requires a rescan of the WAL before the grid is auto re-activated on startup. The time to do this is increasing with the size of the persistent store - it does not appear to be related to the size of the WAL.
2. Deactivate the grid, which leaves the WAL and persistent store in a consistent state. This requires manual re-activation on restart, but does not incur the increasing WAL restart cost.

Is an option like the one below possible?:

3. Suspend the grid, which performs the same steps deactivation does to make the WAL and persistent store consistent, but which leaves the grid activated so the manual activation process is not required on restart.

Thanks,
Raymond.
 

On Thu, Jan 21, 2021 at 4:02 AM andrei <[hidden email]> wrote:

Hi,

Yes, that was to be expected. The main autoactivation scenario is cluster restart. If you are using manual deactivation, you should also manually activate your cluster.

BR,
Andrei

1/20/2021 5:50 AM, Raymond Wilson пишет:
We have been experimenting with using deactivation to shutdown the grid to reduce the time for the grid to start up again.

It appears there is a downside to this: once deactivated the grid does not appear to auto-activate once baseline topology is achieved, which means we will need to run through the bootstrapping protocol of ensuring the grid has restarted correctly before activating it once again.

The baseline topology documentation at https://ignite.apache.org/docs/latest/clustering/baseline-topology does not cover this condition.

Is this expected?

Thanks,
Raymond.


On Wed, Jan 13, 2021 at 11:49 PM Pavel Tupitsyn <[hidden email]> wrote:

On Wed, Jan 13, 2021 at 11:54 AM Raymond Wilson <[hidden email]> wrote:
Of course. Obvious! :)

Sent from my iPhone

On 13/01/2021, at 9:15 PM, Zhenya Stanilovsky <[hidden email]> wrote:





 
Is there an API version of the cluster deactivation?
 
 
On Wed, Jan 13, 2021 at 8:28 PM Zhenya Stanilovsky <arzamas123@...> wrote:


 
Hi Zhenya,
 
Thanks for confirming performing checkpoints more often will help here.
Hi Raymond !
 
I have established this configuration so will experiment with settings little.
 
On a related note, is there any way to automatically trigger a checkpoint, for instance as a pre-shutdown activity?
 
If you shutdown your cluster gracefully = with deactivation [1] further start will not trigger wal readings.
 
 
Checkpoints seem to be much faster than the process of applying WAL updates.
 
Raymond.
 
On Wed, Jan 13, 2021 at 8:07 PM Zhenya Stanilovsky <arzamas123@...> wrote:



 
We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows.
 
This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped.
 
After enabling debug logging we see that the WAL file is scanned, and for every cache, all partitions in the cache are examined, and if there are any uncommitted changes in the WAL file then the partition is updated (I assume this requires reading of the partition itself as a part of this process).
 
We now have ~150Gb of data in our persistent store and we see WAL update times between 5-10 minutes to complete, during which the node is unavailable.
 
We use fairly large WAL files (512Mb) and use 10 segments, with WAL archiving enabled.
 
We anticipate data in persistent storage to grow to Terabytes, and if the startup time continues to grow as storage grows then this makes deploys and restarts difficult.
 
Until now we have been using the default checkpoint time out of 3 minutes which may mean we have significant uncheckpointed data in the WAL files. We are moving to 1 minute checkpoint but don't yet know if this improve startup times. We also use the default 1024 partitions per cache, though some partitions may be large. 
 
Can anyone confirm this is expected behaviour and recommendations for resolving it?
 
Will reducing checking pointing intervals help?
 
Is the entire content of a partition read while applying WAL changes?
 
don`t think so, may be someone else suggest here?
Does anyone else have this issue?
 
Thanks,
Raymond.
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
         
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
         
 
 
 
 
 
 
 
--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@...
         
 
 
 
 
 


--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
[hidden email]








--

Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
[hidden email]