Ignite WAL and WAL archive size estimation

classic Classic list List threaded Threaded
4 messages Options
Venkata Bhagavatula Venkata Bhagavatula
Reply | Threaded
Open this post in threaded view
|

Ignite WAL and WAL archive size estimation

Hi All,


Following is mentioned about the Estimation:

what is est. maximum data volume to be writen on 1 checkpoint?  Is it the size of 1 wal segment or 1 checkpoint buffer size? 

Estimating disk space

WAL Work maximum used size: walSegmentSize * walSegments = 640Mb (default)

in case Default WAL mode - this size is used always,

in case other modes best case is 1 segment * walSegmentSize

WAL Work+WAL Archive max size may be estimated by

  1. average load or
  2. by maximum size.

1st way is applicable if checkpoints are triggered mostly by timer trigger. 
Wal size = 2*Average load(bytes/sec) * trigger interval (sec) * walHistSize (number of checkpoints)
Where 2 multiplier coming from physical & logical WAL Records.

2nd way: Checkpoint is triggered by segments max dirty pages percent. Use persisted data regions max sizes:
sum(Max configured DataRegionConfiguration.maxSize) * 75% - est. maximum data volume to be writen on 1 checkpoint.
Overall WAL size (before archiving) = 2* est. data volume * walHistSize = 1,5 * sum(DataRegionConfiguration.maxSize) * walHistSize 

Note applying WAL compressor may significiantly reduce archive size.


Thanks n Regards,

Chal

Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite WAL and WAL archive size estimation

Hi,

In normal circumstances checkpoint is triggered on timeout, e.g. every 3 minutes (controlled by checkpointFrequency). So, the size of the checkpoint is the amount of data written/updated in a 3-minute interval.
The best way to estimate it in your system is to enable data storage metrics (DataStorageMetrics.setMetricsEnabled(true)) and check the metric getLastCheckpointTotalPagesNumber().

Stan

On Mon, Aug 26, 2019 at 4:59 PM Venkata Bhagavatula <[hidden email]> wrote:
Hi All,


Following is mentioned about the Estimation:

what is est. maximum data volume to be writen on 1 checkpoint?  Is it the size of 1 wal segment or 1 checkpoint buffer size? 

Estimating disk space

WAL Work maximum used size: walSegmentSize * walSegments = 640Mb (default)

in case Default WAL mode - this size is used always,

in case other modes best case is 1 segment * walSegmentSize

WAL Work+WAL Archive max size may be estimated by

  1. average load or
  2. by maximum size.

1st way is applicable if checkpoints are triggered mostly by timer trigger. 
Wal size = 2*Average load(bytes/sec) * trigger interval (sec) * walHistSize (number of checkpoints)
Where 2 multiplier coming from physical & logical WAL Records.

2nd way: Checkpoint is triggered by segments max dirty pages percent. Use persisted data regions max sizes:
sum(Max configured DataRegionConfiguration.maxSize) * 75% - est. maximum data volume to be writen on 1 checkpoint.
Overall WAL size (before archiving) = 2* est. data volume * walHistSize = 1,5 * sum(DataRegionConfiguration.maxSize) * walHistSize 

Note applying WAL compressor may significiantly reduce archive size.


Thanks n Regards,

Chal

akurbanov akurbanov
Reply | Threaded
Open this post in threaded view
|

RE: Ignite WAL and WAL archive size estimation

In reply to this post by Venkata Bhagavatula

Hello,

 

WAL active folder size calculation is correct, that is walSegmentSize * walSegment = 64 * 10 = 640Mb.

 

However, you may completely disregard estimations for WAL archive, as since latest 2.7 version there is a configuration property to limit WAL archive size in bytes, which is obviously more precise way to do so.

 

Ticket for your reference: https://issues.apache.org/jira/browse/IGNITE-6552

Docs link: https://apacheignite.readme.io/docs/write-ahead-log#section-wal-archive

 

Though there is the same link is on the under the hood page in WAL structure part, the ‘estimating disk space’ part has to be updated, thanks for your question!

 

Best regards,

Anton

From: [hidden email]
Sent: Monday, August 26, 2019 4:59 PM
To: [hidden email]
Subject: Ignite WAL and WAL archive size estimation

 

Hi All,

 

 

Following is mentioned about the Estimation:

what is est. maximum data volume to be writen on 1 checkpoint?  Is it the size of 1 wal segment or 1 checkpoint buffer size? 

Estimating disk space

WAL Work maximum used size: walSegmentSize * walSegments = 640Mb (default)

in case Default WAL mode - this size is used always,

in case other modes best case is 1 segment * walSegmentSize

WAL Work+WAL Archive max size may be estimated by

1.      average load or

2.      by maximum size.

1st way is applicable if checkpoints are triggered mostly by timer trigger. 
Wal size = 2*Average load(bytes/sec) * trigger interval (sec) * walHistSize (number of checkpoints)
Where 2 multiplier coming from physical & logical WAL Records.

2nd way: Checkpoint is triggered by segments max dirty pages percent. Use persisted data regions max sizes:
sum(Max configured DataRegionConfiguration.maxSize) * 75% - est. maximum data volume to be writen on 1 checkpoint.
Overall WAL size (before archiving) = 2* est. data volume * walHistSize = 1,5 * sum(DataRegionConfiguration.maxSize) * walHistSize 

Note applying WAL compressor may significiantly reduce archive size.

 

Thanks n Regards,

Chal

 

Venkata Bhagavatula Venkata Bhagavatula
Reply | Threaded
Open this post in threaded view
|

Re: Ignite WAL and WAL archive size estimation

Thanks for your response.

On Mon, Aug 26, 2019 at 10:59 PM Anton <[hidden email]> wrote:

Hello,

 

WAL active folder size calculation is correct, that is walSegmentSize * walSegment = 64 * 10 = 640Mb.

 

However, you may completely disregard estimations for WAL archive, as since latest 2.7 version there is a configuration property to limit WAL archive size in bytes, which is obviously more precise way to do so.

 

Ticket for your reference: https://issues.apache.org/jira/browse/IGNITE-6552

Docs link: https://apacheignite.readme.io/docs/write-ahead-log#section-wal-archive

 

Though there is the same link is on the under the hood page in WAL structure part, the ‘estimating disk space’ part has to be updated, thanks for your question!

 

Best regards,

Anton

From: [hidden email]
Sent: Monday, August 26, 2019 4:59 PM
To: [hidden email]
Subject: Ignite WAL and WAL archive size estimation

 

Hi All,

 

 

Following is mentioned about the Estimation:

what is est. maximum data volume to be writen on 1 checkpoint?  Is it the size of 1 wal segment or 1 checkpoint buffer size? 

Estimating disk space

WAL Work maximum used size: walSegmentSize * walSegments = 640Mb (default)

in case Default WAL mode - this size is used always,

in case other modes best case is 1 segment * walSegmentSize

WAL Work+WAL Archive max size may be estimated by

1.      average load or

2.      by maximum size.

1st way is applicable if checkpoints are triggered mostly by timer trigger. 
Wal size = 2*Average load(bytes/sec) * trigger interval (sec) * walHistSize (number of checkpoints)
Where 2 multiplier coming from physical & logical WAL Records.

2nd way: Checkpoint is triggered by segments max dirty pages percent. Use persisted data regions max sizes:
sum(Max configured DataRegionConfiguration.maxSize) * 75% - est. maximum data volume to be writen on 1 checkpoint.
Overall WAL size (before archiving) = 2* est. data volume * walHistSize = 1,5 * sum(DataRegionConfiguration.maxSize) * walHistSize 

Note applying WAL compressor may significiantly reduce archive size.

 

Thanks n Regards,

Chal