Capacity planning for production deployment on kubernetes

classic Classic list List threaded Threaded
5 messages Options
shivakumar shivakumar
Reply | Threaded
Open this post in threaded view
|

Capacity planning for production deployment on kubernetes

Hi all, 
we are testing field use case before deploying in the field and we want to know whether below resource limits are suitable in production. 
There are 3 nodes (3 pods on kubernetes) running. Each having below configuration 

                           DefaultDataRegion: 60GB 
                                                JVM: 32GB 
Resource allocated for each container: 64GB 

And ignite documents says (JVM+ All DataRegion) should not exceed 70% of total RAM allocated to each node(container). 
but we started testing with the above configuration and up to 9 days ignite cluster was running successfully and there was some data ingestion but suddenly pods crashed and they were unable to recover from the crash. 
does the above resource configuration not good for node recovery??  
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Capacity planning for production deployment on kubernetes

Shivakumar,

Such allocation doesn’t allow full memory utilization, so it’s possible, that nodes will crash because of out of memory errors.
So, it’s better to follow the given recommendation.

If you want us to investigate reasons of the failures, please provide logs and configuration of the failed nodes.

Denis
On 21 Aug 2019, 16:17 +0300, Shiva Kumar <[hidden email]>, wrote:
Hi all, 
we are testing field use case before deploying in the field and we want to know whether below resource limits are suitable in production. 
There are 3 nodes (3 pods on kubernetes) running. Each having below configuration 

                           DefaultDataRegion: 60GB 
                                                JVM: 32GB 
Resource allocated for each container: 64GB 

And ignite documents says (JVM+ All DataRegion) should not exceed 70% of total RAM allocated to each node(container). 
but we started testing with the above configuration and up to 9 days ignite cluster was running successfully and there was some data ingestion but suddenly pods crashed and they were unable to recover from the crash. 
does the above resource configuration not good for node recovery??  
shivakumar shivakumar
Reply | Threaded
Open this post in threaded view
|

Re: Capacity planning for production deployment on kubernetes

Hi Denis, 

Thanks for your response, 
yes in our test also we have seen OOM errors and pod crash. 
so we will follow the recommendation for RAM requirements and also I was checking to ignite documentation on disk space required for WAL + WAL archive. 
here in this link  https://apacheignite.readme.io/docs/write-ahead-log#section-wal-archive

it says: archive size is defined as 4 times the size of the checkpointing buffer and checkpointing buffer is a function of the data region (https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size

but in this link https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-SubfoldersGeneration

under Estimating disk space section it explains something to estimate disk space required for WAL but it is not clear, can you please help me the correct recommendation for calculating the disk space required for WAL+WAL archive. 

In one of my testing, I configured 4GB for data region and 10GB for WAL+WAL archive but our pods crashing as disk mounted for WAL+WAL archive runs out of space. 

[ignite@ignite-cluster-ignite-node-2 ignite]$ df -h
Filesystem      Size  Used Avail Use% Mounted on 
overlay         158G   39G  112G  26% / 
tmpfs            63G     0   63G   0% /dev 
tmpfs            63G     0   63G   0% /sys/fs/cgroup 
/dev/vda1       158G   39G  112G  26% /etc/hosts 
shm              64M     0   64M   0% /dev/shm 
/dev/vdq        9.8G  9.7G   44M 100% /opt/ignite/wal
/dev/vdr         50G  1.4G   48G   3% /opt/ignite/persistence 
tmpfs            63G   12K   63G   1% /run/secrets/kubernetes.io/serviceaccount 
tmpfs            63G     0   63G   0% /proc/acpi 
tmpfs            63G     0   63G   0% /proc/scsi 
tmpfs            63G     0   63G   0% /sys/firmware 


and this is the error message in ignite node: 

"ERROR","JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteCheckedException: Failed to archive WAL segment [srcFile=/opt/ignite/wal/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0000000000000006.wal, dstFile=/opt/ignite/wal/archive/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0000000000000236.wal.tmp]]]"   

On Thu, Aug 22, 2019 at 8:04 PM Denis Mekhanikov <[hidden email]> wrote:
Shivakumar,

Such allocation doesn’t allow full memory utilization, so it’s possible, that nodes will crash because of out of memory errors.
So, it’s better to follow the given recommendation.

If you want us to investigate reasons of the failures, please provide logs and configuration of the failed nodes.

Denis
On 21 Aug 2019, 16:17 +0300, Shiva Kumar <[hidden email]>, wrote:
Hi all, 
we are testing field use case before deploying in the field and we want to know whether below resource limits are suitable in production. 
There are 3 nodes (3 pods on kubernetes) running. Each having below configuration 

                           DefaultDataRegion: 60GB 
                                                JVM: 32GB 
Resource allocated for each container: 64GB 

And ignite documents says (JVM+ All DataRegion) should not exceed 70% of total RAM allocated to each node(container). 
but we started testing with the above configuration and up to 9 days ignite cluster was running successfully and there was some data ingestion but suddenly pods crashed and they were unable to recover from the crash. 
does the above resource configuration not good for node recovery??  
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Capacity planning for production deployment on kubernetes

Please share the whole log file. It might be the case that something goes wrong with volumes you attached to Ignite pods.

-
Denis


On Thu, Aug 22, 2019 at 8:07 AM Shiva Kumar <[hidden email]> wrote:
Hi Denis, 

Thanks for your response, 
yes in our test also we have seen OOM errors and pod crash. 
so we will follow the recommendation for RAM requirements and also I was checking to ignite documentation on disk space required for WAL + WAL archive. 
here in this link  https://apacheignite.readme.io/docs/write-ahead-log#section-wal-archive

it says: archive size is defined as 4 times the size of the checkpointing buffer and checkpointing buffer is a function of the data region (https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size

but in this link https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-SubfoldersGeneration

under Estimating disk space section it explains something to estimate disk space required for WAL but it is not clear, can you please help me the correct recommendation for calculating the disk space required for WAL+WAL archive. 

In one of my testing, I configured 4GB for data region and 10GB for WAL+WAL archive but our pods crashing as disk mounted for WAL+WAL archive runs out of space. 

[ignite@ignite-cluster-ignite-node-2 ignite]$ df -h
Filesystem      Size  Used Avail Use% Mounted on 
overlay         158G   39G  112G  26% / 
tmpfs            63G     0   63G   0% /dev 
tmpfs            63G     0   63G   0% /sys/fs/cgroup 
/dev/vda1       158G   39G  112G  26% /etc/hosts 
shm              64M     0   64M   0% /dev/shm 
/dev/vdq        9.8G  9.7G   44M 100% /opt/ignite/wal
/dev/vdr         50G  1.4G   48G   3% /opt/ignite/persistence 
tmpfs            63G   12K   63G   1% /run/secrets/kubernetes.io/serviceaccount 
tmpfs            63G     0   63G   0% /proc/acpi 
tmpfs            63G     0   63G   0% /proc/scsi 
tmpfs            63G     0   63G   0% /sys/firmware 


and this is the error message in ignite node: 

"ERROR","JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteCheckedException: Failed to archive WAL segment [srcFile=/opt/ignite/wal/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0000000000000006.wal, dstFile=/opt/ignite/wal/archive/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0000000000000236.wal.tmp]]]"   

On Thu, Aug 22, 2019 at 8:04 PM Denis Mekhanikov <[hidden email]> wrote:
Shivakumar,

Such allocation doesn’t allow full memory utilization, so it’s possible, that nodes will crash because of out of memory errors.
So, it’s better to follow the given recommendation.

If you want us to investigate reasons of the failures, please provide logs and configuration of the failed nodes.

Denis
On 21 Aug 2019, 16:17 +0300, Shiva Kumar <[hidden email]>, wrote:
Hi all, 
we are testing field use case before deploying in the field and we want to know whether below resource limits are suitable in production. 
There are 3 nodes (3 pods on kubernetes) running. Each having below configuration 

                           DefaultDataRegion: 60GB 
                                                JVM: 32GB 
Resource allocated for each container: 64GB 

And ignite documents says (JVM+ All DataRegion) should not exceed 70% of total RAM allocated to each node(container). 
but we started testing with the above configuration and up to 9 days ignite cluster was running successfully and there was some data ingestion but suddenly pods crashed and they were unable to recover from the crash. 
does the above resource configuration not good for node recovery??  
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Capacity planning for production deployment on kubernetes

Shiva,

What version of Ignite do you use? 
Before version 2.7 Ignite used a different mechanism to limit the size of the WAL history. It used DataStorageConfiguration#walHistorySize property, which is currently deprecated. This is what’s explained on the internal documentation page.

In Ignite versions starting from 2.7 DataStorageConfiguration#maxWalArchiveSize property is used. It specifies the maximum size of WAL history in bytes.
As you said, it’s 4 times the checkpoint buffer size. Since you didn’t change the size of the checkpoint buffer, by default it’s 1 GB in your case (rules for its calculation can be found here: https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size)
So, maximum WAL history size is 4 GB.
Plus you need to add WAL size itself which is (64 MB per segment) x (10 segments) = 640 megabytes by default.
If you use Ignite 2.7 or newer, then 10 GB should be enough for WAL with archive. Failures should be investigated in this case.

But Ignite 2.6 and older use a different approach, so there is no strict limitation on the WAL archive size.

Denis
On 22 Aug 2019, 22:35 +0300, Denis Magda <[hidden email]>, wrote:
Please share the whole log file. It might be the case that something goes wrong with volumes you attached to Ignite pods.

-
Denis


On Thu, Aug 22, 2019 at 8:07 AM Shiva Kumar <[hidden email]> wrote:
Hi Denis, 

Thanks for your response, 
yes in our test also we have seen OOM errors and pod crash. 
so we will follow the recommendation for RAM requirements and also I was checking to ignite documentation on disk space required for WAL + WAL archive. 
here in this link  https://apacheignite.readme.io/docs/write-ahead-log#section-wal-archive

it says: archive size is defined as 4 times the size of the checkpointing buffer and checkpointing buffer is a function of the data region (https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size

but in this link https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-SubfoldersGeneration

under Estimating disk space section it explains something to estimate disk space required for WAL but it is not clear, can you please help me the correct recommendation for calculating the disk space required for WAL+WAL archive. 

In one of my testing, I configured 4GB for data region and 10GB for WAL+WAL archive but our pods crashing as disk mounted for WAL+WAL archive runs out of space. 

[ignite@ignite-cluster-ignite-node-2 ignite]$ df -h
Filesystem      Size  Used Avail Use% Mounted on 
overlay         158G   39G  112G  26% / 
tmpfs            63G     0   63G   0% /dev 
tmpfs            63G     0   63G   0% /sys/fs/cgroup 
/dev/vda1       158G   39G  112G  26% /etc/hosts 
shm              64M     0   64M   0% /dev/shm 
/dev/vdq        9.8G  9.7G   44M 100% /opt/ignite/wal
/dev/vdr         50G  1.4G   48G   3% /opt/ignite/persistence 
tmpfs            63G   12K   63G   1% /run/secrets/kubernetes.io/serviceaccount 
tmpfs            63G     0   63G   0% /proc/acpi 
tmpfs            63G     0   63G   0% /proc/scsi 
tmpfs            63G     0   63G   0% /sys/firmware 


and this is the error message in ignite node: 

"ERROR","JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteCheckedException: Failed to archive WAL segment [srcFile=/opt/ignite/wal/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0000000000000006.wal, dstFile=/opt/ignite/wal/archive/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0000000000000236.wal.tmp]]]"   

On Thu, Aug 22, 2019 at 8:04 PM Denis Mekhanikov <[hidden email]> wrote:
Shivakumar,

Such allocation doesn’t allow full memory utilization, so it’s possible, that nodes will crash because of out of memory errors.
So, it’s better to follow the given recommendation.

If you want us to investigate reasons of the failures, please provide logs and configuration of the failed nodes.

Denis
On 21 Aug 2019, 16:17 +0300, Shiva Kumar <[hidden email]>, wrote:
Hi all, 
we are testing field use case before deploying in the field and we want to know whether below resource limits are suitable in production. 
There are 3 nodes (3 pods on kubernetes) running. Each having below configuration 

                           DefaultDataRegion: 60GB 
                                                JVM: 32GB 
Resource allocated for each container: 64GB 

And ignite documents says (JVM+ All DataRegion) should not exceed 70% of total RAM allocated to each node(container). 
but we started testing with the above configuration and up to 9 days ignite cluster was running successfully and there was some data ingestion but suddenly pods crashed and they were unable to recover from the crash. 
does the above resource configuration not good for node recovery??