Cluster health

classic Classic list List threaded Threaded
3 messages Options
apohrebniak apohrebniak
Reply | Threaded
Open this post in threaded view
|

Cluster health

Hi all.
I have Ignite in Kubernetes deployed as a standalone application. There are
a couple of caches all with *cacheMode=PARTITIONED* and *backups=1*.
During the cluster updates, K8s updates the pods one by one. There might a
case when the next pod/node had been shut down before all the required
partitions were replicated across the cluster. This is a potential data
loss.
Is there any way to know that all the required partitions were copied to the
newly created node and it's safe to shut down the next one?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ivan Rakov Ivan Rakov
Reply | Threaded
Open this post in threaded view
|

Re: Cluster health

Hello!

This information can be retrieved from cache metrics.
If for every cache
CacheGroupMetricsMXBean#getClusterMovingPartitionsCount returns zero,
rebalancing is not in progress.
I've created a topic on dev list about introducing more simple way to
get the answer.

Best Regards,
Ivan Rakov

On 04.10.2019 10:33, apohrebniak wrote:

> Hi all.
> I have Ignite in Kubernetes deployed as a standalone application. There are
> a couple of caches all with *cacheMode=PARTITIONED* and *backups=1*.
> During the cluster updates, K8s updates the pods one by one. There might a
> case when the next pod/node had been shut down before all the required
> partitions were replicated across the cluster. This is a potential data
> loss.
> Is there any way to know that all the required partitions were copied to the
> newly created node and it's safe to shut down the next one?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
apohrebniak apohrebniak
Reply | Threaded
Open this post in threaded view
|

Re: Cluster health