Rebalancing only to backups

classic Classic list List threaded Threaded
4 messages Options
gupabhi gupabhi
Reply | Threaded
Open this post in threaded view
|

Rebalancing only to backups

(pardon me if this mail is by chance a duplicate - it got bounced back when I sent it earlier from nabble)

Hello,
I have 6 node grid and I've configured it with 1 backup. I want to have
partition rebalancing but only in the following way.
If one of the six nodes goes down, then some primary and backup partitions
go down with it but there is no data loss since the backup for those are
present on one of the other five nodes. So there is only single copy for
these paritions

i. At this point I do not want that the 5 nodes rebalance all the paritions
amongst themselves such that each one has 1 primary and 1 backup
ii. When this 6th node comes back up, I want the partitions in the other
nodes which are living with only a single copy, so hydrate this fresh 6th
one with copies of the partitions that has only one copy before.

Why i? Because I don't want a situation of cascading OOMs
Why ii? Obvious reason so as to have the 2nd copy for all partitions.

Is this possible? If not what's the best way to come close to this?

Thanks,
Abhi
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Rebalancing only to backups

Hi Abhishek,

That's how it works now if you have enabled Persistence. Actually, that's the main reason why BaselineTopology was introduced - we don't want to move a lot of data between nodes if we know that node will return soon after a failure: https://apacheignite.readme.io/docs/baseline-topology
If you want to force rebalance, you can manually change the BaselineTopology.

As far as I know, BaselineTopology concept was also introduced for in-memory caches and will be released as a part of Apache Ignite 2.8. Also, there will be some configurable timeout, after which baseline topology will be changed automatically if you want it.

Best Regards,
Evgenii

чт, 15 авг. 2019 г. в 17:16, Abhishek Gupta (BLOOMBERG/ 731 LEX) <[hidden email]>:
(pardon me if this mail is by chance a duplicate - it got bounced back when I sent it earlier from nabble)

Hello,
I have 6 node grid and I've configured it with 1 backup. I want to have
partition rebalancing but only in the following way.
If one of the six nodes goes down, then some primary and backup partitions
go down with it but there is no data loss since the backup for those are
present on one of the other five nodes. So there is only single copy for
these paritions

i. At this point I do not want that the 5 nodes rebalance all the paritions
amongst themselves such that each one has 1 primary and 1 backup
ii. When this 6th node comes back up, I want the partitions in the other
nodes which are living with only a single copy, so hydrate this fresh 6th
one with copies of the partitions that has only one copy before.

Why i? Because I don't want a situation of cascading OOMs
Why ii? Obvious reason so as to have the 2nd copy for all partitions.

Is this possible? If not what's the best way to come close to this?

Thanks,
Abhi
gupabhi gupabhi
Reply | Threaded
Open this post in threaded view
|

Re: Rebalancing only to backups

In reply to this post by gupabhi
Exactly - what is available for persistence, I was wondering if it is available for in-mem only. So for now I'll just need to configure rebalance mode to non-NONE and live with point 1.

Thanks evgenii!


From: [hidden email] At: 08/15/19 11:37:16
To: [hidden email], [hidden email]
Subject: Re: Rebalancing only to backups

Hi Abhishek,

That's how it works now if you have enabled Persistence. Actually, that's the main reason why BaselineTopology was introduced - we don't want to move a lot of data between nodes if we know that node will return soon after a failure: https://apacheignite.readme.io/docs/baseline-topology
If you want to force rebalance, you can manually change the BaselineTopology.

As far as I know, BaselineTopology concept was also introduced for in-memory caches and will be released as a part of Apache Ignite 2.8. Also, there will be some configurable timeout, after which baseline topology will be changed automatically if you want it.

Best Regards,
Evgenii

чт, 15 авг. 2019 г. в 17:16, Abhishek Gupta (BLOOMBERG/ 731 LEX) <[hidden email]>:
(pardon me if this mail is by chance a duplicate - it got bounced back when I sent it earlier from nabble)

Hello,
I have 6 node grid and I've configured it with 1 backup. I want to have
partition rebalancing but only in the following way.
If one of the six nodes goes down, then some primary and backup partitions
go down with it but there is no data loss since the backup for those are
present on one of the other five nodes. So there is only single copy for
these paritions

i. At this point I do not want that the 5 nodes rebalance all the paritions
amongst themselves such that each one has 1 primary and 1 backup
ii. When this 6th node comes back up, I want the partitions in the other
nodes which are living with only a single copy, so hydrate this fresh 6th
one with copies of the partitions that has only one copy before.

Why i? Because I don't want a situation of cascading OOMs
Why ii? Obvious reason so as to have the 2nd copy for all partitions.

Is this possible? If not what's the best way to come close to this?

Thanks,
Abhi

mmuzaf mmuzaf
Reply | Threaded
Open this post in threaded view
|

Re: Rebalancing only to backups

Hello,

I would not recommend using NONE rebalance mode since it is not stable
enough (this is true, that cache for this mode will not be rebalanced,
but you always should keep in mind then your 6th node return to the
cluster you should call manual rebalancing -- method rebalance() on
cache API).

Currently, it is not possible to clearly achieve your case without
Baseline topology (native persistence must be enabled, can you?), but
you can try these things:
- configure cache eviction policy [1] to avoid OOM
- use rebalanceDelay property [2] -- timeout in milliseconds when the
rebalance procedure should start for a particular cache


[1] https://apacheignite.readme.io/docs/evictions
[2] https://apacheignite.readme.io/v1.1/docs/rebalancing

On Thu, 15 Aug 2019 at 19:06, Abhishek Gupta (BLOOMBERG/ 731 LEX)
<[hidden email]> wrote:

>
> Exactly - what is available for persistence, I was wondering if it is available for in-mem only. So for now I'll just need to configure rebalance mode to non-NONE and live with point 1.
>
> Thanks evgenii!
>
>
> From: [hidden email] At: 08/15/19 11:37:16
> To: Abhishek Gupta (BLOOMBERG/ 731 LEX ) , [hidden email]
> Subject: Re: Rebalancing only to backups
>
> Hi Abhishek,
>
> That's how it works now if you have enabled Persistence. Actually, that's the main reason why BaselineTopology was introduced - we don't want to move a lot of data between nodes if we know that node will return soon after a failure: https://apacheignite.readme.io/docs/baseline-topology
> If you want to force rebalance, you can manually change the BaselineTopology.
>
> As far as I know, BaselineTopology concept was also introduced for in-memory caches and will be released as a part of Apache Ignite 2.8. Also, there will be some configurable timeout, after which baseline topology will be changed automatically if you want it.
>
> Best Regards,
> Evgenii
>
> чт, 15 авг. 2019 г. в 17:16, Abhishek Gupta (BLOOMBERG/ 731 LEX) <[hidden email]>:
>>
>> (pardon me if this mail is by chance a duplicate - it got bounced back when I sent it earlier from nabble)
>>
>> Hello,
>> I have 6 node grid and I've configured it with 1 backup. I want to have
>> partition rebalancing but only in the following way.
>> If one of the six nodes goes down, then some primary and backup partitions
>> go down with it but there is no data loss since the backup for those are
>> present on one of the other five nodes. So there is only single copy for
>> these paritions
>>
>> i. At this point I do not want that the 5 nodes rebalance all the paritions
>> amongst themselves such that each one has 1 primary and 1 backup
>> ii. When this 6th node comes back up, I want the partitions in the other
>> nodes which are living with only a single copy, so hydrate this fresh 6th
>> one with copies of the partitions that has only one copy before.
>>
>> Why i? Because I don't want a situation of cascading OOMs
>> Why ii? Obvious reason so as to have the 2nd copy for all partitions.
>>
>> Is this possible? If not what's the best way to come close to this?
>>
>> Thanks,
>> Abhi
>
>