Force backup on different physical machine

classic Classic list List threaded Threaded
6 messages Options
Humphrey Humphrey
Reply | Threaded
Open this post in threaded view
|

Force backup on different physical machine

Let say I have 2 kubernetes nodes and I have 4 ignite server nodes.
This will result (if kubernetes have 2 pods running on each node) in the
following:

*kubernetes_node1:* ignite_node1, ignite_node2
*kubernetes_node2:* ignite_node3, ignite_node4

I specify that my cache backup = 1

Is there a way to configure that the backup data of the ignite_node1 goes on
the ignite_node3 or ignite_node4 and NOT on ignite_node2 (same physical
machine/kubernetes node)? Is there any configuration for this (I assume it's
something on runtime, because we don't know where kubernetes will schedule
the pod)?

Background:
If kubernetes_node1 goes down, then there won't be any data loss.

Humphrey



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: Force backup on different physical machine

Normally there are two ways to achieve this: excludeNeighbors and AffinityBackupFilter [1]

However, excludeNeighbors won't work across pods on the same k8s node,
since it relies on MAC addresses.

So your best bet is to use ClusterNodeAttributeAffinityBackupFilter:
* set ClusterNodeAttributeAffinityBackupFilter with env var K8S_NODE_NAME as described in [2]
* export K8S_NODE_NAME environment variable in Kubernetes as described in [3]:

      env:
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

This way backups won't end up on the same k8s node.

[1] https://apacheignite.readme.io/docs/affinity-collocation#crash-safe-affinity

On Tue, Jul 7, 2020 at 12:42 PM Humphrey <[hidden email]> wrote:
Let say I have 2 kubernetes nodes and I have 4 ignite server nodes.
This will result (if kubernetes have 2 pods running on each node) in the
following:

*kubernetes_node1:* ignite_node1, ignite_node2
*kubernetes_node2:* ignite_node3, ignite_node4

I specify that my cache backup = 1

Is there a way to configure that the backup data of the ignite_node1 goes on
the ignite_node3 or ignite_node4 and NOT on ignite_node2 (same physical
machine/kubernetes node)? Is there any configuration for this (I assume it's
something on runtime, because we don't know where kubernetes will schedule
the pod)?

Background:
If kubernetes_node1 goes down, then there won't be any data loss.

Humphrey



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Force backup on different physical machine

In reply to this post by Humphrey
You can configure the affinity function (RendezvousAffinityFunction). If you set the backup filter, you can customise which nodes are considered for use as backups:
cacheConfiguration.setBackups(1);
cacheConfiguration.setAffinity(new RendezvousAffinityFunction(1024, (n,p) -> {
return !p.hostNames().equals(n.hostNames());
}));
Having said that, you’re probably best configuring k8s not to put two Ignite server nodes on a single machine.

Regards,
Stephen

On 7 Jul 2020, at 10:42, Humphrey <[hidden email]> wrote:

Let say I have 2 kubernetes nodes and I have 4 ignite server nodes.
This will result (if kubernetes have 2 pods running on each node) in the
following:

*kubernetes_node1:* ignite_node1, ignite_node2
*kubernetes_node2:* ignite_node3, ignite_node4

I specify that my cache backup = 1

Is there a way to configure that the backup data of the ignite_node1 goes on
the ignite_node3 or ignite_node4 and NOT on ignite_node2 (same physical
machine/kubernetes node)? Is there any configuration for this (I assume it's
something on runtime, because we don't know where kubernetes will schedule
the pod)?

Background:
If kubernetes_node1 goes down, then there won't be any data loss.

Humphrey



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


narges saleh narges saleh
Reply | Threaded
Open this post in threaded view
|

Re: Force backup on different physical machine

Hi Stephen,
1) [Having said that, you’re probably best configuring k8s not to put two Ignite server nodes on a single machine.]
Do you mean by having k8s place the VMs in different availability zones?

Then how would you tell ignite to place the backups in a different zone?
I've read this is possible via affinityBackupFilter. Which brings up my next question.

2) Is affinityBackupFilter available with all public cloud platforms or just AWS?

thanks.

On Tue, Jul 7, 2020 at 5:20 AM Stephen Darlington <[hidden email]> wrote:
You can configure the affinity function (RendezvousAffinityFunction). If you set the backup filter, you can customise which nodes are considered for use as backups:
cacheConfiguration.setBackups(1);
cacheConfiguration.setAffinity(new RendezvousAffinityFunction(1024, (n,p) -> {
return !p.hostNames().equals(n.hostNames());
}));
Having said that, you’re probably best configuring k8s not to put two Ignite server nodes on a single machine.

Regards,
Stephen

On 7 Jul 2020, at 10:42, Humphrey <[hidden email]> wrote:

Let say I have 2 kubernetes nodes and I have 4 ignite server nodes.
This will result (if kubernetes have 2 pods running on each node) in the
following:

*kubernetes_node1:* ignite_node1, ignite_node2
*kubernetes_node2:* ignite_node3, ignite_node4

I specify that my cache backup = 1

Is there a way to configure that the backup data of the ignite_node1 goes on
the ignite_node3 or ignite_node4 and NOT on ignite_node2 (same physical
machine/kubernetes node)? Is there any configuration for this (I assume it's
something on runtime, because we don't know where kubernetes will schedule
the pod)?

Background:
If kubernetes_node1 goes down, then there won't be any data loss.

Humphrey



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Force backup on different physical machine

1) I think you’re saying you have two Ignite nodes on one physical machine. What I mean is, don’t do that. All else being equal, one server node per physical machine is the way to go

2) This is an Ignite feature and not limited to any particular cloud vendor. It does rely on your cloud vendor “publishing” the underlying machine (or availability zone) as an environment variable so Ignite can find it

On 21 Jul 2020, at 14:01, narges saleh <[hidden email]> wrote:

Hi Stephen,
1) [Having said that, you’re probably best configuring k8s not to put two Ignite server nodes on a single machine.]
Do you mean by having k8s place the VMs in different availability zones?

Then how would you tell ignite to place the backups in a different zone?
I've read this is possible via affinityBackupFilter. Which brings up my next question.

2) Is affinityBackupFilter available with all public cloud platforms or just AWS?

thanks.

On Tue, Jul 7, 2020 at 5:20 AM Stephen Darlington <[hidden email]> wrote:
You can configure the affinity function (RendezvousAffinityFunction). If you set the backup filter, you can customise which nodes are considered for use as backups:
cacheConfiguration.setBackups(1);
cacheConfiguration.setAffinity(new RendezvousAffinityFunction(1024, (n,p) -> {
return !p.hostNames().equals(n.hostNames());
}));
Having said that, you’re probably best configuring k8s not to put two Ignite server nodes on a single machine.

Regards,
Stephen

On 7 Jul 2020, at 10:42, Humphrey <[hidden email]> wrote:

Let say I have 2 kubernetes nodes and I have 4 ignite server nodes.
This will result (if kubernetes have 2 pods running on each node) in the
following:

*kubernetes_node1:* ignite_node1, ignite_node2
*kubernetes_node2:* ignite_node3, ignite_node4

I specify that my cache backup = 1

Is there a way to configure that the backup data of the ignite_node1 goes on
the ignite_node3 or ignite_node4 and NOT on ignite_node2 (same physical
machine/kubernetes node)? Is there any configuration for this (I assume it's
something on runtime, because we don't know where kubernetes will schedule
the pod)?

Background:
If kubernetes_node1 goes down, then there won't be any data loss.

Humphrey



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/




narges saleh narges saleh
Reply | Threaded
Open this post in threaded view
|

Re: Force backup on different physical machine

Thanks Stephan.
I am saying one ignite-node/pod/k8 node spread across availability zones

On Tue, Jul 21, 2020 at 9:51 AM Stephen Darlington <[hidden email]> wrote:
1) I think you’re saying you have two Ignite nodes on one physical machine. What I mean is, don’t do that. All else being equal, one server node per physical machine is the way to go

2) This is an Ignite feature and not limited to any particular cloud vendor. It does rely on your cloud vendor “publishing” the underlying machine (or availability zone) as an environment variable so Ignite can find it

On 21 Jul 2020, at 14:01, narges saleh <[hidden email]> wrote:

Hi Stephen,
1) [Having said that, you’re probably best configuring k8s not to put two Ignite server nodes on a single machine.]
Do you mean by having k8s place the VMs in different availability zones?

Then how would you tell ignite to place the backups in a different zone?
I've read this is possible via affinityBackupFilter. Which brings up my next question.

2) Is affinityBackupFilter available with all public cloud platforms or just AWS?

thanks.

On Tue, Jul 7, 2020 at 5:20 AM Stephen Darlington <[hidden email]> wrote:
You can configure the affinity function (RendezvousAffinityFunction). If you set the backup filter, you can customise which nodes are considered for use as backups:
cacheConfiguration.setBackups(1);
cacheConfiguration.setAffinity(new RendezvousAffinityFunction(1024, (n,p) -> {
return !p.hostNames().equals(n.hostNames());
}));
Having said that, you’re probably best configuring k8s not to put two Ignite server nodes on a single machine.

Regards,
Stephen

On 7 Jul 2020, at 10:42, Humphrey <[hidden email]> wrote:

Let say I have 2 kubernetes nodes and I have 4 ignite server nodes.
This will result (if kubernetes have 2 pods running on each node) in the
following:

*kubernetes_node1:* ignite_node1, ignite_node2
*kubernetes_node2:* ignite_node3, ignite_node4

I specify that my cache backup = 1

Is there a way to configure that the backup data of the ignite_node1 goes on
the ignite_node3 or ignite_node4 and NOT on ignite_node2 (same physical
machine/kubernetes node)? Is there any configuration for this (I assume it's
something on runtime, because we don't know where kubernetes will schedule
the pod)?

Background:
If kubernetes_node1 goes down, then there won't be any data loss.

Humphrey



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/