Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

classic Classic list List threaded Threaded
16 messages Options
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 

mcherkasov mcherkasov
Reply | Threaded
Open this post in threaded view
|

Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

Hi Raymond,

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,
for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process
of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest
data we need to start all nodes first and only then activate cluster.

Thanks,
Mikhail.

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 




--
Thanks,
Mikhail.
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

RE: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.

Mikhail Mikhail
Reply | Threaded
Open this post in threaded view
|

Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.


Rohan Shetty Rohan Shetty
Reply | Threaded
Open this post in threaded view
|

Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

On Wed, Aug 2, 2017 at 3:18 PM, Michael Cherkasov <[hidden email]> wrote:
when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.



Mikhail Mikhail
Reply | Threaded
Open this post in threaded view
|

Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

>Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

No, you need to deactivate cluster when you going to shutdown the whole cluster. And when you return cluster back to  online, you need to wait when all nodes are in place and then activate it.


2017-08-02 16:22 GMT+03:00 Rohan Shetty <[hidden email]>:
Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

On Wed, Aug 2, 2017 at 3:18 PM, Michael Cherkasov <[hidden email]> wrote:
when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.




Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

RE: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

Michael,

 

Is there a reference implementation in Ignite 2.1 for an agent that listens to topology changes to decide when to set active to true?

 

Thanks,

Raymond.

 

From: Michael Cherkasov [mailto:[hidden email]]
Sent: Thursday, August 3, 2017 1:25 AM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

>Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

No, you need to deactivate cluster when you going to shutdown the whole cluster. And when you return cluster back to  online, you need to wait when all nodes are in place and then activate it.

 

 

2017-08-02 16:22 GMT+03:00 Rohan Shetty <[hidden email]>:

Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

On Wed, Aug 2, 2017 at 3:18 PM, Michael Cherkasov <[hidden email]> wrote:

when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

 

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.

 

 

 

Mikhail Mikhail
Reply | Threaded
Open this post in threaded view
|

Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

Hi Raymond, 

Unfortunately right now there's no auto-activation, restarting cluster is like rare event that should be controlled 
manually. However you can listen for EVT_NODE_JOINED event, when all nodes in place you can activate a cluster.

And you only need this if you have ignite persistence turn on and you have some data on the disk.

Thanks,
Mikhail.

2017-08-03 6:58 GMT+03:00 Raymond Wilson <[hidden email]>:

Michael,

 

Is there a reference implementation in Ignite 2.1 for an agent that listens to topology changes to decide when to set active to true?

 

Thanks,

Raymond.

 

From: Michael Cherkasov [mailto:[hidden email]]
Sent: Thursday, August 3, 2017 1:25 AM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

>Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

No, you need to deactivate cluster when you going to shutdown the whole cluster. And when you return cluster back to  online, you need to wait when all nodes are in place and then activate it.

 

 

2017-08-02 16:22 GMT+03:00 Rohan Shetty <[hidden email]>:

Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

On Wed, Aug 2, 2017 at 3:18 PM, Michael Cherkasov <[hidden email]> wrote:

when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

 

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.

 

 

 


Mikhail Mikhail
Reply | Threaded
Open this post in threaded view
|

Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

In reply to this post by Raymond Wilson
Hi again, 

Checkout the email on dev list: "Cluster auto activation design proposal" 
As you can see this feature is targeted to 2.2.

Thanks,
Mikhail.

2017-08-03 6:58 GMT+03:00 Raymond Wilson <[hidden email]>:

Michael,

 

Is there a reference implementation in Ignite 2.1 for an agent that listens to topology changes to decide when to set active to true?

 

Thanks,

Raymond.

 

From: Michael Cherkasov [mailto:[hidden email]]
Sent: Thursday, August 3, 2017 1:25 AM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

>Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

No, you need to deactivate cluster when you going to shutdown the whole cluster. And when you return cluster back to  online, you need to wait when all nodes are in place and then activate it.

 

 

2017-08-02 16:22 GMT+03:00 Rohan Shetty <[hidden email]>:

Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

On Wed, Aug 2, 2017 at 3:18 PM, Michael Cherkasov <[hidden email]> wrote:

when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

 

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.

 

 

 


Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

RE: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

That Jira ticket is a little light on detail! J

 

From: Michael Cherkasov [mailto:[hidden email]]
Sent: Thursday, August 3, 2017 9:37 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi again, 

 

Checkout the email on dev list: "Cluster auto activation design proposal" 

As you can see this feature is targeted to 2.2.

 

Thanks,

Mikhail.

 

2017-08-03 6:58 GMT+03:00 Raymond Wilson <[hidden email]>:

Michael,

 

Is there a reference implementation in Ignite 2.1 for an agent that listens to topology changes to decide when to set active to true?

 

Thanks,

Raymond.

 

From: Michael Cherkasov [mailto:[hidden email]]
Sent: Thursday, August 3, 2017 1:25 AM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

>Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

No, you need to deactivate cluster when you going to shutdown the whole cluster. And when you return cluster back to  online, you need to wait when all nodes are in place and then activate it.

 

 

2017-08-02 16:22 GMT+03:00 Rohan Shetty <[hidden email]>:

Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

On Wed, Aug 2, 2017 at 3:18 PM, Michael Cherkasov <[hidden email]> wrote:

when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

 

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.

 

 

 

 

Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

RE: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

In reply to this post by Mikhail

Hi Michael,

 

Some more questions:

 

1.       Can I set the grid active once all primaries are active, but before all backups are active? Or do I need to have the entire cluster running before setting active to true?

2.       Is there an assumption that the cluster size is static and known at the time of startup/restart? If so, how are dynamic clusters that grow and shrink according to demand catered for?

3.       Is it usually the responsibility of one of the cluster nodes to set active to true, or is it the responsibility of an external client server node on the grid to detect the point at which the cluster running correctly and set active to true? If a cluster type node is used, does this result in a swarm of activation messages as each node inspects the latest topology changed event and sets active to true?

 

Thanks,

Raymond.

 

 

From: Michael Cherkasov [mailto:[hidden email]]
Sent: Thursday, August 3, 2017 9:27 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond, 

 

Unfortunately right now there's no auto-activation, restarting cluster is like rare event that should be controlled 

manually. However you can listen for EVT_NODE_JOINED event, when all nodes in place you can activate a cluster.

 

And you only need this if you have ignite persistence turn on and you have some data on the disk.

 

Thanks,

Mikhail.

 

2017-08-03 6:58 GMT+03:00 Raymond Wilson <[hidden email]>:

Michael,

 

Is there a reference implementation in Ignite 2.1 for an agent that listens to topology changes to decide when to set active to true?

 

Thanks,

Raymond.

 

From: Michael Cherkasov [mailto:[hidden email]]
Sent: Thursday, August 3, 2017 1:25 AM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

>Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

No, you need to deactivate cluster when you going to shutdown the whole cluster. And when you return cluster back to  online, you need to wait when all nodes are in place and then activate it.

 

 

2017-08-02 16:22 GMT+03:00 Rohan Shetty <[hidden email]>:

Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

On Wed, Aug 2, 2017 at 3:18 PM, Michael Cherkasov <[hidden email]> wrote:

when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

 

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.

 

 

 

 

Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

RE: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

In reply to this post by Mikhail

Sorry, adding a question:

 

4. This raises a more general coordinate question: You can’t create or locate a cache in the grid until the grid is active (when using Ignite persistence). Which means you need to stall any logic in the server nodes that wants to do this until the grid is marked as active. How does such logic detect when the grid becomes active? Should it spin and check periodically, or is there an activation event that can be leveraged?

 

The Ignite persistence example at https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/persistentstore does not address these aspects.

 

Thanks,
Raymond.

 

From: Raymond Wilson [mailto:[hidden email]]
Sent: Friday, August 4, 2017 1:56 PM
To: '[hidden email]'
Subject: RE: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Michael,

 

Some more questions:

 

1.       Can I set the grid active once all primaries are active, but before all backups are active? Or do I need to have the entire cluster running before setting active to true?

2.       Is there an assumption that the cluster size is static and known at the time of startup/restart? If so, how are dynamic clusters that grow and shrink according to demand catered for?

3.       Is it usually the responsibility of one of the cluster nodes to set active to true, or is it the responsibility of an external client server node on the grid to detect the point at which the cluster running correctly and set active to true? If a cluster type node is used, does this result in a swarm of activation messages as each node inspects the latest topology changed event and sets active to true?

 

Thanks,

Raymond.

 

 

From: Michael Cherkasov [[hidden email]]
Sent: Thursday, August 3, 2017 9:27 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond, 

 

Unfortunately right now there's no auto-activation, restarting cluster is like rare event that should be controlled 

manually. However you can listen for EVT_NODE_JOINED event, when all nodes in place you can activate a cluster.

 

And you only need this if you have ignite persistence turn on and you have some data on the disk.

 

Thanks,

Mikhail.

 

2017-08-03 6:58 GMT+03:00 Raymond Wilson <[hidden email]>:

Michael,

 

Is there a reference implementation in Ignite 2.1 for an agent that listens to topology changes to decide when to set active to true?

 

Thanks,

Raymond.

 

From: Michael Cherkasov [mailto:[hidden email]]
Sent: Thursday, August 3, 2017 1:25 AM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

>Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

No, you need to deactivate cluster when you going to shutdown the whole cluster. And when you return cluster back to  online, you need to wait when all nodes are in place and then activate it.

 

 

2017-08-02 16:22 GMT+03:00 Rohan Shetty <[hidden email]>:

Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

On Wed, Aug 2, 2017 at 3:18 PM, Michael Cherkasov <[hidden email]> wrote:

when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

 

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.

 

 

 

 

mcherkasov mcherkasov
Reply | Threaded
Open this post in threaded view
|

Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

Hi Raymond,

1.       Can I set the grid active once all primaries are active, but before all backups are active? Or do I need to have the entire cluster running before setting active to true?

A cluster will work, but it will try to restore backups which are missed and this will cause rebalancing, so it's better to wait for all nodes is restored.

2.       Is there an assumption that the cluster size is static and known at the time of startup/restart? If so, how are dynamic clusters that grow and shrink according to demand catered for?

well, you can add new nodes when a cluster is active or inactive this doesn't really matter, in case if the cluster isn't active, only after activation new nodes will be used for data storing. There's no convenient way to shrink cluster, you can kill nodes wait when rebalancing is done and kill next one and so on.

3.       Is it usually the responsibility of one of the cluster nodes to set active to true, or is it the responsibility of an external client server node on the grid to detect the point at which the cluster running correctly and set active to true? If a cluster type node is used, does this result in a swarm of activation messages as each node inspects the latest topology changed event and sets active to true?

You can use both approaches, but in the case of implementing activating login on server nodes you should surround Ignite.active(true) with try catch, the exception can be thrown due to concurrent activation. Also, you can check whether the node is a coordinator and run activation on it only.

4. This raises a more general coordinate question: You can’t create or locate a cache in the grid until the grid is active (when using Ignite persistence). Which means you need to stall any logic in the server nodes that wants to do this until the grid is marked as active. How does such logic detect when the grid becomes active? Should it spin and check periodically, or is there an activation event that can be leveraged?


Unfortunately, there's no event for this now, so you should spin around Ignite.active()

Thanks,
Mikhail.


On Fri, Aug 4, 2017 at 5:01 AM, Raymond Wilson <[hidden email]> wrote:

Sorry, adding a question:

 

4. This raises a more general coordinate question: You can’t create or locate a cache in the grid until the grid is active (when using Ignite persistence). Which means you need to stall any logic in the server nodes that wants to do this until the grid is marked as active. How does such logic detect when the grid becomes active? Should it spin and check periodically, or is there an activation event that can be leveraged?

 

The Ignite persistence example at https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/persistentstore does not address these aspects.

 

Thanks,
Raymond.

 

From: Raymond Wilson [mailto:[hidden email]]
Sent: Friday, August 4, 2017 1:56 PM
To: '[hidden email]'
Subject: RE: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Michael,

 

Some more questions:

 

1.       Can I set the grid active once all primaries are active, but before all backups are active? Or do I need to have the entire cluster running before setting active to true?

2.       Is there an assumption that the cluster size is static and known at the time of startup/restart? If so, how are dynamic clusters that grow and shrink according to demand catered for?

3.       Is it usually the responsibility of one of the cluster nodes to set active to true, or is it the responsibility of an external client server node on the grid to detect the point at which the cluster running correctly and set active to true? If a cluster type node is used, does this result in a swarm of activation messages as each node inspects the latest topology changed event and sets active to true?

 

Thanks,

Raymond.

 

 

From: Michael Cherkasov [[hidden email]]
Sent: Thursday, August 3, 2017 9:27 PM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond, 

 

Unfortunately right now there's no auto-activation, restarting cluster is like rare event that should be controlled 

manually. However you can listen for EVT_NODE_JOINED event, when all nodes in place you can activate a cluster.

 

And you only need this if you have ignite persistence turn on and you have some data on the disk.

 

Thanks,

Mikhail.

 

2017-08-03 6:58 GMT+03:00 Raymond Wilson <[hidden email]>:

Michael,

 

Is there a reference implementation in Ignite 2.1 for an agent that listens to topology changes to decide when to set active to true?

 

Thanks,

Raymond.

 

From: Michael Cherkasov [mailto:[hidden email]]
Sent: Thursday, August 3, 2017 1:25 AM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

>Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

No, you need to deactivate cluster when you going to shutdown the whole cluster. And when you return cluster back to  online, you need to wait when all nodes are in place and then activate it.

 

 

2017-08-02 16:22 GMT+03:00 Rohan Shetty <[hidden email]>:

Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

On Wed, Aug 2, 2017 at 3:18 PM, Michael Cherkasov <[hidden email]> wrote:

when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

 

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.

 

 

 

 




--
Thanks,
Mikhail.
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

RE: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

Hi Michael,

 

I wrote a simple application to act as an activator for a grid. As this is just a POC, it’s simply a form with a button on it that does this:

 

                // Get an ignite reference to the named grid

                IIgnite ignite = Ignition.TryGetIgnite(“MyGrid”);

 

                // If the grid exists and is not already active then set it to active

                if (ignite != null && !ignite.IsActive())

                  ignite.SetActive(true);

 

In my Ignite server I have this code:

 

            IgniteConfiguration cfg = new IgniteConfiguration();

 

            ConfigureGrid(cfg); // Set it up as a grid with persistence enabled

 

            cfg.GridName = “MyGrid”;

            cfg.IgniteInstanceName = “MyGrid”

 

            IIgnite ignite = Ignition.Start(cfg);

 

            // Wait until the grid is active

            bool isActive = false;

            do

            {

                try

                {

                    isActive = ignite.IsActive();

 

                    if (!isActive)

                    {

                        Thread.Sleep(1000);

                    }

                }

                catch

                {

                    // Ignore it and spin

                    Thread.Sleep(1000);

                }

            } while (!isActive);

 

Note the activator and the server are completely different applications/processes, but running on the same local machine.

 

While the server node that creates the grid has started up I hit the button on the activator app to set it to active. Unfortunately the activator app is unable to get a reference to the “MyGrid” grid, TryGetIgnite always returns nil.

 

Does TryGetIgnite return an Ignite grid reference if Active is not set to true in this instance?

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Saturday, August 5, 2017 2:42 AM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

1.       Can I set the grid active once all primaries are active, but before all backups are active? Or do I need to have the entire cluster running before setting active to true?

 

A cluster will work, but it will try to restore backups which are missed and this will cause rebalancing, so it's better to wait for all nodes is restored.

 

2.       Is there an assumption that the cluster size is static and known at the time of startup/restart? If so, how are dynamic clusters that grow and shrink according to demand catered for?

 

well, you can add new nodes when a cluster is active or inactive this doesn't really matter, in case if the cluster isn't active, only after activation new nodes will be used for data storing. There's no convenient way to shrink cluster, you can kill nodes wait when rebalancing is done and kill next one and so on.

 

3.       Is it usually the responsibility of one of the cluster nodes to set active to true, or is it the responsibility of an external client server node on the grid to detect the point at which the cluster running correctly and set active to true? If a cluster type node is used, does this result in a swarm of activation messages as each node inspects the latest topology changed event and sets active to true?

 

You can use both approaches, but in the case of implementing activating login on server nodes you should surround Ignite.active(true) with try catch, the exception can be thrown due to concurrent activation. Also, you can check whether the node is a coordinator and run activation on it only.

 

4. This raises a more general coordinate question: You can’t create or locate a cache in the grid until the grid is active (when using Ignite persistence). Which means you need to stall any logic in the server nodes that wants to do this until the grid is marked as active. How does such logic detect when the grid becomes active? Should it spin and check periodically, or is there an activation event that can be leveraged?

 

 

Unfortunately, there's no event for this now, so you should spin around Ignite.active()

 

Thanks,

Mikhail.

 

 

On Fri, Aug 4, 2017 at 5:01 AM, Raymond Wilson <[hidden email]> wrote:

Sorry, adding a question:

 

4. This raises a more general coordinate question: You can’t create or locate a cache in the grid until the grid is active (when using Ignite persistence). Which means you need to stall any logic in the server nodes that wants to do this until the grid is marked as active. How does such logic detect when the grid becomes active? Should it spin and check periodically, or is there an activation event that can be leveraged?

 

The Ignite persistence example at https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/persistentstore does not address these aspects.

 

Thanks,
Raymond.

 

From: Raymond Wilson [mailto:[hidden email]]
Sent: Friday, August 4, 2017 1:56 PM
To: '[hidden email]'
Subject: RE: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Michael,

 

Some more questions:

 

1.       Can I set the grid active once all primaries are active, but before all backups are active? Or do I need to have the entire cluster running before setting active to true?

2.       Is there an assumption that the cluster size is static and known at the time of startup/restart? If so, how are dynamic clusters that grow and shrink according to demand catered for?

3.       Is it usually the responsibility of one of the cluster nodes to set active to true, or is it the responsibility of an external client server node on the grid to detect the point at which the cluster running correctly and set active to true? If a cluster type node is used, does this result in a swarm of activation messages as each node inspects the latest topology changed event and sets active to true?

 

Thanks,

Raymond.

 

 

From: Michael Cherkasov [[hidden email]]
Sent: Thursday, August 3, 2017 9:27 PM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond, 

 

Unfortunately right now there's no auto-activation, restarting cluster is like rare event that should be controlled 

manually. However you can listen for EVT_NODE_JOINED event, when all nodes in place you can activate a cluster.

 

And you only need this if you have ignite persistence turn on and you have some data on the disk.

 

Thanks,

Mikhail.

 

2017-08-03 6:58 GMT+03:00 Raymond Wilson <[hidden email]>:

Michael,

 

Is there a reference implementation in Ignite 2.1 for an agent that listens to topology changes to decide when to set active to true?

 

Thanks,

Raymond.

 

From: Michael Cherkasov [mailto:[hidden email]]
Sent: Thursday, August 3, 2017 1:25 AM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

>Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

No, you need to deactivate cluster when you going to shutdown the whole cluster. And when you return cluster back to  online, you need to wait when all nodes are in place and then activate it.

 

 

2017-08-02 16:22 GMT+03:00 Rohan Shetty <[hidden email]>:

Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

On Wed, Aug 2, 2017 at 3:18 PM, Michael Cherkasov <[hidden email]> wrote:

when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

 

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.

 

 

 

 



 

--

Thanks,

Mikhail.

mcherkasov mcherkasov
Reply | Threaded
Open this post in threaded view
|

Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

Hi Raymon,

Activator should start an ignite node, which will connect to the server node, but this node should be client node.

Instead of IIgnite ignite = Ignition.TryGetIgnite(“MyGrid”); 
use:
Ignition.ClientMode = true;
IIgnite ignite = Ignition.Start(config);

Then you can activate cluster.

Thanks,
Mikhail.

On Mon, Aug 7, 2017 at 4:52 AM, Raymond Wilson <[hidden email]> wrote:

Hi Michael,

 

I wrote a simple application to act as an activator for a grid. As this is just a POC, it’s simply a form with a button on it that does this:

 

                // Get an ignite reference to the named grid

                IIgnite ignite = Ignition.TryGetIgnite(“MyGrid”);

 

                // If the grid exists and is not already active then set it to active

                if (ignite != null && !ignite.IsActive())

                  ignite.SetActive(true);

 

In my Ignite server I have this code:

 

            IgniteConfiguration cfg = new IgniteConfiguration();

 

            ConfigureGrid(cfg); // Set it up as a grid with persistence enabled

 

            cfg.GridName = “MyGrid”;

            cfg.IgniteInstanceName = “MyGrid”

 

            IIgnite ignite = Ignition.Start(cfg);

 

            // Wait until the grid is active

            bool isActive = false;

            do

            {

                try

                {

                    isActive = ignite.IsActive();

 

                    if (!isActive)

                    {

                        Thread.Sleep(1000);

                    }

                }

                catch

                {

                    // Ignore it and spin

                    Thread.Sleep(1000);

                }

            } while (!isActive);

 

Note the activator and the server are completely different applications/processes, but running on the same local machine.

 

While the server node that creates the grid has started up I hit the button on the activator app to set it to active. Unfortunately the activator app is unable to get a reference to the “MyGrid” grid, TryGetIgnite always returns nil.

 

Does TryGetIgnite return an Ignite grid reference if Active is not set to true in this instance?

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Saturday, August 5, 2017 2:42 AM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

1.       Can I set the grid active once all primaries are active, but before all backups are active? Or do I need to have the entire cluster running before setting active to true?

 

A cluster will work, but it will try to restore backups which are missed and this will cause rebalancing, so it's better to wait for all nodes is restored.

 

2.       Is there an assumption that the cluster size is static and known at the time of startup/restart? If so, how are dynamic clusters that grow and shrink according to demand catered for?

 

well, you can add new nodes when a cluster is active or inactive this doesn't really matter, in case if the cluster isn't active, only after activation new nodes will be used for data storing. There's no convenient way to shrink cluster, you can kill nodes wait when rebalancing is done and kill next one and so on.

 

3.       Is it usually the responsibility of one of the cluster nodes to set active to true, or is it the responsibility of an external client server node on the grid to detect the point at which the cluster running correctly and set active to true? If a cluster type node is used, does this result in a swarm of activation messages as each node inspects the latest topology changed event and sets active to true?

 

You can use both approaches, but in the case of implementing activating login on server nodes you should surround Ignite.active(true) with try catch, the exception can be thrown due to concurrent activation. Also, you can check whether the node is a coordinator and run activation on it only.

 

4. This raises a more general coordinate question: You can’t create or locate a cache in the grid until the grid is active (when using Ignite persistence). Which means you need to stall any logic in the server nodes that wants to do this until the grid is marked as active. How does such logic detect when the grid becomes active? Should it spin and check periodically, or is there an activation event that can be leveraged?

 

 

Unfortunately, there's no event for this now, so you should spin around Ignite.active()

 

Thanks,

Mikhail.

 

 

On Fri, Aug 4, 2017 at 5:01 AM, Raymond Wilson <[hidden email]> wrote:

Sorry, adding a question:

 

4. This raises a more general coordinate question: You can’t create or locate a cache in the grid until the grid is active (when using Ignite persistence). Which means you need to stall any logic in the server nodes that wants to do this until the grid is marked as active. How does such logic detect when the grid becomes active? Should it spin and check periodically, or is there an activation event that can be leveraged?

 

The Ignite persistence example at https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/persistentstore does not address these aspects.

 

Thanks,
Raymond.

 

From: Raymond Wilson [mailto:[hidden email]]
Sent: Friday, August 4, 2017 1:56 PM
To: '[hidden email]'
Subject: RE: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Michael,

 

Some more questions:

 

1.       Can I set the grid active once all primaries are active, but before all backups are active? Or do I need to have the entire cluster running before setting active to true?

2.       Is there an assumption that the cluster size is static and known at the time of startup/restart? If so, how are dynamic clusters that grow and shrink according to demand catered for?

3.       Is it usually the responsibility of one of the cluster nodes to set active to true, or is it the responsibility of an external client server node on the grid to detect the point at which the cluster running correctly and set active to true? If a cluster type node is used, does this result in a swarm of activation messages as each node inspects the latest topology changed event and sets active to true?

 

Thanks,

Raymond.

 

 

From: Michael Cherkasov [[hidden email]]
Sent: Thursday, August 3, 2017 9:27 PM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond, 

 

Unfortunately right now there's no auto-activation, restarting cluster is like rare event that should be controlled 

manually. However you can listen for EVT_NODE_JOINED event, when all nodes in place you can activate a cluster.

 

And you only need this if you have ignite persistence turn on and you have some data on the disk.

 

Thanks,

Mikhail.

 

2017-08-03 6:58 GMT+03:00 Raymond Wilson <[hidden email]>:

Michael,

 

Is there a reference implementation in Ignite 2.1 for an agent that listens to topology changes to decide when to set active to true?

 

Thanks,

Raymond.

 

From: Michael Cherkasov [mailto:[hidden email]]
Sent: Thursday, August 3, 2017 1:25 AM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

>Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

No, you need to deactivate cluster when you going to shutdown the whole cluster. And when you return cluster back to  online, you need to wait when all nodes are in place and then activate it.

 

 

2017-08-02 16:22 GMT+03:00 Rohan Shetty <[hidden email]>:

Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

On Wed, Aug 2, 2017 at 3:18 PM, Michael Cherkasov <[hidden email]> wrote:

when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

 

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.

 

 

 

 



 

--

Thanks,

Mikhail.




--
Thanks,
Mikhail.
Raymond Wilson Raymond Wilson
Reply | Threaded
Open this post in threaded view
|

RE: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

Hi Michael,

 

Thanks for the reminder. I was (or thought I was) actually doing this in an ancestral constructor of my class, but the pattern I was using at the time used a c# static method so the constructor was never hit and the client server was never started Once I modified it to use the  existing constructor everything is fine. <embarrassing/>

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 8, 2017 5:28 AM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymon,

 

Activator should start an ignite node, which will connect to the server node, but this node should be client node.

 

Instead of IIgnite ignite = Ignition.TryGetIgnite(“MyGrid”); 

use:

Ignition.ClientMode = true;

IIgnite ignite = Ignition.Start(config);

 

Then you can activate cluster.

 

Thanks,

Mikhail.

 

On Mon, Aug 7, 2017 at 4:52 AM, Raymond Wilson <[hidden email]> wrote:

Hi Michael,

 

I wrote a simple application to act as an activator for a grid. As this is just a POC, it’s simply a form with a button on it that does this:

 

                // Get an ignite reference to the named grid

                IIgnite ignite = Ignition.TryGetIgnite(“MyGrid”);

 

                // If the grid exists and is not already active then set it to active

                if (ignite != null && !ignite.IsActive())

                  ignite.SetActive(true);

 

In my Ignite server I have this code:

 

            IgniteConfiguration cfg = new IgniteConfiguration();

 

            ConfigureGrid(cfg); // Set it up as a grid with persistence enabled

 

            cfg.GridName = “MyGrid”;

            cfg.IgniteInstanceName = “MyGrid”

 

            IIgnite ignite = Ignition.Start(cfg);

 

            // Wait until the grid is active

            bool isActive = false;

            do

            {

                try

                {

                    isActive = ignite.IsActive();

 

                    if (!isActive)

                    {

                        Thread.Sleep(1000);

                    }

                }

                catch

                {

                    // Ignore it and spin

                    Thread.Sleep(1000);

                }

            } while (!isActive);

 

Note the activator and the server are completely different applications/processes, but running on the same local machine.

 

While the server node that creates the grid has started up I hit the button on the activator app to set it to active. Unfortunately the activator app is unable to get a reference to the “MyGrid” grid, TryGetIgnite always returns nil.

 

Does TryGetIgnite return an Ignite grid reference if Active is not set to true in this instance?

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Saturday, August 5, 2017 2:42 AM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

1.       Can I set the grid active once all primaries are active, but before all backups are active? Or do I need to have the entire cluster running before setting active to true?

 

A cluster will work, but it will try to restore backups which are missed and this will cause rebalancing, so it's better to wait for all nodes is restored.

 

2.       Is there an assumption that the cluster size is static and known at the time of startup/restart? If so, how are dynamic clusters that grow and shrink according to demand catered for?

 

well, you can add new nodes when a cluster is active or inactive this doesn't really matter, in case if the cluster isn't active, only after activation new nodes will be used for data storing. There's no convenient way to shrink cluster, you can kill nodes wait when rebalancing is done and kill next one and so on.

 

3.       Is it usually the responsibility of one of the cluster nodes to set active to true, or is it the responsibility of an external client server node on the grid to detect the point at which the cluster running correctly and set active to true? If a cluster type node is used, does this result in a swarm of activation messages as each node inspects the latest topology changed event and sets active to true?

 

You can use both approaches, but in the case of implementing activating login on server nodes you should surround Ignite.active(true) with try catch, the exception can be thrown due to concurrent activation. Also, you can check whether the node is a coordinator and run activation on it only.

 

4. This raises a more general coordinate question: You can’t create or locate a cache in the grid until the grid is active (when using Ignite persistence). Which means you need to stall any logic in the server nodes that wants to do this until the grid is marked as active. How does such logic detect when the grid becomes active? Should it spin and check periodically, or is there an activation event that can be leveraged?

 

 

Unfortunately, there's no event for this now, so you should spin around Ignite.active()

 

Thanks,

Mikhail.

 

 

On Fri, Aug 4, 2017 at 5:01 AM, Raymond Wilson <[hidden email]> wrote:

Sorry, adding a question:

 

4. This raises a more general coordinate question: You can’t create or locate a cache in the grid until the grid is active (when using Ignite persistence). Which means you need to stall any logic in the server nodes that wants to do this until the grid is marked as active. How does such logic detect when the grid becomes active? Should it spin and check periodically, or is there an activation event that can be leveraged?

 

The Ignite persistence example at https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/persistentstore does not address these aspects.

 

Thanks,
Raymond.

 

From: Raymond Wilson [mailto:[hidden email]]
Sent: Friday, August 4, 2017 1:56 PM
To: '[hidden email]'
Subject: RE: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Michael,

 

Some more questions:

 

1.       Can I set the grid active once all primaries are active, but before all backups are active? Or do I need to have the entire cluster running before setting active to true?

2.       Is there an assumption that the cluster size is static and known at the time of startup/restart? If so, how are dynamic clusters that grow and shrink according to demand catered for?

3.       Is it usually the responsibility of one of the cluster nodes to set active to true, or is it the responsibility of an external client server node on the grid to detect the point at which the cluster running correctly and set active to true? If a cluster type node is used, does this result in a swarm of activation messages as each node inspects the latest topology changed event and sets active to true?

 

Thanks,

Raymond.

 

 

From: Michael Cherkasov [[hidden email]]
Sent: Thursday, August 3, 2017 9:27 PM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond, 

 

Unfortunately right now there's no auto-activation, restarting cluster is like rare event that should be controlled 

manually. However you can listen for EVT_NODE_JOINED event, when all nodes in place you can activate a cluster.

 

And you only need this if you have ignite persistence turn on and you have some data on the disk.

 

Thanks,

Mikhail.

 

2017-08-03 6:58 GMT+03:00 Raymond Wilson <[hidden email]>:

Michael,

 

Is there a reference implementation in Ignite 2.1 for an agent that listens to topology changes to decide when to set active to true?

 

Thanks,

Raymond.

 

From: Michael Cherkasov [mailto:[hidden email]]
Sent: Thursday, August 3, 2017 1:25 AM


To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

>Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

No, you need to deactivate cluster when you going to shutdown the whole cluster. And when you return cluster back to  online, you need to wait when all nodes are in place and then activate it.

 

 

2017-08-02 16:22 GMT+03:00 Rohan Shetty <[hidden email]>:

Does this mean we have to listen to events of server nodes going up and down and activate and deactivate the cluster?

 

On Wed, Aug 2, 2017 at 3:18 PM, Michael Cherkasov <[hidden email]> wrote:

when all nodes are up, so in latest topology snapshot you can see servers count == servers count you run, then cluster can be activated.

 

2017-08-02 0:51 GMT+03:00 Raymond Wilson <[hidden email]>:

Hi Mikhail,

 

Thanks for the clarifications.

 

Yes, I knew setting active was only required when using the persistence layer, which is the topic of the question J

 

I was interested if there were best practices or approaches for determining when the grid had fully initialized. I realise this is somewhat application specific, but was looking for an established pattern before I invented one myself.

 

In my case I have an affinity function that responds to topology changes which intrinsically would know when it had a ‘quorum’. Is this a typical place for setting active to true.

 

Thanks,

Raymond.

 

From: Mikhail Cherkasov [mailto:[hidden email]]
Sent: Tuesday, August 1, 2017 11:59 PM
To: [hidden email]
Subject: Re: Best practise for setting Ignite.Active to true when using persistence layer in Ignite 2.1

 

Hi Raymond,

 

Ignite cluster is inactive on startup only if persistence is enabled. This is done to avoid unnecessary partition exchanges between nodes,

for example, if you have 3 nodes and 1 backup enabled and you start only 2 of 3 nodes, then they will treat the third node as dead and start process

of restoring data from backup and rebalance data to spread them among 2 nodes, when you add the missed third node back the process will be repeated.

 

So we start cluster as in active. When all nodes are started and ready, so no cluster topology changes aren't expected, you should activate cluster.

 

Also when you turn off cluster, some nodes can still accept request for data update and other nodes won't see it, so understand which node has the latest

data we need to start all nodes first and only then activate cluster.

 

Thanks,

Mikhail.

 

On Tue, Aug 1, 2017 at 5:05 AM, Raymond Wilson <[hidden email]> wrote:

Hi,

 

I am experimenting with a POC looking into using the Ignite persistence layer.

 

One aspect of this is setting the grid to be ‘Active’ after all cache grid nodes have instantiated.

 

In practical terms, what is the best practice for ensuring the cluster is running and in a good state to be set to active? What is the downside of setting active to true before all grid nodes are running?

 

Thanks,

Raymond.

 



 

--

Thanks,

Mikhail.

 

 

 

 



 

--

Thanks,

Mikhail.



 

--

Thanks,

Mikhail.