Iterating through native persistence entries before joining the cluster

classic Classic list List threaded Threaded
6 messages Options
Lukas Polacek Lukas Polacek
Reply | Threaded
Open this post in threaded view
|

Iterating through native persistence entries before joining the cluster

Hi,
in our use case we need to run some C++ code (via JNI) whenever something is pushed into the local Ignite cache. In other words, we need to have Ignite in sync with C++ memory. We have a local listener that listens to EVT_CACHE_OBJECT_PUT events and executes the C++ code, so everything is fine while the node is running. However, we use native persistence, so after a node restart, the local cache is read from the disk but the C++ code hasn't been run for any cache entries, which means that Ignite and C++ memory are out of sync.

Iterating through the local cache entries is only possible once the node has already joined the cluster, but that's too late for us - it needs to be done before joining the cluster.

I've managed to add a lifecycle bean event BEFORE_CLUSTER_JOIN (see http://apache-ignite-users.70518.x6.nabble.com/Register-listeners-before-joining-the-cluster-tc25944.html, a PR is hopefully coming soon), which is triggered before joining the cluster but at that point we cannot access the cache via ignite.cache(...). Is there a way to access all entries in the native persistence at that point or earlier? I'm also fine with modifying the Ignite source code if that's necessary (and simple enough), since we are just prototyping.
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Iterating through native persistence entries before joining the cluster

Hello!

I think you should decouple workload readiness from joining the cluster.

You should let node join cluster first, then iterate entries, and only then allow workload to this node.

Regards,
--
Ilya Kasnacheev


чт, 3 янв. 2019 г. в 13:18, Lukas Polacek <[hidden email]>:
Hi,
in our use case we need to run some C++ code (via JNI) whenever something is pushed into the local Ignite cache. In other words, we need to have Ignite in sync with C++ memory. We have a local listener that listens to EVT_CACHE_OBJECT_PUT events and executes the C++ code, so everything is fine while the node is running. However, we use native persistence, so after a node restart, the local cache is read from the disk but the C++ code hasn't been run for any cache entries, which means that Ignite and C++ memory are out of sync.

Iterating through the local cache entries is only possible once the node has already joined the cluster, but that's too late for us - it needs to be done before joining the cluster.

I've managed to add a lifecycle bean event BEFORE_CLUSTER_JOIN (see http://apache-ignite-users.70518.x6.nabble.com/Register-listeners-before-joining-the-cluster-tc25944.html, a PR is hopefully coming soon), which is triggered before joining the cluster but at that point we cannot access the cache via ignite.cache(...). Is there a way to access all entries in the native persistence at that point or earlier? I'm also fine with modifying the Ignite source code if that's necessary (and simple enough), since we are just prototyping.
Lukas Polacek Lukas Polacek
Reply | Threaded
Open this post in threaded view
|

Re: Iterating through native persistence entries before joining the cluster

Hi,
we already do that by using an IgniteSet which contains consistent IDs of the nodes that are ready, but it's very error-prone and unnecessarily complex. Delaying joining the cluster would make everything simpler.

On Thu, Jan 10, 2019 at 1:38 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

I think you should decouple workload readiness from joining the cluster.

You should let node join cluster first, then iterate entries, and only then allow workload to this node.

Regards,
--
Ilya Kasnacheev


чт, 3 янв. 2019 г. в 13:18, Lukas Polacek <[hidden email]>:
Hi,
in our use case we need to run some C++ code (via JNI) whenever something is pushed into the local Ignite cache. In other words, we need to have Ignite in sync with C++ memory. We have a local listener that listens to EVT_CACHE_OBJECT_PUT events and executes the C++ code, so everything is fine while the node is running. However, we use native persistence, so after a node restart, the local cache is read from the disk but the C++ code hasn't been run for any cache entries, which means that Ignite and C++ memory are out of sync.

Iterating through the local cache entries is only possible once the node has already joined the cluster, but that's too late for us - it needs to be done before joining the cluster.

I've managed to add a lifecycle bean event BEFORE_CLUSTER_JOIN (see http://apache-ignite-users.70518.x6.nabble.com/Register-listeners-before-joining-the-cluster-tc25944.html, a PR is hopefully coming soon), which is triggered before joining the cluster but at that point we cannot access the cache via ignite.cache(...). Is there a way to access all entries in the native persistence at that point or earlier? I'm also fine with modifying the Ignite source code if that's necessary (and simple enough), since we are just prototyping.
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Iterating through native persistence entries before joining the cluster

Hello!

Actually, delaying joining the cluster is complex enough since it will add a lot of unexpected concerns.

There are ClusterGroup's precisely for that purpose.

Regards,
--
Ilya Kasnacheev


пт, 11 янв. 2019 г. в 15:58, Lukas Polacek <[hidden email]>:
Hi,
we already do that by using an IgniteSet which contains consistent IDs of the nodes that are ready, but it's very error-prone and unnecessarily complex. Delaying joining the cluster would make everything simpler.

On Thu, Jan 10, 2019 at 1:38 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

I think you should decouple workload readiness from joining the cluster.

You should let node join cluster first, then iterate entries, and only then allow workload to this node.

Regards,
--
Ilya Kasnacheev


чт, 3 янв. 2019 г. в 13:18, Lukas Polacek <[hidden email]>:
Hi,
in our use case we need to run some C++ code (via JNI) whenever something is pushed into the local Ignite cache. In other words, we need to have Ignite in sync with C++ memory. We have a local listener that listens to EVT_CACHE_OBJECT_PUT events and executes the C++ code, so everything is fine while the node is running. However, we use native persistence, so after a node restart, the local cache is read from the disk but the C++ code hasn't been run for any cache entries, which means that Ignite and C++ memory are out of sync.

Iterating through the local cache entries is only possible once the node has already joined the cluster, but that's too late for us - it needs to be done before joining the cluster.

I've managed to add a lifecycle bean event BEFORE_CLUSTER_JOIN (see http://apache-ignite-users.70518.x6.nabble.com/Register-listeners-before-joining-the-cluster-tc25944.html, a PR is hopefully coming soon), which is triggered before joining the cluster but at that point we cannot access the cache via ignite.cache(...). Is there a way to access all entries in the native persistence at that point or earlier? I'm also fine with modifying the Ignite source code if that's necessary (and simple enough), since we are just prototyping.
Lukas Polacek Lukas Polacek
Reply | Threaded
Open this post in threaded view
|

Re: Iterating through native persistence entries before joining the cluster

Technically, I could delay joining the cluster by adding Thread.sleep() into the BEFORE_NODE_START lifecycle bean (or my newly created BEFORE_CLUSTER_JOIN). I'd like to iterate through the local cache in one of those lifecycle beans - but that doesn't seem possible.

Regarding cluster groups: I can't use IgniteCluster::forAttributes, since attributes are set at the node start. The only remaining option seems IgniteCluster::forPredicate, but that's also not very helpful, since I don't see how to leverage ClusterNode's properties, in particular its metrics.

We have been using IgniteSet as mentioned above to produce an "active cluster group" but I'm looking for something simpler. Today I found a very problematic case which will require yet another special logic to handle it. 

On Fri, Jan 11, 2019 at 4:14 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Actually, delaying joining the cluster is complex enough since it will add a lot of unexpected concerns.

There are ClusterGroup's precisely for that purpose.

Regards,
--
Ilya Kasnacheev


пт, 11 янв. 2019 г. в 15:58, Lukas Polacek <[hidden email]>:
Hi,
we already do that by using an IgniteSet which contains consistent IDs of the nodes that are ready, but it's very error-prone and unnecessarily complex. Delaying joining the cluster would make everything simpler.

On Thu, Jan 10, 2019 at 1:38 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

I think you should decouple workload readiness from joining the cluster.

You should let node join cluster first, then iterate entries, and only then allow workload to this node.

Regards,
--
Ilya Kasnacheev


чт, 3 янв. 2019 г. в 13:18, Lukas Polacek <[hidden email]>:
Hi,
in our use case we need to run some C++ code (via JNI) whenever something is pushed into the local Ignite cache. In other words, we need to have Ignite in sync with C++ memory. We have a local listener that listens to EVT_CACHE_OBJECT_PUT events and executes the C++ code, so everything is fine while the node is running. However, we use native persistence, so after a node restart, the local cache is read from the disk but the C++ code hasn't been run for any cache entries, which means that Ignite and C++ memory are out of sync.

Iterating through the local cache entries is only possible once the node has already joined the cluster, but that's too late for us - it needs to be done before joining the cluster.

I've managed to add a lifecycle bean event BEFORE_CLUSTER_JOIN (see http://apache-ignite-users.70518.x6.nabble.com/Register-listeners-before-joining-the-cluster-tc25944.html, a PR is hopefully coming soon), which is triggered before joining the cluster but at that point we cannot access the cache via ignite.cache(...). Is there a way to access all entries in the native persistence at that point or earlier? I'm also fine with modifying the Ignite source code if that's necessary (and simple enough), since we are just prototyping.
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Iterating through native persistence entries before joining the cluster

Hello!

I guess you could be starting a service instance on every node, doing warmup in init().

Then service requests will be only routed on nodes where warmup is complete.

Regards,
--
Ilya Kasnacheev


чт, 17 янв. 2019 г. в 13:26, Lukas Polacek <[hidden email]>:
Technically, I could delay joining the cluster by adding Thread.sleep() into the BEFORE_NODE_START lifecycle bean (or my newly created BEFORE_CLUSTER_JOIN). I'd like to iterate through the local cache in one of those lifecycle beans - but that doesn't seem possible.

Regarding cluster groups: I can't use IgniteCluster::forAttributes, since attributes are set at the node start. The only remaining option seems IgniteCluster::forPredicate, but that's also not very helpful, since I don't see how to leverage ClusterNode's properties, in particular its metrics.

We have been using IgniteSet as mentioned above to produce an "active cluster group" but I'm looking for something simpler. Today I found a very problematic case which will require yet another special logic to handle it. 

On Fri, Jan 11, 2019 at 4:14 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Actually, delaying joining the cluster is complex enough since it will add a lot of unexpected concerns.

There are ClusterGroup's precisely for that purpose.

Regards,
--
Ilya Kasnacheev


пт, 11 янв. 2019 г. в 15:58, Lukas Polacek <[hidden email]>:
Hi,
we already do that by using an IgniteSet which contains consistent IDs of the nodes that are ready, but it's very error-prone and unnecessarily complex. Delaying joining the cluster would make everything simpler.

On Thu, Jan 10, 2019 at 1:38 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

I think you should decouple workload readiness from joining the cluster.

You should let node join cluster first, then iterate entries, and only then allow workload to this node.

Regards,
--
Ilya Kasnacheev


чт, 3 янв. 2019 г. в 13:18, Lukas Polacek <[hidden email]>:
Hi,
in our use case we need to run some C++ code (via JNI) whenever something is pushed into the local Ignite cache. In other words, we need to have Ignite in sync with C++ memory. We have a local listener that listens to EVT_CACHE_OBJECT_PUT events and executes the C++ code, so everything is fine while the node is running. However, we use native persistence, so after a node restart, the local cache is read from the disk but the C++ code hasn't been run for any cache entries, which means that Ignite and C++ memory are out of sync.

Iterating through the local cache entries is only possible once the node has already joined the cluster, but that's too late for us - it needs to be done before joining the cluster.

I've managed to add a lifecycle bean event BEFORE_CLUSTER_JOIN (see http://apache-ignite-users.70518.x6.nabble.com/Register-listeners-before-joining-the-cluster-tc25944.html, a PR is hopefully coming soon), which is triggered before joining the cluster but at that point we cannot access the cache via ignite.cache(...). Is there a way to access all entries in the native persistence at that point or earlier? I'm also fine with modifying the Ignite source code if that's necessary (and simple enough), since we are just prototyping.