Action performed multiple times when using HA ignite clients using continuous queries

classic Classic list List threaded Threaded
5 messages Options
SunSatION SunSatION
Reply | Threaded
Open this post in threaded view
|

Action performed multiple times when using HA ignite clients using continuous queries

Hi,

We have a scenario where we're using Ignite Kafka Datastream to write to
cache which are setup using PARTITIONED affinity. We're using continuous
queries to identify changes to each cache entry and perform an action if the
entry changes which are running on the client. We have 3 server nodes and 3
client nodes for HA. The problem is that all clients are receiving the cache
changes and the action is performed 3 times, once for each client. We are
unable to use the affinity key of the cache entry as it returns the id of
the server nodes.

Any ideas on the a way that the action is performed once without the client
nodes knowing of each other?

Thanks,
Dorian  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Action performed multiple times when using HA ignite clients using continuous queries

Your clients don’t “know” about each other, so, yes, they’re going to duplicate work. Could you use the Service Grid to run your client? That way it would fail over automatically.

Regards,
Stephen

> On 16 Oct 2019, at 16:04, SunSatION <[hidden email]> wrote:
>
> Hi,
>
> We have a scenario where we're using Ignite Kafka Datastream to write to
> cache which are setup using PARTITIONED affinity. We're using continuous
> queries to identify changes to each cache entry and perform an action if the
> entry changes which are running on the client. We have 3 server nodes and 3
> client nodes for HA. The problem is that all clients are receiving the cache
> changes and the action is performed 3 times, once for each client. We are
> unable to use the affinity key of the cache entry as it returns the id of
> the server nodes.
>
> Any ideas on the a way that the action is performed once without the client
> nodes knowing of each other?
>
> Thanks,
> Dorian  
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/


SunSatION SunSatION
Reply | Threaded
Open this post in threaded view
|

Re: Action performed multiple times when using HA ignite clients using continuous queries

I don't wish to deploy the client code onto each and every server as we would like to keep the separation between business logic and data. Additionally, we are pushing the changes to Kafka and therefore not sure if it's heavy to perform the push the Service grid

On Wed, Oct 16, 2019 at 6:02 PM Stephen Darlington <[hidden email]> wrote:
Your clients don’t “know” about each other, so, yes, they’re going to duplicate work. Could you use the Service Grid to run your client? That way it would fail over automatically.

Regards,
Stephen

> On 16 Oct 2019, at 16:04, SunSatION <[hidden email]> wrote:
>
> Hi,
>
> We have a scenario where we're using Ignite Kafka Datastream to write to
> cache which are setup using PARTITIONED affinity. We're using continuous
> queries to identify changes to each cache entry and perform an action if the
> entry changes which are running on the client. We have 3 server nodes and 3
> client nodes for HA. The problem is that all clients are receiving the cache
> changes and the action is performed 3 times, once for each client. We are
> unable to use the affinity key of the cache entry as it returns the id of
> the server nodes.
>
> Any ideas on the a way that the action is performed once without the client
> nodes knowing of each other?
>
> Thanks,
> Dorian 
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/




--
Two things are infinite: the universe and the human stupidity; and I'm not sure about the universe - Albert Einstein
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Action performed multiple times when using HA ignite clients using continuous queries

Hello!

You can use distributed locks to make sure only one client is working on a chunk of data.

Regards,
--
Ilya Kasnacheev


чт, 17 окт. 2019 г. в 12:37, SunSatION <[hidden email]>:
I don't wish to deploy the client code onto each and every server as we would like to keep the separation between business logic and data. Additionally, we are pushing the changes to Kafka and therefore not sure if it's heavy to perform the push the Service grid

On Wed, Oct 16, 2019 at 6:02 PM Stephen Darlington <[hidden email]> wrote:
Your clients don’t “know” about each other, so, yes, they’re going to duplicate work. Could you use the Service Grid to run your client? That way it would fail over automatically.

Regards,
Stephen

> On 16 Oct 2019, at 16:04, SunSatION <[hidden email]> wrote:
>
> Hi,
>
> We have a scenario where we're using Ignite Kafka Datastream to write to
> cache which are setup using PARTITIONED affinity. We're using continuous
> queries to identify changes to each cache entry and perform an action if the
> entry changes which are running on the client. We have 3 server nodes and 3
> client nodes for HA. The problem is that all clients are receiving the cache
> changes and the action is performed 3 times, once for each client. We are
> unable to use the affinity key of the cache entry as it returns the id of
> the server nodes.
>
> Any ideas on the a way that the action is performed once without the client
> nodes knowing of each other?
>
> Thanks,
> Dorian 
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/




--
Two things are infinite: the universe and the human stupidity; and I'm not sure about the universe - Albert Einstein
stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Action performed multiple times when using HA ignite clients using continuous queries

In reply to this post by SunSatION
You can deploy services to client nodes, for example:
val service = ignite.services(ignite.cluster().forClients())
service.deployClusterSingleton("TestService", new TestServiceImpl())
If you want to have independent clients, as you currently have, I suspect you’d have to implement your own fail over mechanism.

Regards,
Stephen

On 17 Oct 2019, at 10:37, SunSatION <[hidden email]> wrote:

I don't wish to deploy the client code onto each and every server as we would like to keep the separation between business logic and data. Additionally, we are pushing the changes to Kafka and therefore not sure if it's heavy to perform the push the Service grid

On Wed, Oct 16, 2019 at 6:02 PM Stephen Darlington <[hidden email]> wrote:
Your clients don’t “know” about each other, so, yes, they’re going to duplicate work. Could you use the Service Grid to run your client? That way it would fail over automatically.

Regards,
Stephen

> On 16 Oct 2019, at 16:04, SunSatION <[hidden email]> wrote:
>
> Hi,
>
> We have a scenario where we're using Ignite Kafka Datastream to write to
> cache which are setup using PARTITIONED affinity. We're using continuous
> queries to identify changes to each cache entry and perform an action if the
> entry changes which are running on the client. We have 3 server nodes and 3
> client nodes for HA. The problem is that all clients are receiving the cache
> changes and the action is performed 3 times, once for each client. We are
> unable to use the affinity key of the cache entry as it returns the id of
> the server nodes.
>
> Any ideas on the a way that the action is performed once without the client
> nodes knowing of each other?
>
> Thanks,
> Dorian 
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/




--
Two things are infinite: the universe and the human stupidity; and I'm not sure about the universe - Albert Einstein