Load balanced CQ listener/Catch up missed events

classic Classic list List threaded Threaded
9 messages Options
Devakumar J Devakumar J
Reply | Threaded
Open this post in threaded view
|

Load balanced CQ listener/Catch up missed events

Hi,

We have a setup of 3 server nodes and 2 client nodes and 50 caches around.

When we register CQ listener for cache operations from client node, both the
client nodes are getting notified.

1. Is there way to achieve load balanced manner? For e.g) In case of 2
events, event-1 to be  received by
    client-1 and event-2 to be received by client-2 and in-case of one
client node down and other node should
    start receiving all the events?

2. In case of both node failure, is there a way to get only those missed
events?

Thanks,
Devakumar J




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Load balanced CQ listener/Catch up missed events

Hi Devakumar, 

I think it's feasible to implement such logic with custom remote filters:

Also, an alternate option might be a partitioned message bus like Kafka. Servers can send changes to a Kafka topic and the clients can consume the messages on the other side in a load-balanced and fault-tolerant fashion.

-
Denis


On Fri, Jul 17, 2020 at 8:34 AM Devakumar J <[hidden email]> wrote:
Hi,

We have a setup of 3 server nodes and 2 client nodes and 50 caches around.

When we register CQ listener for cache operations from client node, both the
client nodes are getting notified.

1. Is there way to achieve load balanced manner? For e.g) In case of 2
events, event-1 to be  received by
    client-1 and event-2 to be received by client-2 and in-case of one
client node down and other node should
    start receiving all the events?

2. In case of both node failure, is there a way to get only those missed
events?

Thanks,
Devakumar J




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Devakumar J Devakumar J
Reply | Threaded
Open this post in threaded view
|

Re: Load balanced CQ listener/Catch up missed events

This post was updated on .
Hi Denis,

Thanks for your reply.

I will explore the custom implementation of remote filter for load
balancing.

Also currently we dont use kafka in our tech stack, so we dont want to
introduce it just for this purpose.

I was also checking ignite events and distributed message queue as an
alternative for Continuous Queries.

Can we define setup like,
1. Define distributed message queue across the cluster.
2. Each server node publish the selective cache events to the queue.
3. Client nodes consume the queue.

So in this case as per my understanding on distributed queue,
      a. Client nodes will get events in load balanced way.
      b. If any one client node down then other node will start consuming
all the events.
      c. If both client nodes down, queue will still have the entries that
are not yet consumed as it is persisted. So if client nodes come up, it will
resume from where it left.

Is this feasible or do you see any issue in setup in case of server node
failure/ server node re-balancing.

Thanks,
Devakumar J




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Load balanced CQ listener/Catch up missed events

The queue can be a good alternative [1] but the application has to care about faults handling. For example, if a client reads a message from the queue, starts doing some job as the message prescribes and then fails in the middle, then the job is incomplete and nobody else can pick it up. So, you would need to track somewhere else that the message is in fact was fully processed (just toggle the status).

 
-
Denis


On Sat, Jul 18, 2020 at 12:42 AM Devakumar J <[hidden email]> wrote:
Hi Denis,

Thanks for your reply.

I will explore the custom implementation of remote filter for load
balancing.

Also currently we dont use kafka in our tech stack, so we dont want to
introduce it just for this purpose.

I was also checking ignite events and distributed message queue as an
alternative for Continuous Queries.

Can we define setup like,
1. Define distributed message queue across the cluster.
2. Each server node publish the selective cache events to the queue.
3. Client nodes consume the queue.

So in this case as per my understanding on distributed queue,
      a. Client nodes will get events in load balanced way.
      b. If any one client node down then other node will start consuming
all the events.
      c. If both client nodes down, queue will still have the entries that
are not yet consumed as it is persisted. So if client nodes come up, it will
resume from where it left.

Is this feasible or do you see any issue in setup in case of server node
failure/ server node re-balancing.

Thanks,
Devakumar J




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Devakumar J Devakumar J
Reply | Threaded
Open this post in threaded view
|

Re: Load balanced CQ listener/Catch up missed events

Thanks Denis. I will explore further on this.

Some more questions in distributed queue.

1. When i create a queue, should it be created at each server node
separately or it should be created once cluster wide?

2) If cluster wide, can multiple server nodes publish to the same queue?

3) Also is there a way to create the queue at client side and store a part
of client node itself?

Thanks,
Devakumar J



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Devakumar J Devakumar J
Reply | Threaded
Open this post in threaded view
|

Re: Load balanced CQ listener/Catch up missed events

In reply to this post by dmagda
Thanks Denis. I will explore further on this.

Some more questions in distributed queue.

1. When i create a queue, should it be created at each server node
separately or it should be created once cluster wide?

2) If cluster wide, can multiple server nodes publish to the same queue?

3) Also is there a way to create the queue at client side and store a part
of client node itself?

Thanks,
Devakumar J



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Load balanced CQ listener/Catch up missed events

In reply to this post by Devakumar J
1. It depends on your application needs. Most likely a single distributed queue will work if it needs to be accessed by any remote producer or consumer.

2. Yes, refer to the documentation from my previous email.

3. What do you mean under "storing a part of the client node"?

-
Denis


On Wed, Jul 22, 2020 at 9:28 AM Devakumar J <[hidden email]> wrote:
Thanks Denis. I will explore further on this.

Some more questions in distributed queue.

1. When i create a queue, should it be created at each server node
separately or it should be created once cluster wide?

2) If cluster wide, can multiple server nodes publish to the same queue?

3) Also is there a way to create the queue at client side and store a part
of client node itself?

Thanks,
Devakumar J



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Devakumar J Devakumar J
Reply | Threaded
Open this post in threaded view
|

Re: Load balanced CQ listener/Catch up missed events

Hi Denis,

Thanks for the reply.

I mean to ask, is it possible to create distributed queue at client node and
if so where it will be persisted when native persistence enabled.

Should it be under server folder's work folder?

Thanks,
Devakumar J



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Load balanced CQ listener/Catch up missed events

The client can create a queue, and the servers will keep items of that queue. Those items will be stored in memory and disk tiers of the servers. And, yes, physically the records will be persisted in the database folder (that is under the work folder by default). Hope this helps.
 
-
Denis


On Fri, Jul 24, 2020 at 9:36 AM Devakumar J <[hidden email]> wrote:
Hi Denis,

Thanks for the reply.

I mean to ask, is it possible to create distributed queue at client node and
if so where it will be persisted when native persistence enabled.

Should it be under server folder's work folder?

Thanks,
Devakumar J



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/