Communication between Ignite Clusters

classic Classic list List threaded Threaded
7 messages Options
steve.hostettler steve.hostettler
Reply | Threaded
Open this post in threaded view
|

Communication between Ignite Clusters

Hello,

I would like to know what is the best way to integrate two different Apache
Ignite Clusters?
I can see the ignite client and the jdbc integration.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Communication between Ignite Clusters

What do you mean by “integrate”? Replicate data between? One cluster stretched across two data centres?

> On 14 Oct 2020, at 09:20, steve.hostettler <[hidden email]> wrote:
>
> Hello,
>
> I would like to know what is the best way to integrate two different Apache
> Ignite Clusters?
> I can see the ignite client and the jdbc integration.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/


steve.hostettler steve.hostettler
Reply | Threaded
Open this post in threaded view
|

Re: Communication between Ignite Clusters

Hello and thank you for your answer.

Assuming we would have 2 clusters (e.g., we want to scale them
independently, upgrade them at different pace) that are manipulating
different datasets. The output of the first clusters being the input of the
second one.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Communication between Ignite Clusters

There’s nothing “out of the box” I’m afraid. There are commercial options available.

With open source, it would be relatively straight-forward to copy changes over using something like Kafka. Or, assuming you’d want it to be event-driven, you could listen to the first cluster with a thick client, publishing to the second using a thin-client.

The challenge in both cases comes in making sure the two stores stay in sync in the event of outages/downtime.

> On 14 Oct 2020, at 11:07, steve.hostettler <[hidden email]> wrote:
>
> Hello and thank you for your answer.
>
> Assuming we would have 2 clusters (e.g., we want to scale them
> independently, upgrade them at different pace) that are manipulating
> different datasets. The output of the first clusters being the input of the
> second one.
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/


steve.hostettler steve.hostettler
Reply | Threaded
Open this post in threaded view
|

Re: Communication between Ignite Clusters

Thanks a lot for your answer,

So you would use a kafka for instance and then how to push the data? Using a
data streamer to balance the data on the right nodes of the second cluster?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Communication between Ignite Clusters

You could use Ignite’s Kafka adapter — it does both.

To do it “manually” you’d use a continuous query or events to listen to changes. A data streamer or just simple puts depending on the volume to insert into the remote cluster. The data is balanced by virtue of the key; you don’t need to do anything special.

> On 14 Oct 2020, at 12:35, steve.hostettler <[hidden email]> wrote:
>
> Thanks a lot for your answer,
>
> So you would use a kafka for instance and then how to push the data? Using a
> data streamer to balance the data on the right nodes of the second cluster?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/


steve.hostettler steve.hostettler
Reply | Threaded
Open this post in threaded view
|

Re: Communication between Ignite Clusters