Data from multiple MySQL DBs

classic Classic list List threaded Threaded
3 messages Options
Kurt Semba Kurt Semba
Reply | Threaded
Open this post in threaded view
|

Data from multiple MySQL DBs

Hi all,

 

I need to sync data from multiple MySQL databases into Ignite.

 

All those MySQL databases follow the same schema / same tables but obviously contain different data. They are separate instances of the same DB and we want to pull all that data into Ignite to have a central store to query against.

 

What would be the best strategy to solve this?

 

Create a dedicated Cache for each DB instance and then find a way to create a UNION SQL query over all those Caches?

Find a way to define all the MySQL DBs as data sources in the Spring XML cluster config file and hope that Ignite pulls the same type of data from all DBs?

 

I’m open for suggestions 😊

Thanks

Kurt

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Data from multiple MySQL DBs

Hello!

I think the best course will be to read all DBs in user code and use data streamer to populate Ignite.

Regards,
--
Ilya Kasnacheev


вт, 10 сент. 2019 г. в 18:01, Kurt Semba <[hidden email]>:

Hi all,

 

I need to sync data from multiple MySQL databases into Ignite.

 

All those MySQL databases follow the same schema / same tables but obviously contain different data. They are separate instances of the same DB and we want to pull all that data into Ignite to have a central store to query against.

 

What would be the best strategy to solve this?

 

Create a dedicated Cache for each DB instance and then find a way to create a UNION SQL query over all those Caches?

Find a way to define all the MySQL DBs as data sources in the Spring XML cluster config file and hope that Ignite pulls the same type of data from all DBs?

 

I’m open for suggestions 😊

Thanks

Kurt

Kurt Semba Kurt Semba
Reply | Threaded
Open this post in threaded view
|

RE: Data from multiple MySQL DBs

Hi Ilya,

 

Thanks for your suggestion. So what you are suggesting is:

  • Run a Ignite single cluster which has the correct schemas
  • Develop a tool which regularly pulls data from all these MySQL DBs and that uses the Ignite Data Streamer API to update the Caches in the cluster

 

  • That way, all datasets from a specific schema & table from all MySQL DBs will end up in the same Ignite Cache and thus the data will be a UNION of all data from all MySQL DBs.

Kurt

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Wednesday, September 11, 2019 5:10 PM
To: [hidden email]
Subject: Re: Data from multiple MySQL DBs

 

Hello!

 

I think the best course will be to read all DBs in user code and use data streamer to populate Ignite.

 

Regards,

--

Ilya Kasnacheev

 

 

вт, 10 сент. 2019 г. в 18:01, Kurt Semba <[hidden email]>:

Hi all,

 

I need to sync data from multiple MySQL databases into Ignite.

 

All those MySQL databases follow the same schema / same tables but obviously contain different data. They are separate instances of the same DB and we want to pull all that data into Ignite to have a central store to query against.

 

What would be the best strategy to solve this?

 

Create a dedicated Cache for each DB instance and then find a way to create a UNION SQL query over all those Caches?

Find a way to define all the MySQL DBs as data sources in the Spring XML cluster config file and hope that Ignite pulls the same type of data from all DBs?

 

I’m open for suggestions 😊

Thanks

Kurt