Re: Using Apache Ingnite as RDBMS in memory Cache

classic Classic list List threaded Threaded
5 messages Options
Andrés Ivaldi Andrés Ivaldi
Reply | Threaded
Open this post in threaded view
|

Re: Using Apache Ingnite as RDBMS in memory Cache

Hello, I'm new with Apache Ignite, I'd like to know how can I use Apache Ignite cache to bust up RDBMS queries. I saw it in apache conn that is possible perform querys against RDBMS allowing to speed up it's execution

Regards.

--
Ing. Ivaldi Andres
Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Using Apache Ingnite as RDBMS in memory Cache

Hi,

To achieve the performance boost you need to use Apache Ignite SQL engine [1]. However all the data has to be located in caches (RAM). SQL engine doesn’t work against RDBMS.

So you need to preload data into caches from RDBMS using one of existed approaches [2]. After that you can start using Ignite SQL.
If you want to sync up caches with RDBMS then you need to use one of CacheStore interface implementations [3].

Finally, I would recommend you taking look at Ignite examples to see how the product works in practice.


Denis

On Jun 17, 2016, at 12:20 AM, Andrés Ivaldi <[hidden email]> wrote:

Hello, I'm new with Apache Ignite, I'd like to know how can I use Apache Ignite cache to bust up RDBMS queries. I saw it in apache conn that is possible perform querys against RDBMS allowing to speed up it's execution

Regards.

--
Ing. Ivaldi Andres

Jörn Franke Jörn Franke
Reply | Threaded
Open this post in threaded view
|

Re: Using Apache Ingnite as RDBMS in memory Cache

In reply to this post by Andrés Ivaldi

This depends on the type of queries! 
In any case: before you go in-Memory optimize your current data model and exploit your current technology. I have seen in the past often purely designed data model that do not leverage the underlying technology well. 

On 16 Jun 2016, at 23:20, Andrés Ivaldi <[hidden email]> wrote:

Hello, I'm new with Apache Ignite, I'd like to know how can I use Apache Ignite cache to bust up RDBMS queries. I saw it in apache conn that is possible perform querys against RDBMS allowing to speed up it's execution

Regards.

--
Ing. Ivaldi Andres
Andrés Ivaldi Andrés Ivaldi
Reply | Threaded
Open this post in threaded view
|

Re: Using Apache Ingnite as RDBMS in memory Cache

Jörn, you are right with that point, actually we don't have a data model we just query flat data to perform a kind of ROLAP.
We are currently researching for the best option. The idea with Ignite is to use in-memory cache to perform fast queries and as a layer for different kind of data sources (not necessary RDBMS)
We where able to do it with Spark but is too slow for user experience(I've to try 2.0 they said that the speed was improved), also I looked that Ignite can be used as Spark chache with Ignite RDD maybe that could be another approach.

Thanks

On Fri, Jun 17, 2016 at 2:29 AM, Jörn Franke <[hidden email]> wrote:

This depends on the type of queries! 
In any case: before you go in-Memory optimize your current data model and exploit your current technology. I have seen in the past often purely designed data model that do not leverage the underlying technology well. 

On 16 Jun 2016, at 23:20, Andrés Ivaldi <[hidden email]> wrote:

Hello, I'm new with Apache Ignite, I'd like to know how can I use Apache Ignite cache to bust up RDBMS queries. I saw it in apache conn that is possible perform querys against RDBMS allowing to speed up it's execution

Regards.

--
Ing. Ivaldi Andres



--
Ing. Ivaldi Andres
Jörn Franke Jörn Franke
Reply | Threaded
Open this post in threaded view
|

Re: Using Apache Ingnite as RDBMS in memory Cache

What do you mean by flat data?
Csv files? 
You should use Orc or parquet format if you sort the data correctly. Hive+tez+Orc+optionally ignite HDFS cache of the most recent partitions in memory could be an interesting alternative for you.

Similarly spark can have a good performance even in older version. However I recommend spark mostly for iterative machine learning . 

On 17 Jun 2016, at 14:48, Andrés Ivaldi <[hidden email]> wrote:

Jörn, you are right with that point, actually we don't have a data model we just query flat data to perform a kind of ROLAP.
We are currently researchingk for the best option. The idea with Ignite is to use in-memory cache to perform fast queries and as a layer for different kind of data sources (not necessary RDBMS)
We where able to do it with Spark but is too slow for user experience(I've to try 2.0 they said that the speed was improved), also I looked that Ignite can be used as Spark chache with Ignite RDD maybe that could be another approach.

Thanks

On Fri, Jun 17, 2016 at 2:29 AM, Jörn Franke <[hidden email]> wrote:

This depends on the type of queries! 
In any case: before you go in-Memory optimize your current data model and exploit your current technology. I have seen in the past often purely designed data model that do not leverage the underlying technology well. 

On 16 Jun 2016, at 23:20, Andrés Ivaldi <[hidden email]> wrote:

Hello, I'm new with Apache Ignite, I'd like to know how can I use Apache Ignite cache to bust up RDBMS queries. I saw it in apache conn that is possible perform querys against RDBMS allowing to speed up it's execution

Regards.

--
Ing. Ivaldi Andres



--
Ing. Ivaldi Andres