Ignite Performance

classic Classic list List threaded Threaded
3 messages Options
shahidv shahidv
Reply | Threaded
Open this post in threaded view
|

Ignite Performance

Hi,

We are looking for new database technologies because of some performance
issues in existing.
So we tried ignite with 1 node cluster (28gb ram), loaded 90 million records
to the cache.
But we faced several issues like,
-when restarted node, It took 6 hrs to activate again,
-queries are very slower,
-out of memory issue when loading data.
-huge size for data.

We are still learning stage to Ignite  and it may be some configuration
issues.
We have currently large volume of data and processing heavy aggregations.

Anyone please tell me Ignite will be a solution for performance and what is
the issue with my current setup. Following is my config. file,
default-config.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t2499/default-config.xml>  




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
a.dolmatov a.dolmatov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Performance

Hi,
-when restarted node, It took 6 hrs to activate again, 
If you activated cluster on startup, before data loading, then after restart, cluster is active.

-queries are very slower, 
it depends. What SQL queries do you have? JOIN, AGG?

-out of memory issue when loading data. 
Show your CacheConfiguration

-huge size for data.
If you enable QueryEnity for SQL support, Ignite consumes much more memory.

Best Regards, Andrey Dolmatov

вт, 9 июл. 2019 г. в 16:10, shahidv <[hidden email]>:
Hi,

We are looking for new database technologies because of some performance
issues in existing.
So we tried ignite with 1 node cluster (28gb ram), loaded 90 million records
to the cache.
But we faced several issues like,
-when restarted node, It took 6 hrs to activate again,
-queries are very slower,
-out of memory issue when loading data.
-huge size for data.

We are still learning stage to Ignite  and it may be some configuration
issues.
We have currently large volume of data and processing heavy aggregations.

Anyone please tell me Ignite will be a solution for performance and what is
the issue with my current setup. Following is my config. file,
default-config.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t2499/default-config.xml




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
akurbanov akurbanov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Performance

Hi shahidv,

Could you post please the details for configurations of caches, ideas of queries and activation logs?

It's not easy to blindly give some recommendations to follow as there is usually no silver bullet for SQL queries execution in a distributed environment as you have to work on each specific query to mitigate possible bottlenecks: make them as local as possible using data collocation, use some precalculated data if possible, look into join order (larger tables should be joined to smaller ones), probably you might make some small tables replicated -- there are a lot of things that are might be reasonable to be considered, but details for SQL tables/queries are necessary for this. 

Here are couple of things I'd like to know besides previously asked ones:

-- What was server node doing while starting up? Was that storage pages loading, index building, wal files rollover (have you removed by any chance wal/archive folder)?
-- Do you have any kind of subqueries / group by statements in your queries?
-- What is the conditions for OOM? Are you able to load whole data set?
-- As for the data size, what is the cache entry structure and what is the median/average size of the record? What is the estimated size of raw data being inserted into Ignite? Do you have any custom indexes and if so, how many and what columns are there?

вт, 9 июл. 2019 г. в 16:30, Andrey Dolmatov <[hidden email]>:
Hi,
-when restarted node, It took 6 hrs to activate again, 
If you activated cluster on startup, before data loading, then after restart, cluster is active.

-queries are very slower, 
it depends. What SQL queries do you have? JOIN, AGG?

-out of memory issue when loading data. 
Show your CacheConfiguration

-huge size for data.
If you enable QueryEnity for SQL support, Ignite consumes much more memory.

Best Regards, Andrey Dolmatov

вт, 9 июл. 2019 г. в 16:10, shahidv <[hidden email]>:
Hi,

We are looking for new database technologies because of some performance
issues in existing.
So we tried ignite with 1 node cluster (28gb ram), loaded 90 million records
to the cache.
But we faced several issues like,
-when restarted node, It took 6 hrs to activate again,
-queries are very slower,
-out of memory issue when loading data.
-huge size for data.

We are still learning stage to Ignite  and it may be some configuration
issues.
We have currently large volume of data and processing heavy aggregations.

Anyone please tell me Ignite will be a solution for performance and what is
the issue with my current setup. Following is my config. file,
default-config.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t2499/default-config.xml




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/