Ignite performance issues...

classic Classic list List threaded Threaded
2 messages Options
Wolfgang Meyerle Wolfgang Meyerle
Reply | Threaded
Open this post in threaded view
|

Ignite performance issues...

Hi,

I have a question in regards to Apache Ignite performance tuning.

This is my config so far:

https://pastebin.com/NWDzY3RK

I plan to store 2 billion entries in the database. The machine that I have is not great, 32gb ram nothing more.

I wonder how I can tune the performance to speed up things. I know that if I leave out indexing that would speed things are but indices are necessary. Also an SQL access is necessary though not necessarily during inserts. As I'm using C++ I don't know if ignite supports the feature in creating an SQL access later on after the data has been inserted in a cache. Is this possible?

According to my logging I'm able to insert

42108964 entries in 66 minutes

with 610 Insertions per second:610

and an average of 1057 insertions per second (which is not great in my opinion)


According to ignites performance guidelines I segregated already to Wal stores from the persistence storage drive.

I'm also getting Apache ignite errors from time to time:

[21:24:15] Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=db-checkpoint-thread, igniteInstanceName=null, finished=false, heartbeatTs=1605385441327]]]

Tried to find something online and in the Apache Ignite documentation but struggles so hopefully any geek out there can drop a comment...

Regards,


Wolfgang



Mikael-2 Mikael-2
Reply | Threaded
Open this post in threaded view
|

Re: Ignite performance issues...

Hi!

How many nodes will you be running this on ?

What is your use case, what kind of performance is important to you, reading, writing ? do you need access to all data or will you just use a small subset ?

In terms of writing you can get good performance with a streamer instead of using SQL insert if that would be possible.

Mikael


On 2020-11-14 23:35, Wolfgang Meyerle wrote:

Hi,

I have a question in regards to Apache Ignite performance tuning.

This is my config so far:

https://pastebin.com/NWDzY3RK

I plan to store 2 billion entries in the database. The machine that I have is not great, 32gb ram nothing more.

I wonder how I can tune the performance to speed up things. I know that if I leave out indexing that would speed things are but indices are necessary. Also an SQL access is necessary though not necessarily during inserts. As I'm using C++ I don't know if ignite supports the feature in creating an SQL access later on after the data has been inserted in a cache. Is this possible?

According to my logging I'm able to insert

42108964 entries in 66 minutes

with 610 Insertions per second:610

and an average of 1057 insertions per second (which is not great in my opinion)


According to ignites performance guidelines I segregated already to Wal stores from the persistence storage drive.

I'm also getting Apache ignite errors from time to time:

[21:24:15] Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=db-checkpoint-thread, igniteInstanceName=null, finished=false, heartbeatTs=1605385441327]]]

Tried to find something online and in the Apache Ignite documentation but struggles so hopefully any geek out there can drop a comment...

Regards,


Wolfgang