Ignite Load Performance when memory exhausted

classic Classic list List threaded Threaded
2 messages Options
njcstreet njcstreet
Reply | Threaded
Open this post in threaded view
|

Ignite Load Performance when memory exhausted

Hi,

I am evaluating Apache Ignite for a use case where I know we may not have
enough RAM to store the entire database. For example we might store 30 days
of data, but only have enough memory for 15, and in fact most of the queries
will be against the last 5 days of data. I am doing some tests loading the
30 days of data (roughly 2 billion objects) into a test environment.

 

·         The test environment has 6 servers each with 128GB RAM, which I am
allocating roughly 70% to Ignite to allow the OS enough memory.

·         This is using Native Persistence in Ignite 2.8.1.

·         The disks I am using are not as fast as production would be (they
are not SSD) but they are still reasonably fast.

·         The object I am storing is reasonably simple, it has 25 properties
which are integers (2 of these have indexes on) and 4 which are floats, but
there are a lot of them (roughly 2 billion).

·         The cache the data is being loaded into is partitioned.

·         Data is loaded using IDataStreamer – there is a single loader
process which has 10 threads reading files, then submitting into a single
DataStreamer.

 

What I am observing is that:

 

·         Whilst the default region has RAM available, the inserts are quite
quick – even though the data is being written to disk (the WAL is enabled).
I get on average 150K objects inserted per second.

·         Once the default region runs out of RAM, I see an error message
“WARN|org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl|Page
replacements started, pages will be rotated with disk, this will affect
storage performance (consider increasing
DataRegionConfiguration#setMaxSize).”

·         After this, inserts are much slower, going down to about 19K
objects per second.

 

Now I kind of understand what is happening here is that for every new page
in memory that needs to be allocated, Ignite is having to purge another page
from memory (this is implied by the error message). But what I don’t
understand is why the performance impact is so great (150K objects / sec
when memory is available vs 19K per sec when memory is exhausted). Before
and after the region ran out of RAM, the data was being written to disk.
After the region ran out of RAM, I would expect the only difference to be
that it needs to purge some data from RAM to make space for the data that is
loading – I wouldn’t have thought there should be any more disk activity
since through Ignite Persistence the data was already being written to disk.
Maybe I have something mis-configured, or I have misunderstood something?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Load Performance when memory exhausted

Hello!

Until you got this message, you did not need to catch up with disk (except for WAL writing, maybe), but once you see it, good days are over and you need to do checkpoints with all pages which were updated recently.

There are things to tune here, but don't expect miracles.


Regards,
--
Ilya Kasnacheev


вт, 23 июн. 2020 г. в 18:25, njcstreet <[hidden email]>:
Hi,

I am evaluating Apache Ignite for a use case where I know we may not have
enough RAM to store the entire database. For example we might store 30 days
of data, but only have enough memory for 15, and in fact most of the queries
will be against the last 5 days of data. I am doing some tests loading the
30 days of data (roughly 2 billion objects) into a test environment.



·         The test environment has 6 servers each with 128GB RAM, which I am
allocating roughly 70% to Ignite to allow the OS enough memory.

·         This is using Native Persistence in Ignite 2.8.1.

·         The disks I am using are not as fast as production would be (they
are not SSD) but they are still reasonably fast.

·         The object I am storing is reasonably simple, it has 25 properties
which are integers (2 of these have indexes on) and 4 which are floats, but
there are a lot of them (roughly 2 billion).

·         The cache the data is being loaded into is partitioned.

·         Data is loaded using IDataStreamer – there is a single loader
process which has 10 threads reading files, then submitting into a single
DataStreamer.



What I am observing is that:



·         Whilst the default region has RAM available, the inserts are quite
quick – even though the data is being written to disk (the WAL is enabled).
I get on average 150K objects inserted per second.

·         Once the default region runs out of RAM, I see an error message
“WARN|org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl|Page
replacements started, pages will be rotated with disk, this will affect
storage performance (consider increasing
DataRegionConfiguration#setMaxSize).”

·         After this, inserts are much slower, going down to about 19K
objects per second.



Now I kind of understand what is happening here is that for every new page
in memory that needs to be allocated, Ignite is having to purge another page
from memory (this is implied by the error message). But what I don’t
understand is why the performance impact is so great (150K objects / sec
when memory is available vs 19K per sec when memory is exhausted). Before
and after the region ran out of RAM, the data was being written to disk.
After the region ran out of RAM, I would expect the only difference to be
that it needs to purge some data from RAM to make space for the data that is
loading – I wouldn’t have thought there should be any more disk activity
since through Ignite Persistence the data was already being written to disk.
Maybe I have something mis-configured, or I have misunderstood something?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/