Inserts stalled by write-behind process

classic Classic list List threaded Threaded
4 messages Options
bsmets bsmets
Reply | Threaded
Open this post in threaded view
|

Inserts stalled by write-behind process

We are currently using an ignite setup with caches with very large offheap sizes (10GB+) and in these caches we are inserting entries of roughly 30 kB. We flush the entries to disk every minute using write-behind.

What we observe is that after a while, the inserts of new entries into the cache seem to become synchronized with the writing of entries to disk. Considering our disk is much slower than writing in memory, this is effectively throttling our inserts.

We noticed that the WriteBehindTotalCriticalOverflowCount in the metrics is pretty high as well, which leads us to believe that inserts are waiting for entries to be written to disk.

Is this correct, and if so, how can we increase the write-behind backlog?
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Inserts stalled by write-behind process

Hi,

You're right, write-behind store has a back pressure mechanism that starts to update DB synchronously if the queue is too long. Otherwise you will most likely eventually get out of memory error.

The backlog size is controlled by CacheConfiguration.setWriteBehindFlushSize() property. When this size is reached, the store will start flushing to the database in the background. If it will detect that the flushing process is slower than cache updates (i.e., the backlog continuous growing above the setting), it will switch to sync updates.

Makes sense?

-Val
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Inserts stalled by write-behind process

Agree with Valentin. One other thing you may wish to check is whether you are doing a bulk insert to the database, or multiple individual inserts. Bulk insert should perform better, of course.

D.

On Tue, Mar 8, 2016 at 4:50 PM, vkulichenko <[hidden email]> wrote:
Hi,

You're right, write-behind store has a back pressure mechanism that starts
to update DB synchronously if the queue is too long. Otherwise you will most
likely eventually get out of memory error.

The backlog size is controlled by
CacheConfiguration.setWriteBehindFlushSize() property. When this size is
reached, the store will start flushing to the database in the background. If
it will detect that the flushing process is slower than cache updates (i.e.,
the backlog continuous growing above the setting), it will switch to sync
updates.

Makes sense?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Inserts-stalled-by-write-behind-process-tp3390p3399.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

bintisepaha bintisepaha
Reply | Threaded
Open this post in threaded view
|

Re: Inserts stalled by write-behind process

Where do you see this property?
WriteBehindTotalCriticalOverflowCount