Ignite Write Behind performance

classic Classic list List threaded Threaded
13 messages Options
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Ignite Write Behind performance

Hello,

I am struggling with improving ignite transaction performance. However this is another problem

However if I am doing write behind and the writes are slow, I am seeing that the over all performance of teh grid drops.

Is there any configuration that I can do to make ignite performance stable even when the writes are being slow in write behind (due to slow DB)

Currently I have 10 threads in write behind mode.

What should be my system threadpool size assuming I have two caches which participate in transactions and 2 nodes in 10G network.



Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Write Behind performance

Hi,

Most likely your write-behind Threads can’t keep up with volume of data that is aggregated in the write-behind buffer and you get with the situation when from time to time writing to the DB is performed in the sync mode from system Thread(s).
 
I would suggest you playing with the following write-behind configuration to avoid the situation above:
- CacheConfiguration.setWriteBehindFlushSize - increase size of the write-behind buffer;
- CacheConfiguration.setWriteBehindFlushThreadCount;
- CacheConfiguration.setWriteBehindFlushFrequence - decrease this value triggering the flushing more frequently. Default value is 5 seconds.


Denis

> On Jun 2, 2016, at 7:23 PM, amitpa <[hidden email]> wrote:
>
> Hello,
>
> I am struggling with improving ignite transaction performance. However this
> is another problem
>
> However if I am doing write behind and the writes are slow, I am seeing that
> the over all performance of teh grid drops.
>
> Is there any configuration that I can do to make ignite performance stable
> even when the writes are being slow in write behind (due to slow DB)
>
> Currently I have 10 threads in write behind mode.
>
> What should be my system threadpool size assuming I have two caches which
> participate in transactions and 2 nodes in 10G network.
>
>
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Write Behind performance

Danis,

Thanks. I am trying that.

However shouldnt the write behinds not impact the GRID performance at all.
Since we are writing in write behin dmode to avoid the cost of slow disk IO.
I understand that theres a limit, but shouldnt this thread pool be different if technically possible, allowing ignite to continue processing in case of slow write behind  performance.
Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Write Behind performance

As I explained in the situation when the write-behind queue is full a Thread that updates a cache and propagates the changes into the write-behind store can be used to flush the write-behind queue since the queue is full. This situation can affect the performance especially if this happens too often.


Denis

> On Jun 3, 2016, at 11:30 AM, amitpa <[hidden email]> wrote:
>
> Danis,
>
> Thanks. I am trying that.
>
> However shouldnt the write behinds not impact the GRID performance at all.
> Since we are writing in write behin dmode to avoid the cost of slow disk IO.
> I understand that theres a limit, but shouldnt this thread pool be different
> if technically possible, allowing ignite to continue processing in case of
> slow write behind  performance.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5400.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Write Behind performance

I understood this perfectly.

What I mean is :-

Shouldnt the write behind use a separete thread pool so that slow write behind process does not impact the whole grid ?
alexey.goncharuk alexey.goncharuk
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Write Behind performance

Ignite _does_ use a separate thread pool to persist data in write-behind mode, this is the essential difference between write-through and write-behind. However, if your cache load rate is significantly higher than the database insert rate, the write queue will grow faster than background threads can flush data, so eventually you will get an OOME.

In this situation you can
 - block and wait until there is some space in the write queue
 - persist your own data synchronously
Ignite choses the second option and writes transaction data synchronously.

As Denis pointer out, you should tune write-behind parameters to make sure that background threads are working at sufficient insert rate. Also make sure that you use batched inserts in your CacheStore implementation.

Hope this helps,
AG


2016-06-03 10:26 GMT-07:00 amitpa <[hidden email]>:
I understood this perfectly.

What I mean is :-

Shouldnt the write behind use a separete thread pool so that slow write
behind process does not impact the whole grid ?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5412.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Write Behind performance

Thanks we are considering implementing JDBC batch inserts
bintisepaha bintisepaha
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Write Behind performance

amitpa, We would be interested in learning how did this perform for you?
We have implemented spring txn to insert in database for write-behind. Hoever, we see that sometiems write-behind is not even called for some objects that we are certain were just updated in the cache. have you noticed that?
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Write Behind performance

I did test this...For us I think Write behind gets called fine.
bintisepaha bintisepaha
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Write Behind performance

Do you mind sharing your cache configuration for write behind? do you use flush size or flush frequency?
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Write Behind performance

We use both. Which ever is faster gets called first ..
bintisepaha bintisepaha
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Write Behind performance

Thanks for that. we were setting flushsize to 0. how about flush thread count?
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Write Behind performance

We use it to conservative 2 now in our current setting