DataStreamer. allowOverwrite(false) - Will it slow the writes?

classic Classic list List threaded Threaded
4 messages Options
krkumar24061975@gmail.com krkumar24061975@gmail.com
Reply | Threaded
Open this post in threaded view
|

DataStreamer. allowOverwrite(false) - Will it slow the writes?

Hi Guys - If I have my data streamer's allowOverwrite is set to false which
is default, will this cause my read IOPS go up as it need to check if the
key already exist? Bcoz the behavior that we see is that the write
performance goes down significantly after it has inserted few billion rows
and around the same time READ IOPS goes up. We are just curious if
allowOverwrite(false) is causing the read IOPS go up?


Thanx and Regards,
KR Kumar



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: DataStreamer. allowOverwrite(false) - Will it slow the writes?

Hello!

I don't think so. It is supposed to speed it up. Do you have persistence? With persistence, you would expect a slow-down after an initial spike.

Regards,
--
Ilya Kasnacheev


пт, 10 июл. 2020 г. в 15:09, [hidden email] <[hidden email]>:
Hi Guys - If I have my data streamer's allowOverwrite is set to false which
is default, will this cause my read IOPS go up as it need to check if the
key already exist? Bcoz the behavior that we see is that the write
performance goes down significantly after it has inserted few billion rows
and around the same time READ IOPS goes up. We are just curious if
allowOverwrite(false) is causing the read IOPS go up?


Thanx and Regards,
KR Kumar



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
krkumar24061975@gmail.com krkumar24061975@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: DataStreamer. allowOverwrite(false) - Will it slow the writes?

This post was updated on .
Hi - Yes, Ignite native persistence is enabled.

Also does inline indexes consume the heap memory or the memory that's part of data region?

Another thought process is since we have inline indexes on tables and these inline indexes consume memory. Once the size of the indexes is more than the memory available, does it passivate some of the indexes to disk to accommodate the new indexes created and subsequently load the indexes from disk when those keys need to be updated. This will result in constant activation and passivation of indexes from and to the disk which could be contributing to the read IOPS. Please validate this observation and let us know what you think. If this is the case, we could consider increasing the heap or data region?


Thanx and Regards,
KR Kumar



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: DataStreamer. allowOverwrite(false) - Will it slow the writes?

Hello!

With persistence, you will probably burn through the burst time (in-memory no-checkpoint) faster and hit the "slow" persistent epoch earlier.

Then you will need to tune your persistence.


Regards,
--
Ilya Kasnacheev


пт, 10 июл. 2020 г. в 19:07, [hidden email] <[hidden email]>:
Hi - Yes, Ignite native persistence is enabled.

Thanx and Regards,
KR Kumar



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/