DataStreamer transactions question

classic Classic list List threaded Threaded
3 messages Options
hueb1 hueb1
Reply | Threaded
Open this post in threaded view
|

DataStreamer transactions question

Is there a way to make the DataStreamer addData(Map..) transactional?  For example, let's say I have a compute task that must insert its results into a distributed cache.  However, if the node the compute task is running on dies during the middle of the addData call, I don't want it to have partially copied a subset of the key/value pair Map.  Reason for this is to avoid duplicate entries when another node picks up the failed node's work and reinserts the same key entries.  I know I can turn on allowOverwrite to eliminate duplicates, but I've noticed this affects performance too much.  Any recommendations would be appreciated thanks!
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: DataStreamer transactions question

Hi,

Data streamer is not transactional by nature, so if you want to put several values in one transaction, you should use putAll operation on a transactional cache.

Actually I'm not sure I understand what you mean by "duplicates". If you put the same key-value pair twice, you get one entry in cache. Can you clarify the issue?

-Val
hueb1 hueb1
Reply | Threaded
Open this post in threaded view
|

Re: DataStreamer transactions question

Hi Val,

Thank you again for your quick reply as always
You're right, I think I mis-read the documentation.  I believe I can keep overwrite set to false and just skip the entries that are already there during a compute reprocess task. I do expect the same values for the same keys so shouldn't be a problem. Thanks again, and btw, I am really enjoying the Ignite API, I think you guys have made a great product!.

-Eric