Using data streamer for local entries update

classic Classic list List threaded Threaded
5 messages Options
djm132 djm132
Reply | Threaded
Open this post in threaded view
|

Using data streamer for local entries update

Hi,

Is it safe to parallel update local node entries with data streamer ? Here
is the benchmark code, it seems that addData() much faster than individual
put().


val res = Node.ignite.compute(project.indexNodes).broadcast<Long> {
  var n = 0L
  Node.ignite.dataStreamer<String, IndexRecord>(project.index.name).use { ds
->
    ds.allowOverwrite(true)
    project.index.localEntries(CachePeekMode.PRIMARY).forEach { it ->
      it.value.status = IndexRecordStatus.COMPLETED
      ds.addData(it.value.reqKey, it.value)
      n++
    }
  }

  println("Updated $n records")
  n
}




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
djm132 djm132
Reply | Threaded
Open this post in threaded view
|

Re: Using data streamer for local entries update

Backup count = 1 and wal mode = background.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Using data streamer for local entries update

Hello!

Yes, it should be perfectly safe. Data Streamer does batching, thus the speed up. Please note that Data Streamer batches may lock keys, which means you should be careful with doing batch operations in parallel to avoid deadlock.

Regards,
--
Ilya Kasnacheev


пт, 10 янв. 2020 г. в 01:15, djm132 <[hidden email]>:
Backup count = 1 and wal mode = background.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
djm132 djm132
Reply | Threaded
Open this post in threaded view
|

Re: Using data streamer for local entries update

Is localEntries(CachePeekMode.PRIMARY) guarantees that keys will not overlap
for partitioned cache?

Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Using data streamer for local entries update

Hello!

If no rebalance is going on, then the answer is yes.

Regards,
--
Ilya Kasnacheev


пт, 10 янв. 2020 г. в 13:35, djm132 <[hidden email]>:
Is localEntries(CachePeekMode.PRIMARY) guarantees that keys will not overlap
for partitioned cache?

Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/