Ignite server of perNodeParallelOperatoins ?

classic Classic list List threaded Threaded
3 messages Options
java148 java148
Reply | Threaded
Open this post in threaded view
|

Ignite server of perNodeParallelOperatoins ?

Hello,

In IgniteDataStreamer, there is a config: perNodeParallelOperatoins
(int), it is configured in the client side. in the Server side, does it
have similar configuration ? otherwise,  if client has freedom to set
any number of perNodeParallelOperatoins they want,  how server prevent
not crash ?

Thanks. Ed


akorensh akorensh
Reply | Threaded
Open this post in threaded view
|

Re: Ignite server of perNodeParallelOperatoins ?

This post was updated on .
Hi,
   A dataStreamer collects entries into a buffer, and when it is full, sends
the buffer over to a server node. see: perNodeBufferSize(int)
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html#perNodeBufferSize-int-

   perNodeParallelOperations is a client side setting that determines how
many buffers to send to a given server node without waiting for an
acknowledgement. The server node will try to process and acknowledge as many
buffers as it is given. perNodeParallelOperations is a throttling parameter
to make sure receiver nodes are not overloaded.

   IgniteDataStreamer.addData will block when the limit specified
perNodeParallelOperations is reached.

see:
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html#perNodeParallelOperations-int-


from the doc:
  perNodeParallelOperations(int) - sometimes data may be added to the data
streamer via addData(Object, Object) method faster than it can be put in
cache. In this case, new buffered stream messages are sent to remote nodes
before responses from previous ones are received. This could cause unlimited
heap memory utilization growth on local and remote nodes. To control memory
utilization, this setting limits maximum allowed number of parallel buffered
stream messages that are being processed on remote nodes. If this number is
exceeded, then addData(Object, Object) method will block to control memory
utilization. Default is equal to CPU count on remote node multiply by
DFLT_PARALLEL_OPS_MULTIPLIER.

see:
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html

Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Ignite server of perNodeParallelOperatoins ?

In reply to this post by java148
Hello!

Yes, there is Data Streamer Thread Pool Size on server, which corresponds to perNodeParallelOperations.

Server will not run more parallel operations than thread pool size.

Regards,
--
Ilya Kasnacheev


вт, 7 июл. 2020 г. в 23:18, Edward Chen <[hidden email]>:
Hello,

In IgniteDataStreamer, there is a config: perNodeParallelOperatoins
(int), it is configured in the client side. in the Server side, does it
have similar configuration ? otherwise,  if client has freedom to set
any number of perNodeParallelOperatoins they want,  how server prevent
not crash ?

Thanks. Ed