streaming behaviour with streamingPerNodeBufferSize parameter

classic Classic list List threaded Threaded
3 messages Options
shivakumar shivakumar
Reply | Threaded
Open this post in threaded view
|

streaming behaviour with streamingPerNodeBufferSize parameter

Hi
I'm trying to do SQL stream to load data to ignite cluster from my program
(insert sql with streaming set to ON)
My program connects to ignite over JDBC using
org.apache.ignite.IgniteJdbcThinDriver driver (Thin driver)
when i start my application with very small records rate then i don't see
records immediately inserted into my Ignite table but I'm able to see after
inserting few records(>10K records) are pushed or when i close the JDBC
connection.
when I searched about streaming in Ignite document, it says there is a
parameter streamingPerNodeBufferSize with default value set to 1024.
Is this parameter applicable in my scenario with JdbcThinDriver ?
If it is applicable what is 1024 ? Is it 1024 KB(1MB)  ??

Ref:
https://apacheignite-sql.readme.io/v2.5/docs/jdbc-client-driver#streaming-mode


Regards,
Shiva



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: streaming behaviour with streamingPerNodeBufferSize parameter

Hello!

Please check out https://ignite.apache.org/docs/latest/data-streaming and IgniteDataStreamer documentation for the meaning of those settings.

Regards,
--
Ilya Kasnacheev


чт, 10 дек. 2020 г. в 21:44, shivakumar <[hidden email]>:
Hi
I'm trying to do SQL stream to load data to ignite cluster from my program
(insert sql with streaming set to ON)
My program connects to ignite over JDBC using
org.apache.ignite.IgniteJdbcThinDriver driver (Thin driver)
when i start my application with very small records rate then i don't see
records immediately inserted into my Ignite table but I'm able to see after
inserting few records(>10K records) are pushed or when i close the JDBC
connection.
when I searched about streaming in Ignite document, it says there is a
parameter streamingPerNodeBufferSize with default value set to 1024.
Is this parameter applicable in my scenario with JdbcThinDriver ?
If it is applicable what is 1024 ? Is it 1024 KB(1MB)  ??

Ref:
https://apacheignite-sql.readme.io/v2.5/docs/jdbc-client-driver#streaming-mode


Regards,
Shiva



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ibelyakov ibelyakov
Reply | Threaded
Open this post in threaded view
|

Re: streaming behaviour with streamingPerNodeBufferSize parameter

Hi,

Yes, it's applicable. 1024 is a number of records in datastreamer buffer.
More details here:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-IgniteDataStreamer-question-about-units-valid-ranges-for-perNodeBufferSize-autoFlushFrequency-td5431.html

Regards,
Igor

On Fri, Dec 11, 2020 at 6:48 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Please check out https://ignite.apache.org/docs/latest/data-streaming and IgniteDataStreamer documentation for the meaning of those settings.

Regards,
--
Ilya Kasnacheev


чт, 10 дек. 2020 г. в 21:44, shivakumar <[hidden email]>:
Hi
I'm trying to do SQL stream to load data to ignite cluster from my program
(insert sql with streaming set to ON)
My program connects to ignite over JDBC using
org.apache.ignite.IgniteJdbcThinDriver driver (Thin driver)
when i start my application with very small records rate then i don't see
records immediately inserted into my Ignite table but I'm able to see after
inserting few records(>10K records) are pushed or when i close the JDBC
connection.
when I searched about streaming in Ignite document, it says there is a
parameter streamingPerNodeBufferSize with default value set to 1024.
Is this parameter applicable in my scenario with JdbcThinDriver ?
If it is applicable what is 1024 ? Is it 1024 KB(1MB)  ??

Ref:
https://apacheignite-sql.readme.io/v2.5/docs/jdbc-client-driver#streaming-mode


Regards,
Shiva



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/