DataStreamer addData takes lot of time after 500 million writes

classic Classic list List threaded Threaded
3 messages Options
krkumar24061975@gmail.com krkumar24061975@gmail.com
Reply | Threaded
Open this post in threaded view
|

DataStreamer addData takes lot of time after 500 million writes

Hi all  - Why data streamer take lot of time randomly after 500+ million writes, it frequently and very consistently takes lot of time to finish the writes and to the extent of 25 to 45 seconds for write. May be its flushing the data as i have flush frequency set but why not in the beginning and why only in the end. And also I see heap size going up after sometime and trend is consistently upwards. 

Here is the streamer configuration: 

dataStreamer.autoFlushFrequency(10000); 
dataStreamer.perNodeBufferSize(32 * 1024); 
dataStreamer.perNodeParallelOperations(32); 

Not sure if this is of any use but here is the dataStorageConfiguration 

                                <property name="dataStorageConfiguration">
                                        <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
                                                <property name="defaultDataRegionConfiguration">
                                                        <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                                                                <property name="initialSize" value="#{512L *  1024 * 1024}"></property>
                                                                <property name="maxSize" value="#{4L * 1024 * 1024 * 1024}" />
                                                                <property name="persistenceEnabled" value="true" />
                                                                <property name="checkpointPageBufferSize" value="#{2L *1024 * 1024 * 1024}" />
                                                        </bean>
                                                </property>
                                                <property name="pageSize" value="#{4 * 1024}" />
                                                <property name="storagePath" value="${grid.data}" />
                                                <property name="walPath" value="${grid.wal}" />
                                                <property name="walArchivePath" value="${grid.wal}/archive" />
                                                <property name="walMode" value="BACKGROUND" />
                                                <property name="walFlushFrequency" value="5000"></property>
                                        </bean>
                                </property>
Ivan Pavlukhin Ivan Pavlukhin
Reply | Threaded
Open this post in threaded view
|

Re: DataStreamer addData takes lot of time after 500 million writes

Hi,

Is the problem specific to DataStreamer? Will the problem disappear if
insert the same data using IgniteCache.put (or putAll batches)?

вс, 15 сент. 2019 г. в 15:49, KR Kumar <[hidden email]>:

>
> Hi all  - Why data streamer take lot of time randomly after 500+ million writes, it frequently and very consistently takes lot of time to finish the writes and to the extent of 25 to 45 seconds for write. May be its flushing the data as i have flush frequency set but why not in the beginning and why only in the end. And also I see heap size going up after sometime and trend is consistently upwards.
>
> Here is the streamer configuration:
>
> dataStreamer.autoFlushFrequency(10000);
> dataStreamer.perNodeBufferSize(32 * 1024);
> dataStreamer.perNodeParallelOperations(32);
>
> Not sure if this is of any use but here is the dataStorageConfiguration
>
>                                 <property name="dataStorageConfiguration">
>                                         <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
>                                                 <property name="defaultDataRegionConfiguration">
>                                                         <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
>                                                                 <property name="initialSize" value="#{512L *  1024 * 1024}"></property>
>                                                                 <property name="maxSize" value="#{4L * 1024 * 1024 * 1024}" />
>                                                                 <property name="persistenceEnabled" value="true" />
>                                                                 <property name="checkpointPageBufferSize" value="#{2L *1024 * 1024 * 1024}" />
>                                                         </bean>
>                                                 </property>
>                                                 <property name="pageSize" value="#{4 * 1024}" />
>                                                 <property name="storagePath" value="${grid.data}" />
>                                                 <property name="walPath" value="${grid.wal}" />
>                                                 <property name="walArchivePath" value="${grid.wal}/archive" />
>                                                 <property name="walMode" value="BACKGROUND" />
>                                                 <property name="walFlushFrequency" value="5000"></property>
>                                         </bean>
>                                 </property>



--
Best regards,
Ivan Pavlukhin
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: DataStreamer addData takes lot of time after 500 million writes

In reply to this post by krkumar24061975@gmail.com
Hello!

This would happen if you have any non-inlined indexes. You need to load a lot of pages for every write and you are running out of offheap.

4G is a tiny offheap.

Regards,
--
Ilya Kasnacheev


вс, 15 сент. 2019 г. в 15:49, KR Kumar <[hidden email]>:
Hi all  - Why data streamer take lot of time randomly after 500+ million writes, it frequently and very consistently takes lot of time to finish the writes and to the extent of 25 to 45 seconds for write. May be its flushing the data as i have flush frequency set but why not in the beginning and why only in the end. And also I see heap size going up after sometime and trend is consistently upwards. 

Here is the streamer configuration: 

dataStreamer.autoFlushFrequency(10000); 
dataStreamer.perNodeBufferSize(32 * 1024); 
dataStreamer.perNodeParallelOperations(32); 

Not sure if this is of any use but here is the dataStorageConfiguration 

                                <property name="dataStorageConfiguration">
                                        <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
                                                <property name="defaultDataRegionConfiguration">
                                                        <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                                                                <property name="initialSize" value="#{512L *  1024 * 1024}"></property>
                                                                <property name="maxSize" value="#{4L * 1024 * 1024 * 1024}" />
                                                                <property name="persistenceEnabled" value="true" />
                                                                <property name="checkpointPageBufferSize" value="#{2L *1024 * 1024 * 1024}" />
                                                        </bean>
                                                </property>
                                                <property name="pageSize" value="#{4 * 1024}" />
                                                <property name="storagePath" value="${grid.data}" />
                                                <property name="walPath" value="${grid.wal}" />
                                                <property name="walArchivePath" value="${grid.wal}/archive" />
                                                <property name="walMode" value="BACKGROUND" />
                                                <property name="walFlushFrequency" value="5000"></property>
                                        </bean>
                                </property>