Native persistence enabled: Loading the same data again takes longer than initially

classic Classic list List threaded Threaded
5 messages Options
VincentCE VincentCE
Reply | Threaded
Open this post in threaded view
|

Native persistence enabled: Loading the same data again takes longer than initially

Hi!

I made the observation that loading data initially into the ignite-cluster
with native persistence enabled is usually a lot faster than subsequent
loadings of the same data, that is 30 min (initially) vs 52 min (4th time)
for 170 GB of data.

Does this indicate bad configurations from our side or is this expected
behaviour? In fact we are quite happy with the initial loading speed of our
data but will generally need to overwrite significant parts of it (which is
the reasons for my question). I already tried to apply all the suggestion
mentioned in https://apacheignite.readme.io/docs/durable-memory-tuning.

We are using ignite 2.8.1 currently. Here is our data storage configuration:

<property name="\"dataStorageConfiguration\"">
        <bean class="\"org.apache.ignite.configuration.DataStorageConfiguration\"">
                <property name="\"walMode\"" value="\"BACKGROUND\""/>
                <property name="\"writeThrottlingEnabled\"" value="\"true\""/>
                <property name="\"pageSize\"" value="\"#{4 * 1024}\""/>
                <property name=&quot;\&quot;walSegmentSize\&quot;&quot;
value=&quot;\&quot;#{2 * 1000 * 1000 * 1000}\&quot;&quot;/>
                <property name=&quot;\&quot;maxWalArchiveSize\&quot;&quot;
value=&quot;\&quot;#{10L * 1024 * 1024 * 1024}\&quot;&quot;/>
               
                <property name="\"defaultDataRegionConfiguration\"">
                        <bean
class="\"org.apache.ignite.configuration.DataRegionConfiguration\"">
                                <property name="\"persistenceEnabled\"" value="\"true\""/>
                                <property name=&quot;\&quot;checkpointPageBufferSize\&quot;&quot;
value=&quot;\&quot;#{2L * 1024 * 1024 * 1024}\&quot;&quot;/>
                                <property name="\"name\"" value="\"Default_Region\""/>
                                <property name="\"initialSize\"" value="\"${IGNITE_DEFAULT_REGION}\""/>
                                <property name="\"maxSize\"" value="\"${IGNITE_DEFAULT_REGION}\""/>
                        </bean>
                </property>
                <property name="\"storagePath\"" value="\"/persistence\""/>
                <property name="\"walPath\"" value="\"/wal\""/>
                <property name="\"walArchivePath\"" value="\"/wal\""/>
        </bean>

Thanks in advance!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Maxim Muzafarov Maxim Muzafarov
Reply | Threaded
Open this post in threaded view
|

Re: Native persistence enabled: Loading the same data again takes longer than initially

Hello Vincent,


What kind of Ignites API you are using for initial data loading? I
always use this man page [1] for my local tests with simultaneously
disabling WALs if the Ignite native persistence enabled.


[1] https://apacheignite.readme.io/docs/data-loading

On Mon, 30 Nov 2020 at 16:55, VincentCE <[hidden email]> wrote:

>
> Hi!
>
> I made the observation that loading data initially into the ignite-cluster
> with native persistence enabled is usually a lot faster than subsequent
> loadings of the same data, that is 30 min (initially) vs 52 min (4th time)
> for 170 GB of data.
>
> Does this indicate bad configurations from our side or is this expected
> behaviour? In fact we are quite happy with the initial loading speed of our
> data but will generally need to overwrite significant parts of it (which is
> the reasons for my question). I already tried to apply all the suggestion
> mentioned in https://apacheignite.readme.io/docs/durable-memory-tuning.
>
> We are using ignite 2.8.1 currently. Here is our data storage configuration:
>
> <property name="\"dataStorageConfiguration\"">
>         <bean class="\"org.apache.ignite.configuration.DataStorageConfiguration\"">
>                 <property name="\"walMode\"" value="\"BACKGROUND\""/>
>                 <property name="\"writeThrottlingEnabled\"" value="\"true\""/>
>                 <property name="\"pageSize\"" value="\"#{4 * 1024}\""/>
>                 <property name=&quot;\&quot;walSegmentSize\&quot;&quot;
> value=&quot;\&quot;#{2 * 1000 * 1000 * 1000}\&quot;&quot;/>
>                 <property name=&quot;\&quot;maxWalArchiveSize\&quot;&quot;
> value=&quot;\&quot;#{10L * 1024 * 1024 * 1024}\&quot;&quot;/>
>
>                 <property name="\"defaultDataRegionConfiguration\"">
>                         <bean
> class="\"org.apache.ignite.configuration.DataRegionConfiguration\"">
>                                 <property name="\"persistenceEnabled\"" value="\"true\""/>
>                                 <property name=&quot;\&quot;checkpointPageBufferSize\&quot;&quot;
> value=&quot;\&quot;#{2L * 1024 * 1024 * 1024}\&quot;&quot;/>
>                                 <property name="\"name\"" value="\"Default_Region\""/>
>                                 <property name="\"initialSize\"" value="\"${IGNITE_DEFAULT_REGION}\""/>
>                                 <property name="\"maxSize\"" value="\"${IGNITE_DEFAULT_REGION}\""/>
>                         </bean>
>                 </property>
>                 <property name="\"storagePath\"" value="\"/persistence\""/>
>                 <property name="\"walPath\"" value="\"/wal\""/>
>                 <property name="\"walArchivePath\"" value="\"/wal\""/>
>         </bean>
>
> Thanks in advance!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
VincentCE VincentCE
Reply | Threaded
Open this post in threaded view
|

Re: Native persistence enabled: Loading the same data again takes longer than initially

Hi Maxim!

we are generally using IgniteDataStreamers for cache loadings. Regarding
disabling wal: We already thought about that during initial loading but our
business business flow requires several further loadings and at least during
the second load we cannot switch it off anyway. Otherwise according to my
understanding we would risk losing all data e.g. due to a node restart if
wal is turned.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Maxim Muzafarov Maxim Muzafarov
Reply | Threaded
Open this post in threaded view
|

Re: Native persistence enabled: Loading the same data again takes longer than initially

Vincent,

What type of IgniteDataStreamer you are using? allowOverwrite == true/false?
Are you loading data to SQL table or just a regular cache? Are there
any indexes exists?

Yes, disabling WAL means you have a risk of losing data while it still
remains disabled. Then the initial loading completes you have to
re-enable WAL logging. Disabling WAL can significantly increase the
loading speed.

On Mon, 30 Nov 2020 at 17:46, VincentCE <[hidden email]> wrote:

>
> Hi Maxim!
>
> we are generally using IgniteDataStreamers for cache loadings. Regarding
> disabling wal: We already thought about that during initial loading but our
> business business flow requires several further loadings and at least during
> the second load we cannot switch it off anyway. Otherwise according to my
> understanding we would risk losing all data e.g. due to a node restart if
> wal is turned.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Zhenya Stanilovsky Zhenya Stanilovsky
Reply | Threaded
Open this post in threaded view
|

Re: Native persistence enabled: Loading the same data again takes longer than initially

In reply to this post by VincentCE
VincentCE,  how match is your uploaded portions ?
If you upload more than ${IGNITE_DEFAULT_REGION} after restart you will obtain i.e. page replacements (some data pages pages will be requested from disk to memory). Plz check closely for example [1].
If uploading speed is sensible for you, just increase data region size, is possible, disable swap and over commiting [2].
Remove writeThrottlingEnabled — i believe  that default is more usable here.
 
 

From: VincentCE <ve@...>
To: user@...
Cc:
Subject: Native persistence enabled: Loading the same data again takes
longer than initially
Date: Mon, 30 Nov 2020 16:55:17 +0300

Hi!

I made the observation that loading data initially into the ignite-cluster
with native persistence enabled is usually a lot faster than subsequent
loadings of the same data, that is 30 min (initially) vs 52 min (4th time)
for 170 GB of data.

Does this indicate bad configurations from our side or is this expected
behaviour? In fact we are quite happy with the initial loading speed of our
data but will generally need to overwrite significant parts of it (which is
the reasons for my question). I already tried to apply all the suggestion
mentioned in https://apacheignite.readme.io/docs/durable-memory-tuning.

We are using ignite 2.8.1 currently. Here is our data storage
configuration:

<property name="\"dataStorageConfiguration\"">
<bean
class="\"org.apache.ignite.configuration.DataStorageConfiguration\"">
<property name="\"walMode\"" value="\"BACKGROUND\""/>
<property name="\"writeThrottlingEnabled\"" value="\"true\""/>
<property name="\"pageSize\"" value="\"#{4 * 1024}\""/>
<property name=&quot;\&quot;walSegmentSize\&quot;&quot;
value=&quot;\&quot;#{2 * 1000 * 1000 * 1000}\&quot;&quot;/>
<property name=&quot;\&quot;maxWalArchiveSize\&quot;&quot;
value=&quot;\&quot;#{10L * 1024 * 1024 * 1024}\&quot;&quot;/>

<property name="\"defaultDataRegionConfiguration\"">
<bean
class="\"org.apache.ignite.configuration.DataRegionConfiguration\"">
<property name="\"persistenceEnabled\"" value="\"true\""/>
<property name=&quot;\&quot;checkpointPageBufferSize\&quot;&quot;
value=&quot;\&quot;#{2L * 1024 * 1024 * 1024}\&quot;&quot;/>
<property name="\"name\"" value="\"Default_Region\""/>
<property name="\"initialSize\"" value="\"${IGNITE_DEFAULT_REGION}\""/>
<property name="\"maxSize\"" value="\"${IGNITE_DEFAULT_REGION}\""/>
</bean>
</property>
<property name="\"storagePath\"" value="\"/persistence\""/>
<property name="\"walPath\"" value="\"/wal\""/>
<property name="\"walArchivePath\"" value="\"/wal\""/>
</bean>

Thanks in advance!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/