SSD writes get slower and slower

classic Classic list List threaded Threaded
5 messages Options
VT VT
Reply | Threaded
Open this post in threaded view
|

SSD writes get slower and slower

Hello,

I am trying to test use SSD. But as more data are written, the write
performance drops significantly. Is there something I can do to solve this
problem? Or are there any known issues to this problem that might be fixed
in the upcoming versions? The test was done on a single node, locally. The
total data size is small, not enough to fill half of SSD.

Thanks!

For example,

Loaded 10000 keys 239ms.
Loaded 20000 keys 122ms.
Loaded 30000 keys 141ms.
Loaded 40000 keys 110ms.

...

Loaded 8420000 keys 8068ms.
Loaded 8430000 keys 201ms.
Loaded 8440000 keys 187ms.
Loaded 8450000 keys 162ms.
Loaded 8460000 keys 182ms.
Loaded 8470000 keys 164ms.
Loaded 8480000 keys 171ms.
Loaded 8490000 keys 8475ms.

...

Loaded 84370000 keys 5684ms.
Loaded 84380000 keys 4329ms.
Loaded 84390000 keys 5191ms.
Loaded 84400000 keys 7352ms.
Loaded 84410000 keys 11171ms.

...

Loaded 99940000 keys 13352ms.
Loaded 99950000 keys 28771ms.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

RE: SSD writes get slower and slower

Hi,

 

Just to confirm – you’re talking about Ignite’s native persistence, right?

 

Can you share the code and configs you use in your testing?

 

Thanks,

Stan

 

From: [hidden email]
Sent: 23 февраля 2018 г. 12:02
To: [hidden email]
Subject: SSD writes get slower and slower

 

Hello,

 

I am trying to test use SSD. But as more data are written, the write

performance drops significantly. Is there something I can do to solve this

problem? Or are there any known issues to this problem that might be fixed

in the upcoming versions? The test was done on a single node, locally. The

total data size is small, not enough to fill half of SSD.

 

Thanks!

 

For example,

 

Loaded 10000 keys 239ms.

Loaded 20000 keys 122ms.

Loaded 30000 keys 141ms.

Loaded 40000 keys 110ms.

 

...

 

Loaded 8420000 keys 8068ms.

Loaded 8430000 keys 201ms.

Loaded 8440000 keys 187ms.

Loaded 8450000 keys 162ms.

Loaded 8460000 keys 182ms.

Loaded 8470000 keys 164ms.

Loaded 8480000 keys 171ms.

Loaded 8490000 keys 8475ms.

 

...

 

Loaded 84370000 keys 5684ms.

Loaded 84380000 keys 4329ms.

Loaded 84390000 keys 5191ms.

Loaded 84400000 keys 7352ms.

Loaded 84410000 keys 11171ms.

 

...

 

Loaded 99940000 keys 13352ms.

Loaded 99950000 keys 28771ms.

 

 

 

 

--

Sent from: http://apache-ignite-users.70518.x6.nabble.com/

 

VT VT
Reply | Threaded
Open this post in threaded view
|

RE: SSD writes get slower and slower

Hi Stan,

The setting is very simple and straightforward, as follows.

DataStorageConfiguration dsCfg = new DataStorageConfiguration();
dsCfg.setWalMode(WALMode.DEFAULT);
dsCfg.setPageSize(4 * 1024);
dsCfg.setWriteThrottlingEnabled(true);

..

DataRegionConfiguration regionCfg1 = new DataRegionConfiguration();
regionCfg1.setName("region_1");
regionCfg1.setInitialSize(2L * 1024 * 1024 * 1024);
regionCfg1.setMaxSize(10L * 1024 * 1024 * 1024);
regionCfg1.setCheckpointPageBufferSize(2L * 1024 * 1024 * 1024);
regionCfg1.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
regionCfg1.setPersistenceEnabled(true);
regionCfg1.setSwapPath("/data/ignite2/swap/"); //SSD

...

cacheCfg.setDataRegionName("region_1");
cacheCfg.setName(CacheName);
cacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_ASYNC);
cacheCfg.setCacheMode(CacheMode.PARTITIONED);
cacheCfg.setCopyOnRead(false);

...

I used DataStreammer very simply, like the following.

IgniteDataStreamer stmr.addData(key, value);

I have tried multiple settings such as perNodeBufferSize,
perNodeParallelOperations. Still very slow. Please help. Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Dmitriy Govorukhin Dmitriy Govorukhin
Reply | Threaded
Open this post in threaded view
|

Re: SSD writes get slower and slower

Hi,

I guess the problem in "setSwapPath(...)", it is not the path for
persistence.

Try to do something like this:

storePath="/data/ignite2/swap/"

cfg.setDataStorageConfiguration(
     new DataStorageConfiguration()
         .setWriteThrottlingEnabled(true)
         .setPageSize(4 * 1024)
         .setStoragePath(storePath)
         .setWalPath(storePath + "/wal")
         .setWalArchivePath(storePath + "/archive")
         .setDefaultDataRegionConfiguration(
             new DataRegionConfiguration()
                 .setPersistenceEnabled(true)
                 .setInitialSize(2L * 1024 * 1024 * 1024)
                 .setMaxSize(10L * 1024 * 1024 * 1024)
         )
);

Are you sure what you need so strong guaranteed? (WALMode.DEFAULT)
The WALMode.DEFAULT mode provides the strictness of the protection, but
it is the slowest onе.
Use LOG_ONLY mode for data loading. More info about wal mode see:
https://apacheignite.readme.io/docs/write-ahead-log



On 25.02.2018 12:14, VT wrote:

> Hi Stan,
>
> The setting is very simple and straightforward, as follows.
>
> DataStorageConfiguration dsCfg = new DataStorageConfiguration();
> dsCfg.setWalMode(WALMode.DEFAULT);
> dsCfg.setPageSize(4 * 1024);
> dsCfg.setWriteThrottlingEnabled(true);
>
> ..
>
> DataRegionConfiguration regionCfg1 = new DataRegionConfiguration();
> regionCfg1.setName("region_1");
> regionCfg1.setInitialSize(2L * 1024 * 1024 * 1024);
> regionCfg1.setMaxSize(10L * 1024 * 1024 * 1024);
> regionCfg1.setCheckpointPageBufferSize(2L * 1024 * 1024 * 1024);
> regionCfg1.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
> regionCfg1.setPersistenceEnabled(true);
> regionCfg1.setSwapPath("/data/ignite2/swap/"); //SSD
>
> ...
>
> cacheCfg.setDataRegionName("region_1");
> cacheCfg.setName(CacheName);
> cacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_ASYNC);
> cacheCfg.setCacheMode(CacheMode.PARTITIONED);
> cacheCfg.setCopyOnRead(false);
>
> ...
>
> I used DataStreammer very simply, like the following.
>
> IgniteDataStreamer stmr.addData(key, value);
>
> I have tried multiple settings such as perNodeBufferSize,
> perNodeParallelOperations. Still very slow. Please help. Thanks!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: SSD writes get slower and slower

I also think that switching to LOG_ONLY mode should be good enough.

On Mon, Feb 26, 2018 at 6:58 AM, dmitriy.govorukhin <[hidden email]> wrote:
Hi,

I guess the problem in "setSwapPath(...)", it is not the path for persistence.

Try to do something like this:

storePath="/data/ignite2/swap/"

cfg.setDataStorageConfiguration(
    new DataStorageConfiguration()
        .setWriteThrottlingEnabled(true)
        .setPageSize(4 * 1024)
        .setStoragePath(storePath)
        .setWalPath(storePath + "/wal")
        .setWalArchivePath(storePath + "/archive")
        .setDefaultDataRegionConfiguration(
            new DataRegionConfiguration()
                .setPersistenceEnabled(true)
                .setInitialSize(2L * 1024 * 1024 * 1024)
                .setMaxSize(10L * 1024 * 1024 * 1024)
        )
);

Are you sure what you need so strong guaranteed? (WALMode.DEFAULT)
The WALMode.DEFAULT mode provides the strictness of the protection, but it is the slowest onе.
Use LOG_ONLY mode for data loading. More info about wal mode see: https://apacheignite.readme.io/docs/write-ahead-log




On <a href="tel:25.02.2018%2012" value="+12502201812" target="_blank">25.02.2018 12:14, VT wrote:
Hi Stan,

The setting is very simple and straightforward, as follows.

DataStorageConfiguration dsCfg = new DataStorageConfiguration();
dsCfg.setWalMode(WALMode.DEFAULT);
dsCfg.setPageSize(4 * 1024);
dsCfg.setWriteThrottlingEnabled(true);

..

DataRegionConfiguration regionCfg1 = new DataRegionConfiguration();
regionCfg1.setName("region_1");
regionCfg1.setInitialSize(2L * 1024 * 1024 * 1024);
regionCfg1.setMaxSize(10L * 1024 * 1024 * 1024);
regionCfg1.setCheckpointPageBufferSize(2L * 1024 * 1024 * 1024);
regionCfg1.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
regionCfg1.setPersistenceEnabled(true);
regionCfg1.setSwapPath("/data/ignite2/swap/"); //SSD

...

cacheCfg.setDataRegionName("region_1");
cacheCfg.setName(CacheName);
cacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_ASYNC);
cacheCfg.setCacheMode(CacheMode.PARTITIONED);
cacheCfg.setCopyOnRead(false);

...

I used DataStreammer very simply, like the following.

IgniteDataStreamer stmr.addData(key, value);

I have tried multiple settings such as perNodeBufferSize,
perNodeParallelOperations. Still very slow. Please help. Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/