Tuning parameters for performance of ignite Persistence store

classic Classic list List threaded Threaded
5 messages Options
rishi007bansod rishi007bansod
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Tuning parameters for performance of ignite Persistence store

Hi,
     I am trying to use persistent store for my application, w/o persistent store I am getting caching rate of about 280K msgs /sec (1 msg - 512 bytes). Whereas when I use persistent store in Ignite 2.1.0 then throughput decreases to 20 K msgs/sec. So,
1. what are tuning parameters that i can try to improve persistent store efficiency(i have tried increasing checkpoint threads, checkpoint freq but still performance is same)
2. Also in my set up I have 3 disks connected to machine, but persistent store only uses one disk for writing, so how can I improve performance in this case by utilizing all 3 disks?



As shown above only disk sda is used whereas sdb is idle all the time.

Thanks,
Rishikesh
Alexey Kukushkin Alexey Kukushkin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tuning parameters for performance of ignite Persistence store

Hi,

You could try tuning persistence store update reliability vs. performance trade-opff by setting walMode to:

DEFAULT: every update is flushed to disk (sync). Least performant but survives power loss.
LOG_ONLY: OS-managed buffered output (write). Survives process crash but not OS crash.
BACKGROUND: updates are queued in-memory and flushed every 2 seconds.

For example, queue WAL updates in memory and flush them to disk every 5 seconds:
<bean class="org.apache.ignite.configuration.PersistentStoreConfiguration">
    <property name="walMode" value="BACKGROUND"/>
    <property name="walFlushFrequency" value="5000"/>
</bean>

LOG_ONLY could be a good trade-off.

Best regards, 
Alexey



On Tuesday, August 8, 2017, 4:19:37 PM GMT+3, rishi007bansod <[hidden email]> wrote:


Hi,
    I am trying to use persistent store for my application, w/o persistent
store I am getting caching rate of about 280K msgs /sec (1 msg - 512 bytes).
Whereas when I use persistent store in Ignite 2.1.0 then throughput
decreases to 20 K msgs/sec. So,
1. what are tuning parameters that i can try to improve persistent store
efficiency(i have tried increasing checkpoint threads, checkpoint freq but
still performance is same)
2. Also in my set up I have 3 disks connected to machine, but persistent
store only uses one disk for writing, so how can I improve performance in
this case by utilizing all 3 disks?


As shown above only disk sda is used whereas sdb is idle all the time.

Thanks,
Rishikesh



--
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tuning parameters for performance of ignite Persistence store

In reply to this post by rishi007bansod
When you add persistence, latency of each individual updates gets bigger, because you update disk as well. In case you test on the same amount of parallel threads, throughput will obviously go down. However, if you increase the load, i.e. execute more operations in parallel, throughput will go back up as long as you're not running out of resources.

-Val
rishi007bansod rishi007bansod
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tuning parameters for performance of ignite Persistence store

This post was updated on .
In reply to this post by Alexey Kukushkin
Hi,
     I have tried settings you have mentioned for WAL, it improves performance in case of WAL write. But when check pointing process starts(default - after 3 mins), caching process slows down(almost stops). Is there any way by which we can write checkpoint to disk in background, so that caching throughput is improved?
     Also, though I have connected 3 disks to my machine only 1 is getting used while writing, is there any way by which I can use all 3 of them? This single disk "sda" is becoming bottleneck in this case as shown below.

 
     

Thanks,
Rishikesh
Alexey Kukushkin Alexey Kukushkin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tuning parameters for performance of ignite Persistence store

Hi,

By default Ignite runs checkpointing asynchronously in single thread. Configure number of checkpointing threads, for example:

<bean class="org.apache.ignite.configuration.PersistentStoreConfiguration">
    <property name="checkpointingThreads" value="3"/>
</bean>

Right now I am not sure why you see that "caching almost stops": as I said checkpointing is async. If you put data into a memory page being currently checkpoint, it will copy the page on write and continue both the caching and checkpointing in parallel. Thus, the only delay is copying the page but page size is 2K by default (you can customize it) and I do not think copying 2K takes noticeable time.

As for utilizing multiple disks - Ignite is about memory. You need to look for RAID implementations to achieve disk IO performance increase via partitioning. Just FYI - you can customize where Ignite stores persistence database (data file, active WAL segment and WAL archive). For example:

<bean class="org.apache.ignite.configuration.PersistentStoreConfiguration">
    <property name="persistentStorePath" value="/var/ignite-db"/>
    <property name="walStorePath" value="/var2/ignite-db-wal"/>
    <property name="walArchivePath" value="/var3/ignite-db-wal-archive"/>
</bean>

Best regards, 
Alexey

Loading...