Using Ignite Native Persistence as a "temporary durable" cache

classic Classic list List threaded Threaded
3 messages Options
farhad_jabiyev farhad_jabiyev
Reply | Threaded
Open this post in threaded view
|

Using Ignite Native Persistence as a "temporary durable" cache

Hi all,

We have MS SQL database server which contains all data. Our application will fetch some datas from database server and put them to the cache. And then, during 5-10 seconds we will do some updates to that objects and push that changes to ignite in-memory cache. And then, after 5-10 seconds we will take that changes and sync them will database as a bulk operation and then will clear that permanent storage. So, we need some permanent storage to store those updates for 5-10 seconds.
And actually, we will work at most with 100.000 entity.

The idea behind that flow is that we can't now scale DB and users already putting load to the database.

We can't decide whether we have to use Native Persistent or some another 3rd party database like Maria or PostgreSQL for storing that cache operations for 5-10 seconds.

Will Ignite works fast if we will use native persistent and clear the cache periodically?
akorensh akorensh
Reply | Threaded
Open this post in threaded view
|

Re: Using Ignite Native Persistence as a "temporary durable" cache

Hello,
   Ignite native persistence has a good track record, is fast and reliable, you can use it in your application.
   
   Take a look at the following example of how to use a streamer to quickly insert data: https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/persistentstore/PersistentStoreExample.java
   
   Use cache.removeAll() -- to quickly remove all entries once you are done.
Thanks, Alex

On Thu, Sep 5, 2019 at 10:23 AM Farhad Jabiyev <[hidden email]> wrote:
Hi all,

We have MS SQL database server which contains all data. Our application will fetch some datas from database server and put them to the cache. And then, during 5-10 seconds we will do some updates to that objects and push that changes to ignite in-memory cache. And then, after 5-10 seconds we will take that changes and sync them will database as a bulk operation and then will clear that permanent storage. So, we need some permanent storage to store those updates for 5-10 seconds.
And actually, we will work at most with 100.000 entity.

The idea behind that flow is that we can't now scale DB and users already putting load to the database.

We can't decide whether we have to use Native Persistent or some another 3rd party database like Maria or PostgreSQL for storing that cache operations for 5-10 seconds.

Will Ignite works fast if we will use native persistent and clear the cache periodically?
akorensh akorensh
Reply | Threaded
Open this post in threaded view
|

Re: Using Ignite Native Persistence as a "temporary durable" cache

origonal question:
[hidden email]
5:00 AM (5 hours ago)
 to me


 Thanks for answer. But, how Ignite will store that datas to store? Will append to the end of WAL? If, then it is sure faster than storing cache operations in some 3rd part database. 


Ignite does store data to WAL first, but per my benchmarks, storing 100,000 records took a few seconds, and deleting took a few seconds more.

 You can experiment using :


Here is sample code: (modified section in above example)
image.png

On Thu, Sep 5, 2019 at 12:17 PM Alexander Korenshteyn <[hidden email]> wrote:
Hello,
   Ignite native persistence has a good track record, is fast and reliable, you can use it in your application.
   
   Take a look at the following example of how to use a streamer to quickly insert data: https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/persistentstore/PersistentStoreExample.java
   
   Use cache.removeAll() -- to quickly remove all entries once you are done.
Thanks, Alex

On Thu, Sep 5, 2019 at 10:23 AM Farhad Jabiyev <[hidden email]> wrote:
Hi all,

We have MS SQL database server which contains all data. Our application will fetch some datas from database server and put them to the cache. And then, during 5-10 seconds we will do some updates to that objects and push that changes to ignite in-memory cache. And then, after 5-10 seconds we will take that changes and sync them will database as a bulk operation and then will clear that permanent storage. So, we need some permanent storage to store those updates for 5-10 seconds.
And actually, we will work at most with 100.000 entity.

The idea behind that flow is that we can't now scale DB and users already putting load to the database.

We can't decide whether we have to use Native Persistent or some another 3rd party database like Maria or PostgreSQL for storing that cache operations for 5-10 seconds.

Will Ignite works fast if we will use native persistent and clear the cache periodically?