Automatically Persisted Queue

classic Classic list List threaded Threaded
4 messages Options
Isaeed Mohanna Isaeed Mohanna
Reply | Threaded
Open this post in threaded view
|

Automatically Persisted Queue

Hi
I am using Ignite as a compute engine, one of my tasks will process incoming data. Therefore I need a reliable persisted store for my new data until the task process them, I could use ignite cache or use ignite streamer however both these are in memory and my cluster is not kept alive always but will be completely shutdown for upgrades once a week at least.
I know its possible to define persistence for a cache into a database however i do not want to introduce a new database in my landscape so i am looking for a simple approach to persist my incoming data into disk.

1. Is there a way to configure Ignite cache to be automatically stored into disk by ignite?
2. Any persisted queue solution that you can recommend for use with ignite?
Thanks
Regards,
Isaeed Mohanna
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Automatically Persisted Queue

Isaeed Mohanna wrote
I know its possible to define persistence for a cache into a database however i do not want to introduce a new database in my landscape so i am looking for a simple approach to persist my incoming data into disk.

1. Is there a way to configure Ignite cache to be automatically stored into disk by ignite?
2. Any persisted queue solution that you can recommend for use with ignite?
The automatic local disk persistence is not available out of the box in Ignite (to my knowledge it is available in GridGain as enterprise feature).

However, you can still use CacheStore. From Ignite standpoint, CacheStore does not have to be a database, it could be anything, including a file system, to which you wish to persist the data. My advise would be to either provide your own CacheStore implementation, or use CacheJdbcBlobStore if you don't mind plugging in any type of local JDBC database.

On startup you can use IgniteCache.loadCache() method to load the cache from database. This method will delegate to CacheStore.loadCache() for which you can provide your own logic to control the data loading routine.
Isaeed Mohanna Isaeed Mohanna
Reply | Threaded
Open this post in threaded view
|

Re: Automatically Persisted Queue

Thanks for the information,
Assume i have a in memory cache named queue, i have web calls adding items into this queue and ignite tasks processing and deleting data from this queue
If i implemented my own CacheStore or use JDBC Cachestore to SQLite or something similar, several questions:
1. is the cachestore always updated automatically with changes to ignite cache? how big is the performance penalty?
2. My Ignite cache is partitioned across the cluster of three nodes, but my cache store will be in each of the nodes? will the cache store contains the data from all of the nodes? do I need to defined a cache store per node or it can run only in one of the nodes?

Regards,
Isaeed Mohanna

On Thu, May 28, 2015 at 6:27 AM, dsetrakyan <[hidden email]> wrote:
Isaeed Mohanna wrote
> I know its possible to define persistence for a cache into a database
> however i do not want to introduce a new database in my landscape so i am
> looking for a simple approach to persist my incoming data into disk.
>
> 1. Is there a way to configure Ignite cache to be automatically stored
> into disk by ignite?
> 2. Any persisted queue solution that you can recommend for use with
> ignite?

The automatic local disk persistence is not available out of the box in
Ignite (to my knowledge it is available in GridGain as enterprise feature).

However, you can still use CacheStore. From Ignite standpoint, CacheStore
does not have to be a database, it could be anything, including a file
system, to which you wish to persist the data. My advise would be to either
provide your own CacheStore implementation, or use CacheJdbcBlobStore if you
don't mind plugging in any type of local JDBC database.

On startup you can use IgniteCache.loadCache() method to load the cache from
database. This method will delegate to CacheStore.loadCache() for which you
can provide your own logic to control the data loading routine.




-----
D.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Automatically-Persisted-Queue-tp415p420.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Automatically Persisted Queue


On Thu, May 28, 2015 at 12:03 AM, Isaeed Mohanna <[hidden email]> wrote:
1. is the cachestore always updated automatically with changes to ignite cache? how big is the performance penalty?

Generally CacheStore is synchronously updated with every cache operation, unless you configure CacheConfiguration.setWriteBehind(true). If write-behind is enabled, then Ignite will batch store operations and execute them asynchronously in bulks.
 
2. My Ignite cache is partitioned across the cluster of three nodes, but my cache store will be in each of the nodes? will the cache store contains the data from all of the nodes? do I need to defined a cache store per node or it can run only in one of the nodes?

CacheStore will be called for the data stored locally on an individual node. It will contain only the data for that node.