How to tell if a cache has been initialised?

classic Classic list List threaded Threaded
3 messages Options
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|

How to tell if a cache has been initialised?

asked by steve neal

I'm in the process of creating a distributed cache that will load a few million records from a database that is under significant load. The purpose of the cache is to allow complex queries to be run over historic data without further degrading the database performance.

So far I've created an ignite process that loads all the records from the database on startup. When I run a second instance of this process, it loads all the data from the DB again and the caches correctly don't allow the duplicate keys to be stored.

The problem I'm facing, is that when this is released to production environment, all node processes will be started simultaneously and they will all execute this large query to initialise the cache at the same time.

Can you advise me on whether there is a 'correct' way to limit which processes will perform the initial cache load?

I've considered a few options but they all feel like hacks.

Thanks.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|

Re: How to tell if a cache has been initialised?

commented by steve neal

Ok - I think I've answered my own question.

I've created a service that is instantiated as a grid-wide singleton which loads the data on startup.

In case the service gets started following another node shutdown, it checks that the cache is empty before instructing the CacheStore to load the data.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|

Re: How to tell if a cache has been initialised?

commented by dmitriy setrakyan

I think using cluster-singleton is a good solution.

You may also wish to store a flag in some distributed cache stating "Loading Is Finished". This way if a server dies before it finishes the loading, another server will still complete the loading process even if the cache.size() is greater than 0.

I am aware that other users took advantage of the Ignite's Distributed CountDownLatch for this purpose. Whenever the loading was taking place, all cluster members were waiting on the count-down-latch. When the loading is complete, the count-down happens, and all cluster members are able to proceed.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss