Cache not loading correctly

classic Classic list List threaded Threaded
9 messages Options
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Cache not loading correctly

asked by steve neal

I have a problem with a cache loader service that loads data into a cache at process start-up.

If I run a single node, everything works fine and my log message shows the exact same number of items added to the cache as the calls: ignite.jcache(cacheName).localSize(CachePeekMode.PRIMARY) and ignite.jcache(cacheName).size(CachePeekMode.PRIMARY) return.

If I then start more nodes running, then everything works as I'd expect and the full cache is balanced correctly across the cluster. The cache size remains the same overall, and the local caches show numbers indicating that the data has been shared between them correctly.

Problems appear though when I start multiple nodes at the same time. My log message confirms that the same number of records were added to the cache but ignite.jcache(cacheName).localSize(CachePeekMode.PRIMARY) and ignite.jcache(cacheName).size(CachePeekMode.PRIMARY) both return a significantly lower number. On the last run my logs show that 1,163,076 records were added to the cache, but both the above method calls show only 376,879 entries. Running ignite.jcache(cacheName).localSize(CachePeekMode.PRIMARY) on the other nodes in the cluster consistently returns 0.

Have you seen problems like this, and can you suggest a fix? This looks like a cache balancing problem during the discovery phase to me. I don't want to have to start the primary node before the others at this will complicate the deployment and will mean that one node will have to hold all the records until the other nodes are running.

Thanks, Steve.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cache not loading correctly

commented by dmitriy setrakyan

Steave, you should not have to worry when to start the nodes and in which order. If you have a reproducible example, please share it with us. It would help us realize what is happening pretty quickly.

You can use GithubGist service to share the code.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cache not loading correctly

commented by dmitriy setrakyan

Steve, are you still having this issue?

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cache not loading correctly

commented by steve neal

Yes I'm still having the same problem. I've been working on something else for the past few days but am working on this again now.

Am I using the peek modes above correctly? I can't see any detailed documentation on what these different values really mean.

Thanks, Steve.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cache not loading correctly

commented by dmitriy setrakyan

Steave, you are using the pick mode correctly.

However, it looks like you are not waiting for rebalancing to complete and catching the caches in a half-rebalanced state. Can you change the preload mode to SYNC and see if it will fix the issue.

If not, then please provide a code snippet so we can reproduce.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cache not loading correctly

commented by steve neal

Hi Dmitriy,

The cache was set to SYNC mode:

<bean class="org.apache.ignite.configuration.CacheConfiguration"
     p:name="test-cache"
     p:backups="1"
     p:cacheMode="PARTITIONED"
     p:atomicityMode="ATOMIC"
     p:preloadMode="SYNC"
     p:startSize="3000000">

I'll try to get back to this work next week and will narrow the problem down and upload something that emulates it.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss

pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cache not loading correctly

commented by dmitriy setrakyan

Steve, have you had a chance to look into this?

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cache not loading correctly

commented by steve neal

Hi Dmitriy, Yes, and I'm still seeing the same thing. When all nodes are started at the same time, some data goes missing. If I start a single node, it all loads correctly; then when other nodes are started, the data is balanced correctly.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cache not loading correctly

commented by dmitriy setrakyan

Can you provide a small reproducible example? We are specifically interested in how you configure Ignite and how you measure that the data was loaded. I know you already described it in this thread, but a reproducible example will really help us to resolve this issue.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
Loading...