Fastest way to iterate through local cache values

classic Classic list List threaded Threaded
7 messages Options
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Fastest way to iterate through local cache values

This post was updated on .
asked by steve neal

Can you tell me the fastest way is to iterate through all cached values on the local node please?

Currently I'm doing this to test the speed to iterate over 2M values:

IgniteCache<Object, String> jcache = ignite.jcache("test-cache");
AtomicInteger counter = new AtomicInteger();

long start = System.currentTimeMillis();
Iterable<Cache.Entry<Object, String>> ii = jcache.localEntries(CachePeekMode.PRIMARY);
Stream<String> timeStream = StreamSupport.stream(ii.spliterator(), true).map((e) -> e.getValue());
timeStream.forEach((e) -> counter.incrementAndGet());

long end = System.currentTimeMillis();
log.info("time to iterate over (%s) items (%s ms)", counter.get(), (end - start));

but the iteration takes a lot longer than I was hoping for:

time to iterate over (2,000,000) items (542 ms)

My cache configuration looks like this:

<bean class="org.apache.ignite.configuration.CacheConfiguration"
      p:name="test-cache"
      p:backups="1"
      p:cacheMode="PARTITIONED"
      p:atomicityMode="ATOMIC"
      p:preloadMode="SYNC"
      p:startSize="3000000">

Am I accessing the values in an optimal way, or is there a better approach?

Thanks.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fastest way to iterate through local cache values

commented by dmitriy setrakyan

I will try to test the code provided and will post my findings shortly.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fastest way to iterate through local cache values

commented by dmitriy setrakyan

By the way, just of the top of my head, it seems like you are using AtomicLong.incrementAndGet() in a highly concurrent parallel execution and it probably does a lot of spinning. Can you try LongAdder instead?

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fastest way to iterate through local cache values

commented by artem shutak

The issue will be investigated in bounds of https://issues.apache.org/jira/browse/IGNITE-649.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fastest way to iterate through local cache values

commented by artem shutak

In last Ignite version we have copyOnRead property at CacheConfiguration. Try to set it in false and try again. On my machine I has 2 times speed up.

Also, I suggest to measure iteration time only after warm up (not for first iteration).

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fastest way to iterate through local cache values

commented by steve neal

Hi Artem / Dimitriy,

Thanks for the reply. I've updated to RC-3 and have rerun my benchmark with the copyOnRead setting set to false (I had to delete the preloadMode settings, as this seems to have been removed in this version?).

Sadly, things aren't any better:

time to iterate over (2,000,000) items (696 ms)

Is there anything we can do to get this investigated more quickly? My client is a large investment bank and they're keen to move forward with this project. I'll need to look at an alternative solution if this continues to be a problem.

Thanks, Steve.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fastest way to iterate through local cache values

This post was updated on .
commented by artem shutak

Hi, Steve,

According to https://ignite.incubator.apache.org/download.html, the last version of Ignite is 1.0.0 (not RC-3).

But I check it with 1.0.0 and it looks like that my suggestion with copyOnRead do not work properly on 1.0.0. We are planning to release (at several days) 1.1.0, where this setting works properly.

Another suggestions:

check you increase your heap size
check you do not use debug
it's not recommended to use startSize setting
Note, preloadMode is rebalanceMode now.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
Loading...