Lots of cache creation become slow

classic Classic list List threaded Threaded
7 messages Options
ctranxuan ctranxuan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Lots of cache creation become slow

This post was updated on .
Hi,
We are trying for test purposes to create lots of cache on a machine. Something like:

for (int i = 0; i < 10000; i++) {
      IgniteCache<Long, Snapshot> cache;
      cache = ignite.<Long, Snapshot>getOrCreateCache("cache-" + i).withAsync();
      LOGGER.info("starting to read cache #" + cache.getName());
}

What we are noticing is that after ~200 caches created, the cache creation becomes slower and slower. For instance, when reaching the 2000th cache, it takes between 2-3 seconds. Here are some logs extract:

09:34:27.857 starting to read cache #cache-2087
09:34:29.621 starting to read cache #cache-2088
09:34:31.450 starting to read cache #cache-2089
09:34:33.127 starting to read cache #cache-2090

That's may be a naive way of doing that and may be a naive question: but is it normal that the more cache is created, the more time the creation takes?

The program is run on a 12GB RAM machine with 4 CPUs with a Java 8 (1.8.0_121).

Thanks in advance for the answers!
agura agura
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Lots of cache creation become slow

Creation of each cache requires creation and initialization of
internal data structures that leads to increased pressure to GC. Could
you enable GC logs and look at result. I think you will find long GC
pauses.

In order to reduce memory consumption by created caches on creation
stage we can do the following:

- Decrease cache start size (see cacheConfiguration.setStartSize(),
default value is 1 500 000). It safe but can lead to some performance
penalty during dynamic increasing cache size;
- For atomic caches we can decrease size of deferred delete queue size
via JVM parameter (-DIGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE=<your
value>, default value is 200 000 entries per partition). It's static
configuration and can't be changed at runtime;
- Partitions number also affect memory consumption and cache creation
time. But it affect also data distribution so should be configured
carefully. Caches with default configuration uses
RendevousAffinityFunction ith 1024 partitions. This parameter can't be
changed after cache creation and start.

Does it work for you?

On Tue, Apr 11, 2017 at 12:53 PM, ctranxuan
<[hidden email]> wrote:

> Hi,
> We are trying for test purposes to create lots of cache on a machine.
> Something like:
>
> for (int i = 0; i < 10000; i++) {
>       IgniteCache<Long, Snapshot> cache;
>       cache = ignite.<Long, Snapshot>getOrCreateCache("cache-" +
> i).withAsync();
>       LOGGER.info("starting to read cache #" + cache.getName());
> }
>
> What we are noticing is that after ~200 caches created, the cache creation
> becomes slower and slower. For instance, when reaching the 2000th cache, it
> takes between 2-3 seconds. For instance, here some logs:
>
> 09:34:27.857 starting to read cache #cache-2087
> 09:34:29.621 starting to read cache #cache-2088
> 09:34:31.450 starting to read cache #cache-2089
> 09:34:33.127 starting to read cache #cache-2090
>
> That's may be a naive way of doing that and may be a naive question: but is
> it normal that the more cache is created, the more time the creation takes?
>
> The program is run on a 12GB RAM machine with 4 CPUs with a Java 8
> (1.8.0_121).
>
> Thanks in advance for the answers!
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Lots-of-cache-creation-become-slow-tp11875.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
ctranxuan ctranxuan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Lots of cache creation become slow

Thanks a lot for the answer!
We'll try to run tests with these hints.
Alexey Kuznetsov-2 Alexey Kuznetsov-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Lots of cache creation become slow

Cédric,

Just, curious, what for do you need 2000 caches on single node? :)


On Wed, Apr 12, 2017 at 8:47 PM, ctranxuan <[hidden email]> wrote:
Thanks a lot for the answer!
We'll try to run tests with these hints.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Lots-of-cache-creation-become-slow-tp11875p11906.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--
Alexey Kuznetsov
ctranxuan ctranxuan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Lots of cache creation become slow

Well, actually we were interesting in having continuous queries listening multi-tenant caches.

This was the postulate for the architecture of a PoC project. Based on this discussion, we are switching to another architecture postulate where we have one cache with thousands continuous queries listening the changes of thousands keys of the cache (basically 1 continuous query per key).

So, at the beginning, we were investigating how many caches / continuous queries could be supported by a node. May be, it's not the right way to evaluate this?
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Lots of cache creation become slow

Cédric,

Can you clarify why not create 1 continuous query and listen to all the changes for all the keys?

D.

On Thu, Apr 13, 2017 at 8:00 AM, ctranxuan <[hidden email]> wrote:
Well, actually we were interesting in having continuous queries listening
multi-tenant caches.

This was the postulate for the architecture of a PoC project. Based on this
discussion, we are switching to another architecture postulate where we have
one cache with thousands continuous queries listening the changes of
thousands keys of the cache (basically 1 continuous query per key).

So, at the beginning, we were investigating how many caches / continuous
queries could be supported by a node. May be, it's not the right way to
evaluate this?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Lots-of-cache-creation-become-slow-tp11875p11955.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

ctranxuan ctranxuan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Lots of cache creation become slow

Well, that was a new design assumption for the architecture of our PoC.
At the beginning, we wanted some "multi-tenant" caches with 1 continuous query for each. Then, we've tried to switch with one cache and 1 continous query for each key.
But, you're right. We have already started to challenge our PoC with 1 continous query for the whole cache.

Thanks for the advice!
Loading...