Data Cache memory crash

classic Classic list List threaded Threaded
7 messages Options
ali ali
Reply | Threaded
Open this post in threaded view
|

Data Cache memory crash

Hi

I'm trying to use ignite to perform some computations over a large size of data (3M objects)
and quickly hit the GC overhead limit memory.

In order to test i'm using a single machine with one node and 10g of memory.
The example developed is computing the sum of a particular attribute of these objects. the sum is achieved by iterating over all cache objects.  

I tried different configurations , following the user doc found on ignite website, even with off-heap or swap, i have a memory leak.  

It seems that ignite uses arraylist to collect itterated data. (no pagination is done to collect data). ??

I wanted to know the limitation (approximative memory size / numbre of elements that can be achieved with such a configuration) and the best pratices to customise cache and fastest acces to these caches.

I' would be also interested in performance stats if available and recomended configurations.

Thanks
yakov yakov
Reply | Threaded
Open this post in threaded view
|

Re: Data Cache memory crash

Can you please share your code?

--Yakov

2015-05-20 15:51 GMT+03:00 ali <[hidden email]>:
Hi

I'm trying to use ignite to perform some computations over a large size of
data (3M objects)
and quickly hit the GC overhead limit memory.

In order to test i'm using a single machine with one node and 10g of memory.
The example developed is computing the sum of a particular attribute of
these objects. the sum is achieved by iterating over all cache objects.

I tried different configurations , following the user doc found on ignite
website, even with off-heap or swap, i have a memory leak.

It seems that ignite uses arraylist to collect itterated data. (no
pagination is done to collect data). ??

I wanted to know the limitation (approximative memory size / numbre of
elements that can be achieved with such a configuration) and the best
pratices to customise cache and fastest acces to these caches.

I' would be also interested in performance stats if available and recomended
configurations.

Thanks



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-Cache-memory-crash-tp341.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

ali ali
Reply | Threaded
Open this post in threaded view
|

Re: Data Cache memory crash


Here are the interesting points of the code.

The other parts are not very relevant.


private long offHeapMemoryMaxSize = 4 * 1024L * 1024L * 1024L;

    // Cache Configuration
    CacheConfiguration<K, V> cacheConf = new CacheConfiguration<K, V>();
    cacheConf.setName(cacheName);
    cacheConf.setEvictionPolicy(new FifoEvictionPolicy<>(1000));
    cacheConf.setOffHeapMaxMemory(offHeapMemoryMaxSize);
    cacheConf.setSwapEnabled(true);
    cacheConf.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);
    cacheConf.setCacheMode(CacheMode.PARTITIONED);
    ...
   
   
    // Ignite configuration
    IgniteConfiguration conf = new IgniteConfiguration();
    conf.setGridName(name);    
    final FileSwapSpaceSpi fileSwapSpaceSpi = new FileSwapSpaceSpi();
    conf.setSwapSpaceSpi(fileSwapSpaceSpi);
    ...
   
    // cache data access
    cache.query(new ScanQuery<K, V>().setPageSize(256).setLocal(true)).iterator();
    ...
yakov yakov
Reply | Threaded
Open this post in threaded view
|

Re: Data Cache memory crash

Currently Ignite has an issue with page size and scan query. If your scan query does not use any predicates, can you please switch to iteration over entrySet() for now?

Here is the ticket you can track - https://issues.apache.org/jira/browse/IGNITE-872

--Yakov

2015-05-20 16:33 GMT+03:00 ali <[hidden email]>:

Here are the interesting points of the code.

The other parts are not very relevant.


private long offHeapMemoryMaxSize = 4 * 1024L * 1024L * 1024L;

    // Cache Configuration
    CacheConfiguration<K, V> cacheConf = new CacheConfiguration<K, V>();
    cacheConf.setName(cacheName);
    cacheConf.setEvictionPolicy(new FifoEvictionPolicy<>(1000));
    cacheConf.setOffHeapMaxMemory(offHeapMemoryMaxSize);
    cacheConf.setSwapEnabled(true);
    cacheConf.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);
    cacheConf.setCacheMode(CacheMode.PARTITIONED);
    ...


    // Ignite configuration
    IgniteConfiguration conf = new IgniteConfiguration();
    conf.setGridName(name);
    final FileSwapSpaceSpi fileSwapSpaceSpi = new FileSwapSpaceSpi();
    conf.setSwapSpaceSpi(fileSwapSpaceSpi);
    ...

    // cache data access
    cache.query(new ScanQuery<K,
V>().setPageSize(256).setLocal(true)).iterator();
    ...




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-Cache-memory-crash-tp341p344.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

yakov yakov
Reply | Threaded
Open this post in threaded view
|

Re: Data Cache memory crash

Ali this seems to be reproducible with scan queries only when local server carries the data and is in the query topology

--Yakov

2015-05-20 17:22 GMT+03:00 Yakov Zhdanov <[hidden email]>:
Currently Ignite has an issue with page size and scan query. If your scan query does not use any predicates, can you please switch to iteration over entrySet() for now?

Here is the ticket you can track - https://issues.apache.org/jira/browse/IGNITE-872

--Yakov

2015-05-20 16:33 GMT+03:00 ali <[hidden email]>:

Here are the interesting points of the code.

The other parts are not very relevant.


private long offHeapMemoryMaxSize = 4 * 1024L * 1024L * 1024L;

    // Cache Configuration
    CacheConfiguration<K, V> cacheConf = new CacheConfiguration<K, V>();
    cacheConf.setName(cacheName);
    cacheConf.setEvictionPolicy(new FifoEvictionPolicy<>(1000));
    cacheConf.setOffHeapMaxMemory(offHeapMemoryMaxSize);
    cacheConf.setSwapEnabled(true);
    cacheConf.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);
    cacheConf.setCacheMode(CacheMode.PARTITIONED);
    ...


    // Ignite configuration
    IgniteConfiguration conf = new IgniteConfiguration();
    conf.setGridName(name);
    final FileSwapSpaceSpi fileSwapSpaceSpi = new FileSwapSpaceSpi();
    conf.setSwapSpaceSpi(fileSwapSpaceSpi);
    ...

    // cache data access
    cache.query(new ScanQuery<K,
V>().setPageSize(256).setLocal(true)).iterator();
    ...




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-Cache-memory-crash-tp341p344.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Data Cache memory crash

This post was updated on .
In reply to this post by ali
ali wrote
It seems that ignite uses arraylist to collect itterated data. (no pagination is done to collect data). ??
I believe that you are scanning the cache within the process of the node that caches the data. For this approach, I would recommend simple iteration over cache and analyzing values (Cache interface implements Iterable).

If you start another client node, and use Scan query from that node, then the data will be paginated and returned back page-by-page. For this, just start up another node in client mode (Ignition.setClientMode(true) method) and issue the same scan query.
ali wrote
I' would be also interested in performance stats if available and recomended configurations.
Please take a look at Yardstick framework, specifically Ignite benchmarks: https://github.com/yardstick-benchmarks/yardstick-ignite

I have also written a few blogs on benchmarking Ignite:
http://gridgain.blogspot.com/2015/04/benchmarking-data-grids-apache-ignite.html
http://gridgain.blogspot.com/2015/05/benchmarking-data-grids-apache-ignite.html
ali ali
Reply | Threaded
Open this post in threaded view
|

Re: Data Cache memory crash

Hi,

thanks for your feedbacks and precious help.
Using iterable for the cache is giving much better performances.
I will try using a node with mode clientMode set to true,
thanks also fot the benchmark's tools.  

BR