Pagination to fetch complete Cache data

classic Classic list List threaded Threaded
8 messages Options
sujay_kb@ymail.com sujay_kb@ymail.com
Reply | Threaded
Open this post in threaded view
|

Pagination to fetch complete Cache data

This post was updated on .
Can we use pagination while fetching the complete cache data..

I am using below method for fetching complete Cache data, but it is taking too much time :
ScanQuery<String, [CacheObject]> scanQuery = new ScanQuery<String, [CacheObject]>()
QueryCursor<javax.cache.Cache.Entry<String, [CacheObject]>> cursor = cache.query(scanQuery);


I tired using below method also, but i am getting exception:
 QueryCursor<Cache.Entry<Integer,  [CacheObject]>> result =
 cache.query(new SqlQuery<Integer,  [CacheObject]>([CacheObject].class, "select * from  [CacheName]"));
result.getAll()


Exception in thread "main" javax.cache.CacheException: Failed to run reduce query locally.
        at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:673)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$2.iterator(IgniteH2Indexing.java:956)
        at org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:61)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$3.iterator(IgniteH2Indexing.java:990)
        at org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:61)
        at org.apache.ignite.internal.processors.cache.QueryCursorImpl.getAll(QueryCursorImpl.java:73)
        at com.test.ignite.query.QueryFetch2.queryPrices(QueryFetch2.java:83)
        at com.test.ignite.query.QueryFetch2.main(QueryFetch2.java:52)
Caused by: java.lang.NullPointerException
        at org.apache.ignite.internal.processors.query.h2.twostep.GridMergeIndexUnsorted$1.hasNext(GridMergeIndexUnsorted.java:97)


Cache Configurations I am using:

CacheConfiguration<Object, Object>  cacheConfig = new CacheConfiguration<>(cacheName);
                cacheConfig.setCacheMode(CacheMode.REPLICATED);
                cacheConfig.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);
                cacheConfig.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new Duration(TimeUnit.MINUTES, 60)));
                cacheConfig.setEvictionPolicy(new LruEvictionPolicy(50000));
                cacheConfig.setOffHeapMaxMemory(0);
                cacheConfig.setSwapEnabled(false);
                cacheConfig.setBackups(1);
                cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Pagination to fetch complete Cache data

Hi,

Query results are already paginated by default. Page size is 1024 elements by default and can be changed via Query.setPageSize() method. How many entries in cache do you have? Iteration through the large data set is not going to be fast anyway.

SQL should be used for indexed search, and they require additional configuration. Refer to [1] for details.

[1] https://apacheignite.readme.io/docs/sql-queries

-Val
sujay_kb@ymail.com sujay_kb@ymail.com
Reply | Threaded
Open this post in threaded view
|

Re: Pagination to fetch complete Cache data

Thanks for Info on Pagesize, i will try it out

I am having 3 million records i believe its small dataset, then also i am facing this issue

Can you suggest which fetch approach is best for huge dataset (cache),

Regarding previous question/post i have mentioned the exception(Exception in thread "main" javax.cache.CacheException: Failed to run reduce query locally.) while using the Query (select * [CacheName]) - is this a bug? - because with the smaller dataset it works fine.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Pagination to fetch complete Cache data

Hi,

Fetching all entries to the client will never be effective because it involves transferring too much data across network. If you need to process the whole data set, it's usually better to use Compute Grid [1]. Broadcast a closure or execute a map-reduce task and send only results to the client. Is it something that can work for you?

NPE in SQL is a known issue and already fixed. You can build Ignite from master and check if it works for you.

[1] https://apacheignite.readme.io/docs/compute-grid

-Val
sujay_kb@ymail.com sujay_kb@ymail.com
Reply | Threaded
Open this post in threaded view
|

Re: Pagination to fetch complete Cache data

Yes Val, let me try the options

I will get the Ignite from master and check it out

Thanks again Val for timely help
begineer begineer
Reply | Threaded
Open this post in threaded view
|

Re: Pagination to fetch complete Cache data

This post has NOT been accepted by the mailing list yet.
In reply to this post by vkulichenko
setpageSize does not work for me. below is my code where I am setting page size to 10 , but I get all 20 records which are valid for given predicate. I should get 10 records from cursor. Not sure what is wrong in below code ?

public Set<String> getData(int pageSize){
        Ignite ignite = Ignition.start("examples/config/example-ignite.xml");
        CacheConfiguration<Integer, String> config = new CacheConfiguration<>("mycache");
        config.setBackups(1);
        IgniteCache<Integer, String> cache = ignite.getOrCreateCache(config);
        long dataSize = 30;
        for (int i = 1; i < dataSize; i++) {
            cache.put(i, String.valueOf(i));
        }

        ScanQuery<Integer, String> query = new ScanQuery<>((Integer e1, String e2) ->e1>10);
        Set<String> res = cache.query(query.setPageSize(pageSize)).getAll().stream().map(Cache.Entry::getValue).collect(toSet());
        System.out.println(res);
        return res;
    }
begineer begineer
Reply | Threaded
Open this post in threaded view
|

Re: Pagination to fetch complete Cache data

This post has NOT been accepted by the mailing list yet.
Sorry for mistake, correction in below line. Set pageSize to 10. But output remains same. I still get 20 records in result.

Set<String> res = cache.query(query.setPageSize(10)).getAll().stream().map(Cache.Entry::getValue).collect(toSet());
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Pagination to fetch complete Cache data

Hi,

Please properly subscribe to the mailing list so that the community can receive email notifications for you messages. Here is the instruction: http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1

begineer wrote
Sorry for mistake, correction in below line. Set pageSize to 10. But output remains same. I still get 20 records in result.

Set<String> res = cache.query(query.setPageSize(10)).getAll().stream().map(Cache.Entry::getValue).collect(toSet());
QueryCursor allows you to iterate through the whole result set, and the pagination happens internally. This means that client will fetch rows in pages discarding old ones, so that you never have more than 10 rows in local memory at the same time.

-Val