ScanQuery log all entries

classic Classic list List threaded Threaded
6 messages Options
javastuff.sam@gmail.com javastuff.sam@gmail.com
Reply | Threaded
Open this post in threaded view
|

ScanQuery log all entries

This post was updated on .
Hi,

I have a usecase where I have to log cached entries. I am doing below -

ScanQuery<String, Object> scan = new ScanQuery<>();
cache.query(scan).getAll();


Question -
1. With OFF_HEAP cache it is suppose to be huge number of entries in cache. Does getAll() will try to bring all into heap and potential out-of-memory?
2. Is there other way to iterate page by page?
3. Is there any other restriction I can add to scan query like limit 1000 or query timeout?

Thanks,
-Sam
dkarachentsev dkarachentsev
Reply | Threaded
Open this post in threaded view
|

Re: ScanQuery log all entries

Hi Sam,

1. Yes getAll() will bring all values to an on-heap collection, which may lead to OOME, so it should be used with caution.
2. Any query returns QueryCursor that lazily iterates over entries. You may do 1000 iterations over cursor and accumulate result in some collection. This will be just like pagination.
3. You may set page size, by default it's 1024. This means that cursor will pre-fetch from remote node 1024 entries.

-Dmitry
javastuff.sam@gmail.com javastuff.sam@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: ScanQuery log all entries

Thanks Dmitry.

So it means if I use QueryCursor, it will not bring all entries in heap but only 1024 or page size setting.
Is there any example for QueryCursor? Can anybody share a sample code snippet for QueryCursor usage?

Thanks,
-Sam
dkarachentsev dkarachentsev
Reply | Threaded
Open this post in threaded view
|

Re: ScanQuery log all entries

Hi,

Yes, you understand correctly. Here is example:

private static void sqlQuery() {
        IgniteCache<AffinityKey<Long>, Person> cache = Ignition.ignite().cache(PERSON_CACHE);

        // SQL clause which selects salaries based on range.
        String sql = "salary > ? and salary <= ?";

        // People with salaries between 1000 and 2000.
        try (QueryCursor<Cache.Entry<AffinityKey<Long>, Person>> cursor =
                 cache.query(new SqlQuery<AffinityKey<Long>, Person>(Person.class, sql).
                 setArgs(1000, 2000))) {

            // Cursor is Iterable, so you may iterate over it with for each loop.
            for (Cache.Entry<AffinityKey<Long>, Person> entry : cursor)
                System.out.println("Key: " + entry.getKey() + ", Value: " + entry.getValue());
        }
}

It's based on https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheQueryExample.java

-Dmitry.
javastuff.sam@gmail.com javastuff.sam@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: ScanQuery log all entries

Thanks. One question though -

Below loop will automatically go over all pages or just one page with 1024 entries?
       
            for (Cache.Entry<AffinityKey<Long>, Person> entry : cursor)
                System.out.println("Key: " + entry.getKey() + ", Value: " + entry.getValue());

If it is just one page then how to know there is more results and how to iterate on those?

Thanks,
-Sam
dkarachentsev dkarachentsev
Reply | Threaded
Open this post in threaded view
|

Re: ScanQuery log all entries

Sam,

Cursor iterator will go over whole result, but it will not hold in memory all dataset, just part limited by page size. In other words, it fetches 1024 entries, and when iterator points after last entry - loads another page with 1024 entries. So you shouldn't worry when results arrived - they automatically paginated, you just need correctly manage data, e.g. if you collect all of them in list - you may get OOME.

-Dmitry.