Offheap memory consumption + enabled persistence

classic Classic list List threaded Threaded
6 messages Options
mikle-a mikle-a
Reply | Threaded
Open this post in threaded view
|

Offheap memory consumption + enabled persistence

This post was updated on .
Hi!

Problem

I've noticed some weird behavior while working with enabled persistence
(native storage): In a loop, I put some amount of entries to the cluster and delete
them afterwards. When I use the same set of keys each time, I see that
*DataRegionMetrics.getOffheapUsedSize* reaches some limit and don't grow
anymore, i.e. memory is reused. When I do the same action, but with random
keys, I see that mentioned above offheap memory metric always grows and
never stops.

In pictures, test with static keys:
<http://apache-ignite-users.70518.x6.nabble.com/file/t2766/static-keys.png

test with random keys (running ~ 10 hours):
<http://apache-ignite-users.70518.x6.nabble.com/file/t2766/random-keys.png

As I can see, if I disable persistence, everything works as with static
keys.

I really wonder why enabling persistence causes such excessive offheap
consumption, could you explain please?

Code to reproduce

Here are 3 scenarios:
1) TestPersistenceStaticKeys: persistence enabled, static keys - offheap
reaches some limit and stops growing, as expected
2) TestPersistenceRandomKeys: persistence enabled, random keys - offheap
always grow, why?
3) TestRandomKeysWithoutPersistence: persistence disabled, random keys -
offheap reaches limit and stops growing, as expected

package com.test.ignite.memory;

import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.Ignition;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.DataRegionConfiguration;
import org.apache.ignite.configuration.DataStorageConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;

import java.util.concurrent.ThreadLocalRandom;
import java.util.function.Function;
import java.util.stream.IntStream;

public class OffheapConsumption {

    public static class TestPersistenceStaticKeys {
        public static void main(String[] args) {
            runTest(true, i -> i);
        }
    }

    public static class TestPersistenceRandomKeys {
        public static void main(String[] args) {
            runTest(true, i -> ThreadLocalRandom.current().nextInt());
        }
    }

    public static class TestRandomKeysWithoutPersistence {
        public static void main(String[] args) {
            runTest(false, i -> ThreadLocalRandom.current().nextInt());
        }
    }

    private static void runTest(boolean persistenceEnabled, Function<Integer, Integer> idProvider) {
        final DataRegionConfiguration defaultDataRegionConfig = new DataRegionConfiguration();
        defaultDataRegionConfig.setMetricsEnabled(true);
        defaultDataRegionConfig.setPersistenceEnabled(persistenceEnabled);

        final DataStorageConfiguration storageConfiguration = new DataStorageConfiguration();
        storageConfiguration.setDefaultDataRegionConfiguration(defaultDataRegionConfig);

        final CacheConfiguration<Integer, String> cacheConfig = new CacheConfiguration<>();
        cacheConfig.setName("cache");

        final Ignite server1 = Ignition.start(new IgniteConfiguration()
                .setDataStorageConfiguration(storageConfiguration)
                .setCacheConfiguration(cacheConfig)
                .setIgniteInstanceName("server1"));

        final Ignite server2 = Ignition.start(new IgniteConfiguration()
                .setDataStorageConfiguration(storageConfiguration)
                .setCacheConfiguration(cacheConfig)
                .setIgniteInstanceName("server2"));

        server1.cluster().active(true);

        final Ignite client = Ignition.start(new IgniteConfiguration().setClientMode(true));
        final IgniteCache<Integer, String> cache = client.cache("cache");


        for (int i = 0; i < 100; i++) {
            //print data regions used offheap size
            server1.dataRegionMetrics().stream()
                    .filter(drm -> drm.getName().equals("default"))
                    .findFirst()
                    .ifPresent(dataRegionMetrics ->
                            System.out.println(String.format("Data region '%s' used off heap size = %s",
                            dataRegionMetrics.getName(), dataRegionMetrics.getOffheapUsedSize())));

            //put entries
            System.out.println("put entries");
            IntStream.range(0, 1000)
                    .forEach(n -> cache.put(idProvider.apply(n), new String(new byte[1024])));

            //clean all
            System.out.println("clear entries");
            cache.clear();
        }

        client.close();
        server1.close();
        server2.close();
    }
}


--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Offheap memory consumption + enabled persistence

Hi,

Growth of this metric will be stopped after all partitions(1024 by default)
allocate memory for maximum count of entries that can be stored in it
simultaneously. It will take some time for 1024 partitions(not 100
iterations). You can check it by setting partitions count to smaller size:
        cacheConfig.setAffinity(new RendezvousAffinityFunction(false, 8));

Evgenii



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mikle-a mikle-a
Reply | Threaded
Open this post in threaded view
|

Re: Offheap memory consumption + enabled persistence

First, thanks a lot for your reply!

But I am still confused. Did I understood properly that each node should
have enough memory to store full data set?

Previously I thought that the main idea of partitioning is to distribute
data among servers. For example, to distribute 300GB of data among 3 servers
having 100GB each. Now it turns that each server should have 300GB, or I've
understood it wrong?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: Offheap memory consumption + enabled persistence

>But I am still confused. Did I understood properly that each node should
have enough memory to store full data set?
No, it's not the case. It's just preallocating this memory with internal
structures and shows it in the metrics.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Dmitriy Govorukhin Dmitriy Govorukhin
Reply | Threaded
Open this post in threaded view
|

Re: Offheap memory consumption + enabled persistence

In reply to this post by mikle-a
Hi Mikle,

Could you please check what happening with files, they too grow as a metric?
You need to check the size of files in
{IGNITE_HOME}/work/db/cache-{cacheName}/* during random key upload and clean
up. Also, calculate data size on each iteration and print to log.
My assumption in that we have some leak in tracing free space in partition
and each iteration tries to allocate new pages.
In the end, please provide the case with random keys.
- Pure data size on each iteration
- Folder cache size for each iteration
- Chart for DataRegionMetrics.getOffheapUsedSize



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mikle-a mikle-a
Reply | Threaded
Open this post in threaded view
|

Re: Offheap memory consumption + enabled persistence

This post was updated on .
Hi Dmitriy!

I didn't understand about "pure data size" :( Could you please specify how
to calculate it?

Despite this, I've added monitoring for mentioned cache folders and retested
case with random keys and enabled persistence.

Overall test chart:
<http://apache-ignite-users.70518.x6.nabble.com/file/t2766/large-frame.png

Last 30 min:
<http://apache-ignite-users.70518.x6.nabble.com/file/t2766/small-frame.png

As I can see, cache directory grows along with "offheap memory used".

Thanks in advance for your help!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/