Off Heap cache using lots of heap memory

classic Classic list List threaded Threaded
4 messages Options
Neil Wightman Neil Wightman
Reply | Threaded
Open this post in threaded view
|

Off Heap cache using lots of heap memory

Hi All,

I have been trying out ignite for the past few weeks but I am hitting a
strange problem.  I dont know if this problem is in my code or ignite.

Currently I have an OFFHEAP_TIERED cache with ~3 million entries, max
off heap size of 4Gb, but this is showing some very high heap memory
usage.  The cache stores data for 1 hour then expires it and constantly
streams data into this cache via the dataStreamer.addData API.

The problem seems to be with this specific expiry policy configuration :

cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new
CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );

  - with this expiry policy the heap memory usage for the ignite process
is 976Mb, without this expiry policy set its just 156Mb.

The actual off heap memory usage is only 557Mb (assuming the cache
metrics are correct).

I have generated two heap dumps and the memory difference is all in the
GridCacheTtlManager which is retaining a heap of ~889Mb according to
Eclipses MAT.

I have ran my tests with Ignite 1.5.0 Final and 1.6.0 SNAPSHOT from git
(Monday 7th March) and both show the same issue.

The problem with this large heap usage is that I am now getting very
large GC pauses which I cant stop without disabling the expiry policy.

My main question is am I doing something wrong?   Is the Expiry Policy
really supposed to use so much heap memory compared to the off heap
cache size?

Thanks in advance,
Neil

---- config definition ----

final CacheConfiguration<MetricKey, AggregatePack> cacheConfiguration =
new CacheConfiguration<>( "cacheName" );
cacheConfiguration.setBackups( 1 );
cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
cacheConfiguration.setIndexedTypes( CustomKey.class, CustomData.class );
cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L * 1024L );
cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy<>( 6_000_000
) );
cacheConfiguration.setSwapEnabled( true );
cacheConfiguration.setStatisticsEnabled( true );
// This causes large memory usage
cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new
CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );

Vladimir Ozerov Vladimir Ozerov
Reply | Threaded
Open this post in threaded view
|

Re: Off Heap cache using lots of heap memory

Hi Neil,

Could you please attach the code reproducing the problem? 

Vladimir.

On Wed, Mar 9, 2016 at 1:16 PM, Neil Wightman <[hidden email]> wrote:
Hi All,

I have been trying out ignite for the past few weeks but I am hitting a strange problem.  I dont know if this problem is in my code or ignite.

Currently I have an OFFHEAP_TIERED cache with ~3 million entries, max off heap size of 4Gb, but this is showing some very high heap memory usage.  The cache stores data for 1 hour then expires it and constantly streams data into this cache via the dataStreamer.addData API.

The problem seems to be with this specific expiry policy configuration :

cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );

 - with this expiry policy the heap memory usage for the ignite process is 976Mb, without this expiry policy set its just 156Mb.

The actual off heap memory usage is only 557Mb (assuming the cache metrics are correct).

I have generated two heap dumps and the memory difference is all in the GridCacheTtlManager which is retaining a heap of ~889Mb according to Eclipses MAT.

I have ran my tests with Ignite 1.5.0 Final and 1.6.0 SNAPSHOT from git (Monday 7th March) and both show the same issue.

The problem with this large heap usage is that I am now getting very large GC pauses which I cant stop without disabling the expiry policy.

My main question is am I doing something wrong?   Is the Expiry Policy really supposed to use so much heap memory compared to the off heap cache size?

Thanks in advance,
Neil

---- config definition ----

final CacheConfiguration<MetricKey, AggregatePack> cacheConfiguration = new CacheConfiguration<>( "cacheName" );
cacheConfiguration.setBackups( 1 );
cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
cacheConfiguration.setIndexedTypes( CustomKey.class, CustomData.class );
cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L * 1024L );
cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy<>( 6_000_000 ) );
cacheConfiguration.setSwapEnabled( true );
cacheConfiguration.setStatisticsEnabled( true );
// This causes large memory usage
cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );


Neil Wightman Neil Wightman
Reply | Threaded
Open this post in threaded view
|

Re: Off Heap cache using lots of heap memory

Hi Vladimir,

It took me a while to make a small test case but I now have one (see below)

With the setExpiryPolicyFactory method enabled the heap after operation 983Mb, with the setExpiryPolicyFactory commented out it is just 85Mb.

Simple run this class file and once you see the console message attach JVisualVM, then spam the "Perform GC" button.

Also without the expiry time set the test runs with -Xm512m and with expiry the test needs -Xmx2024m.

Thanks
Neil

-- IgniteExpiryIssue.java --
package de.wightman.neil.apache.ignite.test;

import javax.cache.configuration.FactoryBuilder;
import javax.cache.expiry.CreatedExpiryPolicy;
import javax.cache.expiry.Duration;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteDataStreamer;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheMemoryMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.cache.eviction.lru.LruEvictionPolicy;
import org.apache.ignite.cache.query.annotations.QuerySqlField;
import org.apache.ignite.configuration.CacheConfiguration;

public class IgniteExpiryIssue
{

    public static void main( final String[] args ) throws InterruptedException
    {
        final long start = System.currentTimeMillis();
        // configures Tcp discovery for local host only
        try (final Ignite ignite = Ignition.start( "LocalConfig.xml" ))
        {

            final IgniteCache<CustomKey, CustomValue> cache = ignite.getOrCreateCache( cacheConfig() );
            final IgniteDataStreamer<Object, Object> metricStreamer = ignite.dataStreamer( "TestCacheName" );

            final long counter = 1000;
            final long counter2 = 49;
            final long minutes = 60;

            final long fakeBaseTime = System.currentTimeMillis();

            for( long k = 0; k < minutes; k++ )
            {
                final long ts = fakeBaseTime + (minutes * 60 * 1000);
                for( long i = 0; i < counter; i++ )
                {
                    for( long j = 0; j < counter2; j++ )
                    {
                        final CustomKey key = new CustomKey( ts, i, j );

                        final CustomValue value = new CustomValue( 1.0, 1.0, 1.0, 1.0, 10 );
                        metricStreamer.addData( key, value );
                    }
                }
            }

            // debug memory issue here
            System.out.println("Duration = " + (System.currentTimeMillis() - start));
            System.out.println("Cache filled attach JVisualVM to view memory and generate a heap dump.");
            Thread.sleep(Long.MAX_VALUE);
        }
    }


    public static CacheConfiguration<CustomKey, CustomValue> cacheConfig()
    {
        final CacheConfiguration<CustomKey, CustomValue> cacheConfiguration = new CacheConfiguration<>( "TestCacheName" );

        cacheConfiguration.setBackups( 1 );
        cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
        cacheConfiguration.setIndexedTypes( CustomKey.class, CustomValue.class );
        cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
        cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L * 1024L );
        cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy( 6_000_000 ) );
        cacheConfiguration.setSwapEnabled( true );
        cacheConfiguration.setStatisticsEnabled( true );

        // Causes heap usage to be high
        cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );

        return cacheConfiguration;
    }

    public static class CustomKey
    {

        @QuerySqlField(index = true, descending = false, name = "ts")
        private long timeStamp;

        @QuerySqlField(index = true, name = "id1")
        private long id1;

        @QuerySqlField(index = true, name = "id2")
        private long id2;


        public CustomKey( long timeStamp,
                long id1,
                long id2 )
        {
            this.timeStamp = timeStamp;
            this.id1 = id1;
            this.id2 = id2;
        }
    }

    public static class CustomValue
    {

        @QuerySqlField(name = "_d1")
        private double d1;
        @QuerySqlField()
        private double d2;
        @QuerySqlField()
        private double d3;
        @QuerySqlField()
        private double d4;
        @QuerySqlField(name = "_count")
        private long count;


        public CustomValue( double d1,
                double d2,
                double d3,
                double d4,
                long count )
        {
            this.d1 = d1;
            this.d2 = d2;
            this.d3 = d3;
            this.d4 = d4;
            this.count = count;
        }
    }
}
----

On 10/03/16 13:04, Vladimir Ozerov wrote:
Hi Neil,

Could you please attach the code reproducing the problem? 

Vladimir.

On Wed, Mar 9, 2016 at 1:16 PM, Neil Wightman <[hidden email]> wrote:
Hi All,

I have been trying out ignite for the past few weeks but I am hitting a strange problem.  I dont know if this problem is in my code or ignite.

Currently I have an OFFHEAP_TIERED cache with ~3 million entries, max off heap size of 4Gb, but this is showing some very high heap memory usage.  The cache stores data for 1 hour then expires it and constantly streams data into this cache via the dataStreamer.addData API.

The problem seems to be with this specific expiry policy configuration :

cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );

 - with this expiry policy the heap memory usage for the ignite process is 976Mb, without this expiry policy set its just 156Mb.

The actual off heap memory usage is only 557Mb (assuming the cache metrics are correct).

I have generated two heap dumps and the memory difference is all in the GridCacheTtlManager which is retaining a heap of ~889Mb according to Eclipses MAT.

I have ran my tests with Ignite 1.5.0 Final and 1.6.0 SNAPSHOT from git (Monday 7th March) and both show the same issue.

The problem with this large heap usage is that I am now getting very large GC pauses which I cant stop without disabling the expiry policy.

My main question is am I doing something wrong?   Is the Expiry Policy really supposed to use so much heap memory compared to the off heap cache size?

Thanks in advance,
Neil

---- config definition ----

final CacheConfiguration<MetricKey, AggregatePack> cacheConfiguration = new CacheConfiguration<>( "cacheName" );
cacheConfiguration.setBackups( 1 );
cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
cacheConfiguration.setIndexedTypes( CustomKey.class, CustomData.class );
cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L * 1024L );
cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy<>( 6_000_000 ) );
cacheConfiguration.setSwapEnabled( true );
cacheConfiguration.setStatisticsEnabled( true );
// This causes large memory usage
cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );



Vladimir Ozerov Vladimir Ozerov
Reply | Threaded
Open this post in threaded view
|

Re: Off Heap cache using lots of heap memory

Hi Neil,

Thank you for code snippet. I was able to reproduce the problem and created a ticket - https://issues.apache.org/jira/browse/IGNITE-2833
Hope we will be able to fix it soon.

Vladimir.

On Fri, Mar 11, 2016 at 10:13 AM, Neil Wightman <[hidden email]> wrote:
Hi Vladimir,

It took me a while to make a small test case but I now have one (see below)

With the setExpiryPolicyFactory method enabled the heap after operation 983Mb, with the setExpiryPolicyFactory commented out it is just 85Mb.

Simple run this class file and once you see the console message attach JVisualVM, then spam the "Perform GC" button.

Also without the expiry time set the test runs with -Xm512m and with expiry the test needs -Xmx2024m.

Thanks
Neil

-- IgniteExpiryIssue.java --
package de.wightman.neil.apache.ignite.test;

import javax.cache.configuration.FactoryBuilder;
import javax.cache.expiry.CreatedExpiryPolicy;
import javax.cache.expiry.Duration;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteDataStreamer;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheMemoryMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.cache.eviction.lru.LruEvictionPolicy;
import org.apache.ignite.cache.query.annotations.QuerySqlField;
import org.apache.ignite.configuration.CacheConfiguration;

public class IgniteExpiryIssue
{

    public static void main( final String[] args ) throws InterruptedException
    {
        final long start = System.currentTimeMillis();
        // configures Tcp discovery for local host only
        try (final Ignite ignite = Ignition.start( "LocalConfig.xml" ))
        {

            final IgniteCache<CustomKey, CustomValue> cache = ignite.getOrCreateCache( cacheConfig() );
            final IgniteDataStreamer<Object, Object> metricStreamer = ignite.dataStreamer( "TestCacheName" );

            final long counter = 1000;
            final long counter2 = 49;
            final long minutes = 60;

            final long fakeBaseTime = System.currentTimeMillis();

            for( long k = 0; k < minutes; k++ )
            {
                final long ts = fakeBaseTime + (minutes * 60 * 1000);
                for( long i = 0; i < counter; i++ )
                {
                    for( long j = 0; j < counter2; j++ )
                    {
                        final CustomKey key = new CustomKey( ts, i, j );

                        final CustomValue value = new CustomValue( 1.0, 1.0, 1.0, 1.0, 10 );
                        metricStreamer.addData( key, value );
                    }
                }
            }

            // debug memory issue here
            System.out.println("Duration = " + (System.currentTimeMillis() - start));
            System.out.println("Cache filled attach JVisualVM to view memory and generate a heap dump.");
            Thread.sleep(Long.MAX_VALUE);
        }
    }


    public static CacheConfiguration<CustomKey, CustomValue> cacheConfig()
    {
        final CacheConfiguration<CustomKey, CustomValue> cacheConfiguration = new CacheConfiguration<>( "TestCacheName" );

        cacheConfiguration.setBackups( 1 );
        cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
        cacheConfiguration.setIndexedTypes( CustomKey.class, CustomValue.class );
        cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
        cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L * 1024L );
        cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy( 6_000_000 ) );
        cacheConfiguration.setSwapEnabled( true );
        cacheConfiguration.setStatisticsEnabled( true );

        // Causes heap usage to be high
        cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );

        return cacheConfiguration;
    }

    public static class CustomKey
    {

        @QuerySqlField(index = true, descending = false, name = "ts")
        private long timeStamp;

        @QuerySqlField(index = true, name = "id1")
        private long id1;

        @QuerySqlField(index = true, name = "id2")
        private long id2;


        public CustomKey( long timeStamp,
                long id1,
                long id2 )
        {
            this.timeStamp = timeStamp;
            this.id1 = id1;
            this.id2 = id2;
        }
    }

    public static class CustomValue
    {

        @QuerySqlField(name = "_d1")
        private double d1;
        @QuerySqlField()
        private double d2;
        @QuerySqlField()
        private double d3;
        @QuerySqlField()
        private double d4;
        @QuerySqlField(name = "_count")
        private long count;


        public CustomValue( double d1,
                double d2,
                double d3,
                double d4,
                long count )
        {
            this.d1 = d1;
            this.d2 = d2;
            this.d3 = d3;
            this.d4 = d4;
            this.count = count;
        }
    }
}
----


On 10/03/16 13:04, Vladimir Ozerov wrote:
Hi Neil,

Could you please attach the code reproducing the problem? 

Vladimir.

On Wed, Mar 9, 2016 at 1:16 PM, Neil Wightman <[hidden email]> wrote:
Hi All,

I have been trying out ignite for the past few weeks but I am hitting a strange problem.  I dont know if this problem is in my code or ignite.

Currently I have an OFFHEAP_TIERED cache with ~3 million entries, max off heap size of 4Gb, but this is showing some very high heap memory usage.  The cache stores data for 1 hour then expires it and constantly streams data into this cache via the dataStreamer.addData API.

The problem seems to be with this specific expiry policy configuration :

cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );

 - with this expiry policy the heap memory usage for the ignite process is 976Mb, without this expiry policy set its just 156Mb.

The actual off heap memory usage is only 557Mb (assuming the cache metrics are correct).

I have generated two heap dumps and the memory difference is all in the GridCacheTtlManager which is retaining a heap of ~889Mb according to Eclipses MAT.

I have ran my tests with Ignite 1.5.0 Final and 1.6.0 SNAPSHOT from git (Monday 7th March) and both show the same issue.

The problem with this large heap usage is that I am now getting very large GC pauses which I cant stop without disabling the expiry policy.

My main question is am I doing something wrong?   Is the Expiry Policy really supposed to use so much heap memory compared to the off heap cache size?

Thanks in advance,
Neil

---- config definition ----

final CacheConfiguration<MetricKey, AggregatePack> cacheConfiguration = new CacheConfiguration<>( "cacheName" );
cacheConfiguration.setBackups( 1 );
cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
cacheConfiguration.setIndexedTypes( CustomKey.class, CustomData.class );
cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L * 1024L );
cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy<>( 6_000_000 ) );
cacheConfiguration.setSwapEnabled( true );
cacheConfiguration.setStatisticsEnabled( true );
// This causes large memory usage
cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );