After increasing maxsize in DataRegionConfiguration...Ignite throws out of memory exception

classic Classic list List threaded Threaded
4 messages Options
Tunas Tunas
Reply | Threaded
Open this post in threaded view
|

After increasing maxsize in DataRegionConfiguration...Ignite throws out of memory exception

This post was updated on .
I am trying to start Ignite as follows

I am running ignite on windows 10 and i have 32 GB RAM out of which 22 GB is
free available.
If i specify maxsize = 1gb it works fine but for maxsize=2gb it fails., why?

<?xml version="1.0" encoding="UTF-8"?>


<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd">
   
    <bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
       
                        <property name="systemWorkerBlockedTimeout" value="#{5 * 60 * 1000}"/>
                        <property name="failureDetectionTimeout" value="120000"/>
                        <property name="clientFailureDetectionTimeout" value="120000"/>
                        <property name="failureHandler">
                                <bean class="org.apache.ignite.failure.StopNodeFailureHandler">
                                 
                                  <property name="ignoredFailureTypes">
                                        <list>
                                        </list>
                                  </property>
                          </bean>
                        </property>
                       
                        <property name="cacheConfiguration">
                                <list>
                                       
                                        <bean class="org.apache.ignite.configuration.CacheConfiguration">
                                                <property name="atomicityMode" value="ATOMIC"/>
                                                <property name="backups" value="1"/>
                                        </bean>
                                </list>
                        </property>
                       
                        <property name="dataStorageConfiguration">
                                <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
                                        <property name="defaultDataRegionConfiguration">
                                                <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                                                        <property name="persistenceEnabled" value="true" />
                                                         <property name="name" value="Default_Region"/>
                                                         <property name="initialSize" value="#{500L * 1024 * 1024}"/>
                                                         *<property name="maxSize" value="#{1L * 1024 * 1024 * 1024}"/>*
                                                </bean>
                                        </property>
                                       
                                       
                                        <property name="walMode" value="FSYNC"/>
                                        <property name="writeThrottlingEnabled" value="true" />
                                        <property name="pageSize" value="4096"/>
                                </bean>
                        </property>
                                               
                        <property name="discoverySpi">
                                <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                                  <property name="ipFinder">
                                        <bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                                          <property name="addresses">
                                                <list>
                                                  <value>MyHostName1:47500..47501</value>
                                                 
                                                  <value>MyHostName2:47500..47501</value>
                                                </list>
                                          </property>
                                        </bean>
                                  </property>
                                </bean>
                        </property>
                       
                        <property name="communicationSpi">
                                <bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
                                        <property name="localPort" value="48100" />
                                        <property name="localPortRange" value="10" />
                                        <property name="usePairedConnections" value="true"></property>
                                       
                                        <property name="messageQueueLimit" value="1024"></property>
                                       
                                        <property name="directSendBuffer" value="true"></property>
                                        <property name="selectorsCount" value="4"></property>
                                       
                                        <property name="socketWriteTimeout" value="300000" />
                                       
                                </bean>
                        </property>
        </bean>
</beans>


Now as soon i increase <property name="maxSize" value="#{2L * 1024 * 1024 *
1024}"/>  2gb onward it throws out of memory exception

 class Program
    {
        public static IIgnite IgniteServer;
        static void Main(string[] args)
        {
            try
            {
                String EnvironmentPath =Environment.GetEnvironmentVariable("IGNITE_HOME");
                Environment.SetEnvironmentVariable("_JAVA_OPTIONS","-Xms1g");
                var _configPath = Path.Combine(EnvironmentPath,"config\\ignite_server_config.xml");

                if (File.Exists(_configPath))
                {
                    Console.WriteLine(_configPath);

                    IgniteServer = Ignition.Start(_configPath);

                    IgniteServer.GetCluster().SetActive(true);
                    ICollection<IClusterNode> nodes =IgniteServer.GetCluster().ForServers().GetNodes();
                    IgniteServer.GetCluster().SetBaselineTopology(nodes);
                }
            }
            catch(Exception ex)
            {
                Console.WriteLine(ex);
            }

        }
    }


Ignite Log:
[11:40:37,161][INFO][exchange-worker-#43][FsyncModeFileWriteAheadLogManager]
Resolved write ahead log archive directory:
E:\apache-ignite-2.7.6-bin\work\db\wal\archive\node00-8a0f96ea-f6d6-42dd-b9bc-7a61d88fbf88
[11:40:37,161][INFO][exchange-worker-#43][FsyncModeFileWriteAheadLogManager]
Started write-ahead log manager [mode=FSYNC]
[11:40:37,161][INFO][main][IgniteKernal] Performance suggestions for grid (fix if possible)
[11:40:37,161][INFO][main][IgniteKernal] To disable, set DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[11:40:37,161][INFO][main][IgniteKernal]   ^-- Enable server mode for JVM
(add '-server' to JVM options)
[11:40:37,161][INFO][main][IgniteKernal]   ^-- Enable G1 Garbage Collector
(add '-XX:+UseG1GC' to JVM options)
[11:40:37,161][INFO][main][IgniteKernal]   ^-- Specify JVM heap max size
(add '-Xmx<size>[g|G|m|M|k|K]' to JVM options)
[11:40:37,161][INFO][main][IgniteKernal]   ^-- Set max direct memory size if
getting 'OOME: Direct buffer memory' (add
'-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[11:40:37,161][INFO][main][IgniteKernal]   ^-- Disable processing of calls
to System.gc() (add '-XX:+DisableExplicitGC' to JVM options)
[11:40:37,161][INFO][main][IgniteKernal] Refer to this page for more
performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[11:40:37,161][INFO][main][IgniteKernal]
[11:40:37,161][INFO][exchange-worker-#43][PageMemoryImpl] Started page
memory [memoryAllocated=100.0 MiB, pages=24808, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
[11:40:37,161][INFO][main][IgniteKernal] To start Console Management &
Monitoring run ignitevisorcmd.{sh|bat}
[11:40:37,177][INFO][exchange-worker-#43][PageMemoryImpl] Started page
memory [memoryAllocated=100.0 MiB, pages=24808, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
[11:40:37,177][INFO][main][IgniteKernal] Data Regions Configured:
[11:40:37,177][INFO][main][IgniteKernal]   ^-- Default_Region
[initSize=500.0 MiB, maxSize=2.0 GiB, persistence=true]
[11:40:37,177][INFO][main][IgniteKernal]

>>> +----------------------------------------------------------------------+
>>> Ignite ver. 2.7.6#20190911-sha1:21f7ca41c4348909e2fd26ccf59b5b2ce1f4474e
>>> +----------------------------------------------------------------------+
>>> OS name: Windows Server 2008 R2 6.1 x86
>>> CPU(s): 8
>>> Heap: 1.0GB
>>> VM name: 5616@******
>>> Local node [ID=4045A130-7CB5-49EB-B483-BC8B7CF0BDA1, order=1,
>>> clientMode=false]
>>> Local node addresses: [********]
>>> Local ports: TCP:10800 TCP:11211 TCP:47500 TCP:48100

[11:40:37,177][SEVERE][exchange-worker-#43][GridDhtPartitionsExchangeFuture]
Failed to reinitialize local partitions (rebalancing will be stopped):
GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1,
minorTopVer=1], discoEvt=DiscoveryCustomEvent
[customMsg=ChangeGlobalStateMessage
[id=d502b1a8f61-5791bb11-5e5b-4b85-b04c-e1d1f9bbfd6a,
reqId=f9a4d984-f34f-492a-ba10-bbd2d1d0318b,
initiatingNodeId=4045a130-7cb5-49eb-b483-bc8b7cf0bda1, activate=true,
baselineTopology=BaselineTopology [id=0, branchingHash=501879262,
branchingType='Cluster activation',
baselineNodes=[8a0f96ea-f6d6-42dd-b9bc-7a61d88fbf88]],
forceChangeBaselineTopology=false, timestamp=1578570037052],
affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=1],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=4045a130-7cb5-49eb-b483-bc8b7cf0bda1, addrs=[0:0:0:0:0:0:0:1,
10.62.21.53, 10.62.44.30, 10.63.216.30, 127.0.0.1],
sockAddrs=[UEU-IP-WAPP0002.coresit.msci.org/10.63.216.30:47500,
/10.62.44.30:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500,
/10.62.21.53:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1578570033978, loc=true, ver=2.7.6#20190911-sha1:21f7ca41,
isClient=false], topVer=1, nodeId8=4045a130, msg=null,
type=DISCOVERY_CUSTOM_EVT, tstamp=1578570037145]], nodeId=4045a130,
evt=DISCOVERY_CUSTOM_EVT]
java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at
org.apache.ignite.internal.util.GridUnsafe.allocateMemory(GridUnsafe.java:1205)
        at
org.apache.ignite.internal.mem.unsafe.UnsafeMemoryProvider.nextRegion(UnsafeMemoryProvider.java:103)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$6.nextRegion(GridCacheDatabaseSharedManager.java:1131)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.start(PageMemoryImpl.java:356)
        at
org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.startMemoryPolicies(IgniteCacheDatabaseSharedManager.java:224)
        at
org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.onActivate(IgniteCacheDatabaseSharedManager.java:1148)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.onActivate(GridCacheDatabaseSharedManager.java:709)
        at
org.apache.ignite.internal.processors.cache.GridCacheSharedContext.activate(GridCacheSharedContext.java:278)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:1082)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:736)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Unknown Source)
[11:40:37,177][INFO][main][GridDiscoveryManager] Topology snapshot [ver=1,
locNode=4045a130, servers=1, clients=0, state=INACTIVE, CPUs=8,
offheap=2.0GB, heap=1.0GB]
[11:40:37,177][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture]
Finish exchange future [startVer=AffinityTopologyVersion [topVer=1,
minorTopVer=1], resVer=null, err=java.lang.OutOfMemoryError]
[11:40:37,177][INFO][main][GridDiscoveryManager]   ^-- Baseline [id=0,
size=1, online=1, offline=0]
[11:40:37,177][INFO][main][GridDiscoveryManager]   ^-- All baseline nodes
are online, will start auto-activation
[11:40:37,177][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture]
Completed partition exchange
[localNode=4045a130-7cb5-49eb-b483-bc8b7cf0bda1,
exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
[topVer=1, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT,
evtNode=TcpDiscoveryNode [id=4045a130-7cb5-49eb-b483-bc8b7cf0bda1,
addrs=[0:0:0:0:0:0:0:1, 10.62.21.53, 10.62.44.30, 10.63.216.30, 127.0.0.1],
sockAddrs=[UEU-IP-WAPP0002.coresit.msci.org/10.63.216.30:47500,
/10.62.44.30:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500,
/10.62.21.53:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1578570033978, loc=true, ver=2.7.6#20190911-sha1:21f7ca41,
isClient=false], done=true], topVer=null, durationFromInit=32]
[11:40:37,177][SEVERE][exchange-worker-#43][] Critical system error
detected. Will be handled accordingly to configured handler
[hnd=StopNodeFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=[]]], failureCtx=FailureContext [type=CRITICAL_ERROR,
err=java.lang.OutOfMemoryError]]
java.lang.OutOfMemoryError

        at sun.misc.Unsafe.allocateMemory(Native Method)
        at
org.apache.ignite.internal.util.GridUnsafe.allocateMemory(GridUnsafe.java:1205)
        at
org.apache.ignite.internal.mem.unsafe.UnsafeMemoryProvider.nextRegion(UnsafeMemoryProvider.java:103)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$6.nextRegion(GridCacheDatabaseSharedManager.java:1131)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.start(PageMemoryImpl.java:356)
        at
org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.startMemoryPolicies(IgniteCacheDatabaseSharedManager.java:224)
        at
org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.onActivate(IgniteCacheDatabaseSharedManager.java:1148)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.onActivate(GridCacheDatabaseSharedManager.java:709)
        at
org.apache.ignite.internal.processors.cache.GridCacheSharedContext.activate(GridCacheSharedContext.java:278)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:1082)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:736)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Unknown Source)
[11:40:37,192][WARNING][main][] GC server mode is not enabled, this could
lead to less than optimal performance on multi-core machines (to enable see
http://msdn.microsoft.com/en-us/library/ms229357(v=vs.110).aspx).
[11:40:37,192][INFO][main][PluginProcessor] Configured .NET plugins:
[11:40:37,192][WARNING][exchange-worker-#43][FailureProcessor] No deadlocked
threads detected.
[11:40:37,192][INFO][main][PluginProcessor]   ^-- None
[11:40:37,192][WARNING][exchange-worker-#43][FailureProcessor] Thread dump
at 2020/01/09 11:40:37 UTC
Thread [name="wal-file-archiver%null-#44", id=73, state=WAITING, blockCnt=0,
waitCnt=1]
    Lock
[object=o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileArchiver@1212090,
ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at
o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileArchiver.body(FsyncModeFileWriteAheadLogManager.java:1581)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Unknown Source)

Thread [name="session-timeout-worker-#34", id=50, state=TIMED_WAITING,
blockCnt=0, waitCnt=1]
        at java.lang.Thread.sleep(Native Method)
        at
o.a.i.i.processors.rest.GridRestProcessor$4.body(GridRestProcessor.java:495)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Unknown Source)

Thread [name="exchange-worker-#43", id=72, state=RUNNABLE, blockCnt=2,
waitCnt=1]
        at sun.management.ThreadImpl.dumpThreads0(Native Method)
        at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
        at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1368)
        at
o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:128)
        - locked o.a.i.i.processors.failure.FailureProcessor@f8f726
        at
o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:104)
        at
o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2567)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Unknown Source)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Tunas Tunas
Reply | Threaded
Open this post in threaded view
|

Re: After increasing maxsize in DataRegionConfiguration...Ignite throws out of memory exception

Can someone please give me some input. I am stuck on this since last 2 days.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: After increasing maxsize in DataRegionConfiguration...Ignite throws out of memory exception

Hello!

Please ensure that you are using 64 bit JVM. You may not be able to allocate large chunks of memory when using 32-bit VM:

>>> OS name: Windows Server 2008 R2 6.1 x86

Regards,
--
Ilya Kasnacheev


пт, 10 янв. 2020 г. в 08:45, Tunas <[hidden email]>:
Can someone please give me some input. I am stuck on this since last 2 days.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Tunas Tunas
Reply | Threaded
Open this post in threaded view
|

Re: After increasing maxsize in DataRegionConfiguration...Ignite throws out of memory exception

I am not allocating large memory. I just want to allocate at least 2gb of
RAM from available 22GB free space.

I will try with 64-bit JVM.
 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/