heap dump on ignite servers

classic Classic list List threaded Threaded
5 messages Options
ignite_user2016 ignite_user2016
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

heap dump on ignite servers

Hello Igniters / Val,

Since last few weeks, we are seeing out of memory issues on ignite servers, today I was able to capture the heapdumps and that is pointing to following classes -

Our version of Ignite is 1.7.0.

15 instances of "org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache", loaded by "sun.misc.Launcher$AppClassLoader @ 0xc0029998" occupy 453,929,512 (76.41%) bytes.

Biggest instances:
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc426b868 - 34,931,744 (5.88%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc2f94e28 - 34,745,904 (5.85%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc4a0ffc8 - 34,745,896 (5.85%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc2e41e10 - 34,642,904 (5.83%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc100a648 - 34,376,920 (5.79%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc3da2ac0 - 34,376,288 (5.79%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc32991b0 - 34,376,208 (5.79%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc427d798 - 34,376,008 (5.79%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc57cbd28 - 34,270,952 (5.77%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc0a22ae8 - 34,191,040 (5.76%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc0d8ade8 - 34,190,960 (5.76%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc38a3618 - 34,190,920 (5.76%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache @ 0xc50d0ca8 - 33,900,352 (5.71%) bytes.


Keywords
sun.misc.Launcher$AppClassLoader @ 0xc0029998
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache

we are running ignite on 1G memory and dont use ignite for any computing.

Our configuration is based on ip cluster that is as follows -

<?xml version="1.0" encoding="UTF-8"?>




<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/util
        http://www.springframework.org/schema/util/spring-util.xsd">




    <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

        <property name="gridName" value="GRID_NAME"/>

       
        <property name="peerClassLoadingEnabled" value="true"/>
       
        <property name="clientMode" value="false"/>

     
        <property name="gridLogger">
            <bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
                <constructor-arg type="java.lang.String" value="config/log4j2.xml"/>
            </bean>
        </property>

       
        <property name="includeEventTypes">
            <list>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_CLIENT_NODE_RECONNECTED"/>
            </list>
        </property>

       
        <property name="cacheConfiguration">
            <list>
               
               
                   
                   
                   
                   
                   
               
            </list>
        </property>

     

        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                   
                   
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                       
                        <property name="addresses">
                            <list>
                               
                                <value>HOST_IP1:47500..47509</value>
                                <value>HOST_IP2:47500..47509</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>

    </bean>
</beans>


 
what am I missing here ?

thanks for all your help..
Andrew Mashenkov Andrew Mashenkov
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: heap dump on ignite servers

Hi,

Looks like we have a more optimal code in latest versions. See IGNITE-4154 [1]
Try to update to >= 1.8 version.

As a workaround you can try to reduce discovery history size with JVM option IGNITE_AFFINITY_HISTORY_SIZE, which is 500 by default.


On Wed, Jul 26, 2017 at 7:30 AM, ignite_user2016 <[hidden email]> wrote:
Hello Igniters / Val,

Since last few weeks, we are seeing out of memory issues on ignite servers,
today I was able to capture the heapdumps and that is pointing to following
classes -

Our version of Ignite is 1.7.0.

15 instances of
"org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache",
loaded by "sun.misc.Launcher$AppClassLoader @ 0xc0029998" occupy 453,929,512
(76.41%) bytes.

Biggest instances:
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc426b868 - 34,931,744 (5.88%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc2f94e28 - 34,745,904 (5.85%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc4a0ffc8 - 34,745,896 (5.85%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc2e41e10 - 34,642,904 (5.83%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc100a648 - 34,376,920 (5.79%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc3da2ac0 - 34,376,288 (5.79%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc32991b0 - 34,376,208 (5.79%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc427d798 - 34,376,008 (5.79%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc57cbd28 - 34,270,952 (5.77%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc0a22ae8 - 34,191,040 (5.76%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc0d8ade8 - 34,190,960 (5.76%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc38a3618 - 34,190,920 (5.76%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc50d0ca8 - 33,900,352 (5.71%) bytes.


Keywords
sun.misc.Launcher$AppClassLoader @ 0xc0029998
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache

we are running ignite on 1G memory and dont use ignite for any computing.

Our configuration is based on ip cluster that is as follows -

<?xml version="1.0" encoding="UTF-8"?>




<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/util
        http://www.springframework.org/schema/util/spring-util.xsd">




    <bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">

        <property name="gridName" value="GRID_NAME"/>


        <property name="peerClassLoadingEnabled" value="true"/>

        <property name="clientMode" value="false"/>


        <property name="gridLogger">
            <bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
                <constructor-arg type="java.lang.String"
value="config/log4j2.xml"/>
            </bean>
        </property>


        <property name="includeEventTypes">
            <list>
                <util:constant
static-field="org.apache.ignite.events.EventType.EVT_CLIENT_NODE_RECONNECTED"/>
            </list>
        </property>


        <property name="cacheConfiguration">
            <list>








            </list>
        </property>



        <property name="discoverySpi">
            <bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">


                    <bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">

                        <property name="addresses">
                            <list>

                                <value>HOST_IP1:47500..47509</value>
                                <value>HOST_IP2:47500..47509</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>

    </bean>
</beans>



what am I missing here ?

thanks for all your help..




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/heap-dump-on-ignite-servers-tp15679.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--
Best regards,
Andrey V. Mashenkov
Regards, Andrew.
ignite_user2016 ignite_user2016
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: heap dump on ignite servers

Thank you Andrew.. we are upgrading Ignite to 2.0 so hopefully we wont have such issues.



On Wed, Jul 26, 2017 at 10:35 AM, Andrew Mashenkov [via Apache Ignite Users] <[hidden email]> wrote:
Hi,

Looks like we have a more optimal code in latest versions. See IGNITE-4154 [1]
Try to update to >= 1.8 version.

As a workaround you can try to reduce discovery history size with JVM option IGNITE_AFFINITY_HISTORY_SIZE, which is 500 by default.


On Wed, Jul 26, 2017 at 7:30 AM, ignite_user2016 <[hidden email]> wrote:
Hello Igniters / Val,

Since last few weeks, we are seeing out of memory issues on ignite servers,
today I was able to capture the heapdumps and that is pointing to following
classes -

Our version of Ignite is 1.7.0.

15 instances of
"org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache",
loaded by "sun.misc.Launcher$AppClassLoader @ 0xc0029998" occupy 453,929,512
(76.41%) bytes.

Biggest instances:
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc426b868 - 34,931,744 (5.88%) bytes.
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc2f94e28 - 34,745,904 (5.85%) bytes.
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc4a0ffc8 - 34,745,896 (5.85%) bytes.
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc2e41e10 - 34,642,904 (5.83%) bytes.
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc100a648 - 34,376,920 (5.79%) bytes.
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc3da2ac0 - 34,376,288 (5.79%) bytes.
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc32991b0 - 34,376,208 (5.79%) bytes.
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc427d798 - 34,376,008 (5.79%) bytes.
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc57cbd28 - 34,270,952 (5.77%) bytes.
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc0a22ae8 - 34,191,040 (5.76%) bytes.
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc0d8ade8 - 34,190,960 (5.76%) bytes.
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc38a3618 - 34,190,920 (5.76%) bytes.
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc50d0ca8 - 33,900,352 (5.71%) bytes.


Keywords
sun.misc.Launcher$AppClassLoader @ 0xc0029998
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache

we are running ignite on 1G memory and dont use ignite for any computing.

Our configuration is based on ip cluster that is as follows -

<?xml version="1.0" encoding="UTF-8"?>




<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/util
        http://www.springframework.org/schema/util/spring-util.xsd">




    <bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">

        <property name="gridName" value="GRID_NAME"/>


        <property name="peerClassLoadingEnabled" value="true"/>

        <property name="clientMode" value="false"/>


        <property name="gridLogger">
            <bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
                <constructor-arg type="java.lang.String"
value="config/log4j2.xml"/>
            </bean>
        </property>


        <property name="includeEventTypes">
            <list>
                <util:constant
static-field="org.apache.ignite.events.EventType.EVT_CLIENT_NODE_RECONNECTED"/>
            </list>
        </property>


        <property name="cacheConfiguration">
            <list>








            </list>
        </property>



        <property name="discoverySpi">
            <bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">


                    <bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">

                        <property name="addresses">
                            <list>

                                <value>HOST_IP1:47500..47509</value>
                                <value>HOST_IP2:47500..47509</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>

    </bean>
</beans>



what am I missing here ?

thanks for all your help..




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/heap-dump-on-ignite-servers-tp15679.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.



--
Best regards,
Andrey V. Mashenkov
Regards, Andrew.



If you reply to this email, your message will be added to the discussion below:
http://apache-ignite-users.70518.x6.nabble.com/heap-dump-on-ignite-servers-tp15679p15695.html
To start a new topic under Apache Ignite Users, email [hidden email]
To unsubscribe from Apache Ignite Users, click here.
NAML



--
Rishi Yagnik
ignite_user2016 ignite_user2016
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: heap dump on ignite servers

I also applied the IGNITE_AFFINITY_HISTORY_SIZE= 500 on Ignite 1.7 version but ignite heap keeps rising and eventually runs out of memory.


our use case is more with spring caching and we use ignite for spring 2nd level cache.


Is there any setting on cache that can reduce the memory footprint ? OR is there some kind of leak here ?

Please provide your guidance here.

Thanks..



ignite_user2016 ignite_user2016
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: heap dump on ignite servers

any help here ?
Loading...