How much data is actually in RAM?

classic Classic list List threaded Threaded
3 messages Options
javadevmtl javadevmtl
Reply | Threaded
Open this post in threaded view
|

How much data is actually in RAM?

Hi, so I have 3 machines with 8GB RAM and 96GB disk each.

I have configured the persistence as  

            <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
              <property name="persistenceEnabled" value="true"/>

              <property name="name" value="Default_Region"/>
              <property name="maxSize" value="#{72L * 1024 * 1024 * 1024}"/>
            </bean>
          </property>

Looking at the logs:
Topology snapshot [ver=3, locNode=xxxxxx, servers=3, clients=0, state=INACTIVE, CPUs=12, offheap=216.0GB, heap=3.0GB] 

1- If I get this right... Each node is using about 1GB on-heap JVM memory each for the ignite process to run. Correct? Should this be set a bit higher or that's ok?
2- There is total 216GB off-heap so about 72GB off-heap per node. So how much of that is stored in actual RAM in the remainder of what is left from 8GB of the physical host? Is that the 20% value indicated in the docs? Or should maxSize be set to 6L and not 72L giving 2GB free to the OS and ignite process?

 


shaneduan shaneduan
Reply | Threaded
Open this post in threaded view
|

Re: How much data is actually in RAM?

It guess really depends how you use the cluster. For my use case, I have to bump up heap to 8G to support concurrent queries (JDBC); otherwise, Ignite could crash since Ignite need process results in heap. 

On Fri, Jun 14, 2019 at 2:02 PM John Smith <[hidden email]> wrote:
Hi, so I have 3 machines with 8GB RAM and 96GB disk each.

I have configured the persistence as  

            <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
              <property name="persistenceEnabled" value="true"/>

              <property name="name" value="Default_Region"/>
              <property name="maxSize" value="#{72L * 1024 * 1024 * 1024}"/>
            </bean>
          </property>

Looking at the logs:
Topology snapshot [ver=3, locNode=xxxxxx, servers=3, clients=0, state=INACTIVE, CPUs=12, offheap=216.0GB, heap=3.0GB] 

1- If I get this right... Each node is using about 1GB on-heap JVM memory each for the ignite process to run. Correct? Should this be set a bit higher or that's ok?
2- There is total 216GB off-heap so about 72GB off-heap per node. So how much of that is stored in actual RAM in the remainder of what is left from 8GB of the physical host? Is that the 20% value indicated in the docs? Or should maxSize be set to 6L and not 72L giving 2GB free to the OS and ignite process?

 


Mikhail Mikhail
Reply | Threaded
Open this post in threaded view
|

Re: How much data is actually in RAM?

In reply to this post by javadevmtl
Hi John,

if you have native persistence enabled that ignite will evict data to a disk if there's no space in RAM.
BUT you have to specify maxSize for RAM! so Ignite really doesn't have a limit for space, it's just limited by your disk and you will get IgniteOOM at some point.

So your current configuration should fail because Ignite will try to allocate 72 GB in memory on start.

>1- If I get this right... Each node is using about 1GB on-heap JVM memory each for the ignite process to run. Correct? Should this be set a bit higher or that's ok?
Ignite store data in off heap, but heap will be used for everything else, like message processing, marshaling and unmarshaling data and etc.
If you use SQL, ignite will load data from off heap to on heap to send record as a response to client. Usually SQL the main consumer of heap, so you need to tune you heap size based on your needs.

>2- There is total 216GB off-heap so about 72GB off-heap per node. So how much of that is stored in actual RAM in the remainder 
>of what is left from 8GB of the physical host? Is that the 20% value indicated in the docs? Or should maxSize be set to 6L and not 72L giving 2GB free to the OS and ignite process?
As I said, this option defines not max size for storage, but the max size for storage in RAM! Ignite always will use RAM memory that you allocate to it for data storage, only if it's not enough memory for data storage it will evict data to a disk.

Thanks,
Mike.


пт, 14 июн. 2019 г. в 14:02, John Smith <[hidden email]>:
Hi, so I have 3 machines with 8GB RAM and 96GB disk each.

I have configured the persistence as  

            <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
              <property name="persistenceEnabled" value="true"/>

              <property name="name" value="Default_Region"/>
              <property name="maxSize" value="#{72L * 1024 * 1024 * 1024}"/>
            </bean>
          </property>

Looking at the logs:
Topology snapshot [ver=3, locNode=xxxxxx, servers=3, clients=0, state=INACTIVE, CPUs=12, offheap=216.0GB, heap=3.0GB] 

1- If I get this right... Each node is using about 1GB on-heap JVM memory each for the ignite process to run. Correct? Should this be set a bit higher or that's ok?
2- There is total 216GB off-heap so about 72GB off-heap per node. So how much of that is stored in actual RAM in the remainder of what is left from 8GB of the physical host? Is that the 20% value indicated in the docs? Or should maxSize be set to 6L and not 72L giving 2GB free to the OS and ignite process?