loading a cache into memory after startup

classic Classic list List threaded Threaded
4 messages Options
maheshkr76private maheshkr76private
Reply | Threaded
Open this post in threaded view
|

loading a cache into memory after startup

Hi,

After Ignite cluster (w. data, assuming about 50GB of data) starts up we are
experiencing significant delays before data is accessible.


Understandably, ignite is bringing data from backup on disk into memory
based on query.

is there a way, where we as much data as possible from backup (disk) into
memory (allocated in config XML, see below) at one shot after a cluster
start up?

*
                    <bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
                        <property name="persistenceEnabled" value="true"/>
                        <property name="metricsEnabled" value="true"/>
                        <property name="maxSize" value="#{16L * 1024 * 1024
* 1024}"/>
                    </bean>
                </property>
*

Right now, data seems to come into memory based on queries. So when we have
large tables with say 12 million records, some of the queries wait for about
3-4 minutes...

regards







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: loading a cache into memory after startup

Hi!

You can use a scan query, that will iterate over the whole dataset and bring it to memory. You don't need to perform any specific processing. Just touching it is enough for it to appear in memory.
In version 2.8 there will be a new API, allowing preloading partitions. IgniteCache.preloadPartition(...) will let you bring all data associated with the provided partition to memory.
Here is the JIRA ticket, that introduces this ability: https://issues.apache.org/jira/browse/IGNITE-10019

Denis

чт, 13 июн. 2019 г. в 07:44, mahesh76private <[hidden email]>:
Hi,

After Ignite cluster (w. data, assuming about 50GB of data) starts up we are
experiencing significant delays before data is accessible.


Understandably, ignite is bringing data from backup on disk into memory
based on query.

is there a way, where we as much data as possible from backup (disk) into
memory (allocated in config XML, see below) at one shot after a cluster
start up?

*
                    <bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
                        <property name="persistenceEnabled" value="true"/>
                        <property name="metricsEnabled" value="true"/>
                        <property name="maxSize" value="#{16L * 1024 * 1024
* 1024}"/>
                    </bean>
                </property>
*

Right now, data seems to come into memory based on queries. So when we have
large tables with say 12 million records, some of the queries wait for about
3-4 minutes...

regards







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: loading a cache into memory after startup

Denis, thanks for pointing out to that ticket.

Reopened that ticket proposing to reconsider the public API: https://issues.apache.org/jira/browse/IGNITE-8873

Generally, I'm in doubts that IgniteCache.preloadPartition is the right API from the usability standpoint. For instance, Mahesh do you understand how to use it for your case?

-
Denis


On Thu, Jun 13, 2019 at 3:23 AM Denis Mekhanikov <[hidden email]> wrote:
Hi!

You can use a scan query, that will iterate over the whole dataset and bring it to memory. You don't need to perform any specific processing. Just touching it is enough for it to appear in memory.
In version 2.8 there will be a new API, allowing preloading partitions. IgniteCache.preloadPartition(...) will let you bring all data associated with the provided partition to memory.
Here is the JIRA ticket, that introduces this ability: https://issues.apache.org/jira/browse/IGNITE-10019

Denis

чт, 13 июн. 2019 г. в 07:44, mahesh76private <[hidden email]>:
Hi,

After Ignite cluster (w. data, assuming about 50GB of data) starts up we are
experiencing significant delays before data is accessible.


Understandably, ignite is bringing data from backup on disk into memory
based on query.

is there a way, where we as much data as possible from backup (disk) into
memory (allocated in config XML, see below) at one shot after a cluster
start up?

*
                    <bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
                        <property name="persistenceEnabled" value="true"/>
                        <property name="metricsEnabled" value="true"/>
                        <property name="maxSize" value="#{16L * 1024 * 1024
* 1024}"/>
                    </bean>
                </property>
*

Right now, data seems to come into memory based on queries. So when we have
large tables with say 12 million records, some of the queries wait for about
3-4 minutes...

regards







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
maheshkr76private maheshkr76private
Reply | Threaded
Open this post in threaded view
|

Re: loading a cache into memory after startup

I don't know if I even want to call an api to warm up memory. I am thinking,
we could have a setting like the below in config xml, this is userfriendly
and intention of the config file is very clear.

Bottomline: if ignite is used more often from disk, then it loses its value
from a performance standpoint
So, this option (warmupOnStart)  seems mandatory.  

        <property name="dataStorageConfiguration">
            <bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
                <property name="defaultDataRegionConfiguration">
                    <bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
                        <property name="persistenceEnabled" value="true"/>
                        <property name="metricsEnabled" value="true"/>
                        <property name="maxSize" value="#{16L * 1024 * 1024
* 1024}"/>
                         *<property name="warmupOnStart" value="true" />*
                    </bean>
                </property>
            </bean>
        </property>



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/