Cache expiry policy not deleting records from disk(native persistence)

classic Classic list List threaded Threaded
8 messages Options
shivakumar shivakumar
Reply | Threaded
Open this post in threaded view
|

Cache expiry policy not deleting records from disk(native persistence)

Hi all,
I have set cache expiry policy like this


   </property>
   <property name="cacheConfiguration">
            <list>
                <bean id="cache-template-bean" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">
                  <property name="name" value="templateEternal*"/>
                  <property name="cacheMode" value="PARTITIONED"/>
                  <property name="backups" value="1"/>
                  <property name="groupName" value="groupEternal"/>
                  <property name="expiryPolicyFactory">
                    <bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
                      <constructor-arg>
                        <bean class="javax.cache.expiry.Duration">
                          <constructor-arg value="MINUTES"/>
                          <constructor-arg value="10"/>
                        </bean>
                      </constructor-arg>
                    </bean>
                  </property>

                </bean>
            </list>
   </property>


And batch inserting records to one of the table which is created with above cache template.
Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes records started reducing(expiring) when I monitored from sqlline.

0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

248896
--------------------------------
1 row selected (0.86 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

222174
--------------------------------
1 row selected (0.313 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

118154
--------------------------------
1 row selected (0.15 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

76061
--------------------------------
1 row selected (0.106 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

41671
--------------------------------
1 row selected (0.063 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

18455
--------------------------------
1 row selected (0.037 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

0
--------------------------------
1 row selected (0.014 seconds)

 
But in the meantime, the disk space used by the persistence store was in the same usage level instead of decreasing.
 

[ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h /opt/ignite/persistence/; sleep 1s; done
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence

 

This means that expiry policy not deleting records from the disk, but ignite document says when expiry policy is set and native persistence is enabled then it deletes records from disk as well.
Am I missing some configuration?
Any help is appreciated.

Shiva  
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Cache expiry policy not deleting records from disk(native persistence)

Hello,

I guess that generated WAL will take this disk space. Please read about WAL here:

https://apacheignite.readme.io/docs/write-ahead-log

Please provide the size of every folder under /opt/ignite/persistence.

BR,
Andrei

9/6/2019 9:45 PM, Shiva Kumar пишет:
Hi all,
I have set cache expiry policy like this


   </property>
   <property name="cacheConfiguration">
            <list>
                <bean id="cache-template-bean" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">
                  <property name="name" value="templateEternal*"/>
                  <property name="cacheMode" value="PARTITIONED"/>
                  <property name="backups" value="1"/>
                  <property name="groupName" value="groupEternal"/>
                  <property name="expiryPolicyFactory">
                    <bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
                      <constructor-arg>
                        <bean class="javax.cache.expiry.Duration">
                          <constructor-arg value="MINUTES"/>
                          <constructor-arg value="10"/>
                        </bean>
                      </constructor-arg>
                    </bean>
                  </property>

                </bean>
            </list>
   </property>


And batch inserting records to one of the table which is created with above cache template.
Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes records started reducing(expiring) when I monitored from sqlline.

0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

248896
--------------------------------
1 row selected (0.86 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

222174
--------------------------------
1 row selected (0.313 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

118154
--------------------------------
1 row selected (0.15 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

76061
--------------------------------
1 row selected (0.106 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

41671
--------------------------------
1 row selected (0.063 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

18455
--------------------------------
1 row selected (0.037 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

0
--------------------------------
1 row selected (0.014 seconds)

 
But in the meantime, the disk space used by the persistence store was in the same usage level instead of decreasing.
 

[ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h /opt/ignite/persistence/; sleep 1s; done
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence

 

This means that expiry policy not deleting records from the disk, but ignite document says when expiry policy is set and native persistence is enabled then it deletes records from disk as well.
Am I missing some configuration?
Any help is appreciated.

Shiva  
shivakumar shivakumar
Reply | Threaded
Open this post in threaded view
|

Re: Cache expiry policy not deleting records from disk(native persistence)

Hi
I have deployed ignite on kubernetes and configured two seperate persistent volume for WAL and persistence.
The issue Iam facing is same as 

Thanks
Shiva

On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov, <[hidden email]> wrote:

Hello,

I guess that generated WAL will take this disk space. Please read about WAL here:

https://apacheignite.readme.io/docs/write-ahead-log

Please provide the size of every folder under /opt/ignite/persistence.

BR,
Andrei

9/6/2019 9:45 PM, Shiva Kumar пишет:
Hi all,
I have set cache expiry policy like this


   </property>
   <property name="cacheConfiguration">
            <list>
                <bean id="cache-template-bean" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">
                  <property name="name" value="templateEternal*"/>
                  <property name="cacheMode" value="PARTITIONED"/>
                  <property name="backups" value="1"/>
                  <property name="groupName" value="groupEternal"/>
                  <property name="expiryPolicyFactory">
                    <bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
                      <constructor-arg>
                        <bean class="javax.cache.expiry.Duration">
                          <constructor-arg value="MINUTES"/>
                          <constructor-arg value="10"/>
                        </bean>
                      </constructor-arg>
                    </bean>
                  </property>

                </bean>
            </list>
   </property>


And batch inserting records to one of the table which is created with above cache template.
Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes records started reducing(expiring) when I monitored from sqlline.

0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

248896
--------------------------------
1 row selected (0.86 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

222174
--------------------------------
1 row selected (0.313 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

118154
--------------------------------
1 row selected (0.15 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

76061
--------------------------------
1 row selected (0.106 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

41671
--------------------------------
1 row selected (0.063 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

18455
--------------------------------
1 row selected (0.037 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

0
--------------------------------
1 row selected (0.014 seconds)

 
But in the meantime, the disk space used by the persistence store was in the same usage level instead of decreasing.
 

[ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h /opt/ignite/persistence/; sleep 1s; done
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence

 

This means that expiry policy not deleting records from the disk, but ignite document says when expiry policy is set and native persistence is enabled then it deletes records from disk as well.
Am I missing some configuration?
Any help is appreciated.

Shiva  
shivakumar shivakumar
Reply | Threaded
Open this post in threaded view
|

Re: Cache expiry policy not deleting records from disk(native persistence)

I have filed a bug https://issues.apache.org/jira/browse/IGNITE-12152 but this is same as https://issues.apache.org/jira/browse/IGNITE-10862
Any idea on the timeline of these tickets?
In the documentation https://apacheignite.readme.io/v2.7/docs/expiry-policies 
it says when native persistence is enabled "expired entries are removed from both memory and disk tiers" but in the disk it just mark the pages as unwanted pages and same disk space used by these unwanted pages will be used to store new pages but it will not remove unwanted pages from disk and so it will not release disk space used by these unwanted pages.

here is the developer's discussion link
http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html  

On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar <[hidden email]> wrote:
Hi
I have deployed ignite on kubernetes and configured two seperate persistent volume for WAL and persistence.
The issue Iam facing is same as 

Thanks
Shiva

On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov, <[hidden email]> wrote:

Hello,

I guess that generated WAL will take this disk space. Please read about WAL here:

https://apacheignite.readme.io/docs/write-ahead-log

Please provide the size of every folder under /opt/ignite/persistence.

BR,
Andrei

9/6/2019 9:45 PM, Shiva Kumar пишет:
Hi all,
I have set cache expiry policy like this


   </property>
   <property name="cacheConfiguration">
            <list>
                <bean id="cache-template-bean" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">
                  <property name="name" value="templateEternal*"/>
                  <property name="cacheMode" value="PARTITIONED"/>
                  <property name="backups" value="1"/>
                  <property name="groupName" value="groupEternal"/>
                  <property name="expiryPolicyFactory">
                    <bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
                      <constructor-arg>
                        <bean class="javax.cache.expiry.Duration">
                          <constructor-arg value="MINUTES"/>
                          <constructor-arg value="10"/>
                        </bean>
                      </constructor-arg>
                    </bean>
                  </property>

                </bean>
            </list>
   </property>


And batch inserting records to one of the table which is created with above cache template.
Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes records started reducing(expiring) when I monitored from sqlline.

0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

248896
--------------------------------
1 row selected (0.86 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

222174
--------------------------------
1 row selected (0.313 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

118154
--------------------------------
1 row selected (0.15 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

76061
--------------------------------
1 row selected (0.106 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

41671
--------------------------------
1 row selected (0.063 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

18455
--------------------------------
1 row selected (0.037 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

0
--------------------------------
1 row selected (0.014 seconds)

 
But in the meantime, the disk space used by the persistence store was in the same usage level instead of decreasing.
 

[ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h /opt/ignite/persistence/; sleep 1s; done
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence

 

This means that expiry policy not deleting records from the disk, but ignite document says when expiry policy is set and native persistence is enabled then it deletes records from disk as well.
Am I missing some configuration?
Any help is appreciated.

Shiva  
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Cache expiry policy not deleting records from disk(native persistence)

Shiva, 

Hopefully, someone from the dev community will pick this ticket up soon and solve the task. In the meantime, Artem, would you mind documenting this limitation referring to ticket 10862?

-
Denis


On Tue, Sep 10, 2019 at 12:50 AM Shiva Kumar <[hidden email]> wrote:
I have filed a bug https://issues.apache.org/jira/browse/IGNITE-12152 but this is same as https://issues.apache.org/jira/browse/IGNITE-10862
Any idea on the timeline of these tickets?
In the documentation https://apacheignite.readme.io/v2.7/docs/expiry-policies 
it says when native persistence is enabled "expired entries are removed from both memory and disk tiers" but in the disk it just mark the pages as unwanted pages and same disk space used by these unwanted pages will be used to store new pages but it will not remove unwanted pages from disk and so it will not release disk space used by these unwanted pages.

here is the developer's discussion link
http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html  

On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar <[hidden email]> wrote:
Hi
I have deployed ignite on kubernetes and configured two seperate persistent volume for WAL and persistence.
The issue Iam facing is same as 

Thanks
Shiva

On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov, <[hidden email]> wrote:

Hello,

I guess that generated WAL will take this disk space. Please read about WAL here:

https://apacheignite.readme.io/docs/write-ahead-log

Please provide the size of every folder under /opt/ignite/persistence.

BR,
Andrei

9/6/2019 9:45 PM, Shiva Kumar пишет:
Hi all,
I have set cache expiry policy like this


   </property>
   <property name="cacheConfiguration">
            <list>
                <bean id="cache-template-bean" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">
                  <property name="name" value="templateEternal*"/>
                  <property name="cacheMode" value="PARTITIONED"/>
                  <property name="backups" value="1"/>
                  <property name="groupName" value="groupEternal"/>
                  <property name="expiryPolicyFactory">
                    <bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
                      <constructor-arg>
                        <bean class="javax.cache.expiry.Duration">
                          <constructor-arg value="MINUTES"/>
                          <constructor-arg value="10"/>
                        </bean>
                      </constructor-arg>
                    </bean>
                  </property>

                </bean>
            </list>
   </property>


And batch inserting records to one of the table which is created with above cache template.
Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes records started reducing(expiring) when I monitored from sqlline.

0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

248896
--------------------------------
1 row selected (0.86 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

222174
--------------------------------
1 row selected (0.313 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

118154
--------------------------------
1 row selected (0.15 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

76061
--------------------------------
1 row selected (0.106 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

41671
--------------------------------
1 row selected (0.063 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

18455
--------------------------------
1 row selected (0.037 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

0
--------------------------------
1 row selected (0.014 seconds)

 
But in the meantime, the disk space used by the persistence store was in the same usage level instead of decreasing.
 

[ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h /opt/ignite/persistence/; sleep 1s; done
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence

 

This means that expiry policy not deleting records from the disk, but ignite document says when expiry policy is set and native persistence is enabled then it deletes records from disk as well.
Am I missing some configuration?
Any help is appreciated.

Shiva  
rick_tem rick_tem
Reply | Threaded
Open this post in threaded view
|

Re: Cache expiry policy not deleting records from disk(native persistence)

This is very troublesome.  Persistence will kill your disk and LRU doesn't
seem to work for offheap with persistence disabled (see "Data region LRU
offheap algo not working")....It seems the only way to expire items is to
keep it on heap with an expiry policy....

Regards,
Rick



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Artem Budnikov Artem Budnikov
Reply | Threaded
Open this post in threaded view
|

Re: Cache expiry policy not deleting records from disk(native persistence)

In reply to this post by dmagda

Hi Denis,

I updated the page about eviction policies. Not freeing up space on disk was not implemented for reasons explained in the dev-list thread. I'll update the page once more if/when a solution is implemented.

Artem

On 13.09.2019 00:34, Denis Magda wrote:
Shiva, 

Hopefully, someone from the dev community will pick this ticket up soon and solve the task. In the meantime, Artem, would you mind documenting this limitation referring to ticket 10862?

-
Denis


On Tue, Sep 10, 2019 at 12:50 AM Shiva Kumar <[hidden email]> wrote:
I have filed a bug https://issues.apache.org/jira/browse/IGNITE-12152 but this is same as https://issues.apache.org/jira/browse/IGNITE-10862
Any idea on the timeline of these tickets?
In the documentation https://apacheignite.readme.io/v2.7/docs/expiry-policies 
it says when native persistence is enabled "expired entries are removed from both memory and disk tiers" but in the disk it just mark the pages as unwanted pages and same disk space used by these unwanted pages will be used to store new pages but it will not remove unwanted pages from disk and so it will not release disk space used by these unwanted pages.

here is the developer's discussion link
http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html  

On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar <[hidden email]> wrote:
Hi
I have deployed ignite on kubernetes and configured two seperate persistent volume for WAL and persistence.
The issue Iam facing is same as 

Thanks
Shiva

On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov, <[hidden email]> wrote:

Hello,

I guess that generated WAL will take this disk space. Please read about WAL here:

https://apacheignite.readme.io/docs/write-ahead-log

Please provide the size of every folder under /opt/ignite/persistence.

BR,
Andrei

9/6/2019 9:45 PM, Shiva Kumar пишет:
Hi all,
I have set cache expiry policy like this


   </property>
   <property name="cacheConfiguration">
            <list>
                <bean id="cache-template-bean" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">
                  <property name="name" value="templateEternal*"/>
                  <property name="cacheMode" value="PARTITIONED"/>
                  <property name="backups" value="1"/>
                  <property name="groupName" value="groupEternal"/>
                  <property name="expiryPolicyFactory">
                    <bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
                      <constructor-arg>
                        <bean class="javax.cache.expiry.Duration">
                          <constructor-arg value="MINUTES"/>
                          <constructor-arg value="10"/>
                        </bean>
                      </constructor-arg>
                    </bean>
                  </property>

                </bean>
            </list>
   </property>


And batch inserting records to one of the table which is created with above cache template.
Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes records started reducing(expiring) when I monitored from sqlline.

0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

248896
--------------------------------
1 row selected (0.86 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

222174
--------------------------------
1 row selected (0.313 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

118154
--------------------------------
1 row selected (0.15 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

76061
--------------------------------
1 row selected (0.106 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

41671
--------------------------------
1 row selected (0.063 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

18455
--------------------------------
1 row selected (0.037 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

0
--------------------------------
1 row selected (0.014 seconds)

 
But in the meantime, the disk space used by the persistence store was in the same usage level instead of decreasing.
 

[ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h /opt/ignite/persistence/; sleep 1s; done
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence

 

This means that expiry policy not deleting records from the disk, but ignite document says when expiry policy is set and native persistence is enabled then it deletes records from disk as well.
Am I missing some configuration?
Any help is appreciated.

Shiva  
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Cache expiry policy not deleting records from disk(native persistence)

Artem, thanks, could you please share a reference to the updated page? Can't find anything here:

Shiva, I've restarted the discussion on the dev list, to get to the bottom of this gap and how it can be addressed:

-
Denis


On Fri, Sep 13, 2019 at 7:05 AM Artem Budnikov <[hidden email]> wrote:

Hi Denis,

I updated the page about eviction policies. Not freeing up space on disk was not implemented for reasons explained in the dev-list thread. I'll update the page once more if/when a solution is implemented.

Artem

On 13.09.2019 00:34, Denis Magda wrote:
Shiva, 

Hopefully, someone from the dev community will pick this ticket up soon and solve the task. In the meantime, Artem, would you mind documenting this limitation referring to ticket 10862?

-
Denis


On Tue, Sep 10, 2019 at 12:50 AM Shiva Kumar <[hidden email]> wrote:
I have filed a bug https://issues.apache.org/jira/browse/IGNITE-12152 but this is same as https://issues.apache.org/jira/browse/IGNITE-10862
Any idea on the timeline of these tickets?
In the documentation https://apacheignite.readme.io/v2.7/docs/expiry-policies 
it says when native persistence is enabled "expired entries are removed from both memory and disk tiers" but in the disk it just mark the pages as unwanted pages and same disk space used by these unwanted pages will be used to store new pages but it will not remove unwanted pages from disk and so it will not release disk space used by these unwanted pages.

here is the developer's discussion link
http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html  

On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar <[hidden email]> wrote:
Hi
I have deployed ignite on kubernetes and configured two seperate persistent volume for WAL and persistence.
The issue Iam facing is same as 

Thanks
Shiva

On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov, <[hidden email]> wrote:

Hello,

I guess that generated WAL will take this disk space. Please read about WAL here:

https://apacheignite.readme.io/docs/write-ahead-log

Please provide the size of every folder under /opt/ignite/persistence.

BR,
Andrei

9/6/2019 9:45 PM, Shiva Kumar пишет:
Hi all,
I have set cache expiry policy like this


   </property>
   <property name="cacheConfiguration">
            <list>
                <bean id="cache-template-bean" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">
                  <property name="name" value="templateEternal*"/>
                  <property name="cacheMode" value="PARTITIONED"/>
                  <property name="backups" value="1"/>
                  <property name="groupName" value="groupEternal"/>
                  <property name="expiryPolicyFactory">
                    <bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
                      <constructor-arg>
                        <bean class="javax.cache.expiry.Duration">
                          <constructor-arg value="MINUTES"/>
                          <constructor-arg value="10"/>
                        </bean>
                      </constructor-arg>
                    </bean>
                  </property>

                </bean>
            </list>
   </property>


And batch inserting records to one of the table which is created with above cache template.
Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes records started reducing(expiring) when I monitored from sqlline.

0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

248896
--------------------------------
1 row selected (0.86 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

222174
--------------------------------
1 row selected (0.313 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

118154
--------------------------------
1 row selected (0.15 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

76061
--------------------------------
1 row selected (0.106 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

41671
--------------------------------
1 row selected (0.063 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

18455
--------------------------------
1 row selected (0.037 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
--------------------------------

COUNT(ID)
--------------------------------

0
--------------------------------
1 row selected (0.014 seconds)

 
But in the meantime, the disk space used by the persistence store was in the same usage level instead of decreasing.
 

[ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h /opt/ignite/persistence/; sleep 1s; done
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence

 

This means that expiry policy not deleting records from the disk, but ignite document says when expiry policy is set and native persistence is enabled then it deletes records from disk as well.
Am I missing some configuration?
Any help is appreciated.

Shiva