simple file based persistent store support TTL

classic Classic list List threaded Threaded
8 messages Options
shawn.du shawn.du
Reply | Threaded
Open this post in threaded view
|

simple file based persistent store support TTL

Hi,

I am looking for a solution for below simple use case:

#1 Cache entries are created with expiry policy
#2 When restart ignite servers, we can restore the caches which haven't been expired.

As far as my knowledge, we need implement a persistent store and use cache with read/write through and life cycle bean.
There are several approaches:

#1 use MySQL server, if we use this solution:
   a) for each cache, we need record TTL in a extra column.
   b) we need schedule some jobs to delete expired data periodically to avoid disk full.
   c)  when load caches, we need read the TTL column and determine whether the cache is expired.
#2 Use Databases which already support TTL, for example rocksdb. 
   a) RocksDB is library. if we start a RocksDb instance in each ignite server node, thus the persisted data will 
       resides in each server's local file system. I don't know what's the behavior when there are  multiple servers try to load caches. 
       also I don't know how to ensure the data is not lost and consistency when some node is down.

IMO,
 #1 is a bit verbose.
 #2 files based and auto TTL support is good, but need more codes and increase the complexity. 

please comments.  

Thanks
Shawn 

   




vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: simple file based persistent store support TTL

Hi Shawn,

Built in expiration is cache expiration, which by definition doesn't touch database. This means that you need to manually use remove() operation to delete from both cache and store. I think approach #1 is the best way to approach this.

-Val
shawn.du shawn.du
Reply | Threaded
Open this post in threaded view
|

Re: simple file based persistent store support TTL

thanks Val. 

On 02/2/2017 05:27, vkulichenko <[hidden email]> wrote:

Hi Shawn,

Built in expiration is cache expiration, which by definition doesn't touch
database. This means that you need to manually use remove() operation to
delete from both cache and store. I think approach #1 is the best way to
approach this.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/simple-file-based-persistent-store-support-TTL-tp10355p10365.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
shawn.du shawn.du
Reply | Threaded
Open this post in threaded view
|

Re: simple file based persistent store support TTL

In reply to this post by vkulichenko
Hi Val,

do you mean that when cache is expired, I have to call cache.remove(key) manually in order to 
call delete in cachestore?

I try to implement by remote listener like this:

ignite.events().remoteListen(null, new IgnitePredicate<CacheEvent>()
{
@IgniteInstanceResource
private Ignite ignite;

@LoggerResource
private IgniteLogger logger;

@Override
public boolean apply(CacheEvent event)
{
logger.info("remove cache  " + event.cacheName() + "for key " + event.key());
ignite.cache(event.cacheName()).remove(event.key());
return true;
}
}, EventType.EVT_CACHE_OBJECT_EXPIRED);

it seems that it doesn't work. I don't see any logs of calling it. what i missed?

Thanks
Shawn
On 02/2/2017 05:27[hidden email] wrote:
Hi Shawn,

Built in expiration is cache expiration, which by definition doesn't touch
database. This means that you need to manually use remove() operation to
delete from both cache and store. I think approach #1 is the best way to
approach this.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/simple-file-based-persistent-store-support-TTL-tp10355p10365.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: simple file based persistent store support TTL

Hi Shawn,

Did you enable the event (IgniteConfiguration.setIncludeEventTypes(..))?

-Val
shawn.du shawn.du
Reply | Threaded
Open this post in threaded view
|

Re: simple file based persistent store support TTL

Thanks Val, I forgot to enable them. I just got example code and run them.  Now it works.

Thanks
Shawn

On 02/14/2017 11:40[hidden email] wrote:
Hi Shawn,

Did you enable the event (IgniteConfiguration.setIncludeEventTypes(..))?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/simple-file-based-persistent-store-support-TTL-tp10355p10616.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
shawn.du shawn.du
Reply | Threaded
Open this post in threaded view
|

Re: simple file based persistent store support TTL

Hi,

after my code run almost one day, the server throwed many below exceptions, and all seems stop to work(cache,query and compute).

my code is rather simple. what's the possible reason? Thanks in advance.

public boolean apply(CacheEvent cacheEvent)
{
if (cacheEvent.type() == EventType.EVT_CACHE_OBJECT_EXPIRED)
{
logger.info("remove cache entry key:" + cacheEvent.key() + " in cache " + cacheEvent.cacheName());
ignite.cache(cacheEvent.cacheName()).remove(cacheEvent.key());
}
return true;
}

at org.apache.ignite.internal.GridEventConsumeHandler$2.onEvent(GridEventConsumeHandler.java:169)
        at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:770)
        at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:755)
        at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:295)
        at org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:297)
        at org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:204)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:3883)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.onTtlExpired(GridCacheMapEntry.java:3808)
        at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:139)
        at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:103)
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:920)
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leaveNoLock(GridCacheGateway.java:234)
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.leave(GridCacheGateway.java:219)
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.onLeave(IgniteCacheProxy.java:2287)
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.remove(IgniteCacheProxy.java:1476)
        at com.neulion.qos.ignite.listener.CacheEventListener.apply(CacheEventListener.java:29)


Thanks
Shawn

On 02/14/2017 11:44[hidden email] wrote:
Thanks Val, I forgot to enable them. I just got example code and run them.  Now it works.

Thanks
Shawn

On 02/14/2017 11:40[hidden email] wrote:
Hi Shawn,

Did you enable the event (IgniteConfiguration.setIncludeEventTypes(..))?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/simple-file-based-persistent-store-support-TTL-tp10355p10616.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: simple file based persistent store support TTL

What is the exception message? Your trace is truncated.

-Val