Update all data in cache spend a long time

classic Classic list List threaded Threaded
5 messages Options
Justin Ji Justin Ji
Reply | Threaded
Open this post in threaded view
|

Update all data in cache spend a long time

Hi -

I have a cache with more than 1 million records, when I update the whole
records in it, it spends a long time(almost 6 minutes).

Here is the SQL:
"update " + IgniteTableKey.T_DEVICE_ONLINE_STATUS.getCode()+ " set
isOnline=0, mqttTime=" + System.currentTimeMillis() / 1000 + " where 1=1";

And the Cache configuration is :

CacheConfiguration<K, V> cacheCfg = new CacheConfiguration<>();
        cacheCfg.setName(cacheName);
        cacheCfg.setCacheMode(CacheMode.PARTITIONED);
        cacheCfg.setBackups(1);
        cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
       
cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DeviceStatusCacheStore.class));
        cacheCfg.setWriteThrough(true);
        cacheCfg.setWriteBehindEnabled(true);
        cacheCfg.setReadThrough(true);
        cacheCfg.setWriteBehindFlushThreadCount(4);
        cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
        cacheCfg.setWriteBehindFlushSize(409600);
        cacheCfg.setWriteBehindBatchSize(1024);
        cacheCfg.setStoreKeepBinary(true);
        cfg.setCacheConfiguration(cacheCfg);

So I want to know is there a way that can speed up the execution?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Update all data in cache spend a long time

Hello!

Have you tried setting queryParallelism on this cache to some high value (such as your number of CPU cores)?

Note that this query will likely cause lifting all data in cache to heap at once, which is probably not what you expect. My recommendation is also to try splitting this operation into smaller ones, see if they finish faster.

Regards,
--
Ilya Kasnacheev


вт, 9 окт. 2018 г. в 15:47, Justin Ji <[hidden email]>:
Hi -

I have a cache with more than 1 million records, when I update the whole
records in it, it spends a long time(almost 6 minutes).

Here is the SQL:
"update " + IgniteTableKey.T_DEVICE_ONLINE_STATUS.getCode()+ " set
isOnline=0, mqttTime=" + System.currentTimeMillis() / 1000 + " where 1=1";

And the Cache configuration is :

CacheConfiguration<K, V> cacheCfg = new CacheConfiguration<>();
        cacheCfg.setName(cacheName);
        cacheCfg.setCacheMode(CacheMode.PARTITIONED);
        cacheCfg.setBackups(1);
        cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);

cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DeviceStatusCacheStore.class));
        cacheCfg.setWriteThrough(true);
        cacheCfg.setWriteBehindEnabled(true);
        cacheCfg.setReadThrough(true);
        cacheCfg.setWriteBehindFlushThreadCount(4);
        cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
        cacheCfg.setWriteBehindFlushSize(409600);
        cacheCfg.setWriteBehindBatchSize(1024);
        cacheCfg.setStoreKeepBinary(true);
        cfg.setCacheConfiguration(cacheCfg);

So I want to know is there a way that can speed up the execution?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Justin Ji Justin Ji
Reply | Threaded
Open this post in threaded view
|

Re: Update all data in cache spend a long time

Ilya -

Thank for your reply and point out the problem may exist in my code, I will
try to split the operation into smaller ones.

And I still have two problems:

1. Can I split the operation with LIMIT?
2. Where is time spent in the operation?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Update all data in cache spend a long time

Hello!

1. I don't think you should use LIMIT because it is very hard to make sure that all entries are updated exactly once. It is better to partition data by values of some field.
2. It is hard to say outright, did you try queryParallelism?

Regards,
--
Ilya Kasnacheev


вт, 9 окт. 2018 г. в 16:27, Justin Ji <[hidden email]>:
Ilya -

Thank for your reply and point out the problem may exist in my code, I will
try to split the operation into smaller ones.

And I still have two problems:

1. Can I split the operation with LIMIT?
2. Where is time spent in the operation?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Justin Ji Justin Ji
Reply | Threaded
Open this post in threaded view
|

Re: Update all data in cache spend a long time

Thank you, I will try queryParallelism.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/