Data Loading Performance Issue

classic Classic list List threaded Threaded
10 messages Options
diopek diopek
Reply | Threaded
Open this post in threaded view
|

Data Loading Performance Issue

I am having performance issue for loading data into Ignite cache. I tried several ways,
from individual put to putAll to IgniteDataStreamer.addData. As one unit test case example;
- time to retrieve data from data base is 26 secs by populating HashMap.
- if I try doing cache.putAll or dataStreamer.addData to bulk load data into Ignite cache takes another 240 secs. And one large scale batch run cache population took 5 hrs which is quite extreme as actual batch took less than 2hrs after that. Wondering if I am missing some fundementals here. Please advise.
Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Performance Issue

Hi,

Do you use the DB mentioned below as a persistent storage for the Ignite Cache being populated?
If yes, then you can leverage CacheStore interface [1].

If otherwise you migrate your data from the DB then IgniteDataStreamer is the best candidate for such kind of tasks.
It’s not clear from your description below how you actually use the APIs, what the configuration is, how big your data set.
Please provide this info.

[1] https://apacheignite.readme.io/docs/persistent-store#cachestore

Regards,
Denis

> On 14 нояб. 2015 г., at 3:35, diopek <[hidden email]> wrote:
>
> I am having performance issue for loading data into Ignite cache. I tried
> several ways,
> from individual put to putAll to IgniteDataStreamer.addData. As one unit
> test case example;
> - time to retrieve data from data base is 26 secs by populating HashMap.
> - if I try doing cache.putAll or dataStreamer.addData to bulk load data into
> Ignite cache takes another 240 secs. And one large scale batch run cache
> population took 5 hrs which is quite extreme as actual batch took less than
> 2hrs after that. Wondering if I am missing some fundementals here. Please
> advise.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-Loading-Performance-Issue-tp1958.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

diopek diopek
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Performance Issue

Hi Denis, I changed the cache initialization code code, best performance I had by using CacheStore option. Now my caching time reduced to 31 mins in my local development Windows 7 PC (8CPU/32GB RAM) which was taking 5:30 hrs in HP ProLinea DL580 64 CPU/1TB Linux box. Then I moved to jar to more powerful UNIX server which has 64 CPU box. Cache initialization there still taking longer then my local version, almost 2hrs 41mins. Also this box is only dedicated to this module, no other application is running other than my module. Can you please advise how can I diagnose this issue? Thank you
Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Performance Issue

Hi,

Unfortunately I can't give you concrete recommendations because there are tons of the reasons why your loading takes so long time.

Please clarify the following:

1) How big is the data set you're preloading?

2) What is your grid configuration? How many nodes are there? Attache Ignite configuration you use to start the grid nodes.

3) You mentioned that you're getting the data from some DB. What kind of DB are you working with? How do you connect to it from your local machine and from remote machines?

4) Preloading on the servers might work slower because the data is transferred across network while in case of your local laptop the localhost may be leveraged.
 
5) Measure the time that takes to load the data from DB. Probably this is the slowest point.

6) Attache a snippet of your source code you use for data loading.

7) What is CPU load during the cache preloading on all the machines? What kind of VM options do you use (heap size, kind of GC, etc.)

Regards,
Denis
diopek diopek
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Performance Issue

Hi Denis, please see my answers below;

>>1) How big is the data set you're preloading?
~10 GB
>>2) What is your grid configuration? How many nodes are there? Attache Ignite configuration you use to start the grid nodes.
1 node/JVM, My ignite configuration is as the following;
ignite-cache.xml
I also have this code to be used in Cache initialization using as template for my cache config.
<bean id="rwaCacheTemplate" class="org.apache.ignite.configuration.CacheConfiguration">
               
                <property name="atomicityMode" value="ATOMIC" />
                <property name="cacheMode" value="LOCAL" />
                <property name="backups" value="0" />
                <property name="swapEnabled" value="false" />
                <property name="memoryMode" value="ONHEAP_TIERED" />
                <property name="offHeapMaxMemory" value="0" />
        </bean>

>>3) You mentioned that you're getting the data from some DB. What kind of DB are you working with? How do you connect to it from your local machine and from remote machines?
Our DB is Oracle 11g, I am using Spring Data source as you can see in above config file. In my local machine when I ping DB server it is 19 ms on, in other Linux server 0.32 ms

>>4) Preloading on the servers might work slower because the data is transferred across network while in case of your local laptop the localhost may be leveraged.
As I mentioned above item, ironically my PC to DB connection is slower than server connection as app server and DB servers are in same data centers.
 
>>5) Measure the time that takes to load the data from DB. Probably this is the slowest point.
I measured that if I just populate regular Java HashMap with the same database connection. It is 6 times faster that initializing the Ignite cache.

>>6) Attache a snippet of your source code you use for data loading.
// this snippert is from loadCache invoking function client file
public IgniteCache<Integer, ArrayList<RwaRsExistingDTO>> loadExistingCache(Date asOf, Integer scenId, String existingSql) {
               
                StringBuffer rep = new StringBuffer("'");
                rep.append(fastDateFormatddMMMyyyy.format(asOf)).append("'");
                existingSql = existingSql.replace("?", rep);
                String extSQL = existingSql.substring(0, existingSql.indexOf("ORDER"));
                StringBuffer cntExtSQL = new StringBuffer(
                                "SELECT COUNT(*) FROM ( SELECT /*+ parallel (16) */ DISTINCT GOC, ACCT, SUM_AFFIL_CODE, CURRENCY, FRCST_PROD_ID, COUNT(*) TOT FROM (");
                cntExtSQL.append(extSQL).append(" ) GROUP BY GOC,ACCT, SUM_AFFIL_CODE, CURRENCY, FRCST_PROD_ID ");
                cntExtSQL.append(" ORDER BY GOC,ACCT, SUM_AFFIL_CODE, CURRENCY, FRCST_PROD_ID)");
                logger.debug("cntExtSQL::{}", cntExtSQL);

                final int extStartSize = jdbcTemplate.queryForObject(cntExtSQL.toString(), null, Integer.class);
                logger.info("--->>>extHashBucket::" + extStartSize);
                StopWatch sw1 = new StopWatch();
                stopWatchStart(sw1, "loadExistingCache::createCache");
                CacheConfiguration<Integer, ArrayList<RwaRsExistingDTO>> existingCacheCfg = new CacheConfiguration<>("MY_CACHE");
                existingCacheCfg.setStartSize(extStartSize);
                existingCacheCfg.setCacheMode(CacheMode.LOCAL);
                existingCacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(ExistingCacheStore.class));
                existingCacheCfg.setReadThrough(false);
                existingCacheCfg.setWriteThrough(false);
               
                Ignite ignite = Ignition.ignite();
                IgniteCache<Integer, ArrayList<RwaRsExistingDTO>> cache = ignite.createCache(existingCacheCfg);
                stopWatchEnd(sw1);
                stopWatchStart(sw1, "loadExistingCache::loadCache");
                cache.loadCache(null, asOf,scenId,existingSql);
                stopWatchEnd(sw1);
                return cache;
        }
//This snipppet s from another MyCacheStore.java file
@Override
        public void loadCache(final IgniteBiInClosure<Integer, ArrayList<RwaRsExistingDTO>> clo, Object... args) {

                if (args == null || args.length < 2)
                        throw new CacheLoaderException("Expected asOf and scenId parameter is not provided.");

               
                final Date asOf = (Date) args[0];
                final Integer scenId = (Integer) args[1];
                final String existingSql = (String) args[2];
                StringBuffer rep = new StringBuffer("'");

                logger.debug("loadExistingCache::SQL::{}", existingSql);

                ResultSetExtractor<Void> extMapResultSetExtractor = new ResultSetExtractor<Void>() {
                        @Override
                        public Void extractData(ResultSet rs) throws SQLException, DataAccessException {
                                String prevGoc = null, prevAcct = null, prevSac = null, prevCcy = null;
                                Integer prevFpId = null;
                                ArrayList<RwaRsExistingDTO> currDTOList = null, prevDTOList = null;
                                RwaRsExistingDTO dto = null, prevDto = null;
                                final AtomicInteger entryCnt = new AtomicInteger(1);
                                while (rs.next()) {
                                        int i = 1;
                                        dto = new RwaRsExistingDTO();
                                        dto.setAsOf(asOf);
                                        dto.setScnId(scenId);

                                        dto.setGoc(rs.getString(i++));
                                        dto.setAcct(rs.getString(i++));
                                        dto.setSumAffilCode(rs.getString(i++));
                                        dto.setCcyCode(rs.getString(i++));
                                        dto.setFrcstProdId(rs.getInt(i++));

                                        dto.setFrsBu(rs.getString(i++));
                                        dto.setRwaExposureType(rs.getString(i++));
                                        dto.setRiskAssetClass(rs.getString(i++));
                                        dto.setRiskSubAssetClass(rs.getString(i++));
                                        dto.setTreasLiqClass(rs.getString(i++));
                                        dto.setCounterpartyRating(rs.getString(i++));
                                        dto.setClearedStatus(rs.getString(i++));
                                        dto.setMaturityBand(rs.getString(i++));
                                        dto.setDerivativeType(rs.getString(i++));
                                        dto.setStartDate(rs.getDate(i++));
                                        dto.setMaturityDate(rs.getDate(i++));
                                        dto.setAmount(rs.getDouble(i++));
                                        dto.setReplenishFlag("N");


                                        if (dto.getGoc().equals(prevGoc) && dto.getAcct().equals(prevAcct)
                                                        && dto.getSumAffilCode().equals(prevSac) && dto.getCcyCode().equals(prevCcy)
                                                        && dto.getFrcstProdId().equals(prevFpId)) {
                                                prevDTOList.add(prevDto);
                                        } else {
                                                if (prevDto != null) {
                                                        prevDTOList.add(prevDto);
                                                        clo.apply(entryCnt.incrementAndGet(), prevDTOList);
                                                }
                                                currDTOList = new ArrayList<RwaRsExistingDTO>();
                                        }
                                        prevDto = dto;
                                        prevDTOList = currDTOList;
                                        prevGoc = dto.getGoc();
                                        prevAcct = dto.getAcct();
                                        prevSac = dto.getSumAffilCode();
                                        prevCcy = dto.getCcyCode();
                                        prevFpId = dto.getFrcstProdId();
                                }
                                if (prevDto != null) {
                                        prevDTOList.add(prevDto);
                                        clo.apply(entryCnt.incrementAndGet(), prevDTOList);
                                }
                                return null;
                        }

                };

                jdbcTemplate.setFetchSize(SQL_FETCH_SIZE);
                jdbcTemplate.query(existingSql, extMapResultSetExtractor);
        }

>>7) What is CPU load during the cache preloading on all the machines? What kind of VM options do you use (heap size, kind of GC, etc.)
As I mentioned, my local machine is less powerful than UNIX server (8CPU vs. 64CPU and 32GB RAM vs. 1TB RAM and DB Network access is slower than UNIX server (19ms vs. 0.32ms), both JVM is JDK 1.8.0_65 64 bit version and heap configured as -Xms2048m -Xmx20480m

I can summarize my issues as below;

1)Highest priority one, ironically, my local machine batch run time is much lower 42 mins (local) vs. 2hr 56 mins (linux) both running on single node eve though it is less powerful and I have other other apps running at the same time such as Eclipse IDE, SQLdeveloper, Outlook, Word, UNIX box is dedicated app server no other applications running on that server.

2) Second one is in general loading data from DB to Java Collection is much faster than loading into Ignite cache by factor of ~6. Which is not a show stopper but if we need to resolve the first problem as highest priority. I appreciate your help on this. Thanks

Deha
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Performance Issue

Hi Deha,
diopek wrote
1)Highest priority one, ironically, my local machine batch run time is much lower 42 mins (local) vs. 2hr 56 mins (linux) both running on single node eve though it is less powerful and I have other other apps running at the same time such as Eclipse IDE, SQLdeveloper, Outlook, Word, UNIX box is dedicated app server no other applications running on that server.
Can you please clarify what you mean by "batch run time"? Is it somehow connected to data loading via the store or it's a different issue?
diopek wrote
2) Second one is in general loading data from DB to Java Collection is much faster than loading into Ignite cache by factor of ~6. Which is not a show stopper but if we need to resolve the first problem as highest priority. I appreciate your help on this. Thanks
I noticed that you put some lists instead of individual entries into the cache. What is the size of these lists? My suspicion is that the most time is spent for the serialization of the values (JCache spec has pass-by-value semantics, so we have to do this even in LOCAL cache).

I would suggest to store a value per DB row to avoid duplicate data and therefore duplicate serializations. It also looks like loading the data in multithreaded fashion may be helpful - execute the query first and then do DB row parsing and saving into cache in several parallel threads. You can utilize CacheLoadOnlyStoreAdapter for this.

-Val
diopek diopek
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Performance Issue

Hi Val,
Please see my comments below;

Val:Can you please clarify what you mean by "batch run time"? Is it somehow connected to data loading via the store or it's a different issue?
diopek:
Batch run time : first step is cache initialization (most times taking step) and second step doing some computations and generating outputs using this cache populated in first step. Cache loading via store. Currently most important issue is, overall batch is running faster in my local PC (Windows 7, 8CPU/32GB RAM machine) than more powerful Linux server (64 CPU and 1TB RAM, which also has a faster network to Oracle DB). As a side note  deployment package for all servers was built on my Windows server with 64 Windows version of JDK 1.8.0_65, Linux boxes Linux version of that JDK 1.8.0_65 ). I am literally puzzled here, what is causing this delay..
Val:
I noticed that you put some lists instead of individual entries into the cache. What is the size of these lists? My suspicion is that the most time is spent for the serialization of the values (JCache spec has pass-by-value semantics, so we have to do this even in LOCAL cache).

diopek:
Yes my cache structure IgniteCache<Integer, ArrayList<MyBusinessObj>>, as I needed to process records that has certain common attributes together, these are like trade positions that has certain common attributes like date, acct, currency etc. After reading into a cache during the proecessing stage each group split into more granular records like 10 records become 1000 records and then I aggregate (group by) them so number of records shrinks to 500. then I directly write these records into some feed file. During the interim processing/computation the don't get stored back to Ignite cache just stays in regular Java memory till got flush into file.
Also all the grouped records in the cache are being read by multiple partition threads, lets say if cache has 5,000,000 records and there are 5 partition threads each thread reads only its partition of records.
if I read all the records into cache row by row, how can I partition and process records as groups. Of course I can partition rows but I loose the grouping. This is also the reason why I  grouped and inserted into cache at the first place (as DB doesn't have natural partition key). In my uses cases I store sometimes customized objects as well as ArrayList, LindkedHashMap as values. I am aware of that serialization costs but was not sure about its extent.
Currently my first priority just to resolve this Linux deployment slowness first, though this serialization has some cost but I am able to group and partition the records (I am open to suggestions as how can group/partition the records.
Val:
I would suggest to store a value per DB row to avoid duplicate data and therefore duplicate serializations. It also looks like loading the data in multithreaded fashion may be helpful - execute the query first and then do DB row parsing and saving into cache in several parallel threads. You can utilize CacheLoadOnlyStoreAdapter for this.
diopek:
Is there any working example you can point me to ?
Thanks,
Deha
Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Performance Issue

Hi Deha,

Thanks for the additional info. However I didn't manage to open your config file.

In any case I have the following thoughts basing on the current info.

Currently most important issue is, overall batch is running faster in my local PC (Windows 7, 8CPU/32GB RAM machine) than more powerful Linux server (64 CPU and 1TB RAM, which also has a faster network to Oracle DB)


Since you're populating a LOCAL cache and if to take into account that your local machine and the servers use the same configuration and virtual machine options the only reason that should lead to such big performance differences is the speed with which the data retrieved from the DB.

Yeah, you measured connection speed using 'ping' utility but it has nothing to do with actual data retrieving from the DB.

I suggest you doing the following steps:
- measure the time that it takes to get the data from DB. It should be different on your local and server machines;
- check settings of your Oracle/JDBC connector on the server side.


Yes my cache structure IgniteCache<Integer, ArrayList<MyBusinessObj>>, as I needed to process records that has certain common attributes together, these are like trade positions that has certain common attributes like date, acct, currency etc. After reading into a cache during the proecessing stage each group split into more granular records like 10 records become 1000 records and then I aggregate (group by) them so number of records shrinks to 500. then I directly write these records into some feed file. During the interim processing/computation the don't get stored back to Ignite cache just stays in regular Java memory till got flush into file.

Sounds complex for me. Fully share Val's concerns that in the future you will face with the performance issues when these data needs to be processed, requested, updated.

My suggestion is to store different business objects in different caches. After that you can easily use SQL queries with joins to get the data you need from the cache.
https://apacheignite.readme.io/docs/sql-queries

You can utilize CacheLoadOnlyStoreAdapter for this.
diopek:
Is there any working example you can point me to ?


Yes, this is a great suggestion from Val.

We don't have a dedicated example for this kind of store. However, you may refer to some tests that shows how to use it - GridCacheLoadOnlyStoreAdapterSelfTest.

Start using this store with default settings and adjust them depending on your the characteristics of a machine.

Regards,
Denis
diopek diopek
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Performance Issue

Denis
I will look into that CacheLoadOnlyStoreAdapter in Ignite source code.
In the mean time, below I uploaded ignite config file that I use for your review.
Thanks much for your help.
Deha

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd">

        <bean id="placeholderProperties" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
                <property name="locations">
                        <list>
                                <value>classpath:/rwa-ignite.properties</value>
                        </list>
                </property>
                <property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
                <property name="ignoreUnresolvablePlaceholders" value="true" />
                <property name="order" value="1" />
        </bean>
        <bean id="igniteDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
                <property name="driverClassName" value="${ignite.jdbc.driver}" />
                <property name="url" value="${ignite.jdbc.url}" />
                <property name="username" value="${ignite.jdbc.user}" />
                <property name="password" value="${ignite.jdbc.password}" />
                <property name="maxIdle" value="${ignite.jdbc.minPoolSize}" />
                <property name="maxActive" value="${ignite.jdbc.maxPoolSize}" />
                <property name="accessToUnderlyingConnectionAllowed" value="true" />
        </bean>
        <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
                <property name="peerClassLoadingEnabled" value="false" />
                <property name="publicThreadPoolSize" value="1"/>
                <property name="systemThreadPoolSize" value="1"/>
               
                <property name="marshaller">
                        <bean class="org.apache.ignite.marshaller.optimized.OptimizedMarshaller">
                                <property name="requireSerializable" value="false" />
                        </bean>
                </property>

                <property name="localHost" value="127.0.0.1" /
                <property name="discoverySpi">
                        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                                <property name="ipFinder">
                                        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
                                                <property name="addresses">
                                                        <list>
                                                                <value>127.0.0.1:47500..47509</value>
                                                        </list>
                                                </property>
                                        </bean>
                                </property>
                        </bean>
                </property>
        </bean>
</beans>

Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Data Loading Performance Issue

Hi Deha,

These two pools related settings [1] from your configuration file
explicitly reduce the overall performance of your system.
By setting the values to '1' will lead to the situation when a bunch of
logic will start working a single-threaded mode regardless of the
numbers of CPUs available.

Please, just remove these two lines from your config. The defaults are
calculated basing on the number of CPUs.

[1]
<property name="publicThreadPoolSize" value="1"/>
<property name="systemThreadPoolSize" value="1"/>

On 11/19/2015 7:27 AM, diopek wrote:

> Denis
> I will look into that CacheLoadOnlyStoreAdapter in Ignite source code.
> In the mean time, below I uploaded ignite config file that I use for your
> review.
> Thanks much for your help.
> Deha
>
> /<?xml version="1.0" encoding="UTF-8"?>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="
>          http://www.springframework.org/schema/beans
>          http://www.springframework.org/schema/beans/spring-beans.xsd">
>
> <bean id="placeholderProperties"
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
> <list>
> <value>classpath:/rwa-ignite.properties</value>
> </list>
> </property>
> <property name="systemPropertiesModeName"
> value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
> <property name="ignoreUnresolvablePlaceholders" value="true" />
> <property name="order" value="1" />
> </bean>
> <bean id="igniteDataSource" class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
> <property name="driverClassName" value="${ignite.jdbc.driver}" />
> <property name="url" value="${ignite.jdbc.url}" />
> <property name="username" value="${ignite.jdbc.user}" />
> <property name="password" value="${ignite.jdbc.password}" />
> <property name="maxIdle" value="${ignite.jdbc.minPoolSize}" />
> <property name="maxActive" value="${ignite.jdbc.maxPoolSize}" />
> <property name="accessToUnderlyingConnectionAllowed" value="true" />
> </bean>
> <bean id="ignite.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
> <property name="peerClassLoadingEnabled" value="false" />
> <property name="publicThreadPoolSize" value="1"/>
> <property name="systemThreadPoolSize" value="1"/>
>
> <property name="marshaller">
> <bean class="org.apache.ignite.marshaller.optimized.OptimizedMarshaller">
> <property name="requireSerializable" value="false" />
> </bean>
> </property>
>
> <property name=&quot;localHost&quot; value=&quot;127.0.0.1&quot; /
> &lt;property name=&quot;discoverySpi&quot;>
> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="ipFinder">
> <bean
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
> <property name="addresses">
> <list>
> <value>127.0.0.1:47500..47509</value>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
> </bean>
> </beans>/
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-Loading-Performance-Issue-tp1958p2012.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.