Performance Degredation when adding more nodes

classic Classic list List threaded Threaded
6 messages Options
thammoud thammoud
Reply | Threaded
Open this post in threaded view
|

Performance Degredation when adding more nodes

This post has NOT been accepted by the mailing list yet.
Hello,

We're trying to evaluate Ignite and have some concerns about the performance.

We create a cache using the following logic:

        CacheConfiguration cfg = new CacheConfiguration();
        cfg.setName(cacheName);
        cfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);        
        cfg.setCacheMode(CacheMode.PARTITIONED);
        cfg.setBackups(1);
        cfg.setIndexedTypes(PersonKey.class, Person.class);

In a transaction (Committed every 1000 puts), we had a Person with a Person key (Affinity on one field). When using only local 2 nodes, the time it takes to insert 100,000 records is roughly 56 seconds. Very slow. As we add more nodes (up to 7), the numbers get significantly worse (Minutes). We must be doing something very wrong. The machine has lots of memory and CPU.

The thought process is with one backup, a put for one key will talk to a max of two nodes per put. However, the degradation that result from adding more nodes suggest that a lot more chatter is taking place between the nodes. Any pointers will be greatly be appreciated.

Thank you.

thammoud thammoud
Reply | Threaded
Open this post in threaded view
|

Re: Performance Degredation when adding more nodes

This post has NOT been accepted by the mailing list yet.
Forgot to include that we use 1.6.
AndreyVel AndreyVel
Reply | Threaded
Open this post in threaded view
|

Re: Performance Degredation when adding more nodes

In reply to this post by thammoud
Hello thammoud,

Every operation cache.put() takes network roundtrip, by this reason performance of single operation depends on client network adapter. Try to use IgniteDataStreamer for bulk loading.

Example:

        try (IgniteDataStreamer<PersonKey, Person> stmr = Ignition.ignite().dataStreamer(cache.getName())) {
            stmr.perNodeBufferSize(1024); // default 1024

            for (int ind = 0; ind < CACHE_ITEMS; ind++) {
                PersonKey pk = new PersonKey(ind);
                Person person = new Person((long)ind, "First" + ind, "Last" + ind);
                stmr.addData(pk, person);
            }
        }


Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Performance Degredation when adding more nodes

In reply to this post by thammoud
Hi,

Please properly subscribe to the user list (this way we will not have to manually approve your emails). All you need to do is send an email to ì user-subscribe@ignite.apache.orgî and follow simple instructions in the reply.

Do I understand you properly that every transaction performs 1000 put operations? If so it means that the more nodes you have the more of them will participate in every transaction at prepare/commit phases that will lead to more network round-trips.

If my understanding is correct I would suggest the following:
- don't use such big transaction unless there is a reason for that. Split such big transaction on a smaller one that will correspond to your business use case. Describe your use case if you need more details.

- if the keys that are used in transactions, that are running in parallel, are the same then it means that one transaction will block the other. If this is not the case then use several Threads that will perform transactions in parallel to improve the overall throughput.

- use optimistic serializable transactions whenever it's possible [1] because in basic scenarios they should work faster then pessimistic ones.

If to follow this suggestions you shouldn't get any performance degradation when your cluster scales horizontally.


[1] https://apacheignite.readme.io/v1.6/docs/transactions#optimistic-transactions

--
Denis

---------------

Hello,

We're trying to evaluate Ignite and have some concerns about the performance.

We create a cache using the following logic:

        CacheConfiguration cfg = new CacheConfiguration();
        cfg.setName(cacheName);
        cfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);        
        cfg.setCacheMode(CacheMode.PARTITIONED);
        cfg.setBackups(1);
        cfg.setIndexedTypes(PersonKey.class, Person.class);

In a transaction (Committed every 1000 puts), we had a Person with a Person key (Affinity on one field). When using only local 2 nodes, the time it takes to insert 100,000 records is roughly 56 seconds. Very slow. As we add more nodes (up to 7), the numbers get significantly worse (Minutes). We must be doing something very wrong. The machine has lots of memory and CPU.

The thought process is with one backup, a put for one key will talk to a max of two nodes per put. However, the degradation that result from adding more nodes suggest that a lot more chatter is taking place between the nodes. Any pointers will be greatly be appreciated.

Thank you.
thammoud thammoud
Reply | Threaded
Open this post in threaded view
|

Re: Performance Degredation when adding more nodes

In reply to this post by AndreyVel
Thank you. A DS made a huge difference. Now I have 8 nodes with a cache configured for one backup and it takes abut 15 seconds to load the 100k records. Not perfect but much better than minutes. Regarding transactions, I presume that wrapping the whole put operation in a tx.begin() and tx.commit() will make those nodes visible only after txn commit.

IgniteTransactions transactions = ignite.transactions();
                       
try (Transaction tx = transactions.txStart()) {
    try (IgniteDataStreamer<PIDDetailsKey, PIDDetails> ds = ignite.dataStreamer(CACHE_NAME)) {
          for (PIDDetails details : details.values()) {                    
            ds.addData(details.getKey(), details);    
          }                
    }
    tx.commit();
}

and it seems to work. My use case is as follows:

1) For a given account, load transaction details and create the proper PIDDetails
2) Load all the PIDDetails into the cache
3) Commit the transaction to make visible to all

There can only be account loader for a given account Id which I am planning to use an account lock for enforcement.




AndreyVel AndreyVel
Reply | Threaded
Open this post in threaded view
|

Re: Performance Degredation when adding more nodes

Hello thammoud,

Try to use affinity feature for cache. For given accountId all PIDDetails better place in one node.
Example:

cacheCfg.setAffinityMapper(new AffinityKeyMapper() {
    @Override
    public Object affinityKey(Object key) {
        if (key instanceof PIDDetails){
            PIDDetails det = (PIDDetails)key;
            return det.getAccountId();
        }
        return key;
    }

    @Override
    public void reset() {
    }
});

13.06.2016 20:47, thammoud [via Apache Ignite Users] пишет:
Thank you. A DS made a huge difference. Now I have 8 nodes with a cache configured for one backup and it takes abut 15 seconds to load the 100k records. Not perfect but much better than minutes. Regarding transactions, I presume that wrapping the whole put operation in a tx.begin() and tx.commit() will make those nodes visible only after txn commit.

IgniteTransactions transactions = ignite.transactions();
                       
try (Transaction tx = transactions.txStart()) {
    try (IgniteDataStreamer<PIDDetailsKey, PIDDetails> ds = ignite.dataStreamer(CACHE_NAME)) {
          for (PIDDetails details : details.values()) {                    
            ds.addData(details.getKey(), details);    
          }                
    }
    tx.commit();
}

and it seems to work. My use case is as follows:

1) For a given account, load transaction details and create the proper PIDDetails
2) Load all the PIDDetails into the cache
3) Commit the transaction to make visible to all

There can only be account loader for a given account Id which I am planning to use an account lock for enforcement.







If you reply to this email, your message will be added to the discussion below:
http://apache-ignite-users.70518.x6.nabble.com/Performance-Degredation-when-adding-more-nodes-tp5575p5615.html
To unsubscribe from Performance Degredation when adding more nodes, click here.
NAML