Ignite Transaction performance

classic Classic list List threaded Threaded
20 messages Options
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Ignite Transaction performance

I have four caches which I write in a transaction.
I have two nodes running.
The transaction configuration is OPTIMISTIC and READ_COMMITED isolation.

I am observing that sometimes ignite is taking 16milliseconds and sometimes 20-30 milliseconds for commiting a transaction.

This is actually too slow for us, how can I improve the performance of the transactions?
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

Hi,

Can you provide your benchmark so that someone can take a look and investigate?

-Val
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

This post was updated on .
I will try to. But the code is difficult to replicate for obvious reasons. The cache keys are Long and the values are Java POJO's. There are two nodes running on different systems. There is however a massive amount of concurrent transactions. I saw there is an optimization in Ignite 1.6 RC1 for this

One last detail we use Spring Transaction manager for declarative transaction management. I have checked profiled data of the Ignite implementation only to negate the overhead of Spring proxies et all.

Will that help?

I will also try to replicate the situation.
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

There are too many variables to say if this performance is OK or not without having the code. So please provide the test if you want someone in the community to help you.

BTW, 1.6 is released, you can try it: ignite.apache.org/download.cgi#binaries

-Val
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

This post was updated on .
Val,

Understand that. I will try to come up with some code. I think by Friday I will have something that can be replicated...
Lastly one thing what can be the some of the known causes of something like this ?

Regards
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

This post was updated on .

I start one node with the same cache configs as above. I do not execute it . And in another cache I start a ignite node and us ethe code below

 IgniteCache<Integer, String> cache = ignite.getOrCreateCache(cfg1);
    IgniteCache<Integer, String> cache2 = ignite.getOrCreateCache(cfg1);
    IgniteCache<Integer, String> cache3 = ignite.getOrCreateCache(cfg1);
    IgniteTransactions igniteTransactions = ignite.transactions();
    Thread.sleep(2000);
    final long time = System.currentTimeMillis();
    for(int i =0; i< 100000; i ++) {
      try (Transaction tx = igniteTransactions.txStart()) {
          long t = System.nanoTime();
          cache.put(i, String.valueOf(i));
          cache2.put(i, String.valueOf(i));
          cache3.put(i, String.valueOf(i));
          tx.commit();
          System.out.println(System.nanoTime() - t);
          String s = cache.get(i);
         
      }
    }

Both the nodes are in the same machine and still it takes over 22 seconds to commit this work. With one node it takes 2 seconds .It is slower if I increase the put size. This is probably due to 1 pC vs 2pC in Ignite. How should I imporve the performance in this case? I need to have two nodes running.

I am using Ignite 1.6
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

Do you mean that it takes 22 seconds to execute 100000 transactions? This means around 0.2ms per transaction, which seems to be good latency. Note that you're executing them synchronously in a single thread, waiting for the network each time. If you need to increase throughput, you should run more threads (and probably more clients) and execute transactions in parallel.

-Val
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

Val,

Its correct that its so. Hwoever If I add one more node, this adds serious overhead. if I add more threads and start giving more concurrent inputs some transactions start slowing down with some going more than 3 seconds !!!.

However I read about affinity key, does it help while inserting data in Ignite...means will it help if all relevant data is affinity key mapped to a single node even while inserting ?

May be 2 PC will be discarded and 1 PC will only be used, increasing round trips between nodes and increasing performance?
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

Are you executing transactions from a client node or from a server node? Do you have backups?

-Val
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

This post was updated on .
Server Node.

Disabled Backups. But all changes are SYNCED synchronously.
Caches are all partitioned and there are 6 caches in the actual code.

There are some read only caches though, but as they are read in the Transaction I think they might be particpating.

ISOLATION Is READCOMMITED with Optimistic.

There are two nodes.

Will Affinity Key mapping help in transaction commit time by locating all the transaction changes to a single instance.
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

Lastly is there any thread count number to be increased to help in high throughput commit messages...liek does increasing the system threads increase performance?
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

Hi,

With a single node without a client, there is no network, so latency is of course lower than when you add network.

According to you description, your test is more complicated than code snippets your provided before. So please provide a project that I will be able to run and reproduce the performance drop. In this case I will be able to tell something regarding your particular benchmark.

-Val
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

Val,

Will do that, but meanwhile can I increase performance incresing System Thread pool etc?

My machine has CPU and bandwidth limit is still not reached
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

I am creating the test cases. I saw that doing inserts increase the latency severely. I have a question does Ignite treat inserts the same as updates ?

My isolation level is OPTIMISTIC with serializable
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

Hedisruptor.zip

The disruptorserver is to be run first.

The app is the client , sends in 100000, requests at a time.

I get a TPS of 10000 when I run the disruptor server only.

However when I run the IgniteMain to add a secondary server it goes down to 1000.
I understand it will be slower but I expect it tobe half, or even 3000.

Can anyone give me tips to optimize it?

vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

Hi,

Some of your caches are both REPLICATED and FULL_SYNC. This means that each time you update one of these caches, you will send synchronous request to all the nodes, so the latency will almost linearly grow.

You can try switching synchronization mode to PRIMARY_SYNC. In this case you will not synchronously wait for backups, but all the nodes will still be updated which will still slow you down, especially under high load when there is a lot of traffic.

To achieve the best scalability you should use PARTITIONED caches instead. In this mode you will always update the constant number of nodes (primary + zero or more backups depending on the configuration), regardless of the whole number of nodes in the cluster.

Can you try this and let us know if it helps?

-Val
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

Val,

Some of our caches have to REPLICATED , as they are mostly read only caches.
I will try Primary_SYNC and try.

Regards
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

In reply to this post by vkulichenko
Val,

Thanks for the tips.But I saw very little gains 4-10%.
For a large set of nodes this does provid ebenefit. However we plan to deploy two nodes and then take it from there as demand grows.
is there any other way I can imporve what is known in RDBMS terms insert throughput?
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

I have made all caches Atomic which need to be in transaction. Still the performance is no where near a singke node. I get 20000 inserts / sec on a single node now, and around 2000-2500 on a 2 node setup. Is there any thing I can do to increase performance ?

I am open to adding to one more node or other thigs to increase throughput
amitpa amitpa
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Transaction performance

This post was updated on .
Hello, can some one please submit an alternative? I from my side have something interesting to share.

In my example the Disruptor server takes in all UDP request. It also starts a Ignite embedded node.
The IgniteMain code starts another Ignite node.
The disruptor server gets an HTTP request and then it calls does a transaction on the Ignite Grid.


Now if I start a the DisruptorServer first and then the Ignitemain I get a performance increase of 90% almost for nothing ...This is quite crazy I know, but I tested it repeatedly and it happens. If i start the IgniteMain first and then the disruptor server I get a performance hit of almost 90%.


Can anyone please answer what is happening here?