cache update slow

classic Classic list List threaded Threaded
10 messages Options
Coleman, JohnSteven (Agoda) Coleman, JohnSteven (Agoda)
Reply | Threaded
Open this post in threaded view
|

cache update slow

Hi,

 

I’m inserting and then updating 250 cache entries using a couple of threads. So a total of 500 cache puts. The initial 250 writes take a little under 500ms. So each initial put is taking approx. 2ms. While that’s a little faster than writing to database, I would expect RAM write to be much faster, i.e. sub millisecond.

Any ideas why cache write is not so fast and how to improve it?

 

John



This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Maxim.Pudov Maxim.Pudov
Reply | Threaded
Open this post in threaded view
|

Re: cache update slow

Hi, the execution time depends on the configuration of your cache and your
cluster. How many node do you have? What is your cache configuration? Have
you tried Ignite data streamer [1] instead of cache.put(K,V)?

[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Coleman, JohnSteven (Agoda) Coleman, JohnSteven (Agoda)
Reply | Threaded
Open this post in threaded view
|

RE: cache update slow

Hi,

I'm just using the cache created by running a local process, see below. This is just for a POC and works, but I do need to have high performance.

John

          using (var ignite = Ignition.Start())
            {
                QueryEntity[] entities =
                {
                    new QueryEntity(typeof(int), typeof(SimpleTransaction))
                };
                var cfg = new CacheConfiguration
                {
                    OnheapCacheEnabled = true,
                    ReadThrough = false,
                    WriteThrough = false,
                    KeepBinaryInStore = false,
                    QueryEntities = entities,
                    Name = "SimpleTransactions"
                };
                ignite.AddCacheConfiguration(cfg);
                ICache<int, SimpleTransaction> cache = ignite.GetOrCreateCache<int, SimpleTransaction>(cfg);


-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Friday, April 19, 2019 3:37 PM
To: [hidden email]
Subject: Re: cache update slow

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________

Hi, the execution time depends on the configuration of your cache and your cluster. How many node do you have? What is your cache configuration? Have you tried Ignite data streamer [1] instead of cache.put(K,V)?

[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Maxim.Pudov Maxim.Pudov
Reply | Threaded
Open this post in threaded view
|

RE: cache update slow

Thanks for sharing your code. I didn't realise you use .NET. Check out how
you can benefit from data streamer in .NET [1]. It was designed to populate
your cache faster, so it could help you to improve performance.

[1] https://apacheignite-net.readme.io/docs/data-streamers



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Coleman, JohnSteven (Agoda) Coleman, JohnSteven (Agoda)
Reply | Threaded
Open this post in threaded view
|

RE: cache update slow

Hi,

Thanks for the tip. I implemented with data streamer and observe a significant improvement. However it still takes >1ms per cache entry addition which is fast enough for my requirements, but still >500 times slower than DMA. Is this largely a factor of network overhead (even though I use localhost cache), or the underlying caching mechanics?

Regards,
John

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Tuesday, April 23, 2019 8:12 PM
To: [hidden email]
Subject: RE: cache update slow

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________

Thanks for sharing your code. I didn't realise you use .NET. Check out how you can benefit from data streamer in .NET [1]. It was designed to populate your cache faster, so it could help you to improve performance.

[1] https://apacheignite-net.readme.io/docs/data-streamers



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Maxim.Pudov Maxim.Pudov
Reply | Threaded
Open this post in threaded view
|

RE: cache update slow

Glad you met your requirements. I think it is not fair to compare Ignite with
direct memory access, so I can't really say whether this is a good result or
not. In your case .net process starts a java process and communicates with
it via JNI [1]. Also Ignite stores cache data off-heap, which requires
serialisation [2].

[1] https://apacheignite-net.readme.io/docs#section-ignite-and-ignitenet
[2] https://apacheignite.readme.io/docs/durable-memory



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Coleman, JohnSteven (Agoda) Coleman, JohnSteven (Agoda)
Reply | Threaded
Open this post in threaded view
|

RE: cache update slow

Hi,

Yes, comparing to DMA is apples and oranges comparison, but gives an idea of the relative gap in performance.

A better comparison would be to an alike product such as NCache. They claims 20K wps*, thus 20 times faster than my ignite results, but obvs I'd have to compare using my scenario for a valid comparison. But this is more like the kind of gap in performance I'd expect vs DMA. But then NCache product is also quite different from ignite, so what to say?

regards,
John

http://www.alachisoft.com/ncache/ncache-performance-benchmarks.html

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Friday, April 26, 2019 3:17 PM
To: [hidden email]
Subject: RE: cache update slow

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________

Glad you met your requirements. I think it is not fair to compare Ignite with direct memory access, so I can't really say whether this is a good result or not. In your case .net process starts a java process and communicates with it via JNI [1]. Also Ignite stores cache data off-heap, which requires serialisation [2].

[1] https://apacheignite-net.readme.io/docs#section-ignite-and-ignitenet
[2] https://apacheignite.readme.io/docs/durable-memory



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: cache update slow

Hello!

I think that comparing steady state benchmarks of multi-million operations versus 500 operations is misleading.

500 operations is probably not enough to gain full benefits from e.g. JIT.

Regards,
--
Ilya Kasnacheev


пт, 26 апр. 2019 г. в 12:20, Coleman, JohnSteven (Agoda) <[hidden email]>:
Hi,

Yes, comparing to DMA is apples and oranges comparison, but gives an idea of the relative gap in performance.

A better comparison would be to an alike product such as NCache. They claims 20K wps*, thus 20 times faster than my ignite results, but obvs I'd have to compare using my scenario for a valid comparison. But this is more like the kind of gap in performance I'd expect vs DMA. But then NCache product is also quite different from ignite, so what to say?

regards,
John

http://www.alachisoft.com/ncache/ncache-performance-benchmarks.html

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Friday, April 26, 2019 3:17 PM
To: [hidden email]
Subject: RE: cache update slow

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________

Glad you met your requirements. I think it is not fair to compare Ignite with direct memory access, so I can't really say whether this is a good result or not. In your case .net process starts a java process and communicates with it via JNI [1]. Also Ignite stores cache data off-heap, which requires serialisation [2].

[1] https://apacheignite-net.readme.io/docs#section-ignite-and-ignitenet
[2] https://apacheignite.readme.io/docs/durable-memory



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Coleman, JohnSteven (Agoda) Coleman, JohnSteven (Agoda)
Reply | Threaded
Open this post in threaded view
|

RE: cache update slow

Hi,

 

Thanks for that observation. I increased cache test to 100,000 entries and the average write time is far better at around 23K wps. It seems like a lot of latency on the first few hundred writes.

Do you have any benchmarks published?

John

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Friday, April 26, 2019 7:29 PM
To: [hidden email]
Subject: Re: cache update slow

 

Email received from outside the company. If in doubt don't click links nor open attachments!


Hello!

 

I think that comparing steady state benchmarks of multi-million operations versus 500 operations is misleading.

 

500 operations is probably not enough to gain full benefits from e.g. JIT.

 

Regards,

--

Ilya Kasnacheev

 

 

пт, 26 апр. 2019 г. в 12:20, Coleman, JohnSteven (Agoda) <[hidden email]>:

Hi,

Yes, comparing to DMA is apples and oranges comparison, but gives an idea of the relative gap in performance.

A better comparison would be to an alike product such as NCache. They claims 20K wps*, thus 20 times faster than my ignite results, but obvs I'd have to compare using my scenario for a valid comparison. But this is more like the kind of gap in performance I'd expect vs DMA. But then NCache product is also quite different from ignite, so what to say?

regards,
John

http://www.alachisoft.com/ncache/ncache-performance-benchmarks.html

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Friday, April 26, 2019 3:17 PM
To: [hidden email]
Subject: RE: cache update slow

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________

Glad you met your requirements. I think it is not fair to compare Ignite with direct memory access, so I can't really say whether this is a good result or not. In your case .net process starts a java process and communicates with it via JNI [1]. Also Ignite stores cache data off-heap, which requires serialisation [2].

[1] https://apacheignite-net.readme.io/docs#section-ignite-and-ignitenet
[2] https://apacheignite.readme.io/docs/durable-memory



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: cache update slow

Hello!

I don't think that we publish results.

Please take look at our benchmarking approach at

Regards,
--
Ilya Kasnacheev


пн, 29 апр. 2019 г. в 05:57, Coleman, JohnSteven (Agoda) <[hidden email]>:

Hi,

 

Thanks for that observation. I increased cache test to 100,000 entries and the average write time is far better at around 23K wps. It seems like a lot of latency on the first few hundred writes.

Do you have any benchmarks published?

John

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Friday, April 26, 2019 7:29 PM
To: [hidden email]
Subject: Re: cache update slow

 

Email received from outside the company. If in doubt don't click links nor open attachments!


Hello!

 

I think that comparing steady state benchmarks of multi-million operations versus 500 operations is misleading.

 

500 operations is probably not enough to gain full benefits from e.g. JIT.

 

Regards,

--

Ilya Kasnacheev

 

 

пт, 26 апр. 2019 г. в 12:20, Coleman, JohnSteven (Agoda) <[hidden email]>:

Hi,

Yes, comparing to DMA is apples and oranges comparison, but gives an idea of the relative gap in performance.

A better comparison would be to an alike product such as NCache. They claims 20K wps*, thus 20 times faster than my ignite results, but obvs I'd have to compare using my scenario for a valid comparison. But this is more like the kind of gap in performance I'd expect vs DMA. But then NCache product is also quite different from ignite, so what to say?

regards,
John

http://www.alachisoft.com/ncache/ncache-performance-benchmarks.html

-----Original Message-----
From: Maxim.Pudov <[hidden email]>
Sent: Friday, April 26, 2019 3:17 PM
To: [hidden email]
Subject: RE: cache update slow

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________

Glad you met your requirements. I think it is not fair to compare Ignite with direct memory access, so I can't really say whether this is a good result or not. In your case .net process starts a java process and communicates with it via JNI [1]. Also Ignite stores cache data off-heap, which requires serialisation [2].

[1] https://apacheignite-net.readme.io/docs#section-ignite-and-ignitenet
[2] https://apacheignite.readme.io/docs/durable-memory



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.