Possible memory leak in Ignite 2.9

classic Classic list List threaded Threaded
4 messages Options
Kalin Katev Kalin Katev
Reply | Threaded
Open this post in threaded view
|

Possible memory leak in Ignite 2.9

Hi, 

not too long ago I tested apache ignite for my use case on Open JDK 11. The use case consists of writing cache entries with values going up to 800MB in size, the data itself being a simple string. After writing 5 caches entries, 800 MB each, I noticed my Heap space exploding up to 11GB, while the entries themselves were written off-heap. The histogram of the heap dump shows that there are 5 tuples of byte[] arrays with size 800MB and 1000MB that are left dangling on heap.  I am very curious if I did something wrong or if there indeed is an issue in Ignite. All details can be seen on https://stackoverflow.com/questions/64550479/possible-memory-leak-in-apache-ignite

Should I create a jira ticket for this issue? 

Best regards,
Kalin Katev

Resonance GmbH
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak in Ignite 2.9

Hello!

800MB entry is far above of the entry size that we ever expected to see. Even brief holding of these entries on heap will cause problems for you, as well as sending them over communication.

I recommend splitting entries into chunks, maybe. That's what IGFS did basically, we decided to ax it, but you can still use that approach.

Having said that, if you can check the heap dump to see where are these stuck byte arrays referenced from, I may check it.

Regards,
--
Ilya Kasnacheev


вт, 3 нояб. 2020 г. в 11:48, Kalin Katev <[hidden email]>:
Hi, 

not too long ago I tested apache ignite for my use case on Open JDK 11. The use case consists of writing cache entries with values going up to 800MB in size, the data itself being a simple string. After writing 5 caches entries, 800 MB each, I noticed my Heap space exploding up to 11GB, while the entries themselves were written off-heap. The histogram of the heap dump shows that there are 5 tuples of byte[] arrays with size 800MB and 1000MB that are left dangling on heap.  I am very curious if I did something wrong or if there indeed is an issue in Ignite. All details can be seen on https://stackoverflow.com/questions/64550479/possible-memory-leak-in-apache-ignite

Should I create a jira ticket for this issue? 

Best regards,
Kalin Katev

Resonance GmbH
Kalin Katev Kalin Katev
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak in Ignite 2.9

Hi,

I am sorry for responding so late. I can only send screenshots of the heap dump as seen in visualvm. I took screenshots of 2 different byte arrays, one is 800mb large, the other 1gb. Every cached value in ignite (which itself is 800mb) creates a tuple of these arrays. I hope this helps, or you can maybe give me an idea which references are crucial, as there are too many for me to investigate.

Thank you!

Kalin Katev
Resonance GmbH

Am Di., 3. Nov. 2020 um 15:56 Uhr schrieb Ilya Kasnacheev <[hidden email]>:
Hello!

800MB entry is far above of the entry size that we ever expected to see. Even brief holding of these entries on heap will cause problems for you, as well as sending them over communication.

I recommend splitting entries into chunks, maybe. That's what IGFS did basically, we decided to ax it, but you can still use that approach.

Having said that, if you can check the heap dump to see where are these stuck byte arrays referenced from, I may check it.

Regards,
--
Ilya Kasnacheev


вт, 3 нояб. 2020 г. в 11:48, Kalin Katev <[hidden email]>:
Hi, 

not too long ago I tested apache ignite for my use case on Open JDK 11. The use case consists of writing cache entries with values going up to 800MB in size, the data itself being a simple string. After writing 5 caches entries, 800 MB each, I noticed my Heap space exploding up to 11GB, while the entries themselves were written off-heap. The histogram of the heap dump shows that there are 5 tuples of byte[] arrays with size 800MB and 1000MB that are left dangling on heap.  I am very curious if I did something wrong or if there indeed is an issue in Ignite. All details can be seen on https://stackoverflow.com/questions/64550479/possible-memory-leak-in-apache-ignite

Should I create a jira ticket for this issue? 

Best regards,
Kalin Katev

Resonance GmbH

800mb array references.png (595K) Download Attachment
1gb byte array references.png (605K) Download Attachment
1gb byte array threadLocals.png (648K) Download Attachment
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak in Ignite 2.9

Hello!

I think this is expected. Ignite needs to convert an entry (800G array) into packets to send to other nodes or put to persistence, where it will allocate another array (800G more).

I don't think you can do anything here, short of chunking your data.

Regards,
--
Ilya Kasnacheev


пт, 13 нояб. 2020 г. в 17:04, Kalin Katev <[hidden email]>:
Hi,

I am sorry for responding so late. I can only send screenshots of the heap dump as seen in visualvm. I took screenshots of 2 different byte arrays, one is 800mb large, the other 1gb. Every cached value in ignite (which itself is 800mb) creates a tuple of these arrays. I hope this helps, or you can maybe give me an idea which references are crucial, as there are too many for me to investigate.

Thank you!

Kalin Katev
Resonance GmbH

Am Di., 3. Nov. 2020 um 15:56 Uhr schrieb Ilya Kasnacheev <[hidden email]>:
Hello!

800MB entry is far above of the entry size that we ever expected to see. Even brief holding of these entries on heap will cause problems for you, as well as sending them over communication.

I recommend splitting entries into chunks, maybe. That's what IGFS did basically, we decided to ax it, but you can still use that approach.

Having said that, if you can check the heap dump to see where are these stuck byte arrays referenced from, I may check it.

Regards,
--
Ilya Kasnacheev


вт, 3 нояб. 2020 г. в 11:48, Kalin Katev <[hidden email]>:
Hi, 

not too long ago I tested apache ignite for my use case on Open JDK 11. The use case consists of writing cache entries with values going up to 800MB in size, the data itself being a simple string. After writing 5 caches entries, 800 MB each, I noticed my Heap space exploding up to 11GB, while the entries themselves were written off-heap. The histogram of the heap dump shows that there are 5 tuples of byte[] arrays with size 800MB and 1000MB that are left dangling on heap.  I am very curious if I did something wrong or if there indeed is an issue in Ignite. All details can be seen on https://stackoverflow.com/questions/64550479/possible-memory-leak-in-apache-ignite

Should I create a jira ticket for this issue? 

Best regards,
Kalin Katev

Resonance GmbH