Ignite Client Node Stopped With Out Of Memory

classic Classic list List threaded Threaded
9 messages Options
Ravi Makwana Ravi Makwana
Reply | Threaded
Open this post in threaded view
|

Ignite Client Node Stopped With Out Of Memory

Hi,

We are using Apache Ignite 2.7.0 binary and servers are using Linux OS & app servers are using Windows OS.We are using Apache Ignite .Net APIs.

Recently we have noticed that our application is stopping due to a client node throwing Out Of Memory error which we have seen in ignite client node log file.

App server has 32 GB RAM & We are specifying JVM Heap = 8 GB & Data Region Min = 1 GB & Max = 10 GB.

Exception:

class org.apache.ignite.IgniteException: Platform error:System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
   at System.Runtime.InteropServices.Marshal.ReAllocHGlobal(IntPtr pv, IntPtr cb)
   at Apache.Ignite.Core.Impl.Memory.PlatformMemoryUtils.ReallocatePooled(Int64 memPtr, Int32 cap)
   at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.MemoryReallocate(Int64 memPtr, Int64 cap, Int64 unused, Void* arg)
   at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.InLongLongLongObjectOutLong(Int32 type, Int64 val1, Int64 val2, Int64 val3, IntPtr arg)
at org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.loggerLog(PlatformProcessorImpl.java:404)
at org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:460)
at org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:512)
at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67)
at org.apache.ignite.internal.processors.platform.callback.PlatformCallbackUtils.inLongLongLongObjectOutLong(Native Method)
at org.apache.ignite.internal.processors.platform.callback.PlatformCallbackGateway.memoryReallocate(PlatformCallbackGateway.java:840)
at org.apache.ignite.internal.processors.platform.memory.PlatformExternalMemory.reallocate(PlatformExternalMemory.java:48)
at org.apache.ignite.internal.processors.platform.memory.PlatformOutputStreamImpl.ensureCapacity(PlatformOutputStreamImpl.java:305)
at org.apache.ignite.internal.processors.platform.memory.PlatformOutputStreamImpl.unsafeEnsure(PlatformOutputStreamImpl.java:222)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteBinaryObject(BinaryWriterExImpl.java:962)
at org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:744)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:223)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:164)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:151)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.writeObjectDetached(BinaryWriterExImpl.java:1506)
at org.apache.ignite.internal.processors.platform.cache.query.PlatformQueryCursor.write(PlatformQueryCursor.java:43)
at org.apache.ignite.internal.processors.platform.cache.query.PlatformQueryCursor.write(PlatformQueryCursor.java:28)
at org.apache.ignite.internal.processors.platform.cache.query.PlatformAbstractQueryCursor.processOutStream(PlatformAbstractQueryCursor.java:85)
at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.outStream(PlatformTargetProxyImpl.java:93)

Can please suggest what is the root cause for the above issue?

Thanks & Regards,

ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Client Node Stopped With Out Of Memory

Hi Ravi,

The exception indicates that Ignite.NET failed to allocate unmanaged memory on .NET side while trying to pass query data from Java to .NET.

This indicates that your system has run out of memory. Possible reasons are:
* Memory is consumed by other apps
* Memory is consumed by this app
* Memory leak in your code
* Memory leak in Ignite.NET code

I would suggest the following steps:
* Find out which app consumes the memory. Are we sure it is the app with Ignite?
* If yes, run a memory profiler of your choice to figure out what consumes the memory

Thanks,
Pavel

On Sat, Oct 24, 2020 at 2:04 PM Ravi Makwana <[hidden email]> wrote:
Hi,

We are using Apache Ignite 2.7.0 binary and servers are using Linux OS & app servers are using Windows OS.We are using Apache Ignite .Net APIs.

Recently we have noticed that our application is stopping due to a client node throwing Out Of Memory error which we have seen in ignite client node log file.

App server has 32 GB RAM & We are specifying JVM Heap = 8 GB & Data Region Min = 1 GB & Max = 10 GB.

Exception:

class org.apache.ignite.IgniteException: Platform error:System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
   at System.Runtime.InteropServices.Marshal.ReAllocHGlobal(IntPtr pv, IntPtr cb)
   at Apache.Ignite.Core.Impl.Memory.PlatformMemoryUtils.ReallocatePooled(Int64 memPtr, Int32 cap)
   at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.MemoryReallocate(Int64 memPtr, Int64 cap, Int64 unused, Void* arg)
   at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.InLongLongLongObjectOutLong(Int32 type, Int64 val1, Int64 val2, Int64 val3, IntPtr arg)
at org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.loggerLog(PlatformProcessorImpl.java:404)
at org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:460)
at org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:512)
at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67)
at org.apache.ignite.internal.processors.platform.callback.PlatformCallbackUtils.inLongLongLongObjectOutLong(Native Method)
at org.apache.ignite.internal.processors.platform.callback.PlatformCallbackGateway.memoryReallocate(PlatformCallbackGateway.java:840)
at org.apache.ignite.internal.processors.platform.memory.PlatformExternalMemory.reallocate(PlatformExternalMemory.java:48)
at org.apache.ignite.internal.processors.platform.memory.PlatformOutputStreamImpl.ensureCapacity(PlatformOutputStreamImpl.java:305)
at org.apache.ignite.internal.processors.platform.memory.PlatformOutputStreamImpl.unsafeEnsure(PlatformOutputStreamImpl.java:222)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteBinaryObject(BinaryWriterExImpl.java:962)
at org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:744)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:223)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:164)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:151)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.writeObjectDetached(BinaryWriterExImpl.java:1506)
at org.apache.ignite.internal.processors.platform.cache.query.PlatformQueryCursor.write(PlatformQueryCursor.java:43)
at org.apache.ignite.internal.processors.platform.cache.query.PlatformQueryCursor.write(PlatformQueryCursor.java:28)
at org.apache.ignite.internal.processors.platform.cache.query.PlatformAbstractQueryCursor.processOutStream(PlatformAbstractQueryCursor.java:85)
at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.outStream(PlatformTargetProxyImpl.java:93)

Can please suggest what is the root cause for the above issue?

Thanks & Regards,

Ravi Makwana Ravi Makwana
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Client Node Stopped With Out Of Memory

Hi Pavel,

Thanks for the information.

We have noticed our service is taking max 2GB memory which is noticed in task manager and stopping app pool at same time server memory utilization is 80% so we have still memory spare 20%.

Still we are not able to correlate the issue, I would like to share some more logs. Could you suggest what is happening at the moment?

Thanks,


ignite-fd6974c5.0.txt (176K) Download Attachment
ignite-dd4c2e1e.0.txt (184K) Download Attachment
ignite-810a2897.0.txt (177K) Download Attachment
ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Client Node Stopped With Out Of Memory

Looks like the app is running in 32 bit mode, which can't use more than 2GB of memory.
JVM and memory regions pre-allocate all of it, leaving nothing for .NET to use.

Please check the `Platform` column in the Task Manager - does it say `32 bit`?
If yes, then try disabling `Prefer 32 bit` in the project properties.

On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <[hidden email]> wrote:
Hi Pavel,

Thanks for the information.

We have noticed our service is taking max 2GB memory which is noticed in task manager and stopping app pool at same time server memory utilization is 80% so we have still memory spare 20%.

Still we are not able to correlate the issue, I would like to share some more logs. Could you suggest what is happening at the moment?

Thanks,

Ravi Makwana Ravi Makwana
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Client Node Stopped With Out Of Memory

Hi,

Our service is running with 64 bit and we have verified the same in our app server too.

Any finding from the logs?

Is there any way to replicate it?

Thanks,


On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn <[hidden email]> wrote:
Looks like the app is running in 32 bit mode, which can't use more than 2GB of memory.
JVM and memory regions pre-allocate all of it, leaving nothing for .NET to use.

Please check the `Platform` column in the Task Manager - does it say `32 bit`?
If yes, then try disabling `Prefer 32 bit` in the project properties.

On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <[hidden email]> wrote:
Hi Pavel,

Thanks for the information.

We have noticed our service is taking max 2GB memory which is noticed in task manager and stopping app pool at same time server memory utilization is 80% so we have still memory spare 20%.

Still we are not able to correlate the issue, I would like to share some more logs. Could you suggest what is happening at the moment?

Thanks,

ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Client Node Stopped With Out Of Memory

I found a bug in Ignite [1] which probably causes the issue on your side.

Looks like you are running a query (is it a ScanQuery or SqlQuery?) and the size of one results page exceeds 2GB.
Please try using a smaller value for QueryBase.PageSize.

If you use the default value of 1024, your cache entries are very large. What does the data look like?


On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana <[hidden email]> wrote:
Hi,

Our service is running with 64 bit and we have verified the same in our app server too.

Any finding from the logs?

Is there any way to replicate it?

Thanks,


On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn <[hidden email]> wrote:
Looks like the app is running in 32 bit mode, which can't use more than 2GB of memory.
JVM and memory regions pre-allocate all of it, leaving nothing for .NET to use.

Please check the `Platform` column in the Task Manager - does it say `32 bit`?
If yes, then try disabling `Prefer 32 bit` in the project properties.

On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <[hidden email]> wrote:
Hi Pavel,

Thanks for the information.

We have noticed our service is taking max 2GB memory which is noticed in task manager and stopping app pool at same time server memory utilization is 80% so we have still memory spare 20%.

Still we are not able to correlate the issue, I would like to share some more logs. Could you suggest what is happening at the moment?

Thanks,

Ravi Makwana Ravi Makwana
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Client Node Stopped With Out Of Memory





On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn, <[hidden email]> wrote:
I found a bug in Ignite [1] which probably causes the issue on your side.

Looks like you are running a query (is it a ScanQuery or SqlQuery?) and the size of one results page exceeds 2GB.
Please try using a smaller value for QueryBase.PageSize.

If you use the default value of 1024, your cache entries are very large. What does the data look like?


On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana <[hidden email]> wrote:
Hi,

Our service is running with 64 bit and we have verified the same in our app server too.

Any finding from the logs?

Is there any way to replicate it?

Thanks,


On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn <[hidden email]> wrote:
Looks like the app is running in 32 bit mode, which can't use more than 2GB of memory.
JVM and memory regions pre-allocate all of it, leaving nothing for .NET to use.

Please check the `Platform` column in the Task Manager - does it say `32 bit`?
If yes, then try disabling `Prefer 32 bit` in the project properties.

On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <[hidden email]> wrote:
Hi Pavel,

Thanks for the information.

We have noticed our service is taking max 2GB memory which is noticed in task manager and stopping app pool at same time server memory utilization is 80% so we have still memory spare 20%.

Still we are not able to correlate the issue, I would like to share some more logs. Could you suggest what is happening at the moment?

Thanks,

Ravi Makwana Ravi Makwana
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Client Node Stopped With Out Of Memory

Hi Paval,

As we are not setting explicitly QueryBase.Pagesize for SqlQuery and SqlFieldQuery so default will be used as 1024.

We have not found so far any example by looking the same we can try to explicitly set the Query base.Pagesize.

Can we have any reference by checking that we can try to set the lower value and can able to replicate the scenario?


Thanks,

On Wed, 28 Oct, 2020, 7:53 pm Ravi Makwana, <[hidden email]> wrote:




On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn, <[hidden email]> wrote:
I found a bug in Ignite [1] which probably causes the issue on your side.

Looks like you are running a query (is it a ScanQuery or SqlQuery?) and the size of one results page exceeds 2GB.
Please try using a smaller value for QueryBase.PageSize.

If you use the default value of 1024, your cache entries are very large. What does the data look like?


On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana <[hidden email]> wrote:
Hi,

Our service is running with 64 bit and we have verified the same in our app server too.

Any finding from the logs?

Is there any way to replicate it?

Thanks,


On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn <[hidden email]> wrote:
Looks like the app is running in 32 bit mode, which can't use more than 2GB of memory.
JVM and memory regions pre-allocate all of it, leaving nothing for .NET to use.

Please check the `Platform` column in the Task Manager - does it say `32 bit`?
If yes, then try disabling `Prefer 32 bit` in the project properties.

On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <[hidden email]> wrote:
Hi Pavel,

Thanks for the information.

We have noticed our service is taking max 2GB memory which is noticed in task manager and stopping app pool at same time server memory utilization is 80% so we have still memory spare 20%.

Still we are not able to correlate the issue, I would like to share some more logs. Could you suggest what is happening at the moment?

Thanks,

ptupitsyn ptupitsyn
Reply | Threaded
Open this post in threaded view
|

Re: Ignite Client Node Stopped With Out Of Memory

On of these, depending on your query type:
* new ScanQuery<K, V>() { PageSize = 5 }
* new SqlQuery() { PageSize = 5 }



On Wed, Oct 28, 2020 at 5:24 PM Ravi Makwana <[hidden email]> wrote:
Hi Paval,

As we are not setting explicitly QueryBase.Pagesize for SqlQuery and SqlFieldQuery so default will be used as 1024.

We have not found so far any example by looking the same we can try to explicitly set the Query base.Pagesize.

Can we have any reference by checking that we can try to set the lower value and can able to replicate the scenario?


Thanks,

On Wed, 28 Oct, 2020, 7:53 pm Ravi Makwana, <[hidden email]> wrote:




On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn, <[hidden email]> wrote:
I found a bug in Ignite [1] which probably causes the issue on your side.

Looks like you are running a query (is it a ScanQuery or SqlQuery?) and the size of one results page exceeds 2GB.
Please try using a smaller value for QueryBase.PageSize.

If you use the default value of 1024, your cache entries are very large. What does the data look like?


On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana <[hidden email]> wrote:
Hi,

Our service is running with 64 bit and we have verified the same in our app server too.

Any finding from the logs?

Is there any way to replicate it?

Thanks,


On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn <[hidden email]> wrote:
Looks like the app is running in 32 bit mode, which can't use more than 2GB of memory.
JVM and memory regions pre-allocate all of it, leaving nothing for .NET to use.

Please check the `Platform` column in the Task Manager - does it say `32 bit`?
If yes, then try disabling `Prefer 32 bit` in the project properties.

On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <[hidden email]> wrote:
Hi Pavel,

Thanks for the information.

We have noticed our service is taking max 2GB memory which is noticed in task manager and stopping app pool at same time server memory utilization is 80% so we have still memory spare 20%.

Still we are not able to correlate the issue, I would like to share some more logs. Could you suggest what is happening at the moment?

Thanks,