Ignite server send big data to client and make client crash

classic Classic list List threaded Threaded
6 messages Options
shawn.du shawn.du
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Ignite server send big data to client and make client crash

Hi community,

I have a ignite server(1.9.0) and a ignite client.

Recently my client crashed 4 times.  I found it may be caused by server send big data to client. 

see blow picture.  this is AWS Cloud Watch network-out-bytes picture of ignite server. 

It seems that ignite server do something hourly.  My client crashed at the red points.
one time is 19:33, the other is 23:56.

is there any settings to avoid this?

Thanks in advance.


Thanks
Shawn

shawn.du shawn.du
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ignite server send big data to client and make client crash


sorry, Ignore this.

Thanks
Shawn

On 08/11/2017 11:03[hidden email] wrote:
Hi community,

I have a ignite server(1.9.0) and a ignite client.

Recently my client crashed 4 times.  I found it may be caused by server send big data to client. 

see blow picture.  this is AWS Cloud Watch network-out-bytes picture of ignite server. 

It seems that ignite server do something hourly.  My client crashed at the red points.
one time is 19:33, the other is 23:56.

is there any settings to avoid this?

Thanks in advance.


Thanks
Shawn

mcherkasov mcherkasov
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ignite server send big data to client and make client crash

Hi Shawn,

Did you figure out what was wrong?

Thanks,
Mikhail.


On Fri, Aug 11, 2017 at 9:31 AM, shawn.du <[hidden email]> wrote:

sorry, Ignore this.

Thanks
Shawn

On 08/11/2017 11:03[hidden email] wrote:
Hi community,

I have a ignite server(1.9.0) and a ignite client.

Recently my client crashed 4 times.  I found it may be caused by server send big data to client. 

see blow picture.  this is AWS Cloud Watch network-out-bytes picture of ignite server. 

It seems that ignite server do something hourly.  My client crashed at the red points.
one time is 19:33, the other is 23:56.

is there any settings to avoid this?

Thanks in advance.


Thanks
Shawn




--
Thanks,
Mikhail.
Mikhail Getmanov Mikhail Getmanov
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ignite server send big data to client and make client crash



--
Отправлено из мобильного приложения Яндекс.Почты

14:59, 11 августа 2017 г., Mikhail Cherkasov <[hidden email]>:
Hi Shawn,

Did you figure out what was wrong?

Thanks,
Mikhail.


On Fri, Aug 11, 2017 at 9:31 AM, shawn.du <[hidden email]> wrote:










sorry, Ignore this.









Thanks
Shawn







On 08/11/2017 11:03[hidden email] wrote:






Hi community,

I have a ignite server(1.9.0) and a ignite client.

Recently my client crashed 4 times.  I found it may be caused by server send big data to client. 

see blow picture.  this is AWS Cloud Watch network-out-bytes picture of ignite server. 

It seems that ignite server do something hourly.  My client crashed at the red points.
one time is 19:33, the other is 23:56.

is there any settings to avoid this?

Thanks in advance.

<inline-image>BEC4F645-C470-4CE0-BBA5-DF205D72D00C@mailmaster








Thanks
Shawn













--
Thanks,
Mikhail.


shawn.du shawn.du
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ignite server send big data to client and make client crash

In reply to this post by mcherkasov
Hi,

No, still in investigation.  

This is the CPU utilization of ignite server. You can see that there are 3 points which are bigger that others.

Ignite client  disconnected at these time. In fact we use ignite in storm, this will cause storm worker stopped and restart.



We suspect below code, but not sure.

The Interceptor we persist cache entry into file in json format.  there is a rotate policy. currently We use time-based policy,
so file will be moved to another place hourly. This is the reason in the network-out-bytes chart, some big points appears hourly.
But not all of them cause by the interceptor.  The 2nd and the 3rd red point are exceptions. I don't know why they occurred now.

My question is the CPU is not so high, why client is disconnected from server. 

Thanks in advance.
=====================code=============================
public class PersistJsonInterceptor<K, V> extends CacheInterceptorAdapter<K, V>
{
    @IgniteInstanceResource
    private Ignite ignite;
    @LoggerResource
    private IgniteLogger logger
    private transient BlockingQueue<V> queue = new LinkedBlockingQueue<>();
    public void onAfterRemove(Cache.Entry<K, V> entry)
    {
if (worker == null)
        {
            ...
            worker = new WriteWorker(localTmpDir, destDir, partitionId, policy);
            new Thread(worker).start();
        }
        ...
        if (value == null)
        {
            return;
        }
        queue.add(value);
    }
   class WriteWorker implements Runnable
  {
@Override
        public void run()
        {
            while (true)
            {
                try
                {
                    V value = queue.take();
                    init(value.getClass());
                    objectMapper.writeValue(fileWriter, value);
                    fileWriter.append("\n");
                    if (policy.rotateReady(this.file))
                    {
          ...
  this.fileWriter.close();
  policy.rotate(this.file, new File(this.destDir), partitionId);
                        ...
                    }
                }
                catch (Exception e)
                {
                    logger.error("fail to persist data into json file ...", e);
                }
            }
        }
    }
}

Thanks
Shawn

On 08/11/2017 15:59[hidden email] wrote:
Hi Shawn,

Did you figure out what was wrong?

Thanks,
Mikhail.


On Fri, Aug 11, 2017 at 9:31 AM, shawn.du <[hidden email]> wrote:

sorry, Ignore this.

Thanks
Shawn

On 08/11/2017 11:03[hidden email] wrote:
Hi community,

I have a ignite server(1.9.0) and a ignite client.

Recently my client crashed 4 times.  I found it may be caused by server send big data to client. 

see blow picture.  this is AWS Cloud Watch network-out-bytes picture of ignite server. 

It seems that ignite server do something hourly.  My client crashed at the red points.
one time is 19:33, the other is 23:56.

is there any settings to avoid this?

Thanks in advance.


Thanks
Shawn




--
Thanks,
Mikhail.
shawn.du shawn.du
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ignite server send big data to client and make client crash

BTW,

We also use ignite file system and message pub/sub technology. will they do some sync which may cause high cpu/traffic?

Thanks
Shawn

On 08/12/2017 08:04[hidden email] wrote:
Hi,

No, still in investigation.  

This is the CPU utilization of ignite server. You can see that there are 3 points which are bigger that others.

Ignite client  disconnected at these time. In fact we use ignite in storm, this will cause storm worker stopped and restart.



We suspect below code, but not sure.

The Interceptor we persist cache entry into file in json format.  there is a rotate policy. currently We use time-based policy,
so file will be moved to another place hourly. This is the reason in the network-out-bytes chart, some big points appears hourly.
But not all of them cause by the interceptor.  The 2nd and the 3rd red point are exceptions. I don't know why they occurred now.

My question is the CPU is not so high, why client is disconnected from server. 

Thanks in advance.
=====================code=============================
public class PersistJsonInterceptor<K, V> extends CacheInterceptorAdapter<K, V>
{
    @IgniteInstanceResource
    private Ignite ignite;
    @LoggerResource
    private IgniteLogger logger
    private transient BlockingQueue<V> queue = new LinkedBlockingQueue<>();
    public void onAfterRemove(Cache.Entry<K, V> entry)
    {
if (worker == null)
        {
            ...
            worker = new WriteWorker(localTmpDir, destDir, partitionId, policy);
            new Thread(worker).start();
        }
        ...
        if (value == null)
        {
            return;
        }
        queue.add(value);
    }
   class WriteWorker implements Runnable
  {
@Override
        public void run()
        {
            while (true)
            {
                try
                {
                    V value = queue.take();
                    init(value.getClass());
                    objectMapper.writeValue(fileWriter, value);
                    fileWriter.append("\n");
                    if (policy.rotateReady(this.file))
                    {
          ...
  this.fileWriter.close();
  policy.rotate(this.file, new File(this.destDir), partitionId);
                        ...
                    }
                }
                catch (Exception e)
                {
                    logger.error("fail to persist data into json file ...", e);
                }
            }
        }
    }
}

Thanks
Shawn

On 08/11/2017 15:59[hidden email] wrote:
Hi Shawn,

Did you figure out what was wrong?

Thanks,
Mikhail.


On Fri, Aug 11, 2017 at 9:31 AM, shawn.du <[hidden email]> wrote:

sorry, Ignore this.

Thanks
Shawn

On 08/11/2017 11:03[hidden email] wrote:
Hi community,

I have a ignite server(1.9.0) and a ignite client.

Recently my client crashed 4 times.  I found it may be caused by server send big data to client. 

see blow picture.  this is AWS Cloud Watch network-out-bytes picture of ignite server. 

It seems that ignite server do something hourly.  My client crashed at the red points.
one time is 19:33, the other is 23:56.

is there any settings to avoid this?

Thanks in advance.


Thanks
Shawn




--
Thanks,
Mikhail.
Loading...