Possible starvation in striped pool. message

classic Classic list List threaded Threaded
6 messages Options
kestas kestas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Possible starvation in striped pool. message

Hi,
 sometimes we get this message in logs. What does it mean ?

Jul 26, 2017 11:43:25 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: >>> Possible starvation in striped pool.
    Thread name: sys-stripe-3-#4%null%
    Queue: []
    Deadlock: false
    Completed: 17
Thread [name="sys-stripe-3-#4%null%", id=36, state=RUNNABLE, blockCnt=0, waitCnt=18]
        at o.a.i.i.MarshallerContextImpl.getClassName(MarshallerContextImpl.java:348)
        at o.a.i.i.MarshallerContextImpl.getClass(MarshallerContextImpl.java:335)
        at o.a.i.i.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:692)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1745)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1704)
        at o.a.i.i.binary.BinaryUtils.doReadObject(BinaryUtils.java:1728)
        at o.a.i.i.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:2085)
        at o.a.i.i.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2016)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1904)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1704)
        at o.a.i.i.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1964)
        at o.a.i.i.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:674)
        at o.a.i.i.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:164)
        at o.a.i.i.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:843)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1752)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1704)
        at o.a.i.i.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1964)
        at o.a.i.i.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:674)
        at o.a.i.i.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:164)
        at o.a.i.i.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:843)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1752)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1704)
        at o.a.i.i.binary.BinaryUtils.doReadObject(BinaryUtils.java:1728)
        at o.a.i.i.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:2085)
        at o.a.i.i.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2016)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1904)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1704)
        at o.a.i.i.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1964)
        at o.a.i.i.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:674)
        at o.a.i.i.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:164)
        at o.a.i.i.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:843)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1752)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1704)
        at o.a.i.i.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1964)
        at o.a.i.i.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:674)
        at o.a.i.i.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:164)
        at o.a.i.i.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:843)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1752)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1704)
        at o.a.i.i.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1964)
        at o.a.i.i.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:674)
        at o.a.i.i.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:164)
        at o.a.i.i.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:843)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1752)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1704)
        at o.a.i.i.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1964)
        at o.a.i.i.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:674)
        at o.a.i.i.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:164)
        at o.a.i.i.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:843)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1752)
        at o.a.i.i.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1704)
        at o.a.i.i.binary.BinaryObjectImpl.deserializeValue(BinaryObjectImpl.java:794)
        at o.a.i.i.binary.BinaryObjectImpl.value(BinaryObjectImpl.java:142)
        at o.a.i.i.processors.cache.CacheObjectContext.unwrapBinary(CacheObjectContext.java:273)
        at o.a.i.i.processors.cache.CacheObjectContext.unwrapBinaryIfNeeded(CacheObjectContext.java:161)
        at o.a.i.i.processors.cache.CacheObjectContext.unwrapBinaryIfNeeded(CacheObjectContext.java:148)
        at o.a.i.i.processors.cache.GridCacheContext.unwrapBinaryIfNeeded(GridCacheContext.java:1730)
        at o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.setResult(GridPartitionedSingleGetFuture.java:613)
        at o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onResult(GridPartitionedSingleGetFuture.java:475)
        at o.a.i.i.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetResponse(GridDhtCacheAdapter.java:156)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$1200(GridDhtAtomicCache.java:127)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$14.apply(GridDhtAtomicCache.java:416)
        at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$14.apply(GridDhtAtomicCache.java:411)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:863)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:386)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:308)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:100)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:253)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1257)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:885)
        at o.a.i.i.managers.communication.GridIoManager.access$2100(GridIoManager.java:114)
        at o.a.i.i.managers.communication.GridIoManager$7.run(GridIoManager.java:802)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:483)
        at java.lang.Thread.run(Thread.java:745)
slava.koptilin slava.koptilin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Possible starvation in striped pool. message

Hi Kestas,

There are several possible reasons for that.
In your case, I think you are trying to put or get huge objects, that contain internal collections and so on.

>        at o.a.i.i.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2016)
>        at o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1904)
...
>        at o.a.i.i.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2016)
>        at o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1904)

Another possible reason is intensive use of getAll()/putAll() methods from different threads.
In that case, you need to sort collection of keys before, because batch operation on the same entries in different order could lead to deadlock.


Thanks.
kestas kestas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Possible starvation in striped pool. message

Yes, this seems to appear when we start working with large objects. Is there a way to solve this? Does it affect cache put/get operations performance directly ?
yakov yakov
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Possible starvation in striped pool. message

Hi!

Ignite prints stacktrace of an internal thread if it detects that thread runs the same task (single get response in your case) for a long time, which is 30 seconds by default.

Of course, getting large objects from remote nodes is expensive. Can you please estimate the value size?

Possible ways of speeding up the application:
1. Refactor object model to reduce values size
2. use org.apache.ignite.IgniteCache#withKeepBinary - this will return IgniteCache that will not deserialize binary objects, therefore there will be no deserialization. Take a look at this page for info - https://apacheignite.readme.io/docs/binary-marshaller
3. use org.apache.ignite.IgniteCache#invoke(K, org.apache.ignite.cache.CacheEntryProcessor<K,V,T>, java.lang.Object...) instead of pure get() providing the processor that will not be altering the value, but return only some part of the object (e.g. tuple of last name and city instead of returning the entire User object). 
4. confiure NearCache - https://apacheignite.readme.io/docs/near-caches and set copy on read (org.apache.ignite.configuration.CacheConfiguration#isCopyOnRead) to false - however, bringing large objects to client may overload the client and introduce new problems.

I would first try 1-3. Please let us know the results.

--Yakov

kestas kestas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Possible starvation in striped pool. message

I did some testing on #invoke vs #get performance. It works as expected on ATOMIC cache, however on TRANSACTIONAL cache #invoke has even lower performance than pure #get. Is this to be expected?

simplified code:
some = cacheBinary.invoke(1,(mutableEntry, objects) -> mutableEntry.getValue().field("some"));
and
some = cache.get(1).getSome();
Yakov Zhdanov Yakov Zhdanov
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Possible starvation in striped pool. message

Well, we need to take a closer look then. This may be affected by transaction protocol. Viacheslav Koptilin, can you please create a test and see what time goes to?

kestas, you can switch to Ignite.compute().affinityCall("key", () -> {return cacheBinary.get("key").field("f"));}); This should work fine for both transactional and atomic caches.

Thanks!
--
Yakov Zhdanov, Director R&D
GridGain Systems

2017-08-09 16:30 GMT+03:00 kestas <[hidden email]>:
I did some testing on #invoke vs #get performance. It works as expected on
ATOMIC cache, however on TRANSACTIONAL cache #invoke has even lower
performance than pure #get. Is this to be expected?

simplified code:
some = cacheBinary.invoke(1,(mutableEntry, objects) ->
mutableEntry.getValue().field("some"));
and
some = cache.get(1).getSome();



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Possible-starvation-in-striped-pool-message-tp15993p16081.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Loading...