Failed to deserialize in a cluster with a distributed singleton job

classic Classic list List threaded Threaded
3 messages Options
dstieglitz dstieglitz
Reply | Threaded
Open this post in threaded view
|

Failed to deserialize in a cluster with a distributed singleton job

I'm trying to start an Ignite cluster from within a Spring 3 web application running in Tomcat 7.0.53.

I have one cluster service defined (singleton), and a REPLICATED cache. Discovery is multicast, most settings are default (OptimizedMarshaller(true), no peer class loading).

Each node starts fine individually, but when I try to scale the cluster out I get this exception and new nodes do not execute the jobs property although they are reported to be in the cluster. The job requires a bean from the underlying application context but can't find it (BeanNotDefinedException).

See below:
--------------------------

class org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$CacheClassLoader@4f3c8bd9
        at org.apache.ignite.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:239)
        at org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.unmarshal(IgniteCacheObjectProcessorImpl.java:89)
        at org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl.finishUnmarshal(KeyCacheObjectImpl.java:108)
        at org.apache.ignite.internal.processors.cache.GridCacheEntryInfo.unmarshal(GridCacheEntryInfo.java:406)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionSupplyMessage.finishUnmarshal(GridDhtPartitionSupplyMessage.java:249)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:744)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:227)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$700(GridCacheIoManager.java:45)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:792)
        at org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2008)
        at org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:880)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:58)
        at org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:847)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:460)
        at org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:746)
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:419)
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:507)
        at org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:751)
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:419)
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:507)
        at org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:751)
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at org.apache.ignite.internal.processors.datastructures.GridCacheSetItemKey.readExternal(GridCacheSetItemKey.java:102)
        at org.apache.ignite.internal.processors.datastructures.GridCacheSetImpl$CollocatedItemKey.readExternal(GridCacheSetImpl.java:588)
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:451)
        at org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:746)
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at org.apache.ignite.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:236)
        ... 15 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:455)
        ... 35 more
Caused by: java.lang.NullPointerException
        at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getCollection(DataStructuresProcessor.java:804)
        at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.set(DataStructuresProcessor.java:1124)
        at org.apache.ignite.internal.processors.datastructures.GridCacheSetProxy.readResolve(GridCacheSetProxy.java:548)
        ... 40 more
yakov yakov
Reply | Threaded
Open this post in threaded view
|

Re: Failed to deserialize in a cluster with a distributed singleton job

This seems to be a race condition which should be trivial to fix. I suspect that DataStructuredProcessor has not yet received "onKernalStarted", but  started processing messages. I created a ticket - https://issues.apache.org/jira/browse/IGNITE-1007

--Yakov

2015-06-10 1:16 GMT+03:00 dstieglitz <[hidden email]>:
I'm trying to start an Ignite cluster from within a Spring 3 web application
running in Tomcat 7.0.53.

I have one cluster service defined (singleton), and a REPLICATED cache.
Discovery is multicast, most settings are default
(OptimizedMarshaller(true), no peer class loading).

Each node starts fine individually, but when I try to scale the cluster out
I get this exception and new nodes do not execute the jobs property although
they are reported to be in the cluster. The job requires a bean from the
underlying application context but can't find it (BeanNotDefinedException).

See below:
--------------------------

class org.apache.ignite.IgniteCheckedException: Failed to deserialize object
with given class loader:
org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$CacheClassLoader@4f3c8bd9
        at
org.apache.ignite.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:239)
        at
org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.unmarshal(IgniteCacheObjectProcessorImpl.java:89)
        at
org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl.finishUnmarshal(KeyCacheObjectImpl.java:108)
        at
org.apache.ignite.internal.processors.cache.GridCacheEntryInfo.unmarshal(GridCacheEntryInfo.java:406)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionSupplyMessage.finishUnmarshal(GridDhtPartitionSupplyMessage.java:249)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:744)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:227)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$700(GridCacheIoManager.java:45)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:792)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2008)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:880)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:58)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:847)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:460)
        at
org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:746)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:419)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:507)
        at
org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:751)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:419)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:507)
        at
org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:751)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at
org.apache.ignite.internal.processors.datastructures.GridCacheSetItemKey.readExternal(GridCacheSetItemKey.java:102)
        at
org.apache.ignite.internal.processors.datastructures.GridCacheSetImpl$CollocatedItemKey.readExternal(GridCacheSetImpl.java:588)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:451)
        at
org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:746)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at
org.apache.ignite.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:236)
        ... 15 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:455)
        ... 35 more
Caused by: java.lang.NullPointerException
        at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getCollection(DataStructuresProcessor.java:804)
        at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.set(DataStructuresProcessor.java:1124)
        at
org.apache.ignite.internal.processors.datastructures.GridCacheSetProxy.readResolve(GridCacheSetProxy.java:548)
        ... 40 more



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-deserialize-in-a-cluster-with-a-distributed-singleton-job-tp462.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

yakov yakov
Reply | Threaded
Open this post in threaded view
|

Re: Failed to deserialize in a cluster with a distributed singleton job

This fix will be available in GridGain Community Edition 1.1.1 which will be available in a couple of days.

In the meantime you can omit set serialization and serialize set name only. On deserialization you can access the set from Ignite API - org.apache.ignite.Ignite#set - and this way you should not hit the issue. You can inject Ignite instance to your job, closure, predicate, etc with org.apache.ignite.resources.IgniteInstanceResource annotation. For details please refer to corresponding documentation.

--Yakov

2015-06-10 17:44 GMT+03:00 Yakov Zhdanov <[hidden email]>:
This seems to be a race condition which should be trivial to fix. I suspect that DataStructuredProcessor has not yet received "onKernalStarted", but  started processing messages. I created a ticket - https://issues.apache.org/jira/browse/IGNITE-1007

--Yakov

2015-06-10 1:16 GMT+03:00 dstieglitz <[hidden email]>:
I'm trying to start an Ignite cluster from within a Spring 3 web application
running in Tomcat 7.0.53.

I have one cluster service defined (singleton), and a REPLICATED cache.
Discovery is multicast, most settings are default
(OptimizedMarshaller(true), no peer class loading).

Each node starts fine individually, but when I try to scale the cluster out
I get this exception and new nodes do not execute the jobs property although
they are reported to be in the cluster. The job requires a bean from the
underlying application context but can't find it (BeanNotDefinedException).

See below:
--------------------------

class org.apache.ignite.IgniteCheckedException: Failed to deserialize object
with given class loader:
org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$CacheClassLoader@4f3c8bd9
        at
org.apache.ignite.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:239)
        at
org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.unmarshal(IgniteCacheObjectProcessorImpl.java:89)
        at
org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl.finishUnmarshal(KeyCacheObjectImpl.java:108)
        at
org.apache.ignite.internal.processors.cache.GridCacheEntryInfo.unmarshal(GridCacheEntryInfo.java:406)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionSupplyMessage.finishUnmarshal(GridDhtPartitionSupplyMessage.java:249)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:744)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:227)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$700(GridCacheIoManager.java:45)
        at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:792)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2008)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:880)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:58)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:847)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:460)
        at
org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:746)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:419)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:507)
        at
org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:751)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:419)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:507)
        at
org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:751)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at
org.apache.ignite.internal.processors.datastructures.GridCacheSetItemKey.readExternal(GridCacheSetItemKey.java:102)
        at
org.apache.ignite.internal.processors.datastructures.GridCacheSetImpl$CollocatedItemKey.readExternal(GridCacheSetImpl.java:588)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:451)
        at
org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:746)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
        at
org.apache.ignite.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:236)
        ... 15 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:455)
        ... 35 more
Caused by: java.lang.NullPointerException
        at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getCollection(DataStructuresProcessor.java:804)
        at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.set(DataStructuresProcessor.java:1124)
        at
org.apache.ignite.internal.processors.datastructures.GridCacheSetProxy.readResolve(GridCacheSetProxy.java:548)
        ... 40 more



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-deserialize-in-a-cluster-with-a-distributed-singleton-job-tp462.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.