After upgrading 2.7 getting Unexpected error occurred during unmarshalling

classic Classic list List threaded Threaded
15 messages Options
akash shinde akash shinde
Reply | Threaded
Open this post in threaded view
|

After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Hi,

When execute affinity.partition(key), I am getting following exception on Ignite  2.7.

Stacktrace:


2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to authenticate using SASL (unknown error)
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has expired
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has expired, closing socket connection
2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=qzkpr01.p13.eng.in03.com:50024,qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x3673a59f5dd01c7
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to authenticate using SASL (unknown error)
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid = 0x3673a59f5dd01ce, negotiated timeout = 10000
2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted
2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted

Thanks,
Akash
akash shinde akash shinde
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Before submitting the affinity task ignite first gets the affinity cached function (AffinityInfo) by submitting the cluster wide task "AffinityJob". But while in the process of retrieving the output of this AffinityJob, ignite deserializes this output. I am getting exception while deserailizing this output.
In TcpDiscoveryNode.readExternal() method while deserailizing the CacheMetrics object from input stream on 14th iteration I am getting following exception. Complete stack trace is given in this mail chain.

Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.  

This is working fine on Ignite 2.6 version but giving problem on 2.7. 

Is this a bug or am I doing something wrong?

Can someone please help?

On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]> wrote:

Hi,

When execute affinity.partition(key), I am getting following exception on Ignite  2.7.

Stacktrace:


2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to authenticate using SASL (unknown error)
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has expired
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has expired, closing socket connection
2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=qzkpr01.p13.eng.in03.com:50024,qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x3673a59f5dd01c7
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to authenticate using SASL (unknown error)
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid = 0x3673a59f5dd01ce, negotiated timeout = 10000
2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted
2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted

Thanks,
Akash
Mikael Mikael
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Hi!

Any chance you might have one node running 2.6 or something like that ?

It looks like it get a different object that does not match the one expected in 2.7

Mikael

Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
Before submitting the affinity task ignite first gets the affinity cached function (AffinityInfo) by submitting the cluster wide task "AffinityJob". But while in the process of retrieving the output of this AffinityJob, ignite deserializes this output. I am getting exception while deserailizing this output.
In TcpDiscoveryNode.readExternal() method while deserailizing the CacheMetrics object from input stream on 14th iteration I am getting following exception. Complete stack trace is given in this mail chain.

Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.  

This is working fine on Ignite 2.6 version but giving problem on 2.7. 

Is this a bug or am I doing something wrong?

Can someone please help?

On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]> wrote:

Hi,

When execute affinity.partition(key), I am getting following exception on Ignite  2.7.

Stacktrace:


2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to authenticate using SASL (unknown error)
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has expired
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has expired, closing socket connection
2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=qzkpr01.p13.eng.in03.com:50024,qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x3673a59f5dd01c7
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to authenticate using SASL (unknown error)
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid = 0x3673a59f5dd01ce, negotiated timeout = 10000
2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted
2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted

Thanks,
Akash
akash shinde akash shinde
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Hi,

No both nodes, client and server are running on Ignite 2.7 version. I am starting both server and client from Intellij IDE. 

Version printed in Server node log:
Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db

Version in client node log:
Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db  

Thanks,
Akash

On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote:

Hi!

Any chance you might have one node running 2.6 or something like that ?

It looks like it get a different object that does not match the one expected in 2.7

Mikael

Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
Before submitting the affinity task ignite first gets the affinity cached function (AffinityInfo) by submitting the cluster wide task "AffinityJob". But while in the process of retrieving the output of this AffinityJob, ignite deserializes this output. I am getting exception while deserailizing this output.
In TcpDiscoveryNode.readExternal() method while deserailizing the CacheMetrics object from input stream on 14th iteration I am getting following exception. Complete stack trace is given in this mail chain.

Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.  

This is working fine on Ignite 2.6 version but giving problem on 2.7. 

Is this a bug or am I doing something wrong?

Can someone please help?

On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]> wrote:

Hi,

When execute affinity.partition(key), I am getting following exception on Ignite  2.7.

Stacktrace:


2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to authenticate using SASL (unknown error)
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has expired
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has expired, closing socket connection
2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=qzkpr01.p13.eng.in03.com:50024,qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x3673a59f5dd01c7
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to authenticate using SASL (unknown error)
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid = 0x3673a59f5dd01ce, negotiated timeout = 10000
2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted
2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted

Thanks,
Akash
akash shinde akash shinde
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Added  [hidden email].

Should I log Jira for this issue?

Thanks,
Akash



On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote:
Hi,

No both nodes, client and server are running on Ignite 2.7 version. I am starting both server and client from Intellij IDE. 

Version printed in Server node log:
Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db

Version in client node log:
Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db  

Thanks,
Akash

On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote:

Hi!

Any chance you might have one node running 2.6 or something like that ?

It looks like it get a different object that does not match the one expected in 2.7

Mikael

Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
Before submitting the affinity task ignite first gets the affinity cached function (AffinityInfo) by submitting the cluster wide task "AffinityJob". But while in the process of retrieving the output of this AffinityJob, ignite deserializes this output. I am getting exception while deserailizing this output.
In TcpDiscoveryNode.readExternal() method while deserailizing the CacheMetrics object from input stream on 14th iteration I am getting following exception. Complete stack trace is given in this mail chain.

Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.  

This is working fine on Ignite 2.6 version but giving problem on 2.7. 

Is this a bug or am I doing something wrong?

Can someone please help?

On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]> wrote:

Hi,

When execute affinity.partition(key), I am getting following exception on Ignite  2.7.

Stacktrace:


2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to authenticate using SASL (unknown error)
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has expired
2019-01-07 21:23:03,140 6699925 [main-SendThread(qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has expired, closing socket connection
2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=qzkpr01.p13.eng.in03.com:50024,qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x3673a59f5dd01c7
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to authenticate using SASL (unknown error)
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
2019-01-07 21:23:03,296 6700081 [main-SendThread(qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid = 0x3673a59f5dd01ce, negotiated timeout = 10000
2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted
2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, isCancelled=false, retry=null]
org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
 at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
 at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
 at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
 ... 10 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
 at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
 ... 16 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
 ... 18 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 21 common frames omitted
Caused by: java.io.IOException: Failed to deserialize field [name=assignment]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 27 common frames omitted
Caused by: java.io.IOException: Failed to deserialize object [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
 ... 30 common frames omitted
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
 at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ... 41 common frames omitted

Thanks,
Akash
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Hello!

Do you have a reproducer project to reliably confirm this issue?

Regards,
--
Ilya Kasnacheev


ср, 9 янв. 2019 г. в 12:39, Akash Shinde <[hidden email]>:
Added  [hidden email].

Should I log Jira for this issue?

Thanks,
Akash



On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote:

> Hi,
>
> No both nodes, client and server are running on Ignite 2.7 version. I am
> starting both server and client from Intellij IDE.
>
> Version printed in Server node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Version in client node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Thanks,
> Akash
>
> On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote:
>
>> Hi!
>>
>> Any chance you might have one node running 2.6 or something like that ?
>>
>> It looks like it get a different object that does not match the one
>> expected in 2.7
>>
>> Mikael
>> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
>>
>> Before submitting the affinity task ignite first gets the affinity cached
>> function (AffinityInfo) by submitting the cluster wide task "AffinityJob".
>> But while in the process of retrieving the output of this AffinityJob,
>> ignite deserializes this output. I am getting exception while deserailizing
>> this output.
>> In TcpDiscoveryNode.readExternal() method while deserailizing the
>> CacheMetrics object from input stream on 14th iteration I am getting
>> following exception. Complete stack trace is given in this mail chain.
>>
>> Caused by: java.io.IOException: Unexpected error occurred during
>> unmarshalling of an instance of the class:
>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>>
>> This is working fine on Ignite 2.6 version but giving problem on 2.7.
>>
>> Is this a bug or am I doing something wrong?
>>
>> Can someone please help?
>>
>> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> When execute affinity.partition(key), I am getting following exception
>>> on Ignite  2.7.
>>>
>>> Stacktrace:
>>>
>>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired, closing socket connection
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.apache.zookeeper.ZooKeeper - Initiating client connection,
>>> connectString=qzkpr01.p13.eng.in03.com:50024,
>>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka
>>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
>>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO
>>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
>>> 0x3673a59f5dd01c7
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Session establishment complete on server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid =
>>> 0x3673a59f5dd01ce, negotiated timeout = 10000
>>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
>>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>>
>>> Thanks,
>>> Akash
>>>
>>
prasadbhalerao1983 prasadbhalerao1983
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting error during unmarshalling (Works fine on 2.6)


Hi Ilya,

I am not able reproduce this issue in small reproducer project but this is consistently happening in my project. So I debugged this issue and attached the screenshot in this mail.

 

NOTE:  This issue occurs if the statistics are enabled on cache configuration level [cacheCfg.setStatisticsEnabled(true)].

 

As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. All these cache metrics gets serialized successfully on server node. But at the time of deserialization on client node only first metrics get de-serialized successfully but all other metrics till iteration count 13 are de-serialized as null value and on iteration 14 “ref” byte value in “OptimizedObjectInputStream.readObject0()” method is read as 81 and code throws exception.

 

I think this where it is going wrong. The object copy at the time of serialization and de-serialization should be same but that’s not happening in Ignite 2.7 version.

So I debugged this on Ignite 2.6 version. On 2.6 version all 22 cacheMetrics are being de-serialized successfully.


This Looks like a bug to me in serializer/deseriliazer code.  

 

AffinityJob result being serialized on server:

 

 

 

AffinityJob result being de-serialized on client:

 


Thanks,
Prasad
On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Do you have a reproducer project to reliably confirm this issue?

Regards,
--
Ilya Kasnacheev


ср, 9 янв. 2019 г. в 12:39, Akash Shinde <[hidden email]>:
Added  [hidden email].

Should I log Jira for this issue?

Thanks,
Akash



On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote:

> Hi,
>
> No both nodes, client and server are running on Ignite 2.7 version. I am
> starting both server and client from Intellij IDE.
>
> Version printed in Server node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Version in client node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Thanks,
> Akash
>
> On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote:
>
>> Hi!
>>
>> Any chance you might have one node running 2.6 or something like that ?
>>
>> It looks like it get a different object that does not match the one
>> expected in 2.7
>>
>> Mikael
>> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
>>
>> Before submitting the affinity task ignite first gets the affinity cached
>> function (AffinityInfo) by submitting the cluster wide task "AffinityJob".
>> But while in the process of retrieving the output of this AffinityJob,
>> ignite deserializes this output. I am getting exception while deserailizing
>> this output.
>> In TcpDiscoveryNode.readExternal() method while deserailizing the
>> CacheMetrics object from input stream on 14th iteration I am getting
>> following exception. Complete stack trace is given in this mail chain.
>>
>> Caused by: java.io.IOException: Unexpected error occurred during
>> unmarshalling of an instance of the class:
>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>>
>> This is working fine on Ignite 2.6 version but giving problem on 2.7.
>>
>> Is this a bug or am I doing something wrong?
>>
>> Can someone please help?
>>
>> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> When execute affinity.partition(key), I am getting following exception
>>> on Ignite  2.7.
>>>
>>> Stacktrace:
>>>
>>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired, closing socket connection
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.apache.zookeeper.ZooKeeper - Initiating client connection,
>>> connectString=qzkpr01.p13.eng.in03.com:50024,
>>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka
>>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
>>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO
>>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
>>> 0x3673a59f5dd01c7
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Session establishment complete on server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid =
>>> 0x3673a59f5dd01ce, negotiated timeout = 10000
>>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
>>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>>
>>> Thanks,
>>> Akash
>>>
>>
prasadbhalerao1983 prasadbhalerao1983
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

In reply to this post by ilya.kasnacheev


Resending with screenshots.
 

I am not able reproduce this issue in small reproducer project but this is consistently happening in my project. So I debugged this issue and attached the screenshot in this mail.

 

NOTE:  This issue occurs if the statistics are enabled on cache configuration level [cacheCfg.setStatisticsEnabled(true)].

 

As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. All these cache metrics get serialized successfully. But at the time of deserialization on client node only first metrics get de-serialized successfully but all other metrics till iteration count 13 are de-serialized as null value and on iteration 14 “ref” byte value in “OptimizedObjectInputStream.readObject0()” method is read as 81 and code throws exception.

 

I think this where it is going wrong. The object copy at the time of serialization and de-serialization should be same but that’s not happening in Ignite 2.7 version. So I debugged this on Ignite 2.6 version. On 2.6 all 22 cacheMetrics are being de-serialized successfully.

 

AffinityJob result being serialized on server:

 

server.png

 

 

AffinityJob result being de-serialized on client:

 

client.jpg

Thanks,
Prasad


On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Do you have a reproducer project to reliably confirm this issue?

Regards,
--
Ilya Kasnacheev


ср, 9 янв. 2019 г. в 12:39, Akash Shinde <[hidden email]>:
Added  [hidden email].

Should I log Jira for this issue?

Thanks,
Akash



On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote:

> Hi,
>
> No both nodes, client and server are running on Ignite 2.7 version. I am
> starting both server and client from Intellij IDE.
>
> Version printed in Server node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Version in client node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Thanks,
> Akash
>
> On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote:
>
>> Hi!
>>
>> Any chance you might have one node running 2.6 or something like that ?
>>
>> It looks like it get a different object that does not match the one
>> expected in 2.7
>>
>> Mikael
>> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
>>
>> Before submitting the affinity task ignite first gets the affinity cached
>> function (AffinityInfo) by submitting the cluster wide task "AffinityJob".
>> But while in the process of retrieving the output of this AffinityJob,
>> ignite deserializes this output. I am getting exception while deserailizing
>> this output.
>> In TcpDiscoveryNode.readExternal() method while deserailizing the
>> CacheMetrics object from input stream on 14th iteration I am getting
>> following exception. Complete stack trace is given in this mail chain.
>>
>> Caused by: java.io.IOException: Unexpected error occurred during
>> unmarshalling of an instance of the class:
>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>>
>> This is working fine on Ignite 2.6 version but giving problem on 2.7.
>>
>> Is this a bug or am I doing something wrong?
>>
>> Can someone please help?
>>
>> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> When execute affinity.partition(key), I am getting following exception
>>> on Ignite  2.7.
>>>
>>> Stacktrace:
>>>
>>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired, closing socket connection
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.apache.zookeeper.ZooKeeper - Initiating client connection,
>>> connectString=qzkpr01.p13.eng.in03.com:50024,
>>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka
>>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
>>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO
>>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
>>> 0x3673a59f5dd01c7
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Session establishment complete on server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid =
>>> 0x3673a59f5dd01ce, negotiated timeout = 10000
>>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
>>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>>
>>> Thanks,
>>> Akash
>>>
>>
prasadbhalerao1983 prasadbhalerao1983
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

In reply to this post by ilya.kasnacheev
Resending....


I am not able reproduce this issue in small reproducer project but this is consistently happening in my project. So I debugged this issue and attached the screenshot in this mail.

 

NOTE:  This issue occurs if the statistics are enabled on cache configuration level [cacheCfg.setStatisticsEnabled(true)].

 

As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. All these cache metrics get serialized successfully. But at the time of deserialization on client node only first metrics get de-serialized successfully but all other metrics till iteration count 13 are de-serialized as null value and on iteration 14 “ref” byte value in “OptimizedObjectInputStream.readObject0()” method is read as 81 and code throws exception.

 

I think this where it is going wrong. The object copy at the time of serialization and de-serialization should be same but that’s not happening in Ignite 2.7 version. So I debugged this on Ignite 2.6 version. On 2.6 all 22 cacheMetrics are being de-serialized successfully.

 

AffinityJob result being serialized on server:


server.png
server.png


AffinityJob result being de-serialized on client:


client.jpg


Thanks,
Prasad

On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Do you have a reproducer project to reliably confirm this issue?

Regards,
--
Ilya Kasnacheev


ср, 9 янв. 2019 г. в 12:39, Akash Shinde <[hidden email]>:
Added  [hidden email].

Should I log Jira for this issue?

Thanks,
Akash



On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote:

> Hi,
>
> No both nodes, client and server are running on Ignite 2.7 version. I am
> starting both server and client from Intellij IDE.
>
> Version printed in Server node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Version in client node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Thanks,
> Akash
>
> On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote:
>
>> Hi!
>>
>> Any chance you might have one node running 2.6 or something like that ?
>>
>> It looks like it get a different object that does not match the one
>> expected in 2.7
>>
>> Mikael
>> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
>>
>> Before submitting the affinity task ignite first gets the affinity cached
>> function (AffinityInfo) by submitting the cluster wide task "AffinityJob".
>> But while in the process of retrieving the output of this AffinityJob,
>> ignite deserializes this output. I am getting exception while deserailizing
>> this output.
>> In TcpDiscoveryNode.readExternal() method while deserailizing the
>> CacheMetrics object from input stream on 14th iteration I am getting
>> following exception. Complete stack trace is given in this mail chain.
>>
>> Caused by: java.io.IOException: Unexpected error occurred during
>> unmarshalling of an instance of the class:
>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>>
>> This is working fine on Ignite 2.6 version but giving problem on 2.7.
>>
>> Is this a bug or am I doing something wrong?
>>
>> Can someone please help?
>>
>> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> When execute affinity.partition(key), I am getting following exception
>>> on Ignite  2.7.
>>>
>>> Stacktrace:
>>>
>>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired, closing socket connection
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.apache.zookeeper.ZooKeeper - Initiating client connection,
>>> connectString=qzkpr01.p13.eng.in03.com:50024,
>>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka
>>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
>>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO
>>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
>>> 0x3673a59f5dd01c7
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Session establishment complete on server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid =
>>> 0x3673a59f5dd01ce, negotiated timeout = 10000
>>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
>>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>>
>>> Thanks,
>>> Akash
>>>
>>

server.png (213K) Download Attachment
prasadbhalerao1983 prasadbhalerao1983
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling


Resending....


I am not able reproduce this issue in small reproducer project but this is consistently happening in my project. So I debugged this issue and attached the screenshot in this mail.

 

NOTE:  This issue occurs if the statistics are enabled on cache configuration level [cacheCfg.setStatisticsEnabled(true)].

 

As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. All these cache metrics get serialized successfully. But at the time of deserialization on client node only first metrics get de-serialized successfully but all other metrics till iteration count 13 are de-serialized as null value and on iteration 14 “ref” byte value in “OptimizedObjectInputStream.readObject0()” method is read as 81 and code throws exception.

 

I think this where it is going wrong. The object copy at the time of serialization and de-serialization should be same but that’s not happening in Ignite 2.7 version. So I debugged this on Ignite 2.6 version. On 2.6 all 22 cacheMetrics are being de-serialized successfully.

 

AffinityJob result being serialized on server:


server.png
server.png


AffinityJob result being de-serialized on client:


client.jpg


Thanks,
Prasad

On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Do you have a reproducer project to reliably confirm this issue?

Regards,
--
Ilya Kasnacheev


ср, 9 янв. 2019 г. в 12:39, Akash Shinde <[hidden email]>:
Added  [hidden email].

Should I log Jira for this issue?

Thanks,
Akash



On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote:

> Hi,
>
> No both nodes, client and server are running on Ignite 2.7 version. I am
> starting both server and client from Intellij IDE.
>
> Version printed in Server node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Version in client node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Thanks,
> Akash
>
> On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote:
>
>> Hi!
>>
>> Any chance you might have one node running 2.6 or something like that ?
>>
>> It looks like it get a different object that does not match the one
>> expected in 2.7
>>
>> Mikael
>> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
>>
>> Before submitting the affinity task ignite first gets the affinity cached
>> function (AffinityInfo) by submitting the cluster wide task "AffinityJob".
>> But while in the process of retrieving the output of this AffinityJob,
>> ignite deserializes this output. I am getting exception while deserailizing
>> this output.
>> In TcpDiscoveryNode.readExternal() method while deserailizing the
>> CacheMetrics object from input stream on 14th iteration I am getting
>> following exception. Complete stack trace is given in this mail chain.
>>
>> Caused by: java.io.IOException: Unexpected error occurred during
>> unmarshalling of an instance of the class:
>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>>
>> This is working fine on Ignite 2.6 version but giving problem on 2.7.
>>
>> Is this a bug or am I doing something wrong?
>>
>> Can someone please help?
>>
>> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> When execute affinity.partition(key), I am getting following exception
>>> on Ignite  2.7.
>>>
>>> Stacktrace:
>>>
>>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired, closing socket connection
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.apache.zookeeper.ZooKeeper - Initiating client connection,
>>> connectString=qzkpr01.p13.eng.in03.com:50024,
>>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka
>>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
>>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO
>>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
>>> 0x3673a59f5dd01c7
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Session establishment complete on server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid =
>>> 0x3673a59f5dd01ce, negotiated timeout = 10000
>>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
>>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>>
>>> Thanks,
>>> Akash
>>>
>>

client.jpg (139K) Download Attachment
server.png (213K) Download Attachment
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Hello!

I think the problem was introduced by
https://issues.apache.org/jira/browse/IGNITE-6846 which does look very
suspicious, however it is strange that it does not reproduce right away.

I could try and devise a fix but I could not reproduce this behavior in any
of tests. If you could do a reproducer project that would be awesome.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
prasadbhalerao1983 prasadbhalerao1983
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

I tried to reproduce this in standalone program. But the thing is cache metrics map in TcpDiscoveryNode is empty even after setting statisticEnabled to true on all caches.
So the flow does not enter into serializr/deserialize cacheMetrics block.

Any idea how the cacheMetrics gets populated. On which event?


Thanks,
Prasad

On Fri 11 Jan, 2019, 7:55 PM ilya.kasnacheev <[hidden email] wrote:
Hello!

I think the problem was introduced by
https://issues.apache.org/jira/browse/IGNITE-6846 which does look very
suspicious, however it is strange that it does not reproduce right away.

I could try and devise a fix but I could not reproduce this behavior in any
of tests. If you could do a reproducer project that would be awesome.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Hello!

Have you tried enabling Zookeeper in your reproducer? I have a hunch that they are linked: this behavior is affected by zookeeper discovery.

Regards,
--
Ilya Kasnacheev


пт, 11 янв. 2019 г. в 17:44, Prasad Bhalerao <[hidden email]>:
I tried to reproduce this in standalone program. But the thing is cache metrics map in TcpDiscoveryNode is empty even after setting statisticEnabled to true on all caches.
So the flow does not enter into serializr/deserialize cacheMetrics block.

Any idea how the cacheMetrics gets populated. On which event?


Thanks,
Prasad

On Fri 11 Jan, 2019, 7:55 PM ilya.kasnacheev <[hidden email] wrote:
Hello!

I think the problem was introduced by
https://issues.apache.org/jira/browse/IGNITE-6846 which does look very
suspicious, however it is strange that it does not reproduce right away.

I could try and devise a fix but I could not reproduce this behavior in any
of tests. If you could do a reproducer project that would be awesome.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
prasadbhalerao1983 prasadbhalerao1983
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

No I am not using zookeeper discovery.
Using TcpDiscoveryVmIpFinder.

Can someone please explain on what event cacheMetrics in TcpDiscoveryNode gets populated. It is not getting populated in standalone program.

If it gets populated then I might be able to reproduce this case.

On Fri 11 Jan, 2019, 8:28 PM Ilya Kasnacheev <[hidden email] wrote:
Hello!

Have you tried enabling Zookeeper in your reproducer? I have a hunch that they are linked: this behavior is affected by zookeeper discovery.

Regards,
--
Ilya Kasnacheev


пт, 11 янв. 2019 г. в 17:44, Prasad Bhalerao <[hidden email]>:
I tried to reproduce this in standalone program. But the thing is cache metrics map in TcpDiscoveryNode is empty even after setting statisticEnabled to true on all caches.
So the flow does not enter into serializr/deserialize cacheMetrics block.

Any idea how the cacheMetrics gets populated. On which event?


Thanks,
Prasad

On Fri 11 Jan, 2019, 7:55 PM ilya.kasnacheev <[hidden email] wrote:
Hello!

I think the problem was introduced by
https://issues.apache.org/jira/browse/IGNITE-6846 which does look very
suspicious, however it is strange that it does not reproduce right away.

I could try and devise a fix but I could not reproduce this behavior in any
of tests. If you could do a reproducer project that would be awesome.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
prasadbhalerao1983 prasadbhalerao1983
Reply | Threaded
Open this post in threaded view
|

Re: [IGNITE-10925] After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Hi,

I am able to create a reproducer for this issue. I have also created a JIRA IGNITE-10925  for this issue.

Step to Reproduce:

1) First Run com.example.demo.Server class as a java program

2) Then run com.example.demo.Client as java program.


Thanks,
Prasad

On Sat, Jan 12, 2019 at 11:17 AM Prasad Bhalerao <[hidden email]> wrote:
No I am not using zookeeper discovery.
Using TcpDiscoveryVmIpFinder.

Can someone please explain on what event cacheMetrics in TcpDiscoveryNode gets populated. It is not getting populated in standalone program.

If it gets populated then I might be able to reproduce this case.

On Fri 11 Jan, 2019, 8:28 PM Ilya Kasnacheev <[hidden email] wrote:
Hello!

Have you tried enabling Zookeeper in your reproducer? I have a hunch that they are linked: this behavior is affected by zookeeper discovery.

Regards,
--
Ilya Kasnacheev


пт, 11 янв. 2019 г. в 17:44, Prasad Bhalerao <[hidden email]>:
I tried to reproduce this in standalone program. But the thing is cache metrics map in TcpDiscoveryNode is empty even after setting statisticEnabled to true on all caches.
So the flow does not enter into serializr/deserialize cacheMetrics block.

Any idea how the cacheMetrics gets populated. On which event?


Thanks,
Prasad

On Fri 11 Jan, 2019, 7:55 PM ilya.kasnacheev <[hidden email] wrote:
Hello!

I think the problem was introduced by
https://issues.apache.org/jira/browse/IGNITE-6846 which does look very
suspicious, however it is strange that it does not reproduce right away.

I could try and devise a fix but I could not reproduce this behavior in any
of tests. If you could do a reproducer project that would be awesome.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/