OptimizedMarshaller instead of BinaryMarshaller is used for ScanQuery

classic Classic list List threaded Threaded
6 messages Options
Calvin KL Wong, CLSA Calvin KL Wong, CLSA
Reply | Threaded
Open this post in threaded view
|

OptimizedMarshaller instead of BinaryMarshaller is used for ScanQuery

Hi,

 

I have a cache of <Integer, POJO>.

I found that when I use a ScanQuery on that cache, result will be deserialized using OptimizedMarshaller; Ignite sends ‘ordered’ message using GridCacheQueryResonseEntry.  Whereas if I use ContinousQuery, result will be deserialized using BinaryMarshaller.

 

My problem is that ScanQuery won’t be able to handle class versioning.

 

Question:

1.       I am using “keepBinary” and BinaryObject on the cache to avoid versioning related problem.  Is that the recommended approach?  Because it seems strange that “ScanQuery” and “ContinousQuery” have different behaviour.

2.       What is the recommended way to convert BinaryObject into POJO?  Ignite website talks about getting field of each object using ‘BinaryObject.field()’.   However, Ignite uses ‘BinaryReaderExImpl implements BinaryReader’ internally.  How can I get a reference to that object?

 

Thanks,

Calvin

 

Calvin KL Wong

Sr. Lead Engineer, Execution Services

D  +852 2600 7983  |  M  +852 9267 9471  |  T  +852 2600 8888

5/F, One Island East, 18 Westlands Road, Island East, Hong Kong

 

:1. Social Media Icons:CLSA_Social Media Icons_linkedin.png:1. Social Media Icons:CLSA_Social Media Icons_twitter.png:1. Social Media Icons:CLSA_Social Media Icons_youtube.png:1. Social Media Icons:CLSA_Social Media Icons_facebook.png

 

clsa.com

Insights. Liquidity. Capital.

 

CLSA_RGB

 

A CITIC Securities Company

 

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request.

Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

Calvin KL Wong, CLSA Calvin KL Wong, CLSA
Reply | Threaded
Open this post in threaded view
|

RE: OptimizedMarshaller instead of BinaryMarshaller is used for ScanQuery

Ok, I found that I can use binaryObject.deserialize() to get back the POJO.  So, the only remaining question is:

 

When a cache does not have ‘withKeepBinary’ set, why ContinuousQuery can handle class versioning fine whereas ScanQuery cannot?  Is that an expected behaviour?

 

Thanks,

Calvin

 

From: Calvin KL Wong, CLSA [mailto:[hidden email]]
Sent: Monday, July 09, 2018 11:50 AM
To: [hidden email]
Subject: OptimizedMarshaller instead of BinaryMarshaller is used for ScanQuery

 

Hi,

 

I have a cache of <Integer, POJO>.

I found that when I use a ScanQuery on that cache, result will be deserialized using OptimizedMarshaller; Ignite sends ‘ordered’ message using GridCacheQueryResonseEntry.  Whereas if I use ContinousQuery, result will be deserialized using BinaryMarshaller.

 

My problem is that ScanQuery won’t be able to handle class versioning.

 

Question:

1.       I am using “keepBinary” and BinaryObject on the cache to avoid versioning related problem.  Is that the recommended approach?  Because it seems strange that “ScanQuery” and “ContinousQuery” have different behaviour.

2.       What is the recommended way to convert BinaryObject into POJO?  Ignite website talks about getting field of each object using ‘BinaryObject.field()’.   However, Ignite uses ‘BinaryReaderExImpl implements BinaryReader’ internally.  How can I get a reference to that object?

 

Thanks,

Calvin

 

Calvin KL Wong

Sr. Lead Engineer, Execution Services

D  +852 2600 7983  |  M  +852 9267 9471  |  T  +852 2600 8888

5/F, One Island East, 18 Westlands Road, Island East, Hong Kong

 

:1. Social Media Icons:CLSA_Social Media Icons_linkedin.png:1. Social Media Icons:CLSA_Social Media Icons_twitter.png:1. Social Media Icons:CLSA_Social Media Icons_youtube.png:1. Social Media Icons:CLSA_Social Media Icons_facebook.png

 

clsa.com

Insights. Liquidity. Capital.

 

CLSA_RGB

 

A CITIC Securities Company

 

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at
https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request.

Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

RE: OptimizedMarshaller instead of BinaryMarshaller is used forScanQuery

Hi Calvin,

 

It should work the same for all queries. Ideally OptimizedMarshaller shouldn’t even be used (except for JDK classes).

 

How did you check which marshaller is used in each case?

 

Can you share the code of your POJO? Or perhaps a runnable reproducer?

 

Can you also share the logs/check them for warnings? There might be a warning saying that there is a problem in using BinaryMarshaller.

 

Thanks,

Stan

 

From: [hidden email]
Sent: 9 июля 2018 г. 13:40
To: [hidden email]
Subject: RE: OptimizedMarshaller instead of BinaryMarshaller is used forScanQuery

 

Ok, I found that I can use binaryObject.deserialize() to get back the POJO.  So, the only remaining question is:

 

When a cache does not have ‘withKeepBinary’ set, why ContinuousQuery can handle class versioning fine whereas ScanQuery cannot?  Is that an expected behaviour?

 

Thanks,

Calvin

 

From: Calvin KL Wong, CLSA [mailto:[hidden email]]
Sent: Monday, July 09, 2018 11:50 AM
To: [hidden email]
Subject: OptimizedMarshaller instead of BinaryMarshaller is used for ScanQuery

 

Hi,

 

I have a cache of <Integer, POJO>.

I found that when I use a ScanQuery on that cache, result will be deserialized using OptimizedMarshaller; Ignite sends ‘ordered’ message using GridCacheQueryResonseEntry.  Whereas if I use ContinousQuery, result will be deserialized using BinaryMarshaller.

 

My problem is that ScanQuery won’t be able to handle class versioning.

 

Question:

  1. I am using “keepBinary” and BinaryObject on the cache to avoid versioning related problem.  Is that the recommended approach?  Because it seems strange that “ScanQuery” and “ContinousQuery” have different behaviour.
  2. What is the recommended way to convert BinaryObject into POJO?  Ignite website talks about getting field of each object using ‘BinaryObject.field()’.   However, Ignite uses ‘BinaryReaderExImpl implements BinaryReader’ internally.  How can I get a reference to that object?

 

Thanks,

Calvin

 

Calvin KL Wong

Sr. Lead Engineer, Execution Services

D  +852 2600 7983  |  M  +852 9267 9471  |  T  +852 2600 8888

5/F, One Island East, 18 Westlands Road, Island East, Hong Kong

 

:1. Social Media Icons:CLSA_Social Media Icons_linkedin.png:1. Social Media Icons:CLSA_Social Media Icons_twitter.png:1. Social Media Icons:CLSA_Social Media Icons_youtube.png:1. Social Media Icons:CLSA_Social Media Icons_facebook.png

 

clsa.com

Insights. Liquidity. Capital.

 

CLSA_RGB

 

A CITIC Securities Company

 

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at
https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at
https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

 

Calvin KL Wong, CLSA Calvin KL Wong, CLSA
Reply | Threaded
Open this post in threaded view
|

RE: OptimizedMarshaller instead of BinaryMarshaller is used forScanQuery

Hi Stan,

 

Thanks for your response.

 

>> How did you check which marshaller is used in each case?

i)                    from stack trace

ii)                   step with a debugger

 

>> Can you share the code of your POJO?

Attached pojo.zip contains skeleton of the POJO.  The object that was passed between ignite nodes is called WorkingSetLedgerDelta.  It contains objects of other classes.

 

>> Can you also share the logs/check them for warnings? There might be a warning saying that there is a problem in using BinaryMarshaller.

I just checked them – did a grep of ‘-i marsh’, ‘-i warn’,  ‘-i except’, ‘-i error’; no marshaller related problem found.

 

Background:

A newer version of NewOrderRequest extends AbstractRequest is deployed on the service node, but not on the client node.  Client node sends a scan query to server node to get all WorkingSetLedgerDelta, which contains Request objects.

 

About the attached stack traces of the client node

1.       exception.when.using.pojo.with.scan.query.log

Cache: IgniteCache<LedgerKey, LedgerDelta>

Caught exception when trying to deserialize WorkingSetLedgerDelta

 

2.       forced.exception.when.using.binary.object.with.scan.query.with.keep.binary.log

Cache: IgniteCache<BinaryObject, BinaryObject>.withKeepBinary()

I forced an exception from eclipse when I saw that we successfully unmarshalled bytes into the BinaryObjectImpl of WorkingSetLedgerDelta.

 

The main difference between the two stack traces is at:

 

with POJO, the logic unmarshalls WorkingSetLedgerDelta using ‘readSerializable’, which cannot handle versioning.

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:601) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) ~[?:1.8.0_152]

        at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponseEntry.readExternal(GridCacheQueryResponseEntry.java:90) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

 

withKeepBinary, BinaryObjImpl object was received, the logic could unmarshall the BinaryObjImpl of WorkingSetLedgerDelta with no problem.

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:558) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:917) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) ~[?:1.8.0_152]

        at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponseEntry.readExternal(GridCacheQueryResponseEntry.java:90) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

 

 

Thanks,

Calvin

 

 

From: Stanislav Lukyanov [mailto:[hidden email]]
Sent: Monday, July 09, 2018 9:04 PM
To: [hidden email]
Subject: RE: OptimizedMarshaller instead of BinaryMarshaller is used forScanQuery

 

Hi Calvin,

 

It should work the same for all queries. Ideally OptimizedMarshaller shouldn’t even be used (except for JDK classes).

 

How did you check which marshaller is used in each case?

 

Can you share the code of your POJO? Or perhaps a runnable reproducer?

 

Can you also share the logs/check them for warnings? There might be a warning saying that there is a problem in using BinaryMarshaller.

 

Thanks,

Stan

 

From: [hidden email]
Sent: 9 июля 2018 г. 13:40
To: [hidden email]
Subject: RE: OptimizedMarshaller instead of BinaryMarshaller is used forScanQuery

 

Ok, I found that I can use binaryObject.deserialize() to get back the POJO.  So, the only remaining question is:

 

When a cache does not have ‘withKeepBinary’ set, why ContinuousQuery can handle class versioning fine whereas ScanQuery cannot?  Is that an expected behaviour?

 

Thanks,

Calvin

 

From: Calvin KL Wong, CLSA [mailto:[hidden email]]
Sent: Monday, July 09, 2018 11:50 AM
To: [hidden email]
Subject: OptimizedMarshaller instead of BinaryMarshaller is used for ScanQuery

 

Hi,

 

I have a cache of <Integer, POJO>.

I found that when I use a ScanQuery on that cache, result will be deserialized using OptimizedMarshaller; Ignite sends ‘ordered’ message using GridCacheQueryResonseEntry.  Whereas if I use ContinousQuery, result will be deserialized using BinaryMarshaller.

 

My problem is that ScanQuery won’t be able to handle class versioning.

 

Question:

  1. I am using “keepBinary” and BinaryObject on the cache to avoid versioning related problem.  Is that the recommended approach?  Because it seems strange that “ScanQuery” and “ContinousQuery” have different behaviour.
  2. What is the recommended way to convert BinaryObject into POJO?  Ignite website talks about getting field of each object using ‘BinaryObject.field()’.   However, Ignite uses ‘BinaryReaderExImpl implements BinaryReader’ internally.  How can I get a reference to that object?

 

Thanks,

Calvin

 

Calvin KL Wong

Sr. Lead Engineer, Execution Services

D  +852 2600 7983  |  M  +852 9267 9471  |  T  +852 2600 8888

5/F, One Island East, 18 Westlands Road, Island East, Hong Kong

 

:1. Social Media Icons:CLSA_Social Media Icons_linkedin.png:1. Social Media Icons:CLSA_Social Media Icons_twitter.png:1. Social Media Icons:CLSA_Social Media Icons_youtube.png:1. Social Media Icons:CLSA_Social Media Icons_facebook.png

 

clsa.com

Insights. Liquidity. Capital.

 

CLSA_RGB

 

A CITIC Securities Company

 

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at
https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at
https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

 

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request.

Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.


forced.exception.when.using.binary.object.with.scan.query.with.keep.binary.log (15K) Download Attachment
exception.when.using.pojo.with.scan.query.log (20K) Download Attachment
pojo.zip (1K) Download Attachment
Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

RE: OptimizedMarshaller instead of BinaryMarshaller is usedforScanQuery

Hi Calvin,

 

It seems that the code is incomplete.

First, what are the Instant, DeltaMap types? What are the types stored in the Map fields?

Second, do you have some code processing tool generating some code for you? E.g. I see that version() implementation is missing in WorkingSetLedgerDelta – I assume it’s generated.

 

The reason why OptimizedMarshaller is used is most likely that some parts of your objects are Serializable with custom serialization (readObject()/writeObject() methods).

Also, OptimizedMarshaller may be used when processing JDK classes.

The logs should have a message about that though.

 

Are you checking Ignite’s stdout or the log files?

Ignite starts in a quiet mode by default, so not all messages are in the stdout.

There is a path to the full log file printed in the stdout.

 

I’m still not sure whether I grasp your use case in full though.

Any chance you could create a small reproducer project and share it on GitHub?

 

Thanks,

Stan

 

From: [hidden email]
Sent: 10 июля 2018 г. 6:06
To: [hidden email]
Subject: RE: OptimizedMarshaller instead of BinaryMarshaller is usedforScanQuery

 

Hi Stan,

 

Thanks for your response.

 

>> How did you check which marshaller is used in each case?

i)                    from stack trace

ii)                   step with a debugger

 

>> Can you share the code of your POJO?

Attached pojo.zip contains skeleton of the POJO.  The object that was passed between ignite nodes is called WorkingSetLedgerDelta.  It contains objects of other classes.

 

>> Can you also share the logs/check them for warnings? There might be a warning saying that there is a problem in using BinaryMarshaller.

I just checked them – did a grep of ‘-i marsh’, ‘-i warn’,  ‘-i except’, ‘-i error’; no marshaller related problem found.

 

Background:

A newer version of NewOrderRequest extends AbstractRequest is deployed on the service node, but not on the client node.  Client node sends a scan query to server node to get all WorkingSetLedgerDelta, which contains Request objects.

 

About the attached stack traces of the client node

  1. exception.when.using.pojo.with.scan.query.log

Cache: IgniteCache<LedgerKey, LedgerDelta>

Caught exception when trying to deserialize WorkingSetLedgerDelta

 

  1. forced.exception.when.using.binary.object.with.scan.query.with.keep.binary.log

Cache: IgniteCache<BinaryObject, BinaryObject>.withKeepBinary()

I forced an exception from eclipse when I saw that we successfully unmarshalled bytes into the BinaryObjectImpl of WorkingSetLedgerDelta.

 

The main difference between the two stack traces is at:

 

with POJO, the logic unmarshalls WorkingSetLedgerDelta using ‘readSerializable’, which cannot handle versioning.

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:601) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) ~[?:1.8.0_152]

        at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponseEntry.readExternal(GridCacheQueryResponseEntry.java:90) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

 

withKeepBinary, BinaryObjImpl object was received, the logic could unmarshall the BinaryObjImpl of WorkingSetLedgerDelta with no problem.

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:558) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:917) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) ~[?:1.8.0_152]

        at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponseEntry.readExternal(GridCacheQueryResponseEntry.java:90) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

 

 

Thanks,

Calvin

 

 

From: Stanislav Lukyanov [mailto:[hidden email]]
Sent: Monday, July 09, 2018 9:04 PM
To: [hidden email]
Subject: RE: OptimizedMarshaller instead of BinaryMarshaller is used forScanQuery

 

Hi Calvin,

 

It should work the same for all queries. Ideally OptimizedMarshaller shouldn’t even be used (except for JDK classes).

 

How did you check which marshaller is used in each case?

 

Can you share the code of your POJO? Or perhaps a runnable reproducer?

 

Can you also share the logs/check them for warnings? There might be a warning saying that there is a problem in using BinaryMarshaller.

 

Thanks,

Stan

 

From: [hidden email]
Sent: 9 июля 2018 г. 13:40
To: [hidden email]
Subject: RE: OptimizedMarshaller instead of BinaryMarshaller is used forScanQuery

 

Ok, I found that I can use binaryObject.deserialize() to get back the POJO.  So, the only remaining question is:

 

When a cache does not have ‘withKeepBinary’ set, why ContinuousQuery can handle class versioning fine whereas ScanQuery cannot?  Is that an expected behaviour?

 

Thanks,

Calvin

 

From: Calvin KL Wong, CLSA [mailto:[hidden email]]
Sent: Monday, July 09, 2018 11:50 AM
To: [hidden email]
Subject: OptimizedMarshaller instead of BinaryMarshaller is used for ScanQuery

 

Hi,

 

I have a cache of <Integer, POJO>.

I found that when I use a ScanQuery on that cache, result will be deserialized using OptimizedMarshaller; Ignite sends ‘ordered’ message using GridCacheQueryResonseEntry.  Whereas if I use ContinousQuery, result will be deserialized using BinaryMarshaller.

 

My problem is that ScanQuery won’t be able to handle class versioning.

 

Question:

  1. I am using “keepBinary” and BinaryObject on the cache to avoid versioning related problem.  Is that the recommended approach?  Because it seems strange that “ScanQuery” and “ContinousQuery” have different behaviour.
  2. What is the recommended way to convert BinaryObject into POJO?  Ignite website talks about getting field of each object using ‘BinaryObject.field()’.   However, Ignite uses ‘BinaryReaderExImpl implements BinaryReader’ internally.  How can I get a reference to that object?

 

Thanks,

Calvin

 

Calvin KL Wong

Sr. Lead Engineer, Execution Services

D  +852 2600 7983  |  M  +852 9267 9471  |  T  +852 2600 8888

5/F, One Island East, 18 Westlands Road, Island East, Hong Kong

 

:1. Social Media Icons:CLSA_Social Media Icons_linkedin.png:1. Social Media Icons:CLSA_Social Media Icons_twitter.png:1. Social Media Icons:CLSA_Social Media Icons_youtube.png:1. Social Media Icons:CLSA_Social Media Icons_facebook.png

 

clsa.com

Insights. Liquidity. Capital.

 

CLSA_RGB

 

A CITIC Securities Company

 

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at
https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at
https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

 

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at
https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

 

Calvin KL Wong, CLSA Calvin KL Wong, CLSA
Reply | Threaded
Open this post in threaded view
|

RE: OptimizedMarshaller instead of BinaryMarshaller is usedforScanQuery

Hi Stan,

 

Attached is a reproducer I created based on org.apache.ignite.examples.datagrid.CacheContinuousQueryExample.  I am using a simple POJO to reproduce the behaviour I see.

 

Step:

1.       Add those files to ignite-examples project, under package org.apache.ignite.examples.datagrid

2.       Put a conditional breakpoint at OptimizedObjectInputStream.readObject0()

a.       On the return line under “Case SERIALIZABLE”

b.      The condition is:

                                                               i.      curCls.getName().contains("Request")

c.      

3.       Run CacheContinuousQueryServerExample

4.       Run CacheContinuousQueryClientPojoExample

a.       You should hit the break point.  This verifies that OptimizedMarshaller is used.

5.       Continue

6.       Press enter in CacheContinuousQueryServerExample to add more objects into cache.

a.       CacheContinuousQueryClientPojoExample will receive those updates.

b.      You will notice that you won’t get stop at the breakpoint.  This proves that OptimizedMarshaller isn’t being used in this case.

 

You may want to put some attention to the use of GridCacheQueryResponseEntry (which implements Externalizable) by Scan Query, but not by Continuous Query.

 

Thanks,

Calvin

 

From: Stanislav Lukyanov [mailto:[hidden email]]
Sent: Tuesday, July 10, 2018 6:23 PM
To: [hidden email]
Subject: RE: OptimizedMarshaller instead of BinaryMarshaller is usedforScanQuery

 

Hi Calvin,

 

It seems that the code is incomplete.

First, what are the Instant, DeltaMap types? What are the types stored in the Map fields?

Second, do you have some code processing tool generating some code for you? E.g. I see that version() implementation is missing in WorkingSetLedgerDelta – I assume it’s generated.

 

The reason why OptimizedMarshaller is used is most likely that some parts of your objects are Serializable with custom serialization (readObject()/writeObject() methods).

Also, OptimizedMarshaller may be used when processing JDK classes.

The logs should have a message about that though.

 

Are you checking Ignite’s stdout or the log files?

Ignite starts in a quiet mode by default, so not all messages are in the stdout.

There is a path to the full log file printed in the stdout.

 

I’m still not sure whether I grasp your use case in full though.

Any chance you could create a small reproducer project and share it on GitHub?

 

Thanks,

Stan

 

From: [hidden email]
Sent: 10 июля 2018 г. 6:06
To: [hidden email]
Subject: RE: OptimizedMarshaller instead of BinaryMarshaller is usedforScanQuery

 

Hi Stan,

 

Thanks for your response.

 

>> How did you check which marshaller is used in each case?

i)                    from stack trace

ii)                   step with a debugger

 

>> Can you share the code of your POJO?

Attached pojo.zip contains skeleton of the POJO.  The object that was passed between ignite nodes is called WorkingSetLedgerDelta.  It contains objects of other classes.

 

>> Can you also share the logs/check them for warnings? There might be a warning saying that there is a problem in using BinaryMarshaller.

I just checked them – did a grep of ‘-i marsh’, ‘-i warn’,  ‘-i except’, ‘-i error’; no marshaller related problem found.

 

Background:

A newer version of NewOrderRequest extends AbstractRequest is deployed on the service node, but not on the client node.  Client node sends a scan query to server node to get all WorkingSetLedgerDelta, which contains Request objects.

 

About the attached stack traces of the client node

  1. exception.when.using.pojo.with.scan.query.log

Cache: IgniteCache<LedgerKey, LedgerDelta>

Caught exception when trying to deserialize WorkingSetLedgerDelta

 

  1. forced.exception.when.using.binary.object.with.scan.query.with.keep.binary.log

Cache: IgniteCache<BinaryObject, BinaryObject>.withKeepBinary()

I forced an exception from eclipse when I saw that we successfully unmarshalled bytes into the BinaryObjectImpl of WorkingSetLedgerDelta.

 

The main difference between the two stack traces is at:

 

with POJO, the logic unmarshalls WorkingSetLedgerDelta using ‘readSerializable’, which cannot handle versioning.

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:601) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) ~[?:1.8.0_152]

        at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponseEntry.readExternal(GridCacheQueryResponseEntry.java:90) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

 

withKeepBinary, BinaryObjImpl object was received, the logic could unmarshall the BinaryObjImpl of WorkingSetLedgerDelta with no problem.

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:558) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:917) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) ~[?:1.8.0_152]

        at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponseEntry.readExternal(GridCacheQueryResponseEntry.java:90) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]

 

 

Thanks,

Calvin

 

 

From: Stanislav Lukyanov [mailto:[hidden email]]
Sent: Monday, July 09, 2018 9:04 PM
To: [hidden email]
Subject: RE: OptimizedMarshaller instead of BinaryMarshaller is used forScanQuery

 

Hi Calvin,

 

It should work the same for all queries. Ideally OptimizedMarshaller shouldn’t even be used (except for JDK classes).

 

How did you check which marshaller is used in each case?

 

Can you share the code of your POJO? Or perhaps a runnable reproducer?

 

Can you also share the logs/check them for warnings? There might be a warning saying that there is a problem in using BinaryMarshaller.

 

Thanks,

Stan

 

From: [hidden email]
Sent: 9 июля 2018 г. 13:40
To: [hidden email]
Subject: RE: OptimizedMarshaller instead of BinaryMarshaller is used forScanQuery

 

Ok, I found that I can use binaryObject.deserialize() to get back the POJO.  So, the only remaining question is:

 

When a cache does not have ‘withKeepBinary’ set, why ContinuousQuery can handle class versioning fine whereas ScanQuery cannot?  Is that an expected behaviour?

 

Thanks,

Calvin

 

From: Calvin KL Wong, CLSA [mailto:[hidden email]]
Sent: Monday, July 09, 2018 11:50 AM
To: [hidden email]
Subject: OptimizedMarshaller instead of BinaryMarshaller is used for ScanQuery

 

Hi,

 

I have a cache of <Integer, POJO>.

I found that when I use a ScanQuery on that cache, result will be deserialized using OptimizedMarshaller; Ignite sends ‘ordered’ message using GridCacheQueryResonseEntry.  Whereas if I use ContinousQuery, result will be deserialized using BinaryMarshaller.

 

My problem is that ScanQuery won’t be able to handle class versioning.

 

Question:

  1. I am using “keepBinary” and BinaryObject on the cache to avoid versioning related problem.  Is that the recommended approach?  Because it seems strange that “ScanQuery” and “ContinousQuery” have different behaviour.
  2. What is the recommended way to convert BinaryObject into POJO?  Ignite website talks about getting field of each object using ‘BinaryObject.field()’.   However, Ignite uses ‘BinaryReaderExImpl implements BinaryReader’ internally.  How can I get a reference to that object?

 

Thanks,

Calvin

 

Calvin KL Wong

Sr. Lead Engineer, Execution Services

D  +852 2600 7983  |  M  +852 9267 9471  |  T  +852 2600 8888

5/F, One Island East, 18 Westlands Road, Island East, Hong Kong

 

:1. Social Media Icons:CLSA_Social Media Icons_linkedin.png:1. Social Media Icons:CLSA_Social Media Icons_twitter.png:1. Social Media Icons:CLSA_Social Media Icons_youtube.png:1. Social Media Icons:CLSA_Social Media Icons_facebook.png

 

clsa.com

Insights. Liquidity. Capital.

 

CLSA_RGB

 

A CITIC Securities Company

 

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at
https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at
https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

 

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at
https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.

 

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request.

Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.


reproducer.zip (6K) Download Attachment