AWS Java SDK marshalling errors, best practices?

classic Classic list List threaded Threaded
3 messages Options
hueb1 hueb1
Reply | Threaded
Open this post in threaded view
|

AWS Java SDK marshalling errors, best practices?

I have two nodes on different EC2 machines. I'm trying to run a compute task using ExecutorService framework. The compute task uses the AWS Java SDK to perform AmazonS3 API calls.  I have Peer class loading enabled.  I originally was getting Serialization exceptions saying that AmazonS3 is not serializable.  

First question: What is the best practice to get around this Serialization exception when using third party packages that are not trivial to make serializable?

To get around this for now, I change the OptimizationMarshaller to not require serialization. Now I'm getting

[17:49:49,861][SEVERE][main][GridExecutorService] Failed to get task result: ComputeTaskInternalFuture [ses=GridTaskSessionImpl [taskName=ignite_poc.aws.S3Util$BlockDownloader, dep=GridDeployment [ts=1439315380598, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@30a4effe, clsLdrId=8adb1ed1f41-df0e0a53-f6ff-4439-a6d8-a6014e38e031, userVer=0, loc=true, sampleClsName=o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, pendingUndeploy=false, undeployed=false, usage=0], taskClsName=ignite_poc.aws.S3Util$BlockDownloader, sesId=13eb1ed1f41-df0e0a53-f6ff-4439-a6d8-a6014e38e031, startTime=1439315383145, endTime=9223372036854775807, taskNodeId=df0e0a53-f6ff-4439-a6d8-a6014e38e031, clsLdr=sun.misc.Launcher$AppClassLoader@30a4effe, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, subjId=df0e0a53-f6ff-4439-a6d8-a6014e38e031, mapFut=IgniteFuture [orig=GridFutureAdapter [resFlag=2, res=null, startTime=1439315383145, endTime=1439315383145, ignoreInterrupts=false, lsnr=null, state=DONE]]], super=GridFutureAdapter [resFlag=1, res=class o.a.i.IgniteException: Remote job threw user exception (override or implement ComputeTask.result(..) method if you would like to have automatic failover for this exception)., startTime=1439315383145, endTime=1439315383276, ignoreInterrupts=false, lsnr=null, state=DONE]]
class org.apache.ignite.IgniteCheckedException: Failed to find class with given class loader for unmarshalling (make sure same version of all classes are available on all nodes or enable peer-class-loading): sun.misc.Launcher$AppClassLoader@305f387c
        at org.apache.ignite.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:242)
        at org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:382)
        at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1046)
        at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1728)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:690)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$1500(GridIoManager.java:58)
        at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:653)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.amazonaws.http.conn.$Proxy2
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)

I copied all dependent libraries (AWS included) into the libs folder on each of the two nodes prior to starting them up, so each node should have all required libs.  This works fine when executing in a single node, but the classpath seems to get messed up when I move to multiple nodes.  
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: AWS Java SDK marshalling errors, best practices?

Hi!

I'm guessing that you implemented the job you're submitting into executor service as an anonymous class. Any anonymous class has a link to the parent class which is serialized as well (along with all its fields) - this can cause such unexpected behavior.

If this is the case, I would recommend to convert the anonymous class to a private static class. This way you will gain full control on what is actually serialized and will be able to make sure that nothing unnecessary is sent across network.

Let us know if it helps you.

-Val
hueb1 hueb1
Reply | Threaded
Open this post in threaded view
|

Re: AWS Java SDK marshalling errors, best practices?

SOLVED:  The error was mine.  I was passing an instantiated AmazonS3Client object as a parameter to the IgniteCallable implementing class.  I'm guessing that instance maintained some open streams, thereby making serialization impossible much like trying to serialize an open file handle across the network.