Help with Task Not Serializable Exception using IgniteRDD

classic Classic list List threaded Threaded
2 messages Options
javadba javadba
Reply | Threaded
Open this post in threaded view

Help with Task Not Serializable Exception using IgniteRDD

This post has NOT been accepted by the mailing list yet.
The following  simple IgniteRDD code does not work. The closure is very simple and does not contain any references to the enclosing class.    

override def setUp(cfg: BenchmarkConfiguration) {
  sc = new SparkContext("local[4]", "BenchmarkTest")
  sqlContext = new HiveContext(sc)
  ic = new IgniteContext[RddKey, RddVal](sc,
    () ⇒ configuration("client", client = true))
  icCache = ic.fromCache(PARTITIONED_CACHE_NAME)
     icCache.savePairs(  sc.parallelize({
      (0 until 1000).map{ n => (n.toLong, s"Value for key $n")}
        }, 10)) // Error happens here: this is "line 89"


So then - why is it generating a Task Not Serializable exception?

Here is the Stacktrace:

org.apache.spark.SparkException: Task not serializable
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166)
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158)
    at org.apache.spark.SparkContext.clean(SparkContext.scala:1623)
    at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:805)
    at org.apache.ignite.spark.IgniteRDD.savePairs(IgniteRDD.scala:170)
    at org.yardstickframework.spark.SparkAbstractBenchmark.setUp(SparkAbstractBenchmark.scala:89)
    at org.yardstickframework.spark.SparkCoreRDDBenchmark.setUp(SparkCoreRDDBenchmark.scala:18)
    at org.yardstickframework.spark.SparkCoreRDDBenchmark$.main(SparkCoreRDDBenchmark.scala:72)
    at org.yardstickframework.spark.SparkNode.start(SparkNode.scala:28)
    at org.yardstickframework.BenchmarkServerStartUp.main(
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(
    at java.lang.reflect.Method.invoke(
    at org.apache.spark.serializer.SerializationDebugger$ObjectStreamClassMethods$.getObjFieldValues$extension(SerializationDebugger.scala:240)

alexey.goncharuk alexey.goncharuk
Reply | Threaded
Open this post in threaded view

Re: Help with Task Not Serializable Exception using IgniteRDD

It looks like your code is compiled against a different version of scala than the ignite or spark modules were compiled. I got similar exceptions while testing when my code was compiled against scala 2.10 and spark was running scala 2.11 or vice-versa. Module com.databricks:spark-csv_2.10:1.1.0 might be the reason for this.