diferrences between IgniteRdd and SparkRdd

classic Classic list List threaded Threaded
3 messages Options
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

diferrences between IgniteRdd and SparkRdd

hi.
first question
if we could create rdd with spark and store in ignite rdd or we only can
create rdd with ignite and share with spark job?

second question
what exactly the piece of bellow code?

object RDDProducer extends App {
val conf = new SparkConf().setAppName("SparkIgnite")
val sc = new SparkContext(conf)
val ic = new IgniteContext[Int, Int](sc, () => new IgniteConfiguration())
val sharedRDD: IgniteRDD[Int,Int] = ic.fromCache("partitioned")
sharedRDD.savePairs(sc.parallelize(1 to 100000, 10).map(i => (i, i)))
}




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: diferrences between IgniteRdd and SparkRdd

Hi,

The main difference between native Spark RDD and IgniteRDD is that Ignite
RDD provides a shared in-memory view on data across different Spark jobs,
workers, or applications, while native Spark RDD cannot be seen by other
Spark jobs or applications.

You can see the attached image that describes it.

spark-ignite-rdd.png
<http://apache-ignite-users.70518.x6.nabble.com/file/t1704/spark-ignite-rdd.png>  

According to your second question:

The provided code do next:

1)Create Spark context
2)Creates Ignite context with provided configuration.
3)Creates an Ignite Shared RDD of Type (Int,Int)
4)Fill the Ignite Shared RDD in with Int pairs.

To get the full example please take a look at ScalarSharedRDDExample.java
that located in source files:

https://github.com/apache/ignite/blob/master/examples/src/main/scala/org/apache/ignite/scalar/examples/spark/ScalarSharedRDDExample.scala

BR,
Andrei




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: diferrences between IgniteRdd and SparkRdd

I have another question. Could we create Rdd in spark separately frim ignite or we must create rdd from ignite and share with spark. In other words, is it possible create spark and ignite Rdd independently? 

On Wednesday, December 26, 2018, aealexsandrov <[hidden email]> wrote:
Hi,

The main difference between native Spark RDD and IgniteRDD is that Ignite
RDD provides a shared in-memory view on data across different Spark jobs,
workers, or applications, while native Spark RDD cannot be seen by other
Spark jobs or applications.

You can see the attached image that describes it.

spark-ignite-rdd.png
<http://apache-ignite-users.70518.x6.nabble.com/file/t1704/spark-ignite-rdd.png

According to your second question:

The provided code do next:

1)Create Spark context
2)Creates Ignite context with provided configuration.
3)Creates an Ignite Shared RDD of Type (Int,Int)
4)Fill the Ignite Shared RDD in with Int pairs.

To get the full example please take a look at ScalarSharedRDDExample.java
that located in source files:

https://github.com/apache/ignite/blob/master/examples/src/main/scala/org/apache/ignite/scalar/examples/spark/ScalarSharedRDDExample.scala

BR,
Andrei




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/