Ignite and spark for deep learning

classic Classic list List threaded Threaded
5 messages Options
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Ignite and spark for deep learning

Hi. two platforms (spark and ignite) use in memory for computing. instead of
loading data into ignite catch we also can loading data to spark memory and
catch it on spark node. if we could do  this (catching on spark node) why we
load data to ignite catch?. loading to ignite catch have benefit only for
sharing rdd between spark jobs ad indexing query index.i want to integrate
spark and ignite for deep learning platform. I want to use DL4J (deep
learning 4 java) as platform for deep learning. I want to use dl4j in spark
node and integrate spark node with ignite. Is there any speed up in this
idea,?if i want to use ignite i can use ignite only for cache data for
spark?or i can use spark and ignite as an engeen process simultaneously?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
zaleslaw zaleslaw
Reply | Threaded
Open this post in threaded view
|

Re: Ignite and spark for deep learning

Dear Mehdi Sey

Yes, both platforms are used for in-memory computing, but they have
different APIs and history of feature creation and different ways of
integration with famous DL frameworks (like DL4j and TensorFlow).

From my point of view, you have no speed up in Ignite + Spark + DL4j
integration.

Caching data in Ignite as a backend for RDD and dataframes first of all is
acceleration of business logic based on SQL queries. Not the same for ML
frameworks.

We have no proof, that usage Ignite as a backend could speed up DL4j or
MLlib algorithms.

Moreover, to avoid this, we wrote own ML library which is more better than
MLlib and runs natively on Ignite.

In my opinon, you should choose Ignite + Ignite ML + TF integration or Spark
+ DL4j to solve your Data Science task (where you need neural networks).





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: Ignite and spark for deep learning

I completely agree with you. I discuss many time with our team that this integration have not any gain about speed up and i have your idea about caching in ignite because in deep learning we have nothing to share in job because every job independently works on it's portion of data. In your opinion one idea can be ignite+ignite ml+ dl4j +spark? What benefit we can achive in this integration. ?ignite ml cant used for deep independently?

On Wednesday, January 9, 2019, zaleslaw <[hidden email]> wrote:
Dear Mehdi Sey

Yes, both platforms are used for in-memory computing, but they have
different APIs and history of feature creation and different ways of
integration with famous DL frameworks (like DL4j and TensorFlow).

From my point of view, you have no speed up in Ignite + Spark + DL4j
integration.

Caching data in Ignite as a backend for RDD and dataframes first of all is
acceleration of business logic based on SQL queries. Not the same for ML
frameworks.

We have no proof, that usage Ignite as a backend could speed up DL4j or
MLlib algorithms.

Moreover, to avoid this, we wrote own ML library which is more better than
MLlib and runs natively on Ignite.

In my opinon, you should choose Ignite + Ignite ML + TF integration or Spark
+ DL4j to solve your Data Science task (where you need neural networks).





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
zaleslaw zaleslaw
Reply | Threaded
Open this post in threaded view
|

Re: Ignite and spark for deep learning

Currently we have no integration with DL4j in ignite ML and have no such
plans in roadmap.
The TF is the best platform for the Deep Learning now and we spend time on
integration with TF first of all.

Maybe, in future, direct integration between DL4j and ignite will be added.
But it costst a hundred hours of development, of course.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: Ignite and spark for deep learning

I have in doubt. In last mail you said that dl4j over ignite have no speed up and you say now maybe develope in future.is it worthwhile for doing it?

On Friday, January 11, 2019, zaleslaw <[hidden email]> wrote:
Currently we have no integration with DL4j in ignite ML and have no such
plans in roadmap.
The TF is the best platform for the Deep Learning now and we spend time on
integration with TF first of all.

Maybe, in future, direct integration between DL4j and ignite will be added.
But it costst a hundred hours of development, of course.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/