deep learning over apache ignite

classic Classic list List threaded Threaded
17 messages Options
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

deep learning over apache ignite

i have an idea as a proposal for my thesis. in a nutshell i want to develop a
framework on top of Apache ignite for processing deep learning application.
as an engine on top of Apache ignite i have considered DL4J ( deep learning
for java). i send this message for you about express your opinion on this
subject. In this regard, make any suggestions or recommendations you have.
I really appreciate.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmitrievanthony dmitrievanthony
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

Hello,

It's a good idea, machine learning in Apache Ignite is actively developing
right now and new features are always appreciated.

At the same time, let me point that deep learning is already supported in
Apache Ignite via integration with TensorFlow as the most popular deep
learning framework. This integrations allows to: use data stored in Apache
Ignite for training, use Apache Ignite as a model storage for training and
use Apache Ignite as a cluster manager for TensorFlow (all these features
together allow to perform efficient distributed training where it's
required).

For more information please see the following resources:
Apache Ignite [Deep Learning With TensorFlow]
<https://apacheignite.readme.io/docs/tensor-flow>  
TensorFlow [Apache Ignite Contrib Readme]
<https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/ignite>  

If you are interested in integration with DL4J anyway, feel free to share
your ideas and we'll try to help you to implement it.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
zaleslaw zaleslaw
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

In reply to this post by mehdi sey
Hi, I'm a contributor of Ignite ML too and I'd like dl4j library too and have
experience with that.
I have a few experiments with integration but the main problem is tensor
algebra implementation and effective Ignite cluster/caches usage.

Could you provide a simple example of your draft ideas on github and I'll be
happy to help with review

Feel free to post the link here in this thread



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

In a nutshell you want to say that this is a not good idea and may not reach a good performancs? You have integrated dl4j with apache ignite too?

On Tuesday, December 18, 2018, zaleslaw <[hidden email]> wrote:
Hi, I'm a contributor of Ignite ML too and I'd like dl4j library too and have
experience with that.
I have a few experiments with integration but the main problem is tensor
algebra implementation and effective Ignite cluster/caches usage.

Could you provide a simple example of your draft ideas on github and I'll be
happy to help with review

Feel free to post the link here in this thread



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

I am in first step and i have not succeeded to integrate dl4j with apache ignite. Is it possible for you how could you integrate them?

On Tuesday, December 18, 2018, Mehdi Seydali <[hidden email]> wrote:
In a nutshell you want to say that this is a not good idea and may not reach a good performancs? You have integrated dl4j with apache ignite too?

On Tuesday, December 18, 2018, zaleslaw <[hidden email]> wrote:
Hi, I'm a contributor of Ignite ML too and I'd like dl4j library too and have
experience with that.
I have a few experiments with integration but the main problem is tensor
algebra implementation and effective Ignite cluster/caches usage.

Could you provide a simple example of your draft ideas on github and I'll be
happy to help with review

Feel free to post the link here in this thread



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
zaleslaw zaleslaw
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

No, currently we doesn't work on integration bw DL4j and Ignite

The first step can be getting data from IgniteCache and wrapping to tensor
and putting to local instance of DL4j. But currently I could help with code
review only not with the example building.

First of all, we should understand what is the goal of integration? Should
it be simple channel to get data from Ignite without saving to file or it
should be more closest integration



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

Our,s goal are two things:
1. Acceleration of running deep neural network on two framework simultaneously ( ignite and spark)
2. Study that developing deep model with dl4j over ignite has better performance and speed up relared to dl4j?



On Tuesday, December 18, 2018, zaleslaw <[hidden email]> wrote:
No, currently we doesn't work on integration bw DL4j and Ignite

The first step can be getting data from IgniteCache and wrapping to tensor
and putting to local instance of DL4j. But currently I could help with code
review only not with the example building.

First of all, we should understand what is the goal of integration? Should
it be simple channel to get data from Ignite without saving to file or it
should be more closest integration



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

In reply to this post by dmitrievanthony
Do you have any recommendation about this idea?is it possible develope this idea and compare with tensoflow over ignite ?

On Tuesday, December 18, 2018, dmitrievanthony <[hidden email]> wrote:
Hello,

It's a good idea, machine learning in Apache Ignite is actively developing
right now and new features are always appreciated.

At the same time, let me point that deep learning is already supported in
Apache Ignite via integration with TensorFlow as the most popular deep
learning framework. This integrations allows to: use data stored in Apache
Ignite for training, use Apache Ignite as a model storage for training and
use Apache Ignite as a cluster manager for TensorFlow (all these features
together allow to perform efficient distributed training where it's
required).

For more information please see the following resources:
Apache Ignite [Deep Learning With TensorFlow]
<https://apacheignite.readme.io/docs/tensor-flow
TensorFlow [Apache Ignite Contrib Readme]
<https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/ignite

If you are interested in integration with DL4J anyway, feel free to share
your ideas and we'll try to help you to implement it.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmitrievanthony dmitrievanthony
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

In reply to this post by mehdi sey
First of all, when you are talking about "acceleration of running deep neural
network" what do you mean, training or inference?
If you are talking about inference, it actually doesn't matter what cluster
management system we use, we need only to run parallel/distributed
inference.
If you are talking about training, how are you going to speed it up? Use
distributed training on multiple machines? Or GPU's? Or maybe you just want
to speed up data preprocessing?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

My dear friend. Our essential goals are just as below:
1. Distributed training for training model 
2. Utilize Gpu in addition to Cpu for accelerate execution. 
 In tensorflow do you implement two above mentioned? How much speed up do you gain relative to pure tensorflow? 

On Tuesday, December 18, 2018, dmitrievanthony <[hidden email]> wrote:
First of all, when you are talking about "acceleration of running deep neural
network" what do you mean, training or inference?
If you are talking about inference, it actually doesn't matter what cluster
management system we use, we need only to run parallel/distributed
inference.
If you are talking about training, how are you going to speed it up? Use
distributed training on multiple machines? Or GPU's? Or maybe you just want
to speed up data preprocessing?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmitrievanthony dmitrievanthony
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

Yes, in TensorFlow on Apache Ignite we support distributed learning as you
described it (please see details in  this documentation
<https://apacheignite.readme.io/docs/ignite-dataset>  ).

Speaking about performance, TensorFlow supports distributed learning itself
(please see details  here
<https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/distribute>
). But to start distributed learning in pure TensorFlow you need to setup
cluster manually, manually distribute training data between cluster nodes
and handle node failures.

In TensorFlow on Apache Ignite we do it for you automatically. Apache Ignite
plays cluster manager role, it starts and maintains TensorFlow cluster with
optimal configuration and handles node failures. At the same time, the
training is fully performed by TensorFlow anyway. So, the training
performance is absolutely equal to the case when you use pure TensorFlow
with proper manually configured and started TensorFlow cluster because we
don't participate in the training process when the cluster is running
properly.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

I have another question? As you know dl4j execute over spark. When we want to integrate dl4j with ignite is it meaningful? For acceleration execution we can do below idea
1. We can using ignite as cache storage and preparing data for processing on dl4j.
2. Job on spark can spawn hirarchy job for accelerate execution.
Do you have any comment?

On Wednesday, December 19, 2018, dmitrievanthony <[hidden email]> wrote:
Yes, in TensorFlow on Apache Ignite we support distributed learning as you
described it (please see details in  this documentation
<https://apacheignite.readme.io/docs/ignite-dataset>  ).

Speaking about performance, TensorFlow supports distributed learning itself
(please see details  here
<https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/distribute>
). But to start distributed learning in pure TensorFlow you need to setup
cluster manually, manually distribute training data between cluster nodes
and handle node failures.

In TensorFlow on Apache Ignite we do it for you automatically. Apache Ignite
plays cluster manager role, it starts and maintains TensorFlow cluster with
optimal configuration and handles node failures. At the same time, the
training is fully performed by TensorFlow anyway. So, the training
performance is absolutely equal to the case when you use pure TensorFlow
with proper manually configured and started TensorFlow cluster because we
don't participate in the training process when the cluster is running
properly.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

In reply to this post by dmitrievanthony
My dear friend. I have a glance on tensorflow on ignite. I have the same idea as you but only one thing is added. In addition distributing job to local data, i will use gpu in addition to cpu core. Are you agree with total idea?  Is there anything for debating?

On Wednesday, December 19, 2018, dmitrievanthony <[hidden email]> wrote:
Yes, in TensorFlow on Apache Ignite we support distributed learning as you
described it (please see details in  this documentation
<https://apacheignite.readme.io/docs/ignite-dataset>  ).

Speaking about performance, TensorFlow supports distributed learning itself
(please see details  here
<https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/distribute>
). But to start distributed learning in pure TensorFlow you need to setup
cluster manually, manually distribute training data between cluster nodes
and handle node failures.

In TensorFlow on Apache Ignite we do it for you automatically. Apache Ignite
plays cluster manager role, it starts and maintains TensorFlow cluster with
optimal configuration and handles node failures. At the same time, the
training is fully performed by TensorFlow anyway. So, the training
performance is absolutely equal to the case when you use pure TensorFlow
with proper manually configured and started TensorFlow cluster because we
don't participate in the training process when the cluster is running
properly.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
zaleslaw zaleslaw
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

In reply to this post by mehdi sey
I will try to answer these questions
1) We can using ignite as cache storage and preparing data for processing on
dl4j?
- yes you can, it's main case, to load data to ignite caches correctly
2) Job on spark can spawn hirarchy job for accelerate execution.
- Ignite can compute piece pf code in parallel and etc, but you need to
implement correct infrastrucure about code tasks spreading and so on over
ignite.
Look at our TF integration and distributing training



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

as you said we can use hierarchical spawning job in spark. there is remained another issue. for example if we have load data from ignite cache and 
a job on spark has been spawned. this job spawn some other jobs. this jobs can be execute on ignite too? if this job fail how about the other job spawned? spark must handle them or ignite? 

On Sat, Dec 22, 2018 at 6:28 PM zaleslaw <[hidden email]> wrote:
I will try to answer these questions
1) We can using ignite as cache storage and preparing data for processing on
dl4j?
- yes you can, it's main case, to load data to ignite caches correctly
2) Job on spark can spawn hirarchy job for accelerate execution.
- Ignite can compute piece pf code in parallel and etc, but you need to
implement correct infrastrucure about code tasks spreading and so on over
ignite.
Look at our TF integration and distributing training



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

In reply to this post by zaleslaw
my dear friend i have another question in my mine. two platforms (spark and ignite) use in memory for computing. instead of loading data into ignite catch we also can loading data to spark memory and catch it. if we could do  this why we load data to ignite catch?. loading to ignite catch have benefit only for sharing rdd between spark jobs ad indexing query index. as you said for only sharing ignite rdd you said that we must load data to ignite?

On Sat, Dec 22, 2018 at 6:28 PM zaleslaw <[hidden email]> wrote:
I will try to answer these questions
1) We can using ignite as cache storage and preparing data for processing on
dl4j?
- yes you can, it's main case, to load data to ignite caches correctly
2) Job on spark can spawn hirarchy job for accelerate execution.
- Ignite can compute piece pf code in parallel and etc, but you need to
implement correct infrastrucure about code tasks spreading and so on over
ignite.
Look at our TF integration and distributing training



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: deep learning over apache ignite

In reply to this post by dmitrievanthony
i have a question in my mind. we know that spark and ignite are in memory computing platform and both of them cache data. if we want to consider caching data in ignite as a benefit, spark also cache data. in this idea if we want to develope dl4j over ignite What will we gain except for ignite computation?

On Tue, Dec 18, 2018 at 7:38 PM dmitrievanthony <[hidden email]> wrote:
First of all, when you are talking about "acceleration of running deep neural
network" what do you mean, training or inference?
If you are talking about inference, it actually doesn't matter what cluster
management system we use, we need only to run parallel/distributed
inference.
If you are talking about training, how are you going to speed it up? Use
distributed training on multiple machines? Or GPU's? Or maybe you just want
to speed up data preprocessing?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/