Apache Spark and Apache Ignite

classic Classic list List threaded Threaded
3 messages Options
Wellington Alves das Neves Wellington Alves das Neves
Reply | Threaded
Open this post in threaded view
|

Apache Spark and Apache Ignite

Hi,

I am currently researching an architecture on AWS - Elastic MapReduce (EMR) to run Genetic Algorithms (GA).
Apache Ignite already has some genetic algorithms (GA) defined, so we would like to do some testing with it integrated with Apache Spark.

But I found little material on how to implement this Apache Ignite GA architecture along with Apache Spark. 

Thks! 

--
Wellington Alves das Neves  
Software Engineer

+55 (17) 99194-7119
[hidden email]
linkedinskypegitlab

zaleslaw zaleslaw
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark and Apache Ignite

Hi, Wellington

I'll be happy to help you if you give me more information of your goal.

I have a few questions, could you answer please?

1. What's your main goal? To solve optimization task on the data in Ignite
or in Spark?
2. What's the average size of initial population?
3. Did you run these  examples
<https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml/genetic>  
to solve, for example knapsack problem?
4. Did you have a look here, docs
https://apacheignite.readme.io/docs/genetic-algorithms

Yes, Apache Ignite and Apache Spark has integration  bridge
<https://apacheignite-fs.readme.io/docs>   which give us ability to use
Ignite instead of .cache() or persist() to keep dataframes in-memory for
intermediate calculations.

But we have no support for GA framework or another ML parts as part of
extended Spark API.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Wellington Alves das Neves Wellington Alves das Neves
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark and Apache Ignite

Hi, 

1. What's your main goal?  
       We need to run Genetic Algorithms in parallel to return the best execution plan for a factory production line.

To solve optimization task on the data in Ignite or in Spark?  
       As I have little experience, I'm evaluating the best architecture we can be applying to this scenario,
       and remembering that we will perform using AWS Elastic MapReduce (EMR) service.
       
        Regarding architecture, could you suggest this scenario?


2. What's the average size of initial population?  
     We are evaluating the size.


Thks!



Em ter., 5 de nov. de 2019 às 08:24, zaleslaw <[hidden email]> escreveu:
Hi, Wellington

I'll be happy to help you if you give me more information of your goal.

I have a few questions, could you answer please?

1. What's your main goal? To solve optimization task on the data in Ignite
or in Spark?
2. What's the average size of initial population?
3. Did you run these  examples
<https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml/genetic
to solve, for example knapsack problem?
4. Did you have a look here, docs
https://apacheignite.readme.io/docs/genetic-algorithms

Yes, Apache Ignite and Apache Spark has integration  bridge
<https://apacheignite-fs.readme.io/docs>   which give us ability to use
Ignite instead of .cache() or persist() to keep dataframes in-memory for
intermediate calculations.

But we have no support for GA framework or another ML parts as part of
extended Spark API.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--
Wellington Alves das Neves  
Software Engineer

+55 (17) 99194-7119
[hidden email]
linkedinskypegitlab