I use spark-submit script to launch my spark application. There is a problem with IgniteContext when starting ignite server nodes on each worker. It seems IgniteContext is trying to launch more than one ignite node instance per worker. Does Ignite works outiside spark-shell?
// Start ignite server node on each worker in server mode. sparkContext.parallelize(1 to workers, workers).foreachPartition(it ⇒ ignite())
As far as I understand, this line works as far as each executor holds a single partition. That is, if executor owns more than one partition IgniteContext will try to launch several Ignite nodes (in the same process); if there are executors with no partitions, no Ignite instance will be launched.
For example, Image a two worker cluster (node1, node2), each worker having two cores. The idea is to launch an Ignite node on each executor. So we call: sparkContext.parallelize(1 to 2, 2).foreachPartition(it ⇒ ignite())
It might be the chance the two partitions are handled by node1. Or it might be the case node1 handle one partition, and node2 the other. (this is what we want). So the question is how to make sure the latter scenario happens?
On 26 May 2016 at 05:19, vkulichenko [via Apache Ignite Users] <[hidden email]> wrote:
Why do you think this is happening? Do you have any exception? Generally, IgniteContext always reuses the Ignite client if it already exists in the current process.
Note that you can still have several clients in several processes. Usually you will have a client on the driver and a client per each executor (not worker!).
If you reply to this email, your message will be added to the discussion below:
IgniteContext.ignite() method handles this situation and reuses the Ignite client if it's already started. So if there are two partitions on the same worker, there still will be only one Ignite started per process. Is it working in a different way for you?