Task flow implementation question

classic Classic list List threaded Threaded
3 messages Options
Pascoe11 Pascoe11
Reply | Threaded
Open this post in threaded view
|

Task flow implementation question

Hello everyone,

So I am trying to put together a task flow chain. A task can have any number of inputs and outputs, what I would call ports and each port has a value stored in the cache.

A task can have numerous predecessors.

For example say I have two nodes which can be executed in parallel: one generates a large 2d sparse array and saves this to the cache and the second generates a vector which is also saved to cache. A successor is linked to these two tasks, and has an input of type array and a second input of type vector, looking like follows:

GEN_MAT (e.g. 15 seconds)  - >
                                                          MAT_VEC_MUL - >
GEN_VEC(e.g. 1 second)       - >

As I have tried to show, the GEN_MAT takes a lot longer, MAT_MUL can only execute once all input ports are set.

My question is how to implement this functionality effectively.
The input and output ports use a KEY:VALUE scheme, so with cache events, all successor nodes can have their input ports listen for their KEY values to be set, and it does work. But it feels very clunky. I am playing around with setting attributes in ComputeTaskSession, but have not managed to get it working. In a way cache events seem like the best option.

Any recommendations or ideas would be really helpful, I am very new to apache ignite and just programming in general.

Thanks and kind regards,
Pascoe
Pavel Vinokurov Pavel Vinokurov
Reply | Threaded
Open this post in threaded view
|

Re: Task flow implementation question

Hi Pascoe,

Please pay attention to the following example:

It demonstrates continuous mapping that could work in your case. 


Thanks,
Pavel

чт, 30 мая 2019 г. в 08:34, Pascoe Scholle <[hidden email]>:
Hello everyone,

So I am trying to put together a task flow chain. A task can have any number of inputs and outputs, what I would call ports and each port has a value stored in the cache.

A task can have numerous predecessors.

For example say I have two nodes which can be executed in parallel: one generates a large 2d sparse array and saves this to the cache and the second generates a vector which is also saved to cache. A successor is linked to these two tasks, and has an input of type array and a second input of type vector, looking like follows:

GEN_MAT (e.g. 15 seconds)  - >
                                                          MAT_VEC_MUL - >
GEN_VEC(e.g. 1 second)       - >

As I have tried to show, the GEN_MAT takes a lot longer, MAT_MUL can only execute once all input ports are set.

My question is how to implement this functionality effectively.
The input and output ports use a KEY:VALUE scheme, so with cache events, all successor nodes can have their input ports listen for their KEY values to be set, and it does work. But it feels very clunky. I am playing around with setting attributes in ComputeTaskSession, but have not managed to get it working. In a way cache events seem like the best option.

Any recommendations or ideas would be really helpful, I am very new to apache ignite and just programming in general.

Thanks and kind regards,
Pascoe


--

Regards

Pavel Vinokurov

Pascoe11 Pascoe11
Reply | Threaded
Open this post in threaded view
|

Re: Task flow implementation question

Thank you! Ill give that a go

On Thu, May 30, 2019 at 11:16 AM Pavel Vinokurov <[hidden email]> wrote:
Hi Pascoe,

Please pay attention to the following example:

It demonstrates continuous mapping that could work in your case. 


Thanks,
Pavel

чт, 30 мая 2019 г. в 08:34, Pascoe Scholle <[hidden email]>:
Hello everyone,

So I am trying to put together a task flow chain. A task can have any number of inputs and outputs, what I would call ports and each port has a value stored in the cache.

A task can have numerous predecessors.

For example say I have two nodes which can be executed in parallel: one generates a large 2d sparse array and saves this to the cache and the second generates a vector which is also saved to cache. A successor is linked to these two tasks, and has an input of type array and a second input of type vector, looking like follows:

GEN_MAT (e.g. 15 seconds)  - >
                                                          MAT_VEC_MUL - >
GEN_VEC(e.g. 1 second)       - >

As I have tried to show, the GEN_MAT takes a lot longer, MAT_MUL can only execute once all input ports are set.

My question is how to implement this functionality effectively.
The input and output ports use a KEY:VALUE scheme, so with cache events, all successor nodes can have their input ports listen for their KEY values to be set, and it does work. But it feels very clunky. I am playing around with setting attributes in ComputeTaskSession, but have not managed to get it working. In a way cache events seem like the best option.

Any recommendations or ideas would be really helpful, I am very new to apache ignite and just programming in general.

Thanks and kind regards,
Pascoe


--

Regards

Pavel Vinokurov