Is it safe to modify the activeJobs and waitingJobs collections in the CollisionContext?

classic Classic list List threaded Threaded
7 messages Options
Paolo Di Tommaso Paolo Di Tommaso
Reply | Threaded
Open this post in threaded view
|

Is it safe to modify the activeJobs and waitingJobs collections in the CollisionContext?

Hello, 

I would create my own collision SPI chaining an already existing SPI implementation. 

Is it safe to modify the content of the `activeJobs` and `waitingJobs` collections of the CollistionContext and then delegate the execution to another SPI instance `onCollision` method? 


Cheers,
Paolo

yakov yakov
Reply | Threaded
Open this post in threaded view
|

Re: Is it safe to modify the activeJobs and waitingJobs collections in the CollisionContext?

Hi Paolo!

This will throw Unsupported operation exception. Moreover, what is the purpose of doing that. Underlying collections will be modified when you call methods on collision context object. Pls take a look at org.apache.ignite.internal.processors.job.GridJobProcessor#handleCollisions method to get an idea.

Thanks!

--Yakov

2016-01-28 18:39 GMT+03:00 Paolo Di Tommaso <[hidden email]>:
Hello, 

I would create my own collision SPI chaining an already existing SPI implementation. 

Is it safe to modify the content of the `activeJobs` and `waitingJobs` collections of the CollistionContext and then delegate the execution to another SPI instance `onCollision` method? 


Cheers,
Paolo


Paolo Di Tommaso Paolo Di Tommaso
Reply | Threaded
Open this post in threaded view
|

Re: Is it safe to modify the activeJobs and waitingJobs collections in the CollisionContext?

OK, but if so I can create my own copy of `activeJobs` and `waitingJobs` collections. 

My idea is to extend the existing JobStealingCollisionSpi strategy by delegation. For example: 

class CustomSpi extends IgniteSpiAdapter implements CollisionSpi {

  private CollisionSpi delegate = new JobStealingCollisionSpi()

  public void onCollision(CollisionContext ctx) { 
    // IMPLEMENT CUSTOM LOGIC TO ACTIVATE PENDING JOBS HERE 

    // then delegate to exiting strategy to handle job stealing 
    Collection<CollisionJobContext> myActiveJobs = // .. my copy 
    Collection<CollisionJobContext> myWaitJobs = // .. my copy 
    
    delegate.onCollision(new CollisionContext() {
                @Override public Collection<CollisionJobContext> activeJobs() {
                    return myActiveJobs;
                }

                @Override public Collection<CollisionJobContext> waitingJobs() {
                    return myWaitJobs;
                }

                @Override public Collection<CollisionJobContext> heldJobs() {
                    return myHeldJobs;
                }
            }
  
  } 

}


Does this make sense? 


Cheers,
Paolo

On Thu, Jan 28, 2016 at 4:48 PM, Yakov Zhdanov <[hidden email]> wrote:
Hi Paolo!

This will throw Unsupported operation exception. Moreover, what is the purpose of doing that. Underlying collections will be modified when you call methods on collision context object. Pls take a look at org.apache.ignite.internal.processors.job.GridJobProcessor#handleCollisions method to get an idea.

Thanks!

--Yakov

2016-01-28 18:39 GMT+03:00 Paolo Di Tommaso <[hidden email]>:
Hello, 

I would create my own collision SPI chaining an already existing SPI implementation. 

Is it safe to modify the content of the `activeJobs` and `waitingJobs` collections of the CollistionContext and then delegate the execution to another SPI instance `onCollision` method? 


Cheers,
Paolo



vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Is it safe to modify the activeJobs and waitingJobs collections in the CollisionContext?

Hi Paolo,

What's the use case? Why do you want to modify the collection before giving it to the SPI?

Thanks,
Valentin
Paolo Di Tommaso Paolo Di Tommaso
Reply | Threaded
Open this post in threaded view
|

Re: Is it safe to modify the activeJobs and waitingJobs collections in the CollisionContext?

Hi Valentin, 

I want to extend to job stealing collision in such a way that the maximum number of active jobs is defined based on the actual resources needed by the jobs (cpus, memory, storage). 

For example, if a node has 60GB of memory and there are 10 pending jobs each of which need 10 GB to run, I want to executed only 6 of them and keep the others in the wait list. 

Currently the `JobStealingCollisionSpi` does not allow that. Instead of reimplementing it from scratch I was thinking to implement this logic in a custom collision SPI and delegate the job stealing mechanism to an instance of `JobStealingCollisionSpi`.


Cheers,
Paolo

On Thu, Jan 28, 2016 at 11:20 PM, vkulichenko <[hidden email]> wrote:
Hi Paolo,

What's the use case? Why do you want to modify the collection before giving
it to the SPI?

Thanks,
Valentin



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Is-it-safe-to-modify-the-activeJobs-and-waitingJobs-collections-in-the-CollisionContext-tp2758p2767.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Is it safe to modify the activeJobs and waitingJobs collections in the CollisionContext?

Paolo,

This actually sounds like a use case for load balancing [1]. In particular, I think you can use AdaptiveLoadBalancingSpi for what you described, but you will have to provide your own implementation of AdaptiveLoadProbe, because we don't currently have the one based on memory utilization.

Will this work for you?

[1] https://apacheignite.readme.io/docs/load-balancing

-Val
Paolo Di Tommaso Paolo Di Tommaso
Reply | Threaded
Open this post in threaded view
|

Re: Is it safe to modify the activeJobs and waitingJobs collections in the CollisionContext?

Not sure that load balancing is what is needed in this case. 

The problem is to make sure that at any time tasks do not consume more resources than the available ones. Thus, a collision spi is the right strategy to handle that

I've ended up implementing a custom collision spi delegating the job stealing to the default implementation. It looks working fine. You can find the code here: 




Best,
Paolo


On Fri, Jan 29, 2016 at 11:26 PM, vkulichenko <[hidden email]> wrote:
Paolo,

This actually sounds like a use case for load balancing [1]. In particular,
I think you can use AdaptiveLoadBalancingSpi for what you described, but you
will have to provide your own implementation of AdaptiveLoadProbe, because
we don't currently have the one based on memory utilization.

Will this work for you?

[1] https://apacheignite.readme.io/docs/load-balancing

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Is-it-safe-to-modify-the-activeJobs-and-waitingJobs-collections-in-the-CollisionContext-tp2758p2779.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.