Unable to enable ML inference storage

classic Classic list List threaded Threaded
4 messages Options
Courtney Robinson Courtney Robinson
Reply | Threaded
Open this post in threaded view
|

Unable to enable ML inference storage

We're adding support for storing and re-using ML models but I haven't found a way to enable this currently.

I've tried programmatically and via XML as in

<property name="pluginConfigurations">
<bean class="org.apache.ignite.ml.util.plugin.MLPluginConfiguration">
<property name="withMdlStorage" value="#{true}" />
<property name="withMdlDescStorage" value="#{true}" />
<property name="mdlStorageBackups" value="2" />
<property name="mdlDescStorageBackups" value="2" />
</bean>
</property>

or
GridSpringResourceContext springGridCtx = new GridSpringResourceContextImpl(ctx);

MLPluginProvider mlPluginProvider = new MLPluginProvider();
cfg.setPluginProviders(mlPluginProvider);
MLPluginConfiguration pluginConfiguration = new MLPluginConfiguration();
pluginConfiguration.setMdlDescStorageBackups(2);
pluginConfiguration.setMdlStorageBackups(2);
pluginConfiguration.setWithMdlDescStorage(true);
pluginConfiguration.setWithMdlStorage(true);
cfg.setPluginConfigurations(pluginConfiguration);

//ignite = Ignition.start(cfg);
//must use this ignition ctx to make local IgniteRepo and other spring resources injectable into affinity run and other compute classes
ignite = IgnitionEx.start(cfg, springGridCtx); ignite.cluster().active(true);

both fail because the MLPluginProvider appears to be loaded and an attempt to initialise it is made before we can call activate on the cluster.

Is there a way to start a node as "active" so that this can be loaded?
Startup fails with the exceptions in the attached log.

Regards,
Courtney Robinson

cluster-inactive-error.txt (174K) Download Attachment
akorensh akorensh
Reply | Threaded
Open this post in threaded view
|

Re: Unable to enable ML inference storage

Hi,

  Looks like you are loading the ML plugin twice (see below)
  Can you try w/out the plugin config first.
  Just copy the ignite-ml module to the libs folder or
  follow these instructions:
https://apacheignite.readme.io/docs/machine-learning#getting-started
   
  If that works, put in your xml config, and try again.

  see:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/ml/inference/ModelStorageExample.java
(model storage)
  and the corresponding:
https://github.com/apache/ignite/blob/master/examples/config/example-ignite-ml.xml 
(this shows how to config ML plugin)


  If it still doesn't work send the config and, if possible, a reproducer
project.
Thanks, Alex



Multiple loadings of ML Plugin

 2020-05-21 10:04:25.382  INFO 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor   : Configured plugins:
[10:04:25]   ^-- ml-inference-plugin 1.0.0
2020-05-21 10:04:25.383  INFO 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor   :   ^-- ml-inference-plugin 1.0.0
2020-05-21 10:04:25.382  INFO [hypi,,,] 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor  [?] : Configured plugins:
[10:04:25]   ^-- null
2020-05-21 10:04:25.383  INFO 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor   :   ^-- null
2020-05-21 10:04:25.383  INFO [hypi,,,] 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor  [?] :   ^-- ml-inference-plugin
1.0.0
[10:04:25]
2020-05-21 10:04:25.383  INFO [hypi,,,] 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor  [?] :   ^-- null
2020-05-21 10:04:25.384  INFO 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor   :



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Courtney Robinson Courtney Robinson
Reply | Threaded
Open this post in threaded view
|

Re: Unable to enable ML inference storage

Hi Alex,
Thanks for replying. We're definitely not loading it twice.
The MLPluginProvider.onIgniteStart is being called being we can set the node to "active".
To work around it we had to override that method and call it after...
public class HypiMLPluginProvider extends MLPluginProvider {

/** {@inheritDoc} */
@Override public void onIgniteStart() {
SystemBootstrap.onBootstrapped(super::onIgniteStart);
}

}

is there a way to start a node as active without explicitly setting it to be we are currently doing 
ignite.cluster().active(true)
after
ignite = IgnitionEx.start(cfg, springGridCtx)
Regards,
Courtney Robinson
Founder and CEO, Hypi
https://hypi.io


On Thu, May 21, 2020 at 9:55 PM akorensh <[hidden email]> wrote:
Hi,

  Looks like you are loading the ML plugin twice (see below)
  Can you try w/out the plugin config first.
  Just copy the ignite-ml module to the libs folder or
  follow these instructions:
https://apacheignite.readme.io/docs/machine-learning#getting-started

  If that works, put in your xml config, and try again.

  see:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/ml/inference/ModelStorageExample.java
(model storage)
  and the corresponding:
https://github.com/apache/ignite/blob/master/examples/config/example-ignite-ml.xml
(this shows how to config ML plugin)


  If it still doesn't work send the config and, if possible, a reproducer
project.
Thanks, Alex



Multiple loadings of ML Plugin

 2020-05-21 10:04:25.382  INFO 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor   : Configured plugins:
[10:04:25]   ^-- ml-inference-plugin 1.0.0
2020-05-21 10:04:25.383  INFO 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor   :   ^-- ml-inference-plugin 1.0.0
2020-05-21 10:04:25.382  INFO [hypi,,,] 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor  [?] : Configured plugins:
[10:04:25]   ^-- null
2020-05-21 10:04:25.383  INFO 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor   :   ^-- null
2020-05-21 10:04:25.383  INFO [hypi,,,] 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor  [?] :   ^-- ml-inference-plugin
1.0.0
[10:04:25]
2020-05-21 10:04:25.383  INFO [hypi,,,] 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor  [?] :   ^-- null
2020-05-21 10:04:25.384  INFO 63933 --- [           main]
o.a.i.i.p.plugin.IgnitePluginProcessor   :



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
akorensh akorensh
Reply | Threaded
Open this post in threaded view
|

Re: Unable to enable ML inference storage

Courtney,
  This config should be enough
      <bean parent="ignite.cfg">
        <property name="pluginConfigurations">
            <bean
class="org.apache.ignite.ml.util.plugin.MLPluginConfiguration">
                <property name="withMdlStorage" value="#{true}" />
                <property name="withMdlDescStorage" value="#{true}" />
            </bean>
        </property>
    </bean>
 
  It will load the ML plugin and initialize model storage.

  You can send me a reproducer and I'll take a look.

Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/