From one cache to another

classic Classic list List threaded Threaded
4 messages Options
alpha centauri alpha centauri
Reply | Threaded
Open this post in threaded view
|

From one cache to another

Hello,

I'm a new Apache Ignite User. I'm trying to prototype a simple use case :
1. create two caches
2. read strings from a file and put them in the first cache
3. compute strings into complex objects put the complex objects into second cache

My code looks like this :

// Part 1
final IgniteCache<Integer, String> cacheString = ignite.getOrCreateCache("somestrings");
final IgniteCache<String, SomeObjects> cache = ignite.getOrCreateCache("someobjects");

// Part 2
try (IgniteDataStreamer<Integer, String> stmr = ignite.dataStreamer(cacheString.getName())) {
                stmr.allowOverwrite(true);
                try (BufferedReader br = Files.newReader(file, StandardCharsets.UTF_8)) {
                    for (String line; (line = br.readLine()) != null; ) {
                        if (line !=null && line.length() > 0) {
                            stmr.addData(index, line);
                        }
                    }

                }
            }

// Part 3 is where I struggle and need help
IgniteCompute compute = ignite.compute();

for (int i = 0; i < cacheString.size(); i++) {
         final int key = i;
         compute.affinityRun(cacheString.getName(), key, new ComplexObjectProcessor(cache, cacheString.get(key)));
}

The issue is that the ComplexObjectProcessor is serialized to my grid nodes and if I put the destination cache, it will crash because the cache cannot be serialized. If I initialize the cache from my ComplexObjectProcessor, it will complain about the fact that Ignite has already been instanciated  outside. ComplexObjectProcessor has a lot of code so an inside/inline class is not an option.
The cacheString contains 100Millions lines so I really want to split the complex object processor with compute tasks.

Any solution? Or code sample would greatly help.

Thks ahead,
AC

agura-2 agura-2
Reply | Threaded
Open this post in threaded view
|

Re: From one cache to another

CONTENTS DELETED
The author has deleted this message.
alpha centauri alpha centauri
Reply | Threaded
Open this post in threaded view
|

RE: From one cache to another


Hello,

Thanks for your answers, guys.

It works now but I struggled a lot with the serialization process.

Maybe somewhere strongly underline the fact in the documentation that ONLY certain type of classes (I think it's closures and some other
inner tasks) are part of the zero deployment mechanisms. For like two hours, I didn't understood why any object inside my tasks would not be automagically deployed on the cluster nodes.

It is still a bit annoying. Once I create a new processor, I have to restart all my nodes to have it properly deployed everywhere. Is there something I'm missing ?

Anyway thanks, Ignite is a really powerfull thing!




Date: Fri, 27 Nov 2015 13:55:08 +0300
Subject: Re: From one cache to another
From: [hidden email]
To: [hidden email]

Hi,

You should not serialize cache instances. Actually your cache is already created on all nodes of your cluster and you can obtain it using ignite.cache(<name>) or ignite.getOrCreateCache(<name>) methods.

I assume that you try to start Ignite instance with default name from your ComplexObjectProcessor but Ignite is started already. You can declare Ignite reference in ComplexObjectProcessor and annotate it with @IgniteInstanceResource. When your processor will be deserialized on remote node Ignite instance will be injected automatically. It looks like this:

public class ComplexObjectProcessor {

    @IgniteInstanceResource
    private Ignite ignite

    ...
}

Make sure that your ComplexObjectProcessor is regular class or inner static class. Non-static inner class instances have reference to enclosing class that will be serialized with inner class.


On Thu, Nov 26, 2015 at 9:08 PM, alpha centauri <[hidden email]> wrote:
Hello,

I'm a new Apache Ignite User. I'm trying to prototype a simple use case :
1. create two caches
2. read strings from a file and put them in the first cache
3. compute strings into complex objects put the complex objects into second cache

My code looks like this :

// Part 1
final IgniteCache<Integer, String> cacheString = ignite.getOrCreateCache("somestrings");
final IgniteCache<String, SomeObjects> cache = ignite.getOrCreateCache("someobjects");

// Part 2
try (IgniteDataStreamer<Integer, String> stmr = ignite.dataStreamer(cacheString.getName())) {
                stmr.allowOverwrite(true);
                try (BufferedReader br = Files.newReader(file, StandardCharsets.UTF_8)) {
                    for (String line; (line = br.readLine()) != null; ) {
                        if (line !=null && line.length() > 0) {
                            stmr.addData(index, line);
                        }
                    }

                }
            }

// Part 3 is where I struggle and need help
IgniteCompute compute = ignite.compute();

for (int i = 0; i < cacheString.size(); i++) {
         final int key = i;
         compute.affinityRun(cacheString.getName(), key, new ComplexObjectProcessor(cache, cacheString.get(key)));
}

The issue is that the ComplexObjectProcessor is serialized to my grid nodes and if I put the destination cache, it will crash because the cache cannot be serialized. If I initialize the cache from my ComplexObjectProcessor, it will complain about the fact that Ignite has already been instanciated  outside. ComplexObjectProcessor has a lot of code so an inside/inline class is not an option.
The cacheString contains 100Millions lines so I really want to split the complex object processor with compute tasks.

Any solution? Or code sample would greatly help.

Thks ahead,
AC




--
Andrey Gura
GridGain Systems, Inc.
Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: From one cache to another

Hi,

Actually if a processor is being deployed on a remote node all other classes, the processor refers to from its code, are deployed as well.

In my understanding you should take into account the deployment mode that is used on your side.

By default SHARED mode is activated which means for you that if you had several nodes that were sending the processor via compute.affinityRun (or other method) then either all of them has to be restarted once the processor is modified (no need to restart the nodes that executed the task and deployed the processor class remotely) or restart only a single node that modified the processor but provide updated class version in  META-INF/ignite.xml. Please refer to DeploymentMode.SHARED documentation for more info.  

--
Denis

On 12/6/2015 3:51 AM, alpha centauri wrote:

Hello,

Thanks for your answers, guys.

It works now but I struggled a lot with the serialization process.

Maybe somewhere strongly underline the fact in the documentation that ONLY certain type of classes (I think it's closures and some other
inner tasks) are part of the zero deployment mechanisms. For like two hours, I didn't understood why any object inside my tasks would not be automagically deployed on the cluster nodes.

It is still a bit annoying. Once I create a new processor, I have to restart all my nodes to have it properly deployed everywhere. Is there something I'm missing ?

Anyway thanks, Ignite is a really powerfull thing!




Date: Fri, 27 Nov 2015 13:55:08 +0300
Subject: Re: From one cache to another
From: [hidden email]
To: [hidden email]

Hi,

You should not serialize cache instances. Actually your cache is already created on all nodes of your cluster and you can obtain it using ignite.cache(<name>) or ignite.getOrCreateCache(<name>) methods.

I assume that you try to start Ignite instance with default name from your ComplexObjectProcessor but Ignite is started already. You can declare Ignite reference in ComplexObjectProcessor and annotate it with @IgniteInstanceResource. When your processor will be deserialized on remote node Ignite instance will be injected automatically. It looks like this:

public class ComplexObjectProcessor {

    @IgniteInstanceResource
    private Ignite ignite

    ...
}

Make sure that your ComplexObjectProcessor is regular class or inner static class. Non-static inner class instances have reference to enclosing class that will be serialized with inner class.


On Thu, Nov 26, 2015 at 9:08 PM, alpha centauri <[hidden email]> wrote:
Hello,

I'm a new Apache Ignite User. I'm trying to prototype a simple use case :
1. create two caches
2. read strings from a file and put them in the first cache
3. compute strings into complex objects put the complex objects into second cache

My code looks like this :

// Part 1
final IgniteCache<Integer, String> cacheString = ignite.getOrCreateCache("somestrings");
final IgniteCache<String, SomeObjects> cache = ignite.getOrCreateCache("someobjects");

// Part 2
try (IgniteDataStreamer<Integer, String> stmr = ignite.dataStreamer(cacheString.getName())) {
                stmr.allowOverwrite(true);
                try (BufferedReader br = Files.newReader(file, StandardCharsets.UTF_8)) {
                    for (String line; (line = br.readLine()) != null; ) {
                        if (line !=null && line.length() > 0) {
                            stmr.addData(index, line);
                        }
                    }

                }
            }

// Part 3 is where I struggle and need help
IgniteCompute compute = ignite.compute();

for (int i = 0; i < cacheString.size(); i++) {
         final int key = i;
         compute.affinityRun(cacheString.getName(), key, new ComplexObjectProcessor(cache, cacheString.get(key)));
}

The issue is that the ComplexObjectProcessor is serialized to my grid nodes and if I put the destination cache, it will crash because the cache cannot be serialized. If I initialize the cache from my ComplexObjectProcessor, it will complain about the fact that Ignite has already been instanciated  outside. ComplexObjectProcessor has a lot of code so an inside/inline class is not an option.
The cacheString contains 100Millions lines so I really want to split the complex object processor with compute tasks.

Any solution? Or code sample would greatly help.

Thks ahead,
AC




--
Andrey Gura
GridGain Systems, Inc.