Streaming test

classic Classic list List threaded Threaded
26 messages Options
12
waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Streaming test

Hello all,

I'm testing streaming input data and call a service deployed. Two things happened quite confusing and could someone in the community help?

1. I print a '.' in StreamVisitor.from method to show it's running, and call service via proxy. It actually ran for over 17K but then stopped/stuck. The file I used to stream data in has over 1M records in it.

2. I also print a '.' in the service method to show it's being called. It only printed couple dots and seemed to stuck as well.

First I thought, the message queue was too small, but increasing it to 10240 didn't change the behavior. <property name="messageQueueLimit" value="10240"/> 

What shall I do next to see where it might have got stuck?

Btw, the memory policy looks like this
                        <bean class="org.apache.ignite.configuration.MemoryPolicyConfiguration">
                            <property name="name" value="customRegion"/>
                           
                            <property name="initialSize" value="#{1  * 1024 * 1024 * 1024 }"/>
                           
                            <property name="swapFilePath" value="~/swap" />

and cache has a expiry policy looks like this.
<property name="expiryPolicyFactory">
                        <bean id="expiryPolicy" class="javax.cache.expiry.CreatedExpiryPolicy"
                              factory-method="factoryOf">
                            <constructor-arg>
                                <bean class="javax.cache.expiry.Duration">
                                    <constructor-arg value="MILLISECONDS"/>
                                    <constructor-arg value="5000"/>
                                </bean>
                            </constructor-arg>
                        </bean>
                    </property>
dkarachentsev dkarachentsev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Hi,

Could you please share your test project?

Thanks!
-Dmitry.
waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

I noticed that when I try to create a atomic sequence in service init method, it hangs. Is there any reason that atomicSequence cannot be used in service.init? If not, what'd be the alternative? AffinityUUID?

ignite.atomicSequence(
                "guid", // Sequence name.
                start,       // Initial value for sequence.
                true     // Create if it does not exist.
        );
waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Hello there, please my test project. You can see 1) The initialization seem didn't finish 2) the streaming client stopped after a while.
If you take out the Atomic sequence part, the stream start to work.
Thank you for helping.

Jessie
waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

This post was updated on .
CONTENTS DELETED
The author has deleted this message.
waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Hello, has anyone got a chance to look at it?
Thank you!
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

I ran the test. Atomic sequence initialization should be moved out of init() because it is called synchronously with node start, and they wait for each other.

After I do this, everything works fine. Although CSV file has only 348 and it gets loaded multiple times in a loop.

Is there anything else I need to do to reproduce the hang? In addition, if you do have the hang, take thread dumps from all nodes and attach them here.

-Val
waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Thank you for running the test.
If I need to initialize Atomic Seq, where shall I do it?

Jessie




On Fri, Jul 21, 2017 at 12:37 PM, vkulichenko <[hidden email]> wrote:
I ran the test. Atomic sequence initialization should be moved out of init()
because it is called synchronously with node start, and they wait for each
other.

After I do this, everything works fine. Although CSV file has only 348 and
it gets loaded multiple times in a loop.

Is there anything else I need to do to reproduce the hang? In addition, if
you do have the hang, take thread dumps from all nodes and attach them here.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p15263.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Jessie,

Ignite#atomicLong method will return existing instance if it is already initialized, so you actually don't need to manage this manually.

-Val
waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Val, thank you for the reply. 

Unfortunately I don't have an existing AtomicLong or AtomicSequence to work with... What if I need to initiate one, where shall I initiate it? We have a small cluster, sometime I only have one node for dev/test. It's a bit surprising why initiate an AtomicSequence would need to wait so long, because there's no other node to wait for...



Jessie

On Mon, Jul 24, 2017 at 4:33 PM, vkulichenko <[hidden email]> wrote:
Jessie,

Ignite#atomicLong method will return existing instance if it is already
initialized, so you actually don't need to manage this manually.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p15525.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Jessie,

atomicLong() method will do this for you automatically. It either creates new instance or returns existing one. There is no need for separate initialization step.

-Val
waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Thank you Val. I'll give it a try.

On Tue, Jul 25, 2017 at 5:16 PM, vkulichenko <[hidden email]> wrote:
Jessie,

atomicLong() method will do this for you automatically. It either creates
new instance or returns existing one. There is no need for separate
initialization step.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p15646.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Val, I replace atomicSequence with atomicLong and used the method below.
I need to specify an initial value, start, according to requirement.
return ignite.atomicLong(
"id", // name.
start, // Initial value
true // Create if it does not exist.
);
But the process got stuck as well.
Shall I do something differently? Appreciate your advice.

Best,
Jessie

On Tue, Jul 25, 2017 at 7:55 PM, Jessie Lin <[hidden email]> wrote:
Thank you Val. I'll give it a try.

On Tue, Jul 25, 2017 at 5:16 PM, vkulichenko <[hidden email]> wrote:
Jessie,

atomicLong() method will do this for you automatically. It either creates
new instance or returns existing one. There is no need for separate
initialization step.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p15646.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Jessie,

What do you mean by "stuck"? Did you check thread dumps? Is it possible you have memory/GC issues?

-Val
waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Val, I think the ignite.atomicLong() function never returned and just hang there. If you run the test project I attached it should be reproducable. 

I don't expect to have memory issue here. As the service init doesn't do much else. And I only have one server node and one client node.

Jessie

On Tue, Aug 8, 2017 at 5:36 PM, vkulichenko <[hidden email]> wrote:
Jessie,

What do you mean by "stuck"? Did you check thread dumps? Is it possible you
have memory/GC issues?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16066.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

I can't reproduce it, your project works fine for me. Can you attach thread dumps?

-Val
waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Val, please see thread print attached.
This is take after a server is run by "bin\ignite.bat config\ignite-writebehind.xml" and the service initialization didn't complete.
Thank you very much for helping out!

"srvc-deploy-#33%null%" #59 prio=5 os_prio=0 tid=0x00000000577b8000 nid=0x1ef8 waiting on condition [0x00000000608fe000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000c09fbdc8> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at org.apache.ignite.internal.util.IgniteUtils.await(IgniteUtils.java:7419)
at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.awaitInitialization(DataStructuresProcessor.java:1112)
at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.atomicLong(DataStructuresProcessor.java:517)
at org.apache.ignite.internal.IgniteKernal.atomicLong(IgniteKernal.java:3436)
at com.sample.SampleServiceImpl.init(SampleServiceImpl.java:63)

Jessie

On Wed, Aug 9, 2017 at 2:29 PM, vkulichenko <[hidden email]> wrote:
I can't reproduce it, your project works fine for me. Can you attach thread
dumps?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16088.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


threadprint_test.txt (52K) Download Attachment
vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Jessie,

You still call atomicLong() method from Service#init(). As I already mentioned, this is causing the startup hang. You should move IgniteAtomicLong creation out of init() method to avoid it.

-Val
waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Val, thanks for pointing it out. Now I call AtomicLong Function from service#execute() and it's working. Thank you very much!

Jessie 

On Thu, Aug 10, 2017 at 3:08 PM, vkulichenko <[hidden email]> wrote:
Jessie,

You still call atomicLong() method from Service#init(). As I already
mentioned, this is causing the startup hang. You should move
IgniteAtomicLong creation out of init() method to avoid it.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16113.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

waterg waterg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Streaming test

Hello Val,

I now put the stream visitor code in a different model and packaged it in a different jar with dependencies.

Interestingly, when I ran the program again, the server node has weird error messages.
The FileConnector is a class in the client jar. Why would the service needs the caller's signature?
Btw, I turned enablePeerClassLoader on, but I still have the same error.

Jessie

Caused by: java.lang.ClassNotFoundException: com.sample.client.FileConnector
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_
91]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_91]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8
.0_91]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_91]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_91]
        at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_91]
        at org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:
8478) ~[sample-client-0.1-SNAPSHOT-jar-with-dependencies.jar:?]
        at org.apache.ignite.internal.MarshallerContextImpl.getClass(MarshallerC
ontextImpl.java:340) ~[sample-client-0.1-SNAPSHOT-jar-with-dependencies.jar:
?]
        at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshallerUt
ils.classDescriptor(OptimizedMarshallerUtils.java:268) ~[sample-client:?]
12
Loading...