IGFS block at startup

classic Classic list List threaded Threaded
6 messages Options
otorreno otorreno
Reply | Threaded
Open this post in threaded view
|

IGFS block at startup

Hi,

 

I am trying to start a fresh 2 nodes Ignite 2.7.0 cluster (using docker-compose) with 2 IGFS configured. When I start both nodes at the same time, almost always one of them starts without problems, but the second one hangs at line 120 of the IgfsMetaManager class (doing an await on a CountDownLatch). Rarely, both nodes progress, so it seems to be a kind of race condition/inconsistent state problem because of the simultaneous start.

 

Have you experienced such issue before? If yes, is there any workaround to overcome it?

 

Best regards, thanks in advance.

Oscar

 

Logo

Oscar Torreno

Software Engineer

m: + 34 675 026 952

e: [hidden email]

C/ Puerta del Mar 18, 2º. 29005, Málaga,Spain

LinkedIn icon  Twitter icon  

 

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: IGFS block at startup

Hello!

Can you please provide complete logs and stack traces from both nodes?

I guess we have a lot of tests where we start several IGFS nodes and they finish just fine.

Regards,
--
Ilya Kasnacheev


пн, 8 июл. 2019 г. в 10:16, Oscar Torreno <[hidden email]>:

Hi,

 

I am trying to start a fresh 2 nodes Ignite 2.7.0 cluster (using docker-compose) with 2 IGFS configured. When I start both nodes at the same time, almost always one of them starts without problems, but the second one hangs at line 120 of the IgfsMetaManager class (doing an await on a CountDownLatch). Rarely, both nodes progress, so it seems to be a kind of race condition/inconsistent state problem because of the simultaneous start.

 

Have you experienced such issue before? If yes, is there any workaround to overcome it?

 

Best regards, thanks in advance.

Oscar

 

Logo

Oscar Torreno

Software Engineer

m: + 34 675 026 952

e: [hidden email]

C/ Puerta del Mar 18, 2º. 29005, Málaga,Spain

LinkedIn icon  Twitter icon  

 

otorreno otorreno
Reply | Threaded
Open this post in threaded view
|

Re: IGFS block at startup

Hello Ilya,

 

Please find attached the docker compose log of both nodes (shapelets-1 and shapelets-2). Shapelets-2 was the one able to start without problems in this case. Attaching the output of jstack for the main Thread of the shapelets-1 node.

 

Regards,

--

Oscar Torreno

 

From: Ilya Kasnacheev <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, 8 July 2019 at 11:25
To: "[hidden email]" <[hidden email]>
Subject: Re: IGFS block at startup

 

Hello!

 

Can you please provide complete logs and stack traces from both nodes?

 

I guess we have a lot of tests where we start several IGFS nodes and they finish just fine.

 

Regards,

--

Ilya Kasnacheev

 

 

пн, 8 июл. 2019 г. в 10:16, Oscar Torreno <[hidden email]>:

Hi,

 

I am trying to start a fresh 2 nodes Ignite 2.7.0 cluster (using docker-compose) with 2 IGFS configured. When I start both nodes at the same time, almost always one of them starts without problems, but the second one hangs at line 120 of the IgfsMetaManager class (doing an await on a CountDownLatch). Rarely, both nodes progress, so it seems to be a kind of race condition/inconsistent state problem because of the simultaneous start.

 

Have you experienced such issue before? If yes, is there any workaround to overcome it?

 

Best regards, thanks in advance.

Oscar

 

Logo

Oscar Torreno

Software Engineer

m: + 34 675 026 952

e: [hidden email]

C/ Puerta del Mar 18, 2º. 29005, Málaga,Spain

LinkedIn icon  Twitter icon  

 


igfs-issue-log.txt (910K) Download Attachment
igfs-issue-trace.txt (9K) Download Attachment
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: IGFS block at startup

Hello!

It is hard to say what is happening here without full stack trace from all threads of both nodes. Can you provide that?

Regards,
--
Ilya Kasnacheev


пн, 8 июл. 2019 г. в 13:48, Oscar Torreno <[hidden email]>:

Hello Ilya,

 

Please find attached the docker compose log of both nodes (shapelets-1 and shapelets-2). Shapelets-2 was the one able to start without problems in this case. Attaching the output of jstack for the main Thread of the shapelets-1 node.

 

Regards,

--

Oscar Torreno

 

From: Ilya Kasnacheev <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, 8 July 2019 at 11:25
To: "[hidden email]" <[hidden email]>
Subject: Re: IGFS block at startup

 

Hello!

 

Can you please provide complete logs and stack traces from both nodes?

 

I guess we have a lot of tests where we start several IGFS nodes and they finish just fine.

 

Regards,

--

Ilya Kasnacheev

 

 

пн, 8 июл. 2019 г. в 10:16, Oscar Torreno <[hidden email]>:

Hi,

 

I am trying to start a fresh 2 nodes Ignite 2.7.0 cluster (using docker-compose) with 2 IGFS configured. When I start both nodes at the same time, almost always one of them starts without problems, but the second one hangs at line 120 of the IgfsMetaManager class (doing an await on a CountDownLatch). Rarely, both nodes progress, so it seems to be a kind of race condition/inconsistent state problem because of the simultaneous start.

 

Have you experienced such issue before? If yes, is there any workaround to overcome it?

 

Best regards, thanks in advance.

Oscar

 

Logo

Oscar Torreno

Software Engineer

m: + 34 675 026 952

e: [hidden email]

C/ Puerta del Mar 18, 2º. 29005, Málaga,Spain

LinkedIn icon  Twitter icon  

 

otorreno otorreno
Reply | Threaded
Open this post in threaded view
|

Re: IGFS block at startup

Hello Ilya,

 

Please find attached the log and the complete stack traces of both nodes.

 

I n this case the node “shapelets-2” was the one blocked at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.awaitInit(IgfsMetaManager.java:177)

 

Best,

--

Oscar Torreno

 

From: Ilya Kasnacheev <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, 8 July 2019 at 14:15
To: "[hidden email]" <[hidden email]>
Subject: Re: IGFS block at startup

 

Hello!

 

It is hard to say what is happening here without full stack trace from all threads of both nodes. Can you provide that?

 

Regards,

--

Ilya Kasnacheev

 

 

пн, 8 июл. 2019 г. в 13:48, Oscar Torreno <[hidden email]>:

Hello Ilya,

 

Please find attached the docker compose log of both nodes (shapelets-1 and shapelets-2). Shapelets-2 was the one able to start without problems in this case. Attaching the output of jstack for the main Thread of the shapelets-1 node.

 

Regards,

--

Oscar Torreno

 

From: Ilya Kasnacheev <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, 8 July 2019 at 11:25
To: "[hidden email]" <[hidden email]>
Subject: Re: IGFS block at startup

 

Hello!

 

Can you please provide complete logs and stack traces from both nodes?

 

I guess we have a lot of tests where we start several IGFS nodes and they finish just fine.

 

Regards,

--

Ilya Kasnacheev

 

 

пн, 8 июл. 2019 г. в 10:16, Oscar Torreno <[hidden email]>:

Hi,

 

I am trying to start a fresh 2 nodes Ignite 2.7.0 cluster (using docker-compose) with 2 IGFS configured. When I start both nodes at the same time, almost always one of them starts without problems, but the second one hangs at line 120 of the IgfsMetaManager class (doing an await on a CountDownLatch). Rarely, both nodes progress, so it seems to be a kind of race condition/inconsistent state problem because of the simultaneous start.

 

Have you experienced such issue before? If yes, is there any workaround to overcome it?

 

Best regards, thanks in advance.

Oscar

 

Logo

Oscar Torreno

Software Engineer

m: + 34 675 026 952

e: [hidden email]

C/ Puerta del Mar 18, 2º. 29005, Málaga,Spain

LinkedIn icon  Twitter icon  

 


nodes-log.txt (1000K) Download Attachment
shapelets1-stack.txt (196K) Download Attachment
shapelets2-stack.txt (91K) Download Attachment
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: IGFS block at startup

In reply to this post by otorreno
Oscar, 

IGFS won't longer be supported by the community:

Switch to the solutions of this kind if you're accelerating Hadoop:

-
Denis


On Mon, Jul 8, 2019 at 12:16 AM Oscar Torreno <[hidden email]> wrote:

Hi,

 

I am trying to start a fresh 2 nodes Ignite 2.7.0 cluster (using docker-compose) with 2 IGFS configured. When I start both nodes at the same time, almost always one of them starts without problems, but the second one hangs at line 120 of the IgfsMetaManager class (doing an await on a CountDownLatch). Rarely, both nodes progress, so it seems to be a kind of race condition/inconsistent state problem because of the simultaneous start.

 

Have you experienced such issue before? If yes, is there any workaround to overcome it?

 

Best regards, thanks in advance.

Oscar

 

Logo

Oscar Torreno

Software Engineer

m: + 34 675 026 952

e: [hidden email]

C/ Puerta del Mar 18, 2º. 29005, Málaga,Spain

LinkedIn icon  Twitter icon