IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs'

classic Classic list List threaded Threaded
7 messages Options
RobertOfFairfax RobertOfFairfax
Reply | Threaded
Open this post in threaded view
|

IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs'

This post has NOT been accepted by the mailing list yet.
Hi,

I have attempted to configure IGFS on a custom CDH Docker image as follows:

$IGNITE_HOME=/home/ignite_home/ignite/apache-ignite-hadoop-1.4.0-bin

where I ran '$IGNITE_HOME/bin/setup-hadoop.sh' and answered 'y' to all questions which provided:

1) New core-site.xml and mapred-site.xml files in /etc/hadoop/conf where core-site.xml has the following properties:

    <property>
        <name>fs.default.name</name>
        <value>igfs://igfs@localhost</value> 
    </property>

    <property>
        <name>fs.igfs.impl</name>
        <value>org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem</value>
    </property>

    ...

2) symbolic links in /usr/lib/hadoop/lib:
 
ignite-core-1.4.0.jar -> /home/ignite_home/ignite/apache-ignite-hadoop-1.4.0-bin/libs/ignite-core-1.4.0.jar

ignite-hadoop-1.4.0.jar -> /home/ignite_home/ignite/apache-ignite-hadoop-1.4.0-bin/libs/ignite-hadoop/ignite-hadoop-1.4.0.jar

ignite-shmem-1.0.0.jar -> /home/ignite_home/ignite/apache-ignite-hadoop-1.4.0-bin/libs/ignite-shmem-1.0.0.jar


As I start hadoop/hdfs, I observe the following errors:

Start HDFS
starting datanode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-datanode-7eac8860ea37.out
 * Failed to start Hadoop datanode. Return value: 1
starting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-7eac8860ea37.out
 * Failed to start Hadoop namenode. Return value: 1
starting secondarynamenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-secondarynamenode-7eac8860ea37.out
 * Failed to start Hadoop secondarynamenode. Return value: 1

Upon review of the secondarynamenode .out file, I see:

Exception in thread "main" java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs'.
        at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:477)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:461)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.getServiceAddress(NameNode.java:454)
        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:231)
        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:194)
        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:675)

Though I observed the above failures, I also ran 'hadoop fs -ls /' which reports "ls: Failed to communicate with IGFS."

Any help would be appreciated. Thanks.
Ivan Veselovsky Ivan Veselovsky
Reply | Threaded
Open this post in threaded view
|

Re: IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs'

This post was updated on .
Hi, Robert,
please properly subscribe to the user list (this way we will not have to manually approve your emails). All you need to do is send an email to “user-subscribe@ignite.apache.org” and follow simple instructions in the reply.


The problem is that "setup-hadoop.sh" changes global Hadoop configs, and after that all Hadoop services, including namenode and datanode are using it. But Hadoop namenode and datanode cannot work over IGFS, namenode explicitly checks that -- this is the cause of the error you see.

Relevant doc page: https://apacheignite.readme.io/docs/file-system says:
-----
Second, you may override default core-site.xml of your Hadoop installation. This will force all Hadoop jobs to pick Ignite jobs tracker by default unless it is overriden on job level somehow. Note that you will not be able to HDFS in this case.
-----
If you change the global configs as "setup-hadoop.sh" does, you will not be able to use HDFS services any more, and will be able to run IGFS in "primary" mode only (see https://apacheignite.readme.io/docs/modes).

Way to fix the situation is described there: https://issues.apache.org/jira/browse/IGNITE-483 .


 "ls: Failed to communicate with IGFS."  is a separate problem that indicates that Hadoop client for some reason fails to connect to IGFS node. First of all, is the Ignite node started correctly? If it runs correctly, but the problem persists, can you please attach configs and logs.

Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs'

Robert,

Did you have a chance to apply the steps described in IGNITE-483?
However probably you're planning to use IGFS is some other mode and you don't need to fix any there.

If something is unclear to you let us know and clarify how you're going to use IGFS (in which mode).
 

--
Denis
RobertOfFairfax RobertOfFairfax
Reply | Threaded
Open this post in threaded view
|

Re: IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs'

Hi Ivan, Denis, 

Thanks for your prompt response. Currently, my goal is to observe performance with and without IGFS using a stream of Pig commands against delimited files. 

> Did you have a chance to apply the steps described in IGNITE-483? 

This is a weekend activity with the following objectives for which I have described in a docker project (https://github.com/rkennington/ignite-docker): 

1) Configure Hadoop to integrate with IGFS based on installing it on cloudera CDH. - as well as confirm it is working with a few CLI commands. Load some comma delimited files to work with.

2) Startup one or more Ignite clients to use it as a resource. 

3) Develop a client that distributes and executes a stream of Pig commands. Continue to use the Docker platform as a development environment to experiment with. 

4) Migrate it to a distributed cluster to observe performance. 

5) Explore other aspects of Ignite use cases. 


> However probably you're planning to use IGFS is some other mode and you don't need to fix any there. 

You may be right. I could be over thinking this problem with old hadoop file system CLI biases for which I may need to unlearn. As I move along, I'm trying to understand what CLI commands I can use to verify my setup. Maybe the

 'hadoop fs -ls /' 

command is not of value in this context. 


> If something is unclear to you let us know and clarify how you're going to use IGFS (in which mode). 

Will do, thanks. 

Robert 
571-882-2580 m 


On Wed, Nov 18, 2015 at 4:57 AM, Denis Magda [via Apache Ignite Users] <[hidden email]> wrote:
Robert,

Did you have a chance to apply the steps described in IGNITE-483?
However probably you're planning to use IGFS is some other mode and you don't need to fix any there.

If something is unclear to you let us know and clarify how you're going to use IGFS (in which mode).
 

--
Denis


To unsubscribe from IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs', click here.
NAML

RobertOfFairfax RobertOfFairfax
Reply | Threaded
Open this post in threaded view
|

Re: IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs'

In reply to this post by Ivan Veselovsky
Hi Ivan,

The hadoop wrapper example in IGNITE-483 with an updated HADOOP_CLASSPATH worked great as well as understanding that you can't get this working until Ignite is running.

In summary, I

1) copied the core-site-ignite.xml and mapred-site-ignite.xml files to $IGNITE_HOME/ignite-conf/

2) renamed them to core-site-ignite.xml and mapred-site-ignite.xml

3) created the hadoop-ignite wrapper script similar to that described in IGNITE-483

4) started Ignite with '$IGNITE_HOME/bin/ignite.sh $IGNITE_HOME/config/default-config.xml'

Further details of it's usage are described here.

Thanks,
Robert
Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs'

Hi Robert,

So, no more issues so far with this and workarounds work fine for you,
right?

--
Denis

On 11/19/2015 5:43 PM, RobertOfFairfax wrote:

> Hi Ivan,
>
> The hadoop wrapper example in  IGNITE-483
> <https://issues.apache.org/jira/browse/IGNITE-483>   with an updated
> HADOOP_CLASSPATH worked great as well as understanding that you can't get
> this working until Ignite is running.
>
> In summary, I
>
> 1) copied the core-site-ignite.xml and mapred-site-ignite.xml files to
> $IGNITE_HOME/ignite-conf/
>
> 2) renamed them to core-site-ignite.xml and mapred-site-ignite.xml
>
> 3) created the hadoop-ignite wrapper script similar to that described in
> IGNITE-483
>
> 4) started Ignite with '$IGNITE_HOME/bin/ignite.sh
> $IGNITE_HOME/config/default-config.xml'
>
> Further details of it's usage are described  here
> <https://github.com/rkennington/ignite-docker>  .
>
> Thanks,
> Robert
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IllegalArgumentException-Invalid-URI-for-NameNode-address-check-fs-defaultFS-igfs-igfs-localhost-is--tp1978p2017.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

RobertOfFairfax RobertOfFairfax
Reply | Threaded
Open this post in threaded view
|

Re: IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs'

Hi Denis, 

All is well. 

Thanks, 
Robert 


On Fri, Nov 20, 2015 at 5:21 AM, Denis Magda [via Apache Ignite Users] <[hidden email]> wrote:
Hi Robert,

So, no more issues so far with this and workarounds work fine for you,
right?

--
Denis

On 11/19/2015 5:43 PM, RobertOfFairfax wrote:

> Hi Ivan,
>
> The hadoop wrapper example in  IGNITE-483
> <https://issues.apache.org/jira/browse/IGNITE-483>   with an updated
> HADOOP_CLASSPATH worked great as well as understanding that you can't get

> this working until Ignite is running.
>
> In summary, I
>
> 1) copied the core-site-ignite.xml and mapred-site-ignite.xml files to
> $IGNITE_HOME/ignite-conf/
>
> 2) renamed them to core-site-ignite.xml and mapred-site-ignite.xml
>
> 3) created the hadoop-ignite wrapper script similar to that described in
> IGNITE-483
>
> 4) started Ignite with '$IGNITE_HOME/bin/ignite.sh
> $IGNITE_HOME/config/default-config.xml'
>
> Further details of it's usage are described  here
> <https://github.com/rkennington/ignite-docker>  .

>
> Thanks,
> Robert
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IllegalArgumentException-Invalid-URI-for-NameNode-address-check-fs-defaultFS-igfs-igfs-localhost-is--tp1978p2017.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.



To unsubscribe from IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs', click here.
NAML