As I start hadoop/hdfs, I observe the following errors:
starting datanode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-datanode-7eac8860ea37.out
* Failed to start Hadoop datanode. Return value: 1
starting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-7eac8860ea37.out
* Failed to start Hadoop namenode. Return value: 1
starting secondarynamenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-secondarynamenode-7eac8860ea37.out
* Failed to start Hadoop secondarynamenode. Return value: 1
Upon review of the secondarynamenode .out file, I see:
Exception in thread "main" java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs'.
Though I observed the above failures, I also ran 'hadoop fs -ls /' which reports "ls: Failed to communicate with IGFS."
Re: IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): igfs://igfs@localhost is not of scheme 'hdfs'
This post was updated on .
please properly subscribe to the user list (this way we will not have to manually approve your emails). All you need to do is send an email to “email@example.com” and follow simple instructions in the reply.
The problem is that "setup-hadoop.sh" changes global Hadoop configs, and after that all Hadoop services, including namenode and datanode are using it. But Hadoop namenode and datanode cannot work over IGFS, namenode explicitly checks that -- this is the cause of the error you see.
Relevant doc page: https://apacheignite.readme.io/docs/file-system says:
Second, you may override default core-site.xml of your Hadoop installation. This will force all Hadoop jobs to pick Ignite jobs tracker by default unless it is overriden on job level somehow. Note that you will not be able to HDFS in this case.
If you change the global configs as "setup-hadoop.sh" does, you will not be able to use HDFS services any more, and will be able to run IGFS in "primary" mode only (see https://apacheignite.readme.io/docs/modes).
"ls: Failed to communicate with IGFS." is a separate problem that indicates that Hadoop client for some reason fails to connect to IGFS node. First of all, is the Ignite node started correctly? If it runs correctly, but the problem persists, can you please attach configs and logs.
1) Configure Hadoop to integrate with IGFS based on installing it on cloudera CDH. - as well as confirm it is working with a few CLI commands. Load some comma delimited files to work with.
2) Startup one or more Ignite clients to use it as a resource.
3) Develop a client that distributes and executes a stream of Pig commands. Continue to use the Docker platform as a development environment to experiment with.
4) Migrate it to a distributed cluster to observe performance.
5) Explore other aspects of Ignite use cases.
> However probably you're planning to use IGFS is some other mode and you don't need to fix any there.
You may be right. I could be over thinking this problem with old hadoop file system CLI biases for which I may need to unlearn. As I move along, I'm trying to understand what CLI commands I can use to verify my setup. Maybe the
'hadoop fs -ls /'
command is not of value in this context.
> If something is unclear to you let us know and clarify how you're going to use IGFS (in which mode).
Will do, thanks.
On Wed, Nov 18, 2015 at 4:57 AM, Denis Magda [via Apache Ignite Users] <[hidden email]> wrote:
Did you have a chance to apply the steps described in IGNITE-483?
However probably you're planning to use IGFS is some other mode and you don't need to fix any there.
If something is unclear to you let us know and clarify how you're going to use IGFS (in which mode).
If you reply to this email, your message will be added to the discussion below:
> this working until Ignite is running.
> In summary, I
> 1) copied the core-site-ignite.xml and mapred-site-ignite.xml files to
> 2) renamed them to core-site-ignite.xml and mapred-site-ignite.xml
> 3) created the hadoop-ignite wrapper script similar to that described in
> 4) started Ignite with '$IGNITE_HOME/bin/ignite.sh