Hadoop File System Cache Setting Problems

classic Classic list List threaded Threaded
2 messages Options
yaoqin yaoqin
Reply | Threaded
Open this post in threaded view
|

Hadoop File System Cache Setting Problems

HI! I'm looking for help from you to set and test Apache Ignite with Hadoop V2 and Spark 1.5.x. I'm confused about the following questions, and hope to get your help.

 

I'd like to set IGFS as Hadoop FileSystem Cache. And I modified "$IGNITE_HOME/examples/config/filesystem/example-igfs.xml" to "$IGNITE_HOME/examples/config/filesystem/yq-example-igfs.xml" by adding the following elements to it:

计算机生成了可选文字:
《I一
Exampleofconfiguredsharedmemoryendpoint.
-一>
<!一
<propertyname=',ipcEndpointConfiguration">
<propertyname='"type''value=''SHME州‘,/>
</bean>
</p户operty>一
-一>
<beanclass="org.apache.ignite.igfs.IgfsIpcEndpointConfiguration
,,沪―
<卜一propertyname='"secondaryFilesystem',>
<beanclass=',org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFi1eSystem''>
<constructor一argvalue="hdfs://master:9000""/>
-</bean>
</property一>
</bean>
万而二沂气月丫洁{
厂月
只之一
二__准式_丫耳狱厂一―
、J丫、了一~勺厂一一“气一口压
<beanclass=,org.apache.ignite.configuration
<propertyname="secondaryFilesystem",>
.FilesystemConfiguration">
、入
<beanclass=',org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFi1eSystem">
<constructor一argvalue='hdfs://master:9000,'/>,
</bean>
be3n>
</list>
</property>
</property>
尸

 

For detail, you can see the Appendix I: yq-example-igfs.xml

 

I can get ignite started by example-igfs.xml rightly, but when I use yq-example-igfs.xml, it always fails  like this:

 

 

 

master:/home/software/apache-ignite # ignite.sh -i

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0

Available configuration files:

+----------------------------+

0:      config/default-config.xml

1:      (?) config/router/default-router.xml

2:      examples/config/example-cache.xml

3:      examples/config/example-ignite.xml

4:      examples/config/filesystem/example-igfs.xml

5:      examples/config/filesystem/yq-example-igfs.xml

6:      examples/src/main/java/org/apache/ignite/examples/misc/springbean/spring-bean.xml

 

Choose configuration file ('c' to cancel) [0]: 5

 

Using configuration: examples/config/filesystem/yq-example-igfs.xml

 

class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context (make sure all classes used in Spring configuration are present at CLASSPATH) [springUrl=file:/home/software/apache-ignite/examples/config/filesystem/yq-example-igfs.xml]

        at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:684)

        at org.apache.ignite.Ignition.start(Ignition.java:349)

        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:289)

Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context (make sure all classes used in Spring configuration are present at CLASSPATH) [springUrl=file:/home/software/apache-ignite/examples/config/filesystem/yq-example-igfs.xml]

        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:370)

        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:87)

        at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:81)

        at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:547)

        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:740)

        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:671)

        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:527)

        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:497)

        at org.apache.ignite.Ignition.start(Ignition.java:346)

        ... 1 more

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ignite.cfg' defined in URL [<a href="file:///\\home\software\apache-ignite\examples\config\filesystem\yq-example-igfs.xml">file:/home/software/apache-ignite/examples/config/filesystem/yq-example-igfs.xml]: Cannot create inner bean 'org.apache.ignite.configuration.FileSystemConfiguration#131276c2' of type [org.apache.ignite.configuration.FileSystemConfiguration] while setting bean property 'fileSystemConfiguration' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.ignite.configuration.FileSystemConfiguration#131276c2' defined in URL [<a href="file:///\\home\software\apache-ignite\examples\config\filesystem\yq-example-igfs.xml">file:/home/software/apache-ignite/examples/config/filesystem/yq-example-igfs.xml]: Cannot create inner bean 'org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem#1c3a4799' of type [org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem] while setting bean property 'secondaryFileSystem'; nested exception is org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem] for bean with name 'org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem#1c3a4799' defined in URL [<a href="file:///\\home\software\apache-ignite\examples\config\filesystem\yq-example-igfs.xml">file:/home/software/apache-ignite/examples/config/filesystem/yq-example-igfs.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem

 

 

 

And in https://apacheignite.readme.io/v1.3/docs/file-system section Configure Hadoop

2) Hadoop must have Ignite JARS ${IGNITE_HOME}\libs\ignite-core-[version].jar and ${IGNITE_HOME}\libs\hadoop\ignite-hadoop-[version].ja" in CLASSPATH.

Does ignite-hadoop-[version].ja mean ignite-yarn-[version].jar when I use Hadoop v2?

 

 



 

 

Vladimir Ozerov Vladimir Ozerov
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop File System Cache Setting Problems

This post was updated on .
yaoqin,

1) When you configure another IGFS at the very least you must specify it's "metaCacheName" and "dataCacheName" properties. This is not the case for your file system.

2) ClassNotFoundException exception occurs becuase the class "org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem" which is a part of Ignite Hadoop Accelerator is not in your classpath.
If you build Ignite from sources, then you should specify Hadoop edition during build, e.g. "mvn clean install -Prelease -Dignite.edition=hadoop -DskipTests". Once build is finished "ignite-hadoop" artifact with required JAR will appear in your local Maven repository.
Then you should add Ignite Hadoop Accelerator to your project classpath. If you use Maven, then it can be done as follows:

<dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-hadoop</artifactId>
    <version>[your_ignite_version]</version>
</dependency>

3) As per your last question - no. "ignite-hadoop-[version].ja" is a typo for "ignite-hadoop-[version].jar". To have this JAR built you should build Ignite with edition set to "hadoop" as desribed above.

Hope this helps.

Vladimir.