ClassNotFoundException when using IgniteHadoopIgfsSecondaryFileSystem

classic Classic list List threaded Threaded
7 messages Options
Juan Rodríguez Hortalá Juan Rodríguez Hortalá
Reply | Threaded
Open this post in threaded view
|

ClassNotFoundException when using IgniteHadoopIgfsSecondaryFileSystem

Hi, 

I'm trying to use IgniteHadoopIgfsSecondaryFileSystem on EMR, and I have been able to make this work using HDFS in the uri. But when I use a different scheme for another file system that is working ok with `hadoop fs` and other applications, I get 

```
[hadoop@ip-10-0-0-242 ~]$  hadoop fs -ls igfs://[hidden email]:10500/
ls: Generic IGFS error occurred.
````

and then in the node ip-10-0-0-85.ec2.internal I see in the logs, at the same time the request is made, and exception `class org.apache.ignite.IgniteException: java.lang.ClassNotFoundException` and `Class not found` for the class implementing Hadoop FileSystem that is extending org.apache.hadoop.fs.FileSystem that is associated to that scheme in core-site.xml. 

Before I added the path to core-site.xml in configPaths for CachingHadoopFileSystemFactory I was getting "org.apache.ignite.IgniteException: No FileSystem for scheme". I have tried running setup-hadoop.sh  in that slave, and that creates some symlinks, but I have the same error. The weird thing here is that when I check the node config with ignite visor I see the jar that contains this class listed in java.class.path. 

Any idea how might I keep debugging this?

Thanks a lot in advance. 

Juan 
Alexey Kukushkin Alexey Kukushkin
Reply | Threaded
Open this post in threaded view
|

Re: ClassNotFoundException when using IgniteHadoopIgfsSecondaryFileSystem

Hi,

I do not see any other reasons for a "class not found" exception but the class is not on the CLASSPATH. Can you make sure "ignite-core-xxx.jar" and "ignite-hadoop-xxx.jar" are on the hadoop's CLASSPATH. 
Juan Rodríguez Hortalá Juan Rodríguez Hortalá
Reply | Threaded
Open this post in threaded view
|

Re: ClassNotFoundException when using IgniteHadoopIgfsSecondaryFileSystem

Hi, 

Thanks for your answer. I have both libs/ignite-core and libs/ignite-hadoop/ignite-hadoop, and also the jar for the other hadoop file system, in the list of jars visible on the java.class.path section that is returned by the config command of the Ignite Visor. Is the order of the jars relevant here? Are jars expected in any particular order? I also added the jars for ignite-core, ignite-hadoop and ignite-shmem to HADOOP_CLASSPATH in /etc/hadoop/conf/hadoop-env.sh. 

This is working with <property name="uri" value="hdfs://ip-10-0-0-242.ec2.internal:8020/"/>, but it fails when I use another uri with the scheme for the other hadoop file system. So I think the problem is not finding the ignite jars, but the jars for the other hadoop file system. For launching the IGFS daemons I use: 

```
source /etc/default/hadoop # gets HADOOP_HOME
source /etc/hadoop/conf/hadoop-env.sh # gets IGNITE_HOME and HADOOP_CLASSPATH
export USER_LIBS=${HADOOP_CLASSPATH}
${IGNITE_HOME}/bin/ignite.sh ignite-conf.xml
```

I'm not really sure about the role of ${USER_LIBS} here. Any other ideas? Maybe some pointer to some tutorial about using IgniteHadoopIgfsSecondaryFileSystem with a hadoop compatible file system different to HDFS?

Thanks again, 

Juan 


On Wed, Dec 6, 2017 at 3:24 AM, Alexey Kukushkin <[hidden email]> wrote:
Hi,

I do not see any other reasons for a "class not found" exception but the class is not on the CLASSPATH. Can you make sure "ignite-core-xxx.jar" and "ignite-hadoop-xxx.jar" are on the hadoop's CLASSPATH. 

Alexey Kukushkin Alexey Kukushkin
Reply | Threaded
Open this post in threaded view
|

Re: ClassNotFoundException when using IgniteHadoopIgfsSecondaryFileSystem

I just replied you on another thread with details on what you need to do to connect to a remote IGFS cluster. You still need Ignite configuration and Ignite-Hadoop dependencies on your spark nodes to connect to remote IGFS. You also need to add some Hadoop properties to make it find IGFS configuration.
Juan Rodríguez Hortalá Juan Rodríguez Hortalá
Reply | Threaded
Open this post in threaded view
|

Re: ClassNotFoundException when using IgniteHadoopIgfsSecondaryFileSystem

Hi Alexey, 

Thanks for your answers. I was finally able to fix this, I was missing some additional jars in the lib directory of the ignite installation in the worker nodes. The exception thrown was by org.apache.hadoop.conf.Configuration.getClassByName was a bit misleading because it was referring to the file system class, while the class that was missing in the classpath was a different one. It is working fine now. 

Thanks a lot for your help!

On Thu, Dec 7, 2017 at 3:24 AM, Alexey Kukushkin <[hidden email]> wrote:
I just replied you on another thread with details on what you need to do to connect to a remote IGFS cluster. You still need Ignite configuration and Ignite-Hadoop dependencies on your spark nodes to connect to remote IGFS. You also need to add some Hadoop properties to make it find IGFS configuration.

mehdi sey mehdi sey
Reply | Threaded
Open this post in threaded view
|

Re: ClassNotFoundException when using IgniteHadoopIgfsSecondaryFileSystem

In reply to this post by Alexey Kukushkin
hi. i want to start ignite node with a configuration name as
example-igfs.xml. i have alter this configuration for using IGFS as cache
layer for HDFS. but when i execute the below command  for start ignite node
i encounter with error:
/usr/local/apache-ignite-fabric-2.6.0-bin/bin/ignite.sh
/usr/local/apache-ignite-fabric-2.6.0-bin/examples/config/filesystem/example-igfs.xml

but after executing above command i will encounter below error:
/  _/ ___/ |/ /  _/_  __/ __/
[09:57:48]  _/ // (7 7    // /  / / / _/  
[09:57:48] /___/\___/_/|_/___/ /_/ /___/  
[09:57:48]
[09:57:48] ver. 2.6.0#20180710-sha1:669feacc
[09:57:48] 2018 Copyright(C) Apache Software Foundation
[09:57:48]
[09:57:48] Ignite documentation: http://ignite.apache.org
[09:57:48]
[09:57:48] Quiet mode.
[09:57:48]   ^-- Logging to file
'/usr/local/apache-ignite-fabric-2.6.0-bin/work/log/ignite-246509e8.0.log'
[09:57:48]   ^-- Logging by 'JavaLogger [quiet=true, config=null]'
[09:57:48]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[09:57:48]
[09:57:48] OS: Linux 4.15.0-43-generic amd64
[09:57:48] VM information: Java(TM) SE Runtime Environment 1.8.0_192-ea-b04
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.192-b04
[09:57:48] Configured plugins:
[09:57:48]   ^-- None
[09:57:48]
[09:57:48] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0]]
[09:57:48] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
[09:57:48] Security status [authentication=off, tls/ssl=off]
[09:57:49,412][SEVERE][main][IgniteKernal] Exception during start
processors, node will be stopped and close connections
java.lang.NoClassDefFoundError: com/google/common/base/Preconditions
        at
org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:361)
        at
org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:374)
        at
org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:456)
        at
org.apache.ignite.internal.processors.hadoop.impl.HadoopUtils.safeCreateConfiguration(HadoopUtils.java:334)
        at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopBasicFileSystemFactoryDelegate.start(HadoopBasicFileSystemFactoryDelegate.java:129)
        at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopCachingFileSystemFactoryDelegate.start(HadoopCachingFileSystemFactoryDelegate.java:58)
        at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopIgfsSecondaryFileSystemDelegateImpl.start(HadoopIgfsSecondaryFileSystemDelegateImpl.java:413)
        at
org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem.start(IgniteHadoopIgfsSecondaryFileSystem.java:276)
        at
org.apache.ignite.internal.processors.igfs.IgfsImpl.<init>(IgfsImpl.java:185)
        at
org.apache.ignite.internal.processors.igfs.IgfsContext.<init>(IgfsContext.java:102)
        at
org.apache.ignite.internal.processors.igfs.IgfsProcessor.start(IgfsProcessor.java:116)
        at
org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1739)
        at
org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:990)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
        at
org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
        at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1069)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:955)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:854)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:724)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:693)
        at org.apache.ignite.Ignition.start(Ignition.java:352)
        at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
Caused by: java.lang.ClassNotFoundException:
com.google.common.base.Preconditions
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 23 more
[09:57:49,414][SEVERE][main][IgniteKernal] Got exception while starting
(will rollback startup routine).
java.lang.NoClassDefFoundError: com/google/common/base/Preconditions
        at
org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:361)
        at
org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:374)
        at
org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:456)
        at
org.apache.ignite.internal.processors.hadoop.impl.HadoopUtils.safeCreateConfiguration(HadoopUtils.java:334)
        at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopBasicFileSystemFactoryDelegate.start(HadoopBasicFileSystemFactoryDelegate.java:129)
        at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopCachingFileSystemFactoryDelegate.start(HadoopCachingFileSystemFactoryDelegate.java:58)
        at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopIgfsSecondaryFileSystemDelegateImpl.start(HadoopIgfsSecondaryFileSystemDelegateImpl.java:413)
        at
org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem.start(IgniteHadoopIgfsSecondaryFileSystem.java:276)
        at
org.apache.ignite.internal.processors.igfs.IgfsImpl.<init>(IgfsImpl.java:185)
        at
org.apache.ignite.internal.processors.igfs.IgfsContext.<init>(IgfsContext.java:102)
        at
org.apache.ignite.internal.processors.igfs.IgfsProcessor.start(IgfsProcessor.java:116)
        at
org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1739)
        at
org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:990)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
        at
org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
        at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1069)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:955)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:854)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:724)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:693)
        at org.apache.ignite.Ignition.start(Ignition.java:352)
        at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
Caused by: java.lang.ClassNotFoundException:
com.google.common.base.Preconditions
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 23 more
[09:57:49] Ignite node stopped OK [uptime=00:00:01.862]
java.lang.NoClassDefFoundError: com/google/common/base/Preconditions
        at
org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:361)
        at
org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:374)
        at
org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:456)
        at
org.apache.ignite.internal.processors.hadoop.impl.HadoopUtils.safeCreateConfiguration(HadoopUtils.java:334)
        at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopBasicFileSystemFactoryDelegate.start(HadoopBasicFileSystemFactoryDelegate.java:129)
        at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopCachingFileSystemFactoryDelegate.start(HadoopCachingFileSystemFactoryDelegate.java:58)
        at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopIgfsSecondaryFileSystemDelegateImpl.start(HadoopIgfsSecondaryFileSystemDelegateImpl.java:413)
        at
org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem.start(IgniteHadoopIgfsSecondaryFileSystem.java:276)
        at
org.apache.ignite.internal.processors.igfs.IgfsImpl.<init>(IgfsImpl.java:185)
        at
org.apache.ignite.internal.processors.igfs.IgfsContext.<init>(IgfsContext.java:102)
        at
org.apache.ignite.internal.processors.igfs.IgfsProcessor.start(IgfsProcessor.java:116)
        at
org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1739)
        at
org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:990)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
        at
org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
        at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1069)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:955)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:854)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:724)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:693)
        at org.apache.ignite.Ignition.start(Ignition.java:352)
        at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
Caused by: java.lang.ClassNotFoundException:
com.google.common.base.Preconditions
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 23 more
Failed to start grid: com/google/common/base/Preconditions
Note! You may use 'USER_LIBS' environment variable to specify your
classpath.


==================================
this is my example-igfs file configuration:

<?xml version="1.0" encoding="UTF-8"?>






<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd">





   
    <description>
        Spring file for ignite configuration with client available
endpoints.
    </description>

   
    <bean id="propertyConfigurer"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="systemPropertiesModeName"
value="SYSTEM_PROPERTIES_MODE_FALLBACK"/>
        <property name="searchSystemEnvironment" value="true"/>
    </bean>

   
    <bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
       

       
           
       
        <property name="connectorConfiguration">
            <bean
class="org.apache.ignite.configuration.ConnectorConfiguration">
                <property name="port" value="11211"/>
            </bean>
        </property>
         
        <property name="fileSystemConfiguration">
            <list>
                <bean
class="org.apache.ignite.configuration.FileSystemConfiguration">
                    <property name="name" value="igfs"/>

                   
                    <property name="blockSize" value="#{128 * 1024}"/>
                    <property name="perNodeBatchSize" value="512"/>
                    <property name="perNodeParallelBatchCount" value="16"/>

                   
                    <property name="prefetchBlocks" value="32"/>

                   

                   
                   
                   
                   
                    <property name="secondaryFileSystem">
                        <bean
class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
                            <property name="fileSystemFactory">
                                <bean
class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
                                    <property name="uri"
value="hdfs://localhost:9000/"/>
                                   
                                </bean>
                            </property>
                        </bean>
                    </property>

                     
                   
                   

                   
                   
                    <property name="ipcEndpointConfiguration">
                        <bean
class="org.apache.ignite.igfs.IgfsIpcEndpointConfiguration">
                            <property name="type" value="SHMEM" />
                            <property name="host" value="127.0.0.1" />
                            <property name="port" value="10500" />
                        </bean>
                    </property>
                   
                </bean>
            </list>
        </property>

       
        <property name="discoverySpi">
            <bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                   
                   
                   
                    <bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
                        <property name="addresses">
                            <list>
                               
                                <value>127.0.0.1:47500..47509</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</beans>



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: ClassNotFoundException when using IgniteHadoopIgfsSecondaryFileSystem

Hello!

Looks like there is some dependency problem in hadoop modules. Do you have
steps to reproduce?

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/