无法通过Java代码访问HDFS

问题描述

我只想通过Java代码访问Hadoop文件系统，但是我似乎不断出现异常

public class hdfsClient {

public hdfsClient() {}

public void addFile(String source, String dest) throws IOException{
    Configuration conf = new Configuration();
    conf.addResource(new Path("/usr/local/hadoop/etc/hadoop/core-site.xml"));
    conf.addResource(new Path("/usr/local/hadoop/etc/hadoop/hdfs-site.xml"));
    FileSystem fs = null;
    try {
        fs = FileSystem.get(conf);
    } catch (Exception e) {
        System.out.println("Error in getting the fileSystem");
        e.printStackTrace();
    }
}

现在主文件是这样的

public class testMain {
public static void main(String[] args) throws Exception{
    // TODO Auto-generated method stub
    hdfsClient client = new hdfsClient();

    if (args[0].equals("add")) {
        if (args.length < 3) {
            System.out.println("Usage: hdfsclient add <local_path> " +
            "<hdfs_path>");
            System.exit(1);
        }

        client.addFile(args[1], args[2]);
    }
}

}

我在eclipse中创建了这些文件并导出为JAR，然后使用

java -jar <jarname> add <path in local system> <path in hadoop>

确切的命令是

java -jar add.jar add /home/aman/test.txt /

我收到以下错误

org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
at org.apache.hadoop.ipc.Client.call(Client.java:1113)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:124)
at crud.crud.hdfsClient.addFile(hdfsClient.java:28)
at crud.crud.testMain.main(testMain.java:16)

我尝试了整整两天的任何帮助，但无法解决任何帮助的问题

PS：从jps输出

16341 Jps
14985 NameNode
20704 -- process information unavailable
15655 NodeManager
15146 DataNode
15349 SecondaryNameNode
15517 ResourceManager

我找到了解决方案，我在pom.xml文件中使用了hadoop核心依赖关系，而hadoop核心是hadoop 1.X软件包的一部分，其余的依赖关系来自hadoop 2.X，因此存在版本冲突。 删除hadoop核心依赖项可以解决此问题。