hbase 进行 javaAPI 操作时报了如下错误 :
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=32, exceptions:
Tue Dec 17 10:21:47 CST 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68473: row 'myUser,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hadoop01,60020,1576455547324, seqNum=0
Caused by: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
报错详情如下:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=32, exceptions:
Tue Dec 17 10:21:47 CST 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68473: row 'myUser,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hadoop01,60020,1576455547324, seqNum=0at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:320)at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:247)at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:62)at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:862)at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:421)at com.czxy.HBaseAPI.createTable(HBaseAPI.java:40)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)at org.junit.runners.ParentRunner.run(ParentRunner.java:363)at org.junit.runner.JUnitCore.run(JUnitCore.java:137)at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68473: row 'myUser,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hadoop01,60020,1576455547324, seqNum=0at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:169)at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yetat org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1099)at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2311)at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33648)at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332)at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:408)at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:204)at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:65)at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:397)at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:371)at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)... 4 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.ServerNotRunningYetException): org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yetat org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1099)at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2311)at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33648)at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272)at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:400)... 10 moreProcess finished with exit code -1问题排查 :
1.一开是看到如下这段
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=32, exceptions:
Tue Dec 17 10:21:47 CST 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68473: row 'myUser,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hadoop01,60020,1576455547324, seqNum=0
去百度找,百度上都说是hosts文件的文件,但经过实际排查却发现跟hosts文件没一点关系
2.后来看到了 Server is not running yet 这个错,发现确实是集群处于安全模式,不知道什么时候namenode掉了,然后重启了一下集群,并查看集群状态直到集群退出了安全模式再次运行,发现还是报错.
3.经过排查,发现还需要重启hbase(保险起见顺便把zookeeper一起重启了)
然后问题解决.
报错原因:
集群处于安全模式
解决 :
1.查看集群状态
hdfs dfsadmin -safemode get
//如果返回Safe mode is OFF 就说明没问题
//如果返回Safe mode is ON 就说明集群正处于安全模式(强制退出即可)
hdfs dfsadmin -safemode leave//如果返回其他就说明集群出问题了 例如 返回如下内容
safemode: Call From hadoop01/192.168.100.100 to hadoop01:8020 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused检查之后发现是namenode掉了,重启了一下集群.建议执行完上述的操作之后重启hbase