hadoop 问题

Last updated on May 23, 2025 am

🧙 Questions

Inconsistent checkpoint fields
Cannot set priority of namenode process
Failed to start namenode
return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
Permission denied: user=ispong, access=WRITE
Incompatible clusterIDs in /data/hadoop/data

☄️ Ideas

Incompatible clusterIDs in /data/hadoop/data

java.io.IOException: Incompatible clusterIDs in /data/hadoop/data: namenode clusterID = CID-5cdee252-fcdc-46bc-9f6d-3c49954be037; datanode clusterID = CID-24a49d8e-00c2-4694-8777-09624b048260
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:744)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:294)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:407)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:387)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:559)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1748)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1684)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:392)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:282)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829)
        at java.lang.Thread.run(Thread.java:750)
2022-09-13 11:15:06,755 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid 53e083ca-eaca-43dc-afa1-4fd74c4bf011) service to isxcode/172.26.34.191:9000. Exiting. 
java.io.IOException: All specified directories have failed to load.
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:560)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1748)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1684)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:392)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:282)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829)
        at java.lang.Thread.run(Thread.java:750)
2022-09-13 11:15:06,755 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid 53e083ca-eaca-43dc-afa1-4fd74c4bf011) service to isxcode/172.26.34.191:9000

解决方案

# 删除冲突/data/hadoop/data中冲突住主键\
rm -rf /data/hadoop/data/
hdfs namenode -format

Inconsistent checkpoint fields

2021-01-20 15:44:23,768 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint
java.io.IOException: Inconsistent checkpoint fields.
LV = -65 namespaceID = 1632612953 cTime = 1609239411999 ; clusterId = CID-6685ddb6-3f79-4376-a6fa-121a575f2c25 ; blockpoolId = BP-1401685203-172.23.39.204-1609239411999.
Expecting respectively: -65; 126596671; 1608190733065; CID-6685ddb6-3f79-4376-a6fa-121a575f2c25; BP-2042470031-172.23.39.204-1608190733065.
        at org.apache.hadoop.hdfs.server.namenode.CheckpointSignature.validateStorageInfo(CheckpointSignature.java:143)
        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:550)
        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:360)
        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:325)
        at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:485)
        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:321)
        at java.lang.Thread.run(Thread.java:748)

解决方案

sudo vim hadoop/dfs/name/current/VERSION
sudo vim hadoop/dfs/namesecondary/current/VERSION

VERSION文件中的值全部保持一直
namespaceID、clusterID、blockpoolID、layoutVersion、cTime

Cannot set priority of namenode process

Starting namenodes on [0.0.0.0]
0.0.0.0: ERROR: Cannot set priority of namenode process 28593
Starting datanodes
Starting secondary namenodes [isxcode]
Starting resourcemanager
Starting nodemanagers

解决方案

jps
kill -9 ${hadoop process}

虽然执行了 stop-all.sh, 但可能并未执行成功
找出所有的hadoop进程，强行关闭掉，然后重新启动start-all.sh

Failed to start namenode

2021-01-20 16:55:34,761 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: Gap in transactions. Expected to be able to read up until at least txid 84026 but unable to find any edit logs containing txid 83992
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.checkForGaps(FSEditLog.java:1786)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1744)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1719)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:716)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:338)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1197)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:779)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:673)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:760)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1014)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:987)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1756)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1821)
2021-01-20 16:55:34,762 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.io.IOException: Gap in transactions. Expected to be able to read up until at least txid 84026 but unable to find any edit logs containing txid 83992

解决方案

hadoop namenode -recover

重置namenode组件

return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

2021-01-20 17:33:32 ERROR:Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
2021-01-20 17:33:32 ERROR Current task status: ERROR

解决方案

export HADOOP_CLIENT_OPTS="-Xmx1024m $HADOOP_CLIENT_OPTS"

由于hadoop主机内存分配不足，可以尝试扩大hadoop的JVM内存

datanode与namenode 集群id不同

java.io.IOException: Incompatible clusterIDs in /data/hadoop/data: namenode clusterID = CID-20c2b620-23bb-465c-85b2-67d1d9a246da; datanode clusterID = CID-3d016f91-b337-4b7e-84b4-c15f6d59a286
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:744)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:294)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:407)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:387)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:559)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1748)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1684)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:392)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:288)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:830)
        at java.lang.Thread.run(Thread.java:750)

cat /data/hadoop/name/current/VERSION
clusterID=CID-20c2b620-23bb-465c-85b2-67d1d9a246da

vim /data/hadoop/data/current/VERSION
# 集群id保持一致
clusterID=CID-20c2b620-23bb-465c-85b2-67d1d9a246da

java.lang.VerifyError: Bad return type location:org/apache/hadoop/hdfs/DFSClient.getQuotaUsage

hadoop版本依赖不

🔗 Links

hadoop website

hadoop

hadoop 问题

https://ispong.isxcode.com/hadoop/hadoop/hadoop 问题/

Author

ispong

Posted on

April 21, 2021

Licensed under

docker实战开启远程服务 Previous

java 安装 Next