Why is Namenode stuck in safe mode and how to fix it?

When namenode is started or restarted, namenode will be in safemode for a period of time. At this time you will not be able to use your Hadoop cluster fully. Write operations to HDFS will fail and because of that your MapReduce jobs will also fail.

Why does the namenode get stuck in safemode?

When namenode is in safemode, namenode will bring the FSIMAGE file, which has the metadata of HDFS, to memory. It will wait for all the blocks for all the files in HDFS to be available. More importantly, it will wait for the minimum replication for all blocks to be met and  available.

Let’s say you have a file and the file is divided into 5 blocks and the minimum replication factor for this file is 2. This means the file is in good “health” as long as each block of the file is replicated twice in the cluster. If one of the blocks in the file is only replicated once in the cluster then the minimum replication factor for this file is not met. This also means the file is not in good health. Namenode will stay in safemode if the minimum replication factor is not met for more number of blocks.

Why is the minimum replication factor not met for certain blocks?

There are many reasons for the minimum replication factor not met for certain blocks. Here are few reasons –

• Some nodes in the cluster could be down which makes the blocks on the nodes not available.
• Some storage devices on the nodes could be down which makes the blocks on the nodes not available
• Some blocks could be corrupted

How do I check the health of HDFS?

Hadoop fsck (file system check) command is a great to inspect the health of the filesystem.

hdfs fsck /  will give you a report like below which will help you check the health of the cluster and give you a percentage of under replicated blocks and a count of missing replicas.

hdfs fsck / 

…..
…..
…..

Total size:    1943153298 B (Total open files size: 3422 B)
 Total dirs:    137
 Total files:   830
 Total symlinks:                0 (Files currently being written: 2)
 Total blocks (validated):      819 (avg. block size 2372592 B) (Total open file blocks (not validated): 2)
  ********************************
  UNDER MIN REPL'D BLOCKS:      1 (0.12210012 %)
  dfs.namenode.replication.min: 1
  CORRUPT FILES:        1
  MISSING BLOCKS:       1
  MISSING SIZE:         28160 B
  CORRUPT BLOCKS:       1
  ********************************

 Minimally replicated blocks:   818 (99.8779 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       818 (99.8779 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     1.997558
 Corrupt blocks:                1
 Missing replicas:              818 (33.292633 %)
 Number of data-nodes:          2
 Number of racks:               1

FSCK ended at Tue Nov 10 15:51:30 UTC 2020 in 366 milliseconds

How to identify corrupted file/blocks in HDFS?

hdfs fsck -list-corruptfileblocks

will list the files with corrupted blocks

[hdfs@wk1 osboxes]$ hdfs fsck -list-corruptfileblocks

Connecting to namenode via http://ms1.hirw.com:50070/fsck?ugi=hdfs&listcorruptfileblocks=1&path=%2F
The list of corrupt files under path '/' are:
blk_1073742622  /ranger/audit/hdfs/20200813/hdfs_ranger_audit_ms1.hirw.com.log
The filesystem under path '/' has 1 CORRUPT files

Once you identify the corrupted blocks you can remove them with hdfs dfs -rm command. But a write operation like delete is not possible when the namenode is in safemode so we need to come out of safemode first.

How to forcefully come out of safemode?

hdfs dfsadmin -safemode leave will forcefully make namenode to leave safemode.

hdfs dfsadmin -safemode leave

Delete the corrupted file once you are out of the safemode so you don’t run in to this problem again after restarting namenode.

Lower the threshold of under replicated blocks

dfs.namenode.safemode.threshold-pct  specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.namenode.replication.min . Values less than or equal to 0 mean not to wait for any particular percentage of blocks before exiting safemode. 

dfs.safemode.threshold.pct  is the corresponding property in older versions of Hadoop

<configuration> 
  <property> 
    <name>dfs.namenode.safemode.threshold-pct</name> 
    <value>0</value> 
  </property> 
</configuration>

A word of caution

We don’t recommend changing this option to 0 as this will hide the underlying problem of corrupted blocks and could result in lot of blocks not meeting the minimum replication number leaving your files in the filesystem vulnerable.