Issue Description
While setting up BDR (Backup and Disaster Recovery) on 2 remote BDA (Big data appliance) racks. We have managed to finish pre-requisites, added Source BDA as Peer, made the HDFS directory to replicate snapshottable on both sides. Finally, we get a "HDFS replication command failed." error when we run our initial load. We have tested Snapshot creation for the same HDFS directory on both BDAs, and it completed successfully.
- Main Err: "User performing the MapReduce job must be a superuser to preserve user and group permissions: admin"
- Diff-based replication not used "Used diff: false" --> Err: "No snapshottable directories have found. Reason: either run-as-user does not have permissions to get snapshottable directories or source path is not snapshottable
Issue Detail
Default supergroup here is “supergroup” and by default that group is does not exists in the BDA cluster nodes. That can be verified from cloudera manager.
From Cloudera Manager, navigate to HDFS > Configuration > Superuser Group
Solution
We need to create group “supergroup” on all cluster nodes and after that should create a user lets say “admin” and add this user to supergroup.
I have used below set of commands to perform required tasks and commands needs to be executed on the first node of cluster. And this needs to be done on both source and target cluster.
# dcli -C groupadd -g 20020 supergroup
# dcli -C useradd -u 20020 -g hadoop -G hadoop,hive,supergroup -m admin
# dcli -C id admin
# dcli -C passwd -S admin
# hash=$(echo '<hidden>' | openssl passwd -1 -stdin);
# dcli -C "usermod --pass='$hash' admin"
# dcli -C passwd -S admin
# sudo -u hdfs hadoop fs -mkdir /user/admin
# sudo -u hdfs hadoop fs -chown admin:hadoop /user/admin
# sudo -u hdfs hadoop fs -ls -d /user/admin
# dcli -C chmod 755 /home/admin
# dcli -C mkdir -p /home/admin/DATA
# dcli -C ls -ld /home/admin
Creating Replication Schedule
Now while creating replication schedule we have to use the user “admin” for Run as Username and Run on Peer as Username:
Reference
[1] https://support.oracle.com/epmos/main/hadoop/BDR-doAs-Fails-Error-User-performing-the-MapReduce-job-must-be-a-superuser-to-preserve-user-and-group-permissions
[2] https://support.oracle.com/epmos/main/hadoop/Create-HDFS-Superuser-Group
