This post is somewhat dated material. Several years back, when YARN was first making headways and vendors starting adopting it as part of Hadoop 2.x, there were many times where I needed to downgrade to MapReduce v1. I had written a lot of stuff for MRV1, and there were times where downgrading was the best approach to getting things back up and running. For those that may need to, here are my notes for downgrading from YARN to MRv1:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 |
# remove YARN configuration sudo yum remove hadoop-conf-pseudo # stop YARN sudo service hadoop-yarn-resourcemanager stop sudo service hadoop-yarn-nodemanager stop sudo service hadoop-mapreduce-historyserver stop # stop HDFS sudo for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x stop ; done # Install MRv1 sudo yum install hadoop-0.20-conf-pseudo # Remove cache dir sudo rm -rf /var/lib/hadoop-hdfs/cache/ # format namenode sudo -u hdfs hdfs namenode -format # start HDFS sudo for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done # make /tmp directories and set permissions/ownership sudo -u hdfs hadoop fs -mkdir /tmp sudo -u hdfs hadoop fs -chmod -R 1777 /tmp sudo -u hdfs hadoop fs -mkdir -p /var/lib/hadoop-hdfs/cache/mapred/mapred/staging sudo -u hdfs hadoop fs -chmod 1777 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging sudo -u hdfs hadoop fs -chown -R mapred /var/lib/hadoop-hdfs/cache/mapred sudo -u hdfs mkdir -p /var/lib/hadoop-hdfs/cache/mapred/mapred/local/ sudo chown -R mapred /var/lib/hadoop-hdfs/cache/mapred # check dir structure sudo -u hdfs hadoop fs -ls -R / # start MRv1 for x in `cd /etc/init.d ; ls hadoop-0.20-mapreduce-*` ; do sudo service $x start ; done # make user directory for your <user> sudo -u hdfs hadoop fs -mkdir /user/cloudera sudo -u hdfs hadoop fs -chown cloudera /user/cloudera # test hadoop fs -mkdir input hadoop fs -put /etc/hadoop/conf/*.xml input hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar grep input output 'dfs[a-z.]+' |
all should be good on localhost:50030 and 50070
