[研究] Hadoop 2.2.0 編譯 (CentOS 6.4 x64)

[研究] Hadoop 2.2.0 編譯 (CentOS 6.4 x64)

2013-11-06

Hadoop 是個架設雲端的系統，它參考Google Filesystem，以Java開發，提供HDFS與MapReduce API。

官方網站
http://hadoop.apache.org/

安裝參考
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0/BUILDING.txt

為何要自己辛苦編譯，後面會解釋。

基本需求

* Unix System
* JDK 1.6+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

# 安裝基本套件
[root@localhost ~]# su root

[root@localhost ~]# yum -y install gcc  gcc-c++  svn  cmake git zlib zlib-devel openssl openssl-devel rsync
或
[root@localhost ~]# yum -y groupinstall 'Development Tools'
[root@localhost ~]# yum -y install cmake zlib-devel openssl openssl-devel rsync

# 安裝 JDK ( Java Development Kit 7 Update 45 )，Maven 需要
# 手動下載 jdk-7u45-linux-x64.rpm
# http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html#javasejdk

[root@localhost ~]# rpm -ivh  jdk-7u45-linux-x64.rpm

# 目前預設是 OpenJDK，敝人把預設換成 Oracle JDK

[root@localhost ~]# java -version
java version "1.7.0_09-icedtea"
OpenJDK Runtime Environment (rhel-2.3.4.1.el6_3-x86_64)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

[root@localhost ~]# alternatives --install /usr/bin/java java /usr/java/jdk1.7.0_45/jre/bin/java 100

[root@localhost ~]# alternatives --config java

There are 3 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
   2           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java
   3           /usr/java/jdk1.7.0_45/jre/bin/java

Enter to keep the current selection[+], or type selection number: 3

[root@localhost ~]# export JAVA_HOME=/usr/java/jdk1.7.0_45

[root@localhost ~]# java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
[root@localhost ~]#

# 安裝 Apache Maven 3.1.1
cd  /usr/local/src
wget http://ftp.tc.edu.tw/pub/Apache/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.tar.gz
tar zxvf apache-maven-3.1.1-bin.tar.gz -C /usr/local
ln  -s  /usr/local/apache-maven-3.1.1/bin/mvn  /usr/bin/mvn

# 安裝 FindBugs
cd  /usr/local/src
wget http://prdownloads.sourceforge.net/findbugs/findbugs-2.0.2.tar.gz?download
tar zxvf findbugs-2.0.2.tar.gz -C /usr/local/
ln -s /usr/local/findbugs-2.0.2/bin/findbugs  /usr/bin/findbugs

# 安裝 Protoc 2.5.0
cd  /usr/local/src
wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
tar zxvf protobuf-2.5.0.tar.gz -C /usr/local/src
cd /usr/local/src/protobuf-2.5.0
./configure
make
make install
ln -s /usr/local/bin/protoc /usr/bin/protoc

# 編譯 Hadoop 2.2.0

cd  /usr/local/src
wget http://ftp.mirror.tw/pub/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
tar zxvf hadoop-2.2.0-src.tar.gz -C /usr/local/src
cd  /usr/local/src/hadoop-2.2.0-src/

[root@localhost ~]# mvn package -Pdist,native -DskipTests -Dtar
...(略，mvn package 花費時間和網路頻寬、電腦快慢有關，十幾分鐘到超過半小時都遇過)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31:08.714s
[INFO] Finished at: Wed Nov 06 16:32:30 CST 2013
[INFO] Final Memory: 104M/367M
[INFO] ------------------------------------------------------------------------


*****************************************************************************

自己編譯 hadoop 有甚麼用途嗎？
官方提供下載的 hadoop-2.2.0.tar.gz 就是已經編譯好的

問題出在這裡

[root@localhost src]# cd /usr/local/src
[root@localhost src]# wget http://ftp.mirror.tw/pub/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz
[root@localhost src]# tar zxvf hadoop-2.2.0.tar.gz -C /usr/local
[root@localhost src]# cd /usr/local/hadoop-2.2.0/lib/native
[root@localhost native]# file *
libhadoop.a:        current ar archive
libhadooppipes.a:   current ar archive
libhadoop.so:       symbolic link to `libhadoop.so.1.0.0'
libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
libhadooputils.a:   current ar archive
libhdfs.a:          current ar archive
libhdfs.so:         symbolic link to `libhdfs.so.0.0.0'
libhdfs.so.0.0.0:   ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped

hadoop-2.2.0.tar.gz 提供的某些 Library 是 32-bit 的，如果在 x86_64 的作業系統上跑，會出現

Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /home/hadoop/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

不過訊息說這只是 "WARN" (警告)，不是 ERROR、FAIL、FAILURE 等級的問題
是否真的不會產生甚麼大問題，不知道....

來看一下在 x86_64 作業系統上編譯出來的

[root@localhost native]# cd /usr/local/src/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/lib/native
[root@hadoop1 native]# file *
libhadoop.a:        current ar archive
libhadooppipes.a:   current ar archive
libhadoop.so:       symbolic link to `libhadoop.so.1.0.0'
libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
libhadooputils.a:   current ar archive
libhdfs.a:          current ar archive
libhdfs.so:         symbolic link to `libhdfs.so.0.0.0'
libhdfs.so.0.0.0:   ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

如果你使用 x86_64 作業系統，可考慮把 x86 的另外備份，x86_64 版的拷貝過來覆蓋掉 x86 版的

[root@localhost src]# cd /usr/local/hadoop-2.2.0/lib/native
[root@localhost native]# cp libhadoop.so.1.0.0  libhadoop.so.1.0.0.x86
[root@localhost native]# cp libhdfs.so.0.0.0  libhdfs.so.0.0.0.x86

[root@localhost native]# cd /usr/local/src/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/lib/native

[root@localhost native]# cp libhadoop.so.1.0.0  /usr/local/hadoop-2.2.0/lib/native
cp: overwrite `/usr/local/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0'? y

[root@localhost native]# cp -f  libhdfs.so.0.0.0    /usr/local/hadoop-2.2.0/lib/native
cp: overwrite `/usr/local/hadoop-2.2.0/lib/native/libhdfs.so.0.0.0'? y

為了節省大家時間，我把 x86_64 版的 native 目錄整個壓縮成 hadoop-2.2.0-native_x86_64.tar.gz，找了免註冊、免費空間放上去

http://www.sendspace.com/file/5rm0j1

https://folders.io/get/98YgDc

************************

後來根據
[研究] hadoop-2.2.0.tar.gz 有哪些檔案是 32-bit 專用的
http://shaurong.blogspot.tw/2013/11/hadoop-220targz-32-bit.html

又做了一份 hadoop-2.2.0-x86-x86_64.tar.gz
以 64-bit 為主，但是保留 x86 版本4個檔案，檔名添加了 .x86 作為識別，必要時候拷貝覆蓋過去使用。

(完)


相關

[研究] Hadoop 2.2.0 編譯 (CentOS 6.4 x64)
http://shaurong.blogspot.tw/2013/11/hadoop-220-centos-64-x64.html

[研究] Hadoop 2.2.0 Single Cluster 安裝 (二)(CentOS 6.4 x64)
http://shaurong.blogspot.tw/2013/11/hadoop-220-single-cluster-centos-64-x64_7.html

[研究] Hadoop 2.2.0 Single Cluster 安裝 (一)(CentOS 6.4 x64)
http://shaurong.blogspot.tw/2013/11/hadoop-220-single-cluster-centos-64-x64.html

[研究] Hadoop 1.2.1 (rpm)安裝 (CentOS 6.4 x64)
http://shaurong.blogspot.tw/2013/10/hadoop-121-rpm-centos-64-x64.html

[研究] Hadoop 1.2.1 (bin)安裝 (CentOS 6.4 x64)
http://shaurong.blogspot.tw/2013/07/hadoop-112-centos-64-x64.html

[研究] Hadoop 1.2.1 安裝 (CentOS 6.4 x64)
http://forum.icst.org.tw/phpbb/viewtopic.php?t=80035

[研究] 雲端軟體 Hadoop 1.0.0 安裝 (CentOS 6.2 x86)
http://forum.icst.org.tw/phpbb/viewtopic.php?t=21166

[研究] 雲端軟體 Hadoop 0.20.2 安裝 (CentOS 5.5 x86)
http://forum.icst.org.tw/phpbb/viewtopic.php?t=18513

[研究] 雲端軟體 Hadoop 0.20.2 安裝 (CentOS 5.4 x86)
http://forum.icst.org.tw/phpbb/viewtopic.php?t=17974