2015-05-23
官方網站
https://spark.apache.org/
需要套件
CentOS
JDK >=6
Maven >= 3.0.4
Hadoop
Spark
Hive
MySQL-Server
安裝參考
https://spark.apache.org/docs/latest/building-spark.html
安裝
# 為了省事,避免意外的情況,關閉 SELinux (Security Linux ) 和 iptables
# 立刻關閉 SELinuxsetenforce 0
# 設定 reboot 後自動關閉 SELinux#vi /etc/selinux/config#找到#SELINUX=#設為#SELINUX=disabled
sed -i -e "s@SELINUX=enforcing@#SELINUX=enforcing@" /etc/selinux/config
sed -i -e "s@SELINUX=permissive@#SELINUX=permissive@" /etc/selinux/config
sed -i -e "/SELINUX=/aSELINUX=disabled" /etc/selinux/config
# 立刻停掉 iptables#service iptables stop #service ip6tables stop systemctl stop firewalld # 設定 reboot 後自動關閉 iptable
#chkconfig iptables off #chkconfig ip6tables off systemctl disable firewalld
cd /usr/local
# 安裝基本套件
yum -y install java java-devel maven mariadb-server mysql
# 設定 Java 環境
#find / -name java
echo 'export JAVA_HOME=/usr/lib/jvm/java' >> /etc/profile
echo 'export PATH=$PATH:$JAVA_HOME/bin' >> /etc/profile
echo 'export CLASSPATH=$JAVA_HOME/lib/ext:$JAVA_HOME/lib/tools.jar' >> /etc/profile
source /etc/profile
# 安裝 Apache Hadoop
cd /usr/local
wget http://ftp.twaren.net/Unix/Web/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz
tar zxvf hadoop-2.5.2.tar.gz
echo 'export HADOOP_HOME=/usr/local/hadoop-2.5.2' >> /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/bin' >> /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/sbin' >> /etc/profile
echo 'export HADOOP_PREFIX=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_COMMON_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_MAPRED_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop' >> /etc/profile
echo 'export HADOOP_HDFS_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_YARN_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export YARN_CONF_DIR=$HADOOP_CONF_DIR' >> /etc/profile
source /etc/profile
# 安裝 Apache Hive,目前最新 1.2.0 版
cd /usr/local
wget http://apache.stu.edu.tw/hive/stable/apache-hive-1.2.0-bin.tar.gz
tar xzvf apache-hive-1.2.0-bin.tar.gz
export HIVE_HOME=/usr/local/apache-hive-1.2.0-bin
export PATH=$HIVE_HOME/bin:$PATH
# 安裝 Apache Spark
# 照官方資訊說不保證 Hadoop >= 2.5 可以正常,試試看吧,mvn 安裝在 Core i5 + SSD 上約1小時
# https://spark.apache.org/docs/latest/building-spark.html
# build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
#wget http://www.apache.org/dyn/closer.cgi/spark/spark-1.3.1/spark-1.3.1.tgz
# Apache 網站這份有問題,別抓cd /usr/local
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.3.1.tgz
tar zxvf spark-1.3.1.tgz
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
cd spark-1.3.1
# 注意 Hadoop 2.5.x 和 2.4.x 都是設定 2.4
#mvn -Pyarn -Dyarn.version=2.4.0 -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -Phive-1.2.0 -Phive-thriftserver -DskipTests clean package
mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:04:14.172s
[INFO] Finished at: Sun May 24 08:48:42 CST 2015
[INFO] Final Memory: 90M/607M
[INFO] ------------------------------------------------------------------------
[root@localhost spark-1.3.1]#
相關文章
[研究] Apache Spark 1.3.1 + Hadoop 2.5.2 安裝
http://shaurong.blogspot.com/2015/05/apache-spark-131-hadoop-252.html
[研究] Apache Maven 3.0.5 (yum) 安裝 (CentOS 7.1 x64_86)
http://shaurong.blogspot.com/2015/05/apache-maven-305-yum-centos-71-x6486.html
[研究] Hadoop 2.7.0 Single Cluster 安裝 (CentOS 7.1 x86_64)
http://shaurong.blogspot.com/2015/05/hadoop-270-single-cluster-centos-71.html
Spark-1.3.1與Hive整合實現查詢分析
http://wrox.cn/article/1036050/
沒有留言:
張貼留言