[研究] Storm 0.9.0-rc3 即時資料處理框架 安裝 (CentOS 6.4 x64)

[研究] Storm 0.9.0-rc3 即時資料處理框架 安裝 (CentOS 6.4 x64)

2013-12-05

Storm是twitter開源的一套即時資料處理框架，基於該框架你可以通過簡單的程式設計來實現對資料流程的即時處理變換。

Storm的設定檔一般存放在$STORM_HOME/conf下，通常名為storm.yaml。

官方網站
http://storm-project.net/

Wiki
https://github.com/nathanmarz/storm/wiki?spm=0.0.0.0.T4va1I

Storm 邏輯
http://storm-project.net/images/topology.png

Components of a Storm cluster
https://github.com/nathanmarz/storm/wiki/images/storm-cluster.png

Stream
https://github.com/nathanmarz/storm/wiki/images/topology.png

安裝

cd  /usr/local
wget  https://dl.dropboxusercontent.com/s/t8m516l2kadt7c6/storm-0.9.0-rc3.zip
unzip  storm-0.9.0-rc3.zip

設定Storm參數 (Nimbus伺服器與ZooKeeper伺服器)
(如果只是本機，可以不管)

vi /usr/local/storm-0.9.0-rc3/conf/storm.yaml

########### These MUST be filled in for a storm configuration
# storm.zookeeper.servers:
#     - "server1"
#     - "server2"
#
# nimbus.host: "nimbus"
#
#
# ##### These may optionally be filled in:
#
## List of custom serializations
# topology.kryo.register:
#     - org.mycompany.MyType
#     - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
#     - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
#     - "server1"
#     - "server2"

## Metrics Consumers
# topology.metrics.consumer.register:
#   - class: "backtype.storm.metrics.LoggingMetricsConsumer"
#     parallelism.hint: 1
#   - class: "org.mycompany.MyMetricsConsumer"
#     parallelism.hint: 1
#     argument:
#       - endpoint: "metrics-collector.mycompany.org"

啟動 Nimbus服務

/usr/local/storm-0.9.0-rc3/bin/storm  nimbus  &

驗證

[root@localhost local]# ps aux | grep storm
root     29662 28.6  9.2 1964612 93680 pts/1   Sl   06:01   0:03 java -server -Dstorm.options= -Dstorm.home=/usr/local/storm-0.9.0-rc3 -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/storm-0.9.0-rc3/storm-netty-0.9.0-rc3.jar:/usr/local/storm-0.9.0-rc3/storm-core-0.9.0-rc3.jar:/usr/local/storm-0.9.0-rc3/storm-console-logging-0.9.0-rc3.jar:/usr/local/storm-0.9.0-rc3/lib/tools.macro-0.1.0.jar:/usr/local/storm-0.9.0-rc3/lib/netty-3.6.3.Final.jar:/usr/local/storm-0.9.0-rc3/lib/jetty-6.1.26.jar:/usr/local/storm-0.9.0-rc3/lib/disruptor-2.10.1.jar:/usr/local/storm-0.9.0-rc3/lib/logback-core-1.0.6.jar:/usr/local/storm-0.9.0-rc3/lib/mockito-all-1.9.5.jar:/usr/local/storm-0.9.0-rc3/lib/clj-time-0.4.1.jar:/usr/local/storm-0.9.0-rc3/lib/minlog-1.2.jar:/usr/local/storm-0.9.0-rc3/lib/curator-client-1.0.1.jar:/usr/local/storm-0.9.0-rc3/lib/kryo-2.17.jar:/usr/local/storm-0.9.0-rc3/lib/servlet-api-2.5.jar:/usr/local/storm-0.9.0-rc3/lib/junit-3.8.1.jar:/usr/local/storm-0.9.0-rc3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/storm-0.9.0-rc3/lib/clojure-1.4.0.jar:/usr/local/storm-0.9.0-rc3/lib/ring-servlet-0.3.11.jar:/usr/local/storm-0.9.0-rc3/lib/joda-time-2.0.jar:/usr/local/storm-0.9.0-rc3/lib/logback-classic-1.0.6.jar:/usr/local/storm-0.9.0-rc3/lib/libthrift7-0.7.0-2.jar:/usr/local/storm-0.9.0-rc3/lib/commons-codec-1.4.jar:/usr/local/storm-0.9.0-rc3/lib/snakeyaml-1.11.jar:/usr/local/storm-0.9.0-rc3/lib/reflectasm-1.07-shaded.jar:/usr/local/storm-0.9.0-rc3/lib/ring-devel-0.3.11.jar:/usr/local/storm-0.9.0-rc3/lib/jline-0.9.94.jar:/usr/local/storm-0.9.0-rc3/lib/json-simple-1.1.jar:/usr/local/storm-0.9.0-rc3/lib/tools.cli-0.2.2.jar:/usr/local/storm-0.9.0-rc3/lib/jetty-util-6.1.26.jar:/usr/local/storm-0.9.0-rc3/lib/clj-stacktrace-0.2.2.jar:/usr/local/storm-0.9.0-rc3/lib/commons-exec-1.1.jar:/usr/local/storm-0.9.0-rc3/lib/clout-1.0.1.jar:/usr/local/storm-0.9.0-rc3/lib/objenesis-1.2.jar:/usr/local/storm-0.9.0-rc3/lib/hiccup-0.3.6.jar:/usr/local/storm-0.9.0-rc3/lib/httpclient-4.1.1.jar:/usr/local/storm-0.9.0-rc3/lib/jgrapht-0.8.3.jar:/usr/local/storm-0.9.0-rc3/lib/asm-4.0.jar:/usr/local/storm-0.9.0-rc3/lib/curator-framework-1.0.1.jar:/usr/local/storm-0.9.0-rc3/lib/jzmq-2.1.0.jar:/usr/local/storm-0.9.0-rc3/lib/httpcore-4.1.jar:/usr/local/storm-0.9.0-rc3/lib/commons-io-1.4.jar:/usr/local/storm-0.9.0-rc3/lib/core.incubator-0.1.0.jar:/usr/local/storm-0.9.0-rc3/lib/zookeeper-3.3.3.jar:/usr/local/storm-0.9.0-rc3/lib/carbonite-1.5.0.jar:/usr/local/storm-0.9.0-rc3/lib/compojure-1.1.3.jar:/usr/local/storm-0.9.0-rc3/lib/ring-core-1.1.5.jar:/usr/local/storm-0.9.0-rc3/lib/slf4j-api-1.6.5.jar:/usr/local/storm-0.9.0-rc3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm-0.9.0-rc3/lib/commons-fileupload-1.2.1.jar:/usr/local/storm-0.9.0-rc3/lib/math.numeric-tower-0.0.1.jar:/usr/local/storm-0.9.0-rc3/lib/servlet-api-2.5-20081211.jar:/usr/local/storm-0.9.0-rc3/lib/commons-logging-1.1.1.jar:/usr/local/storm-0.9.0-rc3/lib/commons-lang-2.5.jar:/usr/local/storm-0.9.0-rc3/lib/guava-13.0.jar:/usr/local/storm-0.9.0-rc3/lib/tools.logging-0.2.3.jar:/usr/local/storm-0.9.0-rc3/conf -Xmx1024m -Dlogfile.name=nimbus.log -Dlogback.configurationFile=/usr/local/storm-0.9.0-rc3/logback/cluster.xml backtype.storm.daemon.nimbus
root     29706  0.0  0.0 103244   844 pts/1    S+   06:01   0:00 grep storm
[root@localhost local]#

啟動 Storm UI (一網頁介面，提供Storm叢集相關狀態)

/usr/local/storm-0.9.0-rc3/bin/storm ui &

驗證

[root@localhost local]# ps aux | grep storm | grep ui
root     29714 11.6  8.0 1694404 81160 pts/1   Sl   06:05   0:03 java -server -Dstorm.options= -Dstorm.home=/usr/local/storm-0.9.0-rc3 -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/storm-0.9.0-rc3/storm-netty-0.9.0-rc3.jar:/usr/local/storm-0.9.0-rc3/storm-core-0.9.0-rc3.jar:/usr/local/storm-0.9.0-rc3/storm-console-logging-0.9.0-rc3.jar:/usr/local/storm-0.9.0-rc3/lib/tools.macro-0.1.0.jar:/usr/local/storm-0.9.0-rc3/lib/netty-3.6.3.Final.jar:/usr/local/storm-0.9.0-rc3/lib/jetty-6.1.26.jar:/usr/local/storm-0.9.0-rc3/lib/disruptor-2.10.1.jar:/usr/local/storm-0.9.0-rc3/lib/logback-core-1.0.6.jar:/usr/local/storm-0.9.0-rc3/lib/mockito-all-1.9.5.jar:/usr/local/storm-0.9.0-rc3/lib/clj-time-0.4.1.jar:/usr/local/storm-0.9.0-rc3/lib/minlog-1.2.jar:/usr/local/storm-0.9.0-rc3/lib/curator-client-1.0.1.jar:/usr/local/storm-0.9.0-rc3/lib/kryo-2.17.jar:/usr/local/storm-0.9.0-rc3/lib/servlet-api-2.5.jar:/usr/local/storm-0.9.0-rc3/lib/junit-3.8.1.jar:/usr/local/storm-0.9.0-rc3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/storm-0.9.0-rc3/lib/clojure-1.4.0.jar:/usr/local/storm-0.9.0-rc3/lib/ring-servlet-0.3.11.jar:/usr/local/storm-0.9.0-rc3/lib/joda-time-2.0.jar:/usr/local/storm-0.9.0-rc3/lib/logback-classic-1.0.6.jar:/usr/local/storm-0.9.0-rc3/lib/libthrift7-0.7.0-2.jar:/usr/local/storm-0.9.0-rc3/lib/commons-codec-1.4.jar:/usr/local/storm-0.9.0-rc3/lib/snakeyaml-1.11.jar:/usr/local/storm-0.9.0-rc3/lib/reflectasm-1.07-shaded.jar:/usr/local/storm-0.9.0-rc3/lib/ring-devel-0.3.11.jar:/usr/local/storm-0.9.0-rc3/lib/jline-0.9.94.jar:/usr/local/storm-0.9.0-rc3/lib/json-simple-1.1.jar:/usr/local/storm-0.9.0-rc3/lib/tools.cli-0.2.2.jar:/usr/local/storm-0.9.0-rc3/lib/jetty-util-6.1.26.jar:/usr/local/storm-0.9.0-rc3/lib/clj-stacktrace-0.2.2.jar:/usr/local/storm-0.9.0-rc3/lib/commons-exec-1.1.jar:/usr/local/storm-0.9.0-rc3/lib/clout-1.0.1.jar:/usr/local/storm-0.9.0-rc3/lib/objenesis-1.2.jar:/usr/local/storm-0.9.0-rc3/lib/hiccup-0.3.6.jar:/usr/local/storm-0.9.0-rc3/lib/httpclient-4.1.1.jar:/usr/local/storm-0.9.0-rc3/lib/jgrapht-0.8.3.jar:/usr/local/storm-0.9.0-rc3/lib/asm-4.0.jar:/usr/local/storm-0.9.0-rc3/lib/curator-framework-1.0.1.jar:/usr/local/storm-0.9.0-rc3/lib/jzmq-2.1.0.jar:/usr/local/storm-0.9.0-rc3/lib/httpcore-4.1.jar:/usr/local/storm-0.9.0-rc3/lib/commons-io-1.4.jar:/usr/local/storm-0.9.0-rc3/lib/core.incubator-0.1.0.jar:/usr/local/storm-0.9.0-rc3/lib/zookeeper-3.3.3.jar:/usr/local/storm-0.9.0-rc3/lib/carbonite-1.5.0.jar:/usr/local/storm-0.9.0-rc3/lib/compojure-1.1.3.jar:/usr/local/storm-0.9.0-rc3/lib/ring-core-1.1.5.jar:/usr/local/storm-0.9.0-rc3/lib/slf4j-api-1.6.5.jar:/usr/local/storm-0.9.0-rc3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm-0.9.0-rc3/lib/commons-fileupload-1.2.1.jar:/usr/local/storm-0.9.0-rc3/lib/math.numeric-tower-0.0.1.jar:/usr/local/storm-0.9.0-rc3/lib/servlet-api-2.5-20081211.jar:/usr/local/storm-0.9.0-rc3/lib/commons-logging-1.1.1.jar:/usr/local/storm-0.9.0-rc3/lib/commons-lang-2.5.jar:/usr/local/storm-0.9.0-rc3/lib/guava-13.0.jar:/usr/local/storm-0.9.0-rc3/lib/tools.logging-0.2.3.jar:/usr/local/storm-0.9.0-rc3:/usr/local/storm-0.9.0-rc3/conf -Xmx768m -Dlogfile.name=ui.log -Dlogback.configurationFile=/usr/local/storm-0.9.0-rc3/logback/cluster.xml backtype.storm.ui.core
[root@localhost local]#

然後開啟瀏覽器連上網址http://nimbus-ip:8080，其中nimbus-ip請換成Nimbus伺服器的IP，啟動後連至伺服器畫面詳見圖9。

啟動Supervisor伺服器

/usr/local/storm-0.9.0-rc3/bin/storm supervisor &

驗證

[root@localhost local]# ps aux | grep storm | grep supervisor
root     29875 20.8  8.1 1174592 82608 pts/1   Sl   06:25   0:03 java -server -Dstorm.options= -Dstorm.home=/usr/local/storm-0.9.0-rc3 -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/storm-0.9.0-rc3/storm-netty-0.9.0-rc3.jar:/usr/local/storm-0.9.0-rc3/storm-core-0.9.0-rc3.jar:/usr/local/storm-0.9.0-rc3/storm-console-logging-0.9.0-rc3.jar:/usr/local/storm-0.9.0-rc3/lib/tools.macro-0.1.0.jar:/usr/local/storm-0.9.0-rc3/lib/netty-3.6.3.Final.jar:/usr/local/storm-0.9.0-rc3/lib/jetty-6.1.26.jar:/usr/local/storm-0.9.0-rc3/lib/disruptor-2.10.1.jar:/usr/local/storm-0.9.0-rc3/lib/logback-core-1.0.6.jar:/usr/local/storm-0.9.0-rc3/lib/mockito-all-1.9.5.jar:/usr/local/storm-0.9.0-rc3/lib/clj-time-0.4.1.jar:/usr/local/storm-0.9.0-rc3/lib/minlog-1.2.jar:/usr/local/storm-0.9.0-rc3/lib/curator-client-1.0.1.jar:/usr/local/storm-0.9.0-rc3/lib/kryo-2.17.jar:/usr/local/storm-0.9.0-rc3/lib/servlet-api-2.5.jar:/usr/local/storm-0.9.0-rc3/lib/junit-3.8.1.jar:/usr/local/storm-0.9.0-rc3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/storm-0.9.0-rc3/lib/clojure-1.4.0.jar:/usr/local/storm-0.9.0-rc3/lib/ring-servlet-0.3.11.jar:/usr/local/storm-0.9.0-rc3/lib/joda-time-2.0.jar:/usr/local/storm-0.9.0-rc3/lib/logback-classic-1.0.6.jar:/usr/local/storm-0.9.0-rc3/lib/libthrift7-0.7.0-2.jar:/usr/local/storm-0.9.0-rc3/lib/commons-codec-1.4.jar:/usr/local/storm-0.9.0-rc3/lib/snakeyaml-1.11.jar:/usr/local/storm-0.9.0-rc3/lib/reflectasm-1.07-shaded.jar:/usr/local/storm-0.9.0-rc3/lib/ring-devel-0.3.11.jar:/usr/local/storm-0.9.0-rc3/lib/jline-0.9.94.jar:/usr/local/storm-0.9.0-rc3/lib/json-simple-1.1.jar:/usr/local/storm-0.9.0-rc3/lib/tools.cli-0.2.2.jar:/usr/local/storm-0.9.0-rc3/lib/jetty-util-6.1.26.jar:/usr/local/storm-0.9.0-rc3/lib/clj-stacktrace-0.2.2.jar:/usr/local/storm-0.9.0-rc3/lib/commons-exec-1.1.jar:/usr/local/storm-0.9.0-rc3/lib/clout-1.0.1.jar:/usr/local/storm-0.9.0-rc3/lib/objenesis-1.2.jar:/usr/local/storm-0.9.0-rc3/lib/hiccup-0.3.6.jar:/usr/local/storm-0.9.0-rc3/lib/httpclient-4.1.1.jar:/usr/local/storm-0.9.0-rc3/lib/jgrapht-0.8.3.jar:/usr/local/storm-0.9.0-rc3/lib/asm-4.0.jar:/usr/local/storm-0.9.0-rc3/lib/curator-framework-1.0.1.jar:/usr/local/storm-0.9.0-rc3/lib/jzmq-2.1.0.jar:/usr/local/storm-0.9.0-rc3/lib/httpcore-4.1.jar:/usr/local/storm-0.9.0-rc3/lib/commons-io-1.4.jar:/usr/local/storm-0.9.0-rc3/lib/core.incubator-0.1.0.jar:/usr/local/storm-0.9.0-rc3/lib/zookeeper-3.3.3.jar:/usr/local/storm-0.9.0-rc3/lib/carbonite-1.5.0.jar:/usr/local/storm-0.9.0-rc3/lib/compojure-1.1.3.jar:/usr/local/storm-0.9.0-rc3/lib/ring-core-1.1.5.jar:/usr/local/storm-0.9.0-rc3/lib/slf4j-api-1.6.5.jar:/usr/local/storm-0.9.0-rc3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm-0.9.0-rc3/lib/commons-fileupload-1.2.1.jar:/usr/local/storm-0.9.0-rc3/lib/math.numeric-tower-0.0.1.jar:/usr/local/storm-0.9.0-rc3/lib/servlet-api-2.5-20081211.jar:/usr/local/storm-0.9.0-rc3/lib/commons-logging-1.1.1.jar:/usr/local/storm-0.9.0-rc3/lib/commons-lang-2.5.jar:/usr/local/storm-0.9.0-rc3/lib/guava-13.0.jar:/usr/local/storm-0.9.0-rc3/lib/tools.logging-0.2.3.jar:/usr/local/storm-0.9.0-rc3/conf -Xmx256m -Dlogfile.name=supervisor.log -Dlogback.configurationFile=/usr/local/storm-0.9.0-rc3/logback/cluster.xml backtype.storm.daemon.supervisor
[root@localhost local]#

更新畫面，應該可以看到多一台 supervisor

(完)

相關

Storm配置項詳解
http://www.alidata.org/archives/2118

Storm入門教程 第三章 Storm安裝部署步驟
http://blog.linezing.com/2013/01/how-to-install-and-deploy-storm-cluster