Start hadoop, hive, hue server in the virtual hadoop clusters based on docker

######start hadoop####### refer to "http://tashan10.com/yong-dockerda-jian-hadoopwei-fen-bu-shi-ji-qun/"

1.open three terminals and execute one of the following three commands respectively for each terminal to start three containers from the image at first time.

1)docker run -ti -h master --name master centos:hadoop_hive

2)docker run -ti -h slave1 --name slave1 centos:hadoop_hive

3)docker run -ti -h slave2 --name slave2 centos:hadoop_hive

if it is not the first time to start the three node container, we can execute "docker ps -a" to see the name of those three container:

#####################################################################################################################################

datahub@datahub:~$ docker ps -a

CONTAINER ID        IMAGE                COMMAND             CREATED             STATUS              PORTS               NAMES

beefe2d6832c        centos:hadoop_hive   "/bin/bash"         2 days ago          Up 2 days                               slave2

ff3aa0faed1f        centos:hadoop_hive   "/bin/bash"         2 days ago          Up 2 days                               slave1

5738358d2048        centos:hadoop_hive   "/bin/bash"         2 days ago          Up 2 days                               master

5166f7fdead5        centos:Hue           "/bin/bash"         2 days ago          Up 2 days                               hue_server

#####################################################################################################################################

1)docker start master  docker attach master

2)docker start slave1  docker attach slave1

3)docker start slave2  docker attach slave2

notice:这里有几个问题：

    (1)Docker容器中的ip地址是启动之后自动分配的，且不能手动更改

    (2)hostname、hosts配置在容器内修改了，只能在本次容器生命周期内有效。如果容器退出了，重新启动，这两个配置将被还原。且这两个配置无法通过commit命令写入镜像

2.配置hosts

    1.通过ifconfig命令获取各节点ip。环境不同获取的ip可能不一样，例如我本机获取的ip如下：

        master:172.17.0.2

        slave1:172.17.0.6

        slave2:172.17.0.7

    使用nano /etc/hosts命令将如下配置写入各节点的hosts文件，注意修改ip地址：

    172.17.0.2        master

    172.17.0.6        slave1

    172.17.0.7        slave2

3.配置slaves

(1)在master节点容器中执行如下命令：

root@master:~# cd $HADOOP_CONFIG_HOME/

root@master:~/soft/apache/hadoop/hadoop-2.6.0/etc/hadoop# nano slaves 

(2)将如下slave节点的hostname信息写入该文件：

slave1

slave2

(3)启动Hadoop

在master节点上执行start-all.sh命令，启动Hadoop

4.check the hadoop run state

在each node上执行jps命令，结果如下：

master节点:

root@master:~/soft/apache/hadoop/hadoop-2.6.0/etc/hadoop# jps

1223 Jps

992 SecondaryNameNode

813 NameNode

1140 ResourceManager

slave1节点:

root@slave1:~/soft/apache/hadoop/hadoop-2.6.0/etc/hadoop# jps

258 NodeManager

352 Jps

159 DataNode

slave2节点:

root@slave2:~/soft/apache/hadoop/hadoop-2.6.0/etc/hadoop# jps

371 Jps

277 NodeManager

178 DataNode

在master节点上通过命令hdfs dfsadmin -report查看DataNode是否正常启动

######start hive######(all of these steps below are done in master node)

we choose mysql as the metastore of hive(username:hive1,password:hive1)

after we finish start hadoop:

1.[root@master hadoop]# service mysqld start

2.source /etc/profile 

(we have add the hive path into /etc/profile,but we still need to execute "source /etc/profile" to update the

state and then we can use the command "hive", else the hive command will call the error "hive command not find")

3.start hive metastore service in the backend:

[root@master hadoop]# hive --service metastore 2>&1 >> /var/log.log &

###because we start metastore service and hive client both in master, so we should start metastore service on backend,then we can start hive client in the shell.Use this command:hive --service metastore 2>&1 >> /var/log.log &###

4.start hive client:

[root@master hadoop]# hive

5.Ctrl+C quit hive client,and start hiverserver2 for hueserver to connect:

[root@master hadoop]# hive --service hiverserver2 2>&1 >> /var/log.log &

#notice#:for Step 3 and 4, there maybe some information like "no hbase" or "SLF4J" like the following details:

####################################################

[root@master hadoop]# hive --service metastore 2>&1 >> /var/log.log &

[1] 1032

[root@master hadoop]# which: no hbase in (/opt/jdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/soft/apache/hadoop/hadoop-2.7.2/bin:/root/soft/apache/hadoop/hadoop-2.7.2/sbin:/root/soft/apache/hadoop/hadoop-2.7.2/bin:/root/soft/apache/hadoop/hadoop-2.7.2/sbin:/usr/local/apache-hive-2.0.1-bin/bin)

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/apache-hive-2.0.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/root/soft/apache/hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

[root@master hadoop]# hive

which: no hbase in (/opt/jdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/soft/apache/hadoop/hadoop-2.7.2/bin:/root/soft/apache/hadoop/hadoop-2.7.2/sbin:/root/soft/apache/hadoop/hadoop-2.7.2/bin:/root/soft/apache/hadoop/hadoop-2.7.2/sbin:/usr/local/apache-hive-2.0.1-bin/bin)

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/apache-hive-2.0.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/root/soft/apache/hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in file:/usr/local/apache-hive-2.0.1-bin/conf/hive-log4j2.properties

Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

hive> 

####################################################

#it is OK, so do not be worried!Our purpose is the "hive>".

if we visit "http://[ip of master]:50070" in the explorer, we can see some hdfs information on web.

# some records #:

some command used usually

1.docker cp /home/datahub/hive-site.xml 9168dfad0358:/usr/local/apache-hive-2.0.1-bin/conf

2.docker commit -m "modify hive-site.xml" 9168dfad0358  centos:hadoop_hive

3.docker ps                #see the detail information of running container

4.docker run -ti -v /home/datahub/Hue:/root/soft/hue_project/Hue --name hue_server centos:Hue /bin/bash       

####important#### 

running a new container, meanwhile we set the container name(--name),and upload the dir of host to docker container(-v)

####important####

## /home/datahub/Hue is the dir of host,/root/soft/hue_project/Hue is the dir of container

5.sudo docker run -itd --name hue_server centos:Hue /bin/bash   #创建一个守护态的Docker容器

docker attach hue_server    #connect a running container "docker exec -ti hue_server /bin/bash" is ok, too. 

6.sudo docker run hue_server   #run a stop bue exist container by name/container_id

7.Remove all stopped containers:  docker rm $(docker ps -a -q)

8."ctrl+p" and then "ctrl+q" to quit container with the container still running. 

9.docker exec -ti hue_server /bin/bash  #open another one terminal of a running container.

#######start hue#########

1.(1)start from a new commit image and we mounted the host dir:docker run -ti -v /home/datahub/Hue:/root/soft/hue_project/Hue --name hue_server centos:Hue /bin/bash

  (2)start from a exist container by name/ID:

     $sudo docker start hue_server

     $docker attach hue_server 

2.if mysql is stopped: service mysqld start

3.go to ${HUE_HOME}:

  cd /root/soft/hue_project/Hue
4.build/env/bin/hue runserver 0.0.0.0:80

5.http://172.17.0.5/