Druid:Druid.io 部署&使用文档
2016-12-23 19:57
471 查看
参考:
http://lxw1234.com/archives/2015/11/554.htm 海量数据实时OLAP分析系统-Druid.io安装配置和体验http://druid.io/docs/0.9.2/design/design.html Druid官网搭建
Druid.io 部署&使用文档
1.集群模式下部署
Prerequisites : Java 7 or higher & Zookeeper & mysql下载Druid.io :
curl -O http://static.druid.io/artifacts/releases/druid-0.9.1.1-bin.tar.gz tar -xzf druid-0.9.1.1-bin.tar.gz cd druid-0.9.1.1
文件夹目录结构 :
LICENSE - the license files.
bin/ - scripts related to the single-machine quickstart.
conf/* - template configurations for a clustered setup.
conf-quickstart/* - configurations for the single-machine quickstart.
extensions/* - all Druid extensions.
hadoop-dependencies/* - Druid Hadoop dependencies.
lib/* - all included software packages for core Druid.
quickstart/* - files related to the single-machine quickstart.
所有配置文件均在 conf/* 目录下.
配饰HDFS为Druid.io的deep storage & 配置zk & 配置mysql
修改 conf/druid/_common/common.runtime.properties 文件.# # Extensions # # This is not the full list of Druid extensions, but common ones that people often use. You may need to change this list # based on your particular setup. #使用 "mysql-metadata-storage" 作为metadata的存储 #使用 "druid-hdfs-storage" 作为 deep storage #使用 "druid-parquet-extensions" 向druid中插入parquet数据 druid.extensions.loadList=["druid-kafka-eight", "druid-histogram", "druid-datasketches", "mysql-metadata-storage", "druid-hdfs-storage", "druid-avro-extensions", "druid-parquet-extensions"] # If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory # and uncomment the line below to point to your directory. #druid.extensions.hadoopDependenciesDir=/my/dir/hadoop-dependencies # # Logging # # Log all runtime properties on startup. Disable to avoid logging properties on startup: druid.startup.logging.logProperties=true # # Zookeeper # druid.zk.service.host=tagtic-slave01:2181,tagtic-slave02:2181,tagtic-slave03:2181 druid.zk.paths.base=/druid # # Metadata storage # # For MySQL: druid.metadata.storage.type=mysql druid.metadata.storage.connector.connectURI=jdbc:mysql://tagtic-master:3306/druid druid.metadata.storage.connector.user=root druid.metadata.storage.connector.password=123456 # # Deep storage # # For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp): druid.storage.type=hdfs druid.storage.storageDirectory=/druid/segments # # Indexing service logs # # For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp): druid.indexer.logs.type=hdfs druid.indexer.logs.directory=/druid/indexing-logs # # Service discovery # druid.selectors.indexing.serviceName=druid/overlord druid.selectors.coordinator.serviceName=druid/coordinator # # Monitoring # druid.monitoring.monitors=["com.metamx.metrics.JvmMonitor"] druid.emitter=logging druid.emitter.logging.logLevel=info
将 Hadoop 的配置文件(core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml) cp 到 conf/druid/_common 目录下
修改 conf/druid/middleManager/runtime.properties 文件.
druid.service=druid/middleManager druid.port=18091 # Number of tasks per middleManager druid.worker.capacity=3 # Task launch parameters # **CDH版本添加 -Dhadoop.mapreduce.job.classloader=true 来解决hadoop indexer导入时jar包冲突问题** druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dhadoop.mapreduce.job.classloader=true druid.indexer.task.baseTaskDir=var/druid/task # HTTP server threads druid.server.http.numThreads=25 # Processing threads and buffers druid.processing.buffer.sizeBytes=536870912 druid.processing.numThreads=2 # Hadoop indexing druid.indexer.task.hadoopWorkingPath=/tmp/druid-indexing druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.6.0"]
Hadoop集群版本必须和Druid.io中版本同一,可以通过pull-deps下载相同hadoop-dependencies版本,e.g. :
java -classpath "lib/*" io.druid.cli.Main tools pull-deps --defaultVersion 0.9.1.1 -c io.druid.extensions:mysql-metadata-storage:0.9.1.1 -c druid-hdfs-storage -h org.apache.hadoop:hadoop-client:2.6.0
项目中Druid.io配置端口号:
druid.service=druid/coordinator druid.port=18081
druid.service=druid/broker druid.port=18082
druid.service=druid/historical druid.port=18083
druid.service=druid/overlord druid.port=18090
druid.service=druid/middleManager druid.port=18091
2.启动Druid.io
java `cat conf/druid/coordinator/jvm.config | xargs` -cp conf/druid/_common:conf/druid/coordinator:lib/* io.druid.cli.Main server coordinator &>> logs/coordinator.log & java `cat conf/druid/overlord/jvm.config | xargs` -cp conf/druid/_common:conf/druid/overlord:lib/* io.druid.cli.Main server overlord &>> logs/overlord.log & java `cat conf/druid/historical/jvm.config | xargs` -cp conf/druid/_common:conf/druid/historical:lib/* io.druid.cli.Main server historical &>> logs/historical.log & java `cat conf/druid/middleManager/jvm.config | xargs` -cp conf/druid/_common:conf/druid/middleManager:lib/* io.druid.cli.Main server middleManager &>> logs/middleManager.log & java `cat conf/druid/broker/jvm.config | xargs` -cp conf/druid/_common:conf/druid/broker:lib/* io.druid.cli.Main server broker &>> logs/broker.log &
3.从HDFS导入数据到Druid.io
批量导入Batch Data Ingestion导入Parquet文件
Druid作业查看 Coordinator : http://tagtic-master:18090/console.html
Druid集群查看Cluster : http://tagtic-master:18081/#/
解决传入数据时区问题 Hadoop Configuration,在conf/druid/_common/mapred-site.xml中添加
<property> <name>mapreduce.map.java.opts</name> <value>-server -Xmx1536m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</value> </property> <property> <name>mapreduce.reduce.java.opts</name> <value>-server -Xmx2560m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</value> </property>
如果您喜欢我写的博文,读后觉得收获很大,不妨小额赞助我一下,让我有动力继续写出高质量的博文,感谢您的赞赏!!!
相关文章推荐
- Druid.io 部署&使用文档
- 使用未写入文档参数&quot;_ALLOW_RESETLOGS_CORRUPTION&quot;进行崩溃恢复经典
- 使用 WebLogic 10.0 + Workshop for WebLogic Platform 开发&部署 Java EE 应用
- 使用数据容灾恢复软件Acronis Backup & Recovery恢复用户文档文件夹
- dojo文档工具rstwiki windows环境部署及使用
- MyEclipse中Tomcat的部署及其中的一个问题 & SVN使用
- OPC 技术文档之 IOPCBrowseServerAddressSpace 的使用(vc)
- OPC 技术文档之 IOPCBrowseServerAddressSpace 的使用
- 使用未写入文档参数"_ALLOW_RESETLOGS_CORRUPTION"进行崩溃恢复
- VS6.0单文档工程中,View界面想使用CListCtrl & list = GetListCtrl(); 问题总结
- VC 使用MSXML创建SVG文档中的新结点时出现xmlns="" 属性解决方法
- Redis的部署使用文档 推荐
- Redis的部署使用文档
- 使用Ant在Weblogic部署的官方文档
- HTML中,content="text/html; charset=gb2312";XML中 encoding="UTF-8"。其实都是指的 “程序员告诉别人,我的此html文档所使用的字符编码,但是
- Sencha Touch 2 官方文档翻译之 Using Store & Using Proxy(使用数据存储和代理)
- 使用带Arduino IDE & WIZ820io的ATmega1284P