hadoop下c++程序-天气实例
2015-12-26 21:25
369 查看
非常希望能在hadoop上做c++程序。自己对c++还是有点情节的,依据《hadoop权威指南中文第二版》Hadoop的Pipes进行了试验,并測试成功
Makefile文件(自己进行了改动):
源数据文件:
0067011990999991950051507004+68750+023550FM-12+038299999V0203301N00671220001CN9999999N9+00001+99999999999
0043011990999991950051512004+68750+023550FM-12+038299999V0203201N00671220001CN9999999N9+00221+99999999999
0043011990999991950051518004+68750+023550FM-12+038299999V0203201N00261220001CN9999999N9-00111+99999999999
0043012650999991949032412004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+01111+99999999999
0043012650999991949032418004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+00781+99999999999
上传到HDFS:hdfs dfs -put sample.txt
make后生成了可运行文件上传到HDFS: hdfs dfs -put max_temperature /bin
运行方法: hadoop pipes -D hadoop.pipes.java.recordreader=true -D hadoop.pipes.java.recordwriter=true -input /user/root/sample.txt -output /output -program /bin/max_temperature
数据输出结果:
#include <algorithm> #include <limits.h> #include <stdint.h> #include <string> #include "Pipes.hh" #include "TemplateFactory.hh" #include "StringUtils.hh" class MaxTemperatureMapper : public HadoopPipes::Mapper { public: MaxTemperatureMapper(HadoopPipes::TaskContext& context) { } void map(HadoopPipes::MapContext& context) { std::string line = context.getInputValue(); std::string year = line.substr(15, 4); std::string airTemperature = line.substr(87, 5); std::string q = line.substr(92, 1); if (airTemperature != "+9999" && (q == "0" || q == "1" || q == "4" || q == "5" || q == "9")) { context.emit(year, airTemperature); } } }; class MapTemperatureReducer : public HadoopPipes::Reducer { public: MapTemperatureReducer(HadoopPipes::TaskContext& context) { } void reduce(HadoopPipes::ReduceContext& context) { int maxValue = INT_MIN; while (context.nextValue()) { maxValue = std::max(maxValue, HadoopUtils::toInt(context.getInputValue())); } context.emit(context.getInputKey(), HadoopUtils::toString(maxValue)); } }; int main(int argc, char *argv[]) { return HadoopPipes::runTask(HadoopPipes::TemplateFactory<MaxTemperatureMapper, MapTemperatureReducer>()); }注意:和书上不一样的地方:limit.h头文件
Makefile文件(自己进行了改动):
.SUFFIXES:.h .c .cpp .o CC=g++ CPPFLAGS = -m64 RM = rm SRCS = max_temperature.cpp PROGRAM = max_temperature INC_PATH = -I$(HADOOP_DEV_HOME)/include LIB_PATH = -L$(HADOOP_DEV_HOME)/lib/native LIBS = -lhadooppipes -lcrypto -lhadooputils -lpthread $(PROGRAM):$(SRCS) $(CC) $(CPPFLAGS) $(INC_PATH) $< -Wall $(LIB_PATH) $(LIBS) -g -O2 -o $@ .PHONY:clean clean: $(RM) $(PROGRAM)
源数据文件:
0067011990999991950051507004+68750+023550FM-12+038299999V0203301N00671220001CN9999999N9+00001+99999999999
0043011990999991950051512004+68750+023550FM-12+038299999V0203201N00671220001CN9999999N9+00221+99999999999
0043011990999991950051518004+68750+023550FM-12+038299999V0203201N00261220001CN9999999N9-00111+99999999999
0043012650999991949032412004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+01111+99999999999
0043012650999991949032418004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+00781+99999999999
上传到HDFS:hdfs dfs -put sample.txt
make后生成了可运行文件上传到HDFS: hdfs dfs -put max_temperature /bin
运行方法: hadoop pipes -D hadoop.pipes.java.recordreader=true -D hadoop.pipes.java.recordwriter=true -input /user/root/sample.txt -output /output -program /bin/max_temperature
数据输出结果:
相关文章推荐
- Visual studio 2015 C++ 中嵌入python文件的环境搭建
- 宏定义中的#和##
- C语言宏定义技巧
- 黑马程序员-C语言-基础语法
- 输入两个正整数m和n,求其最大公约数和最小公倍数
- 判断完数、输出完数---C++
- c语言数据类型的转换
- 当线程函数为C++类成员函数时
- 稳定排序和不稳定排序
- 字符串操作(C++程序设计第7周)
- 整数的输出格式(C++程序设计第7周)
- 实数的输出格式(C++程序设计第7周)
- 位操作在java和C/C++中的区别
- C代码中如何调用C++ C++中如何调用C
- 实战c++中的智能指针unique_ptr系列-- 使用unique_ptr来避免if多层嵌套
- 实战c++中的智能指针unique_ptr系列-- 使用unique_ptr来避免if多层嵌套
- extern "c"作用
- 如何用C语言封装 C++的类,在 C里面使用
- C语言学习笔记---伊能C语言学习笔记----寄存器变量
- C++11 并发指南一(C++11 多线程初探)