InputFormat到key-value生成流程
2015-09-01 21:06
399 查看
public abstract class InputFormat<K, V> {
public abstract
List<InputSplit> getSplits(JobContext context
) throws IOException, InterruptedException;
public abstract
RecordReader<K,V> createRecordReader(InputSplit split,
TaskAttemptContext context
) throws IOException,
InterruptedException;
}
public abstract class FileInputFormat<K, V> extends InputFormat<K, V> {
public class TextInputFormat extends FileInputFormat<LongWritable, Text> {
getSplits方法,获得对输入文件的切分数量,每一个split对应一个map。
创建RecordReader,该RecordReader接收切分好的split,实现nextKeyValue、getCurrentKey、getCurrentValue。
如下所示,每个map类都会继承Mapper类,在Mapper类中,run方法会调用InputFormat中的RecordReader来获得key、value
public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT> {
/**
* Expert users can override this method for more complete control over the
* execution of the Mapper.
* @param context
* @throws IOException
*/
public void run(Context context) throws IOException, InterruptedException {
setup(context);
try {
while (context.nextKeyValue()) {
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
} finally {
cleanup(context);
}
}
}
public abstract
List<InputSplit> getSplits(JobContext context
) throws IOException, InterruptedException;
public abstract
RecordReader<K,V> createRecordReader(InputSplit split,
TaskAttemptContext context
) throws IOException,
InterruptedException;
}
public abstract class FileInputFormat<K, V> extends InputFormat<K, V> {
public class TextInputFormat extends FileInputFormat<LongWritable, Text> {
getSplits方法,获得对输入文件的切分数量,每一个split对应一个map。
创建RecordReader,该RecordReader接收切分好的split,实现nextKeyValue、getCurrentKey、getCurrentValue。
如下所示,每个map类都会继承Mapper类,在Mapper类中,run方法会调用InputFormat中的RecordReader来获得key、value
public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT> {
/**
* Expert users can override this method for more complete control over the
* execution of the Mapper.
* @param context
* @throws IOException
*/
public void run(Context context) throws IOException, InterruptedException {
setup(context);
try {
while (context.nextKeyValue()) {
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
} finally {
cleanup(context);
}
}
}
相关文章推荐
- iPhone第三节:UITableView
- UIProgressView 进度条
- Codeforces 52A 123-sequence
- UITableView<三>
- UGUI基本控件(二)
- UIToolbar工具栏类
- 尽量用pass-by-reference-to-const(const引用)替换pass-by-value(传值)
- 尽量用pass-by-reference-to-const(const引用)替换pass-by-value(传值)
- UICollectionView 集合视图
- Android UI设计小知识——富文本
- setValue:forUndefinedKey:]: this class is not key value coding-compliant for the key解决办法
- UIView与CALayer的区别
- UITableView<二>
- UIView动画和CAAnimation动画
- 页面之间传值方式的总结,五种方式,通知,block,代理,单例,NSUERDEFALUT,
- Android UI设计小知识——渐变色背景的制作
- HDU 5014 Number Sequence
- UITableView <一>
- UIView动画 防多星球环绕转动
- Android UI设计小知识——按钮背景的制作