您的位置:首页 > 产品设计 > UI/UE

InputFormat到key-value生成流程

2015-09-01 21:06 399 查看
public abstract class InputFormat<K, V> {

public abstract
List<InputSplit> getSplits(JobContext context
) throws IOException, InterruptedException;

public abstract
RecordReader<K,V> createRecordReader(InputSplit split,
TaskAttemptContext context
) throws IOException,
InterruptedException;

}

public abstract class FileInputFormat<K, V> extends InputFormat<K, V> {

public class TextInputFormat extends FileInputFormat<LongWritable, Text> {

getSplits方法,获得对输入文件的切分数量,每一个split对应一个map。
创建RecordReader,该RecordReader接收切分好的split,实现nextKeyValue、getCurrentKey、getCurrentValue。

如下所示,每个map类都会继承Mapper类,在Mapper类中,run方法会调用InputFormat中的RecordReader来获得key、value

public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT> {
/**
* Expert users can override this method for more complete control over the
* execution of the Mapper.
* @param context
* @throws IOException
*/
public void run(Context context) throws IOException, InterruptedException {
setup(context);
try {
while (context.nextKeyValue()) {
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
} finally {
cleanup(context);
}
}
}
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
标签: