MapReduce中有三个步骤用于划分大数据集, 给mapper和reducer提供数据
InputSplit第一个是InputSplit, 它把数据划分成若干块提供给mapper
默认情况下是根据数据文件的block, 来划分, 一个block对应一个mapper, 优先在block所在的机器上启动mapper
如果要重构这个 InputSplit 函数的话, 要去 InputFormat 里重构 getSplits 函数
https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/mapred/InputFormat.html
在streaming中:
-inputformat JavaClassNameOptionalClass you supply should return key/value pairs of Text class. If not specified, TextInputFormat is used as the default-outputformat JavaClassNameOptionalClass you supply should take key/value pairs of Text class. If not specified, TextOutputformat is used as the default这两个参数指定姚世勇inputformat class
Partitionpartition用于把结果分配给不同的reducer, 一般继承自 "org.apache.hadoop.mapreduce.Partitioner" 这个类
Grouping这个概念比较难理解, 意思是在数据给reducer前再进行一次分组, 一组数据给到同一个reducer执行一次, 他们的key用的是分组中第一个数据的key
https://stackoverflow.com/questions/14728480/what-is-the-use-of-grouping-comparator-in-hadoop-map-reduce
最佳答案中 a-1和a-2因为grouping的关系合并成了 a-1为key的一组数据给reducer处理
那么在streaming中Partition和Grouping该怎么处理呢?
在streaming中可以用命令行参数指定Partition的类:
-partitioner JavaClassNameOptionalClass that determines which reduce a key is sent to也可以用另一种参数结合sort命令来指定:
-D stream.map.output.field.separator. \-D stream.num.map.output.key.fields4 \-D mapred.text.key.partitioner.options-k1,2 \
这里指定了分割符, 并且分割出来前4个field是key, 并用第一和第二个field来做partition
-D mapreduce.partition.keycomparator.options-k1,2 -k3,3nr -k4,4nr
linux中的sort命令:
sort -k1 -k2n -k3nr #表示优先根据第一列排序, 再根据第二列排序且第二列是数字,再根据第三列排序它是数字而且要逆序来排
grouping在streaming的模式中没有相应实现, 但是可以利用partition来代替.
附表:
ParameterOptional/RequiredDescription-input directoryname or filenameRequiredInput location for mapper-output directorynameRequiredOutput location for reducer-mapper executable or JavaClassNameRequiredMapper executable-reducer executable or JavaClassNameRequiredReducer executable-file filenameOptionalMake the mapper, reducer, or combiner executable available locally on the compute nodes-inputformat JavaClassNameOptionalClass you supply should return key/value pairs of Text class. If not specified, TextInputFormat is used as the default-outputformat JavaClassNameOptionalClass you supply should take key/value pairs of Text class. If not specified, TextOutputformat is used as the default-partitioner JavaClassNameOptionalClass that determines which reduce a key is sent to-combiner streamingCommand or JavaClassNameOptionalCombiner executable for map output-cmdenv namevalueOptionalPass environment variable to streaming commands-inputreaderOptionalFor backwards-compatibility: specifies a record reader class (instead of an input format class)-verboseOptionalVerbose output-lazyOutputOptionalCreate output lazily. For example, if the output format is based on FileOutputFormat, the output file is created only on the first call to output.collect (or Context.write)-numReduceTasksOptionalSpecify the number of reducers-mapdebugOptionalScript to call when map task fails-reducedebugOptionalScript to call when reduce task fails