Mapreduce 之 nativetask

zhixingheyi_tian2026-08-02 15:43

hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/NativeTask.h

cpp 复制代码

#define NATIVE_SORT_TYPE "native.sort.type"
#define MAPRED_SORT_AVOID "mapreduce.sort.avoidance"
#define NATIVE_SORT_MAX_BLOCK_SIZE "native.sort.blocksize.max"
#define MAPRED_COMPRESS_MAP_OUTPUT "mapreduce.map.output.compress"
#define MAPRED_MAP_OUTPUT_COMPRESSION_CODEC "mapreduce.map.output.compress.codec"
#define MAPRED_MAPOUTPUT_KEY_CLASS "mapreduce.map.output.key.class"
#define MAPRED_OUTPUT_KEY_CLASS "mapreduce.job.output.key.class"
#define MAPRED_MAPOUTPUT_VALUE_CLASS "mapreduce.map.output.value.class"
#define MAPRED_OUTPUT_VALUE_CLASS "mapreduce.job.output.value.class"
#define MAPRED_IO_SORT_MB "mapreduce.task.io.sort.mb"
#define MAPRED_NUM_REDUCES "mapreduce.job.reduces"
#define MAPRED_COMBINE_CLASS_OLD "mapred.combiner.class"
#define MAPRED_COMBINE_CLASS_NEW "mapreduce.job.combine.class"

#define NATIVE_LOG_DEVICE "native.log.device"

/dev/shm/yarn/log/application_1785138663180_0056/container_1785138663180_0056_01_000003

cpp 复制代码

2026-07-28 14:00:01,533 INFO [main] org.apache.hadoop.mapred.nativetask.NativeBatchProcessor: NativeHandler: direct buffer size: 1048576
2026-07-28 14:00:01,541 INFO [main] org.apache.hadoop.mapred.nativetask.util.OutputUtil: nativetask.output.manager = org.apache.hadoop.mapred.nativetask.util.NativeTaskOutputFiles
2026-07-28 14:00:01,542 INFO [main] org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator: Native output collector can be successfully enabled!
2026-07-28 14:00:01,542 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator
2026-07-28 14:00:05,032 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
2026-07-28 14:00:07,532 WARN [Thread-12] mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
2026-07-28 14:00:12,619 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1785138663180_0056_m_000001_0 is done. And is in the process of committing
2026-07-28 14:00:12,701 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1785138663180_0056_m_000001_0' done.
2026-07-28 14:00:12,705 INFO [main] org.apache.hadoop.mapred.Task: Final Counters for attempt_1785138663180_0056_m_000001_0: Counters: 29