4.MapReduce 序列化

目录

概述

序列化是分布式计算中很重要的一环境,好的序列化方式,可以大大减少分布式计算中,网络传输的数据量。

序列化

序列化

对象 --> 字节序例 :存储到磁盘或者网络传输

MR 、Spark、Flink :分布式的执行框架 必然会涉及到网络传输

java 中的序列化:Serializable

Hadoop 中序列化特点: 紧凑、速度、扩展性、互操作

Spark 中使用了其它的序例化框架 Kyro

反序例化

字节序例 ---> 对象

java自带的两种

Serializable

此处是 java 自带的 序例化 方式,这种方式简单方便,但体积大,不利于大数据量网络传输。

java 复制代码
public class JavaSerDemo {

    public static void main(String[] args) throws IOException, ClassNotFoundException {
        Person person = new Person(1, "张三", 33);
        ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream("download/person.obj"));
        out.writeObject(person);

        ObjectInputStream in = new ObjectInputStream(new FileInputStream("download/person.obj"));
        Object o = in.readObject();
        System.out.println(o);
    }


    static class Person implements Serializable {
        private int id;
        private String name;
        private int age;

        public Person(int id, String name, int age) {
            this.id = id;
            this.name = name;
            this.age = age;
        }

        @Override
        public String toString() {
            return "Person{" +
                    "id=" + id +
                    ", name='" + name + '\'' +
                    ", age=" + age +
                    '}';
        }

        public int getId() {
            return id;
        }

        public void setId(int id) {
            this.id = id;
        }

        public String getName() {
            return name;
        }

        public void setName(String name) {
            this.name = name;
        }

        public int getAge() {
            return age;
        }

        public void setAge(int age) {
            this.age = age;
        }
    }
}

非Serializable

java 复制代码
public class DataSerDemo {

    public static void main(String[] args) throws IOException {

        Person person = new Person(1, "张三", 33);
        DataOutputStream out = new DataOutputStream(new FileOutputStream("download/person2.obj"));
        out.writeInt(person.getId());
        out.writeUTF(person.getName());
        out.close();

        DataInputStream in = new DataInputStream(new FileInputStream("download/person2.obj"));
        // 这里要注意,上面以什么顺序写出去,这里就要以什么顺序读取
        int id = in.readInt();
        String name = in.readUTF();
        in.close();
        System.out.println("id:" + id + " name:" + name);

    }

    /**
     *  注意: 不需要继承 Serializable
     */
    static class Person {
        private int id;
        private String name;
        private int age;

        public Person(int id, String name, int age) {
            this.id = id;
            this.name = name;
            this.age = age;
        }

        @Override
        public String toString() {
            return "Person{" +
                    "id=" + id +
                    ", name='" + name + '\'' +
                    ", age=" + age +
                    '}';
        }

        public int getId() {
            return id;
        }

        public void setId(int id) {
            this.id = id;
        }

        public String getName() {
            return name;
        }

        public void setName(String name) {
            this.name = name;
        }

        public int getAge() {
            return age;
        }

        public void setAge(int age) {
            this.age = age;
        }
    }
}

hadoop序例化

官方地址速递

The key and value classes have to be serializable by the framework and hence need to implement the Writable interface. Additionally, the key classes have to implement the WritableComparable interface to facilitate sorting by the framework.

注意:Writable 两个方法,一个 write ,readFields

java 复制代码
@InterfaceAudience.Public
@InterfaceStability.Stable
public interface Writable {

  void write(DataOutput out) throws IOException;

  void readFields(DataInput in) throws IOException;
}

实践

java 复制代码
public class PersonWritable implements Writable {

    private int id;
    private String name;
    private int age;
    // 消费金额
    private int consumption;
    // 消费总金额
    private long consumptions;


    public PersonWritable() {
    }

    public PersonWritable(int id, String name, int age, int consumption) {
        this.id = id;
        this.name = name;
        this.age = age;
        this.consumption = consumption;
    }

    public PersonWritable(int id, String name, int age, int consumption, long consumptions) {
        this.id = id;
        this.name = name;
        this.age = age;
        this.consumption = consumption;
        this.consumptions = consumptions;
    }

    public int getId() {
        return id;
    }

    public void setId(int id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public int getAge() {
        return age;
    }

    public void setAge(int age) {
        this.age = age;
    }

    public int getConsumption() {
        return consumption;
    }

    public void setConsumption(int consumption) {
        this.consumption = consumption;
    }

    public long getConsumptions() {
        return consumptions;
    }

    public void setConsumptions(long consumptions) {
        this.consumptions = consumptions;
    }

    @Override
    public String toString() {
        return
                "id=" + id +
                        ", name='" + name + '\'' +
                        ", age='" + age + '\'' +
                        ", consumption=" + consumption + '\'' +
                        ", consumptions=" + consumptions;
    }

    @Override
    public void write(DataOutput out) throws IOException {
        out.writeInt(id);
        out.writeUTF(name);
        out.writeInt(age);
        out.writeInt(consumption);
        out.writeLong(consumptions);
    }

    @Override
    public void readFields(DataInput in) throws IOException {
        id = in.readInt();
        name = in.readUTF();
        age = in.readInt();
        consumption = in.readInt();
        consumptions = in.readLong();
    }
}
java 复制代码
/**
 * 统计 个人 消费
 */
public class PersonStatistics {

    static class PersonStatisticsMapper extends Mapper<LongWritable, Text, IntWritable, PersonWritable> {
        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            String[] split = value.toString().split(",");
            int id = Integer.parseInt(split[0]);
            String name = split[1];
            int age = Integer.parseInt(split[2]);
            int consumption = Integer.parseInt(split[3]);
            PersonWritable writable = new PersonWritable(id, name, age, consumption, 0);
            context.write(new IntWritable(id), writable);
        }
    }

    static class PersonStatisticsReducer extends Reducer<IntWritable, PersonWritable, NullWritable, PersonWritable> {
        @Override
        protected void reduce(IntWritable key, Iterable<PersonWritable> values, Context context) throws IOException, InterruptedException {
            long count = 0L;
            PersonWritable person = null;
            for (PersonWritable data : values) {
                if (Objects.isNull(person)) {
                    person = data;
                }
                count = count + data.getConsumption();
            }
            person.setConsumptions(count);

            PersonWritable personWritable = new PersonWritable(person.getId(), person.getName(), person.getAge(), person.getConsumption(), count);

            context.write(NullWritable.get(), personWritable);
        }
    }

    public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {
        Configuration configuration = new Configuration();

        String sourcePath = "data/person.data";
        String distPath = "downloadOut/person-out.data";

        FileUtil.deleteIfExist(configuration, distPath);

        Job job = Job.getInstance(configuration, "person statistics");
        job.setJarByClass(PersonStatistics.class);
        //job.setCombinerClass(PersonStatistics.PersonStatisticsReducer.class);
        job.setMapperClass(PersonStatisticsMapper.class);
        job.setReducerClass(PersonStatisticsReducer.class);
        job.setMapOutputKeyClass(IntWritable.class);
        job.setMapOutputValueClass(PersonWritable.class);
        job.setOutputKeyClass(NullWritable.class);
        job.setOutputValueClass(PersonWritable.class);

        FileInputFormat.addInputPath(job, new Path(sourcePath));
        FileOutputFormat.setOutputPath(job, new Path(distPath));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}
bash 复制代码
# person.data
1,张三,30,10
1,张三,30,20
2,李四,25,5

上述执行结果如下:

分片/InputFormat & InputSplit

官方文档速递

java 复制代码
org.apache.hadoop.mapreduce.InputFormat
org.apache.hadoop.mapreduce.InputSplit

日志

执行 序列化 测试小程序,关注以下日志

bash 复制代码
# 总共加载一个文件,分隔成一个
2024-01-06 09:19:42,363 [main] [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] [INFO] - Total input files to process : 1
2024-01-06 09:19:42,487 [main] [org.apache.hadoop.mapreduce.JobSubmitter] [INFO] - number of splits:1

结束

至此,MapReduce 序列化 至此结束,如有疑问,欢迎评论区留言。

相关推荐
Learn Beyond Limits18 小时前
Error metrics for skewed datasets|倾斜数据集的误差指标
大数据·人工智能·python·深度学习·机器学习·ai·吴恩达
IT研究室19 小时前
大数据毕业设计选题推荐-基于大数据的宫颈癌风险因素分析与可视化系统-Spark-Hadoop-Bigdata
大数据·hadoop·spark·毕业设计·源码·数据可视化·bigdata
武子康19 小时前
Java-118 深入浅出 MySQL ShardingSphere 分片剖析:SQL 支持范围、限制与优化实践
java·大数据·数据库·分布式·sql·mysql·性能优化
IT毕设梦工厂19 小时前
大数据毕业设计选题推荐-基于大数据的高级大豆农业数据分析与可视化系统-Hadoop-Spark-数据可视化-BigData
大数据·数据分析·课程设计
专注数据的痴汉19 小时前
「数据获取」《中国服务业统计与服务业发展(2014)》
大数据·人工智能
镜舟科技20 小时前
告别 Hadoop,拥抱 StarRocks!政采云数据平台升级之路
大数据·starrocks·数据仓库·hadoop·存算分离
毕设源码-赖学姐20 小时前
【开题答辩全过程】以 基于Hadoop电商数据的可视化分析为例,包含答辩的问题和答案
大数据·hadoop·分布式
时序数据说20 小时前
时序数据库IoTDB的核心优势
大数据·数据库·物联网·开源·时序数据库·iotdb
计算机毕设残哥21 小时前
HDFS存储农业大数据的秘密是什么?高级大豆数据分析与可视化系统架构设计思路
大数据·hadoop·python·hdfs·数据分析·spark·django
IT毕设实战小研21 小时前
2026届大数据毕业设计选题推荐-基于大数据旅游数据分析与推荐系统 爬虫数据可视化分析
大数据·人工智能·爬虫·机器学习·架构·数据分析·课程设计