4.MapReduce 序列化

流月up2024-01-09 10:08

目录

概述

序列化是分布式计算中很重要的一环境，好的序列化方式，可以大大减少分布式计算中，网络传输的数据量。

序列化

序列化

对象 --> 字节序例 ：存储到磁盘或者网络传输

MR 、Spark、Flink ：分布式的执行框架 必然会涉及到网络传输

java 中的序列化：Serializable

Hadoop 中序列化特点： 紧凑、速度、扩展性、互操作

Spark 中使用了其它的序例化框架 Kyro

反序例化

字节序例 ---> 对象

java自带的两种

Serializable

此处是 java 自带的 序例化 方式，这种方式简单方便，但体积大，不利于大数据量网络传输。

java 复制代码 
public class JavaSerDemo {

    public static void main(String[] args) throws IOException, ClassNotFoundException {
        Person person = new Person(1, "张三", 33);
        ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream("download/person.obj"));
        out.writeObject(person);

        ObjectInputStream in = new ObjectInputStream(new FileInputStream("download/person.obj"));
        Object o = in.readObject();
        System.out.println(o);
    }


    static class Person implements Serializable {
        private int id;
        private String name;
        private int age;

        public Person(int id, String name, int age) {
            this.id = id;
            this.name = name;
            this.age = age;
        }

        @Override
        public String toString() {
            return "Person{" +
                    "id=" + id +
                    ", name='" + name + '\'' +
                    ", age=" + age +
                    '}';
        }

        public int getId() {
            return id;
        }

        public void setId(int id) {
            this.id = id;
        }

        public String getName() {
            return name;
        }

        public void setName(String name) {
            this.name = name;
        }

        public int getAge() {
            return age;
        }

        public void setAge(int age) {
            this.age = age;
        }
    }
}

非Serializable

java 复制代码 
public class DataSerDemo {

    public static void main(String[] args) throws IOException {

        Person person = new Person(1, "张三", 33);
        DataOutputStream out = new DataOutputStream(new FileOutputStream("download/person2.obj"));
        out.writeInt(person.getId());
        out.writeUTF(person.getName());
        out.close();

        DataInputStream in = new DataInputStream(new FileInputStream("download/person2.obj"));
        // 这里要注意，上面以什么顺序写出去，这里就要以什么顺序读取
        int id = in.readInt();
        String name = in.readUTF();
        in.close();
        System.out.println("id:" + id + " name:" + name);

    }

    /**
     *  注意： 不需要继承 Serializable
     */
    static class Person {
        private int id;
        private String name;
        private int age;

        public Person(int id, String name, int age) {
            this.id = id;
            this.name = name;
            this.age = age;
        }

        @Override
        public String toString() {
            return "Person{" +
                    "id=" + id +
                    ", name='" + name + '\'' +
                    ", age=" + age +
                    '}';
        }

        public int getId() {
            return id;
        }

        public void setId(int id) {
            this.id = id;
        }

        public String getName() {
            return name;
        }

        public void setName(String name) {
            this.name = name;
        }

        public int getAge() {
            return age;
        }

        public void setAge(int age) {
            this.age = age;
        }
    }
}

hadoop序例化

官方地址速递

The key and value classes have to be serializable by the framework and hence need to implement the Writable interface. Additionally, the key classes have to implement the WritableComparable interface to facilitate sorting by the framework.

注意：Writable 两个方法，一个 write ,readFields

java 复制代码 
@InterfaceAudience.Public
@InterfaceStability.Stable
public interface Writable {

  void write(DataOutput out) throws IOException;

  void readFields(DataInput in) throws IOException;
}

实践

java 复制代码 
public class PersonWritable implements Writable {

    private int id;
    private String name;
    private int age;
    // 消费金额
    private int consumption;
    // 消费总金额
    private long consumptions;


    public PersonWritable() {
    }

    public PersonWritable(int id, String name, int age, int consumption) {
        this.id = id;
        this.name = name;
        this.age = age;
        this.consumption = consumption;
    }

    public PersonWritable(int id, String name, int age, int consumption, long consumptions) {
        this.id = id;
        this.name = name;
        this.age = age;
        this.consumption = consumption;
        this.consumptions = consumptions;
    }

    public int getId() {
        return id;
    }

    public void setId(int id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public int getAge() {
        return age;
    }

    public void setAge(int age) {
        this.age = age;
    }

    public int getConsumption() {
        return consumption;
    }

    public void setConsumption(int consumption) {
        this.consumption = consumption;
    }

    public long getConsumptions() {
        return consumptions;
    }

    public void setConsumptions(long consumptions) {
        this.consumptions = consumptions;
    }

    @Override
    public String toString() {
        return
                "id=" + id +
                        ", name='" + name + '\'' +
                        ", age='" + age + '\'' +
                        ", consumption=" + consumption + '\'' +
                        ", consumptions=" + consumptions;
    }

    @Override
    public void write(DataOutput out) throws IOException {
        out.writeInt(id);
        out.writeUTF(name);
        out.writeInt(age);
        out.writeInt(consumption);
        out.writeLong(consumptions);
    }

    @Override
    public void readFields(DataInput in) throws IOException {
        id = in.readInt();
        name = in.readUTF();
        age = in.readInt();
        consumption = in.readInt();
        consumptions = in.readLong();
    }
}
java 复制代码 
/**
 * 统计 个人 消费
 */
public class PersonStatistics {

    static class PersonStatisticsMapper extends Mapper<LongWritable, Text, IntWritable, PersonWritable> {
        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            String[] split = value.toString().split(",");
            int id = Integer.parseInt(split[0]);
            String name = split[1];
            int age = Integer.parseInt(split[2]);
            int consumption = Integer.parseInt(split[3]);
            PersonWritable writable = new PersonWritable(id, name, age, consumption, 0);
            context.write(new IntWritable(id), writable);
        }
    }

    static class PersonStatisticsReducer extends Reducer<IntWritable, PersonWritable, NullWritable, PersonWritable> {
        @Override
        protected void reduce(IntWritable key, Iterable<PersonWritable> values, Context context) throws IOException, InterruptedException {
            long count = 0L;
            PersonWritable person = null;
            for (PersonWritable data : values) {
                if (Objects.isNull(person)) {
                    person = data;
                }
                count = count + data.getConsumption();
            }
            person.setConsumptions(count);

            PersonWritable personWritable = new PersonWritable(person.getId(), person.getName(), person.getAge(), person.getConsumption(), count);

            context.write(NullWritable.get(), personWritable);
        }
    }

    public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {
        Configuration configuration = new Configuration();

        String sourcePath = "data/person.data";
        String distPath = "downloadOut/person-out.data";

        FileUtil.deleteIfExist(configuration, distPath);

        Job job = Job.getInstance(configuration, "person statistics");
        job.setJarByClass(PersonStatistics.class);
        //job.setCombinerClass(PersonStatistics.PersonStatisticsReducer.class);
        job.setMapperClass(PersonStatisticsMapper.class);
        job.setReducerClass(PersonStatisticsReducer.class);
        job.setMapOutputKeyClass(IntWritable.class);
        job.setMapOutputValueClass(PersonWritable.class);
        job.setOutputKeyClass(NullWritable.class);
        job.setOutputValueClass(PersonWritable.class);

        FileInputFormat.addInputPath(job, new Path(sourcePath));
        FileOutputFormat.setOutputPath(job, new Path(distPath));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}
bash 复制代码 
# person.data
1,张三,30,10
1,张三,30,20
2,李四,25,5

上述执行结果如下：

分片/InputFormat & InputSplit

官方文档速递

java 复制代码 
org.apache.hadoop.mapreduce.InputFormat
org.apache.hadoop.mapreduce.InputSplit

日志

执行 序列化 测试小程序，关注以下日志

bash 复制代码 
# 总共加载一个文件，分隔成一个
2024-01-06 09:19:42,363 [main] [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] [INFO] - Total input files to process : 1
2024-01-06 09:19:42,487 [main] [org.apache.hadoop.mapreduce.JobSubmitter] [INFO] - number of splits:1

结束

至此，MapReduce 序列化 至此结束，如有疑问，欢迎评论区留言。

相关推荐
2601_965969062 小时前
AI求职能力证明指南：认证证书、项目作品与实际工作成果
大数据
EnCi Zheng2 小时前
AI Agent工具调用防护层怎么写-一份能直接跑的Python四层拦截实现
大数据·开发语言·python
byte轻骑兵2 小时前
【TextIn xParse + WorkBuddy】把8小时的标书审查压进1小时：招标要求逐条核验与废标风险预警工作台
大数据·数据库·aigc·textin·ai数据层基础设施
计算机毕业编程指导师3 小时前
2027计算机毕设选题指南：网站AI化、大数据预测+算法（XGBoost/LightGBM）、深度学习三方向57题+选题方法论
大数据·python·计算机·毕业设计·课程设计·毕设选题·1024程序员节
链上日记3 小时前
凯敏雅克签约恒远善汇团队为进军中国市场打下坚实的基石
大数据
注塑机数据采集联网3 小时前
中科晟达注塑机 MES 系统：从车间到数字化的智能制造实践
大数据·数据仓库·人工智能·物联网·制造·数据库架构
段一凡-华北理工大学3 小时前
高炉能源系统智能化：热风炉、煤气与极致能效~系列文章02：走进热风炉：蓄热、换炉、送风的工艺机理
大数据·数据库·能源·高炉能源管控·热风炉·热风炉工作原理
EnCi Zheng4 小时前
AI Agent工具误调用怎么解决-从参数校验到高危二次确认的四层工程防线
大数据·数据库·人工智能
安托智造13 小时前
MBD：从二维图纸到三维模型的制造革命（下）
大数据·人工智能·制造·mbd
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02如何新建文件夹？ 电脑新建文件夹的4种方法03GitHub 镜像站点04历年考研数学一、数学二、数学三真题试卷及答案PDF05几个好用的ip纯净度检测网站06苹果iPA游戏软件资源下载网站07昇腾910B系列芯片完全指南：四款型号对比、服务器选型与大模型部署实战08ai安全工具:CyberStrikeAI安装部署与使用09蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接10乙二醇水溶液物性参数