Redisson同时使用jackson、fastjson、kryo、protostuff序列化(含效率对比)

原创不易,转载请注明出处:

https://blog.csdn.net/q258523454/article/details/129988906


前言

我们项目中会用到各种序列化工具,到底哪一种是最适合我们的? 需要从序列化、反序列化、存储大小各个方法面来衡量,我之前有实测过jackson、fastjson、kryo、protostuff序列化的数据,大概情况如下:

CPU: i7 2.80GHz

内存: 16GB

系统: 64位

备注:每次插入1个 StudentObject, 每个 StudentObject中含有 100000 个 Student 对象

600次,共6000万个对象,实测数据.

|------------|-------|--------|-------|
| 方式 | 序列化 | 反序列化 | 存储大小 |
| fastjson | 70 ms | 43 ms | 4.3 M |
| jackson | 68 ms | 135 ms | 7.6 M |
| kryo | 19 ms | 20 ms | 1.7 M |
| protoStuff | 14 ms | 30 ms | 2.1 M |

首选: kryo,其次 protoStuff

代码实现

下面的代码将会实现,一个项目中,Redisson同时使用多个序列化工具,让Redisson用不同的序列化来作客户端。文末也有总结。

maven包

XML 复制代码
<dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-configuration-processor</artifactId>
            <optional>true</optional>
        </dependency>

        <!-- ------------------------------------------------------mybatis 数据库 BEGIN ------------------------------------------------------------------------------ -->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>8.0.13</version>
        </dependency>

        <dependency>
            <groupId>com.github.pagehelper</groupId>
            <artifactId>pagehelper-spring-boot-starter</artifactId>
            <version>1.2.13</version>
        </dependency>

        <dependency>
            <groupId>com.baomidou</groupId>
            <artifactId>mybatis-plus-boot-starter</artifactId>
            <version>3.4.0</version>
            <exclusions>
                <exclusion>
                    <artifactId>mybatis</artifactId>
                    <groupId>org.mybatis</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.mybatis</groupId>
            <artifactId>mybatis</artifactId>
            <version>3.5.6</version>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.46</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>

        <!-- redisson -->
        <dependency>
            <groupId>org.redisson</groupId>
            <artifactId>redisson-spring-boot-starter</artifactId>
            <version>3.17.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.11</version>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.24</version>
        </dependency>

        <dependency>
            <groupId>com.dyuproject.protostuff</groupId>
            <artifactId>protostuff-core</artifactId>
            <version>1.1.3</version>
        </dependency>
        <dependency>
            <groupId>com.dyuproject.protostuff</groupId>
            <artifactId>protostuff-runtime</artifactId>
            <version>1.1.3</version>
        </dependency>

        <dependency>
            <groupId>com.esotericsoftware</groupId>
            <artifactId>kryo</artifactId>
            <version>5.4.0</version>
        </dependency>
    </dependencies>

Redisson抽象类

为了让各个序列化工具复用代码,我们先定义 RedissonClientInterface

java 复制代码
public interface RedissonClientInterface {

    RedissonClient getRedissonClient();

}

将公共能力抽象到 AbstractRedissonService

java 复制代码
public abstract class AbstractRedissonService implements RedissonClientInterface {

    /**
     * 默认保存时间
     */
    private static final long DEFAULT_EXPIRE_TIME_SECONDS = 3600L;


    public RLock getLock(String key) {
        return getRedissonClient().getLock(key);
    }

    /**
     * 设置key-value
     */
    public void set(String key, Object value, long seconds) {
        if (Objects.isNull(value)) {
            return;
        }
        if (seconds <= 0) {
            seconds = DEFAULT_EXPIRE_TIME_SECONDS;
        }
        getRedissonClient().getBucket(key).set(value, seconds, TimeUnit.SECONDS);
    }

    /**
     * 批量设置key-value
     */
    public <T> void setBatch(Map<String, T> map, long seconds) {
        if (seconds <= 0) {
            seconds = DEFAULT_EXPIRE_TIME_SECONDS;
        }
        RBatch batch = getRedissonClient().createBatch();
        long finalSeconds = seconds;
        map.forEach((k, v) -> {
            batch.getBucket(k).setAsync(v, finalSeconds, TimeUnit.SECONDS);
        });
        batch.execute();
    }


    /**
     * 获取value
     * 没有则返回 null
     */
    public Object get(String key) {
        return getRedissonClient().getBucket(key).get();
    }

    /**
     * 批量获取value
     */
    public List<?> getBatch(List<String> keys) {
        RBatch batch = getRedissonClient().createBatch();
        keys.forEach(key -> batch.getBucket(key).getAsync());
        BatchResult<?> execute = batch.execute();
        return execute.getResponses();
    }


    /**
     * 设置key-value,同时返回旧值
     * 旧值不存在则返回 null
     */
    public Object getAndSet(String key, Object value, long seconds) {
        if (seconds <= 0) {
            seconds = DEFAULT_EXPIRE_TIME_SECONDS;
        }
        return getRedissonClient().getBucket(key).getAndSet(value, seconds, TimeUnit.SECONDS);
    }

    /**
     * 获取所有的指定前缀 keys
     * 例如: "test:*"
     */
    public Set<String> getKeys(String prefix) {
        Iterable<String> keysByPattern = getRedissonClient().getKeys().getKeysByPattern(prefix);
        Set<String> keys = new HashSet<>();
        for (String key : keysByPattern) {
            keys.add(key);
        }
        return keys;
    }

    /**
     * 删除key
     */
    public boolean removeKey(String key) {
        return getRedissonClient().getBucket(key).delete();
    }

    /**
     * 删除key
     */
    public void removeKeyAsync(String key) {
        getRedissonClient().getBucket(key).deleteAsync();

    }

    /**
     * 批量删除
     */
    public void removeKeyBatch(Collection<String> keys) {
        RBatch batch = getRedissonClient().createBatch();
        keys.forEach(key -> batch.getBucket(key).deleteAsync());
        batch.execute();
    }
}

定义FastJson序列化编码

继承实现Redisson的BaseCodec,自定义fastjson编码器

XML 复制代码
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.parser.ParserConfig;
import com.alibaba.fastjson.serializer.SerializerFeature;
import com.dyuproject.protostuff.Schema;
import com.dyuproject.protostuff.runtime.RuntimeSchema;

import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufAllocator;
import io.netty.buffer.ByteBufInputStream;
import io.netty.buffer.ByteBufOutputStream;

import org.redisson.client.codec.BaseCodec;
import org.redisson.client.handler.State;
import org.redisson.client.protocol.Decoder;
import org.redisson.client.protocol.Encoder;

import java.io.IOException;

public class FastJsonCodec extends BaseCodec {

    private static final Schema SCHEMA = RuntimeSchema.getSchema(ProtoStuffDataWrapper.class);

    @Override
    public Decoder<Object> getValueDecoder() {
        return decoder;
    }

    @Override
    public Encoder getValueEncoder() {
        return encoder;
    }

    private final Encoder encoder = new Encoder() {
        @Override
        public ByteBuf encode(Object in) throws IOException {
            ByteBuf out = ByteBufAllocator.DEFAULT.buffer();
            try {
                ByteBufOutputStream os = new ByteBufOutputStream(out);
                JSON.writeJSONString(os, in, SerializerFeature.WriteClassName);
                return os.buffer();
            } catch (IOException e) {
                // ByteBuf 只有在出现异常的时候才需要主动释放,因为否则重复释放会报错: IllegalReferenceCountException: refCnt: 0
                out.release();
                throw e;
            } catch (Exception e) {
                out.release();
                throw new IOException(e);
            }
        }
    };

    private final Decoder<Object> decoder = new Decoder<Object>() {
        public Object decode(ByteBuf buf, State state) throws IOException {
            // 添加待序列化对象的扫描包,如果不添加扫描包,无法实现序列化和反序列化
            // 另外待序列化的对象必须有一个默认的构造器,否则也无法实现序列化和反序列化。
            ParserConfig.getGlobalInstance().addAccept("redisson.entity");
            return JSON.parseObject(new ByteBufInputStream(buf), Object.class);
        }
    };
}

定义ProtoStuff序列化编码

注意,我们要先定义Protostuff 包装类,Protostuff 是基于POJO进行序列化和反序列化操作, 有时候对非定义POJO要用这个类辅助.

java 复制代码
**
 * Protostuff 包装类
 * Protostuff 是基于POJO进行序列化和反序列化操作, 有时候对非定义POJO要用这个类辅助.
 * 作用:
 * 1.主要用于对 Map、List、String、Enum 等进行序列化/反序列化
 * 2.对未知的 Object.class 可以用这个通用包装类
 */
class ProtoStuffDataWrapper<T> {
    private T data;

    public ProtoStuffDataWrapper() {
    }

    public ProtoStuffDataWrapper(T data) {
        super();
        this.data = data;
    }

    public T getData() {
        return data;
    }

    public void setData(T data) {
        this.data = data;
    }
}

实现Protostuff序列化

java 复制代码
import com.dyuproject.protostuff.LinkedBuffer;
import com.dyuproject.protostuff.ProtostuffIOUtil;
import com.dyuproject.protostuff.Schema;
import com.dyuproject.protostuff.runtime.RuntimeSchema;

import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufAllocator;
import io.netty.buffer.ByteBufInputStream;
import io.netty.buffer.ByteBufOutputStream;
import lombok.extern.slf4j.Slf4j;

import org.redisson.client.codec.BaseCodec;
import org.redisson.client.handler.State;
import org.redisson.client.protocol.Decoder;
import org.redisson.client.protocol.Encoder;

import java.io.IOException;

@Slf4j
public class ProtoStuffCodec extends BaseCodec {

    /**
     * 继承 Redisson Codec 实现后, 因为没有Object.Class信息,因此必须用一个封装类来定义 Schema, 否则无法 prostuff 序列化
     */
    private static final Schema SCHEMA = RuntimeSchema.getSchema(ProtoStuffDataWrapper.class);

    @Override
    public Decoder<Object> getValueDecoder() {
        return decoder;
    }

    @Override
    public Encoder getValueEncoder() {
        return encoder;
    }


    @SuppressWarnings({"unchecked", "rawtypes"})
    private final Encoder encoder = new Encoder() {
        @Override
        public ByteBuf encode(Object in) throws IOException {
            if (null == in) {
                throw new NullPointerException();
            }
            LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);
            ByteBuf out = ByteBufAllocator.DEFAULT.buffer();
            ByteBufOutputStream os = new ByteBufOutputStream(out);
            try {
                ProtostuffIOUtil.writeTo(os, new ProtoStuffDataWrapper(in), SCHEMA, buffer);
                return os.buffer();
            } catch (Exception e) {
                out.release();
                throw e;
            } finally {
                // 注意:finally 不要调用 release() 释放 ByteBuf, 因为重复释放会报错: io.netty.util.IllegalReferenceCountException: refCnt: 0
                // ProtostuffIOUtil.writeTo() 会调用 ByteBufOutputStream.write() 已经进行'软'置空
                buffer.clear();
            }
        }
    };

    @SuppressWarnings({"unchecked", "rawtypes"})
    private final Decoder<Object> decoder = new Decoder<Object>() {
        @Override
        public Object decode(ByteBuf buf, State state) throws IOException {
            ProtoStuffDataWrapper ProtoStuffDataWrapper = new ProtoStuffDataWrapper<>();
            ProtostuffIOUtil.mergeFrom(new ByteBufInputStream(buf), ProtoStuffDataWrapper, SCHEMA);
            return ProtoStuffDataWrapper.getData();
        }
    };
}

定义Jackson序列化编码

无需实现,Redisson自带:org.redisson.codec.JsonJacksonCodec

定义Kryo序列化编码

无需实现,Redisson自带:org.redisson.codec.Kryo5Codec

注入客户端 RedissonClient

每个序列化单独用一个客户端

java 复制代码
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import redisson.codec.FastJsonCodec;
import redisson.codec.ProtoStuffCodec;

import org.redisson.Redisson;
import org.redisson.api.RedissonClient;
import org.redisson.client.codec.Codec;
import org.redisson.codec.JsonJacksonCodec;
import org.redisson.codec.Kryo5Codec;
import org.redisson.config.ClusterServersConfig;
import org.redisson.config.Config;
import org.redisson.config.SingleServerConfig;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.util.CollectionUtils;

import java.util.List;

@Slf4j
@Data
@Configuration
@ConfigurationProperties(prefix = "spring.redis")
public class RedissonConfig {

    private Cluster cluster;

    private String host;

    private String port;

    private String password;

    @Data
    public static class Cluster {

        private List<String> nodes;

        /**
         * Maximum number of redirects to follow when executing commands across the cluster.
         */
        private Integer maxRedirects;
    }

    /**
     * RedissonClient bean
     */
    @Primary
    @Bean(name = "redissonClient", destroyMethod = "shutdown")
    public RedissonClient redissonClient() {
        return getRedissonClient(new JsonJacksonCodec());
    }

    @Bean(name = "kryoRedisson", destroyMethod = "shutdown")
    public RedissonClient kryoRedisson() {
        return getRedissonClient(new Kryo5Codec());
    }

    @Bean(name = "protoStuffRedisson", destroyMethod = "shutdown")
    public RedissonClient protoStuffRedisson() {
        return getRedissonClient(new ProtoStuffCodec());
    }

    @Bean(name = "fastJsonRedisson", destroyMethod = "shutdown")
    public RedissonClient fastJsonRedisson() {
        return getRedissonClient(new FastJsonCodec());
    }

    /**
     * 根据 codec 创建 RedissonClient
     *
     * @param codec codec
     * @return RedissonClient
     */
    public RedissonClient getRedissonClient(Codec codec) {
        Config redissonConfig = new Config();
        redissonConfig.setCodec(codec);
        // 设置'看门狗'续命时间, 默认30秒
        redissonConfig.setLockWatchdogTimeout(30 * 1000);

        if (null != cluster && !CollectionUtils.isEmpty(cluster.getNodes())) {
            List<String> nodes = cluster.getNodes();
            String[] clusterNodes = new String[nodes.size()];
            for (int i = 0; i < nodes.size(); i++) {
                clusterNodes[i] = "redis://" + nodes.get(i);
            }
            // 集群模式
            ClusterServersConfig clusterServersConfig = redissonConfig.useClusterServers();
            clusterServersConfig.addNodeAddress(clusterNodes)
                    .setPassword(password)
                    .setSlaveConnectionMinimumIdleSize(24)
                    .setMasterConnectionMinimumIdleSize(24);

        } else {
            // 单点模式
            SingleServerConfig singleServerConfig = redissonConfig.useSingleServer();
            singleServerConfig.setAddress("redis://" + host + ":" + port)
                    .setPassword(password);
        }
        return Redisson.create(redissonConfig);
    }
}

注入客户端Service

kryoRedisson 为例**,**其他的一样,把Qualifier的名字改一下即可(redissonClient、 kryoRedisson、 protoStuffRedisson、 fastJsonRedisson)

java 复制代码
import redisson.service.base.AbstractRedissonService;

import org.redisson.api.RedissonClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Component;


@Component
public class RedissonKryoService extends AbstractRedissonService {

    @Autowired
    @Qualifier("kryoRedisson")
    private RedissonClient redissonClient;

    @Override
    public RedissonClient getRedissonClient() {
        return redissonClient;
    }
}

结果测试

测试代码如下,运行即可,其他的序列化工具测试写法一样,替换service即可。

java 复制代码
import lombok.extern.slf4j.Slf4j;
import redisson.entity.Student;
import redisson.entity.StudentListObject;
import redisson.service.RedissonKryoService;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.util.StopWatch;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.Set;
import java.util.UUID;

@Slf4j
@RestController
public class KryoController {

    private static final long EXPIRE_TIME_SECONDS = 600;

    public static final String PREFIX = "test:kryo:studentList:";

    @Autowired
    private RedissonKryoService redissonKryoService;

    @GetMapping(value = "/redisson/kryo/set")
    public String studentSet() {
        Student student = new Student();
        student.setId(1);
        student.setAge(19);
        student.setName("zhangsan");
        redissonKryoService.set("kryo_student", student, EXPIRE_TIME_SECONDS);
        return "ok";
    }

    @GetMapping(value = "/redisson/kryo/get")
    public String studentGet() {
        return "" + redissonKryoService.get("kryo_student");
    }

    /**
     * 测试平均速度:
     * 序列化效率 比 JackSon、FastJson 好
     * 100次插入,总耗时:2184,平均耗时:21
     * 100次插入,总耗时:1714,平均耗时:17
     * 100次插入,总耗时:1627,平均耗时:16
     * 100次插入,总耗时:1641,平均耗时:16
     * 100次插入,总耗时:2451,平均耗时:24
     * 100次插入,总耗时:1995,平均耗时:19
     */
    @GetMapping(value = "/redisson/kryo/set/list")
    public String studentSetList() {
        // 为了测试准确, 每次插入前, 先清空旧数据
        Set<String> keys = redissonKryoService.getKeys(PREFIX + "*");
        redissonKryoService.removeKeyBatch(keys);
        log.info("清空 " + PREFIX + keys.size() + "个");

        // 创建插入对象(每个对象List有n个)
        int n = 100000;
        StudentListObject studentListObject = new StudentListObject();
        studentListObject.setName("studentListObject");
        studentListObject.setStudentList(StudentUtil.getRandomStudentList(n));

        // 循环插入100次,测试效率
        int count = 100;
        StopWatch stopWatch = new StopWatch();
        for (int i = 0; i < count; i++) {
            String uuid = UUID.randomUUID().toString().replaceAll("-", "").substring(0, 10);
            stopWatch.start("" + i);
            redissonKryoService.set(PREFIX + uuid, studentListObject, EXPIRE_TIME_SECONDS);
            stopWatch.stop();
        }
        // log.info(stopWatch.prettyPrint());
        log.info(count + "次插入,总耗时:{},平均耗时:{}", stopWatch.getTotalTimeMillis(), stopWatch.getTotalTimeMillis() / stopWatch.getTaskCount());
        return stopWatch.getTotalTimeMillis() + "";
    }

    /**
     * 测试平均速度:
     * 反序列化效率 比 JackSon、FastJson 好
     * 100次查询,总耗时:2657,平均耗时:26
     * 100次查询,总耗时:1962,平均耗时:19
     * 100次查询,总耗时:2055,平均耗时:20
     * 100次查询,总耗时:1980,平均耗时:19
     * 100次查询,总耗时:1910,平均耗时:19
     * 100次查询,总耗时:1968,平均耗时:19
     */
    @GetMapping(value = "/redisson/kryo/get/list")
    public String studentGetList() {
        // 获取所有key
        Set<String> keys = redissonKryoService.getKeys(PREFIX + "*");

        Object list = null;
        StopWatch stopWatch = new StopWatch();
        // 循环依次查询
        int i = 0;
        for (String key : keys) {
            stopWatch.start("" + i);
            list = redissonKryoService.get(key);
            stopWatch.stop();
            i++;
        }
        // log.info(stopWatch.prettyPrint());
        log.info(keys.size() + "次查询,总耗时:{},平均耗时:{}", stopWatch.getTotalTimeMillis(), stopWatch.getTotalTimeMillis() / stopWatch.getTaskCount());
        return list + "";
    }
}

总结

各个序列化效率总结,可以看我这篇文章:

https://blog.csdn.net/q258523454/article/details/129953047

jackson、fastjson、kryo、protobuf等序列化效率对比【全】_kryo jackson-CSDN博客

我们通过同时使用多个序列化工具,以及实际数据测试的效果来看。

综合性能推荐使用:kryoprotoStuff


原创不易,转载请注明出处:

https://blog.csdn.net/q258523454/article/details/129988906

相关推荐
cnsxjean3 分钟前
SpringBoot集成Minio实现上传凭证、分片上传、秒传和断点续传
java·前端·spring boot·分布式·后端·中间件·架构
hadage23313 分钟前
--- stream 数据流 java ---
java·开发语言
《源码好优多》31 分钟前
基于Java Springboot汽配销售管理系统
java·开发语言·spring boot
kingbal36 分钟前
SpringCloud:Injection of resource dependencies failed
后端·spring·spring cloud
小林想被监督学习1 小时前
Java后端如何进行文件上传和下载 —— 本地版
java·开发语言
Erosion20201 小时前
SPI机制
java·java sec
刘天远1 小时前
django实现paypal订阅记录
后端·python·django
ℳ₯㎕ddzོꦿ࿐1 小时前
Spring Boot集成MyBatis-Plus:自定义拦截器实现动态表名切换
spring boot·后端·mybatis
Clown951 小时前
go-zero(十) 数据缓存和Redis使用
redis·缓存·golang
逸风尊者2 小时前
开发也能看懂的大模型:RNN
java·后端·算法