Netty源码—2.Reactor线程模型一

大纲

1.关于NioEventLoop的问题整理

2.理解Reactor线程模型主要分三部分

3.NioEventLoop的创建

4.NioEventLoop的启动

1.关于NioEventLoop的问题整理

一.默认下Netty服务端起多少线程及何时启动?

答:默认是2倍CPU核数个线程。在调用EventExcutor的execute(task)方法时,会判断当前线程是否为Netty的Reactor线程,也就是判断当前线程是否为NioEventLoop对应的线程实体。如果是,则说明Netty的Reactor线程已经启动了。如果不是,则说明是外部线程调用EventExcutor的execute()方法。于是会先调用startThread()方法判断当前线程是否已被启动,如果还没有被启动就启动当前线程作为Netty的Reactor线程。

二.Netty是如何解决JDK空轮询的?

答:Netty会判断如果当前阻塞的一个Select()操作并没有花那么长时间,那么就说明此时有可能触发了空轮询Bug。默认情况下如果这个现象达到512次,那么就重建一个Selector,并且把之前Selector上所有的key重新移交到新Selector上。通过以上这种处理方式来避免JDK空轮询Bug。

三.Netty是如何保证异步串行无锁化的?

答:异步串行无锁化有两个场景。

场景一:拿到客户端一个Channel,不需要对该Channel进行同步,直接就可以多线程并发读写。

场景二:ChannelHandler里的所有操作都是线程安全的,不需要进行同步。

Netty在所有外部线程去调用EventLoop或者Channel的方法时,会通过inEventLoop()方法来判断出当前线程是外部线程(非NioEventLoop的线程实体)。在这种情况下,会把所有操作都封装成一个Task放入MPSC队列,然后在NioEventLoop的执行逻辑也就是run()方法里,这些Task会被逐个执行。

2.理解Reactor线程模型主要分三部分

一.NioEventLoop的创建

二.NioEventLoop的启动

三.NioEventLoop的执行

3.NioEventLoop的创建

(1)创建入口

(2)确定NioEventLoop的个数

(3)NioEventLoopGroup的创建流程

(4)创建线程执行器ThreadPerTaskExecutor

(5)创建NioEventLoop

(6)创建线程选择器EventExecutorChooser

(7)NioEventLoopGroup的创建总结

(1)创建入口

ini 复制代码
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup();

(2)确定NioEventLoop的个数

由NioEventLoopGroup的构造方法来确定NioEventLoop的个数。如果NioEventLoopGroup没有传递构造参数,那么NioEventLoop线程的个数为CPU核数的2倍。如果NioEventLoopGroup传递了参数n,那么NioEventLoop线程的个数就是n。

(3)NioEventLoopGroup的创建流程

NioEventLoopGroup的构造方法会触发创建流程。

一.创建线程执行器ThreadPerTaskExecutor

每次调用ThreadPerTaskExecutor.execute()方法时都会创建一个线程。

二.创建NioEventLoop

NioEventLoop对应NioEventLoopGroup线程池里的线程,NioEventLoopGroup的构造方法会用一个for循环通过调用newChild()方法来创建NioEventLoop线程。

三.创建线程选择器EventExecutorChooser

线程选择器的作用是用于给每个新连接分配一个NioEventLoop线程,也就是从NioEventLoopGroup线程池中选择一个NioEventLoop线程来处理新连接。

scala 复制代码
//MultithreadEventLoopGroup implementations which is used for NIO Selector based Channels.
public class NioEventLoopGroup extends MultithreadEventLoopGroup {

    //Create a new instance using the default number of threads, 
    //the default ThreadFactory and the SelectorProvider which is returned by SelectorProvider#provider().
    public NioEventLoopGroup() {
        this(0);
    }
    
    //Create a new instance using the specified number of threads, 
    //ThreadFactory and the SelectorProvider which is returned by SelectorProvider#provider().
    public NioEventLoopGroup(int nThreads) {
        this(nThreads, (Executor) null);
    }
    
    public NioEventLoopGroup(int nThreads, Executor executor) {
        this(nThreads, executor, SelectorProvider.provider());
    }
    
    public NioEventLoopGroup(int nThreads, Executor executor, final SelectorProvider selectorProvider) {
        this(nThreads, executor, selectorProvider, DefaultSelectStrategyFactory.INSTANCE);
    }
    
    public NioEventLoopGroup(int nThreads, Executor executor, final SelectorProvider selectorProvider, final SelectStrategyFactory selectStrategyFactory) { 
        super(nThreads, executor, selectorProvider, selectStrategyFactory, RejectedExecutionHandlers.reject());
    }
    ...
}

//Abstract base class for EventLoopGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventLoopGroup extends MultithreadEventExecutorGroup implements EventLoopGroup {
    private static final InternalLogger logger = InternalLoggerFactory.getInstance(MultithreadEventLoopGroup.class);
    private static final int DEFAULT_EVENT_LOOP_THREADS;
    static {
        DEFAULT_EVENT_LOOP_THREADS = Math.max(1, SystemPropertyUtil.getInt("io.netty.eventLoopThreads", Runtime.getRuntime().availableProcessors() * 2));
        if (logger.isDebugEnabled()) logger.debug("-Dio.netty.eventLoopThreads: {}", DEFAULT_EVENT_LOOP_THREADS);
    }

    protected MultithreadEventLoopGroup(int nThreads, Executor executor, Object... args) {
        super(nThreads == 0 ? DEFAULT_EVENT_LOOP_THREADS : nThreads, executor, args);
    }
    ...
}

//Abstract base class for EventExecutorGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {
    private final EventExecutor[] children;
    private final EventExecutorChooserFactory.EventExecutorChooser chooser;
    ...
    //Create a new instance.
    protected MultithreadEventExecutorGroup(int nThreads, Executor executor, Object... args) {
        this(nThreads, executor, DefaultEventExecutorChooserFactory.INSTANCE, args);
    }
    
    //Create a new instance.
    //@param nThreads,the number of threads that will be used by this instance.
    //@param executor,the Executor to use, or null if the default should be used.
    //@param chooserFactory,the EventExecutorChooserFactory to use.
    //@param args,arguments which will passed to each #newChild(Executor, Object...) call
    protected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {
        if (nThreads <= 0) throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));
        //1.创建ThreadPerTaskExecutor线程执行器
        if (executor == null) executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());
        //2.创建NioEventLoop
        children = new EventExecutor[nThreads];
        for (int i = 0; i < nThreads; i ++) {
            ...
            //创建每一个NioEventLoop时,都会调用newChild()方法给每一个NioEventLoop配置一些核心参数
            //传入线程执行器executor去创建NioEventLoop
            children[i] = newChild(executor, args);
        }
        //3.创建线程选择器
        chooser = chooserFactory.newChooser(children);
        ...
    }
    ...
}

创建NioEventLoopGroup的脉络如下:

scss 复制代码
new NioEventLoopGroup() //线程组,线程个数默认为2 * CPU核数
  new ThreadPerTaskExecutor() //创建线程执行器,作用是负责创建NioEventLoop对应的线程
  for(...) { newChild() } //构造NioEventLoop,创建NioEventLoop线程组
  chooserFactory.newChooser() //线程选择器,用于给每个新连接分配一个NioEventLoop线程

(4)创建线程执行器ThreadPerTaskExecutor

ThreadPerTaskExecutor的作用是:每次调用它的execute()方法执行Runnable任务时,都会通过threadFactory.newThread()创建出一个线程,然后把要执行的Runnable任务传递进该线程进行执行。

其中成员变量threadFactory是在传参给ThreadPerTaskExecutor的构造方法时,由newDefaultThreadFactory()方法构建的,也就是一个DefaultThreadFactory对象。

所以线程执行器ThreadPerTaskExecutor在通过threadFactory.newThread()创建线程时,其实就是调用DefaultThreadFactory的newThread()方法。

而DefaultThreadFactory.newThread()方法创建出来的线程实体,是Netty经过优化之后的FastThreadLocalThread对象,这个线程实体在操作ThreadLocal时,要比JDK快。

ThreadPerTaskExecutor线程执行器总结:

一.每次执行ThreadPerTaskExecutor的execute()方法时,都会创建出一个FastThreadLocalThread的线程实体,所以Netty的线程实体都是由ThreadPerTaskExecutor创建的。

二.FastThreadLocalThread线程实体的命名规则是:nioEventLoop-自增的线程池编号-自增的线程数编号。

java 复制代码
//Abstract base class for EventExecutorGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {
    private final EventExecutor[] children;
    private final EventExecutorChooserFactory.EventExecutorChooser chooser;
    ...    
    //Create a new instance.
    protected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {
        if (nThreads <= 0) throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));
        //1.创建ThreadPerTaskExecutor线程执行器
        if (executor == null) executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());
        //2.创建NioEventLoop
        children = new EventExecutor[nThreads];
        for (int i = 0; i < nThreads; i ++) {
            ...
            //创建每一个NioEventLoop时,都会调用newChild()方法给每一个NioEventLoop配置一些核心参数
            //传入线程执行器executor去创建NioEventLoop
            children[i] = newChild(executor, args);
        }
        //3.创建线程选择器
        chooser = chooserFactory.newChooser(children);
        ...
    }
    
    protected ThreadFactory newDefaultThreadFactory() {
        //getClass()是获取该方法所属的对象类型,也就是NioEventLoopGroup类型
        //因为是通过NioEventLoopGroup的构造方法层层调用到这里的
        return new DefaultThreadFactory(getClass());
    }
    ...
}

public final class ThreadPerTaskExecutor implements Executor {
    private final ThreadFactory threadFactory;
    public ThreadPerTaskExecutor(ThreadFactory threadFactory) {
        if (threadFactory == null) throw new NullPointerException("threadFactory");
        this.threadFactory = threadFactory;
    }
    
    @Override
    public void execute(Runnable command) {
        //调用DefaultThreadFactory的newThread()方法执行Runnable任务
        threadFactory.newThread(command).start();
    }
}

//A ThreadFactory implementation with a simple naming rule.
public class DefaultThreadFactory implements ThreadFactory {
    private static final AtomicInteger poolId = new AtomicInteger();
    private final AtomicInteger nextId = new AtomicInteger();
    private final boolean daemon;
    private final int priority;
    protected final ThreadGroup threadGroup;
    ...
    public DefaultThreadFactory(Class<?> poolType) {
        this(poolType, false, Thread.NORM_PRIORITY);
    }
    
    public DefaultThreadFactory(Class<?> poolType, boolean daemon, int priority) {
        //toPoolName()方法会把NioEventLoopGroup的首字母变成小写
        this(toPoolName(poolType), daemon, priority);
    }
    
    public DefaultThreadFactory(String poolName, boolean daemon, int priority) {
        this(poolName, daemon, priority, 
        System.getSecurityManager() == null? Thread.currentThread().getThreadGroup() : System.getSecurityManager().getThreadGroup());
    }
    
    public DefaultThreadFactory(String poolName, boolean daemon, int priority, ThreadGroup threadGroup) {
        ...
        //prefix用来标记线程名字的前缀
        prefix = poolName + '-' + poolId.incrementAndGet() + '-';
        this.daemon = daemon;
        this.priority = priority;
        this.threadGroup = threadGroup;
    }
    
    @Override
    public Thread newThread(Runnable r) {
        Thread t = newThread(new DefaultRunnableDecorator(r), prefix + nextId.incrementAndGet());
        if (t.isDaemon()) {
            if (!daemon) t.setDaemon(false);
        } else {
            if (daemon) t.setDaemon(true);
        }
        if (t.getPriority() != priority) t.setPriority(priority);
        return t;
    }
    
    protected Thread newThread(Runnable r, String name) {
        return new FastThreadLocalThread(threadGroup, r, name);
    }
    ...
}

(5)创建NioEventLoop

说明一:

由MultithreadEventExecutorGroup的构造方法可知,Netty会使用for循环 + newChild()方法来创建nThreads个NioEventLoop,而且一个NioEventLoop对应一个线程实体FastThreadLocalThread。

scala 复制代码
//Abstract base class for EventExecutorGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {
    private final EventExecutor[] children;
    private final EventExecutorChooserFactory.EventExecutorChooser chooser;
    ...    
    //Create a new instance.
    protected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {
        if (nThreads <= 0) throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));
        //1.创建ThreadPerTaskExecutor线程执行器
        if (executor == null) executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());
        //2.创建NioEventLoop
        children = new EventExecutor[nThreads];
        for (int i = 0; i < nThreads; i ++) {
            ...
            //创建每一个NioEventLoop时,都会调用newChild()方法给每一个NioEventLoop配置一些核心参数
            //传入线程执行器executor去创建NioEventLoop
            children[i] = newChild(executor, args);
        }
        //3.创建线程选择器
        chooser = chooserFactory.newChooser(children);
        ...
    }
    
    //Create a new EventExecutor which will later then accessible via the #next() method.
    //This method will be called for each thread that will serve this MultithreadEventExecutorGroup.
    protected abstract EventExecutor newChild(Executor executor, Object... args) throws Exception;
    ...
}

说明二:

MultithreadEventExecutorGroup的newChild()抽象方法是由NioEventLoopGroup实现的,所以在执行NioEventLoopGroup的默认构造方法时也会执行其newChild()方法。

NioEventLoopGroup的newChild()方法需要传递一个executor参数,该参数就是执行NioEventLoopGroup构造方法开始时创建的线程执行器,之后newChild()方法会返回一个新创建的NioEventLoop对象。

scala 复制代码
//MultithreadEventLoopGroup implementations which is used for NIO Selector based Channels.
public class NioEventLoopGroup extends MultithreadEventLoopGroup {
    ...
    @Override
    protected EventLoop newChild(Executor executor, Object... args) throws Exception {
        //executor是执行NioEventLoopGroup构造方法开始时创建的线程执行器ThreadPerTaskExecutor
        //this指的是NioEventLoopGroup,表示新创建的NioEventLoop对象归属哪个NioEventLoopGroup
        return new NioEventLoop(this, executor, (SelectorProvider) args[0],
            ((SelectStrategyFactory) args[1]).newSelectStrategy(), (RejectedExecutionHandler) args[2]);
    }
    ...
}

说明三:

创建NioEventLoop对象时,NioEventLoop的构造方法会通过调用其openSelector()方法来创建一个Selector,所以一个Selector就和一个NioEventLoop绑定了,而一个Selector可以将多个连接绑定在一起来负责监听这些连接的读写事件。

在NioEventLoop的openSelector()方法中,Netty会通过反射对Selector底层的数据结构进行优化(Hash Set => 数组)。

scala 复制代码
//SingleThreadEventLoop implementation which register the Channel's to a Selector and so does the multi-plexing of these in the event loop.
public final class NioEventLoop extends SingleThreadEventLoop {
    //The NIO Selector.
    Selector selector;
    private final SelectorProvider provider;
    private final SelectStrategy selectStrategy;
    ...
    NioEventLoop(NioEventLoopGroup parent, Executor executor, SelectorProvider selectorProvider,
        SelectStrategy strategy, RejectedExecutionHandler rejectedExecutionHandler) {
        //调用其父类SingleThreadEventLoop的构造方法
        super(parent, executor, false, DEFAULT_MAX_PENDING_TASKS, rejectedExecutionHandler);
        if (selectorProvider == null) throw new NullPointerException("selectorProvider");
        if (strategy == null) throw new NullPointerException("selectStrategy");
        this.provider = selectorProvider;
        this.selector = openSelector();//创建一个Selector
        this.selectStrategy = strategy;
    }
    
    private Selector openSelector() {
        final Selector selector;
        try {
            selector = provider.openSelector();
            ...
        } catch(IOException e) {
            ...
        }
        ...
        return selector;
    }
    ...
}

说明四:

NioEventLoop的构造方法还会调用其父类的父类SingleThreadEventExecutor的构造方法。SingleThreadEventExecutor的构造方法里有两个关键的操作:一是把线程执行器保存起来,因为后面创建NioEventLoop对应的线程时要用到。二是创建一个MPSC任务队列,因为Netty中所有异步执行的本质都是通过该任务队列来协调完成的。

scala 复制代码
//Abstract base class for EventLoops that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventLoop extends SingleThreadEventExecutor implements EventLoop {
    private final Queue<Runnable> tailTasks;
    ...
    protected SingleThreadEventLoop(EventLoopGroup parent, Executor executor, 
        boolean addTaskWakesUp, int maxPendingTasks, RejectedExecutionHandler rejectedExecutionHandler) {
        //调用其父类SingleThreadEventExecutor的构造方法
        super(parent, executor, addTaskWakesUp, maxPendingTasks, rejectedExecutionHandler);
        //调用父类SingleThreadEventExecutor的newTaskQueue()方法
        tailTasks = newTaskQueue(maxPendingTasks);
    }
    ...
}

//Abstract base class for OrderedEventExecutor's that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {
    private final boolean addTaskWakesUp;
    private final Executor executor;
    private final int maxPendingTasks;
    private final Queue<Runnable> taskQueue;
    private final RejectedExecutionHandler rejectedExecutionHandler;
    ...
    //Create a new instance
    protectedSingleThreadEventExecutor(EventExecutorGroup parent, ThreadFactory threadFactory,
          boolean addTaskWakesUp, int maxPendingTasks, RejectedExecutionHandler rejectedHandler) {
        this(parent, new ThreadPerTaskExecutor(threadFactory), addTaskWakesUp, maxPendingTasks, rejectedHandler);
    }
    
    //Create a new instance
    //@param parent,the EventExecutorGroup which is the parent of this instance and belongs to it
    //@param executor,the Executor which will be used for executing
    //@param addTaskWakesUp,true if and only if invocation of #addTask(Runnable) will wake up the executor thread
    //@param maxPendingTasks,the maximum number of pending tasks before new tasks will be rejected.
    //@param rejectedHandler,the RejectedExecutionHandler to use.
    protected SingleThreadEventExecutor(EventExecutorGroup parent, Executor executor,
          boolean addTaskWakesUp, int maxPendingTasks, RejectedExecutionHandler rejectedHandler) {
        super(parent);
        this.addTaskWakesUp = addTaskWakesUp;
        this.maxPendingTasks = Math.max(16, maxPendingTasks);
        //关键操作一:把线程执行器保存起来
        this.executor = ObjectUtil.checkNotNull(executor, "executor");
        //关键操作二:创建一个MPSC任务队列
        this.taskQueue = newTaskQueue(this.maxPendingTasks);
        this.rejectedExecutionHandler = ObjectUtil.checkNotNull(rejectedHandler, "rejectedHandler");
    }
    
    //Create a new Queue which will holds the tasks to execute.
    //NioEventLoop会重写这个newTaskQueue()方法
    protected Queue<Runnable> newTaskQueue(int maxPendingTasks) {
        return new LinkedBlockingQueue<Runnable>(maxPendingTasks);
    }
    ...
}

//SingleThreadEventLoop implementation which register the Channel's to a Selector and so does the multi-plexing of these in the event loop.
public final class NioEventLoop extends SingleThreadEventLoop {
    ...
    //创建一个MPSC任务队列
    @Override
    protected Queue<Runnable> newTaskQueue(int maxPendingTasks) {
        //This event loop never calls takeTask()
        return PlatformDependent.newMpscQueue(maxPendingTasks);
    }
    ...
}

MPSC队列也就是多生产者单消费者队列。单消费者是指某个NioEventLoop对应的线程(执行其run()方法的那个线程)。多生产者就是这个NioEventLoop对应的线程之外的线程,通常情况下就是我们的业务线程。比如,一些线程在调用writeAndFlush()方法时可以不用考虑线程安全而随意调用,那么这些线程就是多生产者。

MPSC队列是通过JCTools这个工具包来实现的,Netty的高性能很大程度上要归功于这个工具包。MPSC的全称是Muti Producer Single Consumer。Muti Producer对应的是外部线程,Single Consumer对应的是Netty的NioEventLoop线程。外部线程在执行Netty的一些任务时,如果判断不是由NioEventLoop对应的线程执行的,就会直接放入一个任务队列里,然后由一个NioEventLoop对应的线程去执行。

创建NioEventLoop总结:

NioEventLoopGroup的newChild()方法创建NioEventLoop时做了三项事情:一.创建一个Selector用于轮询注册到该NioEventLoop上的连接,二.创建一个MPSC任务队列,三.保存线程执行器到NioEventLoop。

(6)创建线程选择器EventExecutorChooser

说明一:

在传统的BIO编程中,一个新连接被创建后,通常需要给这个连接绑定一个Selector,之后这个连接的整个生命周期都由这个Selector管理。

说明二:

创建NioEventLoop时会创建一个Selector,所以一个Selector会对应一个NioEventLoop,一个NioEventLoop上会有一个Selector。线程选择器的作用就是为一个连接在NioEventLoopGroup中选择一个NioEventLoop,从而将该连接绑定到这个NioEventLoop的Selector上。

说明三:

根据MultithreadEventExecutorGroup的构造方法,会使用DefaultEventExecutorChooserFactory的newChooser()方法来创建线程选择器。创建好线程选择器EventExecutorChooser之后,便可以通过其next()方法获取一个NioEventLoop。

Netty通过判断NioEventLoopGroup中的NioEventLoop个数是否是2的幂来创建不同的线程选择器。但不管是哪一种选择器,最终效果都是从第一个NioEventLoop开始遍历到最后一个NioEventLoop,然后再从第一个NioEventLoop开始,如此循环。

java 复制代码
//Abstract base class for EventExecutorGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {
    private final EventExecutor[] children;
    private final EventExecutorChooserFactory.EventExecutorChooser chooser;
    ...
    //Create a new instance.
    protected MultithreadEventExecutorGroup(int nThreads, Executor executor, Object... args) {
        this(nThreads, executor, DefaultEventExecutorChooserFactory.INSTANCE, args);
    }
    
    //Create a new instance.
    //@param nThreads,the number of threads that will be used by this instance.
    //@param executor,the Executor to use, or null if the default should be used.
    //@param chooserFactory,the EventExecutorChooserFactory to use.
    //@param args,arguments which will passed to each #newChild(Executor, Object...) call
    protected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {
        if (nThreads <= 0) throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));
        //1.创建ThreadPerTaskExecutor线程执行器
        if (executor == null) executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());
        //2.创建NioEventLoop
        children = new EventExecutor[nThreads];
        for (int i = 0; i < nThreads; i ++) {
            ...
            //创建每一个NioEventLoop时,都会调用newChild()方法给每一个NioEventLoop配置一些核心参数
            //传入线程执行器executor去创建NioEventLoop
            children[i] = newChild(executor, args);
        }
        //3.创建线程选择器,chooserFactory就是传入的DefaultEventExecutorChooserFactory实例
        chooser = chooserFactory.newChooser(children);
        ...
    }
    ...
}

//Default implementation which uses simple round-robin to choose next EventExecutor.
@UnstableApi
public final class DefaultEventExecutorChooserFactory implements EventExecutorChooserFactory {
    public static final DefaultEventExecutorChooserFactory INSTANCE = new DefaultEventExecutorChooserFactory();
    private DefaultEventExecutorChooserFactory() { }
    
    @SuppressWarnings("unchecked")
    @Override
    public EventExecutorChooser newChooser(EventExecutor[] executors) {
        if (isPowerOfTwo(executors.length)) {
            //如果NioEventLoop个数是2的幂,则进行位与运算
            return new PowerOfTowEventExecutorChooser(executors);
        } else {
            //如果NioEventLoop个数不是2的幂,则进行普通取模运算
            return new GenericEventExecutorChooser(executors);
        }
    }

    private static boolean isPowerOfTwo(int val) {
        return (val & -val) == val;
    }

    private static final class PowerOfTowEventExecutorChooser implements EventExecutorChooser {
        private final AtomicInteger idx = new AtomicInteger();
        private final EventExecutor[] executors;
        PowerOfTowEventExecutorChooser(EventExecutor[] executors) {
            this.executors = executors;
        }
        @Override
        public EventExecutor next() {
            return executors[idx.getAndIncrement() & executors.length - 1];
        }
    }

    private static final class GenericEventExecutorChooser implements EventExecutorChooser {
        private final AtomicInteger idx = new AtomicInteger();
        private final EventExecutor[] executors;
        GenericEventExecutorChooser(EventExecutor[] executors) {
            this.executors = executors;
        }
        @Override
        public EventExecutor next() {
            return executors[Math.abs(idx.getAndIncrement() % executors.length)];
        }
    }
}

说明四:

创建NioEventLoopGroup的最后一个步骤就是创建线程选择器chooser,创建线程选择器的流程如下:

scss 复制代码
chooserFactory.newChooser() //创建线程选择器的入口,chooser的作用就是为新连接绑定一个NioEventLoop
  DefaultEventExecutorChooserFactory.isPowerOfTwo() //判断NioEventLoop个数是否为2的幂
    PowerOfTowEventExecutorChooser //优化
      index++ & (length - 1) //位与运算
    GenericEventExecutorChooser //普通
      abs(index++ % length) //取模运算

(7)NioEventLoopGroup的创建总结

默认情况下,NioEventLoopGroup会创建2倍CPU核数个NioEventLoop。一个NioEventLoop和一个Selector以及一个MPSC任务队列一一对应。

NioEventLoop线程的命名规则是nioEventLoopGroup-xx-yy,其中xx表示全局第xx个NioEventLoopGroup线程池,yy表示这个NioEventLoop在这个NioEventLoopGroup中是属于第yy个。

线程选择器chooser的作用是为一个连接选择一个NioEventLoop,可通过线程选择器的next()方法返回一个NioEventLoop。如果NioEventLoop的个数为2的幂,则next()方法会使用位与运算进行优化。

一个NioEventLoopGroup会有一个线程执行器executor、一个线程选择器chooser、一个数组children存放2倍CPU核数个NioEventLoop。

4.NioEventLoop的启动

(1)启动NioEventLoop的两大入口

(2)判断当前线程是否是NioEventLoop线程

(3)创建一个线程并启动

(4)NioEventLoop的启动总结

(1)启动NioEventLoop的两大入口

入口一:服务端启动,注册服务端Channel到Selector时

入口二:新连接接入,通过chooser绑定一个NioEventLoop时

下面先看入口一:

调用ServerBootstrap的bind()方法其实会调用AbstractBootstrap的doBind()方法,然后会调用AbstractBootstrap的initAndRegister()方法,接着执行config().group().register(channel)注册服务端Channel。最后会逐层深入调用到AbstractChannel.AbstractUnsafe的register()方法,来启动一个NioEventLoop将服务端Channel注册到Selector上。

scala 复制代码
//Bootstrap sub-class which allows easy bootstrap of ServerChannel
public class ServerBootstrap extends AbstractBootstrap<ServerBootstrap, ServerChannel> {
    ...
    ...
}

//AbstractBootstrap is a helper class that makes it easy to bootstrap a Channel. 
//It support method-chaining to provide an easy way to configure the AbstractBootstrap.
//When not used in a ServerBootstrap context, the #bind() methods are useful for connectionless transports such as datagram (UDP).
public abstract class AbstractBootstrap<B extends AbstractBootstrap<B, C>, C extends Channel> implements Cloneable {
    ...
    //Create a new Channel and bind it.
    public ChannelFuture bind(int inetPort) {
        //首先根据端口号创建一个InetSocketAddress对象,然后调用重载方法bind()
        return bind(new InetSocketAddress(inetPort));
    }
    
    //Create a new Channel and bind it.
    public ChannelFuture bind(SocketAddress localAddress) {
        //验证服务启动需要的必要参数
        validate();
        if (localAddress == null) throw new NullPointerException("localAddress");
        return doBind(ObjectUtil.checkNotNull(localAddress, "localAddress"));
    }
    
    private ChannelFuture doBind(final SocketAddress localAddress) {
        final ChannelFuture regFuture = initAndRegister();//1.初始化和注册Channel
        final Channel channel = regFuture.channel();
        ...
        doBind0(regFuture, channel, localAddress, promise);//2.绑定服务端端口
        ...
        return promise;
    }
    
    final ChannelFuture initAndRegister() {
        Channel channel = null;
        ...
        //1.创建服务端Channel
        channel = channelFactory.newChannel();
        //2.初始化服务端Channel
        init(channel);
        ...
        //3.注册服务端Channel,比如通过NioEventLoopGroup的register()方法进行注册
        ChannelFuture regFuture = config().group().register(channel);
        ...
        return regFuture;
    }
    ...
}

//Bootstrap sub-class which allows easy bootstrap of ServerChannel
public class ServerBootstrap extends AbstractBootstrap<ServerBootstrap, ServerChannel> {
    private final ServerBootstrapConfig config = new ServerBootstrapConfig(this);
    ...
    @Override
    public final ServerBootstrapConfig config() {
        return config;
    }
    ...
}

public abstract class AbstractBootstrapConfig<B extends AbstractBootstrap<B, C>, C extends Channel> {
    protected final B bootstrap;
    ...
    protected AbstractBootstrapConfig(B bootstrap) {
        this.bootstrap = ObjectUtil.checkNotNull(bootstrap, "bootstrap");
    }
    
    //Returns the configured EventLoopGroup or null if non is configured yet.
    public final EventLoopGroup group() {
        //比如返回一个NioEventLoopGroup对象
        return bootstrap.group();
    }
    ...
}

//MultithreadEventLoopGroup implementations which is used for NIO Selector based Channels.
public class NioEventLoopGroup extends MultithreadEventLoopGroup {
    ...
    ...
}

//Abstract base class for EventLoopGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventLoopGroup extends MultithreadEventExecutorGroup implements EventLoopGroup {
    ...
    @Override
    public ChannelFuture register(Channel channel) {
        //先通过next()方法获取一个NioEventLoop,然后通过NioEventLoop.register()方法注册服务端Channel
        return next().register(channel);
    }
    
    @Override
    public EventLoop next() {
        return (EventLoop) super.next();
    }
    ...
}

//Abstract base class for EventExecutorGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {
    private final EventExecutorChooserFactory.EventExecutorChooser chooser;
    ...
    @Override
    public EventExecutor next() {
        //通过线程选择器chooser选择一个NioEventLoop
        return chooser.next();
    }    
    ...
}

//SingleThreadEventLoop implementation which register the Channel's to a Selector and so does the multi-plexing of these in the event loop.
public final class NioEventLoop extends SingleThreadEventLoop {
    ...
    ...
}

//Abstract base class for EventLoops that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventLoop extends SingleThreadEventExecutor implements EventLoop {
    ...
    @Override
    public ChannelFuture register(Channel channel) {
        return register(new DefaultChannelPromise(channel, this));
    }

    @Override
    public ChannelFuture register(final ChannelPromise promise) {
        ObjectUtil.checkNotNull(promise, "promise");
        //调用AbstractUnsafe的register()方法
        promise.channel().unsafe().register(this, promise);
        return promise;
    }
    ...
}

//A skeletal Channel implementation.
public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {
    private volatile EventLoop eventLoop;
    ...
    //Unsafe implementation which sub-classes must extend and use.
    protected abstract class AbstractUnsafe implements Unsafe {
        ...
        @Override
        public final void register(EventLoop eventLoop, final ChannelPromise promise) {
            ...
            //绑定事件循环器,即绑定一个NioEventLoop到该Channel上
            AbstractChannel.this.eventLoop = eventLoop;
            //注册Selector,并启动一个NioEventLoop
            if (eventLoop.inEventLoop()) {
                register0(promise);
            } else {
                ...
                //通过启动这个NioEventLoop线程来调用register0()方法将这个服务端Channel注册到Selector上
                //其实执行的是SingleThreadEventExecutor的execute()方法
                eventLoop.execute(new Runnable() {
                    @Override
                    public void run() {
                        register0(promise);
                    }
                });
                ...
            }
        }
    }
    ...
}

(2)判断当前线程是否是NioEventLoop线程

调用NioEventLoop的inEventLoop()方法可以判断当前线程是否是Netty的Reactor线程,也就是NioEventLoop对应的线程实体。NioEventLoop的线程实体被创建之后,会将该线程实体保存到NioEventLoop父类的成员变量thread中。

服务端启动、注册服务端Channel到Selector,执行到AbstractUnsafe.register()方法中的eventLoop.inEventLoop()代码时,会将main方法对应的主线程传递进来与this.thread进行比较。由于this.thread此时并未赋值,所以为空,因此inEventLoop()方法返回false,于是便会执行eventLoop.execute()代码创建一个线程并启动。

scala 复制代码
//SingleThreadEventLoop implementation which register the Channel's 
//to a Selector and so does the multi-plexing of these in the event loop.
public final class NioEventLoop extends SingleThreadEventLoop {
    ...
    ...
}

//Abstract base class for EventLoops that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventLoop extends SingleThreadEventExecutor implements EventLoop {
    ...
    ...
}

//Abstract base class for OrderedEventExecutor's that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {
    ...
    ...
}

//Abstract base class for EventExecutors that want to support scheduling.
public abstract class AbstractScheduledEventExecutor extends AbstractEventExecutor {
    ...
    ...
}

//Abstract base class for {@link EventExecutor} implementations.
public abstract class AbstractEventExecutor extends AbstractExecutorService implements EventExecutor {
    ...
    @Override
    public boolean inEventLoop() {
        //注册服务端Channel时是通过主线程进行注册的,Thread.currentThread()对应的就是main线程
        //调用SingleThreadEventExecutor.inEventLoop()方法
        return inEventLoop(Thread.currentThread());
    }
    ...
}

//Abstract base class for OrderedEventExecutor's that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {
    private volatile Thread thread;
    ...
    @Override
    public boolean inEventLoop(Thread thread) {
        return thread == this.thread;//此时线程还没创建,this.thread为null
    }
    ...
}

(3)创建一个线程并启动

AbstractUnsafe.register()方法准备将服务端Channel注册到Selector上时,首先在判断条件中执行eventLoop.inEventLoop()代码发现为false,于是便执行eventLoop.execute()代码创建一个线程并启动它去执行注册任务。而执行eventLoop.execute()代码其实就是调用SingleThreadEventExecutor的execute()方法。

java 复制代码
//Abstract base class for EventLoops that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventLoop extends SingleThreadEventExecutor implements EventLoop {
    ...
    @Override
    public ChannelFuture register(Channel channel) {
        return register(new DefaultChannelPromise(channel, this));
    }

    @Override
    public ChannelFuture register(final ChannelPromise promise) {
        ObjectUtil.checkNotNull(promise, "promise");
        //调用AbstractUnsafe的register()方法,并把NioEventLoop自己当作参数传入
        promise.channel().unsafe().register(this, promise);
        return promise;
    }
    ...
}

//A skeletal Channel implementation.
public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {
    private volatile EventLoop eventLoop;
    ...
    //Unsafe implementation which sub-classes must extend and use.
    protected abstract class AbstractUnsafe implements Unsafe {
        ...
        @Override
        public final void register(EventLoop eventLoop, final ChannelPromise promise) {
            ...
            //绑定事件循环器,即绑定一个NioEventLoop到该Channel上
            AbstractChannel.this.eventLoop = eventLoop;
            //注册Selector,并启动一个NioEventLoop
            if (eventLoop.inEventLoop()) {
                register0(promise);
            } else {
                ...
                //通过启动这个NioEventLoop线程来调用register0()方法将这个服务端Channel注册到Selector上
                //其实执行的是SingleThreadEventExecutor的execute()方法
                eventLoop.execute(new Runnable() {
                    @Override
                    public void run() {
                        register0(promise);
                    }
                });
                ...
            }
        }
    }
    ...
}

//Abstract base class for OrderedEventExecutor's that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {
    private volatile Thread thread;
    //创建NioEventLoop时会通过构造方法传入NioEventLoopGroup的线程执行器executor
    private final Executor executor;
    ...
    @Override
    public void execute(Runnable task) {
        if (task == null) throw new NullPointerException("task");
        boolean inEventLoop = inEventLoop();
        //判断当前线程是否是Netty的Reactor线程
        if (inEventLoop) {
            addTask(task);
        } else {
            startThread();
            addTask(task);
            if (isShutdown() && removeTask(task)) reject();
        }
        if (!addTaskWakesUp && wakesUpForTask(task)) wakeup(inEventLoop);
    }
    
    private void startThread() {
        //判断Reactor线程有没有被启动;如果没有被启动,则通过CAS调用doStartThread()方法启动线程
        if (STATE_UPDATER.get(this) == ST_NOT_STARTED) {
            if (STATE_UPDATER.compareAndSet(this, ST_NOT_STARTED, ST_STARTED)) {
                doStartThread();
            }
        }
    }

    private void doStartThread() {
        assert thread == null;
        //executor.execute()方法会创建出一个FastThreadLocalThread线程来执行Runnable任务
        //所以在Runnable的run()方法中,Thread.currentThread()指的是这个FastThreadLocalThread线程
        executor.execute(new Runnable() {
            @Override
            public void run() {
                //Thread.currentThread()指的是FastThreadLocalThread线程
                thread = Thread.currentThread();
                ...
                SingleThreadEventExecutor.this.run();//启动线程
                ...
            }
        });
    }
    
    //具体的run()方法由子类比如NioEventLoop来实现
    protected abstract void run();
    ...
}

SingleThreadEventExecutor的execute()方法的说明如下:

一.这个方法也可能会被用户代码使用,如ctx.executor().execute(task)。所以execute()方法里又调用inEventLoop()方法进行了一次外部线程判断,确保执行task任务时不会遇到线程问题。

二.如果当前线程不是Netty的Reactor线程,则调用startThread()方法启动一个Reactor线程。在startThread()方法中首先会判断当前NioEventLoop对应的Reactor线程实体有没有被启动。如果没有被启动,则通过设置CAS成功后调用doStartThread()方法启动线程。

三.执行doStartThread()方法时,会调用NioEventLoop的内部成员变量executor的execute()方法。executor就是线程执行器ThreadPerTaskExecutor,它的作用是每次执行Runnable任务时都会创建一个线程来执行。也就是executor.execute()方法会通过DefaultThreadFactory的newThread()方法,创建出一个FastThreadLocalThread线程来执行Runnable任务。

四.doStartThread()方法的Runnable任务会由一个FastThreadLocalThread线程来执行。在Runnable任务的run()方法里,会保存ThreadPerTaskExecutor创建出来的FastThreadLocalThread对象到SingleThreadEventExecutor的成员变量thread中,然后调用SingleThreadEventExecutor的run()方法。

java 复制代码
//Abstract base class for EventExecutorGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {
    private final EventExecutor[] children;
    private final EventExecutorChooserFactory.EventExecutorChooser chooser;
    ...    
    //Create a new instance.
    protected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {
        if (nThreads <= 0) throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));
        //1.创建ThreadPerTaskExecutor线程执行器
        if (executor == null) executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());
        //2.创建NioEventLoop
        children = new EventExecutor[nThreads];
        for (int i = 0; i < nThreads; i ++) {
            ...
            //创建每一个NioEventLoop时,都会调用newChild()方法给每一个NioEventLoop配置一些核心参数
            //传入线程执行器executor去创建NioEventLoop
            children[i] = newChild(executor, args);
        }
        //3.创建线程选择器
        chooser = chooserFactory.newChooser(children);
        ...
    }
    
    protected ThreadFactory newDefaultThreadFactory() {
        //getClass()是获取该方法所属的对象类型,也就是NioEventLoopGroup类型
        //因为是通过NioEventLoopGroup的构造方法层层调用到这里的
        return new DefaultThreadFactory(getClass());
    }
    ...
}

public final class ThreadPerTaskExecutor implements Executor {
    private final ThreadFactory threadFactory;
    public ThreadPerTaskExecutor(ThreadFactory threadFactory) {
        if (threadFactory == null) throw new NullPointerException("threadFactory");
        this.threadFactory = threadFactory;
    }
    
    @Override
    public void execute(Runnable command) {
        //调用DefaultThreadFactory的newThread()方法执行Runnable任务
        threadFactory.newThread(command).start();
    }
}

//A ThreadFactory implementation with a simple naming rule.
public class DefaultThreadFactory implements ThreadFactory {
    private static final AtomicInteger poolId = new AtomicInteger();
    private final AtomicInteger nextId = new AtomicInteger();
    private final boolean daemon;
    private final int priority;
    protected final ThreadGroup threadGroup;
    ...
    public DefaultThreadFactory(Class<?> poolType) {
        this(poolType, false, Thread.NORM_PRIORITY);
    }
    
    public DefaultThreadFactory(Class<?> poolType, boolean daemon, int priority) {
        //toPoolName()方法会把NioEventLoopGroup的首字母变成小写
        this(toPoolName(poolType), daemon, priority);
    }
    
    public DefaultThreadFactory(String poolName, boolean daemon, int priority) {
        this(poolName, daemon, priority, 
           System.getSecurityManager() == null ? Thread.currentThread().getThreadGroup() : System.getSecurityManager().getThreadGroup());
    }
    
    public DefaultThreadFactory(String poolName, boolean daemon, int priority, ThreadGroup threadGroup) {
        ...
        //prefix用来标记线程名字的前缀
        prefix = poolName + '-' + poolId.incrementAndGet() + '-';
        this.daemon = daemon;
        this.priority = priority;
        this.threadGroup = threadGroup;
    }
    
    @Override
    public Thread newThread(Runnable r) {
        Thread t = newThread(new DefaultRunnableDecorator(r), prefix + nextId.incrementAndGet());
        if (t.isDaemon()) {
            if (!daemon) t.setDaemon(false);
        } else {
            if (daemon) t.setDaemon(true);
        }
        if (t.getPriority() != priority) t.setPriority(priority);
        return t;
    }
    
    protected Thread newThread(Runnable r, String name) {
        return new FastThreadLocalThread(threadGroup, r, name);
    }
    ...
}

NioEventLoop是如何与一个线程实体绑定的? NioEventLoop会通过线程执行器ThreadPerTaskExecutor创建一个FastThreadLocalThread,然后再将该FastThreadLocalThread线程保存到其成员变量中,从而实现与一个线程实体进行绑定。

(4)NioEventLoop的启动总结

一.在注册服务端Channel的过程中,主线程最终会调用AbstractUnsafe的register()方法。该方法首先会将一个NioEventLoop绑定到这个服务端Channel上,然后把实际注册Selector的逻辑封装成一个Runnable任务,接着调用NioEventLoop的execute()方法来执行这个Runnable任务。

二.NioEventLoop的execute()方法其实就是其父类SingleThreadEventExecutor的execute()方法,它会先判断当前调用execute()方法的线程是不是Netty的Reactor线程,如果不是就调用startThread()方法来创建一个Reactor线程。

三.startThread()方法会通过线程执行器ThreadPerTaskExecutor的execute()方法来创建一个线程。这个线程是一个FastThreadLocalThread线程,这个线程需要执行如下逻辑:把线程保存到NioEventLoop的成员变量thread中,然后调用NioEventLoop的run()方法来启动NioEventLoop。

NioEventLoop的启动流程如下:

scss 复制代码
bind() -> initAndRegister() -> config().group().register() -> eventloop.execute() //入口
  startThread() -> doStartThread() //创建线程
    ThreadPerTaskExecutor.execute() //线程执行器创建FastThreadLocalThread线程
      thread = Thread.currentThread() //保存FastThreadLocalThread线程到NioEventLoop的成员变量中
      NioEventLoop.run() //启动NioEventLoop

NioEventLoop的启动流程说明如下:

首先bind()方法会将具体绑定端口的操作封装成一个Runnable任务,然后调用NioEventLoop的execute()方法,接着Netty会判断调用execute()方法的线程是否是NIO线程,如果发现不是就会调用startThread()方法开始创建线程。

创建线程是通过线程执行器ThreadPerTaskExecutor来创建的。线程执行器的作用是每执行一个任务都会创建一个线程,而且创建出来的线程就是NioEventLoop底层的一个FastThreadLocalThread线程。

创建完FastThreadLocalThread线程后会执行一个Runnable任务,该Runnable任务首先会将这个线程保存到NioEventLoop对象。保存的目的是为了判断后续对NioEventLoop的相关执行线程是否为本身。如果不是就将封装好的一个任务放入TaskQueue中进行串行执行,实现线程安全。该Runnable任务然后会调用NioEventLoop的run()方法,从而启动NioEventLoop。NioEventLoop的run()方法是驱动Netty运转的核心方法。

相关推荐
9527出列1 天前
Netty源码分析--客户端连接接入流程解析
网络协议·netty
马尚来2 天前
【韩顺平】尚硅谷Netty视频教程
后端·netty
马尚道4 天前
【韩顺平】尚硅谷Netty视频教程
netty
马尚道4 天前
Netty核心技术及源码剖析
源码·netty
moxiaoran57534 天前
java接收小程序发送的protobuf消息
websocket·netty·protobuf
马尚来5 天前
尚硅谷 Netty核心技术及源码剖析 Netty模型 详细版
源码·netty
马尚来5 天前
Netty核心技术及源码剖析
后端·netty
失散1311 天前
分布式专题——35 Netty的使用和常用组件辨析
java·分布式·架构·netty
hanxiaozhang201813 天前
Netty面试重点-1
网络·网络协议·面试·netty
mumu1307梦23 天前
SpringAI 实战:解决 Netty 超时问题,优化 OpenAiApi 配置
java·spring boot·netty·超时·timeout·openapi·springai