Dubbo3.2.x服务调用源码分析

源码调试的准备过程前两篇文章有提及，在此就不介绍。

前两篇文章，我们讲了服务暴露/注册、引用/发现的源码流程，此篇，讲一下dubbo服务发现后，是如何一步步调用的。

前篇提到，服务发现得到invoker的代理之后，去执行sayHello()。

consumer侧调用调试

通过调试，会进入到InvokerInvocationHandler，很明显，这里使用的是jdk的代理模式

@Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
    if (method.getDeclaringClass() == Object.class) {
        return method.invoke(invoker, args);
    }
    // 通过反射获取方法名、参数类型等
    String methodName = method.getName();
    Class<?>[] parameterTypes = method.getParameterTypes();
    if (parameterTypes.length == 0) {
        if ("toString".equals(methodName)) {
            return invoker.toString();
        } else if ("$destroy".equals(methodName)) {
            invoker.destroy();
            return null;
        } else if ("hashCode".equals(methodName)) {
            return invoker.hashCode();
        }
    } else if (parameterTypes.length == 1 && "equals".equals(methodName)) {
        return invoker.equals(args[0]);
    }
    // 封装RpcInvocation参数
    RpcInvocation rpcInvocation = new RpcInvocation(serviceModel, method.getName(), invoker.getInterface().getName(), protocolServiceKey, method.getParameterTypes(), args);

    if (serviceModel instanceof ConsumerModel) {
        rpcInvocation.put(Constants.CONSUMER_MODEL, serviceModel);
        rpcInvocation.put(Constants.METHOD_MODEL, ((ConsumerModel) serviceModel).getMethodModel(method));
    }
    // 继续向下进行
    return InvocationUtil.invoke(invoker, rpcInvocation);
}

这个方法里，封装了RpcInvocation，继续向下直接来到 org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker#invoke

@Override
    public Result invoke(final Invocation invocation) throws RpcException {
        checkWhetherDestroyed();

        // binding attachments into invocation.
//        Map<String, Object> contextAttachments = RpcContext.getClientAttachment().getObjectAttachments();
//        if (contextAttachments != null && contextAttachments.size() != 0) {
//            ((RpcInvocation) invocation).addObjectAttachmentsIfAbsent(contextAttachments);
//        }

        InvocationProfilerUtils.enterDetailProfiler(invocation, () -> "Router route.");
        List<Invoker<T>> invokers = list(invocation);
        InvocationProfilerUtils.releaseDetailProfiler(invocation);
        // 初始化负载均衡
        LoadBalance loadbalance = initLoadBalance(invokers, invocation);
        RpcUtils.attachInvocationIdIfAsync(getUrl(), invocation);

        InvocationProfilerUtils.enterDetailProfiler(invocation, () -> "Cluster " + this.getClass().getName() + " invoke.");
        try {
            return doInvoke(invocation, invokers, loadbalance);
        } finally {
            InvocationProfilerUtils.releaseDetailProfiler(invocation);
        }
    }

这段代码中会根据spi扩展机制初始化负载均衡器，默认负载均衡策略是random

接下来向下看关键方法FailoverClusterInvoker#doInvoke

这个方法里涉及了容错机制

@Override
@SuppressWarnings({"unchecked", "rawtypes"})
public Result doInvoke(Invocation invocation, final List<Invoker<T>> invokers, LoadBalance loadbalance) throws RpcException {
    List<Invoker<T>> copyInvokers = invokers;
    checkInvokers(copyInvokers, invocation);
    String methodName = RpcUtils.getMethodName(invocation);
    // 这里计算的是重试次数，默认为3次
    int len = calculateInvokeTimes(methodName);
    // retry loop.
    RpcException le = null; // last exception.
    List<Invoker<T>> invoked = new ArrayList<Invoker<T>>(copyInvokers.size()); // invoked invokers.
    Set<String> providers = new HashSet<String>(len);
    for (int i = 0; i < len; i++) {
        //Reselect before retry to avoid a change of candidate `invokers`.
        //NOTE: if `invokers` changed, then `invoked` also lose accuracy.
        if (i > 0) {
            // 这里若进行了重试，则会重新选择和检查invokers
            checkWhetherDestroyed();
            copyInvokers = list(invocation);
            // check again
            checkInvokers(copyInvokers, invocation);
        }
        // 根据负载均衡机制选择唯一的invoker
        Invoker<T> invoker = select(loadbalance, invocation, copyInvokers, invoked);
        invoked.add(invoker);
        RpcContext.getServiceContext().setInvokers((List) invoked);
        boolean success = false;
        try {
            Result result = invokeWithContext(invoker, invocation);
            if (le != null && logger.isWarnEnabled()) {
                logger.warn(CLUSTER_FAILED_MULTIPLE_RETRIES,"failed to retry do invoke","","Although retry the method " + methodName
                    + " in the service " + getInterface().getName()
                    + " was successful by the provider " + invoker.getUrl().getAddress()
                    + ", but there have been failed providers " + providers
                    + " (" + providers.size() + "/" + copyInvokers.size()
                    + ") from the registry " + directory.getUrl().getAddress()
                    + " on the consumer " + NetUtils.getLocalHost()
                    + " using the dubbo version " + Version.getVersion() + ". Last error is: "
                    + le.getMessage(),le);
            }
            success = true;
            return result;
        } catch (RpcException e) {
            if (e.isBiz()) { // biz exception.
                throw e;
            }
            le = e;
        } catch (Throwable e) {
            le = new RpcException(e.getMessage(), e);
        } finally {
            if (!success) {
                providers.add(invoker.getUrl().getAddress());
            }
        }
    }
    throw new RpcException(le.getCode(), "Failed to invoke the method "
            + methodName + " in the service " + getInterface().getName()
            + ". Tried " + len + " times of the providers " + providers
            + " (" + providers.size() + "/" + copyInvokers.size()
            + ") from the registry " + directory.getUrl().getAddress()
            + " on the consumer " + NetUtils.getLocalHost() + " using the dubbo version "
            + Version.getVersion() + ". Last error is: "
            + le.getMessage(), le.getCause() != null ? le.getCause() : le);
}

在此方法中，涉及到容错机制，若失败会进行重试。另外，还会根据负载均衡器选出唯一的用于执行的invoker。

继续向下调试，会进入到一连串的FilterChain，这里用到了责任链设计模式，执行了一些过滤或增强逻辑，我们直接向下执行，看到关键方法DubboInvoker#doInvoke

@Override
protected Result doInvoke(final Invocation invocation) throws Throwable {
    RpcInvocation inv = (RpcInvocation) invocation;
    final String methodName = RpcUtils.getMethodName(invocation);
    inv.setAttachment(PATH_KEY, getUrl().getPath());
    inv.setAttachment(VERSION_KEY, version);
    // 这里获取通信所用的数据交换Client
    ExchangeClient currentClient;
    if (clients.length == 1) {
        currentClient = clients[0];
    } else {
        currentClient = clients[index.getAndIncrement() % clients.length];
    }
    try {
        // 我们这里用到的是双工通信，此处为false
        boolean isOneway = RpcUtils.isOneway(getUrl(), invocation);
        // 这里会计算超时时间，默认1s
        int timeout = RpcUtils.calculateTimeout(getUrl(), invocation, methodName, DEFAULT_TIMEOUT);
        if (timeout <= 0) {
            return AsyncRpcResult.newDefaultAsyncResult(new RpcException(RpcException.TIMEOUT_TERMINATE,
                "No time left for making the following call: " + invocation.getServiceName() + "."
                    + invocation.getMethodName() + ", terminate directly."), invocation);
        }

        invocation.setAttachment(TIMEOUT_KEY, String.valueOf(timeout));

        RpcContext.getServiceContext().setRemoteAddress(currentClient.getRemoteAddress());

        if (isOneway) {
            boolean isSent = getUrl().getMethodParameter(methodName, Constants.SENT_KEY, false);
            currentClient.send(inv, isSent);
            return AsyncRpcResult.newDefaultAsyncResult(invocation);
        } else {
            ExecutorService executor = getCallbackExecutor(getUrl(), inv);
            // 发起服务调用请求
            CompletableFuture<AppResponse> appResponseFuture =
                currentClient.request(inv, timeout, executor).thenApply(obj -> (AppResponse) obj);
            // save for 2.6.x compatibility, for example, TraceFilter in Zipkin uses com.alibaba.xxx.FutureAdapter
            FutureContext.getContext().setCompatibleFuture(appResponseFuture);
            AsyncRpcResult result = new AsyncRpcResult(appResponseFuture, inv);
            result.setExecutor(executor);
            return result;
        }
    } catch (TimeoutException e) {
        throw new RpcException(RpcException.TIMEOUT_EXCEPTION, "Invoke remote method timeout. method: " + invocation.getMethodName() + ", provider: " + getUrl() + ", cause: " + e.getMessage(), e);
    } catch (RemotingException e) {
        throw new RpcException(RpcException.NETWORK_EXCEPTION, "Failed to invoke remote method: " + invocation.getMethodName() + ", provider: " + getUrl() + ", cause: " + e.getMessage(), e);
    }
}

这个方法就看到开始通信的影子了，开始创建数据交换的Client，计算超时时间等。后续会创建一个线程池，去执行服务调用请求。

从currentClient.request进去之后，一路调试，我们来看下下面的方法。HeaderExchangeChannel#request方法

@Override
public CompletableFuture<Object> request(Object request, int timeout, ExecutorService executor) throws RemotingException {
    if (closed) {
        throw new RemotingException(this.getLocalAddress(), null, "Failed to send request " + request + ", cause: The channel " + this + " is closed!");
    }
    // create request.
    Request req = new Request();
    req.setVersion(Version.getProtocolVersion());
    req.setTwoWay(true);
    req.setData(request);
    DefaultFuture future = DefaultFuture.newFuture(channel, req, timeout, executor);
    try {
        channel.send(req);
    } catch (RemotingException e) {
        future.cancel();
        throw e;
    }
    return future;
}

这里会new一个DefaultFuture，有必要看一下它的构造方法

private DefaultFuture(Channel channel, Request request, int timeout) {
    this.channel = channel;
    this.request = request;
    // 这个请求id很重要，由于是异步请求，响应信息的匹配就是靠它
    this.id = request.getId();
    this.timeout = timeout > 0 ? timeout : channel.getUrl().getPositiveParameter(TIMEOUT_KEY, DEFAULT_TIMEOUT);
    // put into waiting map.
    FUTURES.put(id, this);
    CHANNELS.put(id, channel);
}

这个构造方法，回去填充DefaultFuture中的本地缓存FUTURES和CHANNELS

之后，我们看一下channel.send()方法，一路调试，走到 AbstractClient#send方法。

@Override
public void send(Object message, boolean sent) throws RemotingException {
    if (needReconnect && !isConnected()) {
        // 这里会去建立netty网络连接
        connect(); 
    }
    // 获取信道
    Channel channel = getChannel();
    //TODO Can the value returned by getChannel() be null? need improvement.
    if (channel == null || !channel.isConnected()) {
        throw new RemotingException(this, "message can not send, because channel is closed . url:" + getUrl());
    }
    // 发送信号
    channel.send(message, sent);
}

这个方法首先回去建立网络连接，然后再通过信道发送请求信号。

channel.send(message, sent)中的sent代表是否要等待发送完成，此处为false，message也就是前面封装好的RpcInvocation

下面是NettyChannel#send

/**
 * Send message by netty and whether to wait the completion of the send.
 *
 * @param message message that need send.
 * @param sent    whether to ack async-sent
 * @throws RemotingException throw RemotingException if wait until timeout or any exception thrown by method body that surrounded by try-catch.
 */
@Override
public void send(Object message, boolean sent) throws RemotingException {
    // whether the channel is closed
    super.send(message, sent);

    boolean success = true;
    int timeout = 0;
    try {
        ChannelFuture future = channel.writeAndFlush(message);
        if (sent) {
            // wait timeout ms
            timeout = getUrl().getPositiveParameter(TIMEOUT_KEY, DEFAULT_TIMEOUT);
            success = future.await(timeout);
        }
        Throwable cause = future.cause();
        if (cause != null) {
            throw cause;
        }
    } catch (Throwable e) {
        removeChannelIfDisconnected(channel);
        throw new RemotingException(this, "Failed to send message " + PayloadDropper.getRequestWithoutData(message) + " to " + getRemoteAddress() + ", cause: " + e.getMessage(), e);
    }
    if (!success) {
        throw new RemotingException(this, "Failed to send message " + PayloadDropper.getRequestWithoutData(message) + " to " + getRemoteAddress()
            + "in timeout(" + timeout + "ms) limit");
    }
}

这里回去进行请求消息的发送，并且根据sent判断，是否要记录超时时间和success，如果sent是true，也就代表了要等待响应返回，同时获得超时时间和超时结果success，若success为false则记录超时日志。

再向下调试，就会发现，后续走到netty代码中了

所以我们下面的调试，是站在provider端进行的，看一下，请求信号来了之后，provider端都干了那些事情

Provider侧调用调试

下面我们先调试启动provider（不要打任何断点），启动之后，在NettyServerHanlder的channelRead方法打断点，再直接启动consumer。

@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
    // 获取channel
    NettyChannel channel = NettyChannel.getOrAddChannel(ctx.channel(), url, handler);
    // 请求处理
    handler.received(channel, msg);
    // 触发qos监控处理
    ctx.fireChannelRead(msg);
}

到这里之后，先去获得channel，再进行请求处理，最后会触发qos监控，我们重点看下请求处理过程

一路调试，期间有一系列的handler，最后来到AllChannelHandler#received

@Override
public void received(Channel channel, Object message) throws RemotingException {
    ExecutorService executor = getPreferredExecutorService(message);
    try {
        executor.execute(new ChannelEventRunnable(channel, handler, ChannelState.RECEIVED, message));
    } catch (Throwable t) {
        if (message instanceof Request && t instanceof RejectedExecutionException) {
            sendFeedback(channel, (Request) message, t);
            return;
        }
        throw new ExecutionException(message, channel, getClass() + " error when process received event .", t);
    }

在这里会去拿到一个dubbo的线程池，然后放入一个实现了Runnable接口的ChannelEventRunnable，由于是异步执行，后续调试不好调，我们直接在ChannelEventRunnable的run方法打断点，发现会执行run方法，run方法中会继续向下走，走到DecodeHandler#received中，会对请求的数据进行解码

@Override
public void received(Channel channel, Object message) throws RemotingException {
    if (message instanceof Decodeable) {
        decode(message);
    }

    if (message instanceof Request) {
        // 解码
        decode(((Request) message).getData());
    }

    if (message instanceof Response) {
        decode(((Response) message).getResult());
    }

    handler.received(channel, message);
}

随后继续向下调用handler.received，走到HeaderExchangeHandler#received

@Override
public void received(Channel channel, Object message) throws RemotingException {
    final ExchangeChannel exchangeChannel = HeaderExchangeChannel.getOrAddChannel(channel);
    if (message instanceof Request) {
        // handle request.
        Request request = (Request) message;
        if (request.isEvent()) {
            handlerEvent(channel, request);
        } else {
            if (request.isTwoWay()) {
                // 双向通信走这里
                handleRequest(exchangeChannel, request);
            } else {
                // 。。。。。。
}

这里会获取数据交换通道ExchangeChannel，之后继续向下处理请求。（这里链路式的去调用一些不同的handler，有点责任链模式的样子）

void handleRequest(final ExchangeChannel channel, Request req) throws RemotingException {
    Response res = new Response(req.getId(), req.getVersion());
    if (req.isBroken()) {
        Object data = req.getData();

        String msg;
        if (data == null) {
            msg = null;
        } else if (data instanceof Throwable) {
            msg = StringUtils.toString((Throwable) data);
        } else {
            msg = data.toString();
        }
        res.setErrorMessage("Fail to decode request due to: " + msg);
        res.setStatus(Response.BAD_REQUEST);

        channel.send(res);
        return;
    }
    // find handler by message class.
    // 解码后的RpcInvocation
    Object msg = req.getData();
    try {
        // 对请求进行异步处理
        CompletionStage<Object> future = handler.reply(channel, msg);
        // 处理完拿到结果
        future.whenComplete((appResult, t) -> {
            try {
                if (t == null) {
                    res.setStatus(Response.OK);
                    res.setResult(appResult);
                } else {
                    res.setStatus(Response.SERVICE_ERROR);
                    res.setErrorMessage(StringUtils.toString(t));
                }
                // 将结果回传
                channel.send(res);
            } catch (RemotingException e) {
                logger.warn(
                        TRANSPORT_FAILED_RESPONSE,
                        "",
                        "",
                        "Send result to consumer failed, channel is " + channel + ", msg is " + e);
            }
        });
    } catch (Throwable e) {
        res.setStatus(Response.SERVICE_ERROR);
        res.setErrorMessage(StringUtils.toString(e));
        channel.send(res);
    }
}

这个方法就比较清楚了，new一个response，然后将msg（解码后的RpcInvocation）和channel传入reply进行调用，后续通过future拿到执行结果，并send回consumer端。

根据调试，下面来到DubboProtocol#reply方法

@Override
public CompletableFuture<Object> reply(ExchangeChannel channel, Object message) throws RemotingException {

    if (!(message instanceof Invocation)) {
        throw new RemotingException(
                channel,
                "Unsupported request: "
                        + (message == null
                                ? null
                                : (message.getClass().getName() + ": " + message))
                        + ", channel: consumer: " + channel.getRemoteAddress() + " --> provider: "
                        + channel.getLocalAddress());
    }

    Invocation inv = (Invocation) message;
    Invoker<?> invoker = inv.getInvoker() == null ? getInvoker(channel, inv) : inv.getInvoker();
    // switch TCCL
    if (invoker.getUrl().getServiceModel() != null) {
        Thread.currentThread()
                .setContextClassLoader(
                        invoker.getUrl().getServiceModel().getClassLoader());
    }
    // need to consider backward-compatibility if it's a callback
    if (Boolean.TRUE.toString().equals(inv.getObjectAttachmentWithoutConvert(IS_CALLBACK_SERVICE_INVOKE))) {
        String methodsStr = invoker.getUrl().getParameters().get("methods");
        boolean hasMethod = false;
        if (methodsStr == null || !methodsStr.contains(",")) {
            hasMethod = inv.getMethodName().equals(methodsStr);
        } else {
            String[] methods = methodsStr.split(",");
            for (String method : methods) {
                if (inv.getMethodName().equals(method)) {
                    hasMethod = true;
                    break;
                }
            }
        }
        if (!hasMethod) {
            logger.warn(
                    PROTOCOL_FAILED_REFER_INVOKER,
                    "",
                    "",
                    new IllegalStateException("The methodName " + inv.getMethodName()
                                    + " not found in callback service interface ,invoke will be ignored."
                                    + " please update the api interface. url is:"
                                    + invoker.getUrl())
                            + " ,invocation is :" + inv);
            return null;
        }
    }
    RpcContext.getServiceContext().setRemoteAddress(channel.getRemoteAddress());
    Result result = invoker.invoke(inv);
    return result.thenApply(Function.identity());
}

这里会获取Invoker，设置Rpc上下文中的remoteAddress，然后进入到invoker.invoke，这里会继续执行一系列FilterChain

在经历一系列FilterChain的invoke之后，直接来到AbstractProxyInvoker#invoke方法

@Override
public Result invoke(Invocation invocation) throws RpcException {
    ProfilerEntry originEntry = null;
    try {
        if (ProfilerSwitch.isEnableSimpleProfiler()) {
            Object fromInvocation = invocation.get(Profiler.PROFILER_KEY);
            if (fromInvocation instanceof ProfilerEntry) {
                ProfilerEntry profiler = Profiler.enter(
                        (ProfilerEntry) fromInvocation, "Receive request. Server biz impl invoke begin.");
                invocation.put(Profiler.PROFILER_KEY, profiler);
                originEntry = Profiler.setToBizProfiler(profiler);
            }
        }
        // 这里进行doInvoke
        Object value = doInvoke(
                proxy, invocation.getMethodName(), invocation.getParameterTypes(), invocation.getArguments());

        CompletableFuture<Object> future = wrapWithFuture(value, invocation);
        CompletableFuture<AppResponse> appResponseFuture = future.handle((obj, t) -> {
            AppResponse result = new AppResponse(invocation);
            if (t != null) {
                if (t instanceof CompletionException) {
                    result.setException(t.getCause());
                } else {
                    result.setException(t);
                }
            } else {
                result.setValue(obj);
            }
            return result;
        });
        return new AsyncRpcResult(appResponseFuture, invocation);
    } catch (InvocationTargetException e) {
        if (RpcContext.getServiceContext().isAsyncStarted()
                && !RpcContext.getServiceContext().stopAsync()) {
            logger.error(
                    PROXY_ERROR_ASYNC_RESPONSE,
                    "",
                    "",
                    "Provider async started, but got an exception from the original method, cannot write the exception back to consumer because an async result may have returned the new thread.",
                    e);
        }
        return AsyncRpcResult.newDefaultAsyncResult(null, e.getTargetException(), invocation);
    } catch (Throwable e) {
        throw new RpcException(
                "Failed to invoke remote proxy method " + invocation.getMethodName() + " to " + getUrl()
                        + ", cause: " + e.getMessage(),
                e);
    } finally {
        if (ProfilerSwitch.isEnableSimpleProfiler()) {
            Object fromInvocation = invocation.get(Profiler.PROFILER_KEY);
            if (fromInvocation instanceof ProfilerEntry) {
                ProfilerEntry profiler = Profiler.release((ProfilerEntry) fromInvocation);
                invocation.put(Profiler.PROFILER_KEY, profiler);
            }
        }
        Profiler.removeBizProfiler();
        if (originEntry != null) {
            Profiler.setToBizProfiler(originEntry);
        }
    }
}

javassistProxyFactory中的doInvoke如下

@Override
public <T> Invoker<T> getInvoker(T proxy, Class<T> type, URL url) {
    try {
        // TODO Wrapper cannot handle this scenario correctly: the classname contains '$'
        final Wrapper wrapper =
                Wrapper.getWrapper(proxy.getClass().getName().indexOf('$') < 0 ? proxy.getClass() : type);
        return new AbstractProxyInvoker<T>(proxy, type, url) {
            @Override
            protected Object doInvoke(T proxy, String methodName, Class<?>[] parameterTypes, Object[] arguments)
                    throws Throwable {
                return wrapper.invokeMethod(proxy, methodName, parameterTypes, arguments);
            }
        };
    }

这里new一个Invoker执行doInvoke，之后，通过包装类的invokeMethod方法，会去执行我们目标接口

随后，服务执行完的结果，会被封装到一个Response对象中，利用channel.send进行回传。

至此，服务调用的过程就结束了。

总结

整个流程还是比较简单的，主要就是consumer端依靠Netty网络连接向producer端发起一个请求，将封装好的RpcInvocation传给producer，其中包含了要调用接口的接口名、方法名、Invoker、Proxy、Request等各种信息。Producer拿到请求数据之后，先进行解码，解码之后会做一些增强的扩展操作（FilterChain中的一系列invoke），之后会根据RpcInvocation找到producer端对应的接口信息，并通过这些信息创建Invoker，来进行最后的调用，并将结果通过网络连接返回给consumer。