如何用webRTC实现远程桌面控制？(超详细记录，带源码)

先说一下项目背景：有两个web端，一个客服，一个用户。客服可以发起远程协助，用户端同意之后，可进行远程桌面操作。

就这个需求，需要准备哪些东西呢？

1.因为纯web端无法操作系统鼠标，所以需要使用electron开发一个桌面应用。

2.因为需要一个相对稳定的交互通信，所以使用webRTC来实现。（传输层除了tcp协议，还有UDP协议，与tcp不同的是，UDP建立的是长连接，速度快，但有可能会丢失数据包，所以更适合做视频，流媒体通信。WebRTC就是基于UDP协议的，所以很适合做远程桌面）

3.websocket服务（信令服务）webRTC建立连接的时候，需要通过websocket服务交换信息。(使用socket.io库)

4.两个web端（vue3）

在开始写代码之前，先梳理一下实现的步骤。

客户端：

1.客服点击按钮，通过websocket向用户端发起远程控制询问

2.收到用户同意，创建webrtc协议

3.监听鼠标键盘事件，并通过webrtc协议发送

用户端：

1.用户端收到消息，弹窗提示是否接受

2.点击接受，通过websocket向客服发送消息

3.跳转url，唤起electron应用（electron需注册协议）

electron端：

1.创建webrtc协议

2.监听鼠标键盘事件操作对应鼠标键盘

socket服务端：

1.发起/接受远程控制的消息询问

2.创建webrtc时的数据转发

socket服务端实现

实际项目的一些逻辑判断我就先不管了，只实现一个简易的demo。

当客户发送ready的时候，默认用户一定是同意的，就不弹窗了。

socket直接使用广播的形式，也不再判断房间号，用户是否匹配等问题。

javascript 复制代码

const Koa = require("koa");
const { createServer } = require("http");
const { Server } = require("socket.io");

const app = new Koa();
const httpServer = createServer(app.callback());
const io = new Server(httpServer, {
  /* options */
});

io.on("connection", (socket) => {
  console.log("connect success");
  socket.on("ready", () => {
    // console.log("ready");
    socket.broadcast.emit("ready");
  });

  socket.on("offer", (params) => {
    socket.broadcast.emit("offer", params);
  });

  socket.on("answer", (params) => {
    socket.broadcast.emit("answer", params);
  });

  socket.on("toCustomCandidate", (params) => {
    socket.broadcast.emit("toCustomCandidate", params);
  });

  socket.on("toUserCandidate", (params) => {
    socket.broadcast.emit("toUserCandidate", params);
  });

  socket.on("mousemove", (params) => {
    socket.broadcast.emit("mousemove", params);
  });

  socket.on("stream", (params) => {
    console.log("params", params);
  });
});

httpServer.listen(3000, () => {
  console.log("Server running on http://localhost:3000");
});

用户端

用户端比较简单，监听到ready事件之后，直接跳转url。

dart 复制代码

socket.on("ready", async () => {
    window.open("remote://test?age=1")
})

remote这个协议需要在electron中实现。

electron端

1.注册remote协议，以便通过url唤起

ini 复制代码

const scheme = "remote";
let isSet = false
protocol.registerSchemesAsPrivileged([
  {
    scheme: scheme,
    privileges: {
      bypassCSP: true,
    },
  },
]);

app.removeAsDefaultProtocolClient(scheme);

if (process.env.NODE_ENV === "development" && process.platform === "win32") {
    isSet = app.setAsDefaultProtocolClient(scheme, process.execPath, [
      path.resolve(process.argv[1]),
    ]);
} else {
    isSet = app.setAsDefaultProtocolClient(scheme);
}

通过url唤起之后，url上的参数获取苹果跟window略有差别。

苹果端可以通过app.on("open-url", (event, url)=> {})来获取。

在window上，只能通过process.argv中获取。

2.创建PeerConnection实例

csharp 复制代码

    // electron端
    peer.value = new PeerConnection({
      iceServers: [
        {
          urls: ["stun:stun.l.google.com:19302"],
        },
        {
          urls: ["turn:wangxiang.website:3478"],
          username: "admin",
          credential: "admin",
        },
      ],
    });
    let offer = await peer.value.createOffer();
    await peer.value.setLocalDescription(offer);
    socket.emit("offer", offer);
    socket.on("answer", async (answer) => {
        await peer.value.setRemoteDescription(answer);
    });

2个web端创建PeerConnection实例方式基本是一样的。（其中iceServers表示ICE服务器的配置，是为了确保两个对等端能够成功通信，我直接用的是网上现成的）

创建实例之后，发起方创建offer，并设置本地描述，同时通过socket发送给接收方，接收方收到offer之后同样的设置远程描述与本地描述。

csharp 复制代码

 // 客服端
  socket.on("offer", async (offer) => {
    if (peer) {
      await peer.setRemoteDescription(offer);
      let remoteAnswer = await peer.createAnswer();
      await peer.setLocalDescription(remoteAnswer);
      socket.emit("answer", {
        remoteAnswer,
        conversationId,
      });
    }
  });

3.添加监听Candidate

csharp 复制代码

  //electron端
  socket.on("toUserCandidate", (candidate) => {
    console.log("toUserCandidate");
    peer.addIceCandidate(candidate);
  });
  
   peer.onicecandidate = (event) => {
      console.log("localPc:", event.candidate, event);
      if (event.candidate) {
        socket.emit("toStaffCandidate", {
          candidate: event.candidate,
          ...params,
        });
      }
    };

typescript 复制代码

  // 客服端
  peer.onicecandidate = (e) => {
    if (e.candidate) {
      socket.emit("toUserCandidate", {
        candidate: e.candidate,
        conversationId,
      });
    }
  };
  socket.on("toStaffCandidate", async (candidate) => {
    await peer.addIceCandidate(candidate);
  });
  // 远程桌面的视频流
  peer.onaddstream = (e: any) => {
    console.log("onaddstream");
    try {
      setVideo(e.stream);
    } catch (ex) {}
  };

4. 处理鼠标键盘事件

javascript 复制代码

    // preload.js
    channel.onmessage = (e) => {
      var eventData = JSON.parse(e.data);
      if (eventData.type === "scroll") {
        ipcRenderer.send("scroll", { x: eventData.x, y: eventData.y });
      } else if (eventData.type === "mousemove") {
        ipcRenderer.send("mousemove", { x: eventData.x, y: eventData.y });
      } else if (eventData.type === "keydown") {
        ipcRenderer.send("keydown", { key: eventData.key });
      } else if (eventData.type === "mousedown") {
        ipcRenderer.send("mousedown", { key: eventData.key });
      } else if (eventData.type === "mouseup") {
        ipcRenderer.send("mouseup", { key: eventData.key });
      } else if (eventData.type === "copy") {
        ipcRenderer.send("copy", { key: eventData.key });
      } else if (eventData.type === "paste") {
        ipcRenderer.send("paste", { key: eventData.key });
      }
    };
    
    // main.js
     ipcMain.on("keydown", (e, { key }) => {
        try {
          robot.keyTap(key);
        } catch (error) {
          log.warn("keydown error", error);
        }
      });

      ipcMain.on("copy", (e, { key }) => {
        robot.keyTap("c", ["control"]);
      });

      ipcMain.on("paste", (e, { key }) => {
        robot.keyTap("v", ["control"]);
      });

      ipcMain.on("mousedown", (e, { key }) => {
        robot.mouseToggle("down");
      });

      ipcMain.on("mouseup", (e, { key }) => {
        robot.mouseToggle("up");
      });

      ipcMain.on("mousemove", (e, { x, y }) => {
        robot.moveMouse(x * (screenWidth / 1280), y * (screenHeight / 720));
      });

鼠标事件的处理用的是robotjs这个库，我安装的时候有点问题，所以用的是@jitsi/robotjs。

这个库在window上打包一切正常，但是在mac上打包未生效。

客户端

客户端的RTCPeerConnection实例跟electron端创建方式是一样的，区别也在介绍electron时说明了。

鼠标事件以及键盘事件，有些特殊的按键以及功能需要做兼容，比如换行，回车按键以及复制粘贴功能。

javascript 复制代码

import { Socket } from "socket.io-client";
import { throttle } from "./help";
const PeerConnection =
  window.RTCPeerConnection ||
  // @ts-ignore
  window.mozRTCPeerConnection ||
  // @ts-ignore
  window.webkitRTCPeerConnection;
const PEERCONFIG = {
  iceServers: [
    {
      urls: ["stun:stun1.l.google.com:19302"],
    },
    {
      urls: ["turn:wangxiang.website:3478"],
      username: "admin",
      credential: "admin",
    },
  ],
};

let peer: RTCPeerConnection;
let isCom = false;
export function initPeer(socket: Socket, conversationId: string) {
  // 1.创建
  peer = new PeerConnection(PEERCONFIG);
  // 2.监听通道数据
  peer.ondatachannel = function (event) {
    const channel = event.channel;
    channel.onopen = function () {
      console.log("onopen");
      eventListen(channel);
    };
    channel.onmessage = function (event) {
      console.log("onmessage");
      console.log(event.data);
    };
  };
  peer.onconnectionstatechange = () => {
    if (peer.connectionState === "disconnected") {
      socket.emit("remoteClose", {
        conversationId,
      });
    }
  };
  // 3.监听offer
  socket.on("offer", async (offer) => {
    console.log("offer");
    if (peer) {
      await peer.setRemoteDescription(offer);
      let remoteAnswer = await peer.createAnswer();
      await peer.setLocalDescription(remoteAnswer);
      socket.emit("answer", {
        remoteAnswer,
        conversationId,
      });
    }
  });
  // 4.监听Candidate
  socket.on("toStaffCandidate", async (candidate) => {
    await peer.addIceCandidate(candidate);
  });
  // 5.监听stream
  // @ts-ignore
  peer.onaddstream = (e: any) => {
    console.log("onaddstream");
    try {
      setVideo(e.stream);
    } catch (ex) {}
  };

  peer.onicecandidate = (e) => {
    if (e.candidate) {
      socket.emit("toUserCandidate", {
        candidate: e.candidate,
        conversationId,
      });
    }
  };

  window.addEventListener("remoteClose", () => {
    socket.emit("remoteClose", {
      conversationId,
    });
  });

  const tip = document.getElementById("remote-tip");
  tip!.style.display = "block";
  const vid2 = document.getElementById("remote-control");
  // @ts-ignore
  vid2.srcObject = null;
}

function eventListen(channel: RTCDataChannel) {
  const vid2 = document.getElementById("remote-control");
  vid2!.addEventListener("wheel", (event: any) => {
    channel.send(
      JSON.stringify({
        x: 0,
        y: event.wheelDelta,
        type: "scroll",
      }),
    );
  });

  vid2!.addEventListener("mousedown", (event) => {
    channel.send(
      JSON.stringify({
        x: event.offsetX,
        y: event.offsetY,
        type: "mousedown",
      }),
    );
  });

  vid2!.addEventListener("mouseup", (event) => {
    channel.send(
      JSON.stringify({
        x: event.offsetX,
        y: event.offsetY,
        type: "mouseup",
      }),
    );
  });

  const moveEvent = throttle((event) => {
    channel.send(
      JSON.stringify({
        x: event.offsetX,
        y: event.offsetY,
        type: "mousemove",
      }),
    );
  }, 50);

  vid2!.addEventListener("mousemove", (event) => {
    moveEvent(event);
  });

  document.addEventListener("keydown", (event) => {
    if (isCom && event.key === "c") {
      console.log("copy");
      channel.send(
        JSON.stringify({
          key: "",
          type: "copy",
        }),
      );
      return;
    }

    if (isCom && event.key === "v") {
      console.log("paste");
      channel.send(
        JSON.stringify({
          key: "",
          type: "paste",
        }),
      );
      return;
    }
    channel.send(
      JSON.stringify({
        key: transKey(event.key),
        type: "keydown",
      }),
    );
  });

  document.addEventListener("keyup", (event) => {
    isCom = false;
  });
}

function transKey(key: string) {
  if (key === "Backspace") {
    return "backspace";
  } else if (key === "ArrowLeft") {
    return "left";
  } else if (key === "ArrowUp") {
    return "up";
  } else if (key === "ArrowRight") {
    return "right";
  } else if (key === "ArrowDown") {
    return "down";
  } else if (key === "Meta") {
    isCom = true;
    return "command";
  } else if (key === "Control") {
    isCom = true;
    return "control";
  } else if (key === "Enter") {
    return "\r";
  } else if (key === "Shift") {
    return "shift";
  }
  return key;
}

function setVideo(stream: RTCTrackEvent["streams"]) {
  const vid2 = document.getElementById("remote-control");
  const tip = document.getElementById("remote-tip");
  tip!.style.display = "none";
  // @ts-ignore
  vid2.srcObject = stream;
  // @ts-ignore
  vid2.onloadedmetadata = function () {
    // @ts-ignore
    vid2.play();
  };
}

这个远程控制功能还是蛮复杂的，主要是用到的端太多了。

我在正式开始做这个功能之前，也是先写了测试demo，等demo流程都走通之后，才开始实现功能+逻辑。因为实际项目中，还有房间号，客服和用户是否匹配，是否重复远程，是否结束等各种逻辑判断。

demo地址 github.com/yeshaojun/w... (这个是没有electron端，两个web端实现桌面共享，但无法同步鼠标操作)

electron github.com/yeshaojun/w... （客服端,跟socket服务用上个demo的，用户端用electron）

如无法实现或访问，可留言。