Troubles Introduced by Fibers

纤程带来的麻烦

At this point, I believe we have figured out how fiber works under the hood. As we can see from the implementation, fiber works by hacking the registers to fool CPU so that it swaps the call stacks and other relevant information for the CPU to make a switch. It is extremely cheap to make such a switch. There is great flexibility introduced with the tech. And it can be quite useful for game engine's job system design.

至此，我相信我们已经弄清楚了纤程在幕后的工作原理。从实现中我们可以看到，Fiber 的工作原理是破解寄存器来欺骗 CPU，以便它交换调用堆栈和其他相关信息，以便 CPU 进行切换。进行这样的转换非常便宜。该技术带来了极大的灵活性。这对于游戏引擎的作业系统设计非常有用。

However, while embracing the benefits of fibers, we have to be aware of the risks and responsibilities we are taking in the mean time to avoid problems.

然而，在享受纤程好处的同时，我们必须意识到我们同时承担的风险和责任，以避免出现问题。

Do not Exit the FiberMain

不要退出 FiberMain

As we learned before, there is ways for comiplers to make sure the returned address is properly setup when a function is called. However, we have to be mindful that a fiber entry function has no return address. It is not called in a conventional way. Do not expect the fiber returns its control back to whoever gave its control in the first place, it won't happen automatically.

正如我们之前了解到的，编译器有多种方法可以确保在调用函数时正确设置返回的地址。然而，我们必须注意，纤程入口函数没有返回地址。它不是以常规方式调用的。不要指望纤程将其控制权返回给最初给予控制权的人，这不会自动发生。

So we have to make sure that the fiber entry function will never exit regularly like other regular functions. What we should do is to make a switch to some other fiber once it is not expected to be executed anymore. It is fine to terminate the fiber even if the fiber entry function is not fully finished. It is actually mandatory to avoid unexpected behavior. An alternative is to setup the return address properly. Though, this will make the fiber implementation a bit more complicated and there is little that we will gain in doing so.

所以我们必须确保纤程入口函数永远不会像其他常规函数那样定期退出。我们应该做的是，一旦预计不再执行它，就切换到其他纤程。即使纤程进入功能尚未完全完成，也可以终止纤程。实际上，避免意外行为是强制性的。另一种方法是正确设置返回地址。不过，这将使纤程的实现变得更加复杂，并且这样做我们不会获得什么好处。

Smart Memory Pointers 智能内存指针

Smart pointers is a mechanism to prevent memory leak. For every piece of heap memory allocation, it will couple this allocation with a smart pointer allocation, which is a small object that controls the life time of the heap memory. As long as the smart pointer itself gets destroyed, the heap allocation coupled with it is gurranteed to be freed as well. If all the smart pointers are allocated on stack, as we know by the end of a program all stack memory gets properly popped, we can easily deduce that all the heap allocation is freed as well. The mechanism also extends to smart pointer themselves allocated on a heap, which itself is controled by another smart pointer on a stack. The memory dellocation will happen recursively to any depth level when the top level smart pointer dies.

智能指针是一种防止内存泄漏的机制。对于每一块堆内存分配，它都会将此分配与智能指针分配耦合起来，智能指针分配是一个控制堆内存生命周期的小对象。只要智能指针本身被破坏，与之相关的堆分配也保证被释放。如果所有智能指针都分配在堆栈上，正如我们所知，在程序结束时所有堆栈内存都会正确弹出，我们可以轻松推断出所有堆分配也被释放。该机制还扩展到在堆上分配的智能指针本身，其本身由堆栈上的另一个智能指针控制。当顶层智能指针死亡时，内存重新分配将递归地发生到任何深度级别。

One of the corner cases that makes this mechanism invalid is fiber. Imagine you have a fiber with its fiber stack on a heap. Inside this fiber, we use a smart pointer allocating some memory on a heap. However, the fiber then gets suspended and never gets resumed before the fiber gets destroyed. What will happen here is that the smart pointer sitting on the fiber stack, which is essentialy on a heap, will be leaked. This is different from allocating a object with a smart pointer as its member variable on a heap, when this object goes out of scope, it will destroy the heap allocation bundled with the smart pointer member. This can be done as the compiler is in a good position to make sure it happens. However, similar case mentioned won't work for fibers as the compiler knows nothing about how we use our fiber stack.

使该机制无效的极端情况之一是纤程。想象一下，您有一条纤程，其纤程堆栈位于一堆上。在该纤程内部，我们使用智能指针在堆上分配一些内存。然而，纤程随后会被暂停，并且在纤程被破坏之前永远不会恢复。这里会发生的是，位于纤程堆栈（本质上是堆）上的智能指针将被泄漏。这与在堆上分配一个以智能指针作为其成员变量的对象不同，当该对象超出范围时，它将破坏与智能指针成员捆绑的堆分配。这是可以完成的，因为编译器可以很好地确保它发生。然而，提到的类似情况不适用于纤程，因为编译器对我们如何使用纤程堆栈一无所知。

So with fibers, it is techinically possible to introduce memory leak even if your whole program's memory allocation is guarded with smart pointers. We certainly need to pay attention to avoid it. One way to make sure it won't happen is to leave a fiber switch right before the end of a FiberMain function and always execute to make the last fiber switch. And even with this, one needs to make sure there is no smart pointer whose life time still exists after the last fiber switch.

因此，对于纤程，即使整个程序的内存分配受到智能指针的保护，从技术上来说也有可能引入内存泄漏。我们当然需要注意避免它。确保不会发生这种情况的一种方法是在 FiberMain 函数结束之前留下纤程切换，并始终执行以进行最后一次纤程切换。即使这样，也需要确保在最后一次纤程切换后没有智能指针的生命周期仍然存在。

Object Destruction 物体破坏

We mentioned that we can't allow fiber entry function to exit normally. This means that we have to yield control to other fibers before it ends. This could mean we may still have active objects living on that stack, like the smart pointers we talked about. In more generalized sense, any objects, besides smart pointers, may need to destruct properly. Same as smart pointers lose its control over memory management, if we have an object taking advantage of its destructor to do something important, it may get skipped as well.

我们提到我们不能让纤程入口功能正常退出。这意味着我们必须在结束之前将控制权交给其他纤程。这可能意味着我们可能仍然有活动对象存在于该堆栈上，就像我们讨论的智能指针一样。从更广义的意义上来说，除了智能指针之外的任何对象都可能需要正确地析构。就像智能指针失去对内存管理的控制一样，如果我们有一个对象利用其析构函数来做一些重要的事情，它也可能会被跳过。

It is programmers' responsibility to make sure when a fiber gets destroyed, nothing left in the fiber needs to be executed. Commonly compilers can safe guard it for us, but not in a fiber environment.

程序员有责任确保当纤程被破坏时，纤程中剩余的任何内容都不需要执行。通常编译器可以为我们保护它，但不能在纤程环境中。

To be clear, compiler behavior is totally normal within a fiber stack. This means that if you have an object that lives on a call stack, which gets popped, the compiler will make sure the destroctor gets called properly. What we need to be careful about is to make sure no pending destructor needs to be executed when fiber gets destroyed.

需要明确的是，编译器行为在纤程堆栈中是完全正常的。这意味着，如果您有一个存在于调用堆栈上的对象，该对象被弹出，则编译器将确保正确调用析构函数。我们需要注意的是确保当 Fiber 被破坏时不需要执行挂起的析构函数。

No Fiber Reset in Windows' Fiber Interface

Windows 纤程接口中没有纤程重置

This is more of an inconvenience than a problem. Fiber is commonly seen in a job system in game engines. Such job systems commonly fixes thread on physical CPU cores through thread affinity. Fiber is like a job container, a job can only be executed when it finds an available idle fiber and a thread. Once it is done executing, the fiber will be put back to a pool of idle fibers. When we put a used fiber back to the idle fiber pool, we don't really care about its previous execution state anymore. A nice thing that can be done is to reset the fiber to its initial state before putting it in the idle fiber pool. This can be easily achieved through assembly implementation as we can simply reset the fiber context like we did when we created the fiber. Of course, we should not reset a fiber converted from a thread as it makes little sense anyway.

这与其说是一个问题，不如说是一种不便。 Fiber 常见于游戏引擎的作业系统中。此类作业系统通常通过线程关联将线程固定在物理 CPU 核心上。 Fiber 就像一个作业容器，作业只有找到可用的空闲 Fiber 和线程时才能执行。一旦执行完成，纤程将被放回到空闲纤程池中。当我们将使用过的纤程放回空闲纤程池时，我们不再关心它之前的执行状态。可以做的一件好事是在将纤程放入空闲纤程池之前将纤程重置为其初始状态。这可以通过装配实现轻松实现，因为我们可以像创建 Fiber 时一样简单地重置 Fiber 上下文。当然，我们不应该重置从线程转换而来的纤程，因为无论如何它都没有什么意义。

That simple solution has a problem as Windows doesn't provide an interface to reset a fiber. An unrealistic solution is to delete the fiber and recreate one every time we need to put it back in the idle pool, which pretty much works the same as no idle fiber pool, unfortunately. Since Windows create fiber interface doesn't allow us to allocate our own stack. Fiber allocation on Windows is couple with a memory allocation under the hood, making it a bit expensive. Given the high frequency of job execution during a frame in a game engine, this is by no means a good solution.

这个简单的解决方案有一个问题，因为 Windows 不提供重置纤程的接口。一种不切实际的解决方案是删除纤程并在每次需要将其放回空闲池时重新创建纤程，不幸的是，这与没有空闲纤程池的工作原理几乎相同。由于 Windows 创建 Fiber 接口不允许我们分配自己的堆栈。 Windows 上的纤程分配与底层的内存分配相结合，使其有点昂贵。考虑到游戏引擎中一帧期间作业执行的频率很高，这绝不是一个好的解决方案。

There are at least two solutions to this problem. One of the solutions is simply to implement an assebmly based fiber interface on Windows. This shouldn't be too hard at all since we have already implemented on both of x64 and Arm64 architecture. It is most likely just a matter of toggling a few macros.

这个问题至少有两种解决方案。解决方案之一就是在 Windows 上实现基于组件的纤程接口。这应该不会太难，因为我们已经在 x64 和 Arm64 架构上实现了。这很可能只是切换几个宏的问题。

The other solution is to put an inifinite loop inside a FiberMain function, like this.

另一个解决方案是在 FiberMain 函数中放置一个无限循环，如下所示。

cpp 复制代码

1void FiberMain(void* arg){
2    while(true){
3        // execute the task here
4        DoTask();
5
6        // yield the control back to another fiber
7        SwitchFiber(current_fiber, other_fiber);
8    }
9}

This goes beyond the topic of a fiber library itself. It is more about a job system. I'll briefly mention a few details here

这超出了纤程库本身的主题。它更多的是关于工作系统。我在这里简单提一下一些细节

An idle fiber should either start from the first line or line 8, which is the end of the last loop iteration.
空闲纤程应该从第一行或第 8 行开始，这是最后一次循环迭代的末尾。
Be mindful that it is totally legit for us to yield the control to any other fiber within the DoTask function. We can yield anywhere deep inside the fiber call stack.
请注意，我们将控制权交给 DoTask 函数内的任何其他纤程是完全合法的。我们可以在 Fiber 调用堆栈深处的任何地方进行屈服。
The other_fiber can be either a waiting-for-task fiber or a previously suspended fiber. Which fiber to pick is topic of job system scheduling problem.
other_fiber 可以是等待任务的纤程，也可以是先前挂起的纤程。选择哪条纤程是作业系统调度问题的主题。

Cross Thread Fiber Execution

跨线程纤程执行

Different job systems have different policies. There is one important decision to make in every fiber based job system. That is about whether to allow a suspended fiber to resume on another thread. There is clearly some trade off here to consider.

不同的就业制度有不同的政策。在每一个基于纤程的工作系统中都需要做出一个重要的决定。那是关于是否允许挂起的纤程在另一个线程上恢复。这里显然需要考虑一些权衡。

If we do allow doing so, we will have to implement all the system provided sychronization primitives like mutex, conditional variable. And we can't use thread local storage as freely as before. This is not to say we can use TLS at all, we just need to be careful that our TLS access pattern should not cross a fiber switch as before the switch, TLS could be from thread A and after the thread it could be from thread B, this will easily crash the problem.
如果我们允许这样做，我们将必须实现所有系统提供的同步原语，如互斥体、条件变量。而且我们不能像以前那样自由地使用线程本地存储了。这并不是说我们完全可以使用 TLS，我们只是需要小心，我们的 TLS 访问模式不应像交换机之前那样跨越纤程交换机，TLS 可能来自线程 A，也可能来自线程 B ，这很容易导致崩溃问题。
If we do not allow it. We can use all the above mentioned forbidden things. However, the load balancing may not be as good as the other way around. Think about there 4 threads ( 4 physical cores ), all of which are pulling for tasks. While the first thread somehow pulls 100 tasks, which all gets suspended soon after execution, while other three threads are just pulling tasks that never gets suspended. After the task pool is exhausted, the other three threads may be done executing at a later point. However, since the 100 tasks are already scheduled to thread 1 and if the system doesn't allow cross thread fiber execution, we will have to wait for thread 1 to finish executing all the 100 tasks to be done, when the other threads are waiting idling.
如果我们不允许的话。我们可以使用上面提到的所有禁止的东西。然而，负载平衡可能不如反之好。想想有 4 个线程（4 个物理核心），所有这些线程都在执行任务。第一个线程以某种方式拉取 100 个任务，这些任务在执行后很快就会挂起，而其他三个线程只是拉取从未挂起的任务。任务池耗尽后，其他三个线程可能会在稍后完成执行。然而，由于这 100 个任务已经被调度到线程 1，并且如果系统不允许跨线程纤程执行，我们将不得不等待线程 1 完成所有要完成的 100 个任务，而其他线程正在等待空转。

In an ideal world, for performance consideration, we should consider allowing cross thread fiber execution. This would certainly mean that we are taking a lot more responsibility than the other way around.

在理想的情况下，出于性能考虑，我们应该考虑允许跨线程纤程执行。这肯定意味着我们要承担比相反更多的责任。

Stay Vigelent Against Compiler Optimization

对编译器优化保持警惕

Compiler optimization has always been our best friend. It optimizes code for us without us doing anything low level. However, in such a fiber environment, where we hack low level registers, things can go very wrong if we are not careful enough.

编译器优化一直是我们最好的朋友。它为我们优化代码，而无需我们做任何低级的事情。然而，在这样的纤程环境中，如果我们不够小心，我们会破解低级寄存器，事情可能会变得非常错误。

To name a concrete example, just now we briefly mentioned that as long as the TLS memory access pattern is not crossing fiber switch, it should be fine. In reality, this turns out to be problematic due to a low level compiler optimization that the compiler is allowed to optimize TLS memory access with cache for better performance. To make it clear, let's take a look at the following code snippet.

举个具体的例子，刚才我们简单提到，只要TLS内存访问模式不是跨纤程交换机，就应该没问题。实际上，由于低级编译器优化，允许编译器通过缓存优化 TLS 内存访问以获得更好的性能，因此这是有问题的。为了清楚起见，让我们看一下下面的代码片段。

c 复制代码

 1thread_local int tls_data = 0;
 2void WINAPI FiberEntry(PVOID arg)
 3{
 40x00B21010  push        ebp  
 50x00B21011  mov         ebp,esp  
 60x00B21013  push        ecx  
 70x00B21014  push        esi  
 80x00B21015  mov         esi,dword ptr fs:[2Ch]  
 90x00B2101C  push        edi  
100x00B2101D  mov         edi,dword ptr [__imp__SwitchToFiber@4 (0B23004h)]  
110x00B21023  nop         dword ptr [eax]  
120x00B21027  nop         word ptr [eax+eax]  
13  while (true)
14  {
15    volatile int k = tls_data;
160x00B21030  mov         eax,dword ptr [esi]  
17    SwitchToFiber(thread_fiber);
180x00B21032  push        dword ptr [thread_fiber (0x0B253F4h)]  
190x00B21038  mov         eax,dword ptr [eax+4]  
200x00B2103E  mov         dword ptr [k],eax  
210x00B21041  call        edi  
22  }
230x00B21043  jmp         FiberEntry+20h (0x0B21030h)  
24}

Above is a mixed view between c++ and assembly code for better visibility. I purposely mark the temporay variable k with volatile to avoid compiler optimizing it out since it is not read anywhere.

上面是 C++ 和汇编代码之间的混合视图，以获得更好的可见性。我特意将临时变量 k 标记为易失性，以避免编译器优化它，因为它在任何地方都不会被读取。

A very subtle bug is hidden in this code. We can notice that the value of the volatile variable k is set from the register eax through line 20. And the eax comes from the esi through instruction at line 16. However, the esi value is loaded before the program enters the loop. So that said the compiler is trying to be smart by assuming that the code loop will always run on the same thread so that it can cache the memory fetch through line 8. This is not a bad assumption most of the time. However, we know that there is a legit risk that loop iterations could been executed on different threads. And this optimization will lead the program to read the TLS of incorrect thread, easily causing crash.

这段代码中隐藏着一个非常微妙的错误。我们可以注意到，易失性变量 k 的值是从寄存器 eax 到第20行设置的。 eax 来自 esi 值在程序进入循环之前加载。也就是说，编译器试图变得聪明，假设代码循环始终在同一个线程上运行，以便它可以缓存第 8 行的内存获取。大多数时候，这不是一个坏假设。但是，我们知道存在循环迭代可能在不同线程上执行的合法风险。而这种优化会导致程序读取到错误线程的TLS，容易导致崩溃。

On Windows, there is a dedicated flag /GT for avoiding such fiber unfriendly optimization. However, such a flag doesn't exist on some other platforms. In that case, what we can do is to prevent the compiler from being smart by isolating the access to TLS inside a non-inlined function. A common approach is to define the access method in a different compilation unit. As mentioned before, we still need to be careful about compilers' link time optimization to inline it again.

在 Windows 上，有一个专用标志 /GT 来避免这种纤程不友好的优化。但是，其他一些平台上不存在这样的标志。在这种情况下，我们能做的就是通过隔离非内联函数内部对 TLS 的访问来防止编译器变得聪明。一种常见的方法是在不同的编译单元中定义访问方法。如前所述，我们仍然需要小心编译器的链接时间优化以再次内联它。

Stepping into a FiberSwitch

步入纤程切换

Besides functionality, debugability is almost equally important.

除了功能之外，可调试性几乎同样重要。

Different from threads, suspended fibers are pretty much invisible to debuggers. For example, if we pause a function in a fiber, even if we have other suspended fibers in the air, Visual Studio's parallel call stack will not have visibility of the suspended fibers. This certainly makes debugging a bit tricky in some cases, especially when it involves sychronization issues. I personally found out that printing log is a viable option to gain more information about suspended fibers.

与线程不同，暂停的纤程对于调试器来说几乎是不可见的。例如，如果我们暂停某个纤程中的某个函数，即使空中还有其他暂停的 Fiber，Visual Studio 的并行调用堆栈也不会看到暂停的 Fiber 的可见性。在某些情况下，这肯定会使调试变得有点棘手，特别是当涉及同步问题时。我个人发现打印日志是获取有关暂停纤程的更多信息的可行选择。

Another detail that we need to pay attention is the ability to step into a fiber switch call. Most of the time, we don't care about the detailed implementation in it, but our bottom line is that we should be able to step through this call to get over to the other side of the fiber switch, the target fiber code. GDB and LLDB work pretty well for this as fiber implementation is done through assembly code. However, Visual Studio has a flag that has big impact on the behavior whenever it comes to steping into a fiber switch. One can locate this flag with the following setup, Project Property Page -> Configuration Properties -> Advanced -> Advanced Properties -> Use Debug Library. If we want to step into the fiber switch function like we do with other regular functions, this needs to be set to true. Otherwise, the debugger will simply step over it without going to the other side of the fiber switch.

我们需要注意的另一个细节是进入纤程交换机呼叫的能力。大多数时候，我们并不关心其中的详细实现，但我们的底线是我们应该能够单步执行此调用以到达 Fiber 交换机的另一端，即目标 Fiber 代码。 GDB 和 LLDB 对此工作得很好，因为 Fiber 实现是通过汇编代码完成的。然而，Visual Studio 有一个标志，每当涉及到进入纤程交换机时，该标志都会对行为产生很大影响。可以通过以下设置找到此标志： Project Property Page -> Configuration Properties -> Advanced -> Advanced Properties -> Use Debug Library

Avoid Making Any Blocking Calls

避免拨打任何阻塞电话

Certain opertions, like IO read, will block the thread execution for a while because of the wait. When such a block happens, the OS commonly put this thread on hold and assign some other thread to the physical core for further execution to utilize the available hardware cores.

某些操作（例如 IO 读取）会因为等待而阻塞线程执行一段时间。当发生此类阻塞时，操作系统通常会暂停该线程，并将其他线程分配给物理核心以进一步执行，以利用可用的硬件核心。

However, fiber based job system commonly use thread affinity to fix threads on physical cores. The number of threads should be the same with the number of physically available cores. Fiber is our new user mode thread concept that allows task switching a lot faster. We should be extremely careful to avoid making blocking calls in a fiber, there will be no other thread in the same application for the OS to schedule anymore to fill in the gap while waiting for IO.

然而，基于纤程的作业系统通常使用线程关联来固定物理核心上的线程。线程数应与物理可用核心数相同。 Fiber 是我们新的用户模式线程概念，它允许任务切换速度更快。我们应该非常小心，避免在纤程中进行阻塞调用，因为在等待 IO 时，同一应用程序中不会再有其他线程可供操作系统调度来填补间隙。

However, we can't avoid IO calls in a game. A solution in game engine is to allocate background threads that is dedicated for such calls. And only execute blocking calls on those threads rather than any fibers to avoid it.

然而，我们无法避免游戏中的 IO 调用。游戏引擎中的一个解决方案是分配专用于此类调用的后台线程。并且仅在这些线程而不是任何纤程上执行阻塞调用来避免这种情况。

Summary 概括

In summary, we mentioned a lot about fibers in this post. Starting from the very basics about CPU architecture, and then a detailed fiber implementation, all the way to the problems with fibers.

总之，我们在这篇文章中提到了很多关于纤程的内容。从 CPU 架构的基础知识开始，然后是详细的纤程实现，一直到纤程的问题。

As we can see, fiber offer great flexibility that is commonly not available through other methods, which is why it is favored by game studios seeking for better performance. Of course, the power of fiber is clearly not limited to game development alone. It can be used in mostly all CPU computational intensive software that cares about performance.

正如我们所看到的，纤程提供了其他方法通常无法提供的巨大灵活性，这就是为什么它受到寻求更好性能的游戏工作室的青睐。当然，纤程的力量显然不仅仅局限于游戏开发。它可用于几乎所有关心性能的 CPU 计算密集型软件 。