openresty容器导出火焰图

我们线上使用的是容器部署的openresty作为网关，用于处理请求转发以及使用lua代理处理部分请求。有时候会出现在流量没有增加特别多，但是openresty的cpu跑满的情况。为了能快速查出瓶颈并解决问题，我们增加了一些监控，用于排查各种情况导致的nginx cpu升高问题。这里主要介绍下 openresty导出cpu火焰图的方式，通过cpu火焰图可以快速发现占用cpu高的函数，从而协助快速定位问题以及进行性能优化。

目前网上的文档大部分都是在主机上导出火焰图的，这篇文章介绍了导出容器部署的openresty cpu火焰图的方式。

这里先给两张示例图，cpu火焰图分为nginx的c代码 和 处理业务的lua代码 两种。

nginx火焰图

lua火焰图

导出火焰图流程

一、 准备工作

1. 安装内核debug包

在导出火焰图之前，有一些准备工作需要处理，首先是安装一些内核的debug信息包和开发包，这样在导出火焰图时程序才知道当前执行的函数名称，不然导出的火焰图只有一些很简略的信息，比如[unknown]，[libc-2.28.so] 这种。

安装时需要先根据系统版本找到对应的debug包，输入命令uname -a查询操作系统版本，这里以3.10.0-1160.76.1.el7.x86_64为例，首先找到下载对应内核debug包的网站，然后在机器上通过wget下载。

需要下载的包
kernel-debuginfo-3.10.0-1160.76.1.el7.x86_64.rpm
kernel-debuginfo-common-x86_64-3.10.0-1160.76.1.el7.x86_64.rpm
kernel-devel-3.10.0-1160.76.1.el7.x86_64.rpm

下载网站参考
http://debuginfo.centos.org/
https://buildlogs.centos.org/c7.2009.u.x86_64/kernel/

执行命令下载
wget http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-3.10.0-1160.76.1.el7.x86_64.rpm 
wget http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-common-x86_64-3.10.0-1160.76.1.el7.x86_64.rpm 
wget https://buildlogs.centos.org/c7.2009.u.x86_64/kernel/20220810161826/3.10.0-1160.76.1.el7.x86_64/kernel-devel-3.10.0-1160.76.1.el7.x86_64.rpm

安装
rpm -ivh *3.10.0-1160.76.1.el7.x86_64.rpm

2. 安装perf，stapxx和FlameGraph

perf用于导出nginx c语言的cpu剖析信息，stapxx用于导出lua代码的cpu剖析信息，而FlameGraph则是用于将导出的cpu剖析信息转换为可阅读的svg格式的火焰图。

• perf：执行yum install -y perf命令即可。
• stapxx： git clone github.com/openresty/s...
• FlameGraph： git clone github.com/brendangreg...

3. 制作导出lua火焰图的工具镜像

这里制作一个镜像，用于导出容器内的openresty的cpu剖析信息。需要制作镜像的原因我在这里说明下。

之前尝试过几种方式导出容器内的openresty的cpu剖析信息，但是都不好处理。

• 直接在主机上导出：因为导出lua的cpu剖析信息的话，需要有对应openresty版本的debug包，不然无法显示函数名称。因此尝试了在主机上编译并安装 容器中使用版本的openresty，然后加上debug信息，但是发现很难编译出和容器中完全一样的nginx二进制，导出cpu信息时会报错说 openresty的构建ID 不一致，并且主机和容器里面的虚拟主机系统有区别，会导致stapxx等程序有问题，比如在主机上运行stapxx时，需要修改对应stapxx/tapset/luajit.sxx的代码，改为32位的，但是容器中启动openresty的操作系统是64位的，就会报错。

• 编译一个包含主机内核的debug包的容器，在容器内运行stapxx导出lua的cpu剖析信息，但是导出的信息会变少甚至报错，因为docker中的操作系统其实不包含内核，因此在docker容器中安装主机的内核debug包会报错，但是安装容器内操作系统的debug包，又会找不到信息。因为实际上运行的操作系统内核只有主机上的内核。

  使用镜像的原因总结：

1. 在主机上很难打包出和容器中完全一致的openresty，因此直接使用线上的容器openresty制作一个镜像，解决openresty版本一致的问题。
2. 需要安装openresty的debug信息包，因为使用容器openresty，这个就好处理了，直接在生成镜像时运行命令debuginfo-install -y openresty-1.21.4.1-1.el8.x86_64 安装
3. 一个主机只有一个内核，就是主机上的操作系统版本的内核，但是容器内启动openresty的操作系统版本可能和主机的不一致，这个怎么处理。 其实不用管容器中的操作系统，因为导出cpu剖析信息时是需要从内核抓取函数代码的，而内核的版本是主机上操作系统的版本，因此在容器中挂载主机上的 /lib/modules，/sys/kernel/debug和/usr/src/kernels目录，这样stapxx就能找到执行函数对应的名称了。

下面提供下打镜像的文件和流程。

dockerfile文件：
FROM nginx:10.9.0
RUN yum -y install systemtap && debuginfo-install -y openresty-1.21.4.1-1.el8.x86_64 
ENTRYPOINT ["tail", "-f", "/dev/null"]

生成镜像：
docker build -t openresty-debug:2.0.0 .

容器启动命令是：
docker run --rm --name debug --pid=host -v /lib/modules:/lib/modules -v /root/debug:/root/debug -v /sys/kernel/debug:/sys/kernel/debug -v /usr/src/kernels:/usr/src/kernels --privileged --entrypoint /bin/bash  openresty-debug:2.0.0  -c "cd /root/debug/stapxx/ && export PATH=\$PWD:\$PATH && ./samples/lj-lua-stacks.sxx --skip-badvars --arg time=30 -x $top_nginx_worker_pid > /root/debug/a.bt"

其中/lib/modules，/sys/kernel/debug和/usr/src/kernels是把主机上的kernel debug包挂载进来
之前安装的stapxx，FlameGraph 我是放在/root/debug文件夹下，如果需要放在其他目录下，可以修改docker启动命令，将目录挂载到容器中。

--pid=host是让容器使用主机的pid空间，才能查到nginx的pid

4. 修改stapxx脚本的代码

直接使用stapxx运行可能会出现如下所示的错误。

semantic error: unable to find member 'gcptr32' for struct GCRef (alternatives: gcptr64): operator '->' at :449:100
        source:     gco = @cast(pt, "GCproto", "/usr/local/openresty/luajit/lib/libluajit-5.1.so.2.1.0")->chunkname->gcptr32
        
或者

semantic error: unable to find member 'fr' for union TValue (alternatives: gcr, i, n, it, u32, u64, ftsz, it64): operator '->' at :75:83
        source:     @cast(@tv, "TValue", "/usr/local/openresty/luajit/lib/libluajit-5.1.so.2.1.0")->fr->tp->ftsz

这是因为openresty会因为系统不同而使用不同的宏编译，具体的openresty代码可以通过 bundule/LuaJIT-2.1-20220411/src/lj_obj.h和lj_arch.h文件进行查看，主要是openresty使用了和stapxx默认代码的不同位数的指针导致报错，这里只需要修改stapxx的 tapset/luajit.sxx的代码，根据报错的提示进行修改就可以了，比如将gcptr32批量替换成gcptr64。

可选，下载openresty代码查看：
wget https://openresty.org/download/openresty-1.21.4.1.tar.gz
主要查看bundule/LuaJIT-2.1-20220411/src/lj_obj.h和lj_arch.h  ，需要对比stapxx中结构和openresty中的结果，然后进行修改

执行命令批量替换：
sed -i 's/gcptr32/gcptr64/g' stapxx/tapset/luajit.sxx
sed -i 's/->fr->tp->ftsz/->ftsz/g' stapxx/tapset/luajit.sxx
sed -i 's/->fr->tp->pcr->ptr32/->ftsz/g' stapxx/tapset/luajit.sxx
sed -i 's/->ptr32/->ptr64/g' stapxx/tapset/luajit.sxx
sed -i 's/->fr->func/->gcr/g' stapxx/tapset/luajit.sxx

二、 导出openresty火焰图

1. 导出nginx的c语言的火焰图

这个比较简单，只要执行perf命令就可以导出，然后通过FlameGraph转成火焰图，其中$top_nginx_worker_pid变量是nginx worker的pid

perf record -F 99 -g -p "$top_nginx_worker_pid" -- sleep 30
perf script | ./FlameGraph/stackcollapse-perf.pl| ./FlameGraph/flamegraph.pl > ./ngx_c_flame.svg

2. 导出lua的火焰图

通过执行docker命令导出lua的cpu剖析信息，然后通过FlameGraph转成火焰图。

docker run --rm --name debug --pid=host -v /lib/modules:/lib/modules -v /root/debug:/root/debug -v /sys/kernel/debug:/sys/kernel/debug -v /usr/src/kernels:/usr/src/kernels --privileged --entrypoint /bin/bash  openresty-debug:2.0.0  -c "cd /root/debug/stapxx/ && export PATH=\$PWD:\$PATH && ./samples/lj-lua-stacks.sxx --skip-badvars --arg time=30 -x $top_nginx_worker_pid > /root/debug/a.bt"

./FlameGraph/stackcollapse-stap.pl a.bt > a.cbt
./FlameGraph/flamegraph.pl a.cbt  > ngx_lua_flame.svg

3. 参考脚本

最后贴一下整体的生成脚本，用于参考

#!/bin/bash

get_top_cpu_nginx_worker_pid() {
    # 使用 top 命令获取进程信息，筛选出 Nginx worker 进程
    local top_pid=$(top -b -n 1 | awk '/nginx/ {print $1, $9}' | sort -k2 -nr | head -n 1 | awk '{print $1}')
    echo "$top_pid"
}

# 调用函数并输出结果
top_nginx_worker_pid=$(get_top_cpu_nginx_worker_pid)
if [ -n "$top_nginx_worker_pid" ]; then
    echo "CPU 使用率最高的 Nginx worker 进程 PID: $top_nginx_worker_pid"
else
    echo "未找到 Nginx worker 进程。"
    exit 1
fi

cd /root/debug

echo "perf nginx cpu flame,wait 30s"
perf record -F 99 -g -p "$top_nginx_worker_pid" -- sleep 30
perf script | ./FlameGraph/stackcollapse-perf.pl| ./FlameGraph/flamegraph.pl > ./ngx_c_flame.svg

echo "perf nginx cpu flame done"

echo "stap nginx lua flame,wait 30s"

docker run --rm --name debug --pid=host -v /lib/modules:/lib/modules -v /root/debug:/root/debug -v /sys/kernel/debug:/sys/kernel/debug -v /usr/src/kernels:/usr/src/kernels --privileged --entrypoint /bin/bash  openresty-debug:2.0.0  -c "cd /root/debug/stapxx/ && export PATH=\$PWD:\$PATH && ./samples/lj-lua-stacks.sxx --skip-badvars --arg time=30 -x $top_nginx_worker_pid > /root/debug/a.bt"

./FlameGraph/stackcollapse-stap.pl a.bt > a.cbt
./FlameGraph/flamegraph.pl a.cbt  > ngx_lua_flame.svg

echo "stap nginx lua flame done"

这里也贴一下stapxx/tapset/luajit.sxx的代码，用于参考：

// module luajit


@use nginx.lua


$*pt := @cast(pt, "GCproto", "$^libluajit_path")


//@define LJ_TLIGHTUD  %( 4294967292 %)
@define LJ_TLIGHTUD  %( 4294901760 %)
@define LJ_TSTR  %( 4294967291 %)
@define LJ_TUDATA  %( 4294967283 %)
@define LJ_TFUNC  %( 4294967287 %)


@define TSTR    %( 4 %)
@define TUPVAL  %( 5 %)
@define TTHREAD %( 6 %)
@define TPROTO  %( 7 %)
@define TFUNC   %( 8 %)
@define TTRACE  %( 9 %)
@define TCDATA  %( 10 %)
@define TTAB    %( 11 %)
@define TUDATA  %( 12 %)

@define CTSHIFT_NUM  %( 28 %)
@define CT_HASSIZE   %( 5 %)
@define CT_ATTRIB    %( 8 %)
@define CTMASK_CID   %( 0xffff %)

@define FF_LUA %( 0 %)
@define FF_C   %( 1 %)

@define FRAME_LUA   %( 0 %)
@define FRAME_CONT  %( 2 %)
@define FRAME_TYPE  %( 3 %)
@define FRAME_P     %( 4 %)
@define FRAME_TYPEP %( (@FRAME_TYPE|@FRAME_P) %)

@define NO_BCPOS    %( ~0 %)

@define sizeof_GCtab  %( &@cast(0, "GCtab", "$^libluajit_path")[1] %)
@define sizeof_TValue  %( &@cast(0, "TValue", "$^libluajit_path")[1] %)
@define sizeof_lua_State %( &@cast(0, "struct lua_State", "$^libluajit_path")[1] %)
@define sizeof_GCfunc %( &@cast(0, "union GCfunc", "$^libluajit_path")[1] %)
@define sizeof_GCupval %( &@cast(0, "GCupval", "$^libluajit_path")[1] %)
@define sizeof_GCstr %( &@cast(0, "GCstr", "$^libluajit_path")[1] %)
@define sizeof_GCudata %( &@cast(0, "GCudata", "$^libluajit_path")[1] %)
@define sizeof_Node %( &@cast(0, "Node", "$^libluajit_path")[1] %)
@define sizeof_GCfuncC %( &@cast(0, "GCfuncC", "$^libluajit_path")[1] %)
@define sizeof_GCfuncL %( &@cast(0, "GCfuncL", "$^libluajit_path")[1] %)
@define sizeof_GCRef %( &@cast(0, "GCRef", "$^libluajit_path")[1] %)
@define sizeof_GCproto %( &@cast(0, "GCproto", "$^libluajit_path")[1] %)
@define sizeof_GCtrace %( &@cast(0, "GCtrace", "$^libluajit_path")[1] %)
@define sizeof_GCcdata %( &@cast(0, "GCcdata", "$^libluajit_path")[1] %)
@define sizeof_IRIns %( &@cast(0, "IRIns", "$^libluajit_path")[1] %)
@define sizeof_IRRef %( &@cast(0, "IRRef", "$^libluajit_path")[1] %)
@define sizeof_SnapShot %( &@cast(0, "SnapShot", "$^libluajit_path")[1] %)
@define sizeof_SnapEntry %( &@cast(0, "SnapEntry", "$^libluajit_path")[1] %)
@define sizeof_K64Array %( &@cast(0, "K64Array", "$^libluajit_path")[1] %)
@define sizeof_ptr %( &@cast(0, "global_State", "$^libluajit_path")->strmask %)
@define sizeof_GCcdataVar %( &@cast(0, "GCcdataVar", "$^libluajit_path")[1] %)
@define sizeof_BCIns %( &@cast(0, "BCIns", "$^libluajit_path")[1] %)

@define strdata(s) %(
    (@s + @sizeof_GCstr)
%)

@define frame_gc(frame) %(
    @cast(@frame, "TValue", "$^libluajit_path")->gcr->gcptr64
%)

@define frame_ftsz(tv) %(
    @cast(@tv, "TValue", "$^libluajit_path")->ftsz
%)

@define frame_type(f) %(
    (@frame_ftsz(@f) & @FRAME_TYPE)
%)

@define frame_typep(f) %(
    (@frame_ftsz(@f) & @FRAME_TYPEP)
%)

@define frame_islua(f) %(
    (@frame_type(@f) == @FRAME_LUA)
%)

@define frame_iscont(f) %(
    (@frame_typep(@f) == @FRAME_CONT)
%)

@define frame_pc(tv) %(
    @cast(@tv, "TValue", "$^libluajit_path")->ftsz
%)

@define frame_contpc(f) %(
    (@frame_pc(@f) - 1 * @sizeof_BCIns)
%)

@define bc_a(i) %(
    ((@i >> 8) & 0xff)
%)

@define frame_prevl(f) %(
    (@f - (1 + @bc_a(user_uint32(@frame_pc(@f) - 4))) * @sizeof_TValue)
%)

@define FRAME_VARG  %( 3 %)

@define frame_isvarg(f) %(
    (@frame_typep(@f) == @FRAME_VARG)
%)

@define frame_sized(f) %(
    (@frame_ftsz(@f) & ~@FRAME_TYPEP)
%)

@define frame_prevd(f) %(
    (@f - @frame_sized(@f))
%)

@define isluafunc(fn) %(
    (@cast(@fn, "GCfunc", "$^libluajit_path")->c->ffid == @FF_LUA)
%)

@define isffunc(fn) %(
    (@cast(@fn, "GCfunc", "$^libluajit_path")->c->ffid > @FF_C)
%)

@define funcproto(fn) %(
    (@cast(@fn, "GCfunc", "$^libluajit_path")->l->pc->ptr64 - @sizeof_GCproto)
%)

@define proto_bc(pt) %(
    (@pt + @sizeof_GCproto)
%)

@define proto_bcpos(pt, pc) %(
    ((@pc - @proto_bc(@pt)) / @sizeof_BCIns)
%)

@define proto_lineinfo(pt) %(
    @cast(@pt, "GCproto", "$^libluajit_path")->lineinfo->ptr64
%)


global luajit_cfunc_cache


function luajit_frame_func(f) {
    gco = @frame_gc(f)
    $*gco := @cast(gco, "GCobj", "$^libluajit_path")
    return &$*gco->fn
}


function luajit_G(L)
{
    $*L := @cast(L, "lua_State", "$^libluajit_path")
    return $*L->glref->ptr64
}


function luajit_gcref(r)
{
    $*r := @cast(r, "GCRef", "$^libluajit_path")
    return $*r->gcptr64
}


function luajit_objlen(o, gct, g)
{
    $*o := @cast(o, "GCobj", "$^libluajit_path")
    if (gct == @TSTR) {
        /*
        if ($*o->str->len == 0) {
            printf("empty string found.\n")
        }
        */
        return $*o->str->len + 1 + @sizeof_GCstr
    }

    if (gct == @TTAB) {
        t = &$*o->tab
        $*t := @cast(t, "GCtab", "$^libluajit_path")
        asize = $*t->asize
        hmask = $*t->hmask

        n = 0
        if (hmask > 0) {
            n += @sizeof_Node * (hmask + 1)
        }

        if (asize > 0 && $*t->colo <= 0) {
            n += @sizeof_TValue * asize
        }

        if ($*t->colo) {
            n += ($*t->colo & 0x7f) * @sizeof_TValue + @sizeof_GCtab

        } else {
            n += @sizeof_GCtab
        }

        return n
    }

    if (gct == @TUDATA) {
        return $*o->ud->len + @sizeof_GCudata
    }

    if (gct == @TPROTO) {
        return $*o->pt->sizept
    }

    if (gct == @TTHREAD) {
        L = &$*o->th
        //printf("open upval: %p\n", luajit_gcref(&$*L->openupval))
        //printf("lua_State: %d\n", @sizeof_lua_State)
        n = @sizeof_lua_State + $*L->stacksize * @sizeof_TValue
        p = &$*L->openupval
        while (p) {
            o = luajit_gcref(p)
            if (o == 0) {
                break;
            }

            $*o := @cast(o, "GCobj", "$^libluajit_path")
            gct = $*o->gch->gct
            size = luajit_objlen(o, gct, g)
            //printf("%s: %d\n", typenames[@TSTR], size)
            n += size
            p = &$*o->gch->nextgc
        }

        return n
    }

    if (gct == @TFUNC) {
        fn = &$*o->fn
        $*fn := @cast(fn, "GCfunc", "$^libluajit_path");
        if (@isluafunc(fn)) {
            n = $*fn->l->nupvalues
            //n = 0
            return @sizeof_GCfuncL - @sizeof_GCRef + @sizeof_GCRef * n
        }

        n = $*fn->c->nupvalues
        //n = 0
        return @sizeof_GCfuncC - @sizeof_TValue + @sizeof_TValue * n
    }

    if (gct == @TUPVAL) {
        return @sizeof_GCupval
    }

    if (gct == @TTRACE) {
        T = o
        $*T := @cast(T, "GCtrace", "$^libluajit_path")
        return ((@sizeof_GCtrace + 7) & ~7)
               + ($*T->nins - $*T->nk) * @sizeof_IRIns
               + $*T->nsnap * @sizeof_SnapShot
               + $*T->nsnapmap * @sizeof_SnapEntry
    }

    $*g := @cast(g, "global_State", "$^libluajit_path")

    if (gct == @TCDATA) {
        cd = o
        $*cd := @cast(cd, "GCcdata", "$^libluajit_path")
        if ($*cd->marked & 0x80) {
            /* cdata is a vector */
            cdatav = cd - @sizeof_GCcdataVar
            $*cdatav := @cast(cdatav, "GCcdataVar", "$^libluajit_path")
            return $*cdatav->len + $*cdatav->extra
        }

        /* cdata is not a vector */
        cts = $*g->ctype_state->ptr64
        $*cts := @cast(cts, "CTState", "$^libluajit_path")
        id = $*cd->ctypeid
        ct = &$*cts->tab[id]
        $*ct := @cast(ct, "CType", "$^libluajit_path")
        while (($*ct->info >> @CTSHIFT_NUM) == @CT_ATTRIB) {
            //printf("XXX skipping c type attribute\n")
            ct = &$*cts->tab[$*ct->info & @CTMASK_CID]
        }

        if (($*ct->info >> @CTSHIFT_NUM) <= @CT_HASSIZE) {
            sz = $*ct->size

        } else {
            sz = @sizeof_ptr
        }

        //printf("GCcdata size: %d\n", @sizeof_GCcdata)
        return @sizeof_GCcdata + sz
    }

    return 0
}


function luajit_jit_state_size(J)
{
    $*J := @cast(J, "jit_State", "$^libluajit_path")

    ir_k64_size = 0
    // In the newer revision of luajit, 64-bit constants are represented as an
    // array of TValue. Uncomment following code if the luajit being used
    // is using linked list to represent 64-bit constants. As the 64-bit
    // constants don't take lots of memory, it does not seems to be a big deal
    // if you don't uncomment following code for the older luajit.
    //
    /*
    $*k := @cast(k, "K64Array", "$^libluajit_path")
    for (k = $*J->k64->ptr64; k; ) {
        ir_k64_size += @sizeof_K64Array
        k = $*k->next->ptr64
    }
    if (ir_k64_size) {
        //printf("64-bit constants: %d\n", ir_k64_size)
    }
    */

    sizesnapmap = $*J->sizesnapmap * @sizeof_SnapEntry
    if (sizesnapmap) {
        //printf("JIT snap map buffer size: %d\n", sizesnapmap)
    }

    sizesnap = $*J->sizesnap * @sizeof_SnapShot
    if (sizesnap) {
        //printf("JIT snap buffer size: %d\n", sizesnap)
    }

    sizeirbuf = ($*J->irtoplim - $*J->irbotlim) * @sizeof_IRIns
    if (sizeirbuf) {
        //printf("JIT IR buffer size: %d\n", sizeirbuf)
    }

    sizetrace = $*J->sizetrace * @sizeof_GCRef
    if (sizetrace) {
        //printf("JIT trace array size: %d\n", sizetrace)
    }

    return ir_k64_size + sizesnapmap + sizesnap + sizeirbuf + sizetrace
}


/* returns a TValue* pointer */
function luajit_index2adr(L, idx)
{
    if (idx > 0) {
        o = $*L->base + (idx - 1) * @sizeof_TValue
        return o < $*L->top ? o : 0
    }

    top = $*L->top

    if (idx != 0 && -idx <= top - $*L->base) {
        return top + idx * @sizeof_TValue;
    }

    return 0
}


/* convert GCstr to stap string */
function luajit_unbox_gcstr(gcs)
{
    if (gcs == 0) {
        return ""
    }

    $*gcs := @cast(gcs, "GCstr", "$^libluajit_path")
    src = @strdata(gcs)
    return user_string_n_warn(src, $*gcs->len)
}


function luajit_tostring(L, idx)
{
    $*o := @cast(o, "TValue", "$^libluajit_path")
    o = luajit_index2adr(L, idx)
    if (o == 0) {
        return "<nil>"
    }

    if ($*o->it == @LJ_TSTR) {
        gco = $*o->gcr->gcptr64
        $*gco := @cast(gco, "GCobj", "$^libluajit_path")
        s = &$*gco->str
        $*s := @cast(s, "GCstr", "$^libluajit_path")
        return user_string_n(s + @sizeof_GCstr, $*s->len)
    }

    return "<unknown>"
}


function luajit_tostringlen(L, idx)
{
    $*o := @cast(o, "TValue", "$^libluajit_path")
    o = luajit_index2adr(L, idx)
    if (o == 0) {
        return -1
    }

    if ($*o->it == @LJ_TSTR) {
        gco = $*o->gcr->gcptr64
        $*gco := @cast(gco, "GCobj", "$^libluajit_path")
        s = &$*gco->str
        $*s := @cast(s, "GCstr", "$^libluajit_path")
        return $*s->len
    }

    return -1
}


function luajit_touserdata(L, idx)
{
    $*o := @cast(o, "TValue", "$^libluajit_path")
    o = luajit_index2adr(L, idx)
    if (o == 0) {
        return 0
    }

    //printf("udata type: %d (%d)\n", $*o->it, ($*o->it >> 15))

    if ($*o->it == @LJ_TUDATA) {
        gco = $*o->gcr->gcptr64
        ud = &$*gco->ud
        $*ud := @cast(ud, "GCudata", "$^libluajit_path")
        return ud + @sizeof_GCudata
    }

    if ($*o->it == @LJ_TLIGHTUD) {
        return $*o->u64 & 0x7fffffffffff
    }

    return 0
}


function luajit_proto_chunkname(pt) {
    gco = $*pt->chunkname->gcptr64
    return &$*gco->str
}


function luajit_debug_framepc(L, T, fn, pt, nextframe) {
    //printf("debug framepc: L=%p, fn=%p, nextframe = %p\\n", L, fn, nextframe)
    if (nextframe == 0) {
        return @NO_BCPOS
    }

    if (@frame_islua(nextframe)) {
        ins = @frame_pc(nextframe);
        //printf("frame is lua, ins = %p\\n", ins)

    } else if (@frame_iscont(nextframe)) {
        //println("frame is cont")
        ins = @frame_contpc(nextframe)

    } else {
        /* TODO: add support for cframe */
    }

    //printf("debug framepc: ins = %p\\n", ins)
    pos = @proto_bcpos(pt, ins) - 1;
    sizebc = $*pt->sizebc
    if (pos > sizebc) {
        //printf("Undo the effects of lj_trace_exit for JLOOP.\n")
        if (!T) {
            T = ins - 1 * @sizeof_BCIns - &@cast(0, "GCtrace", "$^libluajit_path")->startins;
        }
        //printf("startpc: %d\n", $*T->startpc->ptr64)
        pos = @proto_bcpos(pt, $*T->startpc->ptr64);
        //printf("Found adjusted position: %d\n", pos)
    }
    return pos;
}


function luajit_debug_line(pt, pc)
{
    lineinfo = @proto_lineinfo(pt)
    //printf("lj_debug_line: lineinfo = %p, %x <= %x\\n", lineinfo,
           //pc, $pt->sizebc)
    sizebc = $*pt->sizebc

    if (pc <= sizebc && lineinfo) {
        first = $*pt->firstline

        if (pc == sizebc) {
            return first + $*pt->numline
        }

        if (pc-- == 0) {
            return first
        }

        if ($*pt->numline < 256) {
            return first + @cast(lineinfo, "uint8_t", "$^libluajit_path")[pc]
        }

        if ($*pt->numline < 65536) {
            return first + @cast(lineinfo, "uint16_t", "$^libluajit_path")[pc]
        }

        return first + @cast(lineinfo, "uint32_t", "$^libluajit_path")[pc]
    }

    return -1
}


function luajit_debug_frameline(L, T, fn, pt, nextframe)
{
    pc = luajit_debug_framepc(L, T, fn, pt, nextframe)
    if (pc != @NO_BCPOS) {
        if (pc <= $*pt->sizebc) {
            return luajit_debug_line(pt, pc)
        }
    }

    return -1
}


function luajit_debug_dumpstack(L, T, depth, base, simple)
{
    level = 0
    dir = 1
    if (depth < 0) {
        level = ~depth
        depth = dir = -1
    }
    bt = ""
    while (level != depth) {
        /* lj_debug_frame(L, level, &size) {{{ */
        bot = $*L->stack->ptr64
        found_frame = 0
        tmp_level = level
        /* Traverse frames backwards. */
        for (nextframe = frame = base - @sizeof_TValue; frame > bot; ) {
            if (@frame_gc(frame) == L) {
                tmp_level++
            }

            if (tmp_level-- == 0) {
                size = (nextframe - frame) / @sizeof_TValue
                found_frame = 1
                break
            }
            nextframe = frame
            if (@frame_islua(frame)) {
                frame = @frame_prevl(frame)

            } else {
                if (@frame_isvarg(frame)) {
                    tmp_level++;  /* Skip vararg pseudo-frame. */
                }

                frame = @frame_prevd(frame);
            }
        }

        if (!found_frame) {
            frame = 0
            size = tmp_level
        }
        /* }}} */

        if (frame) {
            //nextframe = size ? frame + size * @sizeof_TValue : 0
            fn = luajit_frame_func(frame)
            if (fn == 0) {
                return ""
            }
            if (@isluafunc(fn)) {
                pt = @funcproto(fn)

                if (simple) {
                    line = $*pt->firstline

                } else {
                    line = luajit_debug_frameline(L, T, fn, pt, nextframe)
                    if (line < 0) {
                        line = $*pt->firstline
                        //printf("using firstline %d\n", line)

                    } else {
                        //printf("GOT the lineno %d\n", line)
                    }
                }

                name = luajit_proto_chunkname(pt)  /* GCstr *name */
                if (name == 0) {
                    return ""
                }

                path = luajit_unbox_gcstr(name)
                bt .= sprintf("%s:%d\n", path, line)

            } else if (@isffunc(fn)) {
                bt .= sprintf("builtin#%d\n", $*fn->c->ffid)

            } else {
                /* C function */
                cfunc = $*fn->c->f
                sym = luajit_cfunc_cache[cfunc]
                if (sym == "") {
                    sym = sprintf("C:%s\n", usymname(cfunc))
                    luajit_cfunc_cache[cfunc] = sym
                }
                bt .= sym
            }

        } else if (dir == 1) {
            break

        } else {
            level -= size
        }
        level += dir
    }

    return bt
}


function luajit_cur_thread(g)
{
    $*g := @cast(g, "global_State", "$^libluajit_path")
    gco = $*g->cur_L->gcptr64
    if (gco == 0) {
        return 0
    }

    return &$*gco->th
}


function luajit_get_trace(g, traceno)
{
    GG = g - &@cast(0, "GG_State", "$^libluajit_path")->g
    $*GG := @cast(GG, "GG_State", "$^libluajit_path")
    J = &$*GG->J
    $*J := @cast(J, "jit_State", "$^libluajit_path")
    if (J == 0) {
        return 0
    }

    return $*J->trace[traceno]->gcptr64
}


function luajit_trace_starting_func(g, trace)
{
    $*trace := @cast(trace, "GCtrace", "$^libluajit_path")
    gco = $*trace->startpt->gcptr64
    if (gco == 0) {
        return ""
    }

    pt = &$*gco->pt

    firstline = $*pt->firstline

    name = luajit_proto_chunkname(pt)  /* GCstr *name */
    path = luajit_unbox_gcstr(name)

    return sprintf("%s:%d", path, firstline)
}


function luajit_print_backtrace(simple)
{
    if (@defined(@var("globalL", "$^exec_path"))) {
        mL = @var("globalL", "$^exec_path")

    } else {
        mL = ngx_lua_get_main_lua_vm()
    }

    if (mL == 0) {
        return ""
    }

    g = luajit_G(mL)
    if (g == 0) {
        return ""
    }

    L = luajit_cur_thread(g)
    if (L == 0) {
        return ""
    }

    return luajit_backtrace(L, g, simple)
}


function luajit_backtrace(L, g, simple)
{
    bt = ""

    vmstate = $*g->vmstate
    if (vmstate >= 0 || (vmstate == -3 && $*g->jit_base->ptr64)) {
        /* compiled Lua code */

        T = luajit_get_trace(g, vmstate)
        if (T) {
            if (simple) {
                func = luajit_trace_starting_func(g, T)
                if (func != "") {
                    bt .= sprintf("T:%s\n", func)
                }

            } else {
                //printf("start pc: %d\n", $*T->startpc->ptr64)
            }

            #warn(sprintf("bt: %s", bt))
        }

        base = $*g->jit_base->ptr64
        if (!base) {
            return bt
        }

        bt .= luajit_debug_dumpstack(L, T, $^arg_depth :default(30), base, simple)
        /*
        if (bt != "") {
            warn(sprintf("JIT backtrace: %s\n", bt))
        }
        */

        /*
        if (vmstate == -3) {
            printf("HIT JIT GC: %s===\n", bt)
        }
        */

    } else {

        if (vmstate == -1 && !$*L->cframe) {
            return ""
        }

        if (vmstate == -1 || vmstate == -2 || vmstate == -3) {
            base = $*L->base
            bt .= luajit_debug_dumpstack(L, 0, $^arg_depth, base, simple)
        }
    }

    return bt
}


function luajit_vm_state(g)
{
    return $*g->vmstate
}


function luajit_find_gcstr(g, str)
{
    len = strlen(str)
    strmask = $*g->strmask
    strnum = $*g->strnum
    strhash = $*g->strhash
    $*strhash := @cast(strhash, "GCRef", "$^libluajit_path")
    ret = 0

    n = 0
    done = 0
    for (i = 0; i <= strmask; i++) {
        p = &$*strhash[i]

        while (p) {
            o = luajit_gcref(p)
            if (o == 0) {
                break
            }

            $*o := @cast(o, "GCobj", "$^libluajit_path")
            if ($*o->str->len == len) {
                gcs = &$*o->str
                s = luajit_unbox_gcstr(gcs)
                if (s == str) {
                    done = 1
                    ret = gcs
                    break
                }
            }

            if (++n == strnum) {
                done = 1
                break
            }

            p = &$*o->gch->nextgc
        }

        if (done) {
            break
        }
    }

    return ret
}

function luajit_bucket_depth(p)
{
    n = 0
    while (p) {
        $*o := @cast(o, "GCobj", "$^libluajit_path")
        o = luajit_gcref(p)
        if (o == 0) {
            break
        }

        n++;
        p = &$*o->gch->nextgc
    }
    return n
}