创建bhyve Ubuntu虚拟机子系统见：CBSD创建和管理bhyve容器Ubuntu@FreeBSD_bhyve管理器-CSDN博客
CBSD官网：CBSD --- FreeBSD Jail and Bhyve Management Tools
想快速复制Ubuntu系统，在一台FreeBSD机器有了bhyve Ubuntu子系统后，想把这个子系统复制到另外一台服务器。发现可以使用cbsd bexport和bimport导出和导入。
导出子系统
首先看一下有哪些bhyve虚拟子系统
先用cbsd bls看看有哪些bhyve虚拟子系统
cbsd bls
JNAME JID VM_RAM VM_CURMEM VM_CPUS PCPU VM_OS_TYPE IP4_ADDR STATUS VNC
ub12 1280 2048 0 2 0 linux 192.168.1.12 On 0.0.0.0:5900
导出bhyve虚拟子系统
使用cbsd bexport命令导出
注意，联机时不可以导出。所以如果没有关机，需要首先关闭子系统
关闭子系统
cbsd bstop jname=ub12
Send SIGTERM to ub12. Soft timeout is 30 sec. 1 seconds left [............................Send SIGTERM to ub12. Soft timeout is 30 sec. 0 seconds left [..............................]
bstop done in 10 seconds
导出子系统
cbsd bexport jname=ub12
Exporting (with compress level:6), please stand by: ub12
不过因为原来的系统异乎寻常的慢，所以导出子系统也是特别慢
总共用了13分钟，还可以接受
cbsd bexport jname=ub12
Exporting (with compress level:6), please stand by: ub12
WIP: [0%...100%]
20480+1 records in
20480+1 records out
21474967552 bytes transferred in 816.853627 secs (26289860 bytes/sec)
environment flat size: 20g, images size ub12.img: 6g
exported image file: /usr/jails/export/ub12.img
jexport done in 13 minutes and 42 seconds
将导出的系统cp到新系统
前面导出到/usr/jails/export/ub12.img ，在新系统中scp过来
scp skywalk@192.168.1.5:/usr/jails/export/ub12.img .
导入子系统
在另一台机器使用cbsd bimport导入子系统
cbsd bimport jname=``/tmp/debian1``.img
可以看到只需要3分钟，速度还是挺快的
root@fb14:/usr/jails/export # cbsd bimport ub12.img
Importing image, please stand by: ub12
CBSD Image, version: 4
Image was created on node: fb5
Image was created at: 20240815
Image size: 6g
Environment flat size after extracting: 20g
Compress level: 6
WIP: [0%...100%]
ZVOLs list: dsk1
* dsk1 (20g): WIP: [0%...13%...21%...35%...100%]
Hostname sets to: ub12.my.domain
jimport done in 3 minutes and 41 seconds
启动子系统
cbsd bstart ub12
init_systap: waiting for link: vmx0
VRDP is enabled. VNC bind/port: 192.168.1.250:5900
For attach VM console, use: vncviewer 192.168.1.250:5900
Resolution: 800x600.
VNC pass: cbsd
Warning!!! You are running a system with open VNC port to the world wich is not secure
Please use IP filter or balancer with password to restrict VNC port access
Or change vnc_bind params to 127.0.0.1 and reboot VM after maintenance work
bhyve renice: 1
Waiting for PID..........
PID: 0
Thu Aug 15 21:11:27 CST 2024
cmd: env LIB9P_LOGGING=/usr/jails/jails-system/ub12/cbsd_lib9p.log /usr/bin/nice -n 1 /usr/sbin/bhyve -c cpus=2,sockets=2,cores=1,threads=1 -m 2147483648 -H -A -U 6b143aa7-4afb-11ef-935b-00e2691367f6 -s 0,hostbridge -s 1,ahci-cd,/usr/local/cbsd/upgrade/patch/efirefd.fd,ro -s 7,virtio-blk,/usr/jails/jails-data/ub12-data/dsk1.vhd,sectorsize=512/4096 -s 5,virtio-net,tap2,mtu=1500,mac=00:a0:98:31:84:3b -s 6,virtio-rnd -s 9,fbuf,tcp=0.0.0.0:5900,w=800,h=600,password=cbsd -s 30,xhci,tablet -s 31,lpc -l com1,stdio -l bootrom,/usr/local/cbsd/upgrade/patch/efi.fd,/usr/jails/jails-system/ub12/BHYVE_UEFI_VARS.fd ub12
-----
vm_create: Device not configured
Please use for debug: /usr/local/cbsd/share/bhyverun.sh -c /usr/jails/jails-system/ub12/bhyve.conf
bstart done in 15 seconds
root@fb14:/usr/jails/export # cbsd bls
JNAME JID VM_RAM VM_CURMEM VM_CPUS PCPU VM_OS_TYPE IP4_ADDR STATUS VNC
ub12 0 2048 0 2 0 linux 192.168.1.12 Off 0.0.0.0:5900
没有启动起来，还在查找原因中。
最终找到原因，是因为没有开虚拟化：https://blog.csdn.net/skywalk8163/article/details/141271919
打开虚拟化之后，问题解决，系统能启动起来了！
设置pf防火墙
需要设置防火墙，放开5900端口
先cp配置文件
cp /usr/share/examples/pf/pf.conf /etc/pf.conf
编辑/etc/pf.conf文件，加入：
pass out all keep state
tcp_services = "{ ssh, smtp, domain, www, pop3, auth, pop3s, 8080, 5900 }"
block in all
pass in proto tcp to any port $tcp_services keep state
调整好之后，系统能够用vncviewer连上。
现在的问题是网络不通。
调试
创建bhyve的时候报错vmm没有load
cbsd bconstruct-tui
Warning: Your current environment is not physical. This is a requirement for a working bhyve
Virtual engine detected: vmware
Please refer to this page for more information: https://wiki.freebsd.org/bhyve
Try to run bhyve anyway...
To disable this warning, please set in /usr/jails/etc/bhyve-default-default.conf: skip_bhyve_init_virtual_warning to: 1
Pause for 10 seconds
No kldloaded module: vmm
Please add vmm_load="YES" to /boot/loader.conf and
put kld_list="vmm if_tuntap if_bridge nmdm" into your /etc/rc.conf then reboot the host. Or, for example, run:
sysrc kld_list+="vmm if_tuntap if_bridge nmdm" && service kld restart
root@fb14:~ # kldstat |grep vmm
2 1 0xffffffff81f35000 37c540 vmm.ko
说没有vmm，但是kldstat里面显示有vmm啊！
按照提示，把skip_bhyve_init_virtual_warning=1 写入/usr/jails/etc/bhyve-default-default.conf ，vmm的问题解决。
创建bhyve提示Your current environment is not physical.
cbsd bconstruct-tui
Warning: Your current environment is not physical. This is a requirement for a working bhyve
Virtual engine detected: vmware
Please refer to this page for more information: https://wiki.freebsd.org/bhyve
Try to run bhyve anyway...
To disable this warning, please set in /usr/jails/etc/bhyve-default-default.conf: skip_bhyve_init_virtual_warning to: 1
按照提示，把skip_bhyve_init_virtual_warning=1 写入/usr/jails/etc/bhyve-default-default.conf ，问题解决。
vnc也设置了，pf也打开了，但是5900连不上
发现主机的5900端口根本就没开
原来是虚拟机没有启动成功
cbsd bstart ub12虚拟机启动报错
不管是复制的ub12,还是新create的ub22,都是报错
cmd: env LIB9P_LOGGING=/usr/jails/jails-system/ub22/cbsd_lib9p.log /usr/bin/nice -n 1 /usr/sbin/bhyve -c cpus=6,sockets=6,cores=1,threads=1 -m 4294967296 -H -A -U ceae361c-5b0d-11ef-a876-000c29ff148f -s 0,hostbridge -s 4,virtio-blk,/usr/jails/vm/ub22/dsk1.vhd,sectorsize=512/4096 -s 2,ahci-cd,/usr/jails/src/iso/cbsd-iso-ubuntu-22.04.3-live-server-amd64.iso,ro -s 5,virtio-net,tap2,mtu=1500,mac=00:a0:98:68:0a:de -s 6,virtio-rnd -s 7,fbuf,tcp=127.0.0.1:5901,w=800,h=600,wait,password=cbsd -s 30,xhci,tablet -s 31,lpc -l com1,stdio -l bootrom,/usr/local/cbsd/upgrade/patch/efi.fd,/usr/jails/jails-system/ub22/BHYVE_UEFI_VARS.fd ub22