把Sherpa-onnx部署到安卓端的3568

1.下载Sherpa-onnx源码

git clone https://github.com/k2-fsa/sherpa-onnx

2.准备编译

//先设置ndk地址,你们根据你们实际的ndl地址配置即可

export ANDROID_NDK=../../../android-ndk-r26d

//设置编译rknn

export SHERPA_ONNX_ENABLE_RKNN=ON

//执行sherpa-onnx自带的脚步

./build-android-arm64-v8a.sh

等待,脚本会自动安装所有的内容,然后编译出对应的执行库,这里rknn的库也会编译出来

3.把代码导入android studio

从第一步的源码找到sherpa-onnx\android\SherpaOnnx项目,直接导入对应的android studio,这里我是把第三步的库放进来了,这里我是arm64-v8a,我在assets里面放入了我的rknn模型

4.下载双语模型,官网是已经转化好了的

我这里是使用命令下载的,你们可以直接去官网找到你们要的内容

//下载

wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-rk3568-streaming-zipformer-bilingual-zh-en-2023-02-20.tar.bz2

//解压

tar -jxvf sherpa-onnx-rk3568-streaming-zipformer-bilingual-zh-en-2023-02-20.tar.bz2

这是模型官网地址:发布ASR-models·K2-FSA/夏尔巴-ONNX ·GitHub

5.修改代码,源码自带是没有rknn的,只有一个qnn

我们在activity里面新增代码,我直接把这一段全贴出来

  var assetManager: AssetManager? = application.assets
        if (config.modelConfig.provider == "qnn") {
            Log.i(TAG, "nativelibdir: ${applicationInfo.nativeLibraryDir}")
            OnlineRecognizer.prependAdspLibraryPath(applicationInfo.nativeLibraryDir)

            val transducer = config.modelConfig.transducer
            val qnnConfig = transducer.qnnConfig

            if (qnnConfig.backendLib.isEmpty()) {
                throw IllegalArgumentException("You should provide libQnnHtp.so for qnn")
            }

            config.modelConfig.tokens =
                copyAssetToInternalStorage(config.modelConfig.tokens, this)

            if (transducer.encoder.isNotEmpty()) {
                transducer.encoder =
                    copyAssetToInternalStorage(transducer.encoder, this)
            }

            if (transducer.decoder.isNotEmpty()) {
                transducer.decoder =
                    copyAssetToInternalStorage(transducer.decoder, this)
            }

            if (transducer.joiner.isNotEmpty()) {
                transducer.joiner =
                    copyAssetToInternalStorage(transducer.joiner, this)
            }

            if (qnnConfig.contextBinary.isNotEmpty()) {
                qnnConfig.contextBinary =
                    copyAssetListToInternalStorage(qnnConfig.contextBinary, this)
            }

            if (config.hr.lexicon.isNotEmpty()) {
                config.hr.lexicon = copyAssetToInternalStorage(config.hr.lexicon, this)
            }

            if (config.hr.ruleFsts.isNotEmpty()) {
                config.hr.ruleFsts = copyAssetToInternalStorage(config.hr.ruleFsts, this)
            }

            assetManager = null
        }else if (config.modelConfig.provider == "rknn") {
            // ===================== RKNN 新增分支 =====================
            Log.i(TAG, "Enter RKNN provider file copy logic")
            val transducer = config.modelConfig.transducer

            // 1. 拷贝tokens词表文件
            config.modelConfig.tokens =
                copyAssetToInternalStorage(config.modelConfig.tokens, this)

            // 2. 拷贝三个rknn模型文件 encoder/decoder/joiner
            if (transducer.encoder.isNotEmpty()) {
                transducer.encoder =
                    copyAssetToInternalStorage(transducer.encoder, this)
                Log.i(TAG, "Copy encoder.rknn success: ${transducer.encoder}")
            }
            if (transducer.decoder.isNotEmpty()) {
                transducer.decoder =
                    copyAssetToInternalStorage(transducer.decoder, this)
                Log.i(TAG, "Copy decoder.rknn success: ${transducer.decoder}")
            }
            if (transducer.joiner.isNotEmpty()) {
                transducer.joiner =
                    copyAssetToInternalStorage(transducer.joiner, this)
                Log.i(TAG, "Copy joiner.rknn success: ${transducer.joiner}")
            }

            // 3. 拷贝同音替换词典、文法文件（和QNN逻辑一致）
            if (config.hr.lexicon.isNotEmpty()) {
                config.hr.lexicon = copyAssetToInternalStorage(config.hr.lexicon, this)
            }
            if (config.hr.ruleFsts.isNotEmpty()) {
                config.hr.ruleFsts = copyAssetToInternalStorage(config.hr.ruleFsts, this)
            }
            // RKNN不需要assetManager，置空和QNN保持统一
            assetManager = null
        }

在OnlineRecognizer.kt里面也把我们下载的模型配置进去,上面有示例,基本只要改下文件夹名字就行

这里我们新增了一个1002,我们再把mainactivity的type类型改一下,让代码执行1002的配置,源码默认是0

我们运行一遍

使用,基本都能识别出来,我手上的板子npu算力不够,跑一段时间就挂了,但是能用.

基本上一运行,npu就拉满了,你们如果要用,最好是1T以上的算力,会好点.