图像编辑器 Monica 之各种图形绘制、图像调色

一. 图像编辑器 Monica

Monica 是一款跨平台的桌面图像编辑软件(早期主要是个人为了验证一些算法而产生的)。

其技术栈如下：

• Kotlin 编写 UI(Kotlin Compose Desktop 作为 UI 框架)
• 基于 mvvm 模式，依赖注入使用 koin，编译使用 JDK 17。
• 部分算法使用 Kotlin 实现。
• 其余的算法使用 OpenCV C++ 来实现，Kotlin 通过 jni 来调用。
• Monica 所使用的模型，主要使用 ONNXRuntime 进行部署和推理。
• 其余少部分模型使用 OpenCV DNN 进行部署和推理。
• 本地的算法库使用 C++ 17 编译。

Monica 目前还处于开发阶段，当前版本的可以参见 github 地址：github.com/fengzhizi71...

在这个月里，我完成了 Monica 比较重要的两个功能：图形绘制、图像调色。

二. 图形绘制

Monica 支持在图像上的任意位置绘制线段、圆、三角形、矩形、任意多边形，在任意位置添加文字，以及对这些绘制的图形更改属性。

2.1 形状绘制

下面是展示形状绘制的入口。

以及绘制形状的页面。

Monica 提供了图像上的任意位置绘制各种图形的功能，以及修改图形的属性比如图像的颜色、透明度、是否填充、边框类型。

在图像中绘制形状，主要是调用 Compose 的 Canvas API。在实现绘制功能之前，需要先定义好能够绘制图形的类型。

sealed class Shape {
    data class Line(val from: Offset, val to: Offset, val shapeProperties: ShapeProperties): Shape()

    data class Circle(val center: Offset, val radius:Float, val shapeProperties: ShapeProperties): Shape()

    data class Triangle(val first: Offset, val second: Offset?=null, val third: Offset?=null, val shapeProperties: ShapeProperties): Shape()

    data class Rectangle(val tl: Offset, val bl: Offset, val br: Offset, val tr: Offset, val rectFirst: Offset,val shapeProperties: ShapeProperties): Shape()

    data class Polygon(val points: List<Offset>, val shapeProperties: ShapeProperties): Shape()

    data class Text(val point: Offset, val message:String, val shapeProperties: ShapeProperties): Shape()
}

对于不同图形的绘制，需要确定好相关点的坐标。这块的逻辑比较多，可以查看项目的源码。下面主要讲讲如何绘制图形。

class ShapeDrawingViewModel {

    fun drawShape(canvasDrawer:CanvasDrawer,
                  lines: Map<Offset, Line>,
                  circles: Map<Offset, Circle>,
                  triangles: Map<Offset, Triangle>,
                  rectangles: Map<Offset, Rectangle>,
                  polygons: Map<Offset, Polygon>,
                  texts: Map<Offset, Text>,
                  saveFlag: Boolean = false) {

        lines.forEach {

            val line = it.value

            if (line.from != Offset.Unspecified && !saveFlag) {
                canvasDrawer.point(line.from, line.shapeProperties.color)
            }

            if (line.from != Offset.Unspecified && line.to != Offset.Unspecified) {
                canvasDrawer.line(line.from,line.to, Style(null, line.shapeProperties.color, line.shapeProperties.border, null, fill = line.shapeProperties.fill, scale = 1f, alpha = line.shapeProperties.alpha, bounded = true))
            }
        }

        circles.forEach {

            val circle = it.value

            if (circle.center != Offset.Unspecified && !saveFlag) {
                canvasDrawer.point(circle.center, circle.shapeProperties.color)
            }

            canvasDrawer.circle(circle.center, circle.radius, Style(null, circle.shapeProperties.color, circle.shapeProperties.border, null, fill = circle.shapeProperties.fill, scale = 1f, alpha = circle.shapeProperties.alpha, bounded = true))
        }

        triangles.forEach {
            val triangle = it.value

            if (triangle.first != Offset.Unspecified && !saveFlag) {
                canvasDrawer.point(triangle.first, triangle.shapeProperties.color)
            }

            if (triangle.second != Offset.Unspecified && !saveFlag) {
                canvasDrawer.point(triangle.second!!, triangle.shapeProperties.color)
                canvasDrawer.line(triangle.first,triangle.second, Style(null, triangle.shapeProperties.color, triangle.shapeProperties.border, null, fill = triangle.shapeProperties.fill, scale = 1f, alpha = triangle.shapeProperties.alpha, bounded = true))
            }

            if (triangle.first != Offset.Unspecified && triangle.second != Offset.Unspecified && triangle.third != Offset.Unspecified) {
                val list = mutableListOf<Offset>().apply {
                    add(triangle.first)
                    add(triangle.second!!)
                    add(triangle.third!!)
                }

                canvasDrawer.polygon(list, Style(null, triangle.shapeProperties.color, triangle.shapeProperties.border, null, fill = triangle.shapeProperties.fill, scale = 1f, alpha = triangle.shapeProperties.alpha,  bounded = true))
            }
        }

        rectangles.forEach {
            val rect = it.value

            if (rect.rectFirst!=Offset.Unspecified && !saveFlag) {
                canvasDrawer.point(rect.rectFirst, rect.shapeProperties.color)
            }

            if (rect.tl!=Offset.Unspecified && rect.bl!=Offset.Unspecified && rect.br!=Offset.Unspecified && rect.tr!=Offset.Unspecified) {
                val list = mutableListOf<Offset>().apply {
                    add(rect.tl)
                    add(rect.bl)
                    add(rect.br)
                    add(rect.tr)
                }

                canvasDrawer.polygon(list, Style(null, rect.shapeProperties.color, rect.shapeProperties.border, null, fill = rect.shapeProperties.fill, scale = 1f, alpha = rect.shapeProperties.alpha, bounded = true))
            }
        }

        polygons.forEach {
            val polygon = it.value

            if (polygon.points[0]!=null && polygon.points[0] != Offset.Unspecified && !saveFlag) {
                canvasDrawer.point(polygon.points[0] , polygon.shapeProperties.color)
            }

            if (polygon.points.size>1 && polygon.points[1] != Offset.Unspecified && !saveFlag) {
                canvasDrawer.point(polygon.points[1] , polygon.shapeProperties.color)
                canvasDrawer.line(polygon.points[0], polygon.points[1], Style(null, polygon.shapeProperties.color, Border.Line, null, fill = polygon.shapeProperties.fill, scale = 1f, alpha = polygon.shapeProperties.alpha, bounded = true))
            }

            canvasDrawer.polygon(polygon.points, Style(null, polygon.shapeProperties.color, polygon.shapeProperties.border, null, fill = polygon.shapeProperties.fill, scale = 1f, alpha = polygon.shapeProperties.alpha, bounded = true))
        }

        texts.forEach {
            val text = it.value

            if (text.point!= Offset.Unspecified) {
                val list = mutableListOf<String>().apply {
                    add(text.message)
                }
                canvasDrawer.text(text.point, list, text.shapeProperties.color, text.shapeProperties.fontSize)
            }
        }
    }
    ......

}

2.2 添加文字

Monica 支持在图像的任意位置添加文字，修改文字的属性比如字体大小、颜色。

在图像中添加文字，需要一个可以拖动的 TextField ，这样才能在图像的任意位置添加文字。因此，DraggableTextField 控件如下：

@OptIn(ExperimentalFoundationApi::class)
@Composable
fun DraggableTextField(
    modifier: Modifier = Modifier,
    text: String,
    bitmapWidth: Int,
    bitmapHeight: Int,
    density: Density,
    onTextChanged: (String) -> Unit,
    onDragged: (Offset) -> Unit
) {
    var offset by remember { mutableStateOf(Offset.Zero) }
    val halfWidth = bitmapWidth/2
    val halfHeight = bitmapHeight/2
    val halfTextFieldWidth = 125/density.density
    val halfTextFieldHeight = 65/density.density

    Box(
        modifier = modifier
            .offset { IntOffset(offset.x.roundToInt(), offset.y.roundToInt()) }
            .pointerInput(Unit) {
                detectDragGestures { change ->
                    offset += change
                    if (abs(offset.x) > halfWidth - halfTextFieldWidth || abs(offset.y) > halfHeight - halfTextFieldHeight) {
                        offset -= change
                        return@detectDragGestures
                    }
                }
            }
            .shadow(8.dp)
            .background(Color.White)
            .padding(16.dp)
            .fillMaxWidth()
            .wrapContentHeight(Alignment.Top)
            .clip(RoundedCornerShape(8.dp))
    ) {
        Column {
            TextField (
                value = text,
                onValueChange = onTextChanged,
                modifier = Modifier.width(220.dp)
            )

            confirmButton(true, modifier = Modifier.align(Alignment.End).padding(top = 5.dp)) {
                onDragged.invoke(offset)
            }
        }
    }
}

需要注意的是 DraggableTextField 中的 offset 通过 onDragged 回调给当前图像，但是 offset 要变成图像中的当前的坐标，还需要做一些坐标转换才行。

三. 图像调色

Monica 支持调节图像的对比度、色调、饱和度、亮度、色温等，从而帮助大家调整图像的色彩。

3.1 应用层的设计和调用

该模块功能的实现，最终也是调用了封装 OpenCV 的函数。对于应用层，需要编写好调用 jni 层的代码：

object ImageProcess {

    ......

    /**
     * 初始化图像调色模块
     */
    external fun initColorCorrection(src: ByteArray): Long

    /**
     * 图像调色
     */
    external fun colorCorrection(src: ByteArray, colorCorrectionSettings: ColorCorrectionSettings, cppObjectPtr:Long):IntArray

    /**
     * 删除 ColorCorrection
     */
    external fun deleteColorCorrection(cppObjectPtr:Long): Long

    ......
}

其中，initColorCorrection() 返回的是 Long 类型，其实是一个指针的地址，表示的是一个 C++ 对象。之所以这么做是为了在处理当前图片时，能够复用该 C++ 对象。

viewModel 的 colorCorrection 会调用 ImageProcess 的 colorCorrection() 函数，然后将结果展示到 UI 上。离着这个界面的时候，会清除所有的状态，以及回收所用到的 C++ 对象。

class ColorCorrectionViewModel {

    private val logger: Logger = logger<ColorCorrectionViewModel>()

    var contrast by mutableStateOf(255f )
    var hue by mutableStateOf(180f )
    var saturation by mutableStateOf(255f )
    var lightness by mutableStateOf(255f )
    var temperature by mutableStateOf(255f )
    var highlight by mutableStateOf(255f )
    var shadow by mutableStateOf(255f )
    var sharpen by mutableStateOf(0f )
    var corner by mutableStateOf(0f )

    private var cppObjectPtr:Long = 0

    private var init:AtomicBoolean = AtomicBoolean(false)

    fun colorCorrection(state: ApplicationState, image: BufferedImage, colorCorrectionSettings: ColorCorrectionSettings,
                        success: CVSuccess) {

        logger.info("colorCorrectionSettings = ${GsonUtils.toJson(colorCorrectionSettings)}")

        state.scope.launchWithLoading {
            if (!init.get()) {
                init.set(true)

                val byteArray = image.image2ByteArray()
                cppObjectPtr = ImageProcess.initColorCorrection(byteArray)
            }

            OpenCVManager.invokeCV(image,
                action  = { byteArray -> ImageProcess.colorCorrection(byteArray, colorCorrectionSettings, cppObjectPtr) },
                success = { success.invoke(it) },
                failure = { e ->
                    logger.error("colorCorrection is failed", e)
                })
        }
    }

    ......

    fun clearAllStatus() {
        init.set(false)

        contrast = 255f
        hue = 180f
        saturation = 255f
        lightness = 255f
        temperature = 255f
        highlight = 255f
        shadow = 255f
        sharpen = 0f
        corner = 0f

        colorCorrectionSettings = ColorCorrectionSettings()

        ImageProcess.deleteColorCorrection(cppObjectPtr)
        cppObjectPtr = 0
    }
}

3.2 jni 层的编写

对于 jni 层，需要先在头文件里定义好应用层对应的函数

JNIEXPORT jlong JNICALL Java_cn_netdiscovery_monica_opencv_ImageProcess_initColorCorrection
(JNIEnv* env, jobject,jbyteArray array);

JNIEXPORT jintArray JNICALL Java_cn_netdiscovery_monica_opencv_ImageProcess_colorCorrection
        (JNIEnv* env, jobject,jbyteArray array, jobject jobj,  jlong cppObjectPtr);

JNIEXPORT void JNICALL Java_cn_netdiscovery_monica_opencv_ImageProcess_deleteColorCorrection
        (JNIEnv* env, jobject, jlong cppObjectPtr);

然后，编写对应的实现。

JNIEXPORT jlong JNICALL Java_cn_netdiscovery_monica_opencv_ImageProcess_initColorCorrection
(JNIEnv* env, jobject, jbyteArray array) {
    Mat image = byteArrayToMat(env, array);

    // 创建 C++ 对象并存储指针
    ColorCorrection* cppObject = new ColorCorrection(image);
    return reinterpret_cast<jlong>(cppObject);
}

JNIEXPORT jintArray JNICALL Java_cn_netdiscovery_monica_opencv_ImageProcess_colorCorrection
        (JNIEnv* env, jobject,jbyteArray array, jobject jobj,  jlong cppObjectPtr) {

    ColorCorrection* colorCorrection = reinterpret_cast<ColorCorrection*>(cppObjectPtr);
    ColorCorrectionSettings colorCorrectionSettings;

    Mat image = byteArrayToMat(env, array);

    // 获取 jclass 实例
    jclass jcls = env->FindClass("cn/netdiscovery/monica/ui/controlpanel/colorcorrection/model/ColorCorrectionSettings");
    jfieldID contrastId = env->GetFieldID(jcls, "contrast", "I");
    jfieldID hueId = env->GetFieldID(jcls, "hue", "I");
    jfieldID saturationId = env->GetFieldID(jcls, "saturation", "I");
    jfieldID lightnessId = env->GetFieldID(jcls, "lightness", "I");
    jfieldID temperatureId = env->GetFieldID(jcls, "temperature", "I");
    jfieldID highlightId = env->GetFieldID(jcls, "highlight", "I");
    jfieldID shadowId = env->GetFieldID(jcls, "shadow", "I");
    jfieldID sharpenId = env->GetFieldID(jcls, "sharpen", "I");
    jfieldID cornerId = env->GetFieldID(jcls, "corner", "I");
    jfieldID statusId = env->GetFieldID(jcls, "status", "I");

    colorCorrectionSettings.contrast = env->GetIntField(jobj, contrastId);
    colorCorrectionSettings.hue = env->GetIntField(jobj, hueId);
    colorCorrectionSettings.saturation = env->GetIntField(jobj, saturationId);
    colorCorrectionSettings.lightness = env->GetIntField(jobj, lightnessId);
    colorCorrectionSettings.temperature = env->GetIntField(jobj, temperatureId);
    colorCorrectionSettings.highlight = env->GetIntField(jobj, highlightId);
    colorCorrectionSettings.shadow = env->GetIntField(jobj, shadowId);
    colorCorrectionSettings.sharpen = env->GetIntField(jobj, sharpenId);
    colorCorrectionSettings.corner = env->GetIntField(jobj, cornerId);
    colorCorrectionSettings.status = env->GetIntField(jobj, statusId);

    Mat dst;

    try {
        colorCorrection->doColorCorrection(colorCorrectionSettings, dst);
    } catch(...) {
    }

    jthrowable mException = NULL;
    mException = env->ExceptionOccurred();

    if (mException != NULL) {
      env->ExceptionClear();
      jclass exceptionClazz = env->FindClass("java/lang/Exception");
      env->ThrowNew(exceptionClazz, "jni exception");
      env->DeleteLocalRef(exceptionClazz);

      return env->NewIntArray(0);
    }

    env->DeleteLocalRef(jcls);  // 手动释放局部引用

    return matToIntArray(env, dst);
}

JNIEXPORT void JNICALL Java_cn_netdiscovery_monica_opencv_ImageProcess_deleteColorCorrection
        (JNIEnv* env, jobject, jlong cppObjectPtr) {
    // 删除 C++对象，防止内存泄漏
    ColorCorrection* colorCorrection = reinterpret_cast<ColorCorrection*>(cppObjectPtr);
    delete colorCorrection;
}

jni 层还需要调用 C++ 对应的 ColorCorrection 类，这块因为篇幅原因暂时省略，感兴趣的话可以直接看项目的源码。

同意还有一个值得注意的是，从应用层传递的 ColorCorrectionSettings 对象，需要通过 jobject 转换到 jclass 然后再获取对应的各个属性。在 jni 层也需要定义好对应的 ColorCorrectionSettings 对象。

typedef struct {
    int contrast;
    int hue;
    int saturation;
    int lightness;
    int temperature;
    int highlight;
    int shadow;
    int sharpen;
    int corner;
    int status;
} ColorCorrectionSettings;

四. 总结

Monica 支持了图形绘制、图像调色之后，它才算是一款比较基础的图像编辑器。后面还有很长的路要走，Monica 现有的每一个功能都需要打磨一下。

到农历过年前，我对 Monica 的规划是争取完善 CV 算法快速调参的模块和将部分模型部署到云端。如果能完成这些的话，明年可以做更多有意思的功能。

最后，Monica github 地址：github.com/fengzhizi71...