文章目录
-
- 每日一句正能量
- 前言
- 一、体感音游:AR技术重新定义音乐游戏
-
- [1.1 传统音游的交互瓶颈](#1.1 传统音游的交互瓶颈)
- [1.2 "律动星途"游戏架构](#1.2 "律动星途"游戏架构)
- 二、环境配置与游戏引擎初始化
-
- [2.1 模块依赖配置](#2.1 模块依赖配置)
- [2.2 游戏窗口配置(RhythmGameAbility.ets)](#2.2 游戏窗口配置(RhythmGameAbility.ets))
- 三、核心游戏组件实战
-
- [3.1 音符轨道系统与AR输入映射](#3.1 音符轨道系统与AR输入映射)
- [3.2 悬浮节奏HUD与沉浸光效反馈](#3.2 悬浮节奏HUD与沉浸光效反馈)
- [3.3 主游戏页面:全屏沉浸与光效整合](#3.3 主游戏页面:全屏沉浸与光效整合)
- 四、关键技术总结
-
- [4.1 AR输入-游戏映射体系](#4.1 AR输入-游戏映射体系)
- [4.2 判定系统与光效反馈](#4.2 判定系统与光效反馈)
- [4.3 性能优化策略](#4.3 性能优化策略)
- 五、调试与设备适配
-
- [5.1 调试要点](#5.1 调试要点)
- [5.2 设备适配](#5.2 设备适配)
- 六、总结与展望

每日一句正能量
人生,从外打破是压力,从内打破是成长。
真正的成长,永远源于内心的觉醒和自我驱动。
前言
摘要:HarmonyOS 6(API 23)的 Face AR 与 Body AR 能力为游戏开发开辟了全新的交互维度。本文将实战开发一款"律动星途"体感音游,玩家通过面部表情(挑眉、张嘴、眨眼)触发音符特效,通过肢体动作(双手挥舞、身体摇摆)击打节奏音符,结合悬浮导航与沉浸光感,打造"用脸玩游戏、用身体打节奏"的次世代音游体验。
一、体感音游:AR技术重新定义音乐游戏
1.1 传统音游的交互瓶颈
传统音游依赖触屏点击或按键操作,存在明显的体验天花板:
| 维度 | 传统音游 | 体感音游(AR驱动) |
|---|---|---|
| 输入方式 | 触屏/按键 | 面部表情 + 肢体动作 |
| 沉浸感 | 手指局部操作 | 全身律动参与 |
| 表现力 | 固定动画反馈 | 实时表情驱动特效 |
| 社交性 | 分数排行榜 | 表情PK、动作同步 |
| 疲劳度 | 手指酸痛 | 全身运动、健康娱乐 |
HarmonyOS 6(API 23)的 AR Engine 6.1.0 提供了 Face AR (64种BlendShape表情参数)和 Body AR(20+骨骼关键点)两大核心能力 ,为体感音游提供了精准的"输入设备"------用户的脸和身体。
1.2 "律动星途"游戏架构
┌─────────────────────────────────────────────────────────────┐
│ 游戏渲染层 (Canvas 2D/WebGL) │
│ ┌─────────────────────┐ ┌─────────────────────────────┐ │
│ │ 音符轨道系统 │ │ 粒子特效系统 │ │
│ │ · 4轨面部音符 │ │ · 表情触发光爆 │ │
│ │ · 2轨肢体音符 │ │ · 连击火焰尾迹 │ │
│ │ · 判定线动态调整 │ │ · 完美判定全屏闪光 │ │
│ └─────────────────────┘ └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
┌─────────────────────────────────────────────────────────────┐
│ AR输入层(Face AR + Body AR) │
│ ┌─────────────────────┐ ┌─────────────────────────────┐ │
│ │ Face AR 表情输入 │ │ Body AR 动作输入 │ │
│ │ · 挑眉→上轨道音符 │ │ · 左手抬起→左轨道击打 │ │
│ │ · 张嘴→下轨道音符 │ │ · 右手抬起→右轨道击打 │ │
│ │ · 眨眼→特殊技能 │ │ · 双手张开→全屏清场 │ │
│ │ · 微笑→连击加成 │ │ · 身体摇摆→滑条操作 │ │
│ └─────────────────────┘ └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
┌─────────────────────────────────────────────────────────────┐
│ UI层(悬浮导航 + 沉浸光感) │
│ ┌─────────────────────┐ ┌─────────────────────────────┐ │
│ │ 悬浮节奏HUD │ │ 情绪光效反馈 │ │
│ │ · 动态难度指示 │ │ · 连击数驱动光强 │ │
│ │ · 表情状态面板 │ │ · 判定结果色同步 │ │
│ │ · 手势引导提示 │ │ · BPM脉冲环境光 │ │
│ └─────────────────────┘ └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
二、环境配置与游戏引擎初始化
2.1 模块依赖配置
json
// oh-package.json5
{
"dependencies": {
"@hms.core.ar.arengine": "^6.1.0",
"@hms.core.ar.arview": "^6.1.0",
"@kit.ArkUI": "^6.1.0",
"@kit.AbilityKit": "^6.1.0",
"@kit.SensorServiceKit": "^6.1.0",
"@kit.AudioKit": "^6.1.0"
}
}
2.2 游戏窗口配置(RhythmGameAbility.ets)
代码亮点:音游需要极致的帧率稳定性(120Hz),同时AR追踪需要高帧率(60fps)确保输入精准。配置游戏窗口为高刷模式,隐藏系统UI,启用全屏沉浸。
typescript
// entry/src/main/ets/ability/RhythmGameAbility.ets
import { AbilityConstant, UIAbility, Want } from '@kit.AbilityKit';
import { window } from '@kit.ArkUI';
import { arEngine, ARConfig, ARFeatureType, ARMultiFaceMode } from '@hms.core.ar.arengine';
/**
* 游戏配置
*/
interface GameConfig {
bpm: number; // 节拍速度
difficulty: 'easy' | 'normal' | 'hard' | 'expert';
trackCount: number; // 轨道数
noteSpeed: number; // 音符下落速度
}
export default class RhythmGameAbility extends UIAbility {
private arSession: arEngine.ARSession | null = null;
private gameConfig: GameConfig = {
bpm: 128,
difficulty: 'normal',
trackCount: 6, // 4面部轨 + 2肢体轨
noteSpeed: 800 // 像素/秒
};
onWindowStageCreate(windowStage: window.WindowStage): void {
this.setupGameWindow(windowStage);
}
/**
* 配置音游专用窗口
* 关键设置:
* 1. 120Hz高刷:确保音符下落丝滑
* 2. 全屏沉浸:无系统栏干扰
* 3. 低延迟音频:AudioKit配置
* 4. AR双模态:Face+Body并发追踪
*/
private async setupGameWindow(windowStage: window.WindowStage): Promise<void> {
try {
const mainWindow = windowStage.getMainWindowSync();
// 1. 全屏沉浸
await mainWindow.setWindowLayoutFullScreen(true);
await mainWindow.setWindowSystemBarEnable(false);
await mainWindow.setWindowBackgroundColor('#00000000');
// 2. 120Hz高刷(音游核心)
try {
await mainWindow.setPreferredFrameRate(120);
console.info('Game running at 120Hz');
} catch (e) {
await mainWindow.setPreferredFrameRate(60);
console.info('Fallback to 60Hz');
}
// 3. 防误触
await mainWindow.setWindowGestureDisabled(true);
// 4. 窗口阴影与圆角(PC端)
await mainWindow.setWindowShadowEnabled(true);
await mainWindow.setWindowCornerRadius(8);
AppStorage.setOrCreate('game_config', this.gameConfig);
AppStorage.setOrCreate('game_window', mainWindow);
// 加载游戏内容
windowStage.loadContent('pages/RhythmGamePage', (err) => {
if (err.code) {
console.error('Failed to load game:', JSON.stringify(err));
return;
}
this.initializeAREngine();
});
} catch (error) {
console.error('Game window setup failed:', error);
}
}
/**
* 初始化AR引擎:音游专用配置
* 关键优化:
* - 降低相机分辨率以提升帧率(音游不需要超高精度)
* - 启用表情参数快速通道
* - 骨骼追踪降频至30fps(肢体动作不需要60fps)
*/
private async initializeAREngine(): Promise<void> {
try {
const context = getContext(this);
const isReady = await arEngine.isAREngineReady(context);
if (!isReady) {
console.error('AR Engine not available');
return;
}
this.arSession = new arEngine.ARSession(context);
const config = new ARConfig();
config.featureType = ARFeatureType.ARENGINE_FEATURE_TYPE_FACE |
ARFeatureType.ARENGINE_FEATURE_TYPE_BODY;
config.multiFaceMode = ARMultiFaceMode.MULTIFACE_ENABLE;
config.maxDetectedBodyNum = 1;
config.cameraLensFacing = arEngine.ARCameraLensFacing.FRONT;
// 音游优化:降低分辨率换取更高帧率
config.imageResolution = { width: 1280, height: 720 };
this.arSession.configure(config);
await this.arSession.start();
AppStorage.setOrCreate('ar_session', this.arSession);
console.info('AR Engine ready for rhythm game');
} catch (error) {
console.error('AR Engine init failed:', error);
}
}
onWindowStageDestroy(): void {
if (this.arSession) {
this.arSession.stop();
this.arSession = null;
}
}
}
三、核心游戏组件实战
3.1 音符轨道系统与AR输入映射
代码亮点:6轨制设计------上/下/左/右4轨由Face AR表情控制(挑眉/张嘴/左眨眼/右眨眼),左/右2轨由Body AR手势控制(左手/右手抬起)。音符下落速度与BPM同步,判定精度根据表情强度动态调整。
typescript
// components/RhythmTrackSystem.ets
import { Canvas, CanvasRenderingContext2D } from '@kit.ArkUI';
import { arEngine, ARFaceAnchor, ARBody, ARBodyLandmarkType } from '@hms.core.ar.arengine';
/**
* 音符类型
*/
export enum NoteType {
FACE_UP = 'face_up', // 挑眉触发
FACE_DOWN = 'face_down', // 张嘴触发
FACE_LEFT = 'face_left', // 左眨眼触发
FACE_RIGHT = 'face_right', // 右眨眼触发
BODY_LEFT = 'body_left', // 左手抬起
BODY_RIGHT = 'body_right', // 右手抬起
SPECIAL = 'special' // 组合技
}
/**
* 音符对象
*/
interface Note {
id: string;
type: NoteType;
trackIndex: number;
spawnTime: number; // 生成时间戳
hitTime: number; // 应击打时间戳
y: number; // 当前Y坐标
intensity: number; // 表情强度要求(0-1)
hit: boolean; // 是否已击打
missed: boolean; // 是否错过
effect: string; // 特效类型
}
/**
* 判定结果
*/
export enum Judgment {
PERFECT = 'perfect', // ±50ms
GREAT = 'great', // ±100ms
GOOD = 'good', // ±150ms
MISS = 'miss' // >150ms
}
@Component
export struct RhythmTrackSystem {
private canvasRef: CanvasRenderingContext2D | null = null;
private animationId: number = 0;
private audioContext: AudioContext | null = null;
@State notes: Note[] = [];
@State score: number = 0;
@State combo: number = 0;
@State maxCombo: number = 0;
@State lastJudgment: Judgment | null = null;
@State judgmentTime: number = 0;
@State bpm: number = 128;
@State gameTime: number = 0;
@State startTime: number = 0;
@State isPlaying: boolean = false;
// AR输入状态
@State faceExpression: Map<string, number> = new Map();
@State leftHandUp: boolean = false;
@State rightHandUp: boolean = false;
@State bodyDetected: boolean = false;
@State faceDetected: boolean = false;
// 轨道配置(6轨)
private tracks = [
{ index: 0, type: NoteType.FACE_UP, label: '挑眉', color: '#FF6B6B', xRatio: 0.2 },
{ index: 1, type: NoteType.FACE_LEFT, label: '左眨眼', color: '#4ECDC4', xRatio: 0.35 },
{ index: 2, type: NoteType.BODY_LEFT, label: '左手', color: '#45B7D1', xRatio: 0.5 },
{ index: 3, type: NoteType.BODY_RIGHT, label: '右手', color: '#96CEB4', xRatio: 0.65 },
{ index: 4, type: NoteType.FACE_RIGHT, label: '右眨眼', color: '#FFEAA7', xRatio: 0.8 },
{ index: 5, type: NoteType.FACE_DOWN, label: '张嘴', color: '#DDA0DD', xRatio: 0.95 }
];
private noteSpeed: number = 800; // 像素/秒
private judgmentLineY: number = 0.85; // 判定线位置(屏幕比例)
private spawnLeadTime: number = 2000; // 音符提前生成时间(ms)
aboutToAppear(): void {
this.setupARListening();
this.startGameLoop();
this.generateNotePattern();
}
aboutToDisappear(): void {
if (this.animationId) cancelAnimationFrame(this.animationId);
}
/**
* 监听AR输入数据
*/
private setupARListening(): void {
AppStorage.watch('face_expression', (expr: Map<string, number>) => {
this.faceExpression = expr;
this.checkFaceInput(expr);
});
AppStorage.watch('body_gesture', (gesture: { type: string; hands: { left: boolean; right: boolean } }) => {
if (gesture) {
this.leftHandUp = gesture.hands?.left || false;
this.rightHandUp = gesture.hands?.right || false;
this.checkBodyInput();
}
});
AppStorage.watch('face_detected', (v: boolean) => this.faceDetected = v);
AppStorage.watch('body_detected', (v: boolean) => this.bodyDetected = v);
}
/**
* 检查面部输入并触发对应轨道判定
*/
private checkFaceInput(expressions: Map<string, number>): void {
if (!this.isPlaying) return;
const now = Date.now() - this.startTime;
// 挑眉检测 → 上轨道
const browUp = expressions.get('EYE_BROW_UP_LEFT') || 0;
if (browUp > 0.6) {
this.tryHitNote(NoteType.FACE_UP, browUp);
}
// 张嘴检测 → 下轨道
const jawOpen = expressions.get('JAW_OPEN') || 0;
if (jawOpen > 0.5) {
this.tryHitNote(NoteType.FACE_DOWN, jawOpen);
}
// 左眨眼检测 → 左轨道
const leftBlink = expressions.get('EYE_BLINK_LEFT') || 0;
if (leftBlink > 0.8) {
this.tryHitNote(NoteType.FACE_LEFT, leftBlink);
}
// 右眨眼检测 → 右轨道
const rightBlink = expressions.get('EYE_BLINK_RIGHT') || 0;
if (rightBlink > 0.8) {
this.tryHitNote(NoteType.FACE_RIGHT, rightBlink);
}
// 微笑检测 → 连击加成
const smile = expressions.get('MOUTH_SMILE_LEFT') || 0;
if (smile > 0.7 && this.combo > 10) {
this.triggerComboBonus();
}
}
/**
* 检查肢体输入
*/
private checkBodyInput(): void {
if (!this.isPlaying) return;
// 左手抬起 → 左肢体轨道
if (this.leftHandUp) {
this.tryHitNote(NoteType.BODY_LEFT, 1.0);
}
// 右手抬起 → 右肢体轨道
if (this.rightHandUp) {
this.tryHitNote(NoteType.BODY_RIGHT, 1.0);
}
// 双手同时抬起 → 特殊技能
if (this.leftHandUp && this.rightHandUp) {
this.triggerSpecialSkill();
}
}
/**
* 尝试击打音符
*/
private tryHitNote(type: NoteType, intensity: number): void {
const now = Date.now() - this.startTime;
// 查找对应轨道上最接近判定线的未击打音符
const targetNote = this.notes.find(n =>
n.type === type &&
!n.hit &&
!n.missed &&
Math.abs(n.hitTime - now) < 200 // 200ms判定窗口
);
if (!targetNote) return;
const delta = Math.abs(targetNote.hitTime - now);
let judgment: Judgment;
let points: number;
if (delta <= 50) {
judgment = Judgment.PERFECT;
points = 300;
} else if (delta <= 100) {
judgment = Judgment.GREAT;
points = 200;
} else if (delta <= 150) {
judgment = Judgment.GOOD;
points = 100;
} else {
judgment = Judgment.MISS;
points = 0;
}
// 应用判定
targetNote.hit = true;
this.lastJudgment = judgment;
this.judgmentTime = Date.now();
if (judgment !== Judgment.MISS) {
this.combo++;
this.maxCombo = Math.max(this.maxCombo, this.combo);
this.score += points * (1 + this.combo * 0.01); // 连击加成
// 触发特效
this.triggerHitEffect(targetNote, judgment, intensity);
} else {
this.combo = 0;
}
// 同步光效
AppStorage.setOrCreate('last_judgment', { judgment, combo: this.combo, intensity });
}
/**
* 生成音符谱面(简化版,实际应从音乐文件解析)
*/
private generateNotePattern(): void {
const pattern = [
{ time: 2000, type: NoteType.FACE_UP, intensity: 0.6 },
{ time: 2500, type: NoteType.BODY_LEFT, intensity: 0.8 },
{ time: 3000, type: NoteType.FACE_DOWN, intensity: 0.5 },
{ time: 3500, type: NoteType.BODY_RIGHT, intensity: 0.8 },
{ time: 4000, type: NoteType.FACE_LEFT, intensity: 0.7 },
{ time: 4500, type: NoteType.FACE_RIGHT, intensity: 0.7 },
{ time: 5000, type: NoteType.FACE_UP, intensity: 0.8 },
{ time: 5250, type: NoteType.FACE_DOWN, intensity: 0.8 },
{ time: 5500, type: NoteType.BODY_LEFT, intensity: 0.9 },
{ time: 5750, type: NoteType.BODY_RIGHT, intensity: 0.9 },
// ... 更多音符
];
pattern.forEach((p, i) => {
this.notes.push({
id: `note_${i}`,
type: p.type,
trackIndex: this.tracks.findIndex(t => t.type === p.type),
spawnTime: p.time - this.spawnLeadTime,
hitTime: p.time,
y: -100, // 屏幕上方外
intensity: p.intensity,
hit: false,
missed: false,
effect: 'normal'
});
});
}
/**
* 游戏主循环
*/
private startGameLoop(): void {
this.startTime = Date.now();
this.isPlaying = true;
const loop = () => {
const now = Date.now() - this.startTime;
this.gameTime = now;
this.updateNotes(now);
this.renderFrame();
this.checkMissedNotes(now);
this.animationId = requestAnimationFrame(loop);
};
requestAnimationFrame(loop);
}
/**
* 更新音符位置
*/
private updateNotes(now: number): void {
const canvasHeight = this.canvasRef?.canvas.height || 1920;
const judgmentY = canvasHeight * this.judgmentLineY;
this.notes.forEach(note => {
if (note.hit || note.missed) return;
// 计算音符当前位置
const timeToHit = note.hitTime - now;
const distanceToJudgment = timeToHit / 1000 * this.noteSpeed;
note.y = judgmentY - distanceToJudgment;
// 激活生成
if (now >= note.spawnTime && note.y < -50) {
note.y = -50; // 开始显示
}
});
}
/**
* 检查错过的音符
*/
private checkMissedNotes(now: number): void {
this.notes.forEach(note => {
if (!note.hit && !note.missed && now > note.hitTime + 150) {
note.missed = true;
this.combo = 0;
this.lastJudgment = Judgment.MISS;
this.judgmentTime = Date.now();
}
});
}
/**
* 渲染游戏画面
*/
private renderFrame(): void {
if (!this.canvasRef) return;
const ctx = this.canvasRef;
const w = ctx.canvas.width;
const h = ctx.canvas.height;
// 清空画布
ctx.fillStyle = '#0a0a1a';
ctx.fillRect(0, 0, w, h);
// 绘制背景光效(BPM脉冲)
this.drawBPMBackground(ctx, w, h);
// 绘制轨道线
this.drawTracks(ctx, w, h);
// 绘制判定线
this.drawJudgmentLine(ctx, w, h);
// 绘制音符
this.notes.forEach(note => this.drawNote(ctx, note, w, h));
// 绘制判定反馈
this.drawJudgmentFeedback(ctx, w, h);
// 绘制HUD
this.drawHUD(ctx, w, h);
// 绘制AR状态指示
this.drawARStatus(ctx, w, h);
}
/**
* BPM驱动的背景光效
*/
private drawBPMBackground(ctx: CanvasRenderingContext2D, w: number, h: number): void {
const beatDuration = 60000 / this.bpm; // ms per beat
const beatPhase = (this.gameTime % beatDuration) / beatDuration;
const pulseIntensity = Math.sin(beatPhase * Math.PI * 2) * 0.5 + 0.5;
// 主光晕
const gradient = ctx.createRadialGradient(w/2, h/2, 0, w/2, h/2, Math.max(w, h));
const baseColor = this.getComboColor();
gradient.addColorStop(0, baseColor + Math.floor(pulseIntensity * 30).toString(16).padStart(2, '0'));
gradient.addColorStop(0.5, baseColor + '08');
gradient.addColorStop(1, 'transparent');
ctx.fillStyle = gradient;
ctx.fillRect(0, 0, w, h);
}
/**
* 绘制轨道
*/
private drawTracks(ctx: CanvasRenderingContext2D, w: number, h: number): void {
const judgmentY = h * this.judgmentLineY;
this.tracks.forEach(track => {
const x = w * track.xRatio;
// 轨道线
ctx.strokeStyle = track.color + '30';
ctx.lineWidth = 2;
ctx.beginPath();
ctx.moveTo(x, 0);
ctx.lineTo(x, h);
ctx.stroke();
// 轨道底部标识
ctx.fillStyle = track.color + '60';
ctx.beginPath();
ctx.arc(x, judgmentY + 60, 30, 0, Math.PI * 2);
ctx.fill();
// 轨道标签
ctx.fillStyle = '#FFFFFF';
ctx.font = '12px sans-serif';
ctx.textAlign = 'center';
ctx.fillText(track.label, x, judgmentY + 65);
// 输入状态指示
if (this.isInputActive(track.type)) {
ctx.fillStyle = track.color + '80';
ctx.beginPath();
ctx.arc(x, judgmentY + 60, 35, 0, Math.PI * 2);
ctx.fill();
}
});
}
/**
* 绘制判定线
*/
private drawJudgmentLine(ctx: CanvasRenderingContext2D, w: number, h: number): void {
const y = h * this.judgmentLineY;
ctx.strokeStyle = 'rgba(255,255,255,0.3)';
ctx.lineWidth = 3;
ctx.setLineDash([10, 10]);
ctx.beginPath();
ctx.moveTo(0, y);
ctx.lineTo(w, y);
ctx.stroke();
ctx.setLineDash([]);
}
/**
* 绘制音符
*/
private drawNote(ctx: CanvasRenderingContext2D, note: Note, w: number, h: number): void {
if (note.hit || note.missed || note.y < -100 || note.y > h + 100) return;
const track = this.tracks[note.trackIndex];
const x = w * track.xRatio;
const size = 40 + note.intensity * 20;
// 音符主体
ctx.fillStyle = track.color;
ctx.globalAlpha = 0.9;
ctx.beginPath();
ctx.arc(x, note.y, size / 2, 0, Math.PI * 2);
ctx.fill();
// 音符光晕
ctx.globalAlpha = 0.3;
ctx.beginPath();
ctx.arc(x, note.y, size, 0, Math.PI * 2);
ctx.fill();
// 强度指示环
ctx.strokeStyle = '#FFFFFF';
ctx.lineWidth = 2;
ctx.globalAlpha = note.intensity;
ctx.beginPath();
ctx.arc(x, note.y, size / 2 + 5, 0, Math.PI * 2);
ctx.stroke();
ctx.globalAlpha = 1.0;
}
/**
* 绘制判定反馈
*/
private drawJudgmentFeedback(ctx: CanvasRenderingContext2D, w: number, h: number): void {
if (!this.lastJudgment || Date.now() - this.judgmentTime > 500) return;
const age = Date.now() - this.judgmentTime;
const progress = age / 500;
const alpha = 1 - progress;
const scale = 1 + progress * 0.5;
const colors: Record<Judgment, string> = {
[Judgment.PERFECT]: '#FFD700',
[Judgment.GREAT]: '#00FF88',
[Judgment.GOOD]: '#4A90E2',
[Judgment.MISS]: '#FF4444'
};
const texts: Record<Judgment, string> = {
[Judgment.PERFECT]: 'PERFECT!',
[Judgment.GREAT]: 'GREAT!',
[Judgment.GOOD]: 'GOOD',
[Judgment.MISS]: 'MISS'
};
ctx.save();
ctx.translate(w / 2, h * 0.7);
ctx.scale(scale, scale);
ctx.fillStyle = colors[this.lastJudgment] + Math.floor(alpha * 255).toString(16).padStart(2, '0');
ctx.font = 'bold 48px sans-serif';
ctx.textAlign = 'center';
ctx.fillText(texts[this.lastJudgment], 0, 0);
ctx.restore();
}
/**
* 绘制HUD
*/
private drawHUD(ctx: CanvasRenderingContext2D, w: number, h: number): void {
// 分数
ctx.fillStyle = '#FFFFFF';
ctx.font = 'bold 36px sans-serif';
ctx.textAlign = 'left';
ctx.fillText(`Score: ${Math.floor(this.score)}`, 30, 60);
// 连击
if (this.combo > 0) {
const comboColor = this.combo > 50 ? '#FF6B6B' : this.combo > 20 ? '#FFD700' : '#FFFFFF';
ctx.fillStyle = comboColor;
ctx.font = 'bold 48px sans-serif';
ctx.textAlign = 'center';
ctx.fillText(`${this.combo}`, w / 2, 80);
ctx.font = '20px sans-serif';
ctx.fillText('COMBO', w / 2, 110);
}
// 最大连击
ctx.fillStyle = 'rgba(255,255,255,0.5)';
ctx.font = '16px sans-serif';
ctx.textAlign = 'right';
ctx.fillText(`Max: ${this.maxCombo}`, w - 30, 60);
}
/**
* 绘制AR追踪状态
*/
private drawARStatus(ctx: CanvasRenderingContext2D, w: number, h: number): void {
const statusY = h - 30;
// Face AR状态
ctx.fillStyle = this.faceDetected ? '#00FF88' : '#FF4444';
ctx.beginPath();
ctx.arc(30, statusY, 6, 0, Math.PI * 2);
ctx.fill();
ctx.fillStyle = '#FFFFFF80';
ctx.font = '12px sans-serif';
ctx.textAlign = 'left';
ctx.fillText('Face AR', 45, statusY + 4);
// Body AR状态
ctx.fillStyle = this.bodyDetected ? '#00FF88' : '#FF4444';
ctx.beginPath();
ctx.arc(130, statusY, 6, 0, Math.PI * 2);
ctx.fill();
ctx.fillStyle = '#FFFFFF80';
ctx.fillText('Body AR', 145, statusY + 4);
// 当前表情显示
if (this.faceExpression.size > 0) {
const exprs = Array.from(this.faceExpression.entries())
.filter(([, v]) => v > 0.3)
.map(([k]) => k.replace('EYE_', '').replace('MOUTH_', '').replace('JAW_', ''))
.slice(0, 3);
ctx.fillStyle = '#FFFFFF60';
ctx.fillText(exprs.join(' | '), w / 2, statusY + 4);
}
}
/**
* 获取连击对应的颜色
*/
private getComboColor(): string {
if (this.combo > 50) return '#FF6B6B';
if (this.combo > 30) return '#FFD700';
if (this.combo > 10) return '#4A90E2';
return '#4A90E2';
}
/**
* 检查某轨道的输入是否激活
*/
private isInputActive(type: NoteType): boolean {
switch (type) {
case NoteType.FACE_UP:
return (this.faceExpression.get('EYE_BROW_UP_LEFT') || 0) > 0.5;
case NoteType.FACE_DOWN:
return (this.faceExpression.get('JAW_OPEN') || 0) > 0.4;
case NoteType.FACE_LEFT:
return (this.faceExpression.get('EYE_BLINK_LEFT') || 0) > 0.6;
case NoteType.FACE_RIGHT:
return (this.faceExpression.get('EYE_BLINK_RIGHT') || 0) > 0.6;
case NoteType.BODY_LEFT:
return this.leftHandUp;
case NoteType.BODY_RIGHT:
return this.rightHandUp;
default:
return false;
}
}
/**
* 触发击打特效
*/
private triggerHitEffect(note: Note, judgment: Judgment, intensity: number): void {
// 同步到全局,由粒子系统处理
AppStorage.setOrCreate('hit_effect', {
trackIndex: note.trackIndex,
judgment,
intensity,
timestamp: Date.now()
});
}
/**
* 触发连击加成
*/
private triggerComboBonus(): void {
this.score += 500;
AppStorage.setOrCreate('combo_bonus', { combo: this.combo, timestamp: Date.now() });
}
/**
* 触发特殊技能(双手同时抬起)
*/
private triggerSpecialSkill(): void {
// 清除屏幕上所有音符并得分
let cleared = 0;
this.notes.forEach(note => {
if (!note.hit && !note.missed) {
note.hit = true;
cleared++;
}
});
this.score += cleared * 200;
AppStorage.setOrCreate('special_skill', { cleared, timestamp: Date.now() });
}
build() {
Canvas(this.canvasRef)
.width('100%')
.height('100%')
.backgroundColor('#0a0a1a')
.onReady((context) => {
this.canvasRef = context;
})
}
}
3.2 悬浮节奏HUD与沉浸光效反馈
代码亮点:底部悬浮HUD实时显示AR输入状态和节奏指引,根据判定结果(Perfect/Great/Good/Miss)触发对应颜色的全屏光效脉冲,连击数越高光效越强烈 。
typescript
// components/RhythmHUD.ets
import { window } from '@kit.ArkUI';
import { Judgment } from './RhythmTrackSystem';
@Component
export struct RhythmHUD {
@State combo: number = 0;
@State lastJudgment: Judgment | null = null;
@State judgmentIntensity: number = 1.0;
@State faceDetected: boolean = false;
@State bodyDetected: boolean = false;
@State leftHandUp: boolean = false;
@State rightHandUp: boolean = false;
@State expressionPreview: string = '';
@State bottomAvoidHeight: number = 0;
@State bpm: number = 128;
aboutToAppear(): void {
this.getBottomAvoidArea();
this.setupGameStateListening();
}
private async getBottomAvoidArea(): Promise<void> {
try {
const mainWindow = await window.getLastWindow();
const avoidArea = mainWindow.getWindowAvoidArea(window.AvoidAreaType.TYPE_NAVIGATION_INDICATOR);
this.bottomAvoidHeight = avoidArea.bottomRect.height;
} catch (error) {
console.error('Failed to get avoid area:', error);
}
}
private setupGameStateListening(): void {
AppStorage.watch('last_judgment', (data: { judgment: Judgment; combo: number; intensity: number }) => {
if (data) {
this.lastJudgment = data.judgment;
this.combo = data.combo;
this.judgmentIntensity = data.intensity;
this.triggerJudgmentLightEffect(data.judgment, data.intensity);
}
});
AppStorage.watch('face_detected', (v: boolean) => this.faceDetected = v);
AppStorage.watch('body_detected', (v: boolean) => this.bodyDetected = v);
AppStorage.watch('body_gesture', (g: { hands: { left: boolean; right: boolean } }) => {
if (g) {
this.leftHandUp = g.hands.left;
this.rightHandUp = g.hands.right;
}
});
AppStorage.watch('face_expression', (expr: Map<string, number>) => {
if (expr) {
const active = Array.from(expr.entries())
.filter(([, v]) => v > 0.4)
.map(([k]) => k.replace(/EYE_|MOUTH_|JAW_/g, ''))
.slice(0, 2);
this.expressionPreview = active.join(' ');
}
});
const config = AppStorage.get<{ bpm: number }>('game_config');
if (config) this.bpm = config.bpm;
}
/**
* 触发判定光效
*/
private triggerJudgmentLightEffect(judgment: Judgment, intensity: number): void {
const colors: Record<Judgment, string> = {
[Judgment.PERFECT]: '#FFD700',
[Judgment.GREAT]: '#00FF88',
[Judgment.GOOD]: '#4A90E2',
[Judgment.MISS]: '#FF4444'
};
AppStorage.setOrCreate('judgment_light_pulse', {
color: colors[judgment],
intensity: intensity * (judgment === Judgment.PERFECT ? 1.5 : 1.0),
timestamp: Date.now()
});
}
build() {
Stack({ alignContent: Alignment.Bottom }) {
// 内容占位
Column() {}
.width('100%')
.height('100%')
// 判定结果大字(屏幕中央)
if (this.lastJudgment && Date.now() - (AppStorage.get<number>('judgment_time') || 0) < 600) {
this.buildJudgmentDisplay()
}
// 底部悬浮HUD
Column() {
Stack() {
// 动态光效背景(根据判定结果变色)
Column()
.width('100%')
.height('100%')
.backgroundColor(this.getJudgmentColor())
.opacity(this.getJudgmentOpacity())
.blur(60)
.animation({
duration: 300,
curve: Curve.EaseOut
})
Column()
.width('100%')
.height('100%')
.backgroundBlurStyle(BlurStyle.COMPONENT_THICK)
.opacity(0.85)
Column()
.width('100%')
.height('100%')
.linearGradient({
direction: GradientDirection.Top,
colors: [
['rgba(255,255,255,0.15)', 0.0],
['rgba(255,255,255,0.05)', 0.5],
['transparent', 1.0]
]
})
}
.width('100%')
.height('100%')
.borderRadius(24)
.shadow({
radius: 20,
color: this.getJudgmentColor() + '40',
offsetX: 0,
offsetY: -4
})
// HUD内容
Column({ space: 8 }) {
// 顶部:AR输入状态指示
Row({ space: 16 }) {
// Face AR状态
Row({ space: 6 }) {
Column()
.width(8)
.height(8)
.backgroundColor(this.faceDetected ? '#00FF88' : '#FF4444')
.borderRadius(4)
.shadow({ radius: 4, color: this.faceDetected ? '#00FF88' : '#FF4444' })
Text('面部')
.fontSize(11)
.fontColor(this.faceDetected ? '#00FF88' : '#FF4444')
}
// Body AR状态
Row({ space: 6 }) {
Column()
.width(8)
.height(8)
.backgroundColor(this.bodyDetected ? '#00FF88' : '#FF4444')
.borderRadius(4)
.shadow({ radius: 4, color: this.bodyDetected ? '#00FF88' : '#FF4444' })
Text('肢体')
.fontSize(11)
.fontColor(this.bodyDetected ? '#00FF88' : '#FF4444')
}
// 当前表情
if (this.expressionPreview) {
Text(this.expressionPreview)
.fontSize(11)
.fontColor('#FFFFFF60')
.backgroundColor('rgba(255,255,255,0.08)')
.padding({ left: 8, right: 8, top: 2, bottom: 2 })
.borderRadius(4)
}
// BPM指示
Row({ space: 4 }) {
Column()
.width(6)
.height(6)
.backgroundColor('#FFFFFF')
.borderRadius(3)
.animation({
duration: 60000 / this.bpm,
curve: Curve.EaseInOut,
iterations: -1,
playMode: PlayMode.Alternate
})
.scale({ x: 1.5, y: 1.5 })
Text(`${this.bpm} BPM`)
.fontSize(11)
.fontColor('#FFFFFF60')
}
}
.width('100%')
.justifyContent(FlexAlign.Center)
.padding({ top: 10 })
// 中部:手势输入可视化
Row({ space: 24 }) {
this.buildGestureIndicator('挑眉', 'EYE_BROW_UP_LEFT', '#FF6B6B')
this.buildGestureIndicator('张嘴', 'JAW_OPEN', '#DDA0DD')
this.buildGestureIndicator('左手', 'body_left', '#45B7D1', this.leftHandUp)
this.buildGestureIndicator('右手', 'body_right', '#96CEB4', this.rightHandUp)
this.buildGestureIndicator('左眨眼', 'EYE_BLINK_LEFT', '#4ECDC4')
this.buildGestureIndicator('右眨眼', 'EYE_BLINK_RIGHT', '#FFEAA7')
}
.width('100%')
.justifyContent(FlexAlign.SpaceAround)
.padding({ top: 8, bottom: 8 })
// 底部:连击与判定历史
if (this.combo > 0) {
Row({ space: 8 }) {
Text(`${this.combo}`)
.fontSize(24)
.fontWeight(FontWeight.Bold)
.fontColor(this.combo > 30 ? '#FFD700' : '#FFFFFF')
Text('连击')
.fontSize(12)
.fontColor('#FFFFFF60')
}
}
}
.width('100%')
.padding({ bottom: 8 })
}
.width('94%')
.height(130)
.margin({
bottom: this.bottomAvoidHeight + 16,
left: '3%',
right: '3%'
})
}
.width('100%')
.height('100%')
}
@Builder
buildJudgmentDisplay(): void {
Column() {
const texts: Record<Judgment, string> = {
[Judgment.PERFECT]: 'PERFECT!',
[Judgment.GREAT]: 'GREAT!',
[Judgment.GOOD]: 'GOOD',
[Judgment.MISS]: 'MISS'
};
Text(texts[this.lastJudgment!])
.fontSize(64)
.fontWeight(FontWeight.Bold)
.fontColor(this.getJudgmentColor())
.shadow({ radius: 20, color: this.getJudgmentColor() })
}
.width('100%')
.height('100%')
.justifyContent(FlexAlign.Center)
.animation({
duration: 200,
curve: Curve.Spring
})
}
@Builder
buildGestureIndicator(label: string, key: string, color: string, forceActive: boolean = false): void {
const isActive = forceActive || this.isExpressionActive(key);
Column({ space: 4 }) {
Stack() {
// 激活光晕
if (isActive) {
Column()
.width(44)
.height(44)
.backgroundColor(color)
.borderRadius(12)
.opacity(0.3)
.blur(8)
}
// 图标占位(简化)
Column()
.width(36)
.height(36)
.backgroundColor(isActive ? color : 'rgba(255,255,255,0.1)')
.borderRadius(10)
.opacity(isActive ? 1.0 : 0.5)
}
.width(44)
.height(44)
Text(label)
.fontSize(10)
.fontColor(isActive ? color : 'rgba(255,255,255,0.4)')
}
}
private isExpressionActive(key: string): boolean {
if (key.startsWith('body_')) return false;
return (AppStorage.get<Map<string, number>>('face_expression')?.get(key) || 0) > 0.4;
}
private getJudgmentColor(): string {
if (!this.lastJudgment) return '#4A90E2';
const colors: Record<Judgment, string> = {
[Judgment.PERFECT]: '#FFD700',
[Judgment.GREAT]: '#00FF88',
[Judgment.GOOD]: '#4A90E2',
[Judgment.MISS]: '#FF4444'
};
return colors[this.lastJudgment];
}
private getJudgmentOpacity(): number {
if (!this.lastJudgment) return 0.1;
const baseOpacities: Record<Judgment, number> = {
[Judgment.PERFECT]: 0.25,
[Judgment.GREAT]: 0.2,
[Judgment.GOOD]: 0.15,
[Judgment.MISS]: 0.1
};
return baseOpacities[this.lastJudgment] * this.judgmentIntensity;
}
}
3.3 主游戏页面:全屏沉浸与光效整合
typescript
// pages/RhythmGamePage.ets
import { RhythmTrackSystem } from '../components/RhythmTrackSystem';
import { RhythmHUD } from '../components/RhythmHUD';
@Entry
@Component
struct RhythmGamePage {
@State judgmentPulse: { color: string; intensity: number; timestamp: number } | null = null;
@State comboBonus: { combo: number; timestamp: number } | null = null;
@State specialSkill: { cleared: number; timestamp: number } | null = null;
aboutToAppear(): void {
AppStorage.watch('judgment_light_pulse', (pulse: { color: string; intensity: number; timestamp: number }) => {
this.judgmentPulse = pulse;
});
AppStorage.watch('combo_bonus', (bonus: { combo: number; timestamp: number }) => {
this.comboBonus = bonus;
});
AppStorage.watch('special_skill', (skill: { cleared: number; timestamp: number }) => {
this.specialSkill = skill;
});
}
build() {
Stack() {
// 第一层:动态环境光(BPM驱动+判定反馈)
this.buildDynamicLighting()
// 第二层:游戏主画面
RhythmTrackSystem()
// 第三层:特效覆盖层
if (this.comboBonus && Date.now() - this.comboBonus.timestamp < 1000) {
this.buildComboBonusEffect()
}
if (this.specialSkill && Date.now() - this.specialSkill.timestamp < 1500) {
this.buildSpecialSkillEffect()
}
// 第四层:悬浮HUD
RhythmHUD()
// 第五层:顶部信息栏
this.buildTopInfoBar()
}
.width('100%')
.height('100%')
.backgroundColor('#0a0a1a')
.expandSafeArea(
[SafeAreaType.SYSTEM],
[SafeAreaEdge.TOP, SafeAreaEdge.BOTTOM, SafeAreaEdge.START, SafeAreaEdge.END]
)
}
@Builder
buildDynamicLighting(): void {
Column() {
// BPM基础光晕
Column()
.width('100%')
.height('100%')
.backgroundColor('#4A90E2')
.opacity(0.08)
.blur(100)
// 判定脉冲光效
if (this.judgmentPulse && Date.now() - this.judgmentPulse.timestamp < 300) {
Column()
.width('100%')
.height('100%')
.backgroundColor(this.judgmentPulse.color)
.opacity(this.judgmentPulse.intensity * 0.3)
.blur(50)
.animation({
duration: 300,
curve: Curve.EaseOut
})
}
// 连击加成光效
if (this.comboBonus && Date.now() - this.comboBonus.timestamp < 1000) {
const progress = (Date.now() - this.comboBonus.timestamp) / 1000;
Column()
.width('100%')
.height('100%')
.backgroundColor('#FFD700')
.opacity((1 - progress) * 0.2)
.blur(80)
}
}
.width('100%')
.height('100%')
}
@Builder
buildComboBonusEffect(): void {
Column() {
Text(`x${this.comboBonus?.combo} COMBO BONUS!`)
.fontSize(48)
.fontWeight(FontWeight.Bold)
.fontColor('#FFD700')
.shadow({ radius: 20, color: '#FFD700' })
}
.width('100%')
.height('100%')
.justifyContent(FlexAlign.Center)
.animation({
duration: 1000,
curve: Curve.EaseOut
})
}
@Builder
buildSpecialSkillEffect(): void {
Column() {
Text(`FULL CLEAR!`)
.fontSize(56)
.fontWeight(FontWeight.Bold)
.fontColor('#FF6B6B')
.shadow({ radius: 30, color: '#FF6B6B' })
Text(`${this.specialSkill?.cleared} notes cleared`)
.fontSize(20)
.fontColor('#FFFFFF')
.margin({ top: 12 })
}
.width('100%')
.height('100%')
.justifyContent(FlexAlign.Center)
.backgroundColor('rgba(0,0,0,0.5)')
.animation({
duration: 1500,
curve: Curve.EaseOut
})
}
@Builder
buildTopInfoBar(): void {
Row() {
Text('🎵 律动星途')
.fontSize(18)
.fontWeight(FontWeight.Bold)
.fontColor('#FFFFFF')
Row({ space: 12 }) {
Text('体感音游')
.fontSize(12)
.fontColor('#FFFFFF60')
.backgroundColor('rgba(255,255,255,0.08)')
.padding({ left: 8, right: 8, top: 2, bottom: 2 })
.borderRadius(4)
Text('AR模式')
.fontSize(12)
.fontColor('#00FF8880')
.backgroundColor('rgba(0,255,136,0.1)')
.padding({ left: 8, right: 8, top: 2, bottom: 2 })
.borderRadius(4)
}
}
.width('100%')
.height(44)
.padding({ left: 20, right: 20 })
.justifyContent(FlexAlign.SpaceBetween)
.backgroundBlurStyle(BlurStyle.REGULAR)
.backgroundColor('rgba(0,0,0,0.3)')
}
}
四、关键技术总结
4.1 AR输入-游戏映射体系
| 输入类型 | AR参数 | 游戏映射 | 判定精度 |
|---|---|---|---|
| 挑眉 | EYE_BROW_UP_LEFT > 0.6 |
上轨道音符 | 高(表情明显) |
| 张嘴 | JAW_OPEN > 0.5 |
下轨道音符 | 高 |
| 左眨眼 | EYE_BLINK_LEFT > 0.8 |
左轨道音符 | 极高(瞬态动作) |
| 右眨眼 | EYE_BLINK_RIGHT > 0.8 |
右轨道音符 | 极高 |
| 左手抬起 | LEFT_WRIST.y < LEFT_SHOULDER.y |
左肢体轨道 | 中 |
| 右手抬起 | RIGHT_WRIST.y < RIGHT_SHOULDER.y |
右肢体轨道 | 中 |
| 双手同时 | 上述同时满足 | 全屏清场技能 | 组合判定 |
| 微笑 | MOUTH_SMILE_LEFT > 0.7 |
连击加成 | 辅助功能 |
4.2 判定系统与光效反馈
| 判定等级 | 时间窗口 | 得分 | 光效强度 | 连击影响 |
|---|---|---|---|---|
| PERFECT | ±50ms | 300 | 1.5x | +1连击 |
| GREAT | ±100ms | 200 | 1.0x | +1连击 |
| GOOD | ±150ms | 100 | 0.7x | +1连击 |
| MISS | >150ms | 0 | 0.3x | 连击清零 |
4.3 性能优化策略
typescript
// 1. AR数据降频:表情检测30fps,骨骼检测15fps(音游不需要全速)
// 2. 音符对象池:复用Note对象,避免GC卡顿
// 3. 离屏渲染:复杂粒子特效预渲染
// 4. 音频预加载:使用AudioKit预解码音频资源
// 5. 判定预测:基于BPM预计算下一拍时间,提前准备渲染
五、调试与设备适配
5.1 调试要点
- 延迟校准:AR追踪到画面显示的延迟需<50ms,否则影响判定精度
- 表情阈值调优:不同用户的表情强度差异大,需提供个性化校准
- 光线条件:避免逆光,确保面部和手部清晰可见
- 音频同步:确保音符下落与音乐节拍严格同步
5.2 设备适配
| 设备类型 | 摄像头 | 推荐配置 | 体验等级 |
|---|---|---|---|
| 华为Mate 60 Pro | 前置3D深感 | 全功能 | 极佳 |
| 华为MatePad Pro | 前置TOF | 面部+手势 | 优秀 |
| 华为MateBook X | 外接摄像头 | 面部为主 | 良好 |
| 其他设备 | 普通前置 | 基础面部 | 可用 |
六、总结与展望
本文基于 HarmonyOS 6(API 23)的 Face AR 、Body AR 、悬浮导航 与沉浸光感四大特性,完整实战了一款"律动星途"体感音游。核心创新点:
- 表情即操作:通过Face AR的64种BlendShape参数,将挑眉、张嘴、眨眼等微表情映射为游戏输入,实现"用脸打音游"的全新体验
- 肢体即节奏:通过Body AR的20+骨骼关键点,将双手抬起、身体摇摆映射为节奏击打,全身参与游戏
- 光效即反馈:判定结果(Perfect/Great/Good/Miss)实时驱动全屏光效脉冲,连击数越高光效越强烈,打造"视听一体"的沉浸体验
- 悬浮即HUD:底部悬浮导航实时显示AR输入状态和节奏指引,不遮挡游戏画面,同时提供沉浸光感的视觉层次
未来扩展方向:
- 多人对战:通过分布式软总线实现双人Face AR表情PK,看谁的表情更精准
- AI谱面生成:基于AudioKit分析音乐BPM和节拍,自动生成对应AR输入的谱面
- 健康监测:结合运动健康服务,统计游戏过程中的运动量和卡路里消耗
- UGC谱面:开放谱面编辑器,让玩家自定义表情-音符映射关系
转载自:https://blog.csdn.net/u014727709/article/details/148800052
欢迎 👍点赞✍评论⭐收藏,欢迎指正