一、引言：通用搜索的技术纵深
与垂直搜索不同，通用搜索引擎面对的是跨领域、多模态、异构的海量数据。它不仅要处理文本，还要处理图片、视频、知识图谱等多种形态的数据。在服务端，通用搜索的核心竞争力体现在三个层面：召回层 的全面性与覆盖率，排序层 的精准性与个性化，以及特征计算层的实时性与丰富度。
本文将聚焦于通用搜索服务端的特性功能开发，深入剖析底层召回引擎、排序插件架构以及特征计算系统，给出完整的技术方案和代码实践。一个完整的通用搜索系统应当具备开放的插件化排序能力，支持召回策略的热插拔，同时为特征计算提供统一的接口与治理。
二、系统整体架构
通用搜索服务端遵循插件化、可配置的设计理念，使召回、排序、特征计算等核心环节具备独立演进而非耦合变更的能力。自下而上分为五层：引擎基础层提供Lucene/ES核心索引能力；召回层由多路召回插件组成，支持倒排、向量、个性化等多种策略；排序层加载排序插件，支持LTR模型、GBDT、深度学习等不同算法；特征计算层实时计算文档-查询匹配特征；服务接入层对外提供统一的搜索API。
三、底层召回引擎设计与实现
3.1 多路召回架构
召回是搜索系统的第一道关，决定了后续排序能够处理的上限。单一召回策略难以覆盖所有场景，多路召回+融合成为通用搜索引擎的标准架构。
java
// RecallEngine.java - 召回引擎核心接口
package com.search.engine.recall;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
/**
* 召回结果封装
*/
public class RecallResult {
private String docId;
private float score;
private String source; // 召回来源：inverted/vector/geo/personal
private Map<String, Object> features; // 临时特征，供排序使用
// getters/setters...
}
/**
* 召回上下文
*/
public class RecallContext {
private String query;
private String userId;
private Map<String, Object> params; // 扩展参数
// getters/setters...
}
/**
* 召回插件接口
*/
public interface RecallPlugin {
/**
* 执行召回
* @param context 召回上下文
* @return 召回结果列表
*/
List<RecallResult> recall(RecallContext context);
/**
* 插件名称
*/
String getName();
/**
* 优先级（数值越小越先执行）
*/
int getPriority();
}
/**
* 召回引擎主控
*/
public class RecallEngine {
private List<RecallPlugin> plugins;
private int maxTotalResults = 1000;
private int maxPerPlugin = 200;
public RecallEngine(List<RecallPlugin> plugins) {
this.plugins = plugins.stream()
.sorted((a, b) -> Integer.compare(a.getPriority(), b.getPriority()))
.collect(Collectors.toList());
}
/**
* 执行多路召回（并行）
*/
public List<RecallResult> recall(RecallContext context) {
// 并行执行所有召回插件
List<CompletableFuture<List<RecallResult>>> futures = plugins.stream()
.map(plugin -> CompletableFuture.supplyAsync(() -> {
long start = System.currentTimeMillis();
List<RecallResult> results = plugin.recall(context);
long cost = System.currentTimeMillis() - start;
// 记录监控日志
log.info("Recall plugin {} returned {} results, cost {}ms",
plugin.getName(), results.size(), cost);
// 截断每个插件的返回数量
return results.stream().limit(maxPerPlugin).collect(Collectors.toList());
}))
.collect(Collectors.toList());
// 等待所有召回完成
List<RecallResult> allResults = futures.stream()
.map(CompletableFuture::join)
.flatMap(List::stream)
.collect(Collectors.toList());
// 去重（按docId保留最高分）
Map<String, RecallResult> dedupMap = new LinkedHashMap<>();
for (RecallResult result : allResults) {
String key = result.getDocId();
if (!dedupMap.containsKey(key) ||
dedupMap.get(key).getScore() < result.getScore()) {
dedupMap.put(key, result);
}
}
// 按得分排序并截断
return dedupMap.values().stream()
.sorted((a, b) -> Float.compare(b.getScore(), a.getScore()))
.limit(maxTotalResults)
.collect(Collectors.toList());
}
}3.2 倒排索引召回插件
java
// InvertedRecallPlugin.java - 倒排索引召回
package com.search.engine.recall.plugin;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.*;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.queryparser.classic.QueryParser;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.MMapDirectory;
import org.apache.lucene.document.Document;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
public class InvertedRecallPlugin implements RecallPlugin {
private IndexSearcher searcher;
private Analyzer analyzer;
private QueryParser parser;
public InvertedRecallPlugin(String indexDir) throws Exception {
Directory dir = MMapDirectory.open(Paths.get(indexDir));
this.searcher = new IndexSearcher(DirectoryReader.open(dir));
this.analyzer = new StandardAnalyzer();
this.parser = new QueryParser("content", analyzer);
}
@Override
public List<RecallResult> recall(RecallContext context) {
List<RecallResult> results = new ArrayList<>();
String queryText = context.getQuery();
try {
// 构建布尔查询
BooleanQuery.Builder builder = new BooleanQuery.Builder();
// 主查询：对content字段进行全文检索
Query contentQuery = parser.parse(queryText);
builder.add(contentQuery, BooleanClause.Occur.MUST);
// 标题加权：标题匹配的文档加分
Query titleQuery = parser.parse(queryText);
Query boostedTitle = new BoostQuery(titleQuery, 2.0f);
builder.add(boostedTitle, BooleanClause.Occur.SHOULD);
BooleanQuery booleanQuery = builder.build();
// 执行查询
TopDocs topDocs = searcher.search(booleanQuery, 200);
for (ScoreDoc scoreDoc : topDocs.scoreDocs) {
Document doc = searcher.doc(scoreDoc.doc);
RecallResult result = new RecallResult();
result.setDocId(doc.get("doc_id"));
result.setScore(scoreDoc.score);
result.setSource("inverted");
results.add(result);
}
} catch (Exception e) {
log.error("Inverted recall failed: {}", e.getMessage());
}
return results;
}
@Override
public String getName() {
return "inverted";
}
@Override
public int getPriority() {
return 10;
}
}3.3 向量召回插件
在语义搜索场景中，基于稠密向量的近似最近邻检索是补充倒排召回的关键路径。
java
// VectorRecallPlugin.java - 向量召回
package com.search.engine.recall.plugin;
import ai.djl.modality.embeddings.Embedding;
import ai.djl.modality.embeddings.EmbeddingException;
import org.apache.lucene.util.VectorSimilarityFunction;
import java.util.List;
public class VectorRecallPlugin implements RecallPlugin {
private EmbeddingService embeddingService;
private VectorIndex vectorIndex;
private int topK = 100;
public VectorRecallPlugin(EmbeddingService embeddingService, VectorIndex vectorIndex) {
this.embeddingService = embeddingService;
this.vectorIndex = vectorIndex;
}
@Override
public List<RecallResult> recall(RecallContext context) {
List<RecallResult> results = new ArrayList<>();
try {
// 1. 将Query转换为向量
float[] queryVector = embeddingService.embed(context.getQuery());
// 2. 在向量索引中执行ANN搜索
List<VectorSearchResult> vectorResults = vectorIndex.search(queryVector, topK);
// 3. 转换为标准RecallResult
for (VectorSearchResult vResult : vectorResults) {
RecallResult result = new RecallResult();
result.setDocId(vResult.getDocId());
result.setScore(vResult.getScore());
result.setSource("vector");
results.add(result);
}
} catch (EmbeddingException e) {
log.error("Vector recall failed: {}", e.getMessage());
}
return results;
}
@Override
public String getName() {
return "vector";
}
@Override
public int getPriority() {
return 20;
}
}3.4 个性化召回插件
基于用户画像和历史的个性化召回能够显著提升用户体验，尤其在信息流和推荐式搜索场景中价值突出。
java
// PersonalizedRecallPlugin.java - 个性化召回
package com.search.engine.recall.plugin;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
public class PersonalizedRecallPlugin implements RecallPlugin {
private JedisPool redisPool;
private int maxResults = 50;
public PersonalizedRecallPlugin(JedisPool redisPool) {
this.redisPool = redisPool;
}
@Override
public List<RecallResult> recall(RecallContext context) {
List<RecallResult> results = new ArrayList<>();
String userId = context.getUserId();
if (userId == null || userId.isEmpty()) {
return results;
}
try (Jedis jedis = redisPool.getResource()) {
// 1. 获取用户的兴趣标签（从Redis读取）
String tagKey = "user:" + userId + ":tags";
Set<String> tags = jedis.smembers(tagKey);
if (tags.isEmpty()) {
return results;
}
// 2. 基于标签召回相关文档（根据标签从倒排索引或缓存获取）
for (String tag : tags) {
String docKey = "tag:" + tag + ":docs";
Set<String> docIds = jedis.zrevrange(docKey, 0, 10);
for (String docId : docIds) {
RecallResult result = new RecallResult();
result.setDocId(docId);
result.setScore(0.8f); // 个性化召回基础分
result.setSource("personalized");
results.add(result);
}
}
} catch (Exception e) {
log.error("Personalized recall failed: {}", e.getMessage());
}
return results.stream().limit(maxResults).collect(Collectors.toList());
}
@Override
public String getName() {
return "personalized";
}
@Override
public int getPriority() {
return 30;
}
}四、排序插件系统设计
排序插件系统是通用搜索服务端最核心的可扩展模块，支持多种排序策略的热加载。
java
// RankingPlugin.java - 排序插件接口
package com.search.engine.ranking;
/**
* 排序上下文
*/
public class RankingContext {
private String query;
private String userId;
private List<RecallResult> candidates;
private Map<String, Object> features;
// getters/setters...
}
/**
* 排序结果
*/
public class RankingResult {
private String docId;
private float score;
private Map<String, Float> featureScores; // 各特征贡献
// getters/setters...
}
/**
* 排序插件接口
*/
public interface RankingPlugin {
/**
* 对候选文档进行排序
*/
List<RankingResult> rank(RankingContext context);
/**
* 插件名称
*/
String getName();
/**
* 是否启用
*/
boolean isEnabled();
}
/**
* 基于LTR的排序插件
*/
public class LTRRankingPlugin implements RankingPlugin {
private XGBoostModel model; // 使用XGBoost作为LTR模型
private FeatureExtractor featureExtractor;
private boolean enabled = true;
public LTRRankingPlugin(String modelPath) throws Exception {
this.model = new XGBoostModel(modelPath);
this.featureExtractor = new FeatureExtractor();
}
@Override
public List<RankingResult> rank(RankingContext context) {
List<RankingResult> results = new ArrayList<>();
String query = context.getQuery();
List<RecallResult> candidates = context.getCandidates();
if (candidates == null || candidates.isEmpty()) {
return results;
}
// 提取每个文档的特征
List<float[]> featureMatrix = new ArrayList<>();
List<String> docIds = new ArrayList<>();
for (RecallResult candidate : candidates) {
// 提取特征（需要访问文档的详细字段）
float[] features = featureExtractor.extract(query, candidate.getDocId());
featureMatrix.add(features);
docIds.add(candidate.getDocId());
}
// 模型预测得分
float[] scores = model.predict(featureMatrix);
for (int i = 0; i < docIds.size(); i++) {
RankingResult result = new RankingResult();
result.setDocId(docIds.get(i));
result.setScore(scores[i]);
results.add(result);
}
// 按得分排序
results.sort((a, b) -> Float.compare(b.getScore(), a.getScore()));
return results;
}
@Override
public String getName() {
return "ltr";
}
@Override
public boolean isEnabled() {
return enabled;
}
}
/**
* 基于GBDT的排序插件
*/
public class GBDTRankingPlugin implements RankingPlugin {
private LightGBMModel model;
private FeatureExtractor featureExtractor;
private boolean enabled = true;
@Override
public List<RankingResult> rank(RankingContext context) {
// 实现类似LTRRankingPlugin
// 区别在于使用LightGBM模型
return null;
}
@Override
public String getName() {
return "gbdt";
}
@Override
public boolean isEnabled() {
return enabled;
}
}
/**
* 排序引擎主控
*/
public class RankingEngine {
private List<RankingPlugin> plugins;
private RankingPlugin activePlugin;
public RankingEngine(List<RankingPlugin> plugins) {
this.plugins = plugins;
// 选择第一个启用的插件作为默认
this.activePlugin = plugins.stream()
.filter(RankingPlugin::isEnabled)
.findFirst()
.orElse(null);
}
/**
* 执行排序（可动态切换插件）
*/
public List<RankingResult> rank(RankingContext context) {
if (activePlugin == null) {
// 如果没有可用插件，使用默认排序（按召回得分）
return defaultRank(context);
}
return activePlugin.rank(context);
}
/**
* 切换排序插件
*/
public boolean switchPlugin(String pluginName) {
RankingPlugin target = plugins.stream()
.filter(p -> p.getName().equals(pluginName) && p.isEnabled())
.findFirst()
.orElse(null);
if (target != null) {
this.activePlugin = target;
log.info("Switched to ranking plugin: {}", pluginName);
return true;
}
return false;
}
private List<RankingResult> defaultRank(RankingContext context) {
// 按召回原始得分排序
return context.getCandidates().stream()
.sorted((a, b) -> Float.compare(b.getScore(), a.getScore()))
.map(c -> {
RankingResult r = new RankingResult();
r.setDocId(c.getDocId());
r.setScore(c.getScore());
return r;
})
.collect(Collectors.toList());
}
}五、特征计算系统
特征计算是排序的前置环节，决定了排序模型的上限。在通用搜索场景中，特征计算需要支撑多种查询类型和异构数据。
java
// FeatureExtractor.java - 特征提取器
package com.search.engine.feature;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.TermStatistics;
import java.util.HashMap;
import java.util.Map;
public class FeatureExtractor {
private static final int FEATURE_DIM = 45; // 特征总维度
private IndexSearcher searcher;
private DocCache docCache; // 文档缓存
private QueryLogService queryLogService;
/**
* 提取文档-查询对的所有特征
*/
public float[] extract(String query, String docId) {
// 获取文档详情
Document doc = docCache.getDocument(docId);
if (doc == null) {
return new float[FEATURE_DIM];
}
float[] features = new float[FEATURE_DIM];
int idx = 0;
// ===== 1. 文本匹配特征 (0-15) =====
String title = doc.get("title");
String content = doc.get("content");
String summary = doc.get("summary");
features[idx++] = calculateTermOverlap(query, title); // 标题命中率
features[idx++] = calculateTermOverlap(query, content); // 内容命中率
features[idx++] = calculateTermOverlap(query, summary); // 摘要命中率
features[idx++] = calculateBM25(query, docId); // BM25得分
// ===== 2. 文档质量特征 (16-25) =====
features[idx++] = doc.get("quality_score") != null ?
Float.parseFloat(doc.get("quality_score")) : 0.5f;
features[idx++] = doc.get("page_rank") != null ?
Float.parseFloat(doc.get("page_rank")) : 0.0f;
features[idx++] = calculateFreshness(doc);
features[idx++] = doc.get("view_count") != null ?
Float.parseFloat(doc.get("view_count")) / 10000 : 0.0f;
// ===== 3. 统计特征 (26-35) =====
features[idx++] = title.length() / 100.0f;
features[idx++] = Math.min(content.length() / 1000.0f, 1.0f);
features[idx++] = doc.get("word_count") != null ?
Float.parseFloat(doc.get("word_count")) / 1000 : 0.0f;
features[idx++] = doc.get("has_image") != null &&
doc.get("has_image").equals("true") ? 1.0f : 0.0f;
// ===== 4. 行为特征 (36-44) =====
features[idx++] = getCtr(docId);
features[idx++] = getCollectCount(docId);
features[idx++] = getShareCount(docId);
features[idx++] = getCommentCount(docId);
return features;
}
/**
* 计算词项重叠度
*/
private float calculateTermOverlap(String query, String text) {
if (text == null || text.isEmpty()) return 0.0f;
Set<String> queryTokens = tokenize(query);
Set<String> textTokens = tokenize(text);
if (queryTokens.isEmpty()) return 0.0f;
long overlap = queryTokens.stream().filter(textTokens::contains).count();
return (float) overlap / queryTokens.size();
}
/**
* 计算BM25得分
*/
private float calculateBM25(String query, String docId) {
// 调用BM25评分器
return bm25Ranker.score(query, docId);
}
/**
* 计算时效性得分
*/
private float calculateFreshness(Document doc) {
String publishTime = doc.get("publish_time");
if (publishTime == null) return 0.5f;
try {
long time = Long.parseLong(publishTime);
long now = System.currentTimeMillis();
long daysAgo = (now - time) / (1000 * 24 * 3600);
// 30天内满分，超过365天衰减到0.1
if (daysAgo < 30) return 1.0f;
if (daysAgo < 365) return 1.0f - (daysAgo - 30) / 365.0f * 0.9f;
return 0.1f;
} catch (Exception e) {
return 0.5f;
}
}
private Set<String> tokenize(String text) {
// 使用Jieba/IK分词
return jiebaSegmenter.segment(text);
}
// 行为特征获取
private float getCtr(String docId) {
// 从统计服务获取点击率
return clickStatService.getCTR(docId);
}
private float getCollectCount(String docId) {
return behaviorService.getCollectCount(docId);
}
private float getShareCount(String docId) {
return behaviorService.getShareCount(docId);
}
private float getCommentCount(String docId) {
return behaviorService.getCommentCount(docId);
}
}特征实时计算优化
在大流量场景下，特征计算需要高效缓存和异步批处理。
java
// FeatureCacheService.java - 特征缓存服务
package com.search.engine.feature;
import com.github.benmanes.caffeine.cache.Cache;
import com.github.benmanes.caffeine.cache.Caffeine;
import java.util.concurrent.TimeUnit;
public class FeatureCacheService {
// 使用Caffeine作为本地缓存
private Cache<String, float[]> featureCache;
public FeatureCacheService() {
this.featureCache = Caffeine.newBuilder()
.maximumSize(100000) // 最多缓存10万个特征向量
.expireAfterWrite(5, TimeUnit.MINUTES) // 5分钟过期
.recordStats()
.build();
}
public float[] getOrCompute(String key, FeatureExtractor extractor,
String query, String docId) {
return featureCache.get(key, k -> extractor.extract(query, docId));
}
public void invalidate(String key) {
featureCache.invalidate(key);
}
}六、性能优化与监控
6.1 优化策略
| 优化方向 | 具体措施 | 收益 |
|---|---|---|
| 召回并行 | CompletableFuture并行执行多路召回 | 延迟降低60% |
| 特征缓存 | Caffeine本地缓存热点特征 | 特征计算耗时降低80% |
| 模型量化 | XGBoost模型INT8量化 | 推理速度提升2倍 |
| 索引预热 | 启动时加载常用索引段到内存 | 首次查询延迟降低50% |
| 分片路由 | 根据docId一致性哈希路由 | 分布式吞吐提升3倍 |
6.2 监控指标
java
// SearchMetrics.java - 监控指标采集
package com.search.engine.monitor;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
public class SearchMetrics {
private Timer recallTimer;
private Timer rankTimer;
private Timer featureTimer;
private Counter queryCounter;
public SearchMetrics(MeterRegistry registry) {
this.recallTimer = Timer.builder("search.recall.duration")
.description("召回耗时")
.register(registry);
this.rankTimer = Timer.builder("search.rank.duration")
.description("排序耗时")
.register(registry);
this.featureTimer = Timer.builder("search.feature.duration")
.description("特征计算耗时")
.register(registry);
this.queryCounter = Counter.builder("search.query.total")
.description("查询总数")
.register(registry);
}
public Timer.Sample startRecall() {
return Timer.start();
}
public void stopRecall(Timer.Sample sample) {
sample.stop(recallTimer);
}
public void incrementQueryCount() {
queryCounter.increment();
}
}七、总结
本文从通用搜索服务端视角出发，系统阐述了三个核心模块的设计与实现：
- 召回引擎：采用插件化架构支持倒排索引召回、向量召回和个性化召回的多路并行，通过去重与融合保证召回质量
- 排序插件系统：基于LTR/GBDT的排序插件设计，支持动态切换和热加载，提供灵活的排序策略扩展能力
- 特征计算系统：覆盖文本匹配、文档质量、统计特征和行为特征四类共45维特征，配合缓存机制保障实时性能
通用搜索引擎的工程本质是在召回率、准确率和响应时间之间寻找最优平衡。插件化的架构设计让召回、排序和特征计算各自独立演进，而并行化、缓存和监控则保障了系统在高并发下的稳定性。在LLM时代，语义理解能力将进一步增强通用搜索的深度，但扎实的工程基座依然是不可或缺的根基。