LinuxC++项目开发日志——基于正倒排索引的boost搜索引擎（5——通过cpp-httplib库建立网页模块）

基于正倒排索引的boost搜索引擎

• cpp-httplib库
• • [cpp-httplib 库介绍](#cpp-httplib 库介绍)
  • 常用功能与函数
  • • [1. 服务器相关](#1. 服务器相关)
    • [2. 客户端相关](#2. 客户端相关)
  • 下载与使用
  • • 下载路径
    • 使用方法
    • 简单示例
• 网页模块
• • 仿照其它成熟搜索页面
  • 编写主程序入口
  • 编写网页
• 完整代码
• • common.h
  • Index.hpp
  • Log.hpp
  • main.cc
  • makefile
  • Parser.cc
  • Parser.h
  • Search.hpp
  • Util.hpp
• 结果展示

cpp-httplib库

cpp-httplib 库介绍

cpp-httplib 是一个轻量级的 C++ HTTP 客户端 / 服务器库，由日本开发者 yhirose 开发。它的特点是：

• 单文件设计（仅需包含 httplib.h 即可使用）
• 支持 HTTP 1.1
• 同时提供客户端和服务器功能
• 跨平台（Windows、Linux、macOS 等）
• 无需额外依赖（仅需 C++11 及以上标准）
• 支持 SSL/TLS（需配合 OpenSSL）

常用功能与函数

1. 服务器相关

创建服务器

httplib::Server svr;

注册路由处理函数

// GET 请求处理
svr.Get("/hello", [](const httplib::Request& req, httplib::Response& res) {
  res.set_content("Hello World!", "text/plain");
});

// POST 请求处理
svr.Post("/submit", [](const httplib::Request& req, httplib::Response& res) {
  // 处理表单数据 req.body
  res.set_content("Received!", "text/plain");
});

启动服务器

// 监听 0.0.0.0:8080
if (svr.listen("0.0.0.0", 8080)) {
  // 服务器启动成功
}

Request 类主要成员

• method: 请求方法（GET/POST 等）
• path: 请求路径
• body: 请求体内容
• headers: 请求头集合
• params: URL 查询参数
• get_param(key): 获取查询参数

Response 类主要成员

• status: 状态码（200, 404 等）
• body: 响应体内容
• headers: 响应头集合
• set_content(content, content_type): 设置响应内容和类型
• set_header(name, value): 设置响应头

2. 客户端相关

创建客户端

httplib::Client cli("http://example.com");

发送 GET 请求

auto res = cli.Get("/api/data");
if (res && res->status == 200) {
  // 处理响应 res->body
}

发送 POST 请求

httplib::Params params;
params.emplace("name", "test");
params.emplace("value", "123");

auto res = cli.Post("/api/submit", params);

发送带请求体的 POST

std::string json_data = R"({"key": "value"})";
auto res = cli.Post("/api/json", json_data, "application/json");

下载与使用

下载路径

GitHub 仓库：https://github.com/yhirose/cpp-httplib

直接下载头文件：https://raw.githubusercontent.com/yhirose/cpp-httplib/master/httplib.h

使用方法

1.下载 httplib.h 文件

2.在项目中包含该文件：#include "httplib.h"

3.编译时需指定 C++11 及以上标准（如 g++ -std=c++11 main.cpp）

4.若使用 SSL 功能，需定义 CPPHTTPLIB_OPENSSL_SUPPORT 并链接 OpenSSL 库

5.编译器版本低可能会报错，升级一下编译器即可

简单示例

下面是一个完整的服务器示例：

#include "httplib.h"
#include <iostream>

int main() {
  httplib::Server svr;

  // 处理根路径请求
  svr.Get("/", [](const httplib::Request& req, httplib::Response& res) {
    res.set_content("<h1>Hello World!</h1>", "text/html");
  });

  // 处理带参数的请求
  svr.Get("/greet", [](const httplib::Request& req, httplib::Response& res) {
    auto name = req.get_param_value("name");
    if (name.empty()) {
      res.status = 400;
      res.set_content("Name parameter is required", "text/plain");
    } else {
      res.set_content("Hello, " + name + "!", "text/plain");
    }
  });

  std::cout << "Server running on http://localhost:8080" << std::endl;
  svr.listen("localhost", 8080);
  
  return 0;
}

这个库非常适合快速开发小型 HTTP 服务或客户端，由于其轻量性和易用性，在 C++ 社区中非常受欢迎。

网页模块

仿照其它成熟搜索页面

这是一个大公司建立的成熟的搜索页面，我们写的可以仿照着来。

**经过搜索之后，网页地址上会带上搜索的关键词，从而到数据库内部或者其它建立好的搜索模块中查找，在通过网页映射出来。
**

编写主程序入口

---

**当外部通过网页访问建立好的端口的时候，搜索模块会初始化一次，文档是已经建立好的，先绑定主网页html的路径，然后注册Get方法，网页访问/s的时候实用？word=来带参数，从而出发搜索模块的查找，然后把结果json串返回给浏览器。启动后绑定host和端口号，则开始运行。

#include "Log.hpp"
#include "common.h"
#include "Parser.h"
#include "Search.hpp"
#include "httplib.h"
#include <cstdio>
#include <cstring>
#include <string>
const bool INIT = false;
int main()
{
if(INIT)
{
Parser parser(Orignaldir, Tragetfile);
parser.Init();
}
ns_search::Search search;
httplib::Server svr;
svr.set_base_dir(Basewwwroot);
svr.Get("/s", [&](const httplib::Request& req, httplib::Response& rep){
std::string param = "word";
std::string word;
std::string out;
out.clear();
if(req.has_param(param))
{
word = req.get_param_value(param);
Log(LogModule::DEBUG) << "查找关键词：" << word;
}
// rep.set_content("Search: " + word, "text/plain");
bool b = search.SearchBy(word, out);
if(b)
rep.set_content(out, "application/json");
else
Log(DEBUG) << "查找失败";
});
svr.listen("0.0.0.0", 8080);
return 0;
}

编写网页

编写网页是从一个大概的框架开始先写主要部分，再用css美化，然后注册相关函数。
**```text

Boost搜索引擎

基于正倒排索引的高效文档检索系统

搜索

欢迎使用Boost搜索引擎

这是一个基于C++和Boost库实现的搜索引擎，采用正倒排索引技术，提供高效的文档检索功能。

在搜索框中输入关键词，点击搜索按钮即可查找相关文档。

``` ## 完整代码 ### common.h ```cpp #pragma once #include #include #include #include #include "Log.hpp" using std::cout; using std::endl; using namespace LogModule; const std::string Boost_Url_Head = "https://www.boost.org/doc/libs/1_89_0/doc/html"; // const std::string Boost_Url_Head = "../Data/html"; const std::string Basewwwroot = "./wwwroot"; const std::string Orignaldir = "../Data/html"; const std::string Tragetfile = "../Data/output.txt"; const std::string Output_sep = "\3"; const std::string Line_sep = "\n"; // 定义词典路径（根据实际路径修改） const std::string DICT_PATH = "dict/jieba.dict.utf8"; const std::string HMM_PATH = "dict/hmm_model.utf8"; const std::string USER_DICT_PATH = "dict/user.dict.utf8"; const std::string IDF_PATH = "dict/idf.utf8"; const std::string STOP_WORD_PATH = "dict/stop_words.utf8"; // 不可复制基类 class NonCopyable { protected: // 允许派生类构造和析构 NonCopyable() = default; ~NonCopyable() = default; // 禁止移动操作（可选，根据需求决定） // NonCopyable(NonCopyable&&) = delete; // NonCopyable& operator=(NonCopyable&&) = delete; private: // 禁止拷贝构造和拷贝赋值 NonCopyable(const NonCopyable&) = delete; NonCopyable& operator=(const NonCopyable&) = delete; }; ``` ### Index.hpp ```cpp #pragma once #include "Log.hpp" #include "Util.hpp" #include "common.h" #include #include #include #include #include #include #include #include #include namespace ns_index { //正排索引 typedef struct ForwordElem { std::string title_; std::string content_; std::string url_; size_t doc_id_ = 0; void Set(std::string title, std::string content, std::string url, size_t doc_id) { title_ = title; content_ = content; url_ = url; doc_id_ = doc_id; } }Forword_t; typedef struct InvertedElem { size_t doc_id_ = 0; std::string word_; size_t weight_ = 0; void Set(size_t doc_id, std::string word, size_t weight) { doc_id_ = doc_id; word_ = word; weight_ = weight; } }Inverted_t; typedef std::vector InvertedList; class Index : public NonCopyable { private: Index() = default; public: static Index* GetInstance() { static Index index; return &index; } public: Forword_t* QueryById(size_t id) { if(id < 0 || id >= Forword_Index_.size()) { Log(LogModule::DEBUG) << "id invalid!"; return nullptr; } return &Forword_Index_[id]; } InvertedList* QueryByWord(std::string word) { auto it = Inverted_Index_.find(word); if(it == Inverted_Index_.end()) { //Log(LogModule::DEBUG) << word << " find fail!"; return nullptr; } return &it->second; } size_t count = 0; bool BulidIndex() { if(isInit_) return false; size_t estimated_doc = 10000; size_t estimeted_words = 100000; Forword_Index_.reserve(estimated_doc); Inverted_Index_.reserve(estimeted_words); std::ifstream in(Tragetfile, std::ios::binary | std::ios::in); if(!in.is_open()) { Log(LogModule::ERROR) << "Targetfile open fail!BulidIndex fail!"; return false; } Log(LogModule::INFO) << "索引开始预加载..."; std::string singlefile; while (std::getline(in, singlefile)) { bool b = BuildForwordIndex(singlefile); if(!b) { Log(LogModule::DEBUG) << "Build Forword Index Error!"; continue; } b = BuildInvertedIndex(Forword_Index_.size() - 1); if(!b) { Log(LogModule::DEBUG) << "Build Inverted Index Error!"; continue; } count++; if(count % 1000 == 0) { Log(LogModule::INFO) << "索引文件加载中，当前进度：" << count; //debug //break; } } in.close(); isInit_ = true; Log(LogModule::INFO) << "索引加载完毕!已加载-" << count << "个文件!"; return true; } ~Index() = default; private: typedef struct DocCount { size_t title_cnt_ = 0; size_t content_cnt_ = 0; }DocCount_t; bool BuildForwordIndex(std::string& singlefile) { sepfile.clear(); bool b = ns_util::JiebaUtile::CutDoc(singlefile, sepfile); if(!b) return false; // if(count == 764) // { // Log(LogModule::DEBUG) << "Index Url: " << sepfile[2]; // } if(sepfile.size() != 3) { Log(LogModule::DEBUG) << "Segmentation fail!"; return false; } Forword_t ft; ft.Set(std::move(sepfile[0]), std::move(sepfile[1]) , std::move(sepfile[2]), Forword_Index_.size()); // if(count == 764) // { // Log(LogModule::DEBUG) << "Index Url: " << ft.url_; // } Forword_Index_.push_back(std::move(ft)); return true; } bool BuildInvertedIndex(size_t findex) { Forword_t ft = Forword_Index_[findex]; std::unordered_map map_s; titlesegmentation.clear(); ns_util::JiebaUtile::CutPhrase(ft.title_, titlesegmentation); for(auto& s : titlesegmentation) { boost::to_lower(s); map_s[s].title_cnt_++; } contentsegmentation.clear(); ns_util::JiebaUtile::CutPhrase(ft.content_, contentsegmentation); for(auto& s : contentsegmentation) { boost::to_lower(s); map_s[s].content_cnt_++; //cout << s << "--"; // if(strcmp(s.c_str(), "people") == 0) // { // Log(LogModule::DEBUG) << "意外的people！"; // cout << ft.content_ << "------------end!"; // sleep(100); // } } const int X = 10; const int Y = 1; for(auto& p : map_s) { Inverted_t it; it.Set(findex, p.first , p.second.title_cnt_ * X + p.second.content_cnt_ * Y); InvertedList& list = Inverted_Index_[p.first]; list.push_back(std::move(it)); } return true; } private: std::vector Forword_Index_; std::unordered_map Inverted_Index_; bool isInit_ = false; //内存复用，优化时间 std::vector sepfile; std::vector titlesegmentation; std::vector contentsegmentation; }; }; ``` ### Log.hpp ```cpp #ifndef __LOG_HPP__ #define __LOG_HPP__ #include #include #include #include #include #include #include #include #include #include namespace LogModule { const std::string default_path = "./log/"; const std::string default_file = "log.txt"; enum LogLevel { DEBUG, INFO, WARNING, ERROR, FATAL }; static std::string LogLevelToString(LogLevel level) { switch (level) { case DEBUG: return "DEBUG"; case INFO: return "INFO"; case WARNING: return "WARNING"; case ERROR: return "ERROR"; case FATAL: return "FATAL"; default: return "UNKNOWN"; } } static std::string GetCurrentTime() { std::time_t time = std::time(nullptr); struct tm stm; localtime_r(&time, &stm); char buff[128]; snprintf(buff, sizeof(buff), "%4d-%02d-%02d-%02d-%02d-%02d", stm.tm_year + 1900, stm.tm_mon + 1, stm.tm_mday, stm.tm_hour, stm.tm_min, stm.tm_sec); return buff; } class Logstrategy { public: virtual ~Logstrategy() = default; virtual void syncLog(std::string &message) = 0; }; class ConsoleLogstrategy : public Logstrategy { public: void syncLog(std::string &message) override { std::cerr << message << std::endl; } ~ConsoleLogstrategy() override { } }; class FileLogstrategy : public Logstrategy { public: FileLogstrategy(std::string filepath = default_path, std::string filename = default_file) { _mutex.lock(); _filepath = filepath; _filename = filename; if (std::filesystem::exists(filepath)) // 检测目录是否存在，存在则返回 { _mutex.unlock(); return; } try { // 不存在则递归创建（复数）目录 std::filesystem::create_directories(filepath); } catch (const std::filesystem::filesystem_error &e) { // 捕获异常并打印 std::cerr << e.what() << '\n'; } _mutex.unlock(); } void syncLog(std::string &message) override { _mutex.lock(); std::string path = _filepath.back() == '/' ? _filepath + _filename : _filepath + "/" + _filename; std::ofstream out(path, std::ios::app); if (!out.is_open()) { _mutex.unlock(); std::cerr << "file open fail!" << '\n'; return; } out << message << '\n'; _mutex.unlock(); out.close(); } ~FileLogstrategy() { } private: std::string _filepath; std::string _filename; std::mutex _mutex; }; class Log { public: Log() { _logstrategy = std::make_unique(); } void useconsolestrategy() { _logstrategy = std::make_unique(); printf("转换控制台策略！\n"); } void usefilestrategy() { _logstrategy = std::make_unique(); printf("转换文件策略！\n"); } class LogMessage { public: LogMessage(LogLevel level, std::string file, int line, Log &log) : _loglevel(level) , _time(GetCurrentTime()) , _file(file), _pid(getpid()) , _line(line), _log(log) { std::stringstream ss; ss << "[" << _time << "] " << "[" << LogLevelToString(_loglevel) << "] " << "[" << _pid << "] " << "[" << _file << "] " << "[" << _line << "] " << "- "; _loginfo = ss.str(); } template LogMessage &operator<<(const T &t) { std::stringstream ss; ss << _loginfo << t; _loginfo = ss.str(); //printf("重载<syncLog(_loginfo); } } private: LogLevel _loglevel; std::string _time; pid_t _pid; std::string _file; int _line; std::string _loginfo; Log &_log; }; LogMessage operator()(LogLevel level, std::string filename, int line) { return LogMessage(level, filename, line, *this); } ~Log() { } private: std::unique_ptr _logstrategy; }; static Log logger; #define Log(type) logger(type, __FILE__, __LINE__) #define ENABLE_LOG_CONSOLE_STRATEGY() logger.useconsolestrategy() #define ENABLE_LOG_FILE_STRATEGY() logger.usefilestrategy() } #endif ``` ### main.cc ```cpp #include "Log.hpp" #include "common.h" #include "Parser.h" #include "Search.hpp" #include "httplib.h" #include #include #include const bool INIT = false; int main() { if(INIT) { Parser parser(Orignaldir, Tragetfile); parser.Init(); } ns_search::Search search; httplib::Server svr; svr.set_base_dir(Basewwwroot); svr.Get("/s", [&](const httplib::Request& req, httplib::Response& rep){ std::string param = "word"; std::string word; std::string out; out.clear(); if(req.has_param(param)) { word = req.get_param_value(param); Log(LogModule::DEBUG) << "查找关键词：" << word; } // rep.set_content("Search: " + word, "text/plain"); bool b = search.SearchBy(word, out); if(b) rep.set_content(out, "application/json"); else Log(DEBUG) << "查找失败"; }); svr.listen("0.0.0.0", 8080); return 0; } ``` ### makefile ```cpp # 编译器设置 CXX := g++ CXXFLAGS := -std=c++17 LDFLAGS := LIBS := -lboost_filesystem -lboost_system -ljsoncpp # 目录设置 SRC_DIR := . BUILD_DIR := build TARGET := main # 自动查找源文件 SRCS := $(wildcard $(SRC_DIR)/*.cc) OBJS := $(SRCS:$(SRC_DIR)/%.cc=$(BUILD_DIR)/%.o) DEPS := $(OBJS:.o=.d) # 确保头文件依赖被包含 -include $(DEPS) # 默认目标 all: $(BUILD_DIR) $(TARGET) # 创建构建目录 $(BUILD_DIR): @mkdir -p $(BUILD_DIR) # 链接目标文件生成可执行文件 $(TARGET): $(OBJS) $(CXX) $(OBJS) -o $@ $(LDFLAGS) $(LIBS) @echo "✅ 构建完成: $(TARGET)" # 编译每个.cc文件为.o文件 $(BUILD_DIR)/%.o: $(SRC_DIR)/%.cc $(CXX) $(CXXFLAGS) -MMD -MP -c $< -o $@ # 清理构建文件 clean: rm -rf $(BUILD_DIR) $(TARGET) @echo "🧹 清理完成" # 重新构建 rebuild: clean all # 显示项目信息 info: @echo "📁 源文件: $(SRCS)" @echo "📦 目标文件: $(OBJS)" @echo "🎯 最终目标: $(TARGET)" # 伪目标 .PHONY: all clean rebuild info # 防止与同名文件冲突 .PRECIOUS: $(OBJS) ``` ### Parser.cc ```cpp #include "Parser.h" #include "Log.hpp" #include "Util.hpp" #include "common.h" #include #include #include #include Parser::Parser(fs::path Datap, fs::path Targetp) { Orignalpath_ = Datap; Targetpath_ = Targetp; } // 初始化：录入html路径------解析html数据------分割写入Data------记录Url bool Parser::Init() { if(!LoadHtmlPath()) { Log(LogModule::DEBUG) << "LoadHtmlPath fail!"; return false; } if(!ParseHtml()) { Log(LogModule::DEBUG) << "ParseHtml fail!"; return false; } if(!WriteToTarget()) { Log(LogModule::DEBUG) << "WriteToTarget fail!"; return false; } return true; } bool Parser::LoadHtmlPath() { if(!fs::exists(Orignalpath_) || !fs::is_directory(Orignalpath_)) { Log(LogModule::DEBUG) << "Orignalpath is not exists or invalid!"; return false; } fs::recursive_directory_iterator end_it; fs::recursive_directory_iterator it(Orignalpath_); for(; it != end_it; it++) { if(!it->is_regular_file()) { continue; } if(it->path().extension() != ".html") { continue; } htmlpaths_.push_back(it->path()); //Log(DEBUG) << "path: " << it->path(); } Log(LogModule::DEBUG) << "Found " << htmlpaths_.size() << " HTML files"; return true; } bool Parser::ParseHtml() { if(htmlpaths_.empty()) { Log(LogModule::DEBUG) << "paths is empty!"; return false; } size_t successCount = 0; for(fs::path &p : htmlpaths_) { // 检查路径是否存在 if (!fs::exists(p)) { Log(LogModule::ERROR) << "File not exists: " << p.string(); continue; } std::string out; HtmlInfo_t info; // 读取文件并记录错误 if(!ns_util::FileUtil::ReadFile(p.string(), &out)) { Log(LogModule::ERROR) << "Failed to read file: " << p.string(); continue; } // 解析标题并记录错误 if(!ParseTitle(out, &info.title_)) { Log(LogModule::ERROR) << "Failed to parse title from: " << p.string(); continue; } // 解析内容并记录错误 if(!ParseContent(out, &info.content_)) { Log(LogModule::ERROR) << "Failed to parse content from: " << p.string(); continue; } // 检查URL解析结果 if(!ParseUrl(p, &info.url_)) { Log(LogModule::ERROR) << "Failed to parse URL from: " << p.string(); continue; } htmlinfos_.push_back(std::move(info)); successCount++; } // 可以根据需要判断是否全部成功或部分成功 Log(LogModule::INFO) << "Parse HTML completed. Success: " << successCount << ", Total: " << htmlpaths_.size(); return successCount > 0; } bool Parser::WriteToTarget() { if(htmlinfos_.empty()) { Log(LogModule::DEBUG) << "infos empty!"; return false; } for(HtmlInfo_t &info : htmlinfos_) { output_ += info.title_; output_ += Output_sep; output_ += info.content_; output_ += Output_sep; output_ += info.url_; output_ += Line_sep; } WriteToTargetFile(); return true; } bool Parser::ParseUrl(fs::path p, std::string *out) { fs::path head(Boost_Url_Head); head = head / p.string().substr(Orignaldir.size()); *out = head.string(); //Log(LogModule::DEBUG) << "filename: " << p.filename(); return true; } bool Parser::ParseTitle(std::string& fdata, std::string* title) { if(fdata.empty() || title == nullptr) { Log(LogModule::DEBUG) << "parameter invalid!"; return false; } size_t begin = fdata.find(""); if(begin == std::string::npos || end == std::string::npos) { Log(LogModule::DEBUG) << "title find fail!"; return false; } begin += std::string("