DAY42 SQLite3 : Dictionary Import and Data Query Implementation with C Language

SQLite3 : Dictionary Import and Data Query Implementation with C Language

1. Basic Cognition of SQLite3

1.1 Database Classification (By Scale)

Relational databases are the mainstream for current structured data storage, and can be divided into three categories according to deployment scale and applicable scenarios:

Scale	Representative Products	Applicable Scenarios
Large-Scale	ORACLE	Enterprise-level core businesses, massive data storage, and high concurrency (e.g., finance, e-commerce)
Medium-Scale	MySQL, MSSQL	Small and medium-sized websites, background management systems (e.g., enterprise OA, e-commerce backends)
Small-Scale/Embedded	SQLite3, DB2, PowDB	Embedded devices, local small-scale applications (e.g., microcontrollers, mobile local cache, desktop tools)

1.2 Core Noun Explanation

• DB (Database) : A container for storing data (e.g., 123.db), which is essentially one or a set of files used to organize and store structured data.
• DBMS (Database Management System): Software for operating and managing databases (e.g., SQLite3, MySQL), providing functions such as adding, deleting, modifying, querying, backing up, and restoring data.
• MIS (Management Information System): A business management platform built based on databases (e.g., student score management system, inventory management system).
• OA (Office Automation): A system that digitizes daily office processes relying on databases (e.g., attendance, approval, document management).

1.3 Core Features of SQLite3 (First Choice for Embedded Systems)

1. Open-Source and Free: Complies with the GNU protocol, developed in C language, with traceable and customizable source code.
2. Lightweight and Compact: The core code is only about 10,000 lines, with a total size of less than 10M, occupying minimal system resources.
3. Green and No Installation Required: No need to deploy a server or perform configuration; directly operate local database files.
4. File-Based Database : The entire database corresponds to a local .db file, which can be directly copied, moved, and has strong cross-platform compatibility.
5. Large Capacity Support: The maximum capacity of a single database file can reach 2TB, meeting the storage needs of most small-scale applications and embedded scenarios.
6. Compatible with Standard SQL: Supports most standard SQL statements of relational databases, with low learning costs and convenient migration.

1.4 Installation and Compilation of SQLite3 (Ubuntu Environment)

(1) Online Installation

# Install SQLite3 client (for command-line operations)
sudo apt-get install sqlite3

# Install SQLite3 development dependency library (necessary for C language programming)
sudo apt-get install libsqlite3-dev

(2) Verify Installation

# Check SQLite3 version
sqlite3 --version

# View help documentation
sqlite3 --help

(3) Compilation of C Language Programs

SQLite3 programming requires linking a dedicated library, with the following compilation command format:

gcc your_program.c -o executable_file -lsqlite3
# If the program involves thread operations, additional linking of the pthread library is required
gcc your_program.c -o executable_file -lsqlite3 -lpthread

2. SQLite3 Interactive Command-Line Operation

2.1 Core Terminal Commands (Starting with ".")

SQLite3 command-line operations are divided into "system maintenance commands" (starting with ".") and "standard SQL statements" (ending with ";"). The core system maintenance commands are as follows:

Command	Function Description
.database	View the local file path associated with the current database
.tables	List all data tables in the current database
.schema [table_name]	Display the CREATE TABLE statement of the specified table (display all table structures if no table name is specified)
.q/.quit/.exit	Exit the SQLite3 interactive environment
.header on	Enable header display (display field names in query results)
.dump [table_name]	Export the structure and data of the specified table (or the entire database) to the terminal

(1) Opening and Creating a Database

In SQLite3, opening a non-existent database file will automatically create the database:

# Format: sqlite3 database_file_name.db
sqlite3 123.db

After execution, you will enter the interactive prompt sqlite>, indicating that the database connection is successful. At this point, the 123.db file will be generated in the current directory.

(2) Database Backup and Restoration

# Export the database: fully export 123.db to the dict_backup.sql script file
sqlite3 123.db .dump > dict_backup.sql

# Import the database: import the table structure and data in dict_backup.sql into new_123.db
sqlite3 new_123.db < dict_backup.sql

2.2 Practical Standard SQL Statements (CRUD)

All SQL statements must end with ;. Otherwise, you will enter the line continuation prompt ...>, and you can execute the statement by supplementing ;. The core operations are as follows:

(1) Create Table (DDL Statement)

-- Basic table creation: define fields and types
create table user(id int, name char, age int);

-- Advanced table creation: auto-increment primary key + non-null constraint (recommended)
create table if not exists dict(
    id integer primary key autoincrement,  -- Auto-increment primary key, no need to specify manually
    word text not null,                    -- Word field, non-null constraint
    explain text not null                  -- Explanation field, non-null constraint
);

SQLite3 supports core data types: int (integer), text (string), real (floating-point), blob (binary data).

(2) Drop Table (DDL Statement)

-- Format: drop table table_name;
drop table user;

-- Safe deletion: first check if the table exists (to avoid errors)
drop table if exists user;

(3) Insert Data (DML Statement)

-- Method 1: Insert specified fields
insert into user (id, name, age) values (1, 'Zhang San', 20);

-- Method 2: Insert all fields in the order of table creation
insert into user values (2, 'Li Si', 22);

-- Method 3: Insert into an auto-increment primary key table (omit the id field)
insert into dict (word, explain) values ('apple', 'n. Apple');

(4) Query Data (DQL Statement)

-- Method 1: Query all fields of all records
select * from user;

-- Method 2: Query specified fields
select name, age from user;

-- Method 3: Conditional query (WHERE clause)
select * from user where age > 20 and age < 30;

-- Method 4: Fuzzy query (wildcards)
select * from user where name like 'Zhang_';  -- "_" matches a single arbitrary character
select * from user where name like 'Zhang%';  -- "%" matches multiple arbitrary characters

-- Method 5: Sorted query (ORDER BY)
select * from user where age > 18 order by age desc;  -- "desc" for descending order, "asc" for ascending order (default)

(5) Update Data (DML Statement)

-- Format: update table_name set field1=value1, field2=value2 where condition;
update user set age = 23 where name = 'Li Si';
update dict set explain = 'n. Apple; apple tree' where word = 'apple';

(6) Delete Data (DML Statement)

-- Method 1: Delete data that meets the specified conditions (recommended to avoid accidental deletion of the entire table)
delete from user where id = 1;
delete from dict where word = 'apple';

-- Method 2: Delete all data in the table (use with caution, irreversible)
delete from user;

3. Core SQLite3 C Language Programming Interfaces and Callback Functions

3.1 Three Core Functions

The core framework of SQLite3 C language programming is "Open Database → Execute SQL Statements → Close Database", corresponding to three core functions:

(1) Open/Create Database: sqlite3_open

int sqlite3_open(char *path, sqlite3 **db);

• Function: Open the database file at the specified path; create the file automatically if it does not exist.
• Parameters :
  • path: Path to the database file (e.g., "./123.db").
  • db: Pointer to the database handle, on which all subsequent database operations depend.
• Return Value : Returns SQLITE_OK (0) on success, and a non-zero error code on failure.
• Error Handling : Obtain detailed error information through sqlite3_errmsg(db).

(2) Execute SQL Statements: sqlite3_exec

int sqlite3_exec(sqlite3 *db, char *sql, sqlite3_callback fun, void *arg, char **errmsg);

• Function: Execute any standard SQL statement (supports table creation, insertion, deletion, update, and query).
• Parameters :
  • db: The opened database handle.
  • sql: The SQL statement to be executed (ending with ";").
  • fun: Callback function (only required when executing query statements to receive query results; pass NULL for non-query statements).
  • arg: Custom parameter passed to the callback function; pass NULL if there are no parameters.
  • errmsg: Stores execution error information, which is NULL on success and needs to be manually freed.
• Return Value : Returns SQLITE_OK (0) on success, and a non-zero error code on failure.

(3) Close Database: sqlite3_close

int sqlite3_close(sqlite3 *db);

• Function: Close the opened database and release the database handle and system resources.
• Parameter : db: The opened database handle.
• Return Value : Returns SQLITE_OK (0) on success, and a non-zero error code on failure.
• Note : Before closing, the memory pointed to by errmsg must be released (sqlite3_free(errmsg)) to avoid memory leaks.

3.2 Usage of Callback Functions (Receiving Query Results)

When using sqlite3_exec to execute query statements (select), a callback function is required to receive query results. The core characteristics and examples are as follows:

(1) Core Rules of Callback Functions

1. The callback function will be called multiple times by sqlite3_exec; the number of calls is equal to the number of records in the query result.
2. The return value of the callback function must be 0; otherwise, the query will be terminated, and only the currently obtained records will be returned.
3. The parameters of the callback function are automatically populated by sqlite3_exec and do not need to be assigned manually.

(2) Callback Function Example (Print Query Results)

#include <sqlite3.h>
#include <stdio.h>

// Callback function: receive and print query results
int show(void* arg, int col, char** result, char** title)
{
    // Static variable: ensure the header is printed only once
    static int flag = 0;
    int i = 0;
    
    // First call: print the header (field names)
    if (0 == flag)
    {
        flag = 1;
        for (i = 0; i < col; i++)
        {
            printf("%s\t\t", title[i]);  // title[i]: the field name of the i-th column
        }
        printf("\n");
    }
    
    // Print all field values of the current record
    for (i = 0; i < col; i++)
    {
        printf("%s\t\t", result[i]);  // result[i]: the field value of the i-th column
    }
    printf("\n");
    
    // Must return 0, otherwise the query will be terminated
    return 0;
}

(3) Example of Executing Query Statements

int main()
{
    sqlite3* db = NULL;
    int ret = sqlite3_open("123.db", &db);
    if (ret != SQLITE_OK)
    {
        printf("open db failed:%s\n", sqlite3_errmsg(db));
        sqlite3_close(db);
        return -1;
    }
    
    char* errMsg = NULL;
    char sql_cmd[512] = "select * from user;";
    // Execute the query statement and specify the callback function show
    ret = sqlite3_exec(db, sql_cmd, show, NULL, &errMsg);
    if (ret != SQLITE_OK)
    {
        printf("exec sql failed:%s\n", errMsg);
        sqlite3_free(errMsg);  // Release the memory of error information
        sqlite3_close(db);
        return -1;
    }
    
    sqlite3_close(db);
    return 0;
}

4. Practical Project 1: Batch Import of dict.txt Dictionary File

4.1 Project Requirements

Batch import the dictionary data in /home/linux/dict.txt (each line format: word explanation) into the dict table of 123.db, solving problems such as errors caused by special characters and incomplete data segmentation.

4.2 Core Difficulties and Solutions

Core Difficulty	Solution
SQL syntax errors caused by special characters (e.g., single quotes)	Implement the sql_escape function to convert single quotes ' to ''
Multiple spaces/tabs separating words and explanations	Use strtok with spaces/tabs as delimiters to split words and explanations
Primary key conflicts caused by multiple runs	Set id in the table as an auto-increment primary key (AUTOINCREMENT)
Memory leaks	Timely release errmsg when execution fails, and close files and databases after operations are completed

4.3 Complete Practical Code

#include <sqlite3.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define MAX_LINE_LEN 2048  // Adapt to the maximum length of a single line in the dictionary (including long explanations)
#define MAX_SQL_LEN 4096   // Adapt to the spliced long SQL statement
#define DICT_PATH "/home/linux/dict.txt"  // Path to the dictionary file

// SQL special character escaping: convert ' to '' to avoid SQL syntax errors
void sql_escape(char *dest, const char *src, int dest_len) {
    int i = 0, j = 0;
    while (src[i] != '\0' && j < dest_len - 2) {
        if (src[i] == '\'') {
            dest[j++] = '\'';
            dest[j++] = '\'';
        } else {
            dest[j++] = src[i];
        }
        i++;
    }
    dest[j] = '\0';
}

int main(int argc, char** argv) {
    sqlite3* db = NULL;
    char* errmsg = NULL;
    int ret;

    // 1. Open/create database 123.db
    ret = sqlite3_open("123.db", &db);
    if (ret != SQLITE_OK) {
        fprintf(stderr, "Error: Failed to open database - %s\n", sqlite3_errmsg(db));
        sqlite3_close(db);
        return 1;
    }
    printf("Success: Opened/created database 123.db\n");

    // 2. Create the dict table (auto-increment primary key to avoid duplicate insertion conflicts)
    const char sql_create[] = "CREATE TABLE IF NOT EXISTS dict ("
                              "id INTEGER PRIMARY KEY AUTOINCREMENT,"
                              "word TEXT NOT NULL,"
                              "explain TEXT NOT NULL);";
    ret = sqlite3_exec(db, sql_create, NULL, NULL, &errmsg);
    if (ret != SQLITE_OK) {
        fprintf(stderr, "Error: Failed to create table - %s\n", errmsg);
        sqlite3_free(errmsg);
        sqlite3_close(db);
        return 1;
    }
    printf("Success: Created/verified dict table\n");
    sqlite3_free(errmsg);
    errmsg = NULL;

    // 3. Open the dict.txt file (read-only mode)
    FILE *fp = fopen(DICT_PATH, "r");
    if (fp == NULL) {
        perror("Error: Failed to open dict.txt");
        fprintf(stderr, "Please confirm that the path: %s is correct\n", DICT_PATH);
        sqlite3_close(db);
        return 1;
    }
    printf("Success: Opened dict.txt file, starting to read data\n");

    // 4. Read the dictionary file line by line and batch insert into the database
    char line_buf[MAX_LINE_LEN];
    char sql_insert[MAX_SQL_LEN];
    char word[256], explain[MAX_LINE_LEN];
    char escape_word[512], escape_explain[MAX_LINE_LEN * 2];

    while (fgets(line_buf, MAX_LINE_LEN, fp) != NULL) {
        // Remove newline and carriage return characters at the end of the line to avoid inserting invalid characters
        line_buf[strcspn(line_buf, "\n\r")] = '\0';
        if (strlen(line_buf) == 0) {
            continue;  // Skip empty lines
        }

        // Split word and explanation (supports separation by spaces and tabs)
        char *token = strtok(line_buf, " \t");
        if (token == NULL) {
            fprintf(stderr, "Warning: Invalid line (skipped) - %s\n", line_buf);
            continue;
        }
        strncpy(word, token, sizeof(word) - 1);
        word[sizeof(word) - 1] = '\0';

        // Extract the remaining part as the explanation and remove extra spaces at the beginning
        token = strtok(NULL, "");
        if (token == NULL) {
            fprintf(stderr, "Warning: No explanation (skipped) - Word: %s\n", word);
            continue;
        }
        while (*token == ' ' || *token == '\t') token++;
        strncpy(explain, token, sizeof(explain) - 1);
        explain[sizeof(explain) - 1] = '\0';

        // Escape special characters to avoid SQL syntax errors
        sql_escape(escape_word, word, sizeof(escape_word));
        sql_escape(escape_explain, explain, sizeof(escape_explain));

        // Splice the SQL insert statement
        snprintf(sql_insert, MAX_SQL_LEN,
                 "INSERT INTO dict (word, explain) VALUES ('%s', '%s');",
                 escape_word, escape_explain);

        // Execute the insert operation
        ret = sqlite3_exec(db, sql_insert, NULL, NULL, &errmsg);
        if (ret != SQLITE_OK) {
            fprintf(stderr, "Error: Failed to insert - Word: %s | Error message: %s\n", word, errmsg);
            sqlite3_free(errmsg);
            errmsg = NULL;
            continue;
        }
        printf("Success: Inserted - Word: %s | Explanation: %s\n", word, explain);
    }

    // 5. Release resources
    fclose(fp);
    sqlite3_free(errmsg);
    sqlite3_close(db);

    printf("\nAll operations completed: Data in dict.txt has been inserted into the dict table of 123.db\n");
    return 0;
}

4.4 Execution Steps and Verification

(1) Compile the Code

gcc dict2db.c -o dict2db -lsqlite3 -lpthread

(2) Run the Program

./dict2db

(3) Verify the Insertion Result

# Open the 123.db database
sqlite3 123.db

# View the structure of the dict table
.schema dict

# Query the first 10 dictionary records
select * from dict limit 10;

# Exit the database
.q

5. Practical Project 2: User Information Query and Login Verification

5.1 Project Requirements

Based on the user table of 123.db, implement user information query and login verification, and mark whether the user exists through a callback function.

5.2 Complete Practical Code

#include <sqlite3.h>
#include <stdio.h>
#include <string.h>

// Callback function: mark whether the user exists (pass the flag variable through arg)
int show(void* arg,int col,char** result,char** title)
{
    // Convert arg to an int pointer and modify flag to 1 (indicating the user is found)
    *(int*)arg = 1;
    return 0;
}

int main()
{
    char name[50] = {0};
    char pass[50] = {0};
    
    // Receive user input
    printf("Please enter username: ");
    fgets(name, sizeof(name), stdin);
    // Remove the newline character from the input
    name[strcspn(name, "\n")] = '\0';
    
    printf("Please enter password: ");
    fgets(pass, sizeof(pass), stdin);
    pass[strcspn(pass, "\n")] = '\0';

    // Open the database
    sqlite3* db = NULL;
    int ret = sqlite3_open("123.db",&db);
    if(ret != SQLITE_OK)
    {
        printf("open db failed:%s\n", sqlite3_errmsg(db));
        sqlite3_close(db);
        return -1;
    }

    // Splice the query SQL statement
    char* errMsg = NULL;
    int flag = 0;  // Mark whether the user exists: 0 for non-existent, 1 for existent
    char sql_cmd[512] = {0};
    sprintf(sql_cmd, "select * from user where name = '%s' and pass = '%s'", name, pass);

    // Execute the query and modify flag through the callback function
    ret = sqlite3_exec(db,sql_cmd,show,&flag,&errMsg);
    if(ret != SQLITE_OK)
    {
        printf("exec sql failed:%s\n", errMsg);
        sqlite3_free(errMsg);
        sqlite3_close(db);
        return -1;
    }

    // Output the verification result
    if(flag == 0)
    {
        printf("User not found: %s\n", name);
    }
    else
    {
        printf("User verification passed: %s\n", name);
    }

    // Close the database
    sqlite3_close(db);
    return 0;
}

5.3 Key Instructions

1. Use void* arg to pass the custom variable flag to realize data interaction between the callback function and the main function.
2. Remove the newline character during input processing to avoid query failures caused by the newline character read by fgets.
3. If there is no pass field in the user table, modify the SQL statement and table structure to adapt to actual requirements.

6. Common Problems and Pitfall Avoidance Guide

1. no such column: xxx : String values in SQL statements are not enclosed in single quotes, or special characters are not escaped. Enclose strings in 'xxx' and handle single quotes through sql_escape.
2. double free detected in tcache 2 : errmsg is freed repeatedly. Free errmsg only when sqlite3_exec execution fails, and reset it to NULL after freeing to avoid repeated freeing later.
3. no such table: xxx : Data is inserted directly without creating a table. First execute the CREATE TABLE statement, and it is recommended to use IF NOT EXISTS to avoid errors caused by repeated table creation.
4. Primary key conflict : Duplication occurs when manually specifying id for insertion. Set id as an auto-increment primary key (AUTOINCREMENT) and omit the id field during insertion.
5. Failed to open dict.txt : Incorrect file path or lack of read permission. Confirm the correctness of the path and execute chmod +r file_name to grant read permission.