【源码阅读】 Golang中的database/sql库源码探究

童话ing2024-04-29 9:23

Note：文章待完结

文章目录

前言

在golang中，我们比较熟悉的mysql相关的库就是database/sql，这是golang的内置库，该标准库没有具体实现，只列出第三方库需要实现的具体内容。也就是说，这个库只是定义了接口，并没有具体的实现。Go语言为开发数据库驱动定义了一些标准接口，使用标准接口开发的代码，在迁移数据库时，不需要做任何修改（当然双方数据库都遵守标准接口）。下面我将基于golang1.19的源码探究这个库的实现。

源码地址：https://github.com/golang/go/tree/release-branch.go1.19/src/database/sql

一、整体目录结构

整个目录结构就是这样，包含两个包：sql和driver，这两个包必须一起配合着使用，sql包中主要包含着数据库具体实例、驱动的注册、结果集读取、转换各种定义类型结构等。driver包中主要是与数据库打交道的部分，增删改查的接口定义就在这里面。
sql包：

driver包：

二、driver包

在driver包中，主要有如下的接口定义：

  • Connector：抽象的数据库连接器，需要具备创建数据库连接以及返回从属的数据库驱动的能力。
  • Driver：抽象的数据库驱动，具备创建数据库连接的能力。
  • Conn：抽象的数据库连接，具备预处理 sql 以及开启事务的能力。
  • Tx：抽象的事务，具备提交和回滚的能力。
  • Statement：抽象的请求预处理状态. 具备实际执行 sql 并返回执行结果的能力。
  • Result/Row：抽象的 sql 执行结果。

1、驱动相关driver.Driver

Driver是一个数据库驱动的接口，定义了 Open(name string) ,该方法返回一个数据库的Conn接口:

go 复制代码 
// Driver is the interface that must be implemented by a database
// driver.
//
// Database drivers may implement DriverContext for access
// to contexts and to parse the name only once for a pool of connections,
// instead of once per connection.
type Driver interface {
	// Open returns a new connection to the database.
	// The name is a string in a driver-specific format.
	//
	// Open may return a cached connection (one previously
	// closed), but doing so is unnecessary; the sql package
	// maintains a pool of idle connections for efficient re-use.
	//
	// The returned connection is only used by one goroutine at a
	// time.
	Open(name string) (Conn, error)
}

在上面的源码中，我们可以清晰知道，Driver接口是必须要被所有的数据库驱动程序实现的，提供而一个Open方法用于返回一个连接，这个连接可能是缓存的有效的，也可能是新建的连接。同时也提供了一个DriverContext接口，数据库驱动程序可以实现DriverContext以访问上下文，并仅为连接池解析一次名称，而不是每个连接解析一次。

DriverContext接口提供了一个OpenConnector方法用于返回一个连接器，在连接器中去获取对应的连接。连接器接口Connector提供了两个方法，Connect和Driver，其中Connect用于获取连接，并且可以附带参数ctx，Driver用于获取当前这个连接器的的驱动程序。

go 复制代码 
// If a Driver implements DriverContext, then sql.DB will call
// OpenConnector to obtain a Connector and then invoke
// that Connector's Connect method to obtain each needed connection,
// instead of invoking the Driver's Open method for each connection.
// The two-step sequence allows drivers to parse the name just once
// and also provides access to per-Conn contexts.
type DriverContext interface {
	// OpenConnector must parse the name in the same format that Driver.Open
	// parses the name parameter.
	OpenConnector(name string) (Connector, error)
}

// A Connector represents a driver in a fixed configuration
// and can create any number of equivalent Conns for use
// by multiple goroutines.
//
// A Connector can be passed to sql.OpenDB, to allow drivers
// to implement their own sql.DB constructors, or returned by
// DriverContext's OpenConnector method, to allow drivers
// access to context and to avoid repeated parsing of driver
// configuration.
//
// If a Connector implements io.Closer, the sql package's DB.Close
// method will call Close and return error (if any).
type Connector interface {
	// Connect returns a connection to the database.
	// Connect may return a cached connection (one previously
	// closed), but doing so is unnecessary; the sql package
	// maintains a pool of idle connections for efficient re-use.
	//
	// The provided context.Context is for dialing purposes only
	// (see net.DialContext) and should not be stored or used for
	// other purposes. A default timeout should still be used
	// when dialing as a connection pool may call Connect
	// asynchronously to any query.
	//
	// The returned connection is only used by one goroutine at a
	// time.
	Connect(context.Context) (Conn, error)

	// Driver returns the underlying Driver of the Connector,
	// mainly to maintain compatibility with the Driver method
	// on sql.DB.
	Driver() Driver
}

接下来就是驱动的注册了，database/sql提供了一个驱动注册静态方法，驱动的具体实现中，可以调用该方法注册相关的驱动，但同时只允许注册同一类型的驱动，否则会panic。

go 复制代码 
func Register(name string, driver driver.Driver) {
	driversMu.Lock()
	defer driversMu.Unlock()
	if driver == nil {
		panic("sql: Register driver is nil")
	}
	if _, dup := drivers[name]; dup {
		panic("sql: Register called twice for driver " + name)
	}
	drivers[name] = driver
}

该函数用来注册数据库驱动。当第三方开发者开发数据库驱动时，都会实现init函数，而init中会调用 Register(name string, driver driver.Driver) 完成驱动注册。如<github.com/go-sql-driver/mysql>

2、驱动连接：driver.Conn

go 复制代码 
type Conn interface {
	// Prepare returns a prepared statement, bound to this connection.
	Prepare(query string) (Stmt, error)

	// Close invalidates and potentially stops any current
	// prepared statements and transactions, marking this
	// connection as no longer in use.
	//
	// Because the sql package maintains a free pool of
	// connections and only calls Close when there's a surplus of
	// idle connections, it shouldn't be necessary for drivers to
	// do their own connection caching.
	//
	// Drivers must ensure all network calls made by Close
	// do not block indefinitely (e.g. apply a timeout).
	Close() error

	// Begin starts and returns a new transaction.
	//
	// Deprecated: Drivers should implement ConnBeginTx instead (or additionally).
	Begin() (Tx, error)
}

Prepare：返回与当前连接相关的执行SQL语句的准备状态（Stmt），可以进行查询、删除等操作。
Close：关闭当前的链接，执行释放连接拥有的资源等清理工作。
Begin: // 返回一个代表事务处理的Tx,通过它可以进行查询、更新等操作，或者对事务进行回滚、递交。

新版本中，Begin方法已经不推荐了，被ConnBeginTx代替了, 新版本中的Begin方法多提供了入参ctx和额外的可选参数opts，便于扩展和控制。

go 复制代码 
// ConnBeginTx enhances the Conn interface with context and TxOptions.
type ConnBeginTx interface {
	// BeginTx starts and returns a new transaction.
	// If the context is canceled by the user the sql package will
	// call Tx.Rollback before discarding and closing the connection.
	//
	// This must check opts.Isolation to determine if there is a set
	// isolation level. If the driver does not support a non-default
	// level and one is set or if there is a non-default isolation level
	// that is not supported, an error must be returned.
	//
	// This must also check opts.ReadOnly to determine if the read-only
	// value is true to either set the read-only transaction property if supported
	// or return an error if it is not supported.
	BeginTx(ctx context.Context, opts TxOptions) (Tx, error)
}

3、预处理结构：Stmt

go 复制代码 
// Stmt is a prepared statement. It is bound to a Conn and not
// used by multiple goroutines concurrently.
type Stmt interface {
	// Close closes the statement.
	//
	// As of Go 1.1, a Stmt will not be closed if it's in use
	// by any queries.
	//
	// Drivers must ensure all network calls made by Close
	// do not block indefinitely (e.g. apply a timeout).
	Close() error

	// NumInput returns the number of placeholder parameters.
	//
	// If NumInput returns >= 0, the sql package will sanity check
	// argument counts from callers and return errors to the caller
	// before the statement's Exec or Query methods are called.
	//
	// NumInput may also return -1, if the driver doesn't know
	// its number of placeholders. In that case, the sql package
	// will not sanity check Exec or Query argument counts.
	NumInput() int

	// Exec executes a query that doesn't return rows, such
	// as an INSERT or UPDATE.
	//
	// Deprecated: Drivers should implement StmtExecContext instead (or additionally).
	Exec(args []Value) (Result, error)

	// Query executes a query that may return rows, such as a
	// SELECT.
	//
	// Deprecated: Drivers should implement StmtQueryContext instead (or additionally).
	Query(args []Value) (Rows, error)
}

// StmtExecContext enhances the Stmt interface by providing Exec with context.
type StmtExecContext interface {
	// ExecContext executes a query that doesn't return rows, such
	// as an INSERT or UPDATE.
	//
	// ExecContext must honor the context timeout and return when it is canceled.
	ExecContext(ctx context.Context, args []NamedValue) (Result, error)
}

// StmtQueryContext enhances the Stmt interface by providing Query with context.
type StmtQueryContext interface {
	// QueryContext executes a query that may return rows, such as a
	// SELECT.
	//
	// QueryContext must honor the context timeout and return when it is canceled.
	QueryContext(ctx context.Context, args []NamedValue) (Rows, error)
}

Close：关闭当前的连接状态，但如果当前正在执行query,query还是会有效返回rows数据。
NumInput：返回当前预留参数的个数，当返回>=0时，数据库驱动会智能检查调用者的参数。 当数据库驱动包不知道预留参数的时候，返回-1。
Exec：执行Prepare准备好的SQL,传入参数执行Update/Insert等操作，返回Result数据，Result中包含最后插入的自增主键序号（LastInsertId）和受影响的行数（RowAffected）。
Query：执行Prepare准备好的SQL,传入需要的参数执行select操作，返回Rows结果集。

4、执行结果 driver.Result

go 复制代码 
// Result is the result of a query execution.
type Result interface {
	// LastInsertId returns the database's auto-generated ID
	// after, for example, an INSERT into a table with primary
	// key.
	LastInsertId() (int64, error)

	// RowsAffected returns the number of rows affected by the
	// query.
	RowsAffected() (int64, error)
}

5、查询结果：driver.Rows

go 复制代码 
// Rows is an iterator over an executed query's results.
type Rows interface {
 
	// 该函数返回查询数据库表的字段信息，这个返回的slice和SQL查询的字段一一对应，
    // 而不是返回整张表的所有字段。
	Columns() []string
 
	// 用来关闭Rows迭代器
	Close() error
 
	// 该函数用来返回下一条数据，把数据赋值给dest .
    // dest里面元素必须是driver.Value的值（string除外）,返回的数据里面所有的 string 都必须转换成
    // []byte.如果最后没有数据了，Next 函数返回 io.EOF。
	Next(dest []Value) error
}

可以看到，在新版的源码中，Exec和Query已经被单独拎出去定义了接口，方法中只是为了增加ctx参数，这也是golang为了保持向下兼容而做的，试想，如果直接在原有的接口定义的加入ctx，升级golang版本的时候这块儿肯定得花很大功夫去改造。

6、driver.RowsAffected

RowsAffected 不是别的东西，实际上只是 int64 的别名，但它实现了Result接口，用于底层实现 Result 的表示方式，构建Exec方法返回的结果集。

go 复制代码 
// RowsAffected implements Result for an INSERT or UPDATE operation
// which mutates a number of rows.
type RowsAffected int64
 
var _ Result = RowsAffected(0)
 
func (RowsAffected) LastInsertId() (int64, error) {
	return 0, errors.New("LastInsertId is not supported by this driver")
}
 
func (v RowsAffected) RowsAffected() (int64, error) {
	return int64(v), nil
}

7、driver.Value

Value 其实是一个空接口，可以容纳任何的数据。

go 复制代码 
// diver 的 Value 是驱动必须能够操作的 Value,Value要么是nil,要么是下面任意一种:
//
//   int64
//   float64
//   bool
//   []byte
//   string   [*] 除了Rows.Next，返回的不能是string
//   time.Time
//
type Value interface{}

8、Value定义转换相关

在driver/types.go中，还定义了ValueConverter将一个普通的值（any）转换成driver.Value的接口、Valuer接口用于获取driver.Value等，就不逐个展开了。

go 复制代码 
// ValueConverter is the interface providing the ConvertValue method.
//
// Various implementations of ValueConverter are provided by the
// driver package to provide consistent implementations of conversions
// between drivers. The ValueConverters have several uses:
//
//   - converting from the Value types as provided by the sql package
//     into a database table's specific column type and making sure it
//     fits, such as making sure a particular int64 fits in a
//     table's uint16 column.
//
//   - converting a value as given from the database into one of the
//     driver Value types.
//
//   - by the sql package, for converting from a driver's Value type
//     to a user's type in a scan.
type ValueConverter interface {
	// ConvertValue converts a value to a driver Value.
	ConvertValue(v any) (Value, error)
}

// Valuer is the interface providing the Value method.
//
// Types implementing Valuer interface are able to convert
// themselves to a driver Value.
type Valuer interface {
	// Value returns a driver Value.
	// Value must not panic.
	Value() (Value, error)
}

三、sql包

在sql包中，主要是sql.go中的 DB结构，对应为数据库的具象化实例。DB

go 复制代码 
// DB is a database handle representing a pool of zero or more
// underlying connections. It's safe for concurrent use by multiple
// goroutines.
//
// The sql package creates and frees connections automatically; it
// also maintains a free pool of idle connections. If the database has
// a concept of per-connection state, such state can be reliably observed
// within a transaction (Tx) or connection (Conn). Once DB.Begin is called, the
// returned Tx is bound to a single connection. Once Commit or
// Rollback is called on the transaction, that transaction's
// connection is returned to DB's idle connection pool. The pool size
// can be controlled with SetMaxIdleConns.
type DB struct {
	// Atomic access only. At top of struct to prevent mis-alignment
	// on 32-bit platforms. Of type time.Duration.
	waitDuration int64 // Total time waited for new connections.

	connector driver.Connector
	// numClosed is an atomic counter which represents a total number of
	// closed connections. Stmt.openStmt checks it before cleaning closed
	// connections in Stmt.css.
	numClosed uint64

	mu           sync.Mutex    // protects following fields
	freeConn     []*driverConn // free connections ordered by returnedAt oldest to newest
	connRequests map[uint64]chan connRequest
	nextRequest  uint64 // Next key to use in connRequests.
	numOpen      int    // number of opened and pending open connections
	// Used to signal the need for new connections
	// a goroutine running connectionOpener() reads on this chan and
	// maybeOpenNewConnections sends on the chan (one send per needed connection)
	// It is closed during db.Close(). The close tells the connectionOpener
	// goroutine to exit.
	openerCh          chan struct{}
	closed            bool
	dep               map[finalCloser]depSet
	lastPut           map[*driverConn]string // stacktrace of last conn's put; debug only
	maxIdleCount      int                    // zero means defaultMaxIdleConns; negative means 0
	maxOpen           int                    // <= 0 means unlimited
	maxLifetime       time.Duration          // maximum amount of time a connection may be reused
	maxIdleTime       time.Duration          // maximum amount of time a connection may be idle before being closed
	cleanerCh         chan struct{}
	waitCount         int64 // Total number of connections waited for.
	maxIdleClosed     int64 // Total number of connections closed due to idle count.
	maxIdleTimeClosed int64 // Total number of connections closed due to idle time.
	maxLifetimeClosed int64 // Total number of connections closed due to max connection lifetime limit.

	stop func() // stop cancels the connection opener.
}

几个主要的字段，其他字段大部分都是和连接池参数相关的：
connector：用于创建数据库连接的抽象连接器，由第三方数据库提供具体实现。
mu：互斥锁，保证并发安全。
freeConn：数据库连接池，缓存可用的连接以供后续复用。
connRequests：唤醒通道集合，和阻塞等待连接的协程是一对一的关系。
openerCh：创建连接信号通道. 用于向连接创建协程 opener goroutine 发送信号。
stop：连接创建协程 opener goroutine 的终止器，用于停止该协程。

DB结构主要作用如下：

二、结语

本章中我们学习了，golang中database/sql的源码阅读，了解了整个database/sql最大的特点就是定义接口，不做具体实现，从而让使用方去方便使用不同的驱动实现。同时提供了DB实例，内置连接池，方便管理连接的创建和销毁。

三、参考

1、Go database/sql连接池 - 源码学习

2、Golang sql 标准库源码解析

相关推荐
o(╥﹏╥)30 分钟前
linux（ubuntu ）卡死怎么强制重启
linux·数据库·ubuntu·系统安全
阿里嘎多学长44 分钟前
docker怎么部署高斯数据库
运维·数据库·docker·容器
Yuan_o_1 小时前
Linux 基本使用和程序部署
java·linux·运维·服务器·数据库·后端
Sunyanhui11 小时前
牛客网 SQL36查找后排序
数据库·sql·mysql
老王笔记1 小时前
MHA binlog server
数据库·mysql
lovelin+v175030409662 小时前
安全性升级：API接口在零信任架构下的安全防护策略
大数据·数据库·人工智能·爬虫·数据分析
DT辰白3 小时前
基于Redis的网关鉴权方案与性能优化
数据库·redis·缓存
2401_871213303 小时前
mysql高阶语句
数据库·mysql
Mitch3113 小时前
【漏洞复现】CVE-2021-45788 SQL Injection
sql·web安全·docker·prometheus·metersphere
热门推荐
01构建高效的广告投放系统：应用架构的设计与实现02ARM学习（31）编译器对overlay方式的支持03玄机平台应急响应—webshell查杀04Cube-Studio：开源大模型全链路一站式中台05校验 GPT-4 真实性的三个经典问题：快速区分 GPT-3.5 与 GPT-4，并提供免费测试网站06Docker 夺命连环 15 问07〔AI 绘画〕Stable Diffusion 之 解决绘制多人或面部很小的人物时面部崩坏问题 篇08基于大语言模型（LLM）的合成数据生成、策展和评估的综述09(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明10【一文读懂】NTN（非地面网络）技术介绍