【源码阅读】 Golang中的database/sql库源码探究

Note:文章待完结

文章目录

前言

在golang中,我们比较熟悉的mysql相关的库就是database/sql,这是golang的内置库,该标准库没有具体实现,只列出第三方库需要实现的具体内容。也就是说,这个库只是定义了接口,并没有具体的实现。Go语言为开发数据库驱动定义了一些标准接口,使用标准接口开发的代码,在迁移数据库时,不需要做任何修改(当然双方数据库都遵守标准接口)。下面我将基于golang1.19的源码探究这个库的实现。

源码地址:https://github.com/golang/go/tree/release-branch.go1.19/src/database/sql

一、整体目录结构

整个目录结构就是这样,包含两个包:sql和driver,这两个包必须一起配合着使用,sql包中主要包含着数据库具体实例、驱动的注册、结果集读取、转换各种定义类型结构等。driver包中主要是与数据库打交道的部分,增删改查的接口定义就在这里面。
sql包:

driver包:

二、driver包

在driver包中,主要有如下的接口定义:

  • Connector:抽象的数据库连接器,需要具备创建数据库连接以及返回从属的数据库驱动的能力。
  • Driver:抽象的数据库驱动,具备创建数据库连接的能力。
  • Conn:抽象的数据库连接,具备预处理 sql 以及开启事务的能力。
  • Tx:抽象的事务,具备提交和回滚的能力。
  • Statement:抽象的请求预处理状态. 具备实际执行 sql 并返回执行结果的能力。
  • Result/Row:抽象的 sql 执行结果。

1、驱动相关driver.Driver

Driver是一个数据库驱动的接口,定义了 Open(name string) ,该方法返回一个数据库的Conn接口:

go 复制代码
// Driver is the interface that must be implemented by a database
// driver.
//
// Database drivers may implement DriverContext for access
// to contexts and to parse the name only once for a pool of connections,
// instead of once per connection.
type Driver interface {
	// Open returns a new connection to the database.
	// The name is a string in a driver-specific format.
	//
	// Open may return a cached connection (one previously
	// closed), but doing so is unnecessary; the sql package
	// maintains a pool of idle connections for efficient re-use.
	//
	// The returned connection is only used by one goroutine at a
	// time.
	Open(name string) (Conn, error)
}

在上面的源码中,我们可以清晰知道,Driver接口是必须要被所有的数据库驱动程序实现的,提供而一个Open方法用于返回一个连接,这个连接可能是缓存的有效的,也可能是新建的连接。同时也提供了一个DriverContext接口,数据库驱动程序可以实现DriverContext以访问上下文,并仅为连接池解析一次名称,而不是每个连接解析一次。

DriverContext接口提供了一个OpenConnector方法用于返回一个连接器,在连接器中去获取对应的连接。连接器接口Connector提供了两个方法,Connect和Driver,其中Connect用于获取连接,并且可以附带参数ctx,Driver用于获取当前这个连接器的的驱动程序。

go 复制代码
// If a Driver implements DriverContext, then sql.DB will call
// OpenConnector to obtain a Connector and then invoke
// that Connector's Connect method to obtain each needed connection,
// instead of invoking the Driver's Open method for each connection.
// The two-step sequence allows drivers to parse the name just once
// and also provides access to per-Conn contexts.
type DriverContext interface {
	// OpenConnector must parse the name in the same format that Driver.Open
	// parses the name parameter.
	OpenConnector(name string) (Connector, error)
}

// A Connector represents a driver in a fixed configuration
// and can create any number of equivalent Conns for use
// by multiple goroutines.
//
// A Connector can be passed to sql.OpenDB, to allow drivers
// to implement their own sql.DB constructors, or returned by
// DriverContext's OpenConnector method, to allow drivers
// access to context and to avoid repeated parsing of driver
// configuration.
//
// If a Connector implements io.Closer, the sql package's DB.Close
// method will call Close and return error (if any).
type Connector interface {
	// Connect returns a connection to the database.
	// Connect may return a cached connection (one previously
	// closed), but doing so is unnecessary; the sql package
	// maintains a pool of idle connections for efficient re-use.
	//
	// The provided context.Context is for dialing purposes only
	// (see net.DialContext) and should not be stored or used for
	// other purposes. A default timeout should still be used
	// when dialing as a connection pool may call Connect
	// asynchronously to any query.
	//
	// The returned connection is only used by one goroutine at a
	// time.
	Connect(context.Context) (Conn, error)

	// Driver returns the underlying Driver of the Connector,
	// mainly to maintain compatibility with the Driver method
	// on sql.DB.
	Driver() Driver
}

接下来就是驱动的注册了,database/sql提供了一个驱动注册静态方法,驱动的具体实现中,可以调用该方法注册相关的驱动,但同时只允许注册同一类型的驱动,否则会panic。

go 复制代码
func Register(name string, driver driver.Driver) {
	driversMu.Lock()
	defer driversMu.Unlock()
	if driver == nil {
		panic("sql: Register driver is nil")
	}
	if _, dup := drivers[name]; dup {
		panic("sql: Register called twice for driver " + name)
	}
	drivers[name] = driver
}

该函数用来注册数据库驱动。当第三方开发者开发数据库驱动时,都会实现init函数,而init中会调用 Register(name string, driver driver.Driver) 完成驱动注册。如<github.com/go-sql-driver/mysql>

2、驱动连接:driver.Conn

go 复制代码
type Conn interface {
	// Prepare returns a prepared statement, bound to this connection.
	Prepare(query string) (Stmt, error)

	// Close invalidates and potentially stops any current
	// prepared statements and transactions, marking this
	// connection as no longer in use.
	//
	// Because the sql package maintains a free pool of
	// connections and only calls Close when there's a surplus of
	// idle connections, it shouldn't be necessary for drivers to
	// do their own connection caching.
	//
	// Drivers must ensure all network calls made by Close
	// do not block indefinitely (e.g. apply a timeout).
	Close() error

	// Begin starts and returns a new transaction.
	//
	// Deprecated: Drivers should implement ConnBeginTx instead (or additionally).
	Begin() (Tx, error)
}

Prepare:返回与当前连接相关的执行SQL语句的准备状态(Stmt),可以进行查询、删除等操作。
Close:关闭当前的链接,执行释放连接拥有的资源等清理工作。
Begin: // 返回一个代表事务处理的Tx,通过它可以进行查询、更新等操作,或者对事务进行回滚、递交。

新版本中,Begin方法已经不推荐了,被ConnBeginTx代替了, 新版本中的Begin方法多提供了入参ctx和额外的可选参数opts,便于扩展和控制。

go 复制代码
// ConnBeginTx enhances the Conn interface with context and TxOptions.
type ConnBeginTx interface {
	// BeginTx starts and returns a new transaction.
	// If the context is canceled by the user the sql package will
	// call Tx.Rollback before discarding and closing the connection.
	//
	// This must check opts.Isolation to determine if there is a set
	// isolation level. If the driver does not support a non-default
	// level and one is set or if there is a non-default isolation level
	// that is not supported, an error must be returned.
	//
	// This must also check opts.ReadOnly to determine if the read-only
	// value is true to either set the read-only transaction property if supported
	// or return an error if it is not supported.
	BeginTx(ctx context.Context, opts TxOptions) (Tx, error)
}

3、预处理结构:Stmt

go 复制代码
// Stmt is a prepared statement. It is bound to a Conn and not
// used by multiple goroutines concurrently.
type Stmt interface {
	// Close closes the statement.
	//
	// As of Go 1.1, a Stmt will not be closed if it's in use
	// by any queries.
	//
	// Drivers must ensure all network calls made by Close
	// do not block indefinitely (e.g. apply a timeout).
	Close() error

	// NumInput returns the number of placeholder parameters.
	//
	// If NumInput returns >= 0, the sql package will sanity check
	// argument counts from callers and return errors to the caller
	// before the statement's Exec or Query methods are called.
	//
	// NumInput may also return -1, if the driver doesn't know
	// its number of placeholders. In that case, the sql package
	// will not sanity check Exec or Query argument counts.
	NumInput() int

	// Exec executes a query that doesn't return rows, such
	// as an INSERT or UPDATE.
	//
	// Deprecated: Drivers should implement StmtExecContext instead (or additionally).
	Exec(args []Value) (Result, error)

	// Query executes a query that may return rows, such as a
	// SELECT.
	//
	// Deprecated: Drivers should implement StmtQueryContext instead (or additionally).
	Query(args []Value) (Rows, error)
}

// StmtExecContext enhances the Stmt interface by providing Exec with context.
type StmtExecContext interface {
	// ExecContext executes a query that doesn't return rows, such
	// as an INSERT or UPDATE.
	//
	// ExecContext must honor the context timeout and return when it is canceled.
	ExecContext(ctx context.Context, args []NamedValue) (Result, error)
}

// StmtQueryContext enhances the Stmt interface by providing Query with context.
type StmtQueryContext interface {
	// QueryContext executes a query that may return rows, such as a
	// SELECT.
	//
	// QueryContext must honor the context timeout and return when it is canceled.
	QueryContext(ctx context.Context, args []NamedValue) (Rows, error)
}

Close:关闭当前的连接状态,但如果当前正在执行query,query还是会有效返回rows数据。
NumInput:返回当前预留参数的个数,当返回>=0时,数据库驱动会智能检查调用者的参数。 当数据库驱动包不知道预留参数的时候,返回-1。
Exec:执行Prepare准备好的SQL,传入参数执行Update/Insert等操作,返回Result数据,Result中包含最后插入的自增主键序号(LastInsertId)和受影响的行数(RowAffected)。
Query:执行Prepare准备好的SQL,传入需要的参数执行select操作,返回Rows结果集。

4、执行结果 driver.Result

go 复制代码
// Result is the result of a query execution.
type Result interface {
	// LastInsertId returns the database's auto-generated ID
	// after, for example, an INSERT into a table with primary
	// key.
	LastInsertId() (int64, error)

	// RowsAffected returns the number of rows affected by the
	// query.
	RowsAffected() (int64, error)
}

5、查询结果:driver.Rows

go 复制代码
// Rows is an iterator over an executed query's results.
type Rows interface {
 
	// 该函数返回查询数据库表的字段信息,这个返回的slice和SQL查询的字段一一对应,
    // 而不是返回整张表的所有字段。
	Columns() []string
 
	// 用来关闭Rows迭代器
	Close() error
 
	// 该函数用来返回下一条数据,把数据赋值给dest .
    // dest里面元素必须是driver.Value的值(string除外),返回的数据里面所有的 string 都必须转换成
    // []byte.如果最后没有数据了,Next 函数返回 io.EOF。
	Next(dest []Value) error
}

可以看到,在新版的源码中,Exec和Query已经被单独拎出去定义了接口,方法中只是为了增加ctx参数,这也是golang为了保持向下兼容而做的,试想,如果直接在原有的接口定义的加入ctx,升级golang版本的时候这块儿肯定得花很大功夫去改造。

6、driver.RowsAffected

RowsAffected 不是别的东西,实际上只是 int64 的别名,但它实现了Result接口,用于底层实现 Result 的表示方式,构建Exec方法返回的结果集。

go 复制代码
// RowsAffected implements Result for an INSERT or UPDATE operation
// which mutates a number of rows.
type RowsAffected int64
 
var _ Result = RowsAffected(0)
 
func (RowsAffected) LastInsertId() (int64, error) {
	return 0, errors.New("LastInsertId is not supported by this driver")
}
 
func (v RowsAffected) RowsAffected() (int64, error) {
	return int64(v), nil
}

7、driver.Value

Value 其实是一个空接口,可以容纳任何的数据。

go 复制代码
// diver 的 Value 是驱动必须能够操作的 Value,Value要么是nil,要么是下面任意一种:
//
//   int64
//   float64
//   bool
//   []byte
//   string   [*] 除了Rows.Next,返回的不能是string
//   time.Time
//
type Value interface{}

8、Value定义转换相关

在driver/types.go中,还定义了ValueConverter将一个普通的值(any)转换成driver.Value的接口、Valuer接口用于获取driver.Value等,就不逐个展开了。

go 复制代码
// ValueConverter is the interface providing the ConvertValue method.
//
// Various implementations of ValueConverter are provided by the
// driver package to provide consistent implementations of conversions
// between drivers. The ValueConverters have several uses:
//
//   - converting from the Value types as provided by the sql package
//     into a database table's specific column type and making sure it
//     fits, such as making sure a particular int64 fits in a
//     table's uint16 column.
//
//   - converting a value as given from the database into one of the
//     driver Value types.
//
//   - by the sql package, for converting from a driver's Value type
//     to a user's type in a scan.
type ValueConverter interface {
	// ConvertValue converts a value to a driver Value.
	ConvertValue(v any) (Value, error)
}

// Valuer is the interface providing the Value method.
//
// Types implementing Valuer interface are able to convert
// themselves to a driver Value.
type Valuer interface {
	// Value returns a driver Value.
	// Value must not panic.
	Value() (Value, error)
}

三、sql包

在sql包中,主要是sql.go中的 DB结构,对应为数据库的具象化实例。DB

go 复制代码
// DB is a database handle representing a pool of zero or more
// underlying connections. It's safe for concurrent use by multiple
// goroutines.
//
// The sql package creates and frees connections automatically; it
// also maintains a free pool of idle connections. If the database has
// a concept of per-connection state, such state can be reliably observed
// within a transaction (Tx) or connection (Conn). Once DB.Begin is called, the
// returned Tx is bound to a single connection. Once Commit or
// Rollback is called on the transaction, that transaction's
// connection is returned to DB's idle connection pool. The pool size
// can be controlled with SetMaxIdleConns.
type DB struct {
	// Atomic access only. At top of struct to prevent mis-alignment
	// on 32-bit platforms. Of type time.Duration.
	waitDuration int64 // Total time waited for new connections.

	connector driver.Connector
	// numClosed is an atomic counter which represents a total number of
	// closed connections. Stmt.openStmt checks it before cleaning closed
	// connections in Stmt.css.
	numClosed uint64

	mu           sync.Mutex    // protects following fields
	freeConn     []*driverConn // free connections ordered by returnedAt oldest to newest
	connRequests map[uint64]chan connRequest
	nextRequest  uint64 // Next key to use in connRequests.
	numOpen      int    // number of opened and pending open connections
	// Used to signal the need for new connections
	// a goroutine running connectionOpener() reads on this chan and
	// maybeOpenNewConnections sends on the chan (one send per needed connection)
	// It is closed during db.Close(). The close tells the connectionOpener
	// goroutine to exit.
	openerCh          chan struct{}
	closed            bool
	dep               map[finalCloser]depSet
	lastPut           map[*driverConn]string // stacktrace of last conn's put; debug only
	maxIdleCount      int                    // zero means defaultMaxIdleConns; negative means 0
	maxOpen           int                    // <= 0 means unlimited
	maxLifetime       time.Duration          // maximum amount of time a connection may be reused
	maxIdleTime       time.Duration          // maximum amount of time a connection may be idle before being closed
	cleanerCh         chan struct{}
	waitCount         int64 // Total number of connections waited for.
	maxIdleClosed     int64 // Total number of connections closed due to idle count.
	maxIdleTimeClosed int64 // Total number of connections closed due to idle time.
	maxLifetimeClosed int64 // Total number of connections closed due to max connection lifetime limit.

	stop func() // stop cancels the connection opener.
}

几个主要的字段,其他字段大部分都是和连接池参数相关的:
connector:用于创建数据库连接的抽象连接器,由第三方数据库提供具体实现。
mu:互斥锁,保证并发安全。
freeConn:数据库连接池,缓存可用的连接以供后续复用。
connRequests:唤醒通道集合,和阻塞等待连接的协程是一对一的关系。
openerCh:创建连接信号通道. 用于向连接创建协程 opener goroutine 发送信号。
stop:连接创建协程 opener goroutine 的终止器,用于停止该协程。

DB结构主要作用如下:

二、结语

本章中我们学习了,golang中database/sql的源码阅读,了解了整个database/sql最大的特点就是定义接口,不做具体实现,从而让使用方去方便使用不同的驱动实现。同时提供了DB实例,内置连接池,方便管理连接的创建和销毁。

三、参考

1、Go database/sql连接池 - 源码学习

2、Golang sql 标准库源码解析

相关推荐
tatasix10 分钟前
MySQL UPDATE语句执行链路解析
数据库·mysql
ifanatic17 分钟前
[面试]-golang基础面试题总结
面试·职场和发展·golang
南城花随雪。22 分钟前
硬盘(HDD)与固态硬盘(SSD)详细解读
数据库
儿时可乖了23 分钟前
使用 Java 操作 SQLite 数据库
java·数据库·sqlite
懒是一种态度25 分钟前
Golang 调用 mongodb 的函数
数据库·mongodb·golang
天海华兮28 分钟前
mysql 去重 补全 取出重复 变量 函数 和存储过程
数据库·mysql
XINGTECODE1 小时前
海盗王集成网关和商城服务端功能golang版
开发语言·后端·golang
入 梦皆星河1 小时前
在 Ubuntu/Debian 上安装 Go
ubuntu·golang·debian
gma9991 小时前
Etcd 框架
数据库·etcd
爱吃青椒不爱吃西红柿‍️1 小时前
华为ASP与CSP是什么?
服务器·前端·数据库