【源码阅读】 Golang中的database/sql库源码探究

Note:文章待完结

文章目录

前言

在golang中,我们比较熟悉的mysql相关的库就是database/sql,这是golang的内置库,该标准库没有具体实现,只列出第三方库需要实现的具体内容。也就是说,这个库只是定义了接口,并没有具体的实现。Go语言为开发数据库驱动定义了一些标准接口,使用标准接口开发的代码,在迁移数据库时,不需要做任何修改(当然双方数据库都遵守标准接口)。下面我将基于golang1.19的源码探究这个库的实现。

源码地址:https://github.com/golang/go/tree/release-branch.go1.19/src/database/sql

一、整体目录结构

整个目录结构就是这样,包含两个包:sql和driver,这两个包必须一起配合着使用,sql包中主要包含着数据库具体实例、驱动的注册、结果集读取、转换各种定义类型结构等。driver包中主要是与数据库打交道的部分,增删改查的接口定义就在这里面。
sql包:

driver包:

二、driver包

在driver包中,主要有如下的接口定义:

  • Connector:抽象的数据库连接器,需要具备创建数据库连接以及返回从属的数据库驱动的能力。
  • Driver:抽象的数据库驱动,具备创建数据库连接的能力。
  • Conn:抽象的数据库连接,具备预处理 sql 以及开启事务的能力。
  • Tx:抽象的事务,具备提交和回滚的能力。
  • Statement:抽象的请求预处理状态. 具备实际执行 sql 并返回执行结果的能力。
  • Result/Row:抽象的 sql 执行结果。

1、驱动相关driver.Driver

Driver是一个数据库驱动的接口,定义了 Open(name string) ,该方法返回一个数据库的Conn接口:

go 复制代码
// Driver is the interface that must be implemented by a database
// driver.
//
// Database drivers may implement DriverContext for access
// to contexts and to parse the name only once for a pool of connections,
// instead of once per connection.
type Driver interface {
	// Open returns a new connection to the database.
	// The name is a string in a driver-specific format.
	//
	// Open may return a cached connection (one previously
	// closed), but doing so is unnecessary; the sql package
	// maintains a pool of idle connections for efficient re-use.
	//
	// The returned connection is only used by one goroutine at a
	// time.
	Open(name string) (Conn, error)
}

在上面的源码中,我们可以清晰知道,Driver接口是必须要被所有的数据库驱动程序实现的,提供而一个Open方法用于返回一个连接,这个连接可能是缓存的有效的,也可能是新建的连接。同时也提供了一个DriverContext接口,数据库驱动程序可以实现DriverContext以访问上下文,并仅为连接池解析一次名称,而不是每个连接解析一次。

DriverContext接口提供了一个OpenConnector方法用于返回一个连接器,在连接器中去获取对应的连接。连接器接口Connector提供了两个方法,Connect和Driver,其中Connect用于获取连接,并且可以附带参数ctx,Driver用于获取当前这个连接器的的驱动程序。

go 复制代码
// If a Driver implements DriverContext, then sql.DB will call
// OpenConnector to obtain a Connector and then invoke
// that Connector's Connect method to obtain each needed connection,
// instead of invoking the Driver's Open method for each connection.
// The two-step sequence allows drivers to parse the name just once
// and also provides access to per-Conn contexts.
type DriverContext interface {
	// OpenConnector must parse the name in the same format that Driver.Open
	// parses the name parameter.
	OpenConnector(name string) (Connector, error)
}

// A Connector represents a driver in a fixed configuration
// and can create any number of equivalent Conns for use
// by multiple goroutines.
//
// A Connector can be passed to sql.OpenDB, to allow drivers
// to implement their own sql.DB constructors, or returned by
// DriverContext's OpenConnector method, to allow drivers
// access to context and to avoid repeated parsing of driver
// configuration.
//
// If a Connector implements io.Closer, the sql package's DB.Close
// method will call Close and return error (if any).
type Connector interface {
	// Connect returns a connection to the database.
	// Connect may return a cached connection (one previously
	// closed), but doing so is unnecessary; the sql package
	// maintains a pool of idle connections for efficient re-use.
	//
	// The provided context.Context is for dialing purposes only
	// (see net.DialContext) and should not be stored or used for
	// other purposes. A default timeout should still be used
	// when dialing as a connection pool may call Connect
	// asynchronously to any query.
	//
	// The returned connection is only used by one goroutine at a
	// time.
	Connect(context.Context) (Conn, error)

	// Driver returns the underlying Driver of the Connector,
	// mainly to maintain compatibility with the Driver method
	// on sql.DB.
	Driver() Driver
}

接下来就是驱动的注册了,database/sql提供了一个驱动注册静态方法,驱动的具体实现中,可以调用该方法注册相关的驱动,但同时只允许注册同一类型的驱动,否则会panic。

go 复制代码
func Register(name string, driver driver.Driver) {
	driversMu.Lock()
	defer driversMu.Unlock()
	if driver == nil {
		panic("sql: Register driver is nil")
	}
	if _, dup := drivers[name]; dup {
		panic("sql: Register called twice for driver " + name)
	}
	drivers[name] = driver
}

该函数用来注册数据库驱动。当第三方开发者开发数据库驱动时,都会实现init函数,而init中会调用 Register(name string, driver driver.Driver) 完成驱动注册。如<github.com/go-sql-driver/mysql>

2、驱动连接:driver.Conn

go 复制代码
type Conn interface {
	// Prepare returns a prepared statement, bound to this connection.
	Prepare(query string) (Stmt, error)

	// Close invalidates and potentially stops any current
	// prepared statements and transactions, marking this
	// connection as no longer in use.
	//
	// Because the sql package maintains a free pool of
	// connections and only calls Close when there's a surplus of
	// idle connections, it shouldn't be necessary for drivers to
	// do their own connection caching.
	//
	// Drivers must ensure all network calls made by Close
	// do not block indefinitely (e.g. apply a timeout).
	Close() error

	// Begin starts and returns a new transaction.
	//
	// Deprecated: Drivers should implement ConnBeginTx instead (or additionally).
	Begin() (Tx, error)
}

Prepare:返回与当前连接相关的执行SQL语句的准备状态(Stmt),可以进行查询、删除等操作。
Close:关闭当前的链接,执行释放连接拥有的资源等清理工作。
Begin: // 返回一个代表事务处理的Tx,通过它可以进行查询、更新等操作,或者对事务进行回滚、递交。

新版本中,Begin方法已经不推荐了,被ConnBeginTx代替了, 新版本中的Begin方法多提供了入参ctx和额外的可选参数opts,便于扩展和控制。

go 复制代码
// ConnBeginTx enhances the Conn interface with context and TxOptions.
type ConnBeginTx interface {
	// BeginTx starts and returns a new transaction.
	// If the context is canceled by the user the sql package will
	// call Tx.Rollback before discarding and closing the connection.
	//
	// This must check opts.Isolation to determine if there is a set
	// isolation level. If the driver does not support a non-default
	// level and one is set or if there is a non-default isolation level
	// that is not supported, an error must be returned.
	//
	// This must also check opts.ReadOnly to determine if the read-only
	// value is true to either set the read-only transaction property if supported
	// or return an error if it is not supported.
	BeginTx(ctx context.Context, opts TxOptions) (Tx, error)
}

3、预处理结构:Stmt

go 复制代码
// Stmt is a prepared statement. It is bound to a Conn and not
// used by multiple goroutines concurrently.
type Stmt interface {
	// Close closes the statement.
	//
	// As of Go 1.1, a Stmt will not be closed if it's in use
	// by any queries.
	//
	// Drivers must ensure all network calls made by Close
	// do not block indefinitely (e.g. apply a timeout).
	Close() error

	// NumInput returns the number of placeholder parameters.
	//
	// If NumInput returns >= 0, the sql package will sanity check
	// argument counts from callers and return errors to the caller
	// before the statement's Exec or Query methods are called.
	//
	// NumInput may also return -1, if the driver doesn't know
	// its number of placeholders. In that case, the sql package
	// will not sanity check Exec or Query argument counts.
	NumInput() int

	// Exec executes a query that doesn't return rows, such
	// as an INSERT or UPDATE.
	//
	// Deprecated: Drivers should implement StmtExecContext instead (or additionally).
	Exec(args []Value) (Result, error)

	// Query executes a query that may return rows, such as a
	// SELECT.
	//
	// Deprecated: Drivers should implement StmtQueryContext instead (or additionally).
	Query(args []Value) (Rows, error)
}

// StmtExecContext enhances the Stmt interface by providing Exec with context.
type StmtExecContext interface {
	// ExecContext executes a query that doesn't return rows, such
	// as an INSERT or UPDATE.
	//
	// ExecContext must honor the context timeout and return when it is canceled.
	ExecContext(ctx context.Context, args []NamedValue) (Result, error)
}

// StmtQueryContext enhances the Stmt interface by providing Query with context.
type StmtQueryContext interface {
	// QueryContext executes a query that may return rows, such as a
	// SELECT.
	//
	// QueryContext must honor the context timeout and return when it is canceled.
	QueryContext(ctx context.Context, args []NamedValue) (Rows, error)
}

Close:关闭当前的连接状态,但如果当前正在执行query,query还是会有效返回rows数据。
NumInput:返回当前预留参数的个数,当返回>=0时,数据库驱动会智能检查调用者的参数。 当数据库驱动包不知道预留参数的时候,返回-1。
Exec:执行Prepare准备好的SQL,传入参数执行Update/Insert等操作,返回Result数据,Result中包含最后插入的自增主键序号(LastInsertId)和受影响的行数(RowAffected)。
Query:执行Prepare准备好的SQL,传入需要的参数执行select操作,返回Rows结果集。

4、执行结果 driver.Result

go 复制代码
// Result is the result of a query execution.
type Result interface {
	// LastInsertId returns the database's auto-generated ID
	// after, for example, an INSERT into a table with primary
	// key.
	LastInsertId() (int64, error)

	// RowsAffected returns the number of rows affected by the
	// query.
	RowsAffected() (int64, error)
}

5、查询结果:driver.Rows

go 复制代码
// Rows is an iterator over an executed query's results.
type Rows interface {
 
	// 该函数返回查询数据库表的字段信息,这个返回的slice和SQL查询的字段一一对应,
    // 而不是返回整张表的所有字段。
	Columns() []string
 
	// 用来关闭Rows迭代器
	Close() error
 
	// 该函数用来返回下一条数据,把数据赋值给dest .
    // dest里面元素必须是driver.Value的值(string除外),返回的数据里面所有的 string 都必须转换成
    // []byte.如果最后没有数据了,Next 函数返回 io.EOF。
	Next(dest []Value) error
}

可以看到,在新版的源码中,Exec和Query已经被单独拎出去定义了接口,方法中只是为了增加ctx参数,这也是golang为了保持向下兼容而做的,试想,如果直接在原有的接口定义的加入ctx,升级golang版本的时候这块儿肯定得花很大功夫去改造。

6、driver.RowsAffected

RowsAffected 不是别的东西,实际上只是 int64 的别名,但它实现了Result接口,用于底层实现 Result 的表示方式,构建Exec方法返回的结果集。

go 复制代码
// RowsAffected implements Result for an INSERT or UPDATE operation
// which mutates a number of rows.
type RowsAffected int64
 
var _ Result = RowsAffected(0)
 
func (RowsAffected) LastInsertId() (int64, error) {
	return 0, errors.New("LastInsertId is not supported by this driver")
}
 
func (v RowsAffected) RowsAffected() (int64, error) {
	return int64(v), nil
}

7、driver.Value

Value 其实是一个空接口,可以容纳任何的数据。

go 复制代码
// diver 的 Value 是驱动必须能够操作的 Value,Value要么是nil,要么是下面任意一种:
//
//   int64
//   float64
//   bool
//   []byte
//   string   [*] 除了Rows.Next,返回的不能是string
//   time.Time
//
type Value interface{}

8、Value定义转换相关

在driver/types.go中,还定义了ValueConverter将一个普通的值(any)转换成driver.Value的接口、Valuer接口用于获取driver.Value等,就不逐个展开了。

go 复制代码
// ValueConverter is the interface providing the ConvertValue method.
//
// Various implementations of ValueConverter are provided by the
// driver package to provide consistent implementations of conversions
// between drivers. The ValueConverters have several uses:
//
//   - converting from the Value types as provided by the sql package
//     into a database table's specific column type and making sure it
//     fits, such as making sure a particular int64 fits in a
//     table's uint16 column.
//
//   - converting a value as given from the database into one of the
//     driver Value types.
//
//   - by the sql package, for converting from a driver's Value type
//     to a user's type in a scan.
type ValueConverter interface {
	// ConvertValue converts a value to a driver Value.
	ConvertValue(v any) (Value, error)
}

// Valuer is the interface providing the Value method.
//
// Types implementing Valuer interface are able to convert
// themselves to a driver Value.
type Valuer interface {
	// Value returns a driver Value.
	// Value must not panic.
	Value() (Value, error)
}

三、sql包

在sql包中,主要是sql.go中的 DB结构,对应为数据库的具象化实例。DB

go 复制代码
// DB is a database handle representing a pool of zero or more
// underlying connections. It's safe for concurrent use by multiple
// goroutines.
//
// The sql package creates and frees connections automatically; it
// also maintains a free pool of idle connections. If the database has
// a concept of per-connection state, such state can be reliably observed
// within a transaction (Tx) or connection (Conn). Once DB.Begin is called, the
// returned Tx is bound to a single connection. Once Commit or
// Rollback is called on the transaction, that transaction's
// connection is returned to DB's idle connection pool. The pool size
// can be controlled with SetMaxIdleConns.
type DB struct {
	// Atomic access only. At top of struct to prevent mis-alignment
	// on 32-bit platforms. Of type time.Duration.
	waitDuration int64 // Total time waited for new connections.

	connector driver.Connector
	// numClosed is an atomic counter which represents a total number of
	// closed connections. Stmt.openStmt checks it before cleaning closed
	// connections in Stmt.css.
	numClosed uint64

	mu           sync.Mutex    // protects following fields
	freeConn     []*driverConn // free connections ordered by returnedAt oldest to newest
	connRequests map[uint64]chan connRequest
	nextRequest  uint64 // Next key to use in connRequests.
	numOpen      int    // number of opened and pending open connections
	// Used to signal the need for new connections
	// a goroutine running connectionOpener() reads on this chan and
	// maybeOpenNewConnections sends on the chan (one send per needed connection)
	// It is closed during db.Close(). The close tells the connectionOpener
	// goroutine to exit.
	openerCh          chan struct{}
	closed            bool
	dep               map[finalCloser]depSet
	lastPut           map[*driverConn]string // stacktrace of last conn's put; debug only
	maxIdleCount      int                    // zero means defaultMaxIdleConns; negative means 0
	maxOpen           int                    // <= 0 means unlimited
	maxLifetime       time.Duration          // maximum amount of time a connection may be reused
	maxIdleTime       time.Duration          // maximum amount of time a connection may be idle before being closed
	cleanerCh         chan struct{}
	waitCount         int64 // Total number of connections waited for.
	maxIdleClosed     int64 // Total number of connections closed due to idle count.
	maxIdleTimeClosed int64 // Total number of connections closed due to idle time.
	maxLifetimeClosed int64 // Total number of connections closed due to max connection lifetime limit.

	stop func() // stop cancels the connection opener.
}

几个主要的字段,其他字段大部分都是和连接池参数相关的:
connector:用于创建数据库连接的抽象连接器,由第三方数据库提供具体实现。
mu:互斥锁,保证并发安全。
freeConn:数据库连接池,缓存可用的连接以供后续复用。
connRequests:唤醒通道集合,和阻塞等待连接的协程是一对一的关系。
openerCh:创建连接信号通道. 用于向连接创建协程 opener goroutine 发送信号。
stop:连接创建协程 opener goroutine 的终止器,用于停止该协程。

DB结构主要作用如下:

二、结语

本章中我们学习了,golang中database/sql的源码阅读,了解了整个database/sql最大的特点就是定义接口,不做具体实现,从而让使用方去方便使用不同的驱动实现。同时提供了DB实例,内置连接池,方便管理连接的创建和销毁。

三、参考

1、Go database/sql连接池 - 源码学习

2、Golang sql 标准库源码解析

相关推荐
songqq27几秒前
SQL题:使用hive查询各类型专利top 10申请人,以及对应的专利申请数
数据库·sql
计算机学长felix4 分钟前
基于SpringBoot的“校园交友网站”的设计与实现(源码+数据库+文档+PPT)
数据库·spring boot·毕业设计·交友
小码的头发丝、1 小时前
Django中ListView 和 DetailView类的区别
数据库·python·django
杜杜的man1 小时前
【go从零单排】go中的结构体struct和method
开发语言·后端·golang
幼儿园老大*1 小时前
走进 Go 语言基础语法
开发语言·后端·学习·golang·go
半桶水专家1 小时前
go语言中package详解
开发语言·golang·xcode
llllinuuu1 小时前
Go语言结构体、方法与接口
开发语言·后端·golang
Karoku0661 小时前
【企业级分布式系统】Zabbix监控系统与部署安装
运维·服务器·数据库·redis·mysql·zabbix
cookies_s_s1 小时前
Golang--协程和管道
开发语言·后端·golang
王大锤43911 小时前
golang通用后台管理系统07(后台与若依前端对接)
开发语言·前端·golang