Go Modules 和 Go Proxy

Go Modules

简介

Go 在 1.11 版本引入了新的依赖管理模式 Go Modules，旨在解决一直广为诟病的依赖管理问题。

使用下列命令启用 Go Modules

go env -w GO111MODULE=on # 不建议设为 auto

GOENV

Go Modules 采用语义化版本，详见 https://semver.org/lang/zh-CN/ 。

Go Modules 的使用方法详见：

改造一个现有的项目

场景：项目来自 GitHub，版本号作为 tag 附着在 commit 上。

v0 和 v1

go mods init [模块路径]

go build && go mod tidy

项目根目录下会生成两个文件（需要加入到 git 中）：

go.modpackage.jsongo.sumpackage-lock.json

Go 在编译前会下载（如需）依赖并检查校验和，如果不一致将拒绝编译。

verifying github.com/zzc-tongji/mydictionary/v4@v4.1.0: checksum mismatch

v2+

在语义化版本的定义中，主版本号更改意味着“不兼容的 API 修改”。

对于2.0.0及以上版本，上述模块路径需要加上主版本号。例如：

module github.com/zzc-tongji/mydictionary/v4

go 1.13

require (
    github.com/360EntSecGroup-Skylar/excelize v1.4.1
    github.com/PuerkitoBio/goquery v1.5.0
)

此时，引用模块的地方也需要做相应的修改。例如：

module github.com/zzc-tongji/mydictionary-demo/v4

go 1.13

require github.com/zzc-tongji/mydictionary/v4 v4.1.1

package main

import (
    "bufio"
    "fmt"
    "os"
    "path/filepath"
    "time"

    "github.com/zzc-tongji/mydictionary/v4"
)

// ...

新建一个项目

场景：项目最终发布到 GitHub，版本号作为 tag 附着在 commit 上。

go mods init [模块路径]

下载依赖模块

go get [依赖模块路径]go get [依赖模块路径]@[版本号]vX.Y.Z

更新模块

go.modrequirevX.Y.Z

清理

go mod tidygo.sum

Go Proxy

Go 在 1.13 版本加入了 Go Proxy，并提供了官方镜像。

环境变量

相关的环境变量和默认值如下：

GOPROXY="https://proxy.golang.org,direct"
GOSUMDB="sum.golang.org"
GOPRIVATE=""
GONOPROXY=""
GONOSUMDB=""

GOPROXY

GOPROXY,

默认值：

GOPROXY="https://proxy.golang.org,direct"

根据官方说法：

The default setting for GOPROXY is
"https://proxy.golang.org,direct", which means to try the
Go module mirror run by Google and fall back to a direct connection
if the proxy reports that it does not have the module (HTTP error 404 or 410).

由于众所周知的原因，建议国内的开发者将其设为：

GOPROXY="https://proxy.golang.org,https://goproxy.io,direct"

GOPROXY=offGOPROXY=direct

GOSUMDB

GOSUMDBsum.golang.org

GOSUMDB=off

默认值：

GOSUMDB="sum.golang.org"

由于众所周知的原因，建议国内的开发者将其设为：

GOSUMDB=gosum.io+ce6e7565+AY5qEHUk/qmHc5btzW45JVoENfazw8LielDsaI+lEbq6

GOPRIVATE

GOPRIVATE

,*

官方解释和例子：

The GOPRIVATE environment variable controls which modules the go command considers to be private (not available publicly) and should therefore not use the proxy or checksum database. The variable is a comma-separated list of glob patterns (in the syntax of Go's path.Match) of module path prefixes. For example,

> causes the go command to treat as private any module with a path prefix matching either pattern, including **git.corp.example.com/xyzzy**, **rsc.io/private**, and **rsc.io/private/quux**.



### 踩坑经历


#### 入坑


笔者在更新 [mydictionary](https://github.com/zzc-tongji/mydictionary) 的时候遇到了一个问题。


- 笔者发布了 [mydictionary](https://github.com/zzc-tongji/mydictionary) v4.1.0 版本。用 [mydictionary-demo](https://github.com/zzc-tongji/mydictionary-demo) 拉取这个依赖做测试。
- 在测试过程中发现了 [mydictionary](https://github.com/zzc-tongji/mydictionary) 的一处 bug 。
- 首先，笔者修复了这个 bug 。其次，删除了本地和远端的 v4.1.0 标签。然后，用 `git push -f` 强行修改了这个版本。最后，重新为本地和远端打上了 v4.1.0 标签。
- 当笔者再次用 [mydictionary-demo](https://github.com/zzc-tongji/mydictionary-demo) 拉取这个依赖的时候，发现并没有更新。



#### 思索


笔者做了很多尝试未果，最后发现官方镜像 [https://proxy.golang.org](https://proxy.golang.org) 上有这么一段文字：


> **I removed a bad release from my repository but it still appears in the mirror, what should I do?**
> Whenever possible, the mirror aims to cache content in order to avoid breaking builds for people that depend on your package, so this bad release may still be available in the mirror even if it is not available at the origin. The same situation applies if you delete your entire repository. We suggest creating a new version and encouraging people to use that one instead.



对于每个依赖的每个语义化版本，第一次从源站获取的数据会被官方镜像会永久保留。**也就是说，即使强行修改源站上的这个语义化版本（例如使用 `git push -f`），镜像内容也不会更新。**官方的建议是，弃用这个被强行修改的语义化版本，发布一个新的语义化版本并指引人们去使用。


#### 出坑


事实确实如此，当笔者设定 `GOPROXY=direct` 从源站获取依赖，得到的是更新后的 v4.1.0 版本。


笔者将 sum  写入 [mydictionary-demo](https://github.com/zzc-tongji/mydictionary-demo) 的文件 `go.sum` 中，然后将 `GOPROXY` 还原为默认，最后用命令 `go build` 进行编译。果然，由于校验和不一致，编译中止。

$ cd mydictionary-demo

$ git checkout v4.1.0

$ go build
go: downloading github.com/zzc-tongji/mydictionary/v4 v4.1.0
verifying github.com/zzc-tongji/mydictionary/v4@v4.1.0: checksum mismatch
downloaded: h1:slMg3fcLYvE9iufNqmec2jJUojv0qnGkPsbbiBor6EA=
go.sum: h1:7rgn+0Pys4fw+mXiD026rTzPuHto86BRxUia8FRNQSg=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.

笔者只能按照官方的指引，发布了一个新版本 v4.1.1 来修复这个问题。


#### 心得


为了避免在镜像上留下“脏数据”：


1. 编写测试用例并使用命令 `go test` 测试。
2. 采用命令 `go mod vendor` 和 `go build -mod vendor` 进行本地调试。
3. 更新版本前一定要仔细检查，避免强制推送和重置标签。
4. 设定 `GOPROXY=direct` 绕过镜像（前提是镜像不会主动从源站拉去数据，官方镜像似乎可行）。



## 参考资料


### go help modules


> A module is a collection of related Go packages.
Modules are the unit of source code interchange and versioning.
The go command has direct support for working with modules,
including recording and resolving dependencies on other modules.
Modules replace the old GOPATH-based approach to specifying
which source files are used in a given build.
> Module support
> Go 1.13 includes support for Go modules. Module-aware mode is active by default
whenever a go.mod file is found in, or in a parent of, the current directory.
> The quickest way to take advantage of module support is to check out your
repository, create a go.mod file (described in the next section) there, and run
go commands from within that file tree.
> For more fine-grained control, Go 1.13 continues to respect
a temporary environment variable, GO111MODULE, which can be set to one
of three string values: off, on, or auto (the default).
If GO111MODULE=on, then the go command requires the use of modules,
never consulting GOPATH. We refer to this as the command
being module-aware or running in "module-aware mode".
If GO111MODULE=off, then the go command never uses
module support. Instead it looks in vendor directories and GOPATH
to find dependencies; we now refer to this as "GOPATH mode."
If GO111MODULE=auto or is unset, then the go command enables or disables
module support based on the current directory.
Module support is enabled only when the current directory contains a
go.mod file or is below a directory containing a go.mod file.
> In module-aware mode, GOPATH no longer defines the meaning of imports
during a build, but it still stores downloaded dependencies (in GOPATH/pkg/mod)
and installed commands (in GOPATH/bin, unless GOBIN is set).
> Defining a module
> A module is defined by a tree of Go source files with a go.mod file
in the tree's root directory. The directory containing the go.mod file
is called the module root. Typically the module root will also correspond
to a source code repository root (but in general it need not).
The module is the set of all Go packages in the module root and its
subdirectories, but excluding subtrees with their own go.mod files.
> The "module path" is the import path prefix corresponding to the module root.
The go.mod file defines the module path and lists the specific versions
of other modules that should be used when resolving imports during a build,
by giving their module paths and versions.
> For example, this go.mod declares that the directory containing it is the root
of the module with path example.com/m, and it also declares that the module
depends on specific versions of golang.org/x/text and gopkg.in/yaml.v2:
> ```
module example.com/m

require (
    golang.org/x/text v0.3.0
    gopkg.in/yaml.v2 v2.1.0
)

The go.mod file can also specify replacements and excluded versions
that only apply when building the module directly; they are ignored
when the module is incorporated into a larger build.
For more about the go.mod file, see 'go help go.mod'.
To start a new module, simply create a go.mod file in the root of the
module's directory tree, containing only a module statement.
The 'go mod init' command can be used to do this:

> In a project already using an existing dependency management tool like
godep, glide, or dep, 'go mod init' will also add require statements
matching the existing configuration.
> Once the go.mod file exists, no additional steps are required:
go commands like 'go build', 'go test', or even 'go list' will automatically
add new dependencies as needed to satisfy imports.
> The main module and the build list
> The "main module" is the module containing the directory where the go command
is run. The go command finds the module root by looking for a go.mod in the
current directory, or else the current directory's parent directory,
or else the parent's parent directory, and so on.
> The main module's go.mod file defines the precise set of packages available
for use by the go command, through require, replace, and exclude statements.
Dependency modules, found by following require statements, also contribute
to the definition of that set of packages, but only through their go.mod
files' require statements: any replace and exclude statements in dependency
modules are ignored. The replace and exclude statements therefore allow the
main module complete control over its own build, without also being subject
to complete control by dependencies.
> The set of modules providing packages to builds is called the "build list".
The build list initially contains only the main module. Then the go command
adds to the list the exact module versions required by modules already
on the list, recursively, until there is nothing left to add to the list.
If multiple versions of a particular module are added to the list,
then at the end only the latest version (according to semantic version
ordering) is kept for use in the build.
> The 'go list' command provides information about the main module
and the build list. For example:
> ```
go list -m              # print path of main module
go list -m -f={{.Dir}}  # print root directory of main module
go list -m all          # print build list

go get github.com/gorilla/mux@latest    # same (@latest is default for 'go get')
go get github.com/gorilla/mux@v1.6.2    # records v1.6.2
go get github.com/gorilla/mux@e3702bed2 # records v1.6.2
go get github.com/gorilla/mux@c856192   # records v0.0.0-20180517173623-c85619274f5d

> Module compatibility and semantic versioning
> The go command requires that modules use semantic versions and expects that
the versions accurately describe compatibility: it assumes that v1.5.4 is a
backwards-compatible replacement for v1.5.3, v1.4.0, and even v1.0.0.
More generally the go command expects that packages follow the
"import compatibility rule", which says:
> "If an old package and a new package have the same import path,
the new package must be backwards compatible with the old package."
> Because the go command assumes the import compatibility rule,
a module definition can only set the minimum required version of one
of its dependencies: it cannot set a maximum or exclude selected versions.
Still, the import compatibility rule is not a guarantee: it may be that
v1.5.4 is buggy and not a backwards-compatible replacement for v1.5.3.
Because of this, the go command never updates from an older version
to a newer version of a module unasked.
> In semantic versioning, changing the major version number indicates a lack
of backwards compatibility with earlier versions. To preserve import
compatibility, the go command requires that modules with major version v2
or later use a module path with that major version as the final element.
For example, version v2.0.0 of example.com/m must instead use module path
example.com/m/v2, and packages in that module would use that path as
their import path prefix, as in example.com/m/v2/sub/pkg. Including the
major version number in the module path and import paths in this way is
called "semantic import versioning". Pseudo-versions for modules with major
version v2 and later begin with that major version instead of v0, as in
v2.0.0-20180326061214-4fc5987536ef.
> As a special case, module paths beginning with gopkg.in/ continue to use the
conventions established on that system: the major version is always present,
and it is preceded by a dot instead of a slash: gopkg.in/yaml.v1
and gopkg.in/yaml.v2, not gopkg.in/yaml and gopkg.in/yaml/v2.
> The go command treats modules with different module paths as unrelated:
it makes no connection between example.com/m and example.com/m/v2.
Modules with different major versions can be used together in a build
and are kept separate by the fact that their packages use different
import paths.
> In semantic versioning, major version v0 is for initial development,
indicating no expectations of stability or backwards compatibility.
Major version v0 does not appear in the module path, because those
versions are preparation for v1.0.0, and v1 does not appear in the
module path either.
> Code written before the semantic import versioning convention
was introduced may use major versions v2 and later to describe
the same set of unversioned import paths as used in v0 and v1.
To accommodate such code, if a source code repository has a
v2.0.0 or later tag for a file tree with no go.mod, the version is
considered to be part of the v1 module's available versions
and is given an +incompatible suffix when converted to a module
version, as in v2.0.0+incompatible. The +incompatible tag is also
applied to pseudo-versions derived from such versions, as in
v2.0.1-0.yyyymmddhhmmss-abcdefabcdef+incompatible.
> In general, having a dependency in the build list (as reported by 'go list -m all')
on a v0 version, pre-release version, pseudo-version, or +incompatible version
is an indication that problems are more likely when upgrading that
dependency, since there is no expectation of compatibility for those.
> See [https://research.swtch.com/vgo-import](https://research.swtch.com/vgo-import) for more information about
semantic import versioning, and see [https://semver.org/](https://semver.org/) for more about
semantic versioning.
> Module code layout
> For now, see [https://research.swtch.com/vgo-module](https://research.swtch.com/vgo-module) for information
about how source code in version control systems is mapped to
module file trees.
> Module downloading and verification
> The go command can fetch modules from a proxy or connect to source control
servers directly, according to the setting of the GOPROXY environment
variable (see 'go help env'). The default setting for GOPROXY is
"[https://proxy.golang.org](https://proxy.golang.org),direct", which means to try the
Go module mirror run by Google and fall back to a direct connection
if the proxy reports that it does not have the module (HTTP error 404 or 410).
See [https://proxy.golang.org/privacy](https://proxy.golang.org/privacy) for the service's privacy policy.
If GOPROXY is set to the string "direct", downloads use a direct connection
to source control servers. Setting GOPROXY to "off" disallows downloading
modules from any source. Otherwise, GOPROXY is expected to be a comma-separated
list of the URLs of module proxies, in which case the go command will fetch
modules from those proxies. For each request, the go command tries each proxy
in sequence, only moving to the next if the current proxy returns a 404 or 410
HTTP response. The string "direct" may appear in the proxy list,
to cause a direct connection to be attempted at that point in the search.
Any proxies listed after "direct" are never consulted.
> The GOPRIVATE and GONOPROXY environment variables allow bypassing
the proxy for selected modules. See 'go help module-private' for details.
> No matter the source of the modules, the go command checks downloads against
known checksums, to detect unexpected changes in the content of any specific
module version from one day to the next. This check first consults the current
module's go.sum file but falls back to the Go checksum database, controlled by
the GOSUMDB and GONOSUMDB environment variables. See 'go help module-auth'
for details.
> See 'go help goproxy' for details about the proxy protocol and also
the format of the cached downloaded packages.
> Modules and vendoring
> When using modules, the go command completely ignores vendor directories.
> By default, the go command satisfies dependencies by downloading modules
from their sources and using those downloaded copies (after verification,
as described in the previous section). To allow interoperation with older
versions of Go, or to ensure that all files used for a build are stored
together in a single file tree, 'go mod vendor' creates a directory named
vendor in the root directory of the main module and stores there all the
packages from dependency modules that are needed to support builds and
tests of packages in the main module.
> To build using the main module's top-level vendor directory to satisfy
dependencies (disabling use of the usual network sources and local
caches), use 'go build -mod=vendor'. Note that only the main module's
top-level vendor directory is used; vendor directories in other locations
are still ignored.



### go help module-auth


> The go command tries to authenticate every downloaded module,
checking that the bits downloaded for a specific module version today
match bits downloaded yesterday. This ensures repeatable builds
and detects introduction of unexpected changes, malicious or not.
> In each module's root, alongside go.mod, the go command maintains
a file named go.sum containing the cryptographic checksums of the
module's dependencies.
> The form of each line in go.sum is three fields:
> ```
<module> <version>[/go.mod] <hash>

GOSUMDB="sum.golang.org"
GOSUMDB="sum.golang.org+<publickey>"

> The go command knows the public key of sum.golang.org, and also that the name
sum.golang.google.cn (available inside mainland China) connects to the
sum.golang.org checksum database; use of any other database requires giving
the public key explicitly.
The URL defaults to "https://" followed by the database name.
> GOSUMDB defaults to "sum.golang.org", the Go checksum database run by Google.
See [https://sum.golang.org/privacy](https://sum.golang.org/privacy) for the service's privacy policy.
> If GOSUMDB is set to "off", or if "go get" is invoked with the -insecure flag,
the checksum database is not consulted, and all unrecognized modules are
accepted, at the cost of giving up the security guarantee of verified repeatable
downloads for all modules. A better way to bypass the checksum database
for specific modules is to use the GOPRIVATE or GONOSUMDB environment
variables. See 'go help module-private' for details.
> The 'go env -w' command (see 'go help env') can be used to set these variables
for future go command invocations.



### go help module-private


> The go command defaults to downloading modules from the public Go module
mirror at proxy.golang.org. It also defaults to validating downloaded modules,
regardless of source, against the public Go checksum database at sum.golang.org.
These defaults work well for publicly available source code.
> The GOPRIVATE environment variable controls which modules the go command
considers to be private (not available publicly) and should therefore not use the
proxy or checksum database. The variable is a comma-separated list of
glob patterns (in the syntax of Go's path.Match) of module path prefixes.
For example,
> ```
GOPRIVATE=*.corp.example.com,rsc.io/private

GOPRIVATE=*.corp.example.com
GOPROXY=proxy.example.com

> This would tell the go command and other tools that modules beginning with
a corp.example.com subdomain are private but that the company proxy should
be used for downloading both public and private modules, because
GONOPROXY has been set to a pattern that won't match any modules,
overriding GOPRIVATE.
> The 'go env -w' command (see 'go help env') can be used to set these variables
for future go command invocations.