Vista Acquires a Majority Interest in Sonatype: A Great Day for our Customers, Partners and Community

Go Dependencies in Nexus Repository

Repository Manager | Reading time: 11 minutes

Is this article helpful?

In this guide

Summary

This guide will give you fundamentals on dependency management with Go modules. Modules were added to the Go ecosystem to give you built-in versioning and dependency management. Now you and your fellow developers can adapt Go software development to Nexus Repository. Use this guide to get an understanding of the Go toolset, version control, and environment configuration.

Audience

The target audience for this guide includes:

  • Programmers familiar with development and dependency management in the Go ecosystem.
  • Administrators familiar with Go environment configuration.
  • Anyone interested in learning about the Go ecosystem in Nexus Repository.
  • Anyone with understanding of Nexus Repository Manager basics.

Prerequisites

In order to follow the use cases at the end, you’ll need the following:

  • Go version 1.11 or later
  • a GitHub account

Desired Outcomes

When you finish this guide we expect you to:

  • Identify the shortcomings of pre-module dependencies in Go projects.
  • Understand the importance of shifting to module support in Nexus Repository.
  • Demonstrate proxy configuration and locate project dependencies in the repository manager user interface.
  • Build a sample module with the Go toolset so you can see the results in the browse interface.

Shifting from the “Monorepo”

At one time, developers could only count on monolithic, one-stop repositories to source all their Golang code. While this centralized model has its benefits—namely a simplified dependency management strategy—it may also shore up many concerns. Imagine, a giant repository with endless lines of code. That may keep your admins up late at night thinking of strategies to scale development tools, execute code, and maintain the ecosystem. Lest we forget, the extra vigilance your admin may endure could take a toll on code health or system performance.

Reliance on the monorepo model proved to be a barrier for developers with the desire to work from a sub-project in a more isolated fashion. Working from source offered constraints at the system, project, and development levels:

  • At the system level, Go lacks a native package manager. Developers had to use tools like dep which served as a de facto dependency manager.
  • At the project level, only vendoring and GOPATH were available to maintain projects and dependency requirements brought by developers.
  • At the development level, there was no standard for dependency management.

Understandably so, many high-profit organizations may not see these factors as constraints. Some of these organizations can successfully scale up and maintain productivity. If a source change is necessary the admin could modify the Go “monorepo” with one or more commits. However the admin must take into consideration that rapid CI builds, ownership, accountability, and other caveats may complicate code maintenance when development teams continue to grow.

Package Management Before v1.11/v1.12

Developers that work in earlier versions of Golang use GOPATH for environment configuration. It’s also the location for all source code, binaries, and executables. In this pre-module era, projects were maintained by GOPATH set to a workspace directory (e.g. export GOPATH=$HOME/go). Also, in this era developers relied solely on third party solutions—like Dep—to manage package requirements. Furthermore, there was no other location to store packages and dependencies other than the workspace directory, inside the source project.

In Go v1.11 and beyond you and your fellow developers are no longer constrained to a centralized “monorepo.” Of course, the tools for building executables now is consistent with pre-v1.11 releases. However, a shift to independent package management began in a pilot release of v1.11. Version 1.11 offered beta support for Go modules as an alternative to GOPATH. Then a more stable version was deployed in 1.12. So from 1.12 on you can build applications outside the GOPATH, even though it’s still the place where source code is maintained.

Vendoring

Vendoring is a common technique used by Go developers in versions prior to v1.11. With the go command you can fork projects, build them, and pass the package and its dependencies to a vendor folder for distribution. Along with the platform and project, all dependencies are located in a bloated, mega-sized folder. Constant commits will eventually expand the folder into an enormous stack of Go files, associated metadata, and configuration resources.

Also, vendoring relied on third party tools to isolate parts of unneeded source code. A tool like Dep, for example, is intelligent enough to figure out the versions needed for a project. Often used with GitHub, developers move files into the vendor folder as a solution for code exceptions (such as .gitignore).

However, the issue with this technique is the directory that vends projects also maintains the Go platform itself. So it’s no surprise that vendoring has its pain points. Keeping copies of all source code, for every dependency, in each project poses the risk of losing dependencies all together. Since there’s no central proxy to retrieve packages, vendoring was the only way to manage the platform, project (local and/or in a VCS), and all dependencies brought in by you or your team.

By contrast, Go modules comes with native files to manage dependencies (e.g. go.mod) and a new way to specify Go directories so you and your team can access the exact project versions you need.

About Go Modules

As you move away from vendoring, modules help you organize projects in a more incremental and meaningful way. Go modules is a package management system for Golang. It offers a new standard and revised toolset to help you build proxy servers, where you’re no longer dependent on a monolithic platform. For example, you can stand up a caching proxy and use the Go toolset to fetch and publish Go software.

The reason to use a proxy for Go modules is to maintain more stable packages in your environment. Since you don’t need to rely on the GOPATH, vendoring, or a version control system to manage projects, the repository manager acts as the intermediary proxy. Imagine if your organization doesn’t use one. Take the scenario where your team repeatedly iterates code to a version control system like Github. If a developer from outside your organization commits code exposed to the public, they could break existing dependencies authored by your internal development team. So, using Nexus Repository as the central cache to resolve internal dependencies is recommended to prevent such disruptions.

Versions 1.11 and later also introduced a feature designed specifically to access Go modules: the Module Download API. This feature allows you to send GET requests from the client with the command go get. In return, the argument passes packages downstream to a custom URL in Nexus Repository. This adds another layer of abstraction so you and your developers only consume projects and dependencies you need.

GOPATH to GOPROXY

With your instance or instances of Nexus Repository, relying solely on GOPATH is unnecessary. Instead you can use new environment variable GOPROXY. This is where the value is set as an endpoint and dedicated location for your Go applications. So, if the endpoint is a group repository named go-all, then the variable will be:

export GOPROXY=http://localhost:8081/repository/go-all

After the export, you’ll be able to see the variable with the repository URL string in your .profile (or .bash_profile,.bashrc). At this point, Go projects are cached in Nexus Repository. In effect, a Go command determines that it needs a module. It looks to the endpoint provided in the path. With a few keystrokes, the go command fetches the files from the remote location and downloads them to the repository manager.

go.mod

When v1.11 came along with Go modules, dependency management got an upgrade. Modules allow you to maintain multiple versions of related Go packages. It was introduced to formalize code sharing between Go projects similar to the way formats like Java/Maven, PyPI and npm, maintain theirs. go.mod phases out the need for vendoring because you can import the latest components set to the environment variable.

go.mod is a module defined by a hierarchy of Go source files. This can be found in the project’s root directory. This file is characterized as such:

  • generated after initialization (go mod init [module]).
  • specifies builds with all dependencies and exact versions.
  • imports and tags groups of Go packages versioned together as one.

In a further shift from the constraints of the GOPATH model, older techniques like vendoring are now almost entirely redundant. It has the ability to store all the requirements needed to ensure availability.

Providing a go.mod file with the correct list of requirements is the recommended way to achieve reproducible builds in a scenario where our modules are being used.

Consider this example go.mod:

module github.com/any-go-repo/come-together

require (
    some-repo.com/example/paul v0.0.0-xxx.xxx
    some-repo.com/example/ringo v1.5.4
    some-repo.com/example/john v0.0.0-xxx-xxx
    some-repo.com/example/george v1.2.0
    some-repo.com/example/billy v1.2.0
)

When you run go get, the command downloads new dependencies to the module. In the example, the go.mod you can see the namespaces of all newly retrieved content pulled downstream and made available to your team (some-repo.com). Another file, go.sum, is maintained along with go.mod. go.sum keeps a record of secret hashes with specific project versions. You and your team must ensure both files are committed together when maintaining versions. If not, any conflicts between the contents of go.mod and go.sum could lead to errors.

Configuring a Go Project in Nexus Repository

In versions Nexus Repository 3.17 and higher you can serve and cache modules, remotely, from resources such as GitHub. Currently hosted repositories aren’t available. However you can address two use cases:

  • Configure the repository manager to cache projects
  • Create a simple project and download dependencies to the project

Use Case 1: Proxy Configuration

Start up your local instance of Nexus Repository and work through the steps below. In this procedure (and in Use Case 2) you’ll build and run your projects in a global temporary directory (/tmp).

  1. In Nexus Repository, click Create repository to start anew then Save.

  2. Copy the URL from the new repository: http://localhost:8081/repository/go-demo-proxy/

  3. Paste the URL to GOPROXY with the new endpoint: export GOPROXY=http://localhost:8081/repository/go-demo-proxy

  4. In the terminal, clone a project from GitHub in a directory nested in /tmp:

    mkdir -p /tmp/go-demo
    cd /tmp/go-demo
    git clone git@github.com:gobuffalo/buffalo.git
    ls # to list the contents
    
  5. Go to the directory: cd buffalo.

  6. Build packages from the Git repository: go build.

  7. Browse the repository and locate the following assets for each component proxied visible in tree view:

    .zip 
    .mod
    .info
    

In your terminal you can compare the module structure. For example, check out this cloud.google.com project root. Take a look at this view with the tree command and compare with the Browse UI in the repository manager:

$ cd cloud.google.com/
$ tree
.
└── go
    └── @v
        ├── list
        ├── list.lock
        ├── v0.26.0.info
        ├── v0.26.0.mod
        ├── v0.31.0.info
        ├── v0.31.0.mod
        ├── v0.36.0.info
        └── v0.36.0.mod

Now in this next use case, follow along to add dependencies to your own sample project.

Use Case 2: Create a main.go and Add Dependencies

In this scenario, we’re going to create a new project in a new repository and add dependencies. We’ll call this one another-go. Now, the first step is nearly identical, except you’ll initialize this new project with its own repository ID separate from the previous use case. This also means you need to update the GOPROXY variable so you can browse the project’s dependencies in the new repository.

  1. Create a new directory in /tmp: mkdir -p go-demo2
  2. In Nexus Repository, click Create repository to configure a new directory, then Save.

  3. Copy the URL from your new repository: http://localhost:8081/repository/another-go/

  4. Update GOPROXY with the new endpoint: GOPROXY=http://localhost:8081/repository/another-go/

Alternatively, you can open up your .bash_profile (or .bashrc) and add the URL as the GOPROXY. If you go this route save your updates with source ~/.bash_profile (or source ~/.bashrc)

Now that your caching proxy is configured to receive modules, create a sample module. In this example, let’s use github.com/foo/emoji-message as the module path. You’re going to create a go.mod project then iterate with a new dependency. Additionally, you’ll create a main.go, your first executable program.

Go to your terminal and follow the steps below.

  1. Create a temporary “workspace” directory: mkdir -p /tmp/go-again
  2. Change your directory to the target project location: cd /tmp/go-again
  3. Run go mod init [module path] in your terminal to start the project, e.g. go mod init github.com/foo/emoji-message

    It returns this message: go: creating new go.mod: module github.com/foo/emoji-message.

  4. Enter cat go.mod to view the file. It shows a minimal record of the module name and version.

    module github.com/foo/emoji-message go 1.12

  5. Create a main.go: touch main.go.

  6. Paste the sample code in the file (below).

We’re going to import an emoji project from Github (github.com/kyokomi/emoji) and add a function.

package main 
import ( 
    "fmt"
    "github.com/kyokomi/emoji" 
) 
func main() { 
    fmt.Println("Greetings Emoji!")
    emoji.Println(":speech_balloon: May I take your order?") 
    burgerMessage := emoji.Sprint("I'd like a :hamburger: and :fries:!!")
        fmt.Println(burgerMessage)
}
  1. Use the go command to run the app: go run main.go. This produces the output from the code. Be sure to check out another-go in the Browse UI to see the module and dependencies in tree view.
  2. View the new requirement documented in the go.mod file:

    $ cat go.mod
    module github.com/foo/emoji-message
    
    go 1.12
    
    require github.com/kyokomi/emoji v2.1.0+incompatible
    
  3. Locate the go.sum from the project root: cat go.sum. You’ll see the nested versions of the project, each with an associated cryptographic hash.

Additional Resources

Go over to Sonatype Help to read our help topics on Go Repositories in Nexus Repository Manager 3. Also, you can learn about our offical Go proxy discussed on our blog site.

Additional assistance is also available in the Sonatype Community at https://community.sonatype.com/. Please visit our community for easy access to support from us or your peers, product updates, insights from Nexus experts, free training and help, forums and much more.