Best Practices for Go Monorepo Dependency Management

Introduction

In large Go projects, adopting monorepo architecture can bring many advantages such as code sharing and unified build processes. However, as the project scale grows, dependency management issues become increasingly prominent: different services may need different versions of third-party libraries, dependency version conflicts between shared packages and services, and complex transitive dependency relationships make dependency management tricky.

Based on actual project experience, this article summarizes a set of effective Go monorepo dependency management solutions, successfully solving complex dependency conflict issues through modular dependency management and rational use of replace directives.

Problem Background

In monorepo architecture, multiple services and shared packages (pkg/) may depend on different versions of third-party libraries, leading to:

1. Version conflicts: Different services need different versions of the same dependency (such as Kubernetes, certain framework libraries).
2. Transitive dependency conflicts: Incompatible versions of indirect dependencies.
3. Local dependency management difficulties: Dependency versions of shared packages (pkg/) are difficult to unify.
4. High build complexity: A single go.mod is difficult to meet the needs of all submodules.

Solution

1. Modular Dependency Management

Create independent go.mod files for each submodule to achieve dependency isolation:

my-monorepo/
├── go.mod                          # Root module
├── pkg/
│   ├── go.mod                      # Shared package module
│   ├── k8s/
│   │   └── go.mod                  # k8s toolkit module
│   ├── storage/
│   │   └── go.mod                  # storage toolkit module
│   └── repository/
│       └── go.mod                  # repository toolkit module
├── cmd/
│   └── cli-tool/
│       └── go.mod                  # CLI tool module
└── services/
    ├── service-a/
    │   └── go.mod                  # Service module
    ├── service-b/
    │   └── go.mod
    └── service-c/
        └── go.mod

Advantages:

• Each module can independently manage dependency versions.
• Avoid unnecessary dependency transmission.
• Clearer dependency boundaries.
• Easy to build and test separately.

2. Use replace directives to manage local dependencies

In go.mod of each submodule, use replace directive to reference dependencies in local paths:

Root module (go.mod)

module my-monorepo

go 1.21

replace (
    my-monorepo/pkg => ./pkg
    my-monorepo/pkg/storage => ./pkg/storage
    my-monorepo/pkg/k8s => ./pkg/k8s
)

Service module example (services/service-a/go.mod)

module my-monorepo/services/service-a

go 1.21

replace (
    my-monorepo/pkg => ../../pkg
    my-monorepo/pkg/repository => ../../pkg/repository
)

require (
    my-monorepo/pkg v0.0.0-00010101000000-000000000000
    my-monorepo/pkg/repository v0.0.0-00010101000000-000000000000
    // ... other dependencies
)

Key Points:

• Use relative paths (../../pkg) to reference local modules.
• Use pseudo-version v0.0.0-00010101000000-000000000000 as version identifier for local dependencies.
• Each module only declares local dependencies it directly uses.
• Relative paths are calculated based on the directory where the module is located.

3. Unify Third-party Dependency Versions

3.1 Unify Kubernetes Versions

The dependency relationship of the Kubernetes ecosystem is complex, and different packages may depend on different versions. Unify versions of Kubernetes-related dependencies in go.mod of shared packages:

module my-monorepo/pkg

go 1.21

replace (
    k8s.io/apiextensions-apiserver => k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver v0.0.0-20240417172702-7c48c2bd72b9
    k8s.io/client-go => k8s.io/client-go v0.30.0
    k8s.io/kubernetes => k8s.io/kubernetes v1.30.0
    k8s.io/cri-api => k8s.io/cri-api v0.30.0
)

Reason:

• The dependency relationship of the Kubernetes ecosystem is complex, and different packages may depend on different versions.
• Force unified versions through replace to avoid version conflicts.
• Use packages in the staging directory to ensure version consistency.

3.2 Third-party Dependency Replacement

When using fork versions or internal versions, use replace in relevant modules:

// Replace with internal fork version
replace (
    github.com/example/library/v2 => github.com/my-org/library/v2 v2.1.0-custom
)

Application Scenarios:

• Need to use internal fork of third-party libraries.
• Need to fix bugs but upstream has not merged yet.
• Need to use specific versions of dependencies.

3.3 Dependency Version Selection Principles

1. Unify shared dependency versions: Unify management of shared dependency versions in pkg/go.mod.
2. Independent management of service-specific dependencies: Each service can have its own dependency versions.
3. Use replace to solve conflicts: When version conflicts occur, prioritize using replace to unify versions.

4. Version Strategy

4.1 Go Version

It is recommended to unify all modules to the same Go version to avoid compatibility issues:

go 1.21  // All modules use the same version

If different versions need to be supported due to historical reasons, it is suggested to:

• Gradually migrate to a unified version.
• Clearly state version requirements in documentation.
• Verify compatibility of different versions in CI/CD.

4.2 Dependency Version Management

• Major version dependency: Follow semantic version control, pay special attention to major version changes.
• Test version: Avoid using test versions like -alpha, -beta in production environment.
• Security update: Regularly check for security vulnerabilities in dependencies and update in time.

Dependency Management Best Practices

1. Add New Dependency

Add dependency in shared package (pkg/)

cd pkg
go get github.com/example/[email protected]
go mod tidy

Add dependency in service

cd services/my-service
go get github.com/example/[email protected]
go mod tidy

2. Update Dependency Version

Update shared dependency

1. Update version in pkg/go.mod.
2. Run go mod tidy to update all submodules.
3. Check for version conflicts, use replace to unify versions if necessary.
4. Run tests to ensure compatibility.

Update service-specific dependency

1. Update in go.mod of corresponding service.
2. Run go mod tidy.
3. Run tests for that service.

3. Handle Version Conflicts

When version conflicts occur:

1. Identify conflict source:

bash go mod why github.com/example/package go mod graph | grep github.com/example/package

1. Unify version: Use replace in go.mod of relevant modules to unify version.

go replace github.com/example/package => github.com/example/package v1.2.3

1. Test verification: Ensure all modules can be built and run normally. bash go build ./... go test ./...

4. Local Development

Build specific service

cd services/my-service
go build ./...

Build all modules

# In root directory
go work sync  # If using go.work
# Or build each module separately
for dir in cmd/* services/*; do
    (cd "$dir" && go build ./...)
done

Use Go Workspace (Recommended)

Go 1.18+ introduced workspace feature, which can simplify local development:

# Create workspace
go work init
go work use ./pkg
go work use ./services/service-a
go work use ./cmd/cli-tool

Create go.work file:

go 1.21

use (
    ./pkg
    ./services/service-a
    ./services/service-b
    ./cmd/cli-tool
)

5. Docker Build

In Dockerfile, copy and download dependencies first to utilize Docker layer caching for build optimization:

FROM golang:1.21-alpine AS build

WORKDIR /app

# Copy root go.mod
COPY go.mod ./
COPY go.sum ./

# Copy submodule go.mod
COPY cmd/cli-tool/go.mod ./cmd/cli-tool/
COPY cmd/cli-tool/go.sum ./cmd/cli-tool/
COPY pkg/ ./pkg/

# Download dependencies (utilize Docker cache)
RUN cd cmd/cli-tool && go mod download

# Copy source code and build
COPY . .
RUN cd cmd/cli-tool && go build -o /cli-tool

FROM alpine:latest
COPY --from=build /cli-tool /usr/local/bin/cli-tool
ENTRYPOINT ["/usr/local/bin/cli-tool"]

Optimization tips:

• Copy go.mod and go.sum first, utilize Docker layer cache.
• Download dependencies before copying source code.
• Use multi-stage build to reduce image size.

Dependency Relationship Diagram Example

Root module (my-monorepo)
├── pkg/
│   ├── Unify Kubernetes v0.30.0
│   ├── Unify shared dependency versions
│   └── Shared toolkit dependency
├── pkg/k8s/
│   └── Kubernetes v0.30.0 (Unified version)
├── pkg/storage/
│   └── Storage related dependency
├── pkg/repository/
│   ├── Database operation dependency
│   └── Git operation dependency
├── cmd/cli-tool/
│   ├── CLI framework dependency
│   └── Depends on pkg and pkg/repository
└── services/
    ├── service-a/
    │   ├── Web framework dependency
    │   └── Depends on pkg and pkg/repository
    ├── service-b/
    │   └── Depends on pkg
    └── service-c/
        └── Depends on pkg and pkg/repository

FAQ

Q1: Why need multiple go.mod files?

A: In monorepo, different services may have different dependency needs. Independent go.mod files allow:

• Independent management of dependency versions for each module.
• Avoiding unnecessary dependency transmission.
• Clearer dependency boundaries.
• Easy to build and deploy separately.

Q2: Will replace directive affect production build?

A: No. replace directive only takes effect during local development. During production build:

• If dependency has been published to repository, Go will use version in repository.
• For local path replace, same directory structure needs to be maintained in build environment.
• It is recommended to verify build in CI/CD to ensure production environment can build normally.

Q3: How to ensure all modules use compatible dependency versions?

A:

1. Unify management of shared dependencies in pkg/go.mod.
2. Regularly run go mod tidy and go mod verify.
3. Use CI/CD to check for dependency conflicts.
4. Establish dependency version update process.
5. Use tools like go mod graph to analyze dependency relationships.

Q4: How to handle cross-module dependency updates?

A:

1. First update dependency of shared package (pkg/).
2. Run go mod tidy to update all modules depending on that shared package.
3. Check for version conflicts.
4. Update replace directive in service's go.mod if necessary.
5. Run full test suite.

Q5: When should go.work be used?

A: go.work is suitable for:

• Local development environment.
• Need to modify multiple modules simultaneously.
• Rapid iteration and testing.

Not suitable for:

• Production build (should use standard go.mod).
• CI/CD environment (unless explicitly required).

Tools and Automation

1. Dependency Check Script

#!/bin/bash
# check-deps.sh

echo "Checking for dependency conflicts..."

for dir in cmd/* services/* pkg/*/; do
    if [ -f "$dir/go.mod" ]; then
        echo "Checking $dir"
        (cd "$dir" && go mod verify && go mod tidy -check)
    fi
done

2. Unified Update Script

#!/bin/bash
# update-deps.sh

PACKAGE=$1
VERSION=$2

if [ -z "$PACKAGE" ] || [ -z "$VERSION" ]; then
    echo "Usage: $0 <package> <version>"
    exit 1
fi

for dir in cmd/* services/* pkg/*/; do
    if [ -f "$dir/go.mod" ] && grep -q "$PACKAGE" "$dir/go.mod"; then
        echo "Updating $PACKAGE to $VERSION in $dir"
        (cd "$dir" && go get "$PACKAGE@$VERSION" && go mod tidy)
    fi
done

3. CI/CD Check

Add dependency check step in CI/CD:

# .github/workflows/ci.yml
- name: Verify dependencies
  run: |
    for dir in cmd/* services/* pkg/*/; do
      if [ -f "$dir/go.mod" ]; then
        cd "$dir"
        go mod verify
        go mod tidy -check
      fi
    done

Future Improvement Directions

1. Unify Go Version: Unify all modules to the same Go version.
2. Dependency Version Audit: Regularly audit and update dependency versions to ensure security.
3. Automation Tools: Develop scripts to automatically synchronize dependency versions.
4. Document Dependency Strategy: Clarify which dependencies need unified versions and which can be managed independently.
5. Consider using go.work: Fully utilize Go 1.18+ workspace feature for local development.
6. Dependency Security Scan: Integrate tools like govulncheck for security scanning.

Related Resources

• Go Modules Official Documentation
• Go Modules replace Directive
• Go Workspace Documentation
• Kubernetes Dependency Management Best Practices

Summary

By creating independent go.mod files for submodules and using replace directive to unify management of dependency versions, we can effectively solve complex dependency conflict issues in monorepo. Advantages of this solution include:

1. Dependency Isolation: Each module can manage dependencies independently.
2. Version Unification: Unify versions of critical dependencies through replace.
3. Flexibility: Different services can use different dependency versions (under compatibility premise).
4. Maintainability: Clear dependency relationships, easy to maintain and update.
5. Extensibility: Easy to add new modules and services.

During implementation, should:

• Follow established dependency management strategies.
• Consider impact on existing modules when adding new dependencies.
• Regularly update and audit dependency versions.
• Keep dependency management documentation updated.
• Automate dependency check in CI/CD.

Hope these practical experiences can help you better manage dependencies in Go monorepo projects and avoid common dependency conflict issues.