GOPROXY(ies) on performance
Go modules have been available since July 2018, and are nowadays the officially sanctioned way to manage go dependencies.
With go 1.13, modules are downloaded by default from https://proxy.golang.org (and https://sum.golang.org).
A number of other modules proxy are available in the wild, for example https://athens.azurefd.net, and its underlying open source implementation, the Athens Project.
There are several significant upsides in using a proxy for go modules, and performance is listed prominently as one of these.
The following present numerical results from a test on the buildkit project, establishing a baseline comparison between not using any proxy and the official, google maintained proxy, the Athens public server, and a privately operated instance of Athens.
Test setup
The gist of our test is very simple, and defined inside a Dockerfile as follows:
FROM ghcr.io/dubo-dubon-duponey/base:builder-latestARG GOPROXY=directRUN go get github.com/moby/buildkit@v0.8.3; exit 1
The base image we use contains golang 1.16, and can be replaced by any similar image providing it.
Our self-hosted Athens server is built from: https://github.com/dubo-dubon-duponey/docker-goproxy/blob/work/Dockerfile
Athens runs inside a container, uses the default configuration, with in-memory index and filesystem storage, and is fronted by a Caddy server providing TLS termination and authentication.
We are running these tests on a self-hosted NUC server, wired in a decent (1Gb/s) residential line. Our Athens server runs on the same machine.
After a first cold run, each test is ran 10 times and results are averaged.
Direct
Not using any proxy yields (as expected) very slow results, in the range of 45 minutes.
This is painstakingly slow, and is literally a deal-killer for any significant project.
Golang Proxy
Using Google maintained proxy consistently clocks in at about 1 minute 50 seconds.
This is obviously the reference.
Athens public server
After a first cold run at 8 minutes, Athens public server hovers around 2 minutes, with some runs falling down to up to 6 minutes.
Results are much more erratic than with the official proxy, which has more likely to do with the supporting infrastructure and network backend rather than with the software itself. It’s certainly hard to compete with google resources and availability.
Athens self-hosted
Given Athens is configured to use direct for its backend requests, cold start ranges in the same ballpark as mentioned above.
Subsequent runs clock in around 1 minutes 20 seconds on average.
Caveats
None of this is scientifically rigorous. This is meant to provide a sense of how things are performing, and what you can expect from running your own proxy, using google infrastructure, or embracing direct.
We have not tested GoProxy, neither Thumbai, as neither projects seems to be actively developed, and it seems that Athens is the only serious kid in town at this point.
UPDATE: https://github.com/goproxyio/goproxy may be worth a try.
Take-away
It is hard to rationalize using “direct”.
From the perspective of having critical build-systems not relying on (yet) another third-party piece of infrastructure, “direct” turns out to be actually much worse, since it will make your build dependent on a (likely large) number of VCS that are typically cached / hidden by a module proxy. Furthermore, direct does not provide any of the guarantees of content availability & immutability that proxies (to some extent) advertise.
Athens public server is not meant for production use, as clearly outlined by the project, and while its results are not bad from a performance perspective, it’s unlikely there is any upside in using it instead of the golang official proxy.
Should you self-host Athens then?
You will likely not get a massive performance increase by doing so for simple use-cases: cold-start are likely going to be (much more) costly, compared to relying on golang proxy, and subsequent runs only deliver a marginal upside (but on humongous projects).
Cold starts issues may be alleviated if Athens would itself rely on golang proxy instead of using direct, though that would clearly take-away the benefit of not depending on their availability (but then again, using direct makes you depend on VCS).
That being said, hosting such a service on-premise (ie: as close as possible to where you build) would yield a clear upside if you have many concurrent builds (collocated large teams, or a busy build server), or networking limitations that would benefit from slashing down overall traffic.
These benefits would obviously have to be weighted against the cost of maintaining such a service.
If performance is the only angle, astute caching (for eg retrieving dependencies separately from building, or using go mod vendor) would likely give more bang for the buck — while running Athens alongside buildkitd would add a nice additional layer of resiliency and some additional (if marginal) speedup.
