A beginner’s guide to cross-compiling static CGO PIE binaries (golang 1.16), and other ways to hurt yourself purposedly

9 min readJul 30, 2021

TL;DR? Scroll down to the bottom

What kind of PIE? Apple pie?

Position Independent Code (PIC) describes code that has been built in a way that allows it to run properly regardless of location in memory. This is especially useful for shared libraries, so that they can be loaded in each app memory space.

PIE specifically describes executables that have been built entirely this way.

PIE further enables address space layout randomization, thwarting classes of exploits related to memory corruption issues (or at least making it significantly harder to exploit said vulnerabilities).

Major OS-es and distributions (macOS, iOS, Android, FreeBSD, Ubuntu, Debian, Fedora) have all enabled by default or started supporting PIE during the past decade, and a recent increase in interest for it (or more accurately, concerns over memory safety issues) have prompted gradual phasing out of non-PIE binaries.

As far as golang is concerned, building PIE has been supported since go 1.6, has been made the default on Windows with go 1.16, and is supported on an increasing number of platforms and architectures.

Savvy readers would evidently question the relevance of PIE in the go world. After all, isn’t golang managed memory model actually preventing the problems that ASLR claims to make harder to exploit?

This is true to some extent: if you build without CGO, it is going to require a lot of effort to produce go code with that kind of issue.

Now, if you do use CGO, you are linking in C code (whether third-party or libc, through netcgo for example, which, unfortunately, there are still a number of reasons to use), and it is still painfully obvious in 2021 that memory issues in C code are and will always be a major effing PITA, because, as has been repeatedly demonstrated, even the (self-described) best developers cannot be trusted to never make any mistake while holding a loaded gun standing on one foot on a stool on a banana peel wearing a pink pajama (read: writing convoluted C code managing memory manually).

In simple cases, to enable PIE when building your go app, all you have to do is pass `-buildmode=pie` in your `GOFLAGS`. But of course, the simple cases are also the cases where building with PIE does not matter much…

On static builds

The advent of containerization probably contributed to a renewed interest in building static binaries.

After all, the point of shared libraries was largely to reduce footprint on systems running many programs sharing code, and to ease maintenance and patching-in-place of said shared dependencies, for non-transient systems.

Of course, dynamic libraries (and binaries that use them) suffer from many issues stemming into the fact that said shared libraries may change out of band and break applications in (possibly not so) subtle ways (case in point: dependency hell). When shipping a binary that link dynamically to some library, there is nothing you can do as a developer but pray that it will be run with a truly compatible version of an untampered library it was meant to run with. After all, “promises [of backward compatibility] only bind those who believe in them” (<- yes, libpng, looking at you).

Now, in a world of (pretend) micro-services that (supposedly) run just one thing, where systems are treated increasingly as cattle rather than pets, of continuous deployment and constant rebuilding instead of patching-in-place, and of rapidly increasing storage and raw computing power, the downsides of shared libraries significantly out-weight their (questionable) benefits. Furthermore, out of concerns over attack surface, and optimization at scale, interest in running bare-bones workloads rather than full-blown operating systems (eg: single binaries in a “FROM scratch” container for example — or even Unikernels), have made building statically even more attractive.

While, without CGO, golang does produce static binaries by default, building statically with CGO has always been a little bit arcane. For some time, `-ldflags ‘-extldflags “-static”’ -tags ‘static_build’` was the 50% tribal way to go (* see below about the tribal part), but things got messier with go 1.15, when the linker used by go was switched to default to the go internal linker when using PIE, but only on certain platforms. This has tripped even seasoned developers. Put otherwise: magic is impressive for as long as you do not change the outcome of the trick overnight, depending on the rabbit’s fur color. If you do, it’s no longer magic, it’s just a garden variety mess.

To top it all, the `file` binary, that many rely on to check if a binary is static or not, has a notoriously bad history at operating properly with PIE binaries, and will seemingly randomly report PIE static binaries as either “shared object”, or “dynamic”, depending on `file` version (`ldd` is not always perfect either btw).

Finally, things get even messier with/without CGO, and if you add gccgo to the mix… it’s increasingly confusing for newcomers and veterans alike which compiler / linker is being used in what circumstances, and what kind of rabbit you will get out of it. And, of course, it’s not just go…

(*) many projects make use of the `static_build` tag. So, to build them statically, you do also have to pass this tag along (on top of linker flags). It’s not standard (yet): any random project could very well use a `static` tag instead, or something else entirely.

On cross-compiling (cgo)

Cross-compilation has historically been the privilege of a handful of people, and used to require significant efforts and intricate low-level knowledge (rolling your own cross compilation toolchain, to boot) — not to mention the fact that a large fraction of existing software was not portable at all to start with, or had (borked) build systems deeply tied to the host native toolchain.

Over the years, laudable efforts started to show-up, mostly out of the desire to be able to cross-compile for Windows (or even OSX, dare you) without having to touch the darn thing with a barge pole.

The advent of ARM further propelled cross-compilation into mainstream, up to the point even web developers started jonesing for it, and came-up with their own cross-compi^H trans-pilation so they too could solve the one problem they did not have in the first place — in a less comp-gendered way evidently.

Finally, Apple changing architecture every-time they feel like getting in a romp with their supply chain probably furthered this even more (culminating with the M1).

In 2021, good linux distributions do provide pre-packaged cross-compilation toolchains, and modern (compiled) languages have recognized the necessity to ship by default with a toolchain that can target different architectures. Rust — and golang, elegantly came-up with fantastic cross-compilation off the boot. Just set GOOS, GOARCH, and… magic time!

Mostly… Unless…

You use PIE… in that case, things work out of the box just on arm64 and amd64 — other platforms require an external linker from a cross-compilation toolchain that you have to setup yourself outside of go.

Or… unless…

You need CGO. In that case, you do need an external cross-compilation toolchain either way (compiler and linker), and PIE (finally really useful in that case) will just make things… significantly thornier…

On libc(s)

To compile statically with PIE, with CGO, you do of course also need the appropriate c runtime library (`rcrt1.o`).

Support for that was added in glibc 2.27, in 2018, but only for a few platforms. Even in June 2021, Debian Sid only ships it for amd64 and arm64.

It seems you are pretty much out of luck if you want to support armhf for example, and it is unclear why glibc would not add more platforms after over three years.

Furthermore, glibc has a rather long history of being not specifically (static) friendly, nor that interested in even acknowledging the relevance of other (architectures).

Alternatively, you can go with musl. Comparing libc implementations is out of scope here, though you might want to start reading this, this, and/or this. While musl is not as widely used as glibc evidently, and while there may be issues in building certain type of apps against it (mono was problematic last I checked, and more generally anything that relies on non-standard extensions provided by glibc), it is still a very reasonable choice, and a popular one amongst go nuts and other people looking for a modern, correct, static friendly libc.

Unfortunately though, musl does not use NSS, so, if your reason to use netcgo is mDNS, you are out of luck right now.

Finally note that if you build statically against glibc, and hope to use NSS via netcgo, just… stop… you are loosing your time.

TL;DR

If you do not need CGO:

and do NOT care about PIE
or if you want PIE but only care about arm64 and amd64

Do yourself a huge favor and keep it simple. Overall, remember that CGO is a great piece of tech, but also something you should really try to avoid. Just use go tools:

GOARCH=X go build
GOARCH=X go build -buildmode=pie

The PIE binary will be dynamic, the other one static.

2. If you really need CGO, but not netcgo

If you do not care about the distro you are building on, do yourself a favor here too and go with musl. Really.

If you cannot, glibc will still work, but mind the rabbit hole when things will go haywire (they will).

Typically, you can get a cross-compiled non-PIE static binary with:

dpkg --add-architecture X
apt-get update
apt-get install crossbuild-essential-X libc6-dev:Xexport GOARCH=X
eval "$(dpkg-architecture -A "$(echo "$GOARCH" | sed -e "s/arm$/armhf/" -e "s/ppc64le/ppc64el/" -e "s/386/i386/")")"
export CC=${DEB_TARGET_GNU_TYPE}-gcc
export CGO_ENABLED=1go build -tags "cgo netgo osusergo static_build" -ldflags "-linkmode=external -extldflags -static"

Or a static PIE one — only on arm64 and amd64:

# Use the same preamble as above ^ then:
go build -buildmode=pie -tags "cgo netgo osusergo static_build" -ldflags "-linkmode=external -extldflags -static-pie"

Other platforms have to choose between static (above), or PIE:

# Use the same preamble as above ^ then:
go build -buildmode=pie -tags "cgo netgo osusergo"

Finally, since you are doing CGO, you should really take a serious look at how to harden your binary — linker options (relro, now, defs, noexecstack) and compiler / preprocessor options (-fstack-protector-strong, -fstack-clash-protection, -D_GLIBCXX_ASSERTION, -D_FORTIFY_SOURCE=2).

3. If you do really need netcgo

Probably because you really need NSS.

You are stuck with glibc (as previously mentioned, musl does not do NSS — although, an idea would be to run a localhost patched coreDNS that would provide the resolution you need).

And there is no known way to get a static binary in that case (PIE or otherwise).

This here is still a problem six years in (or should we say, twenty years in). It has been rehashed, over and over again. The resulting binary of compiling statically with netcgo against glibc will SIGSEGV.

What you can do, though, is to compile statically everything but glibc:

go build -tags "cgo netcgo osusergo" -ldflags "-linkmode=external -extldflags '-static-libgcc -static-libstdc++'"

This last approach is probably the best trade-off, since, anyway, even with a fully static binary (that would work), you would still need to ship alongside your binary the dynamic (exact same) version of glibc to use NSS… Mind blowing, right?

Take-away

NSS is a venerable (as in: old enough to buy property in Florida) piece of… tech. It’s of course controversial, and there are good reasons why “obviously musl does not have (or want) NSS”. But then, no out-of-the-box support for mDNS for example is hard to defend.

glibc is even harder to defend nowadays though. Of course, it has been instrumental in the past twenty something years. But then, a wet glitter-stripped doormat, in its place, would have been instrumental just the same. The glory was in the function, not the essence.

There are clear, blatant reasons why modern stacks (like go) chose to do away with it (pretty much the same reasons they do away with openssl).

Then, there are still important pieces of tech written in C that have yet to be ported over to something that does not ^$!$`+/ù (and doing so will be costly). In that context, the ability to use CGO is both god-send and a curse. And one thing should be said about the go toolchain overall: it’s way too easy to produce broken binaries. I’m sure they are working on that, but right now, if you are poking the CGO-bear — be sure to watch your six. Hope that helps.

Notes, further reading, errata

Interesting reads

Credit trailer: auditing binaries

Use readelf — from package binutils — specifically -p .interp and -d.

File can still provide some information (if the version you have is not garbage), though you can get the same in a more reliable way out of readelf.

ldd will let you down.

Also have a look at hardening-check (from devscripts).