Home Wiki > openSUSE:Packaging Go
Sign up | Login

openSUSE:Packaging Go

tagline: From openSUSE


The Packaging Go is a step by step introduction on how to build Golang software packages for openSUSE using the openSUSE Build Service.

About golang

Go (or golang, more search engine friendly) is an expressive, concurrent, garbage collected systems programming language that is type safe and memory safe. It has pointers but no pointer arithmetic. Go has fast builds, clean syntax, garbage collection, methods for any type, and run-time reflection. It feels like a dynamic language but has the speed and safety of a static language.

Naming scheme

Please note that golang packages follow a strict naming convention, with exceptions for convenience.

Golang packages' names begin with a prefix "golang-", follow a string modified from $IMPORTPATH, which is the "url" (actually it's a path) when you `go get`. Technically $IMPORTPATH looks like the "include <xxx.h>" in C, "require 'xx'" in Ruby or "from package import function" in Python, but in a format "import golang.org/google/api". In this case "golang.org/google/api" is the $IMPORTPATH.

It can be taken as three parts: vendor => golang.org, author => google, pkg => api. Here's how we modify it to the following string in golang packages' names:

vendor: usually we use the common known name of the vendor, eg. "github.com" will be "github" and "code.google.com/p/" will be "googlecode". When there's no common known name for a vendor, eg. "gopkg.in", we just skip the "dot" in the vendor url, that is "gopkgin".

author: usually this part is unique enough, most of them are people names, so we just keep it as it is. And it can also be skipped in some cases. eg: "code.google.com/p/xxx", there's no author in the importpath at all (not the "p"). But in other cases it must be explicitly kept, eg "golang.org/x/net", the "x" here means that it's some kind of "standard libraries", so it must be kept. And the "nsf" in "github.com/nsf/gocode" can't be skipped at all.

pkg: usually we just keep it as it is. But when there's a string "go" somewhere in the upstream package name:

  • go-XX: skip the "go", we don't want to see package name like "golang-github-xxx-go-xxx", the "golang-" prefix indentifies it already
  • XX.go: this is old history back to Go1 era. just skip the "go"
  • goXX: keep the "go" because it is "in" the package name, not prefix or suffix.

Versioning scheme

Most golang packages did not follow a strict version scheme yet. You should use the date when you retrieved the upstream (e.g. via git/mercurial/etc. checkout) plus the VCS used, like 0.0.0+gitYYYYMMDD, eg 0.0.0+git20150726, or you can let our Build Service do it for you via a _service file. If the upstream maintainers decide to start using a version scheme, this ensures easy package upgrades in the future.

Development repository & process

Golang packages for openSUSE are developed at devel:languages:go repository in the openSUSE Build Service.

If you want to submit a package to this repository, please:

  • branch it (easy way: visit this repository and click "branch" button for any existing package inside this repository)
  • copy your package from your home repository to the branched repository
  • build
  • submit

This will ensure that your package will build for this repository.

Things you should know before packaging

Golang do have dependencies

Golang provides a method called "go get", which is commonly used by upstream. It will fetch the source code of the package plus source codes for all the dependencies, build them in /tmp, and install the compiled codes in ~/go/pkg/linux_${go_arch}/. (a combination of download, `go build` and `go install`.)

So just like other new scripting languages, e.g. Nodejs, it will confuse new packagers with dependencies. Some may think this package has no other dependencies except the "go" main package, run into this mess, and find it has about 30+ dependencies to package first. Please check your ~/go directory first before asserting that this package has no other dependencies and running into chaos.

Where will Golang find dependencies?

Golang will find dependencies from $GOROOT (where go itself installs) and $GOPATH (where go packages installs). You can run:

go env

to find out yours. Basically, inside the build infrastructure, it will be:

/usr/lib(64)/go/pkg/linux_%{go_arch}/ ($GOROOT)
/usr/lib(64)/go/contrib/pkg/linux_%{go_arch}/ ($GOPATH)
/home/abuild/rpmbuild/BUILD/go/pkg/linux_%{go_arch}/ ($GOPATH)

Within plain environments, it would be:

/usr/lib(64)/go/pkg/linux_%{go_arch}/ ($GOROOT)
/usr/lib(64)/go/contrib/pkg/linux_%{go_arch}/ ($GOPATH)
~/go/pkg/linux_%{go_arch}/ ($GOPATH)

But in the build log, it will be different because the compilers of go (such things like 6g 6l...) will:

1st: try to find the compiled dependencies in the above path

2nd: try to find source code of the dependencies in the paths mentioned below, compile them, install them to the $GOPATH of the user operating go, then use them as dependencies.

Within the build infrastructure:

/usr/lib64/go/src/pkg/$IMPORTPATH ($GOROOT)
/home/abuild/rpmbuild/BUILD/go/src/$IMPORTPATH ($GOPATH)
/usr/lib64/go/contrib/src/$IMPORTPATH ($GOPATH)

Within plain environments:

/usr/lib64/go/src/pkg/$IMPORTPATH ($GOROOT)
~/go/src/$IMPORTPATH ($GOPATH)
/usr/lib64/go/contrib/src/$IMPORTPATH ($GOPATH)

3rd (only able in `go get` mode): download source code to the $GOPATH of the user operating go and repeat the 2nd step.

So if OBS gives you an error, it usually happens at the 2nd step because you carelessly used the `go get` method to replace the default %gobuild macro, which will show such errors like "can't find git" or "network not reachable".

The Go layout

The standard directory layout for go looks like this:

  go
    .. pkg
       .. linux_%{go_arch}
          .. $IMPORTPATHs for packages (compiled codes reside here)
    .. src
       .. pkg
          .. $IMPORTPATHs for packages (source codes reside here)
    .. contrib
       .. pkg (same as above)
       .. src (same as above)

The contrib directory is used for 3rd-party implementations (usually our packages). But in ~/go, Go will place compiled codes in pkg and source code into src directly (no contrib directory at all).

Example template

You can use the following spec file template. Make sure to replace $VARIABLES with appropriate values.

#
# Copyright (c) 2015, $YOUR_NAME <$YOUR_MAIL>
#
# All modifications and additions to the file contributed by third parties
# remain the property of their copyright owners, unless otherwise agreed
# upon. The license for this file, and modifications and additions to the
# file, is the same license as for the pristine package itself (unless the
# license for the pristine package is not an Open Source License, in which
# case the license is the MIT License). An "Open Source License" is a
# license that conforms to the Open Source Definition (Version 1.9)
# published by the Open Source Initiative.
 
# Please submit bugfixes or comments via http://bugs.opensuse.org/
# 

%{go_nostrip}

Name:           golang-$STRING
Version:        0.0.0+git20140916
Release:        0
License:        $LICENSE
Summary:        $SUMMARY
Url:            $HOMEPAGE
Group:          Development/Languages/Other
Source:         $EXACT_UPSTREAM_NAME-%{version}.tar.xz
BuildRequires:  golang-packaging
BuildRequires:  xxx-devel
BuildRequires:  xz
Requires:       xxx-devel
BuildRoot:      %{_tmppath}/%{name}-%{version}-build
%{go_provides}
 
%description
$DESCRIPTION
 
%prep
%setup -q -n $EXACT_UPSTREAM_NAME-%{version}

%build
%goprep $IMPORTPATH_NAMESPACE
%gobuild $IMPORTPATH_MODIFIER

%install
%goinstall
%gosrc

%gofilelist

%check
%gotest $IMPORTPATH_NAMESPACE

%files -f file.lst
%defattr(-,root,root,-)
%doc README LICENSE

%changelog

The complicated explanation

Macros

You can look up the provided macros in the file /etc/rpm/macros.go (when the golang base package "go" is installed).

Some common macros are explained below (But not all of the macros, because some macros like %go_exclusivearch are for internal use and have already been included in other macros).

NOTE: Some macros can not be quoted

Usually, you already have had an idea that %make equals to %{make}, just like shell scripts.

This rule applies to almost every aspect of RPM packaging. But in golang packaging you have to eliminate such careless ideas from your mind.

As I know, %goprep, %gofix, %gobuild, %goinstall, %gosrc, %gotest are macros that can not be quoted because they allow arguments. So please double-check after you run "spec-cleaner -i xxx.spec", which will quote such macros.

%go_ver

It will return golang's version like 1.4.2

%go_arch

It will return golang's architecture: i386 or amd64.

%go_nostrip

It will disable stripping of the binary, this causes quite often issues related to tools written in Go.

%go_requires

Removed in golang-packaging v15.0.0.

It will only Requires go main package for your package.

Warning If your package needs some other packages to build, please be sure to manually add them to Requires as well as BuildRequires. Golang is static build, the common shared library detection mechanism (/sbin/ldconfig) will not find dependencies for it. That is, %go_requires itself is not enough most of the times.

%go_provides

It will add Provides %{name}-devel and %{name}-devel-static for your package.

Unlike C/C++ packages, Golang packages do not have header files. They are statically built so the main package is also the devel package.

%go_recommends

Removed in golang-packaging v15.0.0.

It will add Recommends %{name}-doc sub-package for your package.

Unlike the common RPM packaging, %{name}-doc is actually the source package in golang, instead of documentation packages.

%godoc_package

Removed in golang-packaging v15.0.0.

Like %lang_package, it will create a %{name}-doc sub-package in specfile.

%go_contribdir

go binaries/static libraries will be installed into this directory:

%{_libdir}/go/contrib/pkg/linux_%{go_arch}

%go_contribsrcdir

go sources will be installed into this directory:

%{_datadir}/go/contrib/src/pkg

%goprep

This macro will prepare the build environment for golang.

You have to give an argument $IMPORTPATH to it. If you have seen some golang specfiles, you will find something like this:

%goprep gopkg.in/qml.v1
%goprep github.com/howeyc/fsnotify

These gopkg.in/qml.v1 or github.com/howeyc/fsnotify are actually not URL but PATH, although http://gopkg.in/qml.v1 is reachable (Actually it's the homepage for this library).

%goprep will create a %{_builddir}/go/src/gopkg.in/qml.v1 directory.

Then what's the purpose for this $IMPORTPATH and how to find it?

It is used for importing. eg. in golang code:

import (
      qml gopkg.in/qml.v1
)

And golang will find such library in

%{go_contribsrcdir}
~/go/pkg/linux_amd64/
/home/abuild/rpmbuild/BUILD/go/pkg/linux_amd64/ (This only exists in OBS build environment, that is, ~/rpmbuild/BUILD/go/pkg/linux_amd64/ will not count unless you're building inside it)

And you can see the examples provided by upstream to easily find the $IMPORTPATH in source tarball.

%gobuild

It actually builds and installs the compiled golang codes into %{_builddir}.

By default it will build everything inside %{_builddir}/go/src/$IMPORTPATH, if you want to selectively build some sub-directories inside $IMPORTPATH, please install "go" package and read /etc/rpm/macros.go for $IMPORTPATH_MODIFIER arguments' usage.

%goinstall

It will copy the compiled codes from %{_builddir} to %{buildroot} for RPM package creation.

%gosrc

It will scan the source directory and copy all *.go, *.s, and *.h files to %{go_contribsrcdir}.

%gofilelist

It will scan %{go_contribsrcdir} for files and directories, and generate a file (file.lst) listing each. This file can then be used for the %files macro, see here.

%godoc

No-op in golang-packaging v15.0.0.

It will scan the source directory and copy all *.go files to %{go_contribsrcdir} to create a %{name}-doc package, just like %find_lang but without arguments.

%gotest importpath

It will test all golang codes inside importpath directory.

Warning It will test everything including examples. But one common problem is that upstream sometimes only fix/update the functional codes instead of examples (Debhelper has a function to skip testing examples so upstream may be not able to find problems at all). Some examples may not compile against newer golang or library (It triggers lots of build errors on OBS). It's your choice to disable %check or fix the examples (or even "rm -rf" them). But please be sure not to leave codes that couldn't run for our users.

The changes of BUILD directory layouts

Warning RPM macros for go packaging will distroy the current directory layout in %{_builddir}, so if you do things after the %goprep macro, you will not know the BUILD directory layout unless you read the /etc/rpm/macros.go definitions or this help. ALways keep this in mind.

There're more than one derivatives for this case:

  • If you got a "File or directory not found" error and you want to do a "ls -l" in sections like %install.
  • If you want to selectively put files into %{name}-doc packages instead of putting everything upstream provides. This usually involves file addition/deletions, which further suppose you know the directory layout of the folder you are operating under.

Why?

Golang is a structure sensitive language. It needs such layouts to build and place build results.

Layout after %goprep

After running %goprep, an additional Go-specific directory structure is created. All files from BUILD/%{name}-%{version} are then be copied to the new directory BUILD/go/src/$IMPORTPATH.

Although RPM will still cd into BUILD/%{name}-%{version} when reaching the sections %build or %install, this new directory structure is necessary for the package itself and its dependencies. It will also be set in the GOPATH environment variable which in turn is used when running %gobuild.

Layout after %gobuild

While after running %gobuild, you package will be installed as

BUILD/go/pkg/linux_%{go_arch}/xxx/xxx/%{name}.a

or

BUILD/go/bin/%{name}

Layout after %goinstall

After running %goinstall, the package will be installed to

BUILD/usr/bin/

Static libraries (*.a files) are not installed, and therefore won't be part of the shipped package.

Layout after %gosrc

After running %gosrc, your package source will be installed into

BUILDROOT/%{_datadir}/go/contrib/src/pkg/xxx/xxx/%{name}/

It will copy *.go files, as well as *.s and *.h files. Should any other files be required, these have to be copied manually into this path.

There is no %{name} directory in %{go_contribdir}

From previous chapters we know xxx/xxx/%{name} is not a URL but $IMPORTPATH.

So some people may try to "pushd BUILDROOT/%{go_contribdir}/xxx/xxx/%{name}" and then do "ls -l" or "find" to see what's wrong.

But there'll be no such directory in BUILDROOT/%{go_contribdir}/. Actually only xxx/xxx exists, %{name}.a is actually a static compiled file. And if the source has subdirectories, they will be compiled as xxx/xxx/%{name}/<subdirectory-name>.a or xxx/xxx/%{name}/<subdirectory-name>/<some-other-name>.a. But anyway:

xxx/xxx/%{name}/*.go will be compiled to xxx/xxx/%{name}.a

That is, you will never see anything related even if you can "cd" into the %{name} directory after build.

NOTE: this only applies to BUILDROOT/%{go_contribdir}, BUILDROOT/%{go_contribsrcdir} still has the %{name} directory containing the *.go files.

Dependencies (Requires)

Golang packages are "static" builds, so shlib resolving tool (/sbin/ldconfig) does not work.

Easy cheat sheet

Remember this: Your BuildRequires is also your Requires.

It may sound weird, but to some extent Golang is like Python. You can run Python codes in binary-compiled code (*.pyc) or source code (*.py), and you can run Golang codes in binary-compiled code (*.a) or source code (*.go) too. (But unlike Python, "go run" can't run compiled codes.)

So what you need in your build environment will be what you will need for you runtime environment.

Don't be fooled by the ".a" suffix. In C this means static, you don't need to install shared lib dependencies for it then. But in Golang, it will not absorb all the dependencies. It will not absorb the C dependency codes like those from GTK either.

Learn the hard way

"Pure" Golang packages

If you skip the essential Requires (usually are those essential to build the package, eg. go-log4go for lime), it will cause an error when the user runs the package:

test.go:2:8: cannot find package "code.google.com/p/log4go" in any of:
        /usr/lib64/go/src/pkg/code.google.com/p/log4go (from $GOROOT)
        /home/marguerite/go/src/code.google.com/p/log4go (from $GOPATH)
        /usr/lib64/go/contrib/src/code.google.com/p/log4go

This can be fixed by providing Requires and BuildRequires:

Requires:      golang(code.google.com/p/log4go)
BuildRequires: golang(code.google.com/p/log4go)

Some errors however might seem a bit more complicated to solve:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: ([]interface {}) (0x64c540,0xc208045cc0)
[signal 0xb code=0x1 addr=0x0 pc=0x4165ba]

...
[ follwing with a very looong "goroutine" ]
...

In this case, simple tests on your local can help locating the problem:

  • For a library: You can just download the source from upstream and run go run xxx.go to see where it fails.
  • For an application: Just uninstall all Golang packages except the "go" base package and run your application. This way you can catch all the runtime dependencies.

Golang wrapper packages for C/C++

Golang wrappers for C/C++ libraries, like go-gozmq for zeromq, will use pkg-config to find its runtime C/C++ dependencies, which means if you do not have "pkg-config" and "zeromq-devel" (which provides /usr/lib64/pkgconfig/zeromq.pc) installed, gozmq will not able to detect libzmq4 library even if you have it installed. When you run examples, it will complain:

$ go run client.go 
# command-line-arguments
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lzmq
collect2: error: ld returned 1 exit status

For such Golang wrappers it may therefore be necessary to install a *-devel package in order for the package to build.

Common Golang errors

This section collects some common Golang errors for those packagers who know little about Golang.

undefined reference to SOME_C_FUNCTION

Golang can call functions written in pure C with a tool named cgo. If you see

/*
#include <stdio.h>
*/
import "C"

in a .go file, this file calls some C functions. As a qualified packager, we all more or less have some ideas about CFLAGS and LDFLAGS, thus know "undefined reference" error means the compiler couldn't find related header files or *FLAGS in C. But how could we fix such errors in Golang? Here it is:

  • fix for headers are pretty easy.
  • Here's how to fix CFLAGS and LDFLAGS:
/*
#include <stdio.h>
#cgo pkgconfig: geoip
#cgo LDFLAGS: -lGeoIP
#cgo CFLAGS: -xxxxxx
/*

main redeclared in this block

Golang is a structure sensitive language. If a folder contains multiple files and those files have more than one "main" function, this error triggers. But some old go projects (eg. since Golang 1.1) still didn't put their examples files in separate folders like "examples". We know examples are some small, independent demos relying on this project, which must have a separate "main" function. so the fix is easy, just "mkdir -p" an "examples" folder and put all such files into it in %prep section in specfile (see #Layout after %goprep chapter).

object is [linux amd64 go1.3 X:precisestack] expected [linux amd64 go1.3.1 X:precisestack]

This error means your package is built against go 1.3, but your current Golang environment is go 1.3.1. Usually a rebuild will solve the problem.

stat "bitbucket.org/taruti/ssh.go" No such file or directory

It's because "*.go" is no longer allowable for import path in new versions of Go.

So in this case, you need to change

%goprep bitbucket.org/taruti/ssh.go

to

%goprep bitbucket.org/taruti/ssh-go

and mention that in .changes file so developers using our repository will know the change.

But usually it indicates this project is too old and you should find newer implementations if upstream didn't adapt to the change themselves. In the above case, you should use "code.google.com/p/go.crypto/ssh" which is newer and official.

Packaging tricks for Go

"tags" in go get

Sometimes upstream e.g. gozmq told you to "go get -tags xxx $IMPORTPATH" to get a specific version , and they put files for all tags together in one repository. What's worse, newer tags like "zmq_4_x" needs files from older tags like "zmq_3_x" and "zmq_2_2" to build (Maybe that's why upstream didn't split files into different git branches). Plus the "*_test.go" files, they totally get us confused about which file can be deleted for the build of a specific version.

So we can't simply deleted "zmq_3_x" files for "zmq_4_x" builds, and considering the "*_test.go" files, it'll be a total mess for us to use "%{?suse_version}" to delete files (illogic, changeable all the time). All in all, "go get -tags" is the perfect, most simple, and (maybe) only possible way.

In this case, we need to manually "construct" a "go get" to replace %gobuild. As #Where will Golang find dependencies? shows, "go get" will try to find sources in $GOPATH before actually downloading online (the later will certainly break because OBS build VM doesn't have any network), it indicates that if we put sources at the right place (%goprep) and export the correct envrionment variables, "go get" is usuable on OBS. So we can change commands in specfiles like this:

%build
%goprep github.com/alecthomas/gozmq
%gobuild
%install
%goinstall

to

%build
%goprep github.com/alecthomas/gozmq
# manual build
export IMPORTPATH="github.com/alecthomas/gozmq"
export BUILDFLAGS="-s -v -p 4 -x"
export GOPATH=%{_builddir}/go:%{_libdir}/go/contrib
%if 0%{?suse_version} >= 1230
%if 0%{?suse_version} > 1315
go get --tags zmq_4_x $BUILDFLAGS $IMPORTPATH
%else
go get --tags zmq_3_x $BUILDFLAGS $IMPORTPATH
%endif
%else
go get --tags zmq_2_1 $BUILDFLAGS $IMPORTPATH
%endif
%install
%goinstall

$GOTOOLDIR is not rewritable

Sometimes when building Go official packages e.g. go.tools, some binraries like "godoc" will force to install themselves into %{_libdir}/go/pkg/tool/linux_%{go_arch}, even if you have recontructed the %gobuild macro and exported GOTOOLDIR to %{_builddir}/go/pkg/tool/linux_%{go_arch}.

This is intentional because Go upstream doesn't want official tools like godoc to be separately updated without the rest of go tools. But as go.tools is separately built from go main package, such decision will trigger "open /usr/lib64/go/pkg/tool/linux_amd64: Permission Denied" error on OBS.

We patched go main package and provided a macro called "GOROOT_TARGET". If you:

export GOROOT_TARGET="%{buildroot}%{_libdir}/go"

before %gobuild, you can get this workaround.

Team members

External links