openSUSE:Build Service private instance boot strapping
Summary
A private instance of an OBS is likely to need to operate in a fully independent way for some projects - and even for collaborative deployments it can be a useful facility in the event of connectivity failure.
A normal development project/package in an OBS needs another project to act as a target; this project should contain all the binary packages required for the build root.
This guide talks about the various approaches to setting up a project on a local OBS to act as a build target.
You may also feel the need to (re)build this project from source; this bootstrap problem is also covered.
The Approaches
Repository Binary Import
This approach imports pre-built binary packages from an existing repository into a project. It does still require a prjconf be obtained but it can also be used where you have only access to the download repository.
This is the simplest way to build packages against a stable base where you do not need or want to rebuild the packages locally.
Repository Binary Download On Demand (DOD)
This approach imports the binary file as needed, into the :full directory of the project.
#Use_Repository_Binary_Download_On_Demand_.28DOD.29
OBS Package copy
This approach leads a bootstrap activity, and is performed to build the binaries locally.
If you copy your reference distro from an OBS like OpenSUSE (https://build.opensuse.org/) or the MeeGo community OBS (https://build.pub.meego.com/), obtaining a login is easy and you should prefer this method.
OBS Remote Link
If there is an OBS providing a remote link to the project you require, then this approach provides a simple solution.
This approach is useful when you don't have a login to the remote OBS (eg https://build.meego.com/).
Remote link can be used in a project configuration (osc meta prj PRJNAME) or in a build target definition.
you need to be Admin to create a remote link project.
Syntax for a remote link is
<remoteurl>https://<Mylogin>:<Mypasswd>@api.opensuse.org/</remoteurl>
Note public API do not need a login:password
<remoteurl>https://api.opensuse.org/public</remoteurl>
Remote link project "Raw Config/Meta" sample:
<project name="openSUSE.org">
<title>build.openSUSE.org</title>
<description>This project allows access to all resources on build.opensuse.org OBS instance. This is needed to build against other distributions.</description>
<remoteurl>https://api.opensuse.org/public</remoteurl>
<person userid="aUser" role="maintainer"/>
<person userid="aUser" role="bugowner"/>
</project>
Repository Source Import
If you have access only to the repositories of your source reference target, then your life will be a bit more difficult. My advice would be to recheck if you find you cannot get a login on a public OBS service - such as provided by openSUSE or MeeGo - before proceeding this way.
As mentioned you will not be able to import the project config and you will have to create it by hand. This is too long to be covered in this HowTo.
more on project config
Note: This approach should be used as a last resort.
The bootstrap issue
Note: Each approach method can be used on the same project.
toolchain
In order to rebuild a standalone project, your OBS needs to first build its full reference toolchain.
This process - called Bootstrap - presents the same problem as the Chicken and the Egg, which one came first !
Toolchain binaries are provided by:
Source can be provided by:
In the steps outlined below the approach is to first copy over the source packages and then to seed the project with binary packages. Then rely on the OBS rebuild facility to rebuild and overwrite the binary packages.
A Clean Bootstrap Technique
If you want a complete standalone offline OBS Bootstrap use:
#Repository_Binary_Import + #OBS_Package_copy
If you can't do this, at first do:
(#OBS_Remote_Link or #Repository_Binary_Download_On_Demand_(DOD)) + (#OBS_Package_copy or #Repository_Source_Import)
and after you will be able to do:
#Repository_Binary_Import + #OBS_Package_copy
A step-by-step guide
Naming convention
PROJECT=Meego-test:1.2 PROJECT_SOURCE=MeeGo:1.2.0:oss REPO=standard ARCH=i586 OBSHOST=http://myprivateserver.meego.com:81 OBSHOST_SOURCE=https://api.meego.com Login="The OBS login" Pwd="The OBS passworld"
Note: We assume you have an account (Login/Pwd) to the OBSHOST and eventually OBSHOST_SOURCE.
Note: "standard" is the default name of your Build repository as defined in your project . It might change depending of who created the initial build repo.
Note: We assume that you have installed the 'osc' package on your workstation.
Note: 'osc' does not support public host like 'https://api.meego.com/public'.
Very Important Note
Note: That a project used as a build target must have a 'prjconf' to define what packages are put into the build root. If you are not using an OBS built repository then you must generate a suitable one (possibly basing upon an existing openSUSE OBS project). This is a complex topic - (more on project configs)
How to initialize a project config
The easy way is to initialize from a existing project.
osc -A $OBSHOST_SOURCE meta prjconf $PROJECT_SOURCE > my_project.conf
If you have not access to the $OBSHOST_SOURCE, you also can use 'curl'.
curl $OBSHOST_SOURCE/public/source/$PROJECT_SOURCE/_config > my_project.conf
Now you can import your 'Project Config'.
osc -A $OBSHOST meta prjconf -F my_project.conf $PROJECT
Of course you also can use the web UI to copy/paste the project config.
"$PROJECT>Advanced>Project Config>project configuration"
Setup your OBS
Log on to the Web API See starting your OBS Appliance
The default user Admin, with the password opensuse is available. Create your own login and password and set yourself as Admin.
Log on to the Web UI as Admin and click on the Icon "Setup OBS". Accept the default. It should work.
Create your first repository project
Create by web UI
A project at the root of OBS.
Select from under 'Locations' -> 'Projects'. At the top left click on 'Create a new project '. Give it a name (e.g. Meego-test)
Enter a 'ProjectName'(Meego-test:1.2), 'Tilte'(Meego-test 1.2), 'Description'.
you need to be 'Admin' to create a project at the root of OBS.
A sub project.
Select a Project. Click on 'Actions->Create subproject'
Creating a project as a sub project simplifies the user and project administration.
Enter a 'ProjectName' (1.2), 'Title' (Meego-test 1.2), 'Description'.
Here 1.2 is a sub project of Meego-test
you need to be 'maintainer' of the project to create a sub project.
Configure the project.
You have now a project Meego-test:1.2.
Go into the 'Advanced'->'Raw config/Meta' of the webui and add a <repository> section containing the architectures (inside <project>, usually the last entry).
<project name="Meego-test:1.2"> <title>Meego-test 1.2</title> <description>A Description</description> <person role="maintainer" userid="obsuser"/> <person role="bugowner" userid="obsuser"/> <repository name="standard"> <arch>i586</arch> <arch>armv8el</arch> </repository> </project>
Note: You can have more than one architecture <arch>armv8el</arch>
Create by command line
osc -A $OBSHOST meta prj -F - $PROJECT << OTHEREOF <project name="$PROJECT"> <title>A Title</title> <description>A Description</description> <person role="maintainer" userid="$Login"/> <person role="bugowner" userid="$Login"/> <repository name="$REPO"> <arch>$ARCH</arch> </repository> </project> OTHEREOF
Import your base Linux project
We are now going to import the base project. We will describe the five approaches outlined above.
Use Repository Binary Import
Prepare the OBS Server
In the special case where we want to seed binaries into a project which has sources, you must stop the scheduler as it will create a mess if it starts to build and binaries change underneath it. (Note, use 'shutdown' instead of 'stop' in versions pre-2.3 of OBS as it does a clean shutdown).
rcobsscheduler shutdown
Binary Import from the web
The stable repository (or a snapshot) is imported to the backend repository server (bs_repserver) machine.
- The binaries are copied to a staging area (not directly into /srv/obs).
mkdir -p /data/imports
cd /data/imports
use the rsync command:
rsync -a --progress rsync://mirrors.kernel.org/meego/builds/1.2.90/1.2.90.0.0.20110824.2/repos/ meego_1.2.90.0.0.20110824.2
use the wget command:
wget --directory-prefix=/data/imports --reject index.html* --mirror --no-parent --no-host-directories --cut-dirs=8 http://repo.meego.com/MeeGo/releases/1.2.0/repos/oss/ia32/packages/
- Now create the â:fullâ directory.
mkdir -p /srv/obs/build/$PROJECT/$REPO/$ARCH/:full
cd /srv/obs/build/$PROJECT/$REPO/$ARCH/:full
- Add binaries to the :full directory of the Project
WARNING: Following snippet wrongly strips some parts of the packages. For example different yast rpms gets renamed to yast2.rpm.
# sorted, so that later versions overwrite earlier versions, if any
for i in `find /data/imports/ -name \*.rpm | sort`;
do
j=`basename $i`
k=`echo $j |sed -rn 's/(.*)-.*-.*.rpm/\1.rpm/p'`
mv $i /srv/obs/build/$PROJECT/$REPO/$ARCH/:full/$k
done
Instead of above snippet, something like following can be used to perform the rename. To use it, you need to adjust hardcoded parts of the paths. If you modified this command to be more universal, please contribute it here and remove the original for above.
find . -type f -name "*-*.*.rpm" | perl -pe 'print $_; s/-[\w\.\+\~]*-[\d\.]+\..\w+\.rpm$/.rpm/' | xargs -n2 mv
Remember to change owner of the files:
chown -R obsrun:obsrun /srv/obs/build/$PROJECT
After packages have been transferred, the temporary directory can be removed.
rm -r /data/imports
Note: These RPMs should have the release and version numbers stripped from them. e.g. alsa-utils-1.0.22-2.7.i586.rpm -- should be â alsa-utils.rpm
Binary Import from an other project
mkdir -p /srv/obs/build/$PROJECT/$REPO/$ARCH/:full
cp /srv/obs/build/$PROJECT_old/$REPO_old/$ARCH_old/\:full/* /srv/obs/build/$PROJECT/$REPO/$ARCH/\:full/
chown -R obsrun:obsrun /srv/obs/build/$PROJECT
- Add binaries to the :full directory of the Project. Change all user/group privileges under /srv/obs/build/ to âobsrunâ. If you leave root as owner of :full, it will still build but the scheduler will fail (almost silently) to upgrade :full with the latest built packages. Except in very special cases, it is very unlikely that you want to do so.
chown -R obsrun:obsrun /srv/obs/build/$PROJECT
Initialize OBS
- If the scheduler is not running, start it now.
rcobsscheduler start
Note: Don't do a 'restart' as that puts a shutdown event in the scheduler queue; then, when the scheduler starts, it sees the event and immediately stops. Use shutdown/start instead.
- The back-end is told about the imported packages. This will send an event to the scheduler which will re-index your new ':full' directory and create a file named :full.solv
/usr/lib/obs/server/bs_admin --rescan-repository $PROJECT $REPO $ARCH
This project is now ready to roll, you can start to build against it.
Note :When the project is ready, You can "officially" add it to the OBS server .openSUSE:Build_Service_private_installation#Add_Repositories_targets
Use Repository Binary Download On Demand (DOD)
Modify the 'Raw config/Meta' of project
<project name="$PROJECT"> <title>A Title</title> <description>A Description</description> <person userid="obsuser" role="maintainer"/> <person userid="obsuser" role="bugowner"/> <download arch="$ARCH" metafile="primary.xml" mtype="rpmmd" baseurl="http://repo.meego.com/MeeGo/releases/1.2.0/repos/oss/$ARCH/packages/"/> <download arch="armv8el" metafile="primary.xml" mtype="rpmmd" baseurl="http://repo.meego.com/MeeGo/releases/1.2.0/repos/oss/armv7hl/packages/"/> <useforbuild> <disable/> </useforbuild> <build> <disable/> </build> <publish> <disable/> </publish> <repository name="$REPO"> <arch>$ARCH</arch> <arch>armv8el</arch> </repository> </project>
Note: As you can have more than one architecture <arch>armv8el</arch>, you can have more than one download link.
Download the primary.xml file
Create the directory
mkdir -p /srv/obs/build/$PROJECT/$REPO/$ARCH/:full cd /srv/obs/build/$PROJECT/$REPO/$ARCH/:full
Remove the old file if existing.
rm -f *primary.xml*
Download the primary.xml file and change owner.
wget -q -c http://[REPO_URL]/repodata/c843450ac61964516fa1f83477573da3d61596f243884c9f40b0dcef5f5403f5-primary.xml.gz mv *-primary.xml.gz primary.xml.gz gunzip primary.xml.gz chown -R obsrun:obsrun /srv/obs/build/$PROJECT
Note: If the primary.xml file changes on the server, you must update your own file.
cd /srv/obs/build/$PROJECT/$REPO/$ARCH/ rm \:full.solv cd \:full wget -q -c http://[REPO_URL]/repodata/c843450ac61964516fa1f83477573da3d61596f243884c9f40b0dcef5f5403f5-primary.xml.gz mv *-primary.xml.gz primary.xml.gz gunzip primary.xml.gz chown -R obsrun:obsrun /srv/obs/build/$PROJECT obs_admin --rescan-repository $PROJECT $REPO $ARCH
Initialize Project Conf for (DOD)
Initialize Project Conf for OBS Package Copy
see #How_to_initialize_a_project_config
load prjconf, this causes the scheduler to generate the :full.solv files
This project is now ready to roll, you can start to build against it.
Note :When the project is ready, You can "officially" add it to the OBS server .openSUSE:Build_Service_private_installation#Add_Repositories_targets
Use OBS Package Copy
How to use "osc copypac"
The osc copypac has an option -t which enables copying towards a remote target OBS instance.
#osc copypac --help
copypac: Copy a package
A way to copy package to somewhere else.
It can be done across buildservice instances, if the -t option is used.
In that case, a client-side copy and link expansion are implied.
Using --client-side-copy always involves downloading all files, and
uploading them to the target.
The DESTPAC name is optional; the source packages' name will be used if
DESTPAC is omitted.
usage:
osc copypac SOURCEPRJ SOURCEPAC DESTPRJ [DESTPAC]
Options:
-h, --help show this help message and exit
-e, --expand if the source package is a link then copy the expanded
version of the link
-m TEXT, --message=TEXT
specify message TEXT
-t URL, --to-apiurl=URL
URL of destination api server. Default is the source
api server.
-r rev, --revision=rev
link the specified revision.
-d, --keep-develproject
keep develproject tag in the package metadata
-k, --keep-maintainers
keep original maintainers. Default is remove all and
replace with the one calling the script.
-c, --client-side-copy
do a (slower) client-side copy
Initialize Project Conf for OBS Package Copy
First you need to import the project configuration.
see #How_to_initialize_a_project_config
Copy Source
Then import the project.
Note : You must have an access to $OBSHOST_SOURCE. If not and your source $OBSHOST_SOURCE supports a public anonymous API, you can use obstag/obs2obscopy or consider the following method #OBS_Remote_Link which is reported to be quicker.
PRJ=ProjectToCopy
for i in `osc -A $OBSHOST_SOURCE ls $PROJECT_SOURCE`
do
osc -A $OBSHOST_SOURCE copypac -t $OBSHOST $PROJECT_SOURCE $i $PROJECT
done
Note: As you might have some Links in the project that you import, it is a good idea to keep the source and target project names identical.
Note: If you want to copy a full project with a lot of packages, obstag/obs2obscopy is a good idea.
This project is now ready to build against a repository.
Use OBS Remote Link
Some $OBSHOST_SOURCE ('e.g. build.meego.com) will let you browse anomalously the project but will not easily give you a login. To copy from those $OBSHOST_SOURCE you can create a link project in your local $OBSHOST and then do a copy like if it was a local copy.
You need to be 'Admin' to create a Remote Link.
You have 2 ways to create the remote linked OBS.
Create by web UI
At the 'Welcome' page of the Build Service, you use the the Setup OBS button in the web UI
Local Project Name: meego.com Remote OBS api url: https://api.meego.com/public Title: build.MeeGo.com Description: This project allows access to all resources on build.meego.com instance.
Create by command line
Create a configuration by hand in the project meta data. The example bellow represents a local project called "meego.com" which is a link to the public API of the official MeeGo OBS.
osc -A $OBSHOST meta prj -F - meego.com << OTHEREOF
<project name="meego.com">
<title>build.MeeGo.com</title>
<description>This project allows access to all resources on build.meego.com instance.</description>
<remoteurl>https://api.meego.com/public</remoteurl>
<person role="maintainer" userid="Admin"/>
<person role="bugowner" userid="Admin"/>
</project>
OTHEREOF
The side effects
- You can not use the command 'osc ls' to discover what is available on the remote OBS. The project will be reported empty and you need to use the Web UI to browse the remote OBS directly.
- If the server of $OBSHOST_SOURCE is down or not reachable ('e.g. build.meego.com), your project will not be usable.
How to use a remote link
To access to a remote project (to either copy or build against) you will just have to add the name of your local link before the name of the remote project.
For example if your link is named meego.com to access MeeGo:1.2:oss on the official MeeGo OBS you will use meego.com:MeeGo:1.2:oss
by 'osc':
osc -A $OBSHOST ls meego.com:MeeGo:1.2:oss
by 'curl':
curl $OBSHOST/source/meego.com:MeeGo:1.2:oss
A project can build against a remote repository.
For example :
<repository name="MeeGo_1.2"> <path repository="standard" project="meego.com:MeeGo:1.2:oss"/> <arch>armv8el</arch> <arch>i586</arch> </repository>
Note: When the project is ready, You can "officially" add it to the OBS server. openSUSE:Build_Service_private_installation#Add_Repositories_targets
Use Repository Source Import
How to use "osc importsrcpkg"
You need to download all your source RPMs on a local machine and import them into your project with the command
#osc importsrcpkg --help
importsrcpkg: Import a new package from a src.rpm
A new package dir will be created inside the project dir
(if no project is specified and the current working dir is a
project dir the package will be created in this project). If
the package does not exist on the server it will be created
too otherwise the meta data of the existing package will be
updated (<title /> and <description />).
The src.rpm will be extracted into the package dir. The files
won't be committed unless you explicitly pass the --commit switch.
SRPM is the path of the src.rpm in the local filesystem,or an URL.
Usage:
osc importsrcpkg SRPM
Options:
-h, --help show this help message and exit
-c, --commit commit the new files
--delete-old-files delete existing files from the server
-d description, --description=description
set the description of the package
-t title, --title=title
set a title
-n name, --name=name
specify a package name
-p project, --project=project
specify the path to a project
Initialize Project Conf for Repository Source Import
Initialize Project Conf for OBS Package Copy
see #How_to_initialize_a_project_config
Upload the Packages
On your local machine:
osc -A $OBSHOST checkout $PROJECT cd $PROJECT
Commit to the OBS server:
osc ci -m "Commit the src"
Now Import your source RPM:
osc importsrcpkg ~/tzdata-2010b-2.8.src.rpm osc add tzdata/*
This project is now ready to build against a repository.
Troubleshooting
At that time you should see your project restarting to build. If that would not be the case:
- check that your files in your target :full directory are all owned by the obsrun user. The following command should not return any file name:
find /obs/build ! -user obsrun
- check that tour boot strap project has meta data defined. You must import the meta from the project that you have ysed to create your :full. The osc meta prj command will also work on a linked project.
osc meta prj link:REMOTE_REFERENCE_PROJECT > prj-meta-to-be-copied.meta osc meta prj -F prj-meta-to-be-copied.meta YOUR_LOCAL_BOOTSTRAP_PROJECT
If you find any file or directory use:
chown -R obsrun:obsrun /obs/build (will correct any ownership issue)
- Force the OBS to reindex your new :full directory. It will create a file named :full.solv
obs_admin --rescan-repository $PROJECT $REPO $ARCH
- Check that your RPMs are valid (i.e. not damaged during transfer)
cd /obs/build/$PROJECT/$REPO/$ARCH/:full for I in `ls *.rpm` ; do rpm -qlp $I >/dev/null; if [ $? -ne 0 ] ;then echo $I >>../error.lst ; fi ; done cat ../error.lst (must be empty, all rpm in error needs re-installation)
- Still not working, get a look in the log files in the directory /obs/log.
You can start by having a look at /obs/log/scheduler_$ARCH.log and search from the end for the string "expanding dependencies". You will find from there why the scheduler fails.
tail -f /obs/log/scheduler_$ARCH.log
Creating a first project to test the Bootstrapping
- After creating a dedicated user (testUser) via the web API, relog onto the web UI with your new login.
- Open your home project and create a sub project called "MyTest", "home:testUser>Actions>Create subproject".
- Add a package in your new Home project.
- Simply create a branch "Packages>Create new package based on existing package" with one of the packages recently copied in your new OBS instance. Pick up a small one to speed compilation time (e.g. tzdata).
- Create a package from scratch "Packages>Add" (you need to add source manually).
- Add a repository.
- By a OBS repository UI:Click on the "home:obsuser:testProject>Repositories>Add".
- By a local project on OBS repository UI:"home:obsuser:testProject>Repositories>Add>pick one via advanced interface".
- Project:pick from the list the project/repo that you have just imported and rebuilt.
- Repository:pick from the list repository.
- New name:Give a name to your repo (e.g. my-repository-test,standard,...).
This will request the OBS to build your new project "MyTest" against that repository.
You can now :
- Check out your project "MyTest" with the osc command.
- Modify a file or two.
- Check-in the package/project.
The OBS will rebuild your project "MyTest".
If your reference project changes, the OBS will also rebuild your project "MyTest".