Mirror Infrastructure

From openSUSE

This document describes the various ways to mirror the content of download.opensuse.org and how an official mirror can get established.

Public Servers

rsync.opensuse.org

This server offers public access via rsync protocol to the download.opensuse.org content. It is currently limited to 50 concurrent connections, so you might not get always access to it. Various mirror hosts do also offer rsync services.

rsync modules

general rsync.opensuse.org modules:

opensuse-hotstuff-160gb: The most requested files, which fit into 160 GB. This currently includes the install repo and CD/DVD media of the latest product, its updates, and the most popular other repositories. This is the most suitable module for mirrors with limited disk space. The majority of requests goes on exactly these files.

opensuse-hotstuff-80gb: An even more restricted selection of most popular files, restricted to 80 GB of space. Use this if your mirror has very limited disk space. Still, the majority of requests goes on the files included in this module, so it is highly useful to mirror "only" these files.

opensuse-updates: This rsync module provides the /update tree, with official updates for released openSUSE distributions, starting with openSUSE 10.3. (To mirror the updates for older releases, check rsync://ftp.suse.com/suse/update/.)

opensuse-full: This rsync module provides the complete content of download.opensuse.org, except the SL-OSS-factory directory. The reason to exclude this directory is the high frequency of updates inside. To mirror the SL-OSS-factory directory, we recommend using drpmsync to fetch this directory instead, it decreases the traffic to less than 10% compared to rsync.

opensuse-full-with-factory: The same as the previous one including the SL-OSS-factory directory containing the Factory Distribution. Again, we do not recommend using this module.

opensuse-stable: Only the last stable released version of SUSE Linux gets mirrored. An update happens once about every 8 months.

opensuse-current: Only the last development snapshot (Alpha, Beta or RC). An update happens every month in the Alpha Phase and every week in the Beta phase.

modules for mirroring the Build Service repositories:

buildservice-repos: The complete content

buildservice-repos-main: Everything, but not the home: projects of the users

Updates do happen all the time, whenever a repository from the Build Service got rebuilt and updated. It is also possible to get the updates pushed.

modules for mirroring the drpmsync tree:

opensuse-drpmsync: distribution/SL-OSS-factory/drpmsync tree only (OSS part)

opensuse-drpmsync-nonoss: distribution/SL-Factory-non-oss/drpmsync tree only (non-OSS part)

opensuse-full-with-factory-drpmsync: The same as opensuse-full-with-factory, including the drpmsync tree.

If you want to mirror the drpmsync trees, you need to be aware that the enormous number of files (~200000) will cause a very high load on rsync.opensuse.org. Please avoid it if possible.

drpmsync.opensuse.org

drpmsync is a sync services for the Factory Distribution. It does reduce the transfer heavily by transmitting only xdelta data, if the local rpm is not older than 1 month.

drpmsync provides access to the SL-OSS-factory directory only. This directory contains always the latest build of the distribution. This means it can also contain an inconsistent or very broken state. It is useful for developers to fetch the latest code or for testers to validate the latest version.

A drpmsync client can get found in the make deltarpm package from http://download.opensuse.org/tools/drpmsync/ and it can be called with this line for example:

  drpmsync drpmsync.opensuse.org:8888/Factory /my/directory

Please note that drpmsync.opensuse.org might be under heavy load, we do look for public mirrors who want also to run a drpmsync server. Please contact us, if you do want to run one as well.

Access for registered servers

stage.opensuse.org

Some dedicated mirrors get access to stage.opensuse.org. This server provides the updated content of download.opensuse.org before the official release and has a higher transfer rate than the public servers. You may want to register for access at mailto:ftpadmin@suse.de, if your mirror has at least a 100MBit connection. This server provides as well rsync and drpmsync services.

Pushing support for Build Service updates

stage.opensuse.org does also host all content from the Build Service. Since the updates do happen all the time, whenever a new package set got built it is also possible to get the content pushed, instead of polling for it. The obviously requires rsync write access for stage.opensuse.org on your server. The advantages of that method are that

• the mirror is almost always up to date,
• no need to run rsync calls via all repositories. The pushing will only update the repositories which have changed. This does reduce the IO load of the mirror a lot.
• the redirector running at http://download.opensuse.org/ is aware that the packages got updated and can immediately redirect to the mirror.

How to become a pushed mirror?

The usual way (but we can also support a different way) is to open a rsync module on your server, where stage.opensuse.org gets write access. A login and password is optionally possible, but not really needed. Please write a mail to mailto:ftpadmin@suse.de afterwards where you tell us:

• the server name where to push
• the rsync module to be used
• an eventually needed account and password
• What you want to mirror (everything or only some certain projects)
• What the public download url will be.
• Any kind of a special wish :)

Then, we need rsync read access to scan your mirror for our download redirector. The download redirector database needs to be updated periodically so it reflects the actual files on your mirror. The scanning happens from 195.135.220.0/22.

The push pauses between 1:00 and 2:00 CEST, and between 13:00 and 14:00 CEST. If you want to run a pulling rsync from your end, this would be the best time to do that.

Register Your Mirror

In order to redirect clients to your mirror, we need the following:

• email address for contact
• HTTP URL on your mirror (e.g. http://yourmirror.example.com/pub/opensuse/)
• is your web server large-file capable? (to handle images larger than 2 GB in size)
• read-only rsync access for our scanner -- for scanning which we perform to keep our download redirector database up-to-date. It is done from 195.135.220.0/22.
• FTP URL, if you run an FTP server. Can serve as fallback protocol for scanning, if rsync is not available.

If you provide this data in writing to mailto:ftpadmin@suse.de, we will add your mirror to our mirror database. The mirror database is used by our download server to actively redirect clients to your server. We attempt to distribute requests on a geographical basis per client IP address. The amount of redirects issued also depends on a score which we will determine together with you, in order to match your capacities.

Furthermore, we actively monitor content on mirrors, so that we redirect only to files which actually exists on them. rsync is the most efficient way to do this; scanning through 300.000 files might take only a few minutes with it. The second best method, if rsync is not available, is via FTP, but it is much less efficient (takes considerably longer and places more load on your server). As last resort, we can fall back to HTTP, if neither rsync nor FTP is available. But it crawls. Thus, please do consider adding an rsync module for opensuse content, which allows for much faster scanning of your server.

You may also want to add your mirror to our official mirror list for the released versions or for the development builds, but those lists are not used for the download redirector and might be phased out later. These are wiki pages, simply hit the "Edit" button at the top ;)

How to set up a mirror

See here for a howto: Mirror_Setup_Howto

Planning: Sizes & Update Frequency

(as of May 1st, 2008)

Staying informed

The mirror@opensuse.org mailing list (previously called suse-mirror@suse.de) is low-traffic and used mainly for announcements. It is also a suitable place for discussions around mirroring openSUSE content, should the need arise. To subscribe, please write to mailto:ftpadmin@suse.de and ask to be added (since it is a closed list).

The general contact address is: mailto:ftpadmin@suse.de