Libzypp/Failover

From openSUSE

Concept for libzypp doing failover when downloading packages from download.opensuse.org

Mirrors die all the time, without warning. That's the thing with mirrors. And download.opensuse.org is not Google, or Akamai - it can die, too.

But with the download redirector that we have on download.opensuse.org, there are some unique opportunities which could be implemented in libzypp. One of the features of the redirector is to give a client a dynamic mirror list, instead of redirecting it to a mirror. With that list, the client could do powerful things like failover (when a mirror has died, is not reachable, times out, ...), select the fastest mirror, or other things.

Here is a concept how this could be implemented.

Dynamic mirror lists

There are the following three types of mirror lists:

http://download.opensuse.org/distribution/10.3/repo/oss/suse/repodata/primary.xml.gz?mirrorlist

This is a proof of concept which is mainly useful for humans for diagnostic purposes.

http://download.opensuse.org/distribution/10.3/repo/oss/suse/repodata/primary.xml.gz.metalink

This is a format for metalink clients (see [1]). Please look at it, because it illustrates well the point, which we want to achieve below.

The third type is a terse text-only format. (It could also be sent in XML, machine-parsable text file, JSON, or whatever. But a text file is likely the fastest and easiest to parse. It allows the client to simply ignore everything than the first line, if nothing goes wrong.)

It like this:

# mirrorlist-txt version=1.0
# url baseurl_len mirrorid region:country power
http://ftp.hosteurope.de/mirror/ftp.opensuse.org/distribution/10.3/repo/oss/GPLv3.txt 49 22 EU:de 100
http://ftp5.gwdg.de/pub/opensuse/distribution/10.3/repo/oss/GPLv3.txt 33 44 EU:de 100
http://ftp.uni-ulm.de/mirrors/opensuse/distribution/10.3/repo/oss/GPLv3.txt 39 42 EU:de 100
...

The mirror list is sorted by suitability for that particular client country. The most appropriate mirror (exactly the one that the redirector would otherwise choose to redirect the client to) is right at the top.

To try this out and get a feel for it, try a request like this:

curl -H "Accept: application/mirrorlist-txt" http://download.opensuse.org/distribution/10.3/repo/oss/suse/repodata/primary.xml.gz

How it works (Step 1)

The client will get this list by requesting the file like it normally would (http://host/path/to/file), but with the following request header added:

Accept: application/mirrorlist-txt

Thereby the client indicates its wish (and ability) to get a mirror list for the requested file.

If it makes sense for that file, the server will return the zypp mirrorlist. If so, it'll come with MIME type application/mirrorlist-txt so that it is distinguishable from other responses. If not, the server returns the requested file.

The client indicates its ability to all requests, and it will get a list each time, if the redirector decides so. This is important because

• the redirector offers file-level granularity
• we need that fine granularity
• all requests can be the same, without "special" requests (like, say, append query string to URL -> get list -> pick mirror -> fetch file). Thus the client stays 100% compatible and HTTP compliant and works with different servers.

The client needs to be able to handle three possible cases:

• 200 OK, Content-Type != application/mirrorlist-txt: receive the file. (Like today.)
• 200 OK, Content-Type == application/mirrorlist-txt: follow first URL.
  • in case of failure, follow the second URL from the mirror list.
  • in case of failure, try next. And so on.
• 302 Found: follow the Location header (standard redirect -- like happening today.)

At present, when a mirror doesn't work (times out, unexpectedly doesn't have a file, or returns something broken), the client will simply try again or give up. With this proposal, it'll transparently try other mirrors and will likely succeed.

How it works (Step 2)

So far, we assumed a healthy and reachable download.opensuse.org. But dealing with a non-reachable download.opensuse.org (or one returning garbage) is also possible. For that, the client should:

• do all the above, and in addition save every base URL which it used into a little cache file. For "hard times" :-) That's why the mirror list doesn't only contain URLs, but also a number which indicates the length of the URL pointing to the "repository base".
• in case of failing to get a reply from the redirector (timeout, garbage), the client uses one of the cached baseurls and try to get the file from there.

Thus, this enables the client to fall back to autonomously trying mirrors, and finding what it is looking for. This would be great, wouldn't it? Right now, the client gives up when download.opensuse.org is not available -- even though there are mirrors in abundance that it could fall back to.

How it works (Step 3 - nice to have)

In addition, the mirror lists (and the client-side little cache of base URLs and mirror ids) makes way for some other interesting features. Like:

• optionally run a speed test and choose the fastest mirror
• the client could download from multiple mirrors simultaneously

Moreover, the client could implement its own scheme of mirror selection. This could be "always use mirror XY if possible", or for instance. This is important to be configurable on the client side, because there is no reliable way for the server to find out -- not even by network prefix. Of course, the client could send its preferences along the request to the server -- but it is much easier to keep it on the client side, as there is no need to burden the server with this information.

I would not underestimate the desire of users to use a fixed, certain, preferred mirror, especially a local one!

Thus, the client could

• prefer a certain mirror, every time it is among the listed mirrors
• use the geographical coordinates of the mirrors (which could be included in the list) to implement a more sophisticated scheme which we can't do server wise. (There is no scalable and reliable way to know about the clients location, from the server side.) The client can know about its location, after having been configured with that data.

An additional idea for remote users with Internet connectivity (think Indonesia, New Zealand, ...): avoid the additional round-trip which occurs for each request, by optionally checking download.opensuse.org only once per refresh, and use mirror directly after that. This would use the "mirrorlist request" only once, on a crucial file in the repository, in order to locate mirrors which host this repository.

Performance considerations

For the server, serving mirror lists doesn't make a difference; generating the list is virtually as cheap as returning a redirect. The database query that we use is exactly the same.

On the client-side (which is waiting for network i/o) it won't matter if it needs to parse the list in addition. After all, it is very simple to just pick a mirror out of the list, which is all what the client needs to do, at a minimum.

In fact, it requires nothing else than reading the first line from the mirror list, and extracting the URL from it -- as long as nothing goes wrong. The mirror list is already sorted adequately. And the URL is the ready-made URL to be used.

Just like the client reads the server reply headers now, sees the Location: header, grabs the URL from that and follows it, it would read the mirror list, grab the URL from the first line, and follow it.

That's really all there is to it.

In case of an error then, it can simply try the next one.

Thus, performance won't be an issue.

Example scenarios

Not exhaustive.

Mirror breakage:

• ftp5.gwdg.de is not available for 5 minutes, because a router is rebooted. (Happened this afternoon). After 1-3 minutes the redirection to that mirror is disabled, since our probing notices it. But in the time window before disabling, clients are sent there, and get timeouts. No amount of hardware redundancy on the server will solve this "three minute window". Nor will retrying. The above proposal will make the client seamlessly try the next mirror, and succeed.

• Repository inconsistency introduced by various means, best described as "something messed up and now the md5sum of a patch file doesn't match". Some mirrors might have the old, broken file. Some mirrors might already have the new, repaired file. (This scenario can happen only if a kind of file is modified which is normally not supposed to change without a version number increase, and thus is normally safe to redirect to mirrors for.) An example is https://bugzilla.novell.com/show_bug.cgi?id=352719. The above proposal can make the client try another mirror and (if possible) retrieve the good file from elsewhere.

• A German university hosting a mirror installs a new firewall. Its connection tracking is broken, which causes random connection resets for about 5% of downloaded files. The client does see HTTP 200 OK replies, but ends up with partial files that don't match the repository metadata. In this particular case, retrying to download from the same mirror would eventually succeed (due to the randomness of the problem), but retrying the download from another mirror would have succeeded immediately.
  
  (I'm not making this up. This scenario happened twice. The first time, it took me 3 weeks to debug it because there were only very vague reports. The second time it took only 3 hours. This bug must have caused lots of grief in 3 weeks.)

There will always be breakage which can't be anticipated. Trying another mirror will help in many cases.

Server breakage:

• When a download.o.o outage happens, clients can't check for security and other updates. With a cached list of previously used mirror base URLs, the client can use mirrors as fallback.

• New installations are a different issue, because they don't have "previously used mirrors" yet. Either they retry later (to set up the security update repository, or other repositories), or we may be able to find another solution. Well, a hardcoded second redirector URL would obviously help, if we take it seriously. But it would obviously need a second redirector.

It is important to understand that the fallback is really only a fallback, because mirrors are never 100% exact replicas. For some repositories this matters more (buildservice), for some less (10.3 repo). Security updates are somewhere in between. In the end, only download.o.o delivers files with exact caching control headers, making sure that a client doesn't see stale content. Content from any mirror could have been cached for arbitrary time. And stale content doesn't only mean "client doesn't see the update yet". It can also mean "those files don't fit together" and the client will complain about a (seemingly) broken repository.

A way to test an implementation

If the client sends a User-Agent of "getPrimaryFailover-agent/0.1", the server will insert broken mirror URLs at the top of the list. Thus, the client will need to cope with those. Thus, the implementation can be tested in the live environment:

 % curl -s -A "getPrimaryFailover-agent/0.1" \
   -H "Accept: foobar, application/mirrorlist-txt, */*" \
   http://localhost:80/zrkadlo/distribution/10.3/repo/oss/GPLv3.txt
# mirrorlist-txt version=1.0
# url baseurl_len mirrorid region:country power
http://doesnotexist/distribution/10.3/repo/oss/GPLv3.txt 20 10001 EU:DE 100
http://www.poeml.de/nonexisting_file_for_libzypp 20 10002 EU:DE 100  
http://www.poeml.de:83/nonexisting_file_for_libzypp 23 10003 EU:DE 100
http://ftp.opensuse.org:21/foobar 27 10004 EU:DE 100
http://download.opensuse.org/server-status 42 10005 EU:DE 100
http://widehat.opensuse.org:22/foobar 37 10006 EU:DE 100
[...]