Creating Jigsaw Download Images for DVD and CD Sets

From openSUSE

Revision as of 13:36, 1 October 2005

Introduction

This page should describe how Jigsaw Download images can be created by someone that has access to a set of DVD and/or CD images (most likely someone at SUSE and how a user can use them to download the images. The advantage of Jigsaw Download images is that an arbitrary number of DVD/CD images can be distributed without the need of much space.

Savings

How much does this save?

I did this for SUSE Linux 10.1 alpha1 with the provided CD sets.

The Original CD Sets

601M    SUSE-10.1-CD-OSS-i386-Alpha1-CD1.iso
649M    SUSE-10.1-CD-OSS-i386-Alpha1-CD2.iso
685M    SUSE-10.1-CD-OSS-i386-Alpha1-CD3.iso
665M    SUSE-10.1-CD-OSS-i386-Alpha1-CD4.iso
525M    SUSE-10.1-CD-OSS-i386-Alpha1-CD5.iso
3.1G    total

662M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD1.iso
688M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD2.iso
694M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD3.iso
679M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD4.iso
536M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD5.iso
3.2G    total

643M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD1.iso
686M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD2.iso
687M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD3.iso
662M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD4.iso
536M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD5.iso
3.2G    total

The Jigsaw Download Images

i386

188K    SL-10.1-OSS-alpha1.jigdo
8.6M    SUSE-10.1-CD-OSS-i386-Alpha1-CD1.template
132K    SUSE-10.1-CD-OSS-i386-Alpha1-CD2.template
14M     SUSE-10.1-CD-OSS-i386-Alpha1-CD3.template
156K    SUSE-10.1-CD-OSS-i386-Alpha1-CD4.template
116K    SUSE-10.1-CD-OSS-i386-Alpha1-CD5.template
23M     total

This is obviously enormous. Note that this is only 0.72% of the original size!

ppc

188K    SL-10.1-OSS-alpha1.jigdo
124M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD1.template
200M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD2.template
134M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD3.template
76M     SUSE-10.1-CD-OSS-ppc-Alpha1-CD4.template
195M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD5.template
728M    total

These are 22% of the original size. This is not as good as for i386. Why is that the case? For some reason, SUSE builds the noarch packages on all platforms separately. Because of that the ppc version has noarch packages that are different to those on the FTP server and thus the savings for ppc are lower than for the i386 platform. If they used the same noarch packages for all platforms you could expect a total size of about 23M instead of 728M.

x86_64

188K    SL-10.1-OSS-alpha1.jigdo
118M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD1.template
200M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD2.template
134M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD3.template
77M     SUSE-10.1-CD-OSS-x86_64-Alpha1-CD4.template
177M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD5.template
704M    total

These are 22% of the original size for the same reasons as for ppc.

Creating the Images

After installing the Jigdo package you can use the following script.

#!/bin/bash
#
# mkjigdo - make jigdo files for all ISO images in one directory
# Copyright (C) 2005  Robert Schiele <rschiele@uni-mannheim.de>
#
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation; either version 2 of the License, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along with
# this program; if not, write to the Free Software Foundation, Inc., 51
# Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
#

set -eu

usage()
{
    cat << EOT

Usage: $0 [OPTIONS]
Options:
  -b WORD  Basename of the jigdo file (=BASENAME)
  -c FILE  Configuration file to load
  -i PATH  Path for image files (=ISODIR)
  -j PATH  Path for jigdo files (=JIGDODIR)
  -m WORD  List of mirror servers separated by \`,' (=MIRRORS)
  -r PATH  Relative path on all mirrors (=RELDIR)
  -s PATH  Path for file storage (=SRCDIR)
  -t PATH  Path for temporary files and cache (=TMPDIR)
  -h       Output this help
  -v       Output version info
EOT
}

# set some plain stupid defaults
BASENAME=a
RELDIR=.
ISODIR=.
JIGDODIR=.
TMPDIR=.
SRCDIR=.
MIRRORS="ftp://localhost/pub"

# read standard config file
test -f ~/.$(basename "$0")rc && . ~/.$(basename "$0")rc

# parse command line
while [ "$*" ]; do
    case "$1" in
        -b) BASENAME="$2"; shift;;
        -c) . "$2"; shift;;
        -h) usage; exit;;
        -i) ISODIR="$2"; shift;;
        -j) JIGDODIR="$2"; shift;;
        -m) MIRRORS="${2//,/ }"; shift;;
        -r) RELDIR="$2"; shift;;
        -s) SRCDIR="$2"; shift;;
        -t) TMPDIR="$2"; shift;;
        -v) echo mkj version 0.0; exit 0;;
        *)  usage >&2; exit 1;;
    esac
    shift
done

# run jigdo on all ISO files
JIGDOLIST=
for ISO in "$ISODIR"/*.iso; do
    BASEFILE="${ISO/*\/}"
    BASEFILE="${BASEFILE/.iso}"
    case "$BASEFILE" in
        *.delta) ;;
        *) jigdo-file mt --bzip2 -i "$ISO" -j "$TMPDIR/$BASEFILE.jigdo.tmp" \
            -t "$JIGDODIR/$BASEFILE.template" -c "$TMPDIR/$BASENAME.cache" \
            "$SRCDIR"
            JIGDOLIST="$JIGDOLIST $TMPDIR/$BASEFILE.jigdo.tmp";;
    esac
done

# merge all temporary jigdo files into one file
{
    awk '/^[[]/{if(a==1)a=2}/^\[Image\]/{a=1}{if(a<2)print}' $JIGDOLIST
    echo '[Servers]'
    for i in $MIRRORS; do
        echo A="$i/$RELDIR/"
    done
    echo
    echo '[Parts]'
    awk '/^[[#]/{a=0}/^\[Parts\]/{a=1}/^[^[]/{if(a==1)print}' $JIGDOLIST | \
        sort -t= -k2 -u
} | gzip -c9 > "$JIGDODIR/$BASENAME.jigdo"

You can either specify all options on the command line or put them into the configuration file ~/.mkjigdorc. This is an example I used to make jigdo files for beta4:

BASENAME=SL-10.0-OSS-beta4
RELDIR=distribution/$BASENAME
BASEDIR=/pub/opensuse/$RELDIR/
ISODIR=$BASEDIR/iso
JIGDODIR=result
TMPDIR=tmp
SRCDIR=$BASEDIR/inst-source
MIRRORS="ftp://ftp.opensuse.org/pub/opensuse
         ftp://ftp.gwdg.de/pub/opensuse"

Now just put the .jigdo and all the .template files to the FTP server and you are done.

Downloading the Images

If you want to download some image you first need Jigsaw Download from http://atterer.net/jigdo/ (Linux and Windows version availlable) or special packages for SUSE LINUX 9.2, 9.3, or 10.0 from http://pi3.informatik.uni-mannheim.de/~schiele/suse/ and the .jigdo and .template for the image you want.

Then you start the application as described here. (Maybe someone wants to add some more specific description here ;-)

Here you can find another nice description of the tool in German.