SDB:KIWI-LTSP transcoding

Jump to: navigation, search

Introduction

dvd::rip and transcode are tools for converting between different video and audio formats. dvd::rip is used for ripping the contents of a video DVD (MPEG2 data streams) to disk, eventually cut scenes and start transcode, which then converts the MPEG2 data streams to a desired format (for example xvid and mp3).

Since the transcoding process can take a long time, the work can be split into several fragments which then can be processed by other machines on a network. The best way to accomplish this is to set up a server which controls the transcoding process, from which the (diskless) clients can boot. This was tested on openSUSE 11.1 (as of 10-Jun-2009).

In the following examples the host has the IP address 192.168.0.50 and is named ws50. The clients are named ws51, ws52, ws53, etc. Adjust the addresses to your own setup, if required. "user" specifies a normal user running dvd::rip on the host.

It is important to know that the clients must be assigned a fixed hostname/IP address because you have to specify a list of clients to the dvdrip-master (the controller for the whole cluster). The default behaviour of LTSP is, if you reboot a client inside the default-lease-time (dhcpd.conf), the client is assigned a new hostname and IP address, which is then not seen by dvdrip-master (showing the previously used hostname as "offline").

Requirements

Repositories

The openSUSE-11.1-i586 installation DVD content must be available at /mnt/11.1 (either by copying the contents in there, mounting the DVD or using 'mount -o loop' on an image of the DVD). The following repositories need to be present:

  • Packman
  • openSUSE OSS

Packages

The following packages need to be installed on the host:

  • dvd::rip (>= 0.98.10)
  • transcode (>= 1.1.0)

KIWI-LTSP

Follow the instructions on KIWI-LTSP quick start to install LTSP server.

Setting up LTSP on the host

/etc/sysconfig

The following values should be set in System/Kiwi-ltsp:

  • DHCP_RANGE = 192.168.0.51 192.168.0.100
  • DHCP_SUBNET = 192.168.0.0
  • IMAGETYPE = NFS
  • LTSP_LOCALAPP = transcode ogmtools nfs-client

This will instruct KIWI-LTSP to provide an NFS root-filesystem and transcode, ogmtools and nfs-client as local apps for the clients. The network subnet 192.168.0.0 is used. The DHCP daemon will provide an IP range from .51 to .100 to assign addresses to the network-booted clients.

Setting up repositories required by KIWI

The following repositories are required by kiwi-ltsp-setup (this script is run later):

  • Packman
  • openSUSE OSS

This is an example on what to add to /usr/share/kiwi/image/ltsp/suse-11.1/config.xml. Locate the last repository entry and add: 

<repository type="rpm-md">
    <source path="http://ftp.skynet.be/pub/packman/suse/11.1/"/>
</repository>
<repository type="rpm-md">
    <source path="http://download.opensuse.org/distribution/11.1/repo/oss/suse/"/>
</repository>

KIWI setup

The KIWI setup script can be run now. Do so by invoking as root: 

kiwi-ltsp-setup -s

This might take some time while it downloads required packages, installs them in /srv/kiwi-ltsp-nfs and prepares the host to provide a terminal-service for network-booting openSUSE on a client.

/srv/tftpboot/KIWI/lts.conf (optional)

Modify the lts.conf so that not required features are excluded at boot time (GUI, sound, etc). This can save some RAM on the client. It might then look like this: 

[default]
    SOUND=false
    LOCALDEV=false
    CONFIGURE_X=false
    SERVER=192.168.0.50
    SCREEN_07=shell
    LOCAL_APPS=true
    LDM_NOMAD=false

The variable SERVER is set to the IP address of the host.

/etc/dhcpd.conf

Adjust the "subnet 192.168.0.0" section in dhcpd.conf so that every client gets assigned a unique IP address. It might then look like this: 

subnet 192.168.0.0 netmask 255.255.255.0 {
#  range 192.168.0.51 192.168.0.100;
  default-lease-time 14400;
  max-lease-time 172800;
  filename "pxelinux.0";
  host ws51 {
    hardware ethernet 00:01:23:45:67:89;
    fixed-address 192.168.0.51;
  }
  host ws52 {
    hardware ethernet 00:12:34:56:78:9A;
    fixed-address 192.168.0.52;
  }
  host ws53 {
    hardware ethernet 00:23:45:67:89:AB;
    fixed-address 192.168.0.53;
  }
  host ws54 {
    hardware ethernet 00:34:56:78:9A:BC;
    fixed-address 192.168.0.54;
  }
  host ws55 {
    hardware ethernet 00:45:67:89:AB:CD;
    fixed-address 192.168.0.55;
  }
}

The MAC address ("hardware ethernet" value) is shown as soon as the client tries to PXE-boot.

To apply the changes, run as root: 

/etc/init.d/dhcpd restart

If some error is shown here, there might be a typo (or missing semicolon) in dhcpd.conf. Fix it and re-run the above command.

/etc/hosts

The hosts /etc/hosts must be updated so dvd::rip can use the hostnames (IP addresses are not allowed when doing clustered transcoding). An example would be: 

192.168.0.50    ws50 ws50
192.168.0.51    ws51 
192.168.0.52    ws52 
192.168.0.53    ws53 
192.168.0.54    ws54 
192.168.0.55    ws55

Setting up transcode clustering

Adjusting fping file attribute

Run following as root: 

chmod u+s /usr/sbin/fping

This will set the SUID attribute on fping, which is required by dvdrip-master.

Preparing password-less login to the clients using ssh

Since dvdrip-master will log into the clients using ssh, it will not send a password for logging in. Because of this, every client must download a prepared /root/.ssh/authorized_keys file while booting.

If ~/.ssh/id_rsa.pub does not exist yet, this will create one (run as normal user, leave passphrase empty): 

ssh-keygen -t rsa

As root do: 

cat /home/user/.ssh/id_rsa.pub > /srv/tftpboot/KIWI/authorized_keys

This will add the created public key to the authorized_keys file which will be downloaded by every client at boot time.

Now in /srv/tftpboot/KIWI/config.default locate the line beginning with "CONF=" and add (no spaces or newlines): 

,/KIWI/authorized_keys;/root/.ssh/authorized_keys;192.168.0.50

This tells the client to download /srv/tftpboot/KIWI/authorized_keys from the host (192.168.0.50) and copy it to /root/.ssh/authorized_keys.

Preparing dvdrip-data directory

Since all clients must work with the same data files, they all must have access to one folder on the host. This is accomplished by setting up an NFS exported folder in "user"'s home directory.

As normal user do: 

mkdir ~/dvdrip-data
cat /etc/passwd | grep "user"

This will create the dvdrip-data folder in the home directory of "user". As second it shows some informations about "user" required for the exported folder. It should look like this: 

user:x:1000:100:User:/home/user:/bin/bash

The number 1000 is the numerical user ID (UID) and 100 is the numerical group ID (GID).

Add the following line to /etc/exports 

/home/user/dvdrip-data 192.168.0.0/255.255.255.0(rw,all_squash,async,no_subtree_check,anonuid=1000,anongid=100)

and to apply the changes run as root: 

/etc/init.d/nfsserver restart

This will make the dvdrip-data folder in "user"'s home directory being shared using NFS on the subnet 192.168.0.0. The value "anonuid" and "anongid" must eventually be adjusted to the UID/GID of "user". Both options are required to let the host NFS server assign all files and folders (being created by the clients "root"-user) in there to "user".

Auto-mounting dvdrip-data on the clients

As root create a file /srv/kiwi-ltsp-nfs/etc/init.d/after.local: 

#! /bin/sh
mkdir /root/dvdrip-data
mount.nfs 192.168.0.50:/home/user/dvdrip-data /root/dvdrip-data -o rsize=8192,wsize=8192,timeo=14,intr -w

and (as root) make it executeable: 

chmod a+x /srv/kiwi-ltsp-nfs/etc/init.d/after.local

The script after.local is being run on the client after booting has finished. It creates the dvdrip-data folder in root's home directory. Afterwards the shared NFS folder "dvdrip-data" on the host (192.168.0.50) is being mounted to it.

Prepare dvd::rip to transcode in a cluster

dvdrip-master must be told which clients in the network will do clustered transcoding. This is done using the dvd::rip GUI. Every client can run multiple nodes, which is useful when having more than one CPU on the client. For example a client having a dual-core CPU can run two nodes. The Intel Core i7 processor has 8 virtual CPUs (4 physical cores, with Hyperthreading enabled, each core shows two CPUs), so it can run up to 8 nodes which could then be named wsXXa, wsXXb, wsXXc and so on. The naming scheme is up to the user. The host can be also involved in clustered transcoding, just add it like every other client too. Beside this, it is suggested to leave "free" one CPU on the host, which is then used for doing the network- and disk-traffic generated by the nodes.

Boot up every client to be involved in clustered transcoding. Make sure they are all up and running. As normal user on the host, log into every client: 

ssh root@wsXX

Replace wsXX with the name of the client (ws51, ws52, etc). SSH will answer that the fingerprint is unknown and will be added to the list of known_hosts, write "yes" here. Repeat this until every clients fingerprint has been added. This needs to be done again everytime the user has run "ssh-keygen".

Then run dvd::rip as normal user. Clicking on menu "Cluster" -> "Control Window", the window titled "dvd::rip - Cluster Control" is shown. Press "Add node" to add a each clients nodes to the list of nodes:

  • Name: (name of the node, for example: ws51a, ws51b, ws52a, ws52b, etc).
  • Client name: (name of the client, the node runs on; ws51, ws52, ws53, etc).
  • Local data directory: /root/dvdrip-data
  • Username for connecting via SSH: root
  • Leave all other fields blank.

Press "Test settings" and check the "Results" list. Afterwards press "OK" to add the node. Press "Start node" if desired.

All nodes should be now shown as "online".

Using dvd::rip

Ripping DVDs

For informations about ripping DVDs and creating projects and such, see documentation of dvd::rip.

Transcoding

For transcoding video/audio data using dvd::rip, select "Transcode" tab, select the title, re-fine the settings if desired and press "to cluster". On the then shown "dvd::rip - edit cluster project" window simply press "OK". On the shown "dvd::rip - Cluster Control" window in the "Project queue" list, select the project and press "Start project". When it is finished, one can remove the project from the queue. When there are no more projects left, the clients can be simply powered off.

Retrieved from "https://en.opensuse.org/index.php?title=SDB:KIWI-LTSP_transcoding&oldid=13467"