SUPER preloading

From openSUSE

This article will not be transferred to the new wiki instance! This article does not fit the openSUSE:Wiki Guidelines requirement and cannot be updated. It will not be transferred to the new Wiki instance. Reason Obsolete.

Diese Seite auf deutsch.

Preloading, a sub-project of SUPER

What does preloading do? The basic idea is to load files into the cache that are used by certain applications, e.g. OpenOffice. This makes OpenOffice start up much faster because it reads most of its data from cache, not from the slow harddisk. The preloading is done completely in the background, so it will not slow down anything else. Currently preloading is by default done for OpenOffice, Firefox, Mozilla, Gimp and Khelpcenter. With PePr, preloading is easily done for almost any application.

There are currently 2 ways to include preloading into your system:

• Static preloading: Preloads files according to a fixed list. Since Suse Linux 10.0beta4, this is installed and active by default in the official distribution!
• Personal preloading with PePr: Preloads only the files that you use. Still needs testing, but should add an additional perfomance boost to the static version. In contrast to the static version, PePr is not limited to certain applications but can easily preload anything that you want.

There is also a more technical description of preloading.

Introduction to PePr, your personal preloader

PePr is an extension to the original preloading idea that Andreas and I (Stefan) came up with. It will only preload the files that you actually use. To do this, you have to "teach" PePr which files you frequently use. More details on this below.

PePr is an addition to (and not a replacement for) static preloading. While static preloading can only preload the program itself, PePr will also preload files that are specific for you. This includes files in your home-directory and the translation files for your specific language, making your programs start up even faster. But PePr can also easily preload new applications that are not included in static preloading, like koffice and eclipse.

Current Version of PePr is Release 3 (September 7th). Changes are

• Use "ionice" to also renice disk IO.
• Fix desktop file to use KDE-autostart-phase=2.
• Set umask to 077 for pepr-updates (handling private data).
• Added some "" to allow spaces in filenames.
• Static preload is now installed by default, so don't dynamically preload stuff that is already preloaded by default. Reduced my personal preload file from 81kB to 45kB

Installation of PePr (10.0beta4 or later)

Installation is pretty easy, but currently only works for KDE (help wanted!):

Save the shell scripts to /usr/local/bin and chmod them to 755:
pepr.sh
pepr_parsepriv.sh
pepr_update.sh

Save the auotstart entry for KDE to /opt/kde3/share/autostart/
pepr.desktop

The preload package is already installed by default, no need to do this by hand!
Note: All these programs are licensed under the GPL.

Configuration of PePr

Once all the files are in place, log into KDE with the user for which you want to activate preloading. On a terminal, start "pepr_update.sh". Now start all the applications that you would usually start: Open Office, your E-Mail program, your favorite browser. Also make sure you open files that you usually open. One example is your mailbox: my mailboxes have lots of MB, so preloading these via PePr is a good idea. pepr_update will strace all these apps to find out what files they open. This way, it "learns" how to serve you best.

When you are finished with teaching, go back to your console and press ENTER. Now the pepr_update will ask you for the root password. This is because /usr/bin/prepare_preload needs root privileges for some reason (haven't looked closer at it, the program is not from me). The finished files are now written to ~/.pepr.

To reset your PePr configuration, simply delete ~/.pepr. To deinstall PePr, delete all the files you downloaded.

Feedback and improvements can be sent to nordi at addcom dot de.

Application preloading (without PePr)

Since Suse Linux 10.0beta4, this is installed by default, so there is no need to mess with your system. Just install your system and enjoy! The applications that are preloaded by default are Firefox, OpenOffice, Mozilla, Gimp, Khelpcenter and KDE itself. For more details, have a look at the "preload" package.

There have been questions about how to disable preloading for certain applications that are not installed. First of all, preloading is not done for applications that are not installed. E.g. if you don't have Mozilla installed, nothing will be preloaded. If you still want to remove preloading for a certain application, delete the corresponding files in /etc/preload.d/ and /var/cache/preload/.

Preloading if you don't use KDE

Preloading for applications currently only works if you are using KDE as your windowmanager. To get this working for other WMs you need to edit /etc/init.d/boot.local and at the end of the file insert the following line

   /usr/bin/ionice -c3 /usr/bin/appspreloader.sh &

Once you have done this you can delete the file /opt/kde3/share/autostart/appspreload.desktop. You don't need it any longer.

Additionally, KDE itself also gets preloaded which improves startup time a lot. You can also preload GNOME if you follow these steps:

*download the GNOME preload file and save it to /etc/preload.d/
*as root, run the command "update_preload"
*edit /etc/init.d/boot.local and add the following line:
     /usr/bin/ionice -c3 /sbin/preload < /var/cache/preload/gnome.preload &

This reduces GNOME's startup time from 25 seconds to 10 seconds on my system. Note that if you have little RAM (256MB is little for this purpose) the effect may be much smaller or zero.

Benchmarking

If you want to benchmark this keep one thing in mind: Once you have started an application, all of its data is in cache. If you want to benchmark your system with and without preloading, you need to reboot your system in between!

Benchmark results showing the effect of preloading can be seen here. Notice the difference between a vanilla beta3 (no preloading) and a vanilla beta4 (static preloading)!

TODO

• Build a nice rpm for PePr
• PePr currently only works for KDE. Other WMs need to be supported, I just don't know how to make other WMs autostart the script. Keep in mind it needs to be the very last thing the WM does, because we don't want to increase the startup time. Any help is welcome!

== Adding new applications == (not necessary for PePr) If you want to preload another application, for example, Opera, here is what you need to do:

1. First, you will need to copy this script and save it as parse_strace:

#!/bin/bash

test -z $1 && exit -1

#remove all references to private data: /home/ /root/ /tmp/
#remove all references to not cachable file systems: /proc/ /dev/ /sys/
#grep for "/" at the end in case some unwanted lines slipped through


#filter out everything where (l)stat(64) was called
grep stat ./"$1" | grep -v ENOENT | grep -v fstat \
    | grep -v /proc/ | grep -v /dev/ | grep -v /sys/ \
    | grep -v /home/ | grep -v /root/ | grep -v /tmp/ \
    | grep / | sed 'sI^.*("II' | sed 'sI".*II' \
    | sort | uniq | grep / | sed 'sI^/Istat /I' > "$1_statted"

#filter out everything where a file was opened
grep -v stat ./"$1" | grep open | grep -v ENOENT \
    | grep -v /proc/ | grep -v /dev/ | grep -v /sys/ \
    | grep -v /home/ | grep -v /root/ | grep -v /tmp/ \
    | grep / | sed 'sI^.*("II' | sed 'sI".*II' \
    | sort | uniq | grep / | sed 'sI^/Iopen /I' > "$1_opened"


#sort them according to path: fewer head movements on the disk
#uniq them: if there is an "open" and a "stat" entry for the same file,
#only the "open" will stay there, no reason to do an extra-stat
cat "$1_statted" "$1_opened" | sort -k 2 | uniq --skip-fields=1 > "$1_preload"

2. Find out the location of Opera. This is usually /usr/bin/, but it's best to check.

3. Create a profile of the program with strace -fF -e open,process opera -o opera_name_strace

4. Parse the file with

• sh parse_strace opera_strace

5. The result you need is opera_strace_preload. Open a terminal, become root and do

• mv opera_strace_preload /etc/preload.d/opera

6. Now enter

• update_preload

7. Now edit /usr/bin/appspreloader.sh in a text editor, and add at the end of the file:

• preload "/usr/bin/opera" "opera"

Then save /usr/bin/appspreloader.sh