Boottime/Boot time

From openSUSE

< Boottime(Redirected from Boot time)

Contents

Overview

Background

The SLED desktop team had its semi-annual meeting in July 2006 in the Cambridge office. Federico led the sessions to start a project to reduce the time it takes to boot a Suse Linux system.

Starting January, 2009, SLE is again the focus of efforts to cut down on boot time - this time, across the whole distribution, investigating kernel, system, X and desktop (initially GNOME) startup time. Our basis is SLE 11, with an eye towards getting improvements into openSUSE Factory and upstream. openSUSE 11.1 is being used as a public reference platform.

Goals

The goal is to boot in 20 seconds. This means that from Grub until the desktop is usable and auto-logged-into (file browser + panel), no more than 20 seconds should pass. Stating a goal lets us know when to stop optimizing, and it also lets us see regressions in performance.

Our first deliverable will be a set of packages for SLE 11 that can be used to replace some of the existing packages in order to improve boot performance.

The timeframe for this project is not fixed, but we want to be able to show significant results - a set of packages that reduce SLE 11 boot time to near 20s on reference hardware - by March 1st.

Resources

Collaboration

Reference System

We're performing regular tests on a reference Lenovo S9e system equipped with both an SSD and a mechanical hard drive. Boot timings are provided for both the SSD and regular HD. Hans Petter is currently performing these tests. Send patches his way.

Hardware

Lenovo S9e

  • Atom N270 processor
  • 512MB RAM
  • 4GB SSD
  • 100GB HD
  • Intel Integrated 950 GPU

Software

  • No network connected.
  • openSUSE 11.1 installed from media via USB.
  • GNOME desktop.
  • Single partition mounted on / with noatime.
  • No swap partition.
  • Default package selection.
  • Automatic configuration.
  • Deployed via images.
  • Autologin of "test" user.
  • Allow Yast to probe for all hardware.
  • Boot system once and accept the first-login greeting.
  • Don't modify anything else, and shut the system down.
  • The system is now ready for measurements.

Results

Stopwatch

Stopwatch timings measuring the time from hitting enter in grub, to the desktop and panel look usable, with no bootchart or other instrumentation running.

  • February 4th, 2009: 26.5s
  • February 13th, 2009: 23.3s

Bootcharts

Alternate setup

Lenovo S9e

Distribution modifications

This section will contain detailed descriptions of the changes we've made to the distribution in order to reduce boot times.

Packages

Changes made to individual packages - usually in the form of updated specfiles, patches or in rare cases, upgraded source tarballs.

Image

Image-level changes.

Disabled autostart programs

The following programs should be disabled in the GNOME autostart:

  • at-spi-registryd-wrapper.desktop
  • beagled-autostart.desktop
  • ggreeter.desktop
  • gnome-at-session.desktop
  • gnome-docker.desktop
  • gnome-session-splash.desktop
  • hplip-systray.desktop
  • vino-server.desktop

They can be disabled either by copying the desktop files to the user's $HOME/.config/autostart/ dir and editing the X-GNOME-Autostart-enabled= line, or editing the desktop file directly in /usr/share/gnome/autostart/.

Solid-color background

Set the default GNOME background to be solid-color (instead of a picture):

/usr/bin/gconftool-2 --direct --config-source                     \
  xml:readwrite:/etc/gconf/gconf.xml.defaults --set --type=string \
  /desktop/gnome/background/color_shading_type "solid"

Tools

Bootchart (system/desktop)

Bootchart is a general, yet powerful tool - it gives you a high-level overview of the boot process: Which processes get run, when do they use the CPU and disk, which processes wait for things to happen before they can continue. Bootchart will not give you fine-grained analyses of the various processes, but it lets you find hotspots easily.

Useful for both system and desktop startup.

Installation

  1. Install bootchart with BSD process accounting:
    zypper install acct bootchart
    cp /usr/sbin/accton /sbin/
  2. Make an initrd if you want your profiling to start as early as possible:
    mkinitrd
  3. Add a bootchart item to your grub menu...
    joe /boot/grub/menu.lst

    ...and copy the existing normal boot section to a new entry and make it run bootchart for init (changes in copied section in boldface):

    # Modified by YaST2. Last modification on Fri Jan  9 13:47:31 UTC 2009
    default 0
    timeout 8
    ##YaST - generic_mbr
    gfxmenu (hd0,1)/boot/message
    ##YaST - activate
    
    ###Don't change this comment - YaST2 identifier: Original name: linux###
    title openSUSE 11.1 - 2.6.27.7-9
        root (hd0,1)
        kernel /boot/vmlinuz-2.6.27.7-9-default root=/dev/sda2 resume=/dev/sda1 splash=silent showopts vga=0x332
        initrd /boot/initrd-2.6.27.7-9-default
    
    ###Don't change this comment - YaST2 identifier: Original name: linux-bootchart###
    title openSUSE 11.1 - 2.6.27.7-9 - Bootchart
        root (hd0,1)
        kernel /boot/vmlinuz-2.6.27.7-9-default root=/dev/sda2 init=/sbin/bootchartd resume=/dev/sda1 splash=silent showopts vga=0x332
        initrd /boot/initrd-2.6.27.7-9-default
    

Usage

  1. Restart the computer.
  2. Boot the bootchart entry from grub.
  3. As regular user, issue:
    bootchart
  4. View the resulting image.

Bootgraph (kernel)

There's a script for profiling the kernel startup in kernel/scripts/bootgraph.pl. It has a comment in it explaining its use.

Federico's timeline (desktop)

This is a visualization tool intended for deep analysis of individual apps. It was written for desktop apps, but works with anything that can be straced. It can be used to generate precise analysis, but requires you to instrument the source code manually and recompile the application. The scripts can be checked out from Gitorious.

There's also a short how-to for applying this to Mono apps.

finit

  • Fastinit is a proprietary and much simplified init(8) replacement written by Asus to help the Eeepc to boot faster. Finit is a fastinit reimplementation based on fastinit system calls with gaps filled with frog DNA.

Finit's purpose is to build upon the general workings of fastinit and improve/optimize it for Xandros or alternative distributions running on the Eeepc. Finit comes in three flavors: finit is a replica of the Asus fastinit, finit-mod is an improved drop-in replacement for fastinit, and finit-alt is a generic replacement for other systems. Finit-mod and finit-alt were much improved with patches from Metalshark.

Research

  • Analyzing and Improving GNOME Startup Time (2005). Though dated, this document has a lot to say about methodology. It's very light on the conclusions, though, stating that GNOME startup is I/O-bound and suffers a lot from poor I/O locality (disk seeks).


Action items

  • Fix trivial bug in kernel that causes initrd to be decompressed twice. Savings: 200ms.
  • Bump /sys/proc/sda/queue/nr_requests to 1024 or higher in order to improve readahead performance (CFQ does a better job with more concurrent requests).
  • Boot chart seems to show some problems with our kernel that are not in upstream kernels. Arjan/Olaf to provide details.
  • Compiz - make it take less then 10 secs to start. We suspect the plugin system. David Reveman is looking into this.

Ideas and open questions

Readahead

Looking into sreadahead to do more aggressive readahead.

GConf

GConf is serializing all the requests during bootup, leading to the kernel seeing only one I/O request at a time. Maybe it could be sped up using threads? An idea is to keep complexity down by using a thread pool for reading and writing config files on disk - not parallelizing higher-level CPU-bound operations.

Gettext caching

A quick analysis of an strace from gdm to login, shows we have at least ~50 different ".mo" files being opened, and operated on. Since the contents are hashed, and un-ordered, I anticipate that much of the data is hot in the page-cache. This must consume a large amount of I/O resource in a scattered and hard to trace way.

I propose that we have an LD_PRELOAD hack for the session startup (and boot up?) that incrementally builds a list of the strings we lookup, and creates a small, sorted cache of this data, storable in a single file in the users' home directory. It should intercept the stock gettext calls, and an strace should show no access to a desktop .mo file after a priming run. We should measure the potential win by knocking up the LD_PRELOAD hack, and just returning a single static string "foo" or somesuch from that - to measure the win before investing too much.

Sample LD_PRELOAD intercepting code can be found at: svn+ssh://michael@svn.gnome.org/svn/memprof/trunk

<agruen>I'm wondering if setting LANG/LC_ALL to C shouldn't be enough to disable the .mo files.</agruen>

beagled startup

BeagleDaemon reads a ton of indexes, doing a lot of heavy-duty I/O (sadly) scattered across many directories at startup. We should:

  • defer this until we actually want to do a query, or some indexing
  • stop creating way more indecees that we need - if there is no content, there should be no index:
    • eg. /beagled/KonqBookmarkQueryable/KonqBookmarkQueryable.cs creates 40K of index in 6 files and 4 directories, regardless of whether we have any bookmarks at all to edit.
   public LuceneQueryingDriver (string index_name, int minor_version, bool read_only) 
          : base (index_name, minor_version)
   {
       // FIXME: Maybe the LuceneQueryingDriver should never try to create the index?
       if (Exists ())
           Open (read_only);
       else if (!read_only)
           Create (); // <<-- ! :-)

Looks like we need to avoid doing that, and try to defer work here.

Having dug at this approach - it seems that avoiding opening databases we don't need is more work than necessary : we should just defer the query Lucene thing init until much, much later.

Tomboy startup

There's some great performance work being done upstream to make Tomboy faster and leaner. Tomboy uses too much memory at startup. Even though this is focused on memory use, it also helps startup time, as it avoids unnecessary work.

Miscellaneous

  • Preload - actually do some, and do it with a very low I/O prority, and as aggressively as possible, from early in the boot - cf. moblin.
  • Kill the bash / xrdbcomp nonsense from gnome-session.
  • Back-port the fixes from Behdad to gnome-settings-daemon.
  • Brutally force nautilus to get its background drawn as early as possible.
  • Stop python starting up at all (is this the print applet, does it need a trivial C wrapper to stop it?).
  • Disable tomboy's over-flexible 'plugin' system, or write a simple in-proc icon-applet wrapper around it until it is clicked.
  • Investigate why it takes so long to get to the last stage of startup where we launch applets etc.
  • Some info from IBM: [[1]]
  • Do we really need the network during boot? Maybe only if we have NFS home directories, or a NIS setup, or Active Directory. We can probably delay launching sshd/nscd/ntp until later. Could we benefit from a systemwide D-Bus service to start the network on demand?
  • Preload a bit -> boot a bit -> preload a bit -> boot a bit, etc?
  • Joe Shaw says parallelizing things at boot doesn't really help (Fedora tested this), since everything waits on I/O anyway.
  • There is overhead to loading kernel modules. Can we delay them?
  • Once we add instrumentation to individual processes to monitor them, we can simply use syslog to centralize all this logging.
  • Make the kernel print timestamps before init(1) gets run (i.e. a poor man's kernel profiler).
  • There's a lot of waiting for udev before "/" gets mounted.
  • The USB driver is slow when booting. Can we delay it?
  • We need to track how boot performance degrades over time: what happens after you upgrade (not reinstall) your distro? When you update certain packages and the disk becomes fragmented?
  • Have an opt-in program for openSUSE contributors: automatically mail their performance results back to Novell so that we can see what their machines are doing:
    • CPU, RAM, disk from their machines
    • Date of installation or upgrade (get this from the Yast logs? stat /etc/SuSE-release?)
  • Is it necessary to scan for new hardware, just go with the hardware from the last boot may be way faster. Add a tool for hardware scanning at runtime. This can also be used as a security feature, as new possibly malcious hardware will not be used in the system.
  • Suspicious processes (in 10.3): X startup, apparmor, sshd, nscd.
  • Kernel I/O latency bug - does this affect us?

Other projects

 e.g. `alsasound status` goes from 28ms (bash) to 12 ms (dash), this does not sum up. The resulting boot time 
 is the same with dash and bash.

Discussion

CUPS needs USB to be loaded, to probe for printers. Is CUPS aware of HAL/hotplug?

Quotes from Robert Love and Greg KH:

> > I also have a kernel patch that makes all of the device discovery work 
> > in a multi-threaded way.  It speeds up the kernel boot time a _lot_ but
> > causes some "interesting" issues with some hardware and drivers.  You
> > can very easily blow up SCSI power supplies if you aren't careful, but
> > for my tiny laptop, I shave a full second off of the time the kernel
> > takes to starting up init, which is very respectable.
> 
> I did multi-threaded drive discovery way back when, for MontaVista.  I
> think it was against 2.4.  I don't recall the gains on small machines;
> our goal at the time was very large ones.  On my workstation, scanning
> the entire SCSI LUN space is still the largest consumer of boot time.
> But that might just be the way it is.  Interesting to see what gains (a)
> multi-threading everything on (b) small machines brings.

On my 4-way workstation, with SATA (not scsi, sorry), the speedups are
around 4-5 seconds.  And I need to tweak that to start running init when
only the root disk is found, not to wait around till all devices are
found.

Miscellaneous

Older discussion

services should be separated into three categories: needed, optional, self-destroy. needed service must be started when booting, for example, dbus. optional service should be delay, for example, powersave service can be delayed. self-destroy service should be run only one time, for example, first boot.

of course, all this service can be configurable.

optional services should be loaded on demand when the first user log into the system, he will enable all this optional service(maybe we can also make it run on demand). how we can do this? via dbus, normal users also can run anything(here, we should think about security issue). I called this "daemondaemon", means a daemon manage other daemons.

another topic is "service on demand" as I said before, for example, if the system had not share any files to others, samba should be delayed until we need it. loop back network should be enabled at boottime, but local network(eth0, etc.) can be delayed if we had no need to auth from other machine, or we do not need a remote X login.

as we had done in Everest Linux 0.2, booting time is less than 30 seconds, normally, less than 25 seconds.

we had no need to think about prefetch or preload, it can speed up the application start time, but do nothing for booting time, even make it slower than before.

See also older pages, Boottime/ReduceBoottime Boottime/Fast_boot.