Boot time

From openSUSE

Boot time

The SLED desktop team had its semi-annual meeting in July 2006 in the Cambridge office. User:Federico-mena led the sessions to start a project to reduce the time it takes to boot a Suse Linux system.

Status

• 2006/Aug/02 - The project has not started yet; we just have notes on what we need to do.

• 2006/Oct/01 - Work has begun to create the needed scripts. --MBoman

• 2006/Oct/18 - Scripts are uploaded to Boot_time/Scripts. Please note that they have only been tested on openSUSE 10.2 Alpha5. --mboman 21:25, 17 October 2006 (UTC)

• 2007/Apr/10 - Added Boot_time/Break_It_Down. Describes how to break down the boot process in smaller bits for easy (easier!) analyzing. --mboman 04:40, 10 April 2007 (UTC)

Bugs

• 192998: earlygdm is not installed in runlevel 5, so we don't do preloading appropriately.

• 209358: RFE to move network services from boot to NM dispatcher scripts. Bug marked as invalid due to technical reasons.

• 209414: RFE to have NM support init scripts and dependencies. If this one goes through, we might be able to re-open 209358 Bug marked as invalid for an unknown reason.

The plan

• THE GOAL IS TO BOOT IN 5 SECONDS. This means, from the grub screen until gdm loads there should be no more than 5 seconds. Stating a goal lets us know when to stop optimizing, and it also lets us see regressions in performance.

• We need an easy way to get reproducible results. Rebooting the machine every time you want to get timings is just too painful. An initial idea is to have a script roughly like

   telinit 1
   wipe-the-buffer-cache
   bootchartd
   telinit 5
   wait-for-gdm-to-open
   stop bootchartd

• The basic tool is [Bootchart]. This gives you a high-level overview of the boot process: which processes get run, when do they use the CPU and disk, which processes wait for things to happen before they can continue. Bootchart will not give us fine-grained analyses of the various processes, but it lets us find hotspots easily.

• This is the kind of optimization problem where there is no single big hotspot: we'll have to kill 5% here, 2% there, 3% elsewhere, etc. until we reach a goal.

• Suspicious processes: zmd, X startup, apparmor, sshd, nscd.

Ideas and open questions

Some information here : [[1]]

Do we really need the network during boot? Maybe only if we have NFS home directories, or a NIS setup, or Active Directory. We can probably delay launching sshd/nscd/ntp until later. Could we benefit from a systemwide D-Bus service to start the network on demand?

Zmd definitely doesn't need to start running at boot time.

Preload a bit -> boot a bit -> preload a bit -> boot a bit, etc?

Joe Shaw says parallelizing things at boot doesn't really help (Fedora tested this), since everything waits on I/O anyway.

There is overhead to loading kernel modules. Can we delay them?

Once we add instrumentation to individual processes to monitor them, we can simply use syslog to centralize all this logging.

Make the kernel print timestamps before init(1) gets run (i.e. a poor man's kernel profiler).

There's a lot of waiting for udev before "/" gets mounted.

The USB driver is slow when booting. Can we delay it?

How long does Windows take to boot? XP SP2 takes a long time, but the desktop shows quite early, long before the keyboard is available. Of course we could load a png snapshot of the desktop :-).

We need to track how boot performance degrades over time: what happens after you upgrade (not reinstall) your distro? When you update certain packages and the disk becomes fragmented?

Have an opt-in program for openSUSE contributors: automatically mail their performance results back to Novell so that we can see what their machines are doing:

• CPU, RAM, disk from their machines
• Date of installation or upgrade (get this from the Yast logs? stat /etc/SuSE-release?)

services should be separated into three categaries: needed, optional, self-destroy. needed service must be started when booting, for example, dbus. optional service should be delay, for example, powersave service can be delayed. self-destroy service should be run only one time, for example, first boot.

of course, all this service can be configurable.

optional services should be loaded on demand when the first user log into the system, he will enable all this optional service(maybe we can also make it run on demand). how we can do this? via dbus, normal users also can run anything(here, we should think about security issue). I called this "daemondaemon", means a daemon manage other daemons.

another topic is "service on demand" as I said before, for example, if the system had not share any files to others, samba should be delayed until we need it. loop back network should be enabled at boottime, but local network(eth0, etc.) can be delayed if we had no need to auth from other machine, or we do not need a remote X login.

as we had done in Everest Linux 0.2, booting time is less than 30 seconds, normally, less than 25 seconds.

we had no need to think about prefetch or preload, it can speed up the application start time, but do nothing for booting time, even make it slower than before.

Is it necessary to scan for new hardware, just go with the hardware from the last boot may be way faster. Add a tool for hardware scanning at runtime. This can also be used as a security feature, as new possibly malcious hardware will not be used in the system.

Just an idea: I think the fastest way of really fast boot would be something like a swap-partition or a special sector/space of the partition, which will hold the following data: after the user has set his desktop up, the whole memory/settings are saved as a block in that special partition/sector. Like hibernation. When the user wants to directly boot his standard desktop, he can choose it from grub (fast boot?), and the hard disk can fly over the partition, read that block and put it into the memory and directly start (like hibernation). If new hardware is connected and wants to be recognized during boot up and not on runtime, or another reason, the standard boot can be selected. Preloading of application data should also be worthy to implement this way, isn't it? If there would be such a function in openSUSE today, I'd use it everytime I start my computer. And if I changed my standard desktop, I'd just click: menu->system->configuration->reconfigure fast boot. It asks me whether I really want to set the new fast boot session like the one I have now. I click continue, memory is written, and from now on I can just profit from it. It'd be something really amazing, and in my eyes a new interesting idea to solve that "problem". Booting in 5secs would be really great.

Pending patches

Sun has a patch to GDM to make it preload the gnome-session stuff while GDM is waiting for you to type your username/password.

Discussion

CUPS needs USB to be loaded, to probe for printers. Is CUPS aware of HAL/hotplug?

Quotes from Robert Love and Greg KH:

> > I also have a kernel patch that makes all of the device discovery work 
> > in a multi-threaded way.  It speeds up the kernel boot time a _lot_ but
> > causes some "interesting" issues with some hardware and drivers.  You
> > can very easily blow up SCSI power supplies if you aren't careful, but
> > for my tiny laptop, I shave a full second off of the time the kernel
> > takes to starting up init, which is very respectable.
> 
> I did multi-threaded drive discovery way back when, for MontaVista.  I
> think it was against 2.4.  I don't recall the gains on small machines;
> our goal at the time was very large ones.  On my workstation, scanning
> the entire SCSI LUN space is still the largest consumer of boot time.
> But that might just be the way it is.  Interesting to see what gains (a)
> multi-threading everything on (b) small machines brings.

On my 4-way workstation, with SATA (not scsi, sorry), the speedups are
around 4-5 seconds.  And I need to tweak that to start running init when
only the root disk is found, not to wait around till all devices are
found.

ZMD

AutoYaST may need ZMD in some situations after the initial installation. Starting ZMD on demand there may not be a problem.

Quote from Ingo Tambet:

zmd is also the ZLM remote management daemon so it can't be started on
demand.

There's actually two different daemons in zmd. There's one, very light
weight watcher daemon that only opens communication sockets and listens
for incoming requests. When a request comes in, it'll exec the real
(heavy-weight) deamon and exit. When there's certain time of inactivity,
the real daemon will shut down and exec the watcher daemon. 

Currently, at startup, we always launch the real daemon. What we can do,
(in addition to optimizing the real daemon) is to start the watcher
daemon only, but I'm not sure what it would buy us - the zen-updater
would trigger the start of the real daemon anyway.