SDB:KIWI Cookbook Data Separation
All of KIWI edit
Data separation or handling Partitioning
This example provides a simple approach to separating application data from OS data.
It appears that whenever the use of an OEM image is discussed in any kind of setting it is inevitable that someone asks about partitioning. However, what most people are really after is the separation of application data from the OS data by means of a partition. There is a subtle but important distinction in play that will be addressed below. This recipe shows how to achieve data separation for an OEM image.
Data separation via partitions
Prep Time:
- 20 min
Cook Time:
- 15 min
Ingredients:
- a running openSUSE 11.1, 11.2, or 11.3 system
- an openSUSE 11.1, 11.2, or 11.3 repository
- installed recent release of the KIWI toolset
- about 1 GB free disk space
Partitioning vs. Data separation via partitions
When speaking about partitioning, one has to consider the whole spectrum of possibilities. This includes arbitrarily complicated partitioning schemes that may or may not include partitions for directories that the OS cares about. This type of partitioning is supported during a regular SUSE interactive installation and during an installation via AutoYaST. In this use case a user may, if so desired, create partitions for /opt, /var, /usr, /home, etc. Basically directories that have "special" meaning to the OS.
This works for interactive installation and AutoYaST as a process runs before the actual installation of software occurs on the target system. Once software installation begins packages are simply unpacked into the appropriate target directory and the installer itself does not care whether the target directory is mounted on a specific partition or the given directory is simply part of the root (/) partition.
For Kiwi this scheme does not work. When a Kiwi image is created the installation of software takes place in a directory, provided by the --root argument for the prepare step. In this installation mode there is by definition no notion of partitions, the only thing that exists is a directory. In the create step this directory is "simply" transformed into a disk image, for the vmx or oem image types. During OEM installation the disk image is then dumped to the target storage media and thus we end up with basically a one partition (spare the swap space) system. The reasons for this implementation can be explained as follows:
- There is no process that runs ahead of installation when installing an OEM system from an ISO image or a USB stick onto a target machine. Unlike in the interactive and Auto YaST installation. As soon as the install media boots we are in the Kiwi created process that dumps the image onto the target storage device. Therefore any partitioning scheme would have to be integrated into this self install process.
- Partitioning can be arbitrarily complicated, depending on the target system storage configuration, the use of primary and extended partitions, file system type, partition sizes etc. This quickly leads to a management overhead night mare. General partitioning code with probing, calculation, error handling etc. is not only complicated to implement and maintain inside the extremely minimal self install environment, but also bears the considerable risk that one would end up with a non working system after the OEM dump. The later being a rather embarrassing occurrence if it were to happen at a customer.
- General partitioning is not really needed in the context of appliances. When people speak of partitioning in the context of appliances what is generally desired is a separation of data for the application(s), that are part of the appliance, and the data (such as /usr, /bin, /sbin, etc.) that is considered part of the OS. This is where the subtle but significant difference between general partitioning and data separation via partitioning lies.
With this in mind the remainder of this example will provide all the necessary information to achieve data separation via partitioning without requiring general support of a partitioning feature in Kiwi.
Kiwi image configuration
This example uses a very simple config.xml file as the main focus will be on the creation of the data partition, mounted to a directory called myData.
Basic setup
Create a directory to be used as the Kiwi configuration directory.
The config.xml file
Use your favorite editor and create /tmp/dataSep_exa/config.xml. You may cut and paste the example file below , create your own content from scratch or modify the example to include the packages that you would like to have included.
<?xml version="1.0" encoding="utf-8"?>
<image schemaversion="4.7" name="suse-11.3-oem-dataseparation">
<description type="system">
<author>Robert Schweikert</author>
<contact>rschweikert at novell dot com</contact>
<specification>
openSUSE 11.3 based example showing data separation based on
partitions
</specification>
</description>
<preferences>
<type image="oem" filesystem="ext4" boot="oemboot/suse-11.3" installiso="true">
<oemconfig>
<oem-boot-title>Data-Separation</oem-boot-title>
<oem-home>false</oem-home>
<oem-swap>true</oem-swap>
<oem-swapsize>4096</oem-swapsize>
<oem-systemsize>8192</oem-systemsize>
</oemconfig>
</type>
<version>1.0.0</version>
<packagemanager>zypper</packagemanager>
<keytable>us.map.gz</keytable>
<timezone>US/Eastern</timezone>
<rpm-excludedocs>true</rpm-excludedocs>
</preferences>
<users group="root">
<user pwd="linux" pwdformat="plain" home="/root" name="root"/>
</users>
<users group="users">
<user pwd="linux" pwdformat="plain" home="/home/tux" name="tux"/>
</users>
<repository type="yast2">
<source path="opensuse://11.3/repo/oss/"/>
</repository>
<packages type="image" patternType="plusRecommended">
<package name="kernel-default"/>
<package name="ifplugd"/>
<package name="python"/>
<package name="vim"/>
<opensusePattern name="default"/>
</packages>
<packages type="bootstrap">
<package name="filesystem"/>
<package name="glibc-locale"/>
</packages>
</image>
There is nothing really all that special about the previous configuration file. Note that the <oem-systemsize> element is used to limit the size of the OS image; 8 GB in this example. Within 8 GB one can pretty much install all packages distributed with the standard SUSE distribution. The packages and patterns selected in the example config.xml file above reflect a minimum set, as the example is intended to demonstrate the concept rather than produce a useful appliance. Python is included to execute the script used to create the data separation.
The <oem-systemsize> element limits the storage space used by the root partition, thus leaving any extra space on the storage device untouched. Without the use of the <oem-systemsize> element the root system image will expand to the available storage space on the device thus leaving no room for extra partitioning operations. Therefore, the use of the <oem-systemsize> element is required if you need to create additional partitions after the system has been dumped to the target storage device.
If you are including your own application in the build you might want to add an appropriate amount of disk space to accommodate the binaries for your application. Do not add space for your application data, as we will put the data on a separate partition, which is after all the point of this example.
The config.sh file
The config.sh script for the configuration is not shown as it is pretty much a "standard" config.sh that removes info files, etc. as in previous recipes.
Partition setup
First create a place for the script in our Kiwi configuration tree.
Then with your favorite editor create the file /tmp/dataSep_exa/root/sbin/setUpPartitions. Cut and paste the example script below to follow along the example or modify the script as it applies to your needs.
#!/usr/bin/python
import os
import time
def getSizeAndUnit(valueStr):
"""Extract number and unit information from a string of the format 111MB"""
size = ''
idx = 0
for char in valueStr:
if char.isdigit() or char == '.':
size += char
idx += 1
unit = valueStr[idx:]
return size, unit
diskInfoCmd = '/usr/sbin/parted %s print' %(os.environ["imageDiskDevice"])
diskInfo = os.popen(diskInfoCmd).readlines()
partCnt = 0
diskSize = ''
diskUnit = ''
lastPartEnd = ''
lastPartEndUnit = ''
foundPartTbl = None
numPartitions = 0
for diskLn in diskInfo:
if diskLn.find('Disk') != -1:
diskSize, diskUnit = getSizeAndUnit(diskLn.split(':')[-1].strip())
continue
if diskLn.find('Number') != -1:
foundPartTbl = 1
continue
if foundPartTbl and diskLn.split():
numPartitions += 1
lastPartEnd, lastPartEndUnit = getSizeAndUnit(diskLn.split()[2])
if (lastPartEnd == diskSize) and (lastPartEndUnit == diskUnit):
print "No free space on device"
exit(1)
# Create the partition using all available space
partCmd = '/usr/sbin/parted %s mkpart primary %s%s %s%s >& /dev/null' %(
os.environ["imageDiskDevice"],
lastPartEnd,
lastPartEndUnit,
diskSize,
diskUnit)
os.system(partCmd)
# Force the partition table to be re-read
partReadCmd = 'partx -a %s >& /dev/null' %(os.environ["imageDiskDevice"])
os.system(partReadCmd)
# Create the file system on the new partition
fsCmd = '/sbin/mkfs -t ext4 %s%d >& /dev/null' %(os.environ["imageDiskDevice"],
numPartitions + 1)
os.system(fsCmd)
# Detremine disk id based on existing fstab entry
fstabData = open('/etc/fstab', 'r').readlines()
diskID = ''
for fstabLn in fstabData:
if fstabLn.find('by-id') != -1:
diskID = fstabLn.split()[0]
break
# Create an entry in /etc/fstab
fstab = open('/etc/fstab', 'a')
fstab.write('\n\n#Autogenerated data partition information\n')
fstab.write(diskID[:diskID.index('part')])
fstab.write('part%d' %(numPartitions + 1))
fstab.write(' /myData')
fstab.write(' ext4')
fstab.write(' defaults 1 2\n')
fstab.close()
# Give the system some time to catch up to the changes
time.sleep(10)
# Create a mount point and mount he new partition
os.mkdir('/myData')
mntCmd = '/bin/mount /myData >& /dev/null'
os.system(mntCmd)
# Clean up
os.system('rm /tmp/partSetup')
bootInfo = open('/etc/init.d/boot.local', 'r').readlines()
bootLocal = open('/etc/init.d/boot.local', 'w')
skipBlock = None
for ln in bootInfo:
if ln.find('# Begin init data setup') != -1:
skipBlock = 1
if ln.find('# End init data setup') != -1:
skipBlock = None
continue
if skipBlock:
continue
bootLocal.write(ln)
bootLocal.close()
os.system('rm sbin/setUpPartitions')
Save the file and add execute permissions.
Lets take a closer look at the script. First a function is defined that separates a string of the form "12.6GB" into two strings. The first string returned is the size, "12.6" in this case, and the second one is the unit. "GB" in this example.
Following the function definition we collect the information provided by parted /dev/sda print such that we can parse the data and extract what is useful to us.
In this recipe it is assumed that the image is installed on the first storage device discovered (/dev/sda) and that the data partition will also be located on this device.
The for loop simply examines the output from the parted command and extracts data of interest. The size of the storage device is stored in the diskSize variable extracted from a line in the parted output that starts with Disk. The reported unit for the storage size is stored in the diskUnit variable. The loop also counts the pre-existing partitions (numPartitions counter) and stores the size at which the last existing partition ends in the lastPartEnd variable. The unit of the end of the last partition is stored in the lastPartEndUnit variable.
In this recipe there are two pre-existing partitions, the root partition (/) and the swap partition.
A quick comparison of the end of the last partition and the size of the disk determines if there is space available.
After we know that there is space available the script simply creates a single partition that begins at the end of the previously last partition and takes up all the space available on the device. One can insert an arbitrary algorithm here to create a partition layout as desired. Keep in mind however, that any given storage device may only have 4 primary partitions.
Following the partition creation a file system is created on the partition, ext4 in this example.
Next the script forces the running kernel to re-read the partition table, the partx command, to assure the appropriate device nodes are created in /dev.
After the new device nodes are created the script appends a new line to the /etc/fstab file to make sure our partition is mounted at boot time. After a short sleep, to give the system time to catch up to all the changes, the mount point directory is created and the new partition is mounted.
Last but not least the script cleans up the boot.local file (shown and discussed below) and then removes itself from the system. Leaving no trace of the partitioning magic that just happened.
Kicking of the partitioning process
From the appliance user perspective we want the partitioning setup to be invisible. Further, since everything we need to do can be accomplished on a running system we can simply kick off the partitioning script during the boot process and we do not need to worry about using the Yast Firstboot mechanism.
Create the init.d directory in your Kiwi configuration tree.
With your favorite editor create /tmp/dataSep_exa/root/etc/init.d/boot.local and to follow the example cut and paste the script shown below.
#! /bin/sh
# Begin init data setup
if [ -f /tmp/partSetup ]; then
/sbin/setUpPartitions
fi
# End init data setup
You can see the comments that are used by the partitioning script to clean up and a test for the existence of a trigger file in /tmp.
For a more modular design the partitioning script shouldn't clean up boot.local, that task should be handled by a separate script. However, for this example the mingling of responsibilities will do. In a production environment one should definitely separate the concerns. Anyway, the boot.local script is straight forward. Don't forget to set the execute bit for boot.local and create the trigger file in the Kiwi configuration tree.
This completes the configuration tree setup.
Build and test the Example
There is nothing special about the build when compared to previous recipes.
Now create a disk image with an arbitrary size (250 GB in this example).
With the disk image we can now use qemu to test the OEM image created by Kiwi.
Once the image boots, you will see the self install process as usual, followed by the boot process of the installed system. Once you are at the login screen you can login as root, password linux and list the partition table using the following command:
You will see that there are now 3 partitions on the system. If you run the mount command you'll see that /dev/sda3 is mounted on /myData