After my successful experiments with SLES10 and Xen, I wanted to get my hands really dirty and setup a Gentoo Xen DomU from scratch.

Main source of information was this excellent entry in the Gentoo Wiki. I basically follow the instructions in the section “Gentoo domU using quickpkg”. Compared to the description I used a LVM2 volume for the root-fs, which was partitioned into 2Gb of swap and about 20Gb space for the root-fs (remember the kpartx utility). Since I’m running multiple Gentoo systems, I’m already emerging each new package with the --buildpkg option, so that I had a complete set of binary packages immediately available. Furthermore I’m sharing the /usr/portage tree and the location, were I keep the binary packages, via NFS. This required two additional mount -o bind to mount the trees at the correct positions, so that the content would be available in the chroot environment. I’m running all my other Gentoo systems with the same /etc/make.profile, which is actually the x86/2007.0/desktop profile, although two systems operate more like a server than a desktop system. Anyway, I didn’t bother to change this to server for the Xen DomU.

As it turned out, the biggest obstacle was an out of date or out of sync system set for the invocation of

ROOT=/xen-gentoo/ \
    CONFIG_PROTECT=-/etc \
    FEATURES=-collision-protect \
    emerge --usepkg --emptytree system

The system, on which I did all this, evolved during a couple of years, having been cloned a couple of times, but never really reinstalled from scratch. Apparently the database for the system set become somewhat outdated, therefore I had to repeat the above emerge a couple of times, before I had a basic workable chroot environment. Once I had everything installed, what I wanted and had a consistent system (after some revdep-rebuild‘s) I ended up with a root-fs of about 1.2GB in size.

After the suggestions of the Gentoo Wiki I ended up with this Xen configuration file:

# general
name    = "xen1";
memory  = 256;

# booting
kernel  = "/boot/kernel-genkernel-x86-xen-2.6.20-xen-r4";

uuid = "f9ac60b6-6ad5-90f9-1165-f600453cfe83"

# virtual harddisk
disk = [ "phy:/dev/mapper/xen-xen1,xvda,w" ];
root = "/dev/xvda2 ro";

# virtual network
vif = [ "mac=00:16:3e:51:af:c9" ];
dhcp = "dhcp";

The uuid and the mac address were obtained after the first start of the Xen DomU by executing

xm list -l

Again, in the end I was pretty surprised how smoothly everything went. Next step will be to install the Endian Firewall (Community Edition) into a Xen DomU.

My desktop system in the company is a 2800MHz Intel P4 system running Gentoo Linux. I’ve been using hibernation with TuxOnIce (former Suspend2) with very good success for quite some time now.

However I’ve started to see a problem with it now. I don’t know, if there is any relation to the recent kernel update update to 2.6.22, but after hibernation the second (hyperthreaded) CPU doesn’t come online after a couple of days. Here is an extract from the dmesg-output, were the 2nd CPU came correctly online.

Suspend2 debugging info:
- Suspend core   : 2.2.10
- Kernel Version : 2.6.22-suspend2-r2
- Compiler vers. : 4.1
- Attempt number : 6
- Parameters     : 0 81936 0 1 0 0
- Overall expected compression percentage: 0.
- Compressor is 'lzf'.
  Compressed 833339392 bytes into 574195838 (31 percent compression).
- SwapAllocator active.
  Swap available for image: 268357 pages.
- FileAllocator inactive.
- I/O speed: Write 61 MB/s, Read 69 MB/s.
- Extra pages    : 37 used/500.
Enabling non-boot CPUs ...
SMP alternatives: switching to SMP code
Booting processor 1/1 eip 3000
Initializing CPU#1
Calibrating delay using timer specific routine.. 5584.82 BogoMIPS (lpj=2792411)
CPU: After generic identify, caps: bfebfbff 00000000 00000000 00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
CPU: After all inits, caps: bfebfbff 00000000 00000000 0000b080 00004400 00000000 00000000
CPU1: Intel(R) Pentium(R) 4 CPU 2.80GHz stepping 09
CPU1 is up

As I wrote, this worked perfectly for a couple of days, but today morning, when the system came out of hibernation I’m seeing this:

Enabling non-boot CPUs ...
SMP alternatives: switching to SMP code
Booting processor 1/1 eip 3000
Not responding.
Inquiring remote APIC #1...
... APIC #1 ID: failed
... APIC #1 VERSION: failed
... APIC #1 SPIV: failed
skipping cpu1, didn't come online
Error taking CPU1 up: -5

The last time this happened it could only be fixed by rebooting.

Recently an IBM DB2 expert dropped by to give a colleague and me a short 2 days introduction of this database. I had organized for an unused AMD64 server to be used as a playground. I thought, it would be nice if I could provide each of us with an independent environment. That’s when I thought about Xen. I had enjoyed a brief Xen introduction during the SLES10 boot camp in Nürnberg, therefore I though Xen might be the way to go. I could’ve used VMware Server, but since I intended to install a distributed SAP Netweaver system after out DB2 introduction, I thought the paravirtualisation of Xen might give me a performance benefit.

Anyway the setup of three DomU with SLES10 was a breeze. First the relevant Xen utilities and tools in addition to the kernel packages must be installed. After rebooting I defined three Xen DomU with a nice GUI. Common wisdom on the Net is, that LVM2 slices work better than simple file images. So I sliced the second hard disc of the system into three pieces under control of LVM2. Since I had access to a networked Novell SLES 10 installation server, I considered doing three installs from scratch just as easy as if I would harvest the file tree from some existing installation. Apart from the manual input for the SLES 10 installation process, the whole setup process was graphical. I had three DomUs with SLES10 with only a couple of mouse clicks. The whole process was a very pleasant experience.

After that, I had to shuffle the data within the LVM2 volumes a little bit around, to allow the repartition within of the LVM2 slices. I had SLES 10 installed with 1.5Gb of swap and the rest for the root-fs. However 1.5Gb swap is not enough if you intend to fiddle with SAP Netweaver in a DomU. Configure more than 2Gb, best go directly for 4Gb of swap and of course have enough main memory. This particular system was equipped with 8Gb. I configured 2Gb for each DomU. In the end I really did succeed to install the various SAP instances into the three DomUs in a distributed setup.

During this data shuffling I learned about another important command: kpartx. You need this command, if you want to access the partitions within the logical volume. Consider these outputs:

# lvs
  LV    VG     Attr   LSize  Origin Snap%  Move Log Copy%
  root1 xen    -wi-a- 24.84G
  root2 xen    -wi-a- 24.84G
  root3 xen    -wi-a- 24.84G

The volume group xen is presenting the whole 80Gb disc. Then there are 3 logical volumes, each about 25Gb in size. Using fdisk on one of these logical volumes displays this output:

# fdisk -l /dev/mapper/xen-root1

Disk /dev/mapper/xen-root1: 26.6 GB, 26675773440 bytes
255 heads, 63 sectors/track, 3243 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

                Device Boot      Start         End      Blocks   Id  System
/dev/mapper/xen-root1p1               1         312     2506108+  82  Linux swap / Solaris
/dev/mapper/xen-root1p2   *         313        3243    23543257+  83  Linux

As you can, the SLES10 installation process created two partitions within the logical volume. Now, how can I access root-fs of this DomU? The devices /dev/mapper/xen-root1p* don’t actually exist.

Here is, where the kpartx command comes into the picture. kpartx belongs to the Multipathing tools. It installs specific mappings and actually creates devices, so that you can mount the partitions within the logical volume. The -l option lists, -a adds and -d deletes the partitions devmappings.

# kpartx -l /dev/mapper/xen-root1
xen-root1p1 : 0 5012217 /dev/mapper/xen-root1 63
xen-root1p2 : 0 47086515 /dev/mapper/xen-root1 5012280

# kpartx -a /dev/mapper/xen-root1

# ls -l /dev/mapper/xen-root1*
brw------- 1 root root 253, 2 Sep  3 10:24 /dev/mapper/xen-root1
brw------- 1 root root 253, 5 Sep 12 15:11 /dev/mapper/xen-root1p1
brw------- 1 root root 253, 6 Sep 12 15:11 /dev/mapper/xen-root1p2

Now you can mount the device /dev/mapper/xen-root1p2 and operate in the root-fs of the corresponding Xen DomU.

After this pleasant experience with Xen and SLES10 I wanted to get my hands a little more dirty, this time on my preferred Gentoo distribution. “May the source be with you.” At the time, when I started my first experiments, the latest Xen version in Portage was still 3.0.4. So I downloaded the 3.1 sources from xensource and compiled the corresponding 2.6.18 Linux kernel and even succeeded in creating a initrd/initramfs from the Xen build tree with the Gentoo genkernel utility. After the appropriate modifications for the Grub bootloader the newly created kernel started to boot, but then I hit a wall. The boot would hang during the load of the aic79xx module in the initrd, the module for the onboard hard disc controller. I tried what I could think off, but I couldn’t get beyond this point. Also studying various How-To’s from the Net didn’t supply any hint, what my problem might be.

Anyway, some time passed with other activities. In the meantime the ebuilds for Xen 3.1 became available in the Portage tree and the xen-sources ebuild for the kernel was even updated to 2.6.20. So I retried with this version. However I hit the same block. The initrd wouldn’t finish to load the aic79xx module. I can only say: RTFM. When I finally read the Xen documentation it hit me right between the eyes. The are a couple of arguments, that also need to be specified for the Xen hypervisor. On my particular system the kernel arguments noapic and pci=noacpi are required to boot a standard Linux kernel. Once I supplied similar arguments to the loading of the Xen hypervisor my problem was solved. I actually used the arguments noapic and acpi=off.

So, the thing to learn here, if you need any special arguments for booting like for instance acpi=off you have to check with the Xen documentation, if the arguments have to be issued to the hypervisor as well.

Out of curiosity, since I had the Xen kernel now running, I tried if I could get my VMware virtual machine to run. That didn’t went down so well. To bad in fact, that the VMware disc of the Windows VM became unusable due to the resulting kernel panic.

My default KDE desktop configuration consists of 4 virtual desktops. The first desktop is the Web browsing desktop, the second usually used for programming stuff, the third for audio editing and the fourth for the occasional Windows session either through rdesktop or VMware server.

There is one konsole application running with multiple tabs, which is sticky to all desktops. Within each of the tabs I like to have each of the shells it’s own history file. I could use the tty name to associate each shell history with a unique name, but depend on the activity on the system, the tty name might change within a particular tab. So I looked at the shell environment and found, that konsole passes two dcop related variables.

$ set|grep DCOP 
KONSOLE_DCOP='DCOPRef(konsole-9342,konsole)'
KONSOLE_DCOP_SESSION='DCOPRef(konsole-9342,session-2)'

Looking through the konsole dcop functions with kdcop I came across the currentSession function.

$ dcop $KONSOLE_DCOP currentSession
session-2

Contrary to what you might think, this doesn’t provide you with the session name of the tab the shell is currently executing in. This function returns the session name of the tab, which currently is selected and has the focus.

In the end I included this line in my .bashrc.

HISTFILE=$HOME/.bash_history_$(echo $KONSOLE_DCOP_SESSION|sed -e 's/^[^,]*,//' -e 's/)//')

The sed removes everything from around the session-2 of DCOPRef(konsole-9342,session-2).

I had hoped, that there is some direct dcop function, what would return the session name the shell is currently executing in, but apparently there is no such thing (or I didn’t find it). That would have saved my from the sed invocation.

I’m frequently recording either TV or Radio shows with my two installed Hauppauge TV/Radio cards. The recording may end any time during the night. I would switch off the display and leave the KDE session running (this is important, since otherwise the termination of the session would result in resetting the permissions of devices in the /dev directory and the recording software running under my user-id subsequently wouldn’t be able to open them). To save energy, my intention is to shut the system down, once the recording is complete.

In the past I’ve been using the command “sudo /sbin/halt” either directly from within the recording script or executed by the at-command. This mostly worked OK, but sometimes it also happened, that the shutdown wouldn’t complete and the system stayed powered on. I was always to lazy, to really dig into this, what the possible reasons might have been. Another problem with this approach was, that the KDE session was not cleanly terminated.

Anyway, I have never observed this behaviour, when I correctly terminated the KDE session and selected “System Shutdown”. After a bit of searching and googling I found, that the following is essentially executed:

dcop ksmserver ksmserver logout 0 2 2

Since I changed to this command invocation at the end of my TV/Radio recording sessions, I never had a problem with shutdown’s, which didn’t actually power off the machine.

In the end I completely stopped using the at-command. I’m now using the KDE kalarm utility. It has a very nice graphical user interface. Some times the nicest things are so close to you. You only have to see them.