Eric Smith

Starting with Fedora 16, you can again use Fedora as a Dom0 environment for running virtual machines under xen.

Install xen

It's fairly simple to get Fedora 16 set up as the Dom0. Just install Fedora 16 as usual. After the installation is complete and you have rebooted, install xen:

# yum -y install xen

This will install a large number of package dependencies.

After installing xen, I disable selinux. While it's probably possible to get selinux running in the Dom0, for my testing purposes I've not gone through the hassle.

The next step is to configure Fedora 16 to automatically boot into xen. To do that, you must tell grub2 to use xen as the default:

# grub2-set-default "Xen 4.1"

Configure network bridging

Next, you'll want to enable network bridging. This will allow all of your DomU's to act as if they're on the same network as your Dom0. To do so, edit /etc/sysconfig/network-scripts/ifcfg-virbr0 to contain:

DEVICE=virbr0
TYPE=bridge
ONBOOT=yes
USERCTL=no
BOOTPROTO=dhcp
NM_CONTROLLED=no

And set your primary ethernet interface to use the bridge. For me, the interface is p49p1, so in /etc/sysconfig/network-scripts/ifcfg-p49p1 I have:

DEVICE=p49p1
HWADDR="01:23:45:67:89:ab"  # actual MAC address here
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
BOOTPROTO=none
BRIDGE=virbr0
NM_CONTROLLED=no

After rebooting, I see this output from ifconfig (with various values hidden):

lo      link encap:Local Loopback
        inet addr:127.0.0.1 Mask:255.0.0.0
        inet6 addr: ::1/128 Scope:Host
        UP LOOPBACK RUNNING  MTU:16436  Metric:1
        RX packets:0 errors:0 dropped:0 overruns:0 frame:0
        TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

p49p1   Link encap:Ethernet HWaddr 01:23:45:67:89:ab
        inet6 addr: xxxx::yyy:zzzz:aaaa:bbb/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
        RX packets:201 errors:0 dropped:0 overruns:0 frame:0
        TX packets:140 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:25946 (25.3 KiB)  TX bytes:15704 (15.3 KiB)
        Interrupt:16 Memory:fbce0000-fbd00000

virbr0  Link encap:Ethernet  HWaddr 01:23:45:67:89:ab
        inet addr:10.10.10.2  Bcast:10.10.10.255 Mask:255.255.255.0
        inet6 addr: xxxx::yyy:zzzz:aaaa:bbb/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
        RX packets:197 errors:0 dropped:0 overruns:0 frame:0
        TX packets:148 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:21884 (21.3 KiB)  TX bytes:16488 (16.1 KiB)

Build the Fedora 14 appliance

The next step is to build and install a Fedora 14 appliance. For this, I use boxgrinder:

# yum -y install rubygem-boxgrinder-build

Then, create a Fedora 14 appliance configuration file named f14.appl:

name: f14-jeos
summary: Just Enough Operating System based on Fedora 14
os:
  name: fedora
  version: 14
hardware:
  partitions:
    "/":
      size: 2
packages:
  - @core

Run boxgrinder to create the appliance:

# boxgrinder-build f14.appl

boxgrinder will download all of the Fedora packages from the installation mirrors and create a Fedora 14 appliance, ready to install and run.

Install virt-install, which will be used to launch the appliance:

# yum -y install python-virtinst

Launch the appliance

Then run virt-install to launch the appliance:

# virt-install --graphics=none --name=f14 --ram=768 --import \
--disk=build/appliances/i686/fedora/14/f14-jeos/1.0/fedora-plugin/f14-jeos-sda.raw \
--network=bridge:virbr0

I'm installing with --graphics=none, meaning that only a text mode console is supported (no VNC). You should now be connected to the DomU's console. From within the Fedora 14 DomU you will be able to find your appliance's IP address and be able to connect to it via ssh.

Adding additional storage

For my Fedora 14 instance, I wanted to add some additional storage. The easiest way is to create a file in the Dom0 storage, then mount that in the DomU. In the Dom0, create the file to be used as storage. In my case, it's 65GB of space:

# dd if=/dev/zero of=build-f14-repository bs=1G count=65

Still in Dom0, attach the block device to the DomU:

# xm block-attach f14 file:$PWD/build-f14-repository xvdb w

Then switch to the DomU and format the device:

# mkfs.ext4 /dev/xvdb

Finally, mount the device in the Fedora 14 DomU:

# mount /dev/xvdb /mnt
# df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                1.9G  604M  1.3G  32% /
udev                  362M  116K  362M   1% /dev
tmpfs                 374M     0  374M   0% /dev/shm
/dev/xvda1            1.9G  604M  1.3G  32% /
udev                  362M  116K  362M   1% /dev
/dev/xvdb              64G  180M   61G   1% /mnt

We've added 61GB of usable space on /mnt.

I have at least 100 accounts in my personal and professional life. It's gotten to the point where I just can't remember them all and their associated passwords and other account information. And using a single password everywhere is just a horrible idea. Just one breach at any site stupid enough to store my password in clear text and all of my accounts would be compromised.

So I've started using KeePass to store my passwords. There are two versions of KeePass: I use the 1.x version so I can share my password database among all of my computers: Windows, Linux, and Macs. KeePass stores everything in a database encrypted with a single password. For that password I use a very long, complicated string that includes all of the usual "good things": upper and lower case, numbers, and symbols. But hey, it's the only password I need to remember anymore, so I don't mind!

All of my account passwords are unique and randomly generated, since I never type them anymore. There are some tricks to doing this successfully. Some web sites make random passwords particularly difficult to work with: they disallow long passwords, they don't like some symbols, they require all passwords be a specific length (a truly horrible idea), etc. Some even let you enter long passwords, but then don't let you log in with them. I guess they truncate them at some point, but never tell you about it. But eventually you can figure it out for even the most user-hostile sites.

I also store other account information. I keep vendor phone numbers, account numbers, and answers to security questions. I've decided that I'm never going to give a truthful answer to a security question ever again. My mother's maiden name? For Visa, maybe it's "ptdTpX?mdSY9C". My first pet? For my bank it might be "SMtw*3X8L". They don't care, they just want me to use something that is easy for me to remember. With KeePass, all of these are easy to remember! I believe that eventually every one of my vendors will experience a breach that exposes my private information. The less they know about me, the better.

Of course it's critical that I have good backups of my KeePass database. I use a variety of backup schemes to ensure that I always have at least 5 or 6 copies of the database on geographically diverse machines. More on that another time.

Python 2.6 and greater support str.format(), also known as PEP-3101 Advanced String Formatting. A friend asked me how, given a string to be formatted, he could tell if a variable was refernced by that string and would be expanded.

As he pointed out, this is easy enough if you're using Python 3.2 and have access to str.format_map(). You'd just write a special dict that would remember what keys have been accessed. But before Python 3.2, you're stuck with no good answer.

Given that I wrote the implementation of str.format(), I thought I'd take a crack at this.

My solution was to use string.Formatter. This is a little-known (and even less used) class that exposes the internals of str.format(), but all wrapped up in a Python class with a number of overridable methods. Here was my first attempt:

import string

class Formatter(string.Formatter):
    def __init__(self):
        self.used = set()
    def get_value(self, key, args, kwargs):
        self.used.add(key)
        return ''

def is_used(var, format_string):
    formatter = Formatter()
    formatter.format(format_string)
    return var in formatter.used

Given that, here's how you use it:

>>> is_used('foo', '{foo}')
True
>>> is_used('foo1', '{foo}')
False
>>> is_used('foo', '{{foo}}')
False

However, there's a problem. If you have a format specifier that doesn't make sense for a string, you'll get an exception:

>>> is_used('date', '{date:%Y-%m-%d}')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in is_used
  File "C:\Python26\lib\string.py", line 543, in format
    return self.vformat(format_string, args, kwargs)
  File "C:\Python26\lib\string.py", line 547, in vformat
    result = self._vformat(format_string, args, kwargs, used_args, 2)
  File "C:\Python26\lib\string.py", line 580, in _vformat
    result.append(self.format_field(obj, format_spec))
  File "C:\Python26\lib\string.py", line 597, in format_field
    return format(value, format_spec)
ValueError: Invalid conversion specification

My solution was to add another class that ignores any format specifier:

import string

class AnyFormatSpec:
    def __format__(self, fmt):
        return ''

class Formatter(string.Formatter):
    def __init__(self):
        self.used = set()
    def get_value(self, key, args, kwargs):
        self.used.add(key)
        return AnyFormatSpec()

def is_used(var, format_string):
    formatter = Formatter()
    formatter.format(format_string)
    return var in formatter.used

Now we can check for any variable, with any format specifier:

>>> is_used('date', '{date:%Y-%m-%d}')
True
>>> is_used('pi', '{pi:.12f}')
True
>>> is_used('pi', '{pie}')
False

Arduino is an open source hardware and software platform, built around the Atmel AVR family of microcontrollers. These are very tiny computers by today's standards. They're only 8 bits wide, instead of the 64 bits of most new Intel servers and workstations. They only run at about 16 kilohertz instead of 1+ gigahertz. They only have about 32 kilobytes of permanent storage, instead of the multi-gigabyte disk drives on the machine you're probably reading this on. And finally, they contain only 2 kilobytes of RAM, approximate 1 millionth of the RAM standard in any laptop sold today.

But they're cheap, tiny, run well on a few AA batteries, and are well supported by the "maker" community. This is a large group of people who build their own projects, often hardware and software. Make magazine is devoted to this community.

The event I attended was at HacDC, which is a great space to get help with any sort of project you might be interested in. They have a couple of very impressive homemade 3-D printers, even!

What am I going to do with my Arduino? I'm not so sure yet. I bought enough sensors to build a simple alarm system, and I'm interesting in controlling my Arduino with a standard infra-red remote control. I might also use it to build a wireless mesh network. This project is mostly an excuse to solder some circuits and get by to my electrical engineering roots. I'll create small blog entries here if I do anything interesting.

I bought my Arduino, sensors, motors and a few tools from Adafruit Industries. There are literally hundreds of sites that sell and support Arduino and AVR. I like Adafruit because the woman that runs it has many great tutorials and helpful blog posts.

Today Verizon reconfigured residential FiOS in the Washington DC area to block TCP port 25 to non-Verizon servers. They claim this is for network security reasons, and I understand that this change will ultimately be made to the nationwide FiOS network.

SMTP interception: Not a new issue

For a long time we've recommended to our clients that they use port 1025 for sending email to our servers. Our servers listen to this port in addition to port 25. The main reason we recommend this is that many ISPs, in particular hotels, redirect port 25 to their own email servers. We definitely do not want our email to traverse these servers!

For those clients who have already switched to port 1025, today's action by Verizon was a non-event. For other clients (and indeed my own computers at home!) outbound email was broken until the clients were reconfigured.

Problems with Apple

Switching to port 1025 worked well for our clients using Thunderbird, but Apple Mail presented a unique problem. Old Apple Mail clients (2.1.4) worked, but for versions 3.1 and 4.2 the port setting would not survive a reboot. Fortunately Apple Mail tries multiple ports: 25, 465, and 587. So by adding port 465 to the list of ports we were already listening to, these Apple Mail clients started working without any reconfiguration.

We've recently started investigating Amazon EC2 for some of our computing needs. So far our progress has been excellent. I've been focusing on using the Amazon-supplied Fedora 8 (F8) images, in particular ami-48aa4921, although the problem I describe here applies to all of the F8 AMIs that Amazon provides.

One significant roadblock has been a problem with rsync. In particular, we use the excellent dirvish for our online backups. Unfortunately, Amazon uses the 2.6.21 kernel in its F8 images and this version does not support the lutimes system call. The version of rsync that comes with F8 uses lutimes to set the modification time on directories. lutimes isn't available until the 2.6.22 kernels. For more information on the issue with rsync and lutimes, see the rsync bug entry.

The symptom is errors in the dirvish rsync_error log files of the form:

rsync: failed to set times on "<directory-name>": Function not implemented (38)

Dirvish sees these as fatal errors and marks the images as failed. This prevents dirvish from performing its incremental backups.

Because rsync does not have a runtime switch to ignore lutimes, the easiest way to solve this is to produce a version of rsync that doesn't use the call at all. Unfortunately rsync does not have an autoconf switch to turn off lutimes, so I had to patch configure.in and rebuild. The change is simple, here's the diff I use:

--- rsync-2.6.9/configure.in.orig        2010-01-29 15:37:35.000000000 -0500
+++ rsync-2.6.9/configure.in    2010-01-29 15:38:07.000000000 -0500
@@ -528,7 +528,7 @@
 AC_FUNC_UTIME_NULL
 AC_FUNC_ALLOCA
 AC_CHECK_FUNCS(waitpid wait4 getcwd strdup chown chmod lchmod mknod mkfifo \
-    fchmod fstat ftruncate strchr readlink link utime utimes lutimes strftime \
+    fchmod fstat ftruncate strchr readlink link utime utimes strftime \
     memmove lchown vsnprintf snprintf vasprintf asprintf setsid glob strpbrk \
     strlcat strlcpy strtol mallinfo getgroups setgroups geteuid getegid \
     setlocale setmode open64 lseek64 mkstemp64 mtrace va_copy __va_copy \

I created a new RPM for rsync. Here's the diff for the .spec file:

--- rsync.spec.orig     2008-04-09 10:36:56.000000000 -0400
+++ rsync.spec  2010-01-29 22:21:13.000000000 -0500
@@ -1,7 +1,7 @@
 Summary: A program for synchronizing files over a network.
 Name: rsync
 Version: 2.6.9
-Release: 5%{?dist}
+Release: 5%{?dist}.trueblade.0
 Group: Applications/Internet
 # TAG: for pre versions use
 #Source:       ftp://rsync.samba.org/pub/rsync/rsync-%{version}pre1.tar.gz
@@ -10,6 +10,7 @@
 Patch1: rsync-2.6.9-acl-xattr-delete-bug.patch
 Patch2: rsync-2.6.9-hlink-segv.patch
 Patch3: rsync-3.0.1-xattr-alloc.diff
+Patch4: rsync-2.6.9-disable-lutimes.patch
 BuildRequires: libacl-devel, libattr-devel, autoconf, make, gcc, popt-devel
 Prefix: %{_prefix}
 BuildRoot: /var/tmp/%{name}-root
@@ -33,6 +34,7 @@
 %patch1 -p1 -b .acl_xattrs_bug
 %patch2 -p1 -b .hlink_segv
 %patch3 -p1 -b .xattr-alloc
+%patch4 -p1 -b .lutimes
 
 %build
 rm -fr autom4te.cache
@@ -62,6 +64,10 @@
 %{_mandir}/man5/rsyncd.conf.5*
 
 %changelog
+* Fri Jan 29 2010 Eric V. Smith <eric@trueblade.com> 2.6.9-5.fc8.trueblade.0
+- Added patch4 to remove lutimes, since the EC2 kernel in ami-48aa4921 does
+  not support it.
+
 * Tue Apr  8 2008 Simo Sorce <ssorce@redhat.com> 2.6.9-5.fc8
 - Security release: http://rsync.samba.org/security.html#s3_0_2 

Once I had the new RPM, I signed it and added it to our local RPM repository. Because it has a newer version than the one supplied with F8, it will automatically be picked up by "yum update".

The only remaining complication is that the AMI I'm using supplies its own copy of rsync in /usr/local/bin, in addition to the one supplied by default in /usr/bin. I'm not sure why Amazon did this, because the /usr/local/bin version has the same problem as the /usr/bin one. The /usr/local/bin version does not come from an RPM, so I just delete it using our automated server configuration tool. The files to delete are:

/usr/local/bin/rsync
/usr/local/man/man1/rsync.1
/usr/local/man/man5/rsyncd.conf.5
/usr/local/share/man/man1/rsync.1
/usr/local/share/man/man5/rsyncd.conf.5

Once the new RPM updates rsync and the unneeded /usr/local files are deleted, rsync and dirvish are again working correctly.

I'd rather solve this problem by upgrading the kernel to 2.6.22 or newer, but upgrading the kernel is a non-trivial task with AWS. I'd rather let Amazon handle that issue and instead focus on using the provided images. This way we can more easily upgrade when Amazon produces newer AMIs.

For a number of other approaches (but with fewer specifics), see this thread.

I've been using Verizon FiOS for Internet service for years. In general I've been pleased with the service: it's very fast and very reliable.

I've been running FiOS without using the Verizon supplied router. Instead, I plugged the incoming CAT5 cable directly into my Linux server. This did require that I use a slightly complicated configuration supporting PPPoE, but once I finally got it set up it has been problem-free. My Linux server has also been my DNS server and my DHCP server. I'm sure Verizon doesn't like this configuration, as it gives them less control over and less visibility into my network. But that's fine with me.

However, I recently switched to using FiOS TV. Because of the way the TV set top boxes (STB's) need to communicate upstream for program information, I'm forced to use the ActionTec router that Verizon supplies. As long as I'm forced to use this router I decided to use a more normal, less techy configuration and just let the ActionTec be my DNS and DHCP server. This has generally worked without issue, at least for the first few days.

Last night I decided to change the default domain name that the router uses. It defaults to "home", but it's better for me if it uses "trueblade.com", that way I can more easily resolve domain names. In any network I've ever worked on, this would not be a problem. The only thing it should affect is that when a client asks for a name like "mail", it would first query for "mail.trueblade.com".

However this morning my home network wasn't able to connect to my mail servers. After a lot of poking around I discovered that my internal systems were not able to resolve fully qualified DNS names like mail.trueblade.com. After a lot more poking around, I discovered that the ActionTec would not resolve domain names ending in trueblade.com if its default domain were also trueblade.com.

So the solution was simply to change the default domain name on the ActionTec back to "home", or indeed any other string. That's frustrating, because it means that I can't type domain names like "mail", but I need to use the fully qualified "mail.trueblade.com". But it's only a minor frustration. The only time I don't use fully qualified names is when I'm debugging. All of my systems use fully qualified names for their configuration files.

I'll probably switch away from using the ActionTec as my DNS and DHCP servers. In addition to this problem, you're limited in the amount of configuration you have over the DHCP server in particular. I'll post more when I've made the decision to switch off of the ActionTec for DNS and DHCP.

In order to reliably build our servers, we use kickstart and PXE to give us a simple, repeatable process. During this build process, we need user input to decide exactly which configuration to apply to the system being built.

In order to do that, we run a Python script in the ks.cfg %pre section that uses the snack library to get information from the user. Snack is the (poorly documented) UI toolkit that comes with kickstart/anaconda. Here's how you use it in %pre.

First, you need a %pre section that runs the python interpreter. To do that, start the section with:

%pre --interpreter /usr/bin/python

Next, you need to realize that the kickstart screen you usually see runs on tty3. But the snack UI will show up on tty1. So we use a little routine to switch tty's:

def set_tty(n):
    f = open('/dev/tty%d' % n, 'a')
    os.dup2(f.fileno(), sys.stdin.fileno())
    os.dup2(f.fileno(), sys.stdout.fileno())
    os.dup2(f.fileno(), sys.stderr.fileno())

Next comes the function that actually calls snack to get the user input. Don't worry about the host_config parameter, instead focus on how snack is used. This code build a dialog box with a listbox and an OK button:

def get_user_input(host_config, default=None):
    # get the hostname, from it the other params are computed
    # return the hostname and everything that's derived from it
    # which for now is just the disk layout scheme

    from snack import SnackScreen, Listbox, Grid, Label, Entry, Button, GridForm

    def host_list():
        def hosts():
            return sorted(host_config.keys())

        lb = Listbox(height=len(host_config), returnExit=True)
        for host in hosts():
            lb.append(host, host)
        if default in host_config.keys():
            lb.setCurrent(default)
        return lb

    screen = SnackScreen()
    form = GridForm(screen, 'Select host to configure', 1, 2)
    form.add(host_list(), 0, 0, (0, 0, 0, 1))
    form.add(Button('OK'), 0, 1)

    # now run the form
    result = form.runOnce()
    screen.finish()

    hostname = form.childList[0].current()
    return {'layout': host_config[hostname],
            'hostname': hostname,
            }

Finally, put it all together:

set_tty(1)  # change to tty1, we're called by kickstart with stdout as tty3
user_args = get_user_input(host_config, args.args.get('tb-host'))
set_tty(3)  # restore

If you ever need to find information on snack, it helps to know that it's a wrapper for newt. So you can Google on "snack newt python" to get some useful answers. But be warned that there's not much there and you might have to look through the source code to snack.

Yesterday I dropped my daughter off in Washington, DC at Georgetown Cupcake. If you've ever been to Georgetown, you'll appreciate that I had to park about 10 blocks away. When I was walking back to join her, I realized that I didn't remember all of the turns I'd taken in order to find a parking space. Plus, all the quaint houses look the same, so I was having difficulty retracing my path.

So I used my new G1 (the "Google phone") to find my location and a local map. I did have to turn on the GPS, since I usually leave it off to extend the battery life, but other than that it was simple to find my location and a map, all without breaking stride (which was important because it was starting to rain!).

I was also able to use the phone's voice search to find the store's address. This even worked with the background noise of a jet taking off from the nearby National Airport.

The advanced Plone class was taught by Joel Burton from Plone Bootcamps. I've attended Joel's basic Plone class a few years ago, so when I heard he was teaching his advanced class in Chapel Hill, I decided to attend. I'm very glad I did.

Here's what I learned:

• Plone is designed by some very smart people. Once you understand a handful of concepts, it's simple, elegant, and extensible. It's true that some of the concepts take a while to understand, but I think it's worth it. And Joel helped tremendously.
  
• Plone is improving with each release. Over the years I've been using Plone, it has added locking, easier workflow configuration, a much-improved setup facility, and an install process based on the awesome zc.buildout. If you're not using zc.buildout for your Plone deployments, you should be. And you might want to think of using it for other installations, too.
  
• Joel is an excellent educator. He not only knows Plone inside and out, but he also understands how to teach.
• Zope's acquisition is much maligned, but it's incredibly useful. We had an example in class where we wanted to add a portlet only when we were under certain folders, and to make the content folder specific. I couldn't figure out how to do it, but a 2 line Python method and acquisition solved the problem. Incredible!
  
• Chapel Hill, as you'd expect, has some awesome barbeque places.