Hugh Perkins's tech blog

DFS-R makes it really easy to replicate application file servers.

In my tests, it even automatically breaks open file locks on the files for you, transparently, and silently.

And it’s very fast, bandwidth optimized, and only replicates changes. Transparently and silently, and with only a trivial amount of configuration.

Ok, let’s back up a bit. What do I mean by application file servers? Well, those server that are file servers that contain executable files for various applications in a read-only form. These types of servers make it trivial to update applications across the enterprise, since you don’t need to install anything locally on the pcs. Worst case, for some applications, you can do a quick robocopy at application start to copy/cache stuff locally in a user-writable directory on the computer.

Application file servers work great, except for file locks: which make it non-trivial to update any locked files in place, without turning off the Server service, or using some third-party/proprietary code to unlock the files during the copy.

DFS-R makes this easy, since it seems to handle overwriting locked files automatically and silently and easily.

Server Core is actually fairly cool: you can use ‘sconfig’ to configure it for networking and remote server manager configuration, and then configure it from another machine with server manager installed!

sconfig is a commandline utility, with a menu system, a little like using fdisk or similar.

sconfig and remote administration is available in the R2 version of Windows Server 2008 Server Core, which includes the .Net Framework.

Failover clusters can run on server core servers. Unfortunately sql server is not (yet!) supported on server core installations, somewhat negating the utility.

Hyper-V servers apparently start in server core by default, though it’s a little difficult to test this in an internet cafe since I don’t think they would take very kindly to my booting their computers to my own os

Interesting video showing hyper-v and sconfig in action:

http://edge.technet.com/Media/Hyper-V-Server-2008-R2-Bare-Metal-to-Live-Migration-In-about-an-hour/

New version of usb commandline mounter for debian/ubuntu:
- bug fixes
- works with fstab if an entry already exists
- mount options are now per-fstype

Instructions:
- download from usb commandline automounter
- put usbmount.sh into /opt/bin
- put usbmount.rules into /etc/udev/rules.d

If someone would like to package this up into a debian package complete with man page and so on?

I spent my morning setting up Windows 2008 Failover Clustering in an internet cafe, and it worked!

My eeepc has only a 32-bit processor, but the internet cafe computers, whilst running Windows XP, have 64-bit Athlon processors, and 2gigabytes of ram.

They work quite nicely.

There are guides all over the internet on how to install Windows 2008 Server (dead easy in fact), and the clustering (pretty self-explanatory too).

Some points that were not obvious to me from the available guides:
- whilst the minimum spec for Windows 2008 Enterprise R2 is 40gigabytes ( :-O ) for the hard-drive, in fact the vmware disk-images created only take up about 8gigabytes per vm, even less for standard edition.
– if the vm harddrive files are stored on FAT, then VMWare Player is preferable over VirtualBox, since VirtualBox doesn’t seem to handle spanning over multiple disk files
- freenas really does work really well as an iscsi target for Windows 2008 Failover Clusters (see link in previous post).
– it also only takes up about 50meg of hard-drive space :-O
- I installed the following virtual machines:
– 1 freenas, to be used as an iscsi target
– 1 x Windows 2008 Standard, Server Core, configured to be a dc, and a dns
– 2 x Windows 2008 Server Enterprise R2, Full Installation (free download from Microsoft, no license key required, at least for testing, lab use)
- on a 2 gigabyte memory machine, installations with 512megabytes of ram assigned to each vm ran fluidly
- to run all machines at the same time, after installation, I assigned:
– 200meg to freenas (it crashes on shutdown, but who cares?)
– 256meg to the domain controller (really slow to start up, but didn’t slow down the host, and left more memory to the more important machines)
– 400meg for each cluster node (a little tight, but better than causing excessive paging of the host I felt)
- iSCSI targets can’t contain underscores

I played around with creating a clustered file server, creating a share, and doing things like disconnecting the network for a cluster node, or suspending the node, and seeing what happened.

What happened was:
- failover seemed pretty solid and reliable, very self-healing I felt
- failover seemed for me to take more than seconds but less than 10 seconds
- on client machines, as soon as the failover was complete, files were immediately available, no issues with caching of the underlying node address or anything like that
- on client machines, doing ‘dir’ from a commandprompt whose currentdirectory was a drive mapped to the cluster share caused a hang for 30 seconds or so whilst it figured out the drive wasnt available
– and then an error message, rather than the now-available drive

Other things…

Windows Server 2008 seems really easy to use. Everything seems to be doable from a single console called ‘Server Manager’, by and large. There is a concept of ‘Roles’ and ‘Features’. A ‘Role’ is something like being a file server ,and a feature is somethign like failover-clustering. I guess a role is somethign that is published for client machines to use, and a feature is something that enhances the provision of said roles.

The server manager console is structured by roles and features, and installation of roles and features is from the same console, and trivially easy.

When you install Windows Server 2008, you have the choice between ‘Server Core’ and ‘Full Installation’. Server Core still provides a gui, but containing a single command-prompt, and no possibility of doing ‘explorer’ or ‘mmc’ or anything like that, although taskmgr works ok. Server Core cannot be upgraded to use the full normal gui, and a full installation cannot be downgraded to server core.

I kindof think that full installation probably makes more sense generally, but time might prove me wrong…

I used a Server Core installation for the domain controller to save disk space, and I used Full Installation for the clustering, because I felt that is probably the more normal way of managing clustering. The last thing one needs in an outage is to be looking up command-line arguments!

There does I feel seem to be increasing provision for scripting admin functions from the commandline. For example, on the server core dc, I could use ‘dnsadm’ to add an A record for the freenas box.

I quite enjoyed by Windows Server 2008 experience. Also, I quite liked that it is downloadable for free and needs no license key, at least for transient testing.

Links on how to setup Windows Server 2008 Failover cluster in virtual machines. These are mostly here so I can access them easily whilst I play with this

• Setting up Windows 2008 cluster on vmware
• FreeNAS as iscsi target with SCSI-3 persistent reservations

What’s with the SCSI-3 persistent reservations? Well, after doing a little reading, when one node in a cluster thinks the other has died, it needs to take over the shared resources, specifically the hard-drive, before starting.

However, before taking over the other machine’s resources, it needs to make sure the other node is really dead, and not just temporarily hanging or something.

One approach is to Shoot The Other Node In The Head, “STONITH”, ie pull the power plug on the other. Whichever node ‘wins’, pulls the power plug on the other first, becomes the active node. The other one is … dead.

Another approach is rather less … fun …. or at least doesn’t have such an interesting acronym , but eminently more practical: reserve/release: a node attempts to reserve the resource. If it fails, it assumes the other node already reserved it, and it commits suicide.

Windows Server 2003 used this type of I/O fencing, as these technos are referred to. However, there are so many layers involved in the scsi stack, and it involved using a scsi bus reset, that sometimes commands were converted to other commands and then entirely ignored by the underlying storage system, and failover never happened.

With Windows Server 2008, the I/O fencing used is Persistent Reservations, implemented using SCSI-3 Persistent Reservations. In this form of I/O fencing, each node holds a key to reserve the storage volume. Each node’s key is different. Only one key is activated on the storage volume at a time, so only one node can access it. During failover, the key is changed on the storage volume … somehow … .and then only the corresponding node can access it.

As well as being fundamentally more resilient to race-conditions, the SCSI-3 persistent reservations commands cut through all the various layers in the scsi stack and should be much more reliable.

Unfortunately on my poor Debian system, neither of the two iscsi providers, ‘targets’ in the scsi techno, support SCSI-3 PR semantics as far as I can tell.

FreeNas does so I’m looking through the links above to install freenas in a vm on my system, using virtualbox.

Oh, last night I got iSCSI running on my debian, without the SCSI-3 persistent reservations semantics, but with a target, and an initiator, and it worked fine, which was fun. An ‘initiator’ is the scsi terminology for the client. The initiator was ‘open-iscsi’, and the target was ‘iscsitarget’.

After perusing the available heresay – one can’t really I feel call it much more than that, although this article, by a semi-anonymous Stephen looks pretty convincing, and rather scary, but I don’t have an OCZ so it doesn’t directly apply. Theodore Ts’o‘s name carries some weight in my mind, but he doesn’t seem to provide any hard benchmarking figures to back up his assertions, as far as I can see. Well, anyway, after reading through those articles, I decided block align my lvm extents on 4MB boundaries, on the off-chance this will reduce writes to my, it seems quite fragile, MLC drive.

It seems there are two drives on this 901 eeepc: a 4GB SLC drive as primary, and a 16GB MLC drive. They are apparently not the same. MLC drives (‘multi-layered cell’?) have 10% of the expected write cycles of the SLC drives!.

What I did, just on the off-chance, and because it is fun:
- backed up all my lvs using tar, to an external usb drive
- fdisk -H 256 -S 32 /dev/sdb
- created the following partitions:

Disk /dev/sdb: 16.1 GB, 16139354112 bytes
256 heads, 32 sectors/track, 3847 cylinders
Units = cylinders of 8192 * 512 = 4194304 bytes
Disk identifier: 0xf26d47c4

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               2         960     3928064   8e  Linux LVM
/dev/sdb2             961        1922     3940352   8e  Linux LVM
/dev/sdb3            1923        2885     3944448   8e  Linux LVM
/dev/sdb4            2886        3847     3940352   8e  Linux LVM

- these are all 8192 sector aligned, ie aligned to 4MB blocks
- I created multiple partitions to give me more flexibility in the future

Next up, creating the lvs:
- pvcreate –metadatasize 4090kb /dev/sda
- ditto for other partitions

Creating the lvs is as per normal.

To create the file systems, I used ext4, because I do see a noticeable performance increase of ext4 compared to ext2 and ext3, though it’s subjective, anecdotal, I don’t have any hard objective benchmarks.

I used the following command, assuming an erase block size of up to 512kb:

mkfs.ext4 -E stripe-width=128,resize=16G -L data /dev/vg0/data
# ... ditto for other volumes

… then I restored from tar. The operating systems (debian and centos) booted up just fine with no modifications (I’d already updated the initfs with ext4 earlier, prior to starting the migration to 4MB extent alignment).

I do see a performance improvement, but this might be just from using ext4, which does I feel make a *significant* difference to performance, compared to both ext2 and ext3.

I took LPIC exam 101 this morning, LPIC Exam 101 objectives, with 90.25%, which I feel is not too bad.

Preparation:
- used Ubuntu as my sole home OS for one year
- read ‘LPIC-1 Linux Professional Institute Certification Study Guide, Second Edition’, by Roderick W. Smith for one week
- installed Centos 5.4 and Debian Testing (Squeeze) and played with them a little

The book by Roderick Smith is I feel excellent. It’s all you really need to pass I feel, though obviously playing around on a real system for a while makes the whole thing fairly painless.

The exam was not an adaptive exam :-O 80 questions, for two hours. Insane… I used to quite like the adaptive MCSE exams where you only answer a few questions, and then you’re done. Less easy for questions to leak out, and much fewer questions to answer.

Or maybe I’m just lazy

This is just a list of links I’m looking into really for now, just in case I accidentally toggle the wifi and everything crashes

Very interesting benchmarking about random writes on SSDs
Theodore Ts’o on ssds and erase block size
eee user thread about erase block size

I just found out about getfacl and setfacl. It looks like something I was just thinking was missing on linux.

On Debian testing, simply ‘sudo apt-get install acl’. No need to reboot. It seems the appropriate module is already in the kernel.

setfacl lets you add additional acls – user and group permissions – to a file. How does this integrate with the ‘standard’ linux permissions? Well, a ‘+’ sign appears to the right of the permissions:

$ ls -lh blah.txt
-rw-r--r--+ 1 testuser1 user 6 2009-12-19 15:52 blah.txt

Using getfacl, we can see the additional acls:

$ getfacl blah.txt
# file: blah.txt
# owner: testuser1
# group: user
user::rw-
user:postgres:r--
user:testuser1:r--
group::r--
mask::r--
other::r--

One can use ‘setfacl -m u:targetuser:rw filename.txt’ to grant access to someone.

How does this stack up to the real world? How about: does dreamhost use them? No:

$ getfacl
-bash: getfacl: command not found

…. however it seems it’s mainstream enough that they will have it when they upgrade to the next version of debian:

ACLS on Dreamhost:

“ACL permissions will not work on our servers until we update to the next release of Debian some time later this year.”

Admittedly that is from 2005 ,and no change yet but still…

eee-control is great, but it takes up 11MB of precious ram, so, for my own laptop (but not for my girlfriend’s), I’ve written a tiny script to control it:

To see the current state:

eeepc-state.sh

Example result:

wifi: 1
camera: 0
bluetooth: 0
cardreader: 1

To set a state, do for example:

eeepc-state.sh camera 1

It’s a tiny tiny script:

#!/bin/bash

function checkroot() {
   if [[ $(whoami) != root ]]; then {
      sudo $0 $*
      exit 0
   } fi
}

if [[ $1 == wifi ]]; then {
   checkroot $*
   echo $2 >/sys/class/rfkill/rfkill0/state
} fi
if [[ $1 == bluetooth ]]; then {
   checkroot $*
   echo $2 >/sys/class/rfkill/rfkill1/state
} fi
if [[ $1 == camera ]]; then {
   checkroot $*
   echo $2 >/sys/devices/platform/eeepc/camera
} fi
if [[ $1 == cardreader ]]; then {
   checkroot $*
   echo $2 >/sys/devices/platform/eeepc/cardr
} fi

echo "wifi: $(cat /sys/class/rfkill/rfkill0/state)"
echo "camera: $(cat /sys/devices/platform/eeepc/camera)"
echo "bluetooth: $(cat /sys/class/rfkill/rfkill1/state)"
echo "cardreader: $(cat /sys/devices/platform/eeepc/cardr)"