In trying to find out which shared libraries were now broken, I wrote a very crude script to find out which binaries broke.

In short, it finds executable files that ldd reports as having missing runtime dependencies and attempts to find those in the package directory (so you have a clue what packages it came from).

I renamed libfam to something else and this was the output:

First field is the file that is broken. Second field are packages that have files with that name. Last field are the missing libraries.

If no package uses a broken binary then that field in the output stays blank, wit:

I’m posting this in case someone else finds it useful…

Obligatory plug: It’s only 99¢ (or other local denomination)! Go buy it!

ABC of the Sea is a children-focussed animated book featuring 26 pages of hand-drawn marine life. The illustrations were originally produced by my wife many years ago but never published. Our own child can now read and a print of the book surfaced which she thoroughly enjoyed. We then had the idea to turn it into an iPad application. Each page has been carefully, but subtly animated in a fashion largely consistent with how the creature (or plant) behaves in its natural habitat, though some artistic license has been taken if the real-life movements would be virtually undetectable.

Visually it’s lovely and the simplicity of the images and the text are deliberate so that young-readers will enjoy it, but also so that even children who do not read yet will benefit. Planned updates include more information for each page describing the creature and its habits and narrated names and text.

ABC of the Sea: O is for Octopus

Behind the scenes

However, the real magic is under the hood. For me this was an ideal opportunity to dive into the innards of iOS, its graphics capabilities and more besides. There were a handful of key development areas that made this book a possibility.

Firstly, though Apple has an app, iBooks, with a visually remarkable page turning transition, that effect is not available to other developers. Some of the functionality exists in iOS, but it is undocumented and thus would not pass muster with the app store moderators. As I documented in October last year there are a number of 3rd party attempts to fake this effect but with more than a moments glance the approximation becomes obvious. Based on some OpenGL work by W. Dana Nuon, Implementing iBooks page curling using a conical deformation algorithm, I came up with an implementation of my own and some of that work is present within ABC of the Sea. After some more work I will probably be licensing this mechanism, if there’s interest in it. In particular it provides a framework for a developer to provide his own deformation and position-mapping algorithm should mine not be quite what is desired!

D is for Dolphin

I then had the task of animating the critters. There are many frameworks out there, both opensource and paid-for, but mostly these are geared for gaming with events engines and whatnot. Many of them were in varying degrees of completeness and they all had their own learning curves. Ultimately I decided to roll my own since I knew my needs were very simple. It makes extensive use of the animation built into theUIView and CALayer classes and provides a simple mechanism to define, entirely in property lists, the sprites and their component parts and relative positioning, their locations on the page, the name of a path to apply and the operations that comprise a path. The operations needed were simple: timing, move and rotate. Movement can be direct or use a Bezier path. Both can incorporate randomness. The results are well exercised throughout ABC of the Sea. I will probably also license this code in due course.

The simplest sprites were animated in well under 30 minutes – which includes whatever sprite dissection necessary to turn an image into its component parts. The most complicated pages took a few hours – and most of that is spent working in a graphics application working out how best to decompose a creature.

ABC of the Sea cover page

If you have small children, or simply want to see how my page transition code works, I encourage you to purchase the app!

I was developing a mechanism to verify some data that was generated outside the device. Using a simple key pair generated by OpenSSL at a command line it was very simple to create scripts in Perl and PHP to produce (and sign) and then decode (and validate) some data using this key pair. The functions to add a public or a private key to the keychain are there in iOS but they don’t work as expected.

It’s all in the format

There are many ways to transmit a key for use by another system but there are also some defacto standards. For an RSA public or private key, which has a pretty straightforward internal structure, that would be defined by PKCS#1.

There are two common forms to transmitting and using an RSA public or private key. ASN.1 DER – which is a non-displayable (i.e., not ASCII) format, or the same thing but base64 encoded and referred to as PEM – which is 7-bit ASCII and thus displayable. The base64 format is usually wrapped with something like -----BEGIN PUBLIC KEY----- (which is what OpenSSL produces) or -----BEGIN RSA PUBLIC KEY-----.

You can have OpenSSL generate a key pair very easily and you select the output form right on the command line:

Sign here

So far so good. In PHP we can use the PEM encoded keys directly. For example, to acquire the signature of some data, one could do this:

The format being returned is not a standard, it’s something I made up, but it’s easily transmitted and base64 implementations are pretty universal.

How would we verify the integrity of this message? Here’s a crude example:

With this little nugget of code it becomes trivial to find out if someone has been tampering with the data. This is useful, for example, for software licenses. Assuming the private key is kept secure you have a degree of trust that anything signed by it is also secure.

So what’s the problem with iOS?

iOS provides a security framework but if you try to use it for keys you generated elsewhere then it doesn’t work and gives only very limited indications why it doesn’t work.

iOS does not provide documented direct access to the OpenSSL API. If it’s there it wouldn’t matter – being undocumented means that it would face App Store rejection if their static analysis showed use of the undocumented library.

You could statically link an OpenSSL library that you compiled and it would work but in that situation it’s your app providing the cryptography code and thus you will need to jump some hurdles (you may need a CCATS) to get export clearance for your app (although it’s not that simple in general – read the Export Compliance FAQ in iTunes Connect). In my case, I’m using a signed software license to prevent piracy – digital rights management – and thus does not need US Government clearance – but only if I include only enough code to perform that function. If I link against the whole of OpenSSL then what my app is doing will be less clearly defined.

So the key to a simple life is to work out why the provided functions don’t work.

What’s in a key?

It comes down to format.

The first clue is that if you use iOS to generate a key pair then the resulting form is not ASCII. So that rules out PEM and means that if you have a PEM formatted key, you’ll also need a base64 decoder – which iOS does not provide. There’s plenty of them around. I used NSData+Base64.m which if you Google for you will find many varied implementations and derivations of.

But that is not enough, even though an Apple employee confirms on the dev forums that iOS wants a PKCS#1 key and if you’re doing signature verification, that the signature is of the SHA1 hash of the data, as I would expect.

The next clue came when I noticed the keys produced by iOS don’t have the normal binary ASN.1 preamble to identify itself as a PKCS#1 key. This prompted some searching and I came across this post talking exactly about this issue. I have no idea why it did not come up in my earlier searching. In it Berin explains his similar discovery, but also that the payload still looks like a binary PKCS#1 key, just without the preamble. In his application he had to add the header so he could import the key elsewhere. What I wanted was the reverse.

And so I did the reverse. And it worked.

And then I came across this later post by Berin also doing what I wanted. Doh. Our implementations are not identical, but functionally similar.

Code or it didn’t happen

Assuming you have your OpenSSL generated RSA public key in an NSData object, this method will verify that it is in fact a PKCS#1 key and strip the header:

and the result can be fed into the iOS functions in the expected manner. For example, this code adds a keychain reference to a public key to an array. If you use this, don’t forget to eventually CFRelease() each SecKeyRef you acquire, and don’t forget that tag needs to be unique for every key you add.

Lastly, to verify a message in the same way we did in PHP earlier, the code might look like this:

So, yes. It’s involved. But I think it’s worth it.

Also, for what it is worth, this does work in the Simulator, certainly at least with the current iOS SDK (4.2).

So I wrote my own WordPress plugin, which is demonstrated here.

This example demonstrates image maps and the anchor that encompasses the entire image (this last the WordPress Shadowbox JS plugin picks up). Note how the image maps work hierarchically and provide tool-tips too.

Generated files are kept around instead of being regenerated using a name based on the MD5 hash of the DOT code. If you are logged in and an Admin user then the output is always regenerated, just in case (since the hash doesn’t incorporate any attributes the shortcode tag, for instance).

I plan to release this plugin next week.

Related Reading

I noted in RAID gone Green that my read speed was lower than write speeds and provided some commentary on why that might be. I have a further explanation to add to this – the stripe size of the RAID array.

When I was initially setting up the volumes on the array I was following advice I found online (on the basis I was experimenting and could change it later without harm) to set the stripe size to be the same or close to the block size used in VMFS. Since I was targeting 2TB VMFS volumes and that needed an 8MB block, I opted for the maximum that the Dell PERC supported – 1MB. In hindsight, this may not be the optimum setting.

The default read-ahead size for Linux is something like 512 sectors» which means 256KB. With a stripe size of four times this a noteworthy result is that most of the time we will be reading from only one device. Consequently the performance we will observe will generally look like that of a single device, all the time – we won’t see any benefit from the stripe of the array.

When I tweaked the read-ahead, I started at 4096 sectors and ended up at 16384 and each time the read performance increased almost proportionately. How much does a read-ahead of 16384 equate to? 8MB. With a stripe of 1MB, how many disks does that read-ahead span? Eight. How many disks in my array? Eight. In effect, I was forcing it to read the data from all eight drives in parallel with a single I/O operation. I have not tried it but I suspect the linear gains will end at this value and any further improvements would be marginal at best. Another interesting thought is that if these drives reduce their spin rate aggressively, this also forces all of the drives up to speed a little earlier than it would otherwise and keeps them busy the entire time.

This is not the only factor at work and more tuning may be necessary later. In particular, I doubt I will keep a 1MB stripe size – I believe it will become a hindrance once my workload on this machine becomes much more random in nature, and I suspect an 8MB read-ahead in this particular virtual-machine will have similar problems. For the sake of testing the raw transfer ability of the drives I think it was a worthwhile setup, however, and it does feel very zippy with interactive use.

Related Reading

For those that don’t get it, that’s 4.6TiB available. Why not say “4.6TiB”… or even “4.2TB” or at least provide a switch to do so? Most Unixish tools have -h and -H options to do just this. The same file system, but on the file server:

The difference between the two being whether it uses 1000 or 1024 as the divisor.

Related Reading

Playing with your LUNs

I’ve been a long-time VMware server user, since before ESX existed. In reality, I’ve used virtualisation since before the current market for it existed – my undergraduate university software project at UKC in the early 90′s was as part of a three-guy team to build a virtual machine monitor (VMM) – what is now called a hypervisor. When VMware got started they even emailed me a couple of times to talk about it. Most recently I’ve been running the 64bit variant of VMware Server 2 on my previous local NAS/VM host box which runs Ubuntu. This machine had roughly 6TB of space to play with and VMware could use as much or as little as I wanted.

I recently upgraded hardware and decided to give ESXi 4.1 a go, friends having reported positive things and finding the concept appealing (since it mirrored exactly what I did almost 20 years ago.) One key quirk is a limitation on file system size. In general, VMware provides virtual storage space by mapping the space occupied by a file in the host machine via software to the logical storage provided by a virtual host bus adapter (HBA).» On ESX and ESXi the common approach is for VMware to format storage volumes» using their own file system format called VMFS, optimised for large blocks. For most file systems the maximum size of a single file system is a function of the size of its blocks – the smallest allocatable unit of space – since it addresses blocks with a number and numbers on computers have a finite range. Sometimes, file systems also impose limits on the block size and for VMFS this is 8MB, which translates into a 2TB file system limit. ESX and ESXi also support presenting a raw device, via its LUN» , to virtual machines – but only under certain circumstances, as discussed below.

Moreover, if you present a volume to ESXi (and I presume ESX) that is larger than 2TB in size it gets very confused and, in my case, refused to create a file system larger than ~900MB. This was less than optimal – I had 12TB of space and ESXi was being dumb with it. But, there are workarounds.

Workaround 1: Concatenation of VMFS volumes

First, and simplest, since ESXi has a 2TB limit, is to only present volumes to it that are 2TB or less. Depending on the underlying infrastructure there are a variety of ways to do this – with partitioning or with logical volumes on a hardware RAID controller» or SAN.» VMware can then install VMFS onto these volumes and you can present them to the VM.

Most operating systems offer tools to provide software-RAID. This is usually avoided for the higher RAID levels because of the computational requirements but if your underlying storage infrastructure is already providing the fault tolerance there will be only a minimal impact if you use RAID-0, or simple block-level striping. If your hardware already stripes you will probably want to concatenate rather than stripe but the concept is similar. For FreeBSD you can use memory disk (md) to do this, for Linux you can also use md or, alternatively, the Logical Volume Manager (LVM) will oblige and with Windows one can use dynamic disks and extend our NTFS volumes across them.

In my case, using Ubuntu Server, I opted for LVM and setting up a simple concatenated volume is really very easy. I created four 1.8TB volumes on the RAID controller, created file stores in ESXi, created the virtual disks on those file stores (VMDK) and mapped them to SCSI controller targets in the virtual machine configuration. Linux then sees these four volumes as ordinary SCSI devices, sdb, sbc, sdd and sde. Using the whole of these virtual devices to create one large volume group and then carving it up into two logical volumes for /home and /data is quite easy to do.

And then add these lines to fstab:

Assuming you have /home and /data mount points already created, you can then mount them. Here’s some df output after I loaded the filesystems with some data:

Job done.

Workaround 2: Concatenation of directly mapped LUNs

Wait a moment, doesn’t that all seem horribly inefficient?

That’s a lot of layers that all have to do significant amounts of work in an I/O stack – in particular in having to map block numbers from one view of the storage to another. Worse, most of it runs on your CPU, using up cycles that could be used doing useful work.

We can remove some of this by mapping the volumes from the RAID controller directly into the virtual HBA provided to the client. VM and ESXi provides a virtual hard disk type entitled “Raw Device Mappings” (RDM) which does exactly this. However they only make the feature available to remotely-attached storage (SAN or an HBA with storage attached to external connectors).

I’ve not seen any definitive reason why they imposed this limitation, there’s certainly no technical reason to. My suspicion is to prevent a good number of their customers from shooting themselves in the proverbial foot. By mapping storage in a virtual machine to a physical unit of storage you form a very strong anchor between the machine and that storage. A key part of vSphere, in the upper-echelons of the licensing map, is to be able to migrate your virtual installations between hosts at will. VMDK files nicely abstract this storage into a quantifiable object that VMware has complete control over. Volumes on physical hardware are beyond its control. So, to reduce support calls, I believe they disabled the feature for local storage.

However, it’s only disabled in-so-far-as being able to configure it with the graphical tools is concerned. Using remote command line tools or the tech-support CLI you can use the right magic to create the raw device mapping you need. There are two commands key to this: esxcfg-scsidevs to list the path that VMware uses to reference all of the volumes presented to it by the hardware and vmkfstools to create the mapping. On my system I renamed the volumes under the hosts Configuration → Storage Adapters menu in the vSphere client to make them easier to spot, and on this page you can see the LUN values for each volume. esxcfg-scsidevs told me this:

The four devices I am interested in are “Bell Raid6 Data2” to “Data5“. Take note of the paths listed under “Console Device“.

Next you need to find where the virtual machine configuration lives. Each volume that has been formatted with VMFS is mounted with a path that uses the same ID as its underlying volume – you can see them by typing mount. In that list above, you can see the volume I labelled “Bell Raid1 Boot“. Well, I also labelled the VMFS file system with the same name, and ESXi conveniently installs a link from that name to where it is mounted, so I can cd "/vmfs/volumes/Bell Raid1 Boot/" and end up in the root of that file system. My VM configuration is in a directory off of there and I change directory into that.

The next bit of magic is using vmkfstools. In previous releases of VMware you had to create a raw mapping by cloning another virtual disk and importing it into the LUN. ESXi 4.1 lets you skip that fuss and do just what you want, thus:

This creates four virtual disks that map directly to the logical units presented to us by the HBA.

These VMDK’s can be added to your virtual machine using the vSphere client in the normal way – just tell it that they are existing virtual disks and it works out the rest. It will list them as “Mapped Raw LUN” in the VM configuration. Just make sure you attach them to a SCSI controller of the same type above. Note: You used lsilogic, any reason why?»

After that, I use LVM again, in exactly the same way, to concatenate the volumes together in the VM client, and as a result a good amount of processing and block number mapping has been eliminated.

Workaround 3: Can I directly map a big LUN?

The next logical step in this train of thought is… why use LVM at all? If we can map LUNs on the host to LUNs in the VM, why not create one big volume on the hardware and pass this right through?

I do not have an answer other than “I believe it should work, but I’ve not tested it and anytime I search online I see lots of advice against it.”

Right now my suspicion is that VMware imposed a 2TB limitation for a reason and I suspect that either their physical or their virtual hardware drivers don’t work beyond 2TB. If that is so, then any mapped LUN will also fail because it will not be able to pass through commands with big enough a block number. Or perhaps, if the drivers could work, some of the ancillary features, like snapshots, can’t cope with it.

I may test it at some point, but I suspect the performance gains will be marginal and this setup performs really very well, currently.

Issues with using mapped LUNs

The obvious issue is that if the use of mapped LUNs on local storage is not supported then anyone doing it should not cry if/when it breaks. One very obvious thing to note is that in all of the vSphere configuration and summary screens none of the mapped LUNs appear as being in use making it potentially easy to make mistakes and lose data.

Also worth thinking about is that VMware is not going to be able to do anything sensible with snapshots – experiment with it, but don’t rely on it.

Related Reading

Western Digital RE4-GP

Western Digital released their lower-power “Enterprise” RAID Edition (The RE-GP series) Serial ATA (SATA) drives some time ago, and the 1.5 and 2TB versions (RE4-GP series) last year. They were found to have an issue with certain RAID controllers and received much bad press as a result.

It’s now more than a year on and they have newer firmware, which has been generally well received. I also was toying with reducing the energy footprint of my home-business mass storage needs (not to mention to improve its reliability and performance) so I decided to give them a whirl in my newest server build which has eight Western Digital 2TB RE4-GP's» in RAID-6.

What makes it green?

Western Digital (WDC) has had a “Green” variant of their desktop drive for some time and the key difference is that it nominally runs at a lower rotational speed (5400RPM versus 7200RPM), though in reality it varies the spin rate with load. The RE version follows ostensibly the same idea but incorporates the other RAID-focussed features, such as time-limited error recovery» and a tolerance for vibration.» The area that gives one pause before choosing a drive with a lower rotational speed is it may have a direct impact on the drives data transfer performance. The drive heads can only pass over the physical locations where data is stored as fast as the surface moves beneath them and this translates directly to the rate at which data can be sent to the host. Further, if you want to read data that’s on the other side of the platter, you have to wait for it to arrive – as data density capabilities have increased, so has the volume of data that is more than half a revolution away.

Lower rotational speed has a direct impact on the power required to spin the platters but it comes at the cost of performance. You could go install the 7200RPM SATA drives, or even go with Serially Attached SCSI (SAS) to get 10,000 and 15,000RPM drives – but if they spend any time idle, or less than say 50% busy, you may be wasting energy on it. And they cost more, too.

Most of the literature published by WDC doesn’t explicitly state the range of rotational speeds the RE4-GP drives operate at and most reviewers online either guess a value or have some way of measuring the speed and end up at 5400RPM along with comments like “I’ve not seen it vary with load”. This is disappointing but then I stumbled across a WDC Knowledge Base page which supports the notion that the drives will vary their spin rate between 5400 and 7200RPM. This encouraged me and the fact that all the nay-sayers comments were from last year, and thus probably on old firmware, persuaded me to spin up some RE4-GP.

The Setup

I’ve long been a proponent of home-built machines – you get what you want, sometimes for less cash, with no vendor lock-in. Sometimes, however, I just can’t be bothered with it especially when I just want it to work. In superseding my trusty VMware/NAS host I decided to do something a bit more off-the-shelf. Supermicro and Dell were the obvious candidates for me. I was leery of Dell after reading about vendor lock-in» issues with their RAID controllers. However they didn’t count on a huge customer backlash and quickly backtracked. Now it’s worth mentioning that Dell does offer very good support on the hard drives it supplies, but sometimes I feel that vendors lose sight of the “I” in RAID. Inexpensive.

In the end what sold me, after confirming the current firmware wasn’t vendor crippled, was that Dell had introduced the PowerEdge R515: a dual AMD Opteron box with room for lots of memory and, more importantly, had a 12-drive chassis with an option for two more internal drives. I may not need all of them now, but I like having options down the road. Furthermore, they had some huge discounts for Small Business and for the second processor. It was a bargain. And free shipping too. I ordered it after forcing myself to spend a week considering whether it was a good idea or not.

I had drive trays already and I ordered the front drives from Amazon.com. I kitted the machine out with two 2.8GHz Opterons and 32GB of DRAM. The Dell PERC H700 Raid Controller I ordered with 1GB of NVRAM. Basically, this thing should fly. I ordered two internal 10k RPM SAS drives, from Dell, configured as RAID-1 for the boot device and loaded the eight RE4-GP’s in the front, configured as RAID-6.

This setup has VMware ESXi 4.1 installed on the boot volume, hosted on the internal drives, and a combination of VMFS volumes and LUNs mapped directly into a client VM on the RE4-GP drives.

How hungry are they?

Western Digital specifies the RE4-GP and it’s 7200RPM brother power consumption as follows:

I added the difference column. It shows quite a drop. On a machine with ten drives that would mean at idle the GP uses 44W less than its counterpart and under load the difference is 39W. That’s 1 kWh per day. Where I live, that’s US$75 a year. Add in that the GP model costs less than its friend, there is a decent saving to be had.

Compare this to old NAS where it’s pulling about 150W just for hard drives when busy – and even when idle, they pull more than 110W (whole machine draws about 300 to 350W total). My replacement machine is looking good. In fact, the new Dell servers brag about having more efficient power supplies than previously and include power monitoring sensors, so I can tell you that before I put the RE4-GP drives in it was using a under 200W when I taxed both CPUs. This went up to just over 200W after I inserted them and now, whilst doing some file transfers from another host, the load is hovering at about 230W.

Dell PowerEdge R515 Power Graph

In the graph it is very obvious when the drives are idle or under load. Overall, I’m better off by about 120W. 2.8 kWh per day. About $200 a year. Tangible.

Performance

The crux of the story, however, is what is the penalty of this power reduction? Well the host itself (discounting the hard drives) is using about the same amount of power as the older machine but has more cores (twelve versus four), more memory (32GB DDR3 versus 8GB DDR2) and a better memory management architecture (Opteron memory bandwidth scales with processors and this is a dual-chip board, the older machine was a Core2 Quad chip which is not known for its memory abilities). So we have lots more compute power. What about its storage?

The older machine uses 14 500GB drives (mostly RE2 but some RE3 as they’ve been replaced) drives in RAID-6 on an Areca controller with 1GB of cache memory. This configuration has usually done moderately well, easily reaching 150MBytes/sec on sequential reads once I’ve made sure it’s not fetching from the cache. By itself this is not a useful indicator of overall performance, but indicative of the ability to extract data from the platters which is, as mentioned above, a function of (to a large extent) rotational speed – my minor concern with the RE4-GP series.

This new setup has a major difference from the older one: fewer spindles. Fewer drives means that the load of reading and writing is spread across less hardware. With RAID-5 and RAID-6 setups this decreases some of the work performed by the RAID system when writing (the read-modify-write cycle) but, during reading especially, it means potentially that there is less bandwidth available.

Imagine my surprise, then, when I wrote out a 10 gigabyte file in under a minute. 200MBytes/sec. And reading was no slouch – from the cache it reached 600MBytes/sec at one point (The PERC H700 is a PCI-Express x8 card, so 8Gbits/sec is the physical maximum it can do – 600MBytes/sec is not too far from that limit!) and larger sequential reads ran between 150 and 200MBytes/sec. Note: Why would reads be slower?» Very respectable considering this is from within a guest virtual machine, too. For what it was worth the results were comparable with various file-system types, but ext4 and xfs are what I generally use with Linux. I may later post more comprehensive data from bonnie++.

Final word

It’s only been a handful of days and I’ve not really exercised the new drives, or the new machine, but I’m happy with it so far. It uses less power than the machine it supplants, gives more compute resource, more storage space, more expandability and any concerns I had over the impact of using the low-power version of the hard drives and their indeterminate-but-probably-mostly-less-than-7200RPM rotational speed are diminished.

VMware ESXi Disk Performance Graph

Yes, I’m happy.

Errata

I have posted some post-publication thoughts on the read performance noted above.

Related Reading

A couple of weeks ago I posted about automating Ad-Hoc publishing using some simple shell scripting and a modified version of the BetaBuilder utility by Hunter Hillegas. Based on a comment on his blog I’ve taken this a step further: I’ve fully integrated the publishing mechanism into Xcode. Here’s how…

Aggregate Targets

First step, if you don’t have one already, create a build configuration called something like “Ad Hoc” to differentiate a build from the normal Release and Debug variants. This is easily done: Go to the Project menu → Edit Project Settings (or, right-click or control-click on the project name at the top of Groups & Files or you can even double-click on the icon to open this). Go to the Configurations tab and “Duplicate” the Release configuration. I call this copy “Ad Hoc“. Then you can change the code signing identity of this new configuration to the Ad Hoc profile you generate on the Apple Provisioning Portal. Remember in future that if you change any settings in the Release configuration, you may also need to change them in the Ad Hoc one too. Note: Does it need to be called Ad Hoc?»

Xcode Aggregate Targets

Example Shell Script Target

You now have a new target in Groups & Files, expand it and you’ll see it has a single build step named “Run Script“. Right-click on this, select Get Info and you’re presented with a dialog with a basic “Script“. What I use here looks like:

Quite simply, ipa-packager is a shell script based on the one in a previous blog post that does the packaging for me. If it returns an error code, so do we – this is the signal to Xcode to stop the build process. All the other options to the Run Script dialog we leave at defaults.

Shell Script Configuration

Lastly, we need to make this new target depend on the target of our app. The easiest mechanism to do this is to simply drag-and-drop the app target, under Groups & Files, into the new target. Job done.

Packaging revisited

So, what next? If we try to build the project above it will complain because ipa-packager doesn’t exist. I reworked the script from my previous blog post to work either directly from the command line or using the environment variables that Xcode passes to scripts. The revised version appears below and is attached at the bottom, for download.

I also improved a number of items, particularly dependency checking so it does not re-do work that is unnecessary. Another area I paid attention to is being able to provide some of the settings in either the Xcode project or target build configuration – any user-defined values are also provided in the environment of the script. This is just as well since, because the script is called from its own target, the environment variable identifying the current target is that of the script, not our iOS app! Consequently, I have to provide at least one setting in the project: ADHOC_TARGET_NAME which I set to the name of the app.

One more major improvement is the provision of a local “configuration” file that allows a user of this script to set their own defaults without having to tweak the script itself. An example is given below. I will at some point add a command line tweak to allow for an arbitrary file to be used for this configuration.

Naturally you will wish to alter the publishing URL to one that works for you or, as described in the next section, you can do this in the project configuration inside Xcode.

The observant reader will have noticed references to an ADHOC_TEMPLATE_HTML value. This is not yet used but it will allow a project to provide its own HTML template for the generated index.html file, the same way it does now for the README.txt file.

Configuring from Xcode

So, we have our Xcode target and we have the script that does the magic of packaging and distributing an Ad Hoc build. There’s one component left – the values that we need to pass from Xcode to ipa-packager. There’s more than one place you can do this (project or target level), and you can apply it to just Ad Hoc builds or to all – even though it won’t distribute code for non Ad Hoc builds, you may wish to archive development and release builds all the same. You should decide what functionality you want and where to keep the values. The suggested approach is to keep values unique to a target within that targets configuration and is what I shall demonstrate here.

Xcode Target Build Configuration

We need the Target Info page open for the aggregate “Run Shell Script” target we added earlier (right-click, control-click or double-click on the target under Groups & Files) and select the Build tab. The item “Configuration:” is where you will decide whether to apply the values to one specific configuration (such as Ad Hoc) or to all of them. Then quit simply use the cog-menu at bottom-left to Add User-Defined Setting. There is only value you must specify as barest minimum: ADHOC_TARGET_NAME which needs to refer to the name of the target in the project that contains your app. This should also be the same as the application name – that is, the name of the .app package.

You can also specify any of the values used by ipa-packager though note that the results may be undefined if you try to redefine a value provided by Xcode itself. Where I use a common URL scheme for all my projects, others may wish to specify a unique URL for each project and you can do that here by adding a value for ADHOC_PUBLISHING_URL to their project. See the script for the full list of values it will look for.

Replaceable values

Some of the environment variables that ipa-packager uses have a notion of replaceable values. These are components of a string that are replaced with values that are determined at runtime. The idea is to enable these variables to provide a template for the final value. For example, the default for ADHOC_PUBLISHING_URL is http://localhost/iosbeta/#SHORTBUNDLEID#-#BUILD#. Note: You keep using 'localhost'. Why?» #SHORTBUNDLEID# is replaced with “abcsea” (the last component of org.flirble.abcsea) and #BUILD# is replaced by the current build number which is discovered by reading the key CFBundleVersion from the targets Info.plist file.

Not all variables are checked for replaceable values since it is not a trivial process in a shell script. You can check in the script to see which ones are by looking at the contents of variable replacement_options. At the time of writing, this contained:

The replaceable values are:

These values can appear anywhere within variables that are inspected, and any number of times.

The result

A picture says it all…

Xcode Build Results with ipa-packager

Attachments

Note that some files may be downloaded with a .txt file extension to overcome WordPress security checks. Just rename the file if necessary once downloaded.

• ipa-packager
• adhoc-distribution.conf

Related Reading

Testing front-end availability

To probe each front-end reverse-proxy we use the GET tool that is installed with the LWP package in Perl. This provides a very simple no-bells mechanism to perform an HTTP request. We configure GET with a proxy server, that of the reverse-proxy we wish to test, and the URL of a resource that it should succeed in fetching. Since we want to run this often, we choose something small and with minimal CPU impact. You do need to decide whether to fetch a cacheable object or one that will always force a fetch from the origin-server – this depends on how deeply you want to test the system. For this example, we’ll just fetch this blogs favicon.ico.

Yes, I am using fake IP addresses there.

Fixing LWP for IPv6

Unfortunately, LWP doesn’t handle IPv6 very well. There are two workarounds needed to make it work. Firstly, the code that parses HTTP proxy configuration doesn’t understand the URI form http://[2001:0db8::1]:80/ – anything with square brackets makes it croak with Bad http proxy specification. This is a simple regexp fix, per the diff below. Hopefully this will be fixed in CPAN sometime.

The second problem is more fundamental, but also easier to fix. LWP doesn’t know that IPv6 connections use a different module library from IPv4 (IPv6 in Perl is approximately broken in general as a result). Thankfully someone has a workaround for this, the module Net::INET6Glue::INET_is_INET6 (available in Ubuntu/Debian as package libnet-inet6glue-perl) which does some low-level Perl hackery to make the normal socket routines work for IPv6 also.

Armed with a modified UserAgent.pm and this INET6Glue module, we can do this:

and achieve the desired result.

Dynamic DNS

There are several aspects to updating a DNS record dynamically. You need to have a DNS server that is authoritative for a zone, is the primary authoritative server for that zone, and it must be configured to allow updates for a record, or records, or a sub-zone.
Then you can use a client utility to cause DNS records to be updated with the results of the tests performed above.

DNS server configuration

My setup uses BIND (for better or for worse) and I keep my dynamic records under the zone “dyn.flirble.org“. The record for this reverse-proxy setup has the name “ac“.

The configuration for this zone looks a bit like this:

There are other, stronger, crypto schemes, but this works for my purposes. You can generate a key simply with dnssec-keygen as follows:

Delete the two files generated when you’re done. That string “kmL…Q==” is the one you want. It’s random and you can generate it however you like.

Sending in updates

We’ll use another BIND tool for this, nsupdate. It works by collecting together a batch of commands and them sending them as a unit to the name server. As a result, the operation is (probably) atomic, meaning you can simply erase the existing records and add the new set without worrying about a period of time when you return no A or AAAA records.

I do this as follows:

Putting it all together

Here’s the entire script:

Then you just CNAME your resources to this dynamic entry. Presto, site resilience. I run this script every two minutes from cron.

Pre-loading the cache

Finally, I also have another simple script that I use to pre-load the contents of the caches. This is as simple as recursively iterating the page structure downloading the contents – in this case using wget. There are two options to ensure the cache is loaded: A forced load which always fetches from the origin server, or one that validates the cached object and fetches from the origin only if it’s out of date or not loaded.

This script specifically takes only the reverse-proxies currently listed as available by the dynamic DNS method above. At the moment it skips IPv6 addresses since they point to the same servers as the IPv4 addresses and would be redundant.

Other thoughts

And that’s my poor-guys CDN. It does have limitations, of course. It does not replace origin-server resilience. The reverse-proxy, if it follows RFCs, will frequently re-validate cached objects and if it cannot reach the origin server, will say so. There are tweaks to overcome this but they can have side effects. And there’s of course the issue that the net bandwidth benefit from using a reverse-proxy only comes from cacheable objects, but since images are generally the larger items and generally static, the benefit should be there.

If you’re using something like WordPress and one of the caching plugins, they will make pages look like static html to unregistered users and you may get a caching benefit there, but note that this means that any other plugins have to work well with a static page (server-side functionality is limited on a static cached page!)

It works for me, for now. YMMV.

Related Reading