Of Django Models…

Note to self, make sure app_label matches what you put in settings.py’s INSTALLED_APPS section, or syncdb won’t create tables for your model.

class CIJob(Loop):
    """Defines the base class for a CI Job."""
    class Meta:
        app_label = 'ci_job'

INSTALLED_APPS = (
...
    'frontend.ci_job',

My wife is just as silly as I am, so when I turned 33 a few days ago I was ambushed by soot sprites! More pics on the click through.

laughingsquid: President Obama & Star Trek’s Lt. Uhura Pose...

In the UK I would like Leonard Nimoy to nerve pinch David Cameron..

Font Smoothing - at last!

Recently I have been using Ubuntu 12.04’s development branch and while it sometimes exhibits comedy instability (yesterday the crash reporter kept crashing!) and sometimes is just plain frustrating (I still can’t get my Nvidia graphics card + two external monitors to work smoothly in Unity) I have finally got frustrated and fixed the broken font rendering when using XFCE.

Turns out that the XFCE font configuration utility sets up some, but not all font smoothing settings. This being Linux, there are several places to look for settings and several different types of font settings to look after. Yay. The key one that it seemed to be missing is the fonts.conf / fonts/conf.d setting. What needed to happen was to enable font smoothing for everyone using the machine was get the soft links in /etc/fonts/conf.d to be:

10-antialias.conf, 10-hinting.conf, 10-hinting-full.conf

And remove any other 10- files. I only had to change one on my system.

No PCI cards for me

Looks like my server is suffering from a bit of shoddy hardware. I had previously thought that I had hardware that had poor drivers, or just bad hardware, sitting in my PCI slots, but no, I had an ASM1083 PCIe <-> PCI bridge on my motherboard and it doesn’t play nice.

Arg!

So, my expandability is down to using the single PCIe 1x slot that I have left. If I plug anything into the PCI slots the machine grinds to a halt within a few minutes. Great.

Web site development using an overlay proxy

I have just written some updates to the web site at android-build.linaro.org. It is a build service that uses Jenkins as a back end (running at https://android-build.linaro.org/jenkins) and uses Django and Apache 2 to serve up the UI that you see. The UI uses JavaScript to dynamically create some page parts by querying Jenkins using JSON and querying an API that we have crafted using Django. A simple front page change is relatively simple since you can create a new instance of the entire server and play with it. My problem was that I was going to change the build pages and needed the build history from the live server to test against.
 
My fist thought was that since I was just modifying some JavaScript to use a Greasemonkey script to unload the JavaScript that I was modifying from the page supplied by android-build.linaro.org and replace it with my own. This worked at first, but seemed a bit unreliable because if the functions that I wanted to modify had already run by the time Greasemonkey had replaced it with my new script, nothing would actually change. I also realised very early on that I wanted to add a new API, so what I needed was to run the Django development server to serve up the front end pages, serve up my JavaScript using a static file server and proxy in the Jenkins instance running on android-build.linaro.org. Easy right?
 
Turns out it is. In node.js you can write a couple of lines of code to proxy a remote HTTPS server. I just needed to make it look like http://localhost:<port>/jenkins alongside the Django development server running on the same port on the server root and serve up static files next to it as well. The code is in launchpad and pasted below. I will just mention a couple of problems I found along the way and how they were solved.
 
The first problem I had was that some API calls to my local Django instance didn’t return the same value as those on the real server. This is because the local development server was not a full blown instance of everything on android-build.linaro.org. Wanting to keep using the Django development server so I could debug my API changes I changed the proxy to map my new/changed API calls to my local Django server and the rest to the remote server.
 
The second problem I ran into was that that some parts of android-build.linaro.org are only available over HTTP and others only over HTTPS. Some on both. I had started with just an HTTPS proxy, so I added an HTTP one and mapped the connections using the requested URL. Unfortunately this didn’t work. If you visit android-builds.linaro.org you will get a nice web UI. If you visit the server that it is running on (currently an EC2 instance) you will find the standard Apache 2 “It works” page. This is what I was seeing through my proxy - the server name wasn’t being used as I was expecting. Turns out I needed to add in the option “changeOrigin: true” to the proxy settings.
 
So, the proxy does the following:
  • If a file called manifest.txt is requested, return a local one. This doesn’t exist on any remote servers yet, but doing this allows me to emulate that it does.
  • If a new API call is seen, send it to the local Django development server.
  • If a static file is requested, serve it up from the local disk (allows me to test new JavaScript).
  • If a relative URL starts with /build serve it up using the HTTP proxy to android-build.linaro.org (this part of the server isn’t available over HTTPS).
  • Everything else is served up by proxying it to android-build.linaro.org. This accounts for all API calls I am not acting on locally and all calls to the Jenkins server.
Here is the whole thing. It should be easy to modify for any similar projects.
var http = require('http'),
    httpProxy = require('http-proxy'),
    util = require('util'),
    static = require('node-static');
// Create proxy servers for each remote server we are going to talk to.
var proxy_android_build_https = new httpProxy.HttpProxy({
  target: {
    host: 'android-build.linaro.org',
    port: 443,
    https: true
  }
});
var proxy_android_build_http = new httpProxy.HttpProxy({
  target: {
    host: 'android-build.linaro.org',
    port: 80
  },
  changeOrigin: true // Required because the host machine has many virtual hosts on it.
                     // Go to the ec2 instance - you will just the Apache "It worked!" page.
});
var proxy_django = new httpProxy.HttpProxy({
  target: {
    host: 'localhost',
    port: 9000
  }
});
var proxy_static = new httpProxy.HttpProxy({
  target: {
    host: 'localhost',
    port: 10101
  }
});
// Create a node-static server instance to serve static files.
var file = new(static.Server)('/home/dooferlad/dev/android-build/linaro-android-frontend/');
require('http').createServer(function (request, response)
{
  request.addListener('end', function ()
  {
    // Serve files!
    file.serve(request, response);
  });
}).listen(10101);
// Create an HTTP server - this is the one our browser will talk to. Each
// connection to it is routed based on its URL to one of the above servers.
http.createServer(function (req, res)
{
  util.puts(req.url);
  // Special case - when a manifest.txt is requested, return a static file (this isn't enabled on server yet)
  if(req.url.match(/.*manifest.txt$/))
  {
    util.puts(">static, remap");
    req.url = req.url.replace(/.*manifest.txt/, "/static/manifest.txt");
    proxy_static.proxyRequest(req, res);
  }
  else if(   req.url.match(/^\/api\/proxy-remote-file/)
          || req.url.match(/^\/api\/test-dir-exists/))
  {
    // In the case of proxy requests, make sure that localhost isn't used (not whitelisted)
    req.url = req.url.replace(/http:\/\/localhost\//, "http://android-build.linaro.org/")
    req.url = req.url.replace(/https:\/\/localhost\//, "https://android-build.linaro.org/")
    util.puts(">local");
    proxy_django.proxyRequest(req, res);
  }
  else if(req.url.match(/^\/static\//))
  {
    util.puts(">static");
    proxy_static.proxyRequest(req, res);
  }
  // Some parts of android-build.linaro.org aren't available over https...
  else if(req.url.match(/^\/builds\/.*\/\d+\//))
  {
    util.puts(">remote (*http*)");
    proxy_android_build_http.proxyRequest(req, res);
  }
  else
  {
    util.puts(">remote (httpS)");
    proxy_android_build_https.proxyRequest(req, res);
  }
  util.puts(req.url);
  util.puts("---");
}).listen(8001);

Thinking if contact cards for JoCo Cruise Crazy II.

Thinking if contact cards for JoCo Cruise Crazy II.

Backing up backups…

Since I am paranoid about disk death (I have suffered it several times) and I have machines backing up to, well, a disk, I like to make sure that those backups don’t vanish if a disk dies or if someone deletes a file by accident. I don’t need snapshots, I just need to keep as much history as I can. I also don’t care about keeping old files that have changed - my assumption is that accidental deletions may happen, but changing a backup archive by accident is unlikely.

For this reason I have hacked together a little script to do just that. You can find it here.

So, you using like this: sync_delete_oldest.py source dest

You need rsync and unix find, the script is written in python. If your source disk dies, rsync won’t delete files from the destination disk.

Well, makes me happy anyway.

Crashplan Vs Quassel: Port Fight!

I spend my days on IRC to talk to people I work with and I find that Quassel is a great IRC client and running the back end on my server means that I can always look back at what was discussed over night if I need to. I also always have all my IRC logs wherever I am, which since a lot of important stuff is discussed on IRC is invaluable.

I am also a data security nut. I have suffered the curse of hard disks going to the great spinning magnet in the sky. I haven’t found a decent solution to the problem of keeping large quantities of data in sync between multiple computers reliably without spending a fortune, but mostly I want everything backed up. Today I thought I would give CrashPlan a go. The problem was that the GUI wouldn’t connect to the back end service that does the back up work.

So, what was going on? Both Quassel and CrashPlan default to using port 4242. CrashPlan’s GUI just said that it couldn’t connect to the back end. The back end seemed convinced that it was running (trying to stop it didn’t throw any errors about it not running). A simple “port in use” message from CrashPlan would have been useful!

Since it is relatively easy to move Quassel to another port (quasselcore -p <port number>) this was easy to solve, but debugging took more than a couple of minutes.

Adding a new column to a table in django

Well, the Django book basically covers it, but because I am new to this stuff I was left asking myself what my appname was. Turns out this is just the name of the directory containing models.py, so that is easy. Since I am using PostgreSQL I used the rather great pgadmin3 to insert the new columns I found by running manage.py sqlall <appname> rather than resorting to actually writing any SQL.

Update: South looks like it manages DB changes automatically.