All of this, basically. I love the idea of live music, but so often it turns out to not be quite as a good as you imagine it to be. And, much like Keys, I listen to albums in a long-form way. Sometimes a whole artist’s discography in an evening (and for some, over a day). I much prefer it that way.

Basically this. I don’t read most technology news any more because it’s almost always scraping the bottom of the barrel crap. And bar one or two writers, this default way of thinking is all over Apple “journalism”, too.

It stinks.

This is probably the first post I’ve seen at a more mainstream publication which talks about IoT well and doesn’t get it all wrong.

I also quite like that they use the term, “Sensor Revolution”. We’re mostly talking about adding a layer of sensors to objects, not giving a fridge a web browser.

And, the location stuff reminds me of Meridian, and some of the research I’m interested in (but my stuff is attached to robots).

“As for one’s “personal brand”? Puh-lease. The concept of such a thing is so toxic to human honesty as to make me want to kick a small child. The reason we’ve only now seen one professional sportsman in the United States come out of the closet as gay is precisely because of “personal brand”. We’re all too horrified about offending some hypothetical future employer with our opinions or even our very own lives that we hide away from telling anyone anything. Sorry, but I’m with the kids on this: YOLO. Lives are short, closets are bad, crippling fear is toxic. If your personal brand demands that you live your life in fear of disclosing important parts of your life or your experience, the answer is to reject the whole sodding concept of personal brands."

I’ve — suprisingly — seen a few building designs over the years, and they’re always cool to see. When I was on placement, we moved to a new office building and had to plan out all of the data drops on both floors, and today I saw a bundle of architect’s plans for an interesting project in Plymouth (that, obviously, I can’t talk about.)

But, what’s very cool to see here is SparkFun openly documenting it all. Much of this stuff gets lost in folders of paperwork after the project is completed and all of the contractors move on.

My journey started on the 6:55 train out of Plymouth to get to Exeter for around 8:00, from there we set off on the drive to Newcastle, via Nottingham. It’s quite the trip; we landed at the Life Centre by about 18:00 with just about enough time to setup, grab something to eat, find the hotel and then crash in preparation for Day 1 of the Maker Faire.

Our plan for being there was two fold; one was to evangelise 3D printing (as in, the stuff you can do on your desk) and secondly to help sell a few RepRap kits by promoting Printed Worlds. To do this we bought along a RepRap and spent the weekend printing off whistles. These make a cool demonstration as just off the printer they essentially work (you just have to poke the ball that’s printed inside of the body). We gave away a few to slobbery children (and, er, adults.) We’ll see how successful it ends up being in the next few weeks¹.

For the evangelism side, this worked rather well. In the photo above you can see Dan explaining the purpose of a printed part to a group of people (this is likely a hotend assembly, although it’s hard to tell.)

And, as you can see below, we regularly had a rather huge group of people crowded around. The working RepRap (which was printing those whistles) can be blamed for much of it. It was a great crowd puller.

Overall, we had an interesting mix of people come over to the stand. A small proportion hadn’t heard all that much about 3D printing, had seen the RepRap going and came over to see what was going on. A much larger proportion had heard of 3D printing and were subsequently rather impressed with what we were doing and indeed, impressed with the price of the kit we were promoting. And a few others knew quite a bit (these often had RepRap’s or MakerBots or one of the many others that exist) and were more interested in us personally; that was pretty cool.

Another thing that was nice to see was design and technology teachers talking about what they were either doing, or thinking about doing with a 3D printer. This is already something that Stewart has been doing with local schools and colleges around Exeter; the students can work on their project in the morning (say, learning CAD) and then by the afternoon print it and see it as a real object. As someone who was quite uninspired after making the fifth wooden box at school, I can see this as being quite revolutionary — especially on the motivation front.

At the end of day one, we were fed and then head off in search of a pub. Firstly, we ended up at Newcastle’s Brew Dog Pub (which was rather nice), but then I suggested meeting up with Adam, who was in town, and a few other makers at a Weatherspoon’s closer to the venue. Here we ended up meeting Oli Wood and a bunch of other people. And had a fine time. Day 2 then followed, with slightly less, but still quite a lot of people. After closing time, we packed up, found something to eat and then turned in. We were leaving at 7, after all.

As a “maker” myself, it was fascinating seeing everyone else’s creations. From Pancake Bot to the Pole Dancing Robots in the photo above, there were lots of things to see. There was also a good showing from the many Hack Spaces around the country; notably Nottingham who had a huge stand.

Exhibiting at Maker Faire, and the faire itself was great fun. I hope to be at the next one. But in the mean time, I’ll likely next be at the Brighton Mini Maker Faire in September.

For me, it worked out as a nice breather before the mad rush of finishing off my project (even though I did spend some time reviewing some code and adjusting a few things).

But, more interesting was seeing what other people were coming up with. As an offshoot of Growers Nation (er, pun not intended) we had an open hardware soil analyser, entitled “MudPi”, which is able to collect humidity, temperature, dew point & moisture and is designed to be placed in the soil somewhere. They were commended for being quite close to market. I’ll be interested to see what comes next out of it.

Next, was a collection of projects from mostly Plymouth University students entitled ArduHack that was focused around the ArduSat platform. This is an Arduino based CubeSat, which itself is a project with the aim to reduce the cost of getting satellites into space — to the point where groups of people, researchers (as in, not space ones) can do. Anyway, half of the team was focused upon attaching an earth orientated camera to the ArduSat, so that people would be able to photograph themselves from space; the key bit here was being able to photography themselves — people could essentially operate their own spy satellite for a moment in time.

The other half of the ArduSat project was about bringing some of the sensors “back down to earth”. Using a combination of a Magician Robot Kit, an Arduino, Raspberry Pi and a TI SensorTag Development Kit. The key bit is the last one, it includes an IR temperature sensor, a gyroscope, accelerometer, magnetometer, pressure sensor and a humidity sensor all of which are able to communicate over the Bluetooth Low Energy standard. From here, they used the Raspberry Pi to communicate with the SensorTag, to then control the Arduino which would drive the robot around.

Finally, the team behind WebRover1 where looking at expanding what a set of Lego Mindstorms based robots could do for outreach — getting young people interested in robotics and it’s related STEM subjects. They ended up with new control code and an easy to use front end which would work with most browsers (aimed a touch devices). Their project page gives a better explanation of the user interface design process.

There were several other interesting things by other groups there, too. And, the last two, ArduHack and WebRover1 are up for global judging.

A Continuation of Predict the Sky

A few weeks before the event, I’d emailed around asking to see if any of the old team were interested in spending the weeknd continuing on with Predict the Sky. My hope was to catch up with the work which was done at the Met Office’s Weather for Fun event last year (which I’d missed) and for us to work out where we’d take the project. Personally, I wanted a documented API that we could wave around at people and then from there work on the mobile applications and so forth that we had worked on at the first Space Apps Challenge.

And actually; that’s exactly what we did. Emma was around all weekend, Sophie was with the WebRover1 team (but we grabbed her on a few bits and pieces) and we were joined by a few more people.

In the end, we ended up assembling predictthesky.org, which will contain a description of what the project is all about, the people involved, the API documentation (there’s some examples at the moment, they need actually implementing) and eventually some cool projects that are using it.

We also looked at the way we were going about calculations and the data sources for certain objects. We’ll be using PyEphem for much of it and relying on Space Track for the object references (a project run under contract to the US Department of Defence). A few other data sources will be used for other objects, too.

The whole API will eventually be implemented using Flask and you can find the code (and source for the GitHub Pages based site) under the Predict the Sky Organisation.

My next steps after I finish University (rather soon, now) is to start on the code side, and also build out the documentation — especially for others to contribute who are new to the project.

In the mean time, if you’re interested, shout at me and I’ll make sure something is done about it.

But overall, another great event, one of hopefully many more to come.

I find these kind of applications of 3D printing fascinating. Given something cool, what can someone do to make it useful with the application of a 3D printer?

I don’t usually read tech news, but I have been following along Paul Miller’s departure from the internet. It’s interesting, like Miller, I like long form articles — especially writing them — and for a long time I’ve attempted to get through the mountain of books (plus plenty more awaiting purchase in my Amazon basket) that litter every flat surface around me.

But it never happens. I get caught up with everything else. The whole concept of “work-life balance” is alien to me and instead I attempt to balance a stack of plates many times my height. And when they fall (they always do — there’s always a sudden deadline) I succeed in only catching the stuff that’s closer to me — missing the long term bits I want to be working on.

Of course, I suspect I’d come to the same conclusion as Miller if I attempted something as drastic as this. The internet is where the people are, and I’d miss them.

The work I undertook for my final year project was to investigate the use of Kalman filters for robot localisation and mapping. That wasn’t quite my initial plan, though. That was to build a Quadrotor, then write the code for the localisation and mapping, followed by running a series of tests to see how well it worked. Sadly, I ran out of time and got just past having ¾ of a Quadrotor prototype and a simple 2D simulator which just about demonstrates the thing I was attempting to.

But that’s fine as I did end up with doing quite a lot of working and learning a hell of a lot more than I could have expected.

I also have quite a few plans from where I intend to take the project — mostly the hardware and electronics side — and I’ll be slowly revealing the project as it is and then what I intend to do over time.

For the past few months this has been almost completely consuming. I look forward to finally coming up for some air at Maker Faire UK this weekend.

“We need to be truthful about the limitations of robots and robot intelligence, and measured with our predictions. We can show that real robots are both very different and much more surprising than their fictional counterparts”

Of course, articles such as this by Glenn Greenwald hardly help:

“the most sophisticated robotics use artificial intelligence that [can] seek out and record certain kinds of suspicious activity”

(Although, I do agree on the privacy, et. al. risks of using surveillance tools such as ‘drones’ in the name of “the public good”.)

Sounds familiar.

Why Chef, over it’s main competitor, Puppet? I like it more. But I can’t really describe why. I’m like that sometimes. I wish it were because one had much nicer, more accessable documentation, but neither seems to be the case. Fortunately, Vagrant has lovely documentation and so I’m not going to replicate it. I am going to go into detail about how to use both together and provide an introduction to Chef, though.

Vagrant allows you to start, stop and destroy virtual machines in a nice abstraction which also provides hooks to automate the provisioning of them. I want to be able to quickly spin up a test environment per project and when it’s started I want everything configured for me, dotfiles, libraries, applications, shared folders and so on.

Chef handles the second bit. It’s known as a configuration management tool. It describes what we want the system to look like after it’s run, but we don’t care about how we get there. Chef overuses the cooking analogy a bit, but: A recipe describes the way a specific tool is configured, along with any dependencies. A cookbook contains the recipes we wish to run on a system. I’m mostly going to talk about using “Chef Solo”, that is, Chef without a server as that works best for environments which are quickly created, used and thrown away.

Before You Begin

Firstly, you should install:

• VirtualBox
• Vagrant

Then you should at least skim read the Vagrant documentation. I’ll only describe it lightly. By the end, I’ll have a “test-environment” repository which I’ll clone whenever I need a new to spin something up.

Virtual Machines with Vagrant

Vagrant is the glue that holds all of this together and makes interacting with virtual machines as pleasurable as possible. It uses a simple Ruby DSL¹ to describe the virtual machines that should be managed. All of this is stored in a Vagrantfile.

Vagrant uses the notion of “boxes” to describe preconfigured base VMs for which we can work from. I’m using Ubuntu here. You should also look at Gareth Rushgrove’s Vagrant Boxes site. A simple Vagrant friendly install with a copy of Chef is all you need.

Firstly, create a Vagrantfile and, if you don’t have one already, grab a box:

vagrant init
vagrant box add precise32 \ http://files.vagrantup.com/precise32.box

Then start the Vagrantfile (it already has an example configuration, but I prefer it a little tidier):

Vagrant.configure("2") do |config|
  config.vm.box = "precise32"
end

Now, you can start the virtual machine and ssh into it:

vagrant up
vagrant ssh

And, you can throw the box away by running:

vagrant destroy

This might be the bit where you wish you had an SSD, if you don’t already have one. You’ll do that a lot. This base box configures Ruby, Chef and a few other things for us. And notably it’ll share the working directory in which the Vagrantfile is contained. You should now be good to start thinking about provisioning.

Provisioning with Chef

There are a few versions of Chef. These can be broadly split into “client”, known as “Solo” and “client-server”, in the case of “Chef Server”, “Chef Hosted” or “Chef Private”. These latter three provide a central area in which to manage nodes, cookbooks, roles and so on.

The node is a server (actually, it’s not necessarily a server, because you could use Chef to manage your workstation too) that is managed by Chef. The routine to complete a task is known as a recipe. This is used to install, configure and start services, from the appropriate package management system, or from source and using the correct dependencies. These recipes are grouped together in a “cookbook”.

A set of cookbooks may be configured together so that a node can have a “role” applied to it. This is something like “web” or “database”, causing Chef to configure the appropriate cookbooks for it.

“Data Bags” are collections of data (which may be encrypted) which are used by Chef to aid the install. The idea here is that the data can be kept seperate from the configuration, like usernames, passwords or keys. Below, I’m using this to put a special set of ssh keys in place so I can push code up to GitHub inside the test environment.

One of the few key points to Chef, and similar configuration management tools is that of “idempotence” — every time Chef is run, the same state will be had at the end. So, for example if you run Chef manually and everything is configured, nothing will happen. If you destroy this virtual machine and create another, once Chef has done it’s thing, it’ll be exactly the same as the old one was.

knife is Chef’s main client side tool for interacting with all of this. Whilst it’s designed to be used to interact with the Chef Server, it can also be used locally to assemble cookbooks. Once you hunt around, you’ll also see that knife has plugins available to interact with EC2, OpenStack and many others.

I’m missing a bunch of stuff out here because I’m only interested in using Chef with Vagrant. Look at the “Further Reading” heading at the bottom.

Building Cookbooks

Cookbooks look a little bit like this (with each indent being a subdirectory):

cookbooks
    cookbook-name
        recipes
            default.rb

The root cookbooks represents the default collection of cookbooks that Chef — and especially when used with Vagrant — looks for. Inside here are the cookbooks themselves. A single cookbook can contain multiple recipes, but the one that is called by default is funnily enough, called, default.rb.

A cookbook is then used to describe the tool which needs installing, for example, git. Our cookbook would be called “git” and our default.rb recipe might look like this:

package "git"

This will ensure that the package from the local package management system, “git” is installed.

Of course, that doesn’t necessarily exist on every system’s package manager. Chef has a library called “Ohai” that sniffs out the node and reports what it is and what it can do. And to interact with Ohai, Chef’s DSL provides methods like value_for_platform.

You might wish to use Opscode’s ‘chef-repo’, but I’d prefer to build this up myself, at least for now. It is all about learning it, afterall. The same applies for using knife, which I mentioned before. This can be used to create a cookbook’s file structure for you.

Opscode, and the community maintain a collection of cookbooks that you can use. A lot of these include tested support for lots of different OS styles and Linux distributions. So you’ll probably often find you might wish to lean towards these. Opscode have a community page for shared Cookbooks. And an organisation on GitHub containing all of the ones which they manage themselves. I’ll use some of these later as submodules.

Vagrant Integration

Vagrant supports configuring Chef from inside the Vagrantfile. Typically, you’d define a node.json file. This would contain the “run list” — the list of recipes that a node should manage. Vagrant handles writing, and copying over the VM this node.json file for you on vagrant up or on vagrant reload. You can also assign roles, add recipes or assign a “data bag” that should be used. The configuration for the git cookbook above should look something like this:

Vagrant.configure("2") do |config|
  config.vm.box = "precise32"

  config.vm.provision :chef_solo do |chef|
    chef.add_recipe "git"
  end
end

This will ensure Chef Solo is invoked when you start up your Vagrant VM, adding “git” to it’s run list.

A Cookbook for dotfiles

That’s all well and good. But, a more complex worked example is more useful. I want to be able to run a git clone on my dotfiles and then run the included setup.sh script, but before I do this, I need to also ensure that all of it’s dependencies (and the tools I expect) are installed. On top of this, I want a set of keys copied over so that I can access the likes of GitHub and so forth.

To do this, I’ll create a cookbook called dotfiles which is tasked with installing the client utilities I expect (git, vim, tmux). Followed by configuring the keys. After this, it will pul down a clone of my dotfiles and running it’s setup.sh script.

But before this is installed, I’ll run a few more recipes which install the system libraries and build environment. Then we’ll end up with a Vagrantfile which looks something like this:

Vagrant.configure("2") do |config|
  config.vm.box = "precise32"

  config.vm.provision :chef_solo do |chef|
    chef.json = {
      "dotfiles" => {
        "user" => "vagrant",
        "group" => "vagrant",
        "public_key" => IO.read(File.expand_path("~/.ssh/id_rsa.pub")),
        "private_key" => IO.read(File.expand_path("~/.ssh/id_rsa"))
      }
    }

    chef.add_recipe "build-essential"
    chef.add_recipe "python"
    chef.add_recipe "dotfiles"
  end
end

The recipes will be installed in the same order as they are listed here. The chef.json call passes along a data bag. This is used to provide the target user (Chef is run as root, and I probably won’t always want to be using my dotfiles cookbook like this) and to pass along my own public/private key pair², by using a quick one-liner to suck up the file and pass it along as a string.

In the dotfiles cookbook, default.rb looks like this:

# tools
%w(git vim vim-scripts tmux).each do |pkg|
  package pkg
end

home_dir = "/home/#{node['dotfiles']['user']}"

# setup ssh keys
file "#{home_dir}/.ssh/id_rsa" do
  owner node['dotfiles']['user']
  group node['dotfiles']['group']
  mode "0600"
  content node['dotfiles']['private_key']
  action :create
end

file "#{home_dir}/.ssh/id_rsa.pub" do
  owner node['dotfiles']['user']
  group node['dotfiles']['group']
  mode "0600"
  content node['dotfiles']['public_key']
  action :create
end

# sync dotfiles
git "#{home_dir}/dotfiles" do
  repository "git://github.com/nickcharlton/dotfiles.git"
  reference "master"
  enable_submodules true
  user node['dotfiles']['user']
  group node['dotfiles']['group']
  action :checkout
end

# setup dotfiles
bash "setup_dotfiles" do
  cwd "#{home_dir}/dotfiles"
  user node['dotfiles']['user']
  group node['dotfiles']['group']
  environment "HOME" => home_dir
  code "./setup.sh"
end

You can use a Ruby whitespace array (the %w() bit) to iterate around a list of packages, and pass that to the package method. This works fine for me, as I’m only going to use Debian/Ubuntu. Then, we create the ssh key files, by using the values from the data bag.

For the last bit, we clone a copy of my dotfiles repo (including it’s multiple submodules) and run the setup script. The environment variable of $HOME specifies where the user’s home directory, so we override this so that the script can handle being run by root.

The process of setting up a new environment is now a matter of cloning the “test-environment” repository, then running vagrant up. On my 2009 MacBook Pro, on a crappy DSL connection (there’s a few packages to grab) this takes just under 5 minutes, but without grabbing the packages, the Chef run only takes 30 seconds of that. With a good connection (or a local apt-cache) and something a bit faster than this machine you could cut that down quite a bit.

Further Reading

This is just enough Chef to setup a basic Debian/Ubuntu environment for the way I like things, but this should put you in enough of a position to understand the basic concepts. So the next step would be to jump over to the main Chef documentation. It’s quite readable once you understand the basics.

There’s something to be said here as it being the difference between “hacking” and “software engineering”. Software Engineering is all about the other stuff that comes once you have a something working. It’s about elegance, design patterns, testing and the interactions with operations.

I suppose this is partly why I find DevOps so interesting, it’s merging it all together, demonstrating to both sides of the coin the effort that is required to have cost effective, reliable services.

"Stuart Armstrong, a research fellow at the Future of Humanity Institute, once illustrated this phenomenon to me with has pithy take on recent primate evolution. ‘The difference in intelligence between humans and chimpanzees is tiny,’ he said. ‘But in that difference lies the contrast between 7 billion inhabitants and a permanent place on the endangered species list. That tells us it’s possible for a relatively small intelligence advantage to quickly compound and become decisive.’

A slightly fantasised, but interesting look at the future of humanity and AI.

To start with, I’ve adjusted the leading, measure and font size to make it a little easier to read. This was partly motivated by Dan Barber’s talk at Digpen VI I also rebalanced the header, but unless you visited in the last few days, you won’t notice.

Secondly, I added a “Link” section. I intend to link to things I come across every so often, sometimes with a bit more than a line of commentry. You’ll see more when I read longer and more indepth things.

It’s in the same Atom feed (and on the homepage) but implemented as a seperate directory. I might do something more with it in future. But you will see it displayed differently on the site itself; it’ll always have a “→” and the post metadata will say “Linked on <date>”, rather than “Posted on <date>”.

This is implemented in the same way as posts, but to provide both (posts and links) to the index page and atom feed, I’m using a regex, like so:

setFieldPageList (take 3 . myChronological) 
    "templates/post_full.html" "posts" (regex "^(posts|links)/")

(where take 3 . myChronological is a sorts differently than the default, the template string is where it renders to and “posts” is in the infull.)

I should move to Hakyll 4 and no doubt it’ll be easier than I expect; but I’m waiting until I finish the write up part of my degree. Also known as the next few weeks.

And of course the last two posts make up some of the stuff I’ll start writing up from my project, a retrospect over a month, maybe.

It’s a great idea, but also a nice implementation. Something similar applied to Whisky would be interesting.

But, Matplotlib was designed to produce plots for publications, and so it’s centred around printing. And so we have to deal with DPIs, inches and boundary boxes and a bit of configuration.

So, given the basic sine wave plot below:

import numpy as np
import matplotlib.pyplot as plt

figure = plt.figure()
plt.subplot(111)

X = np.linspace(-np.pi, np.pi, 256, endpoint=True)
C, S = np.cos(X), np.sin(X)

plt.plot(X, C)
plt.plot(X, S)

plt.ylim([-1.0, 1.0])
plt.xlim([-3, 3])

figure.savefig('sine_wave_plot.svg')

The last line handles saving in the simplest of forms. This gives us a standard sized SVG file. A nice way to calculate the resulting size is below. This was helped by this Stack Overflow question about page sizes.

dpi = figure.get_dpi()
size = figure.get_size_inches()
print "DPI: %i" % dpi
print "Size in inches: %i x %i" % (size[0], size[1])
print "Pixels: %i x %i" % (dpi * size[0], dpi * size[1])

Typical screen DPI is 72 (with print usually around 300), so that gives us the scaling factor. But, there’s no reason why this cannot be 100 and using this gives more obvious result. We can then use the simple equation of: pixels ÷ DPI to figure out the inches. So, for a 700 x 650 image, you’d want to specify 7 x 6.5 inches:

figure.set_dpi(100)
figure.set_size_inches(7, 6.5)

The resulting inches measurement will be rounded to the nearest integer, but this will still translate to pixels. So, a 7 x 6.5 inch image will report being 7 x 6, even though in the example above it’ll produce a 700 x 650 image.

To keep the aspect ratio correct, 50 pixels are added in the above example to take into account the y axis. Notably, the plot dispayed with show() won’t respect the same aspect ratio as the saved file.

The next thing to look at is the border around the plot. The simplest thing to configure is that of tight_layout. If turned on, this reduces the white border around the outside of the plot. The same will be applied to each subplot. After enabling this, you get Figure 2.

figure.set_tight_layout(True)

And so, with a little bit of extra effort, it’s quite possible to get perfectly sized and positioned plots using savefig. Using SVG means that the file size is relatively small and it can be scaled without losing quality — much nicer on Retina displays — and they can be embedded like any other image.

Note: You should already know how to work with Matplotlib. If you don’t, I suggest either Matplotlib for Python Developers or the SciPy Lecture Notes.

Primative shapes in Matplotlib are known as patches, and are provided by the patches module. Subclasses of patch provide implementations for Rectangles, Arrows, Ellipses (and then Arcs, Circles) and so on. All of this is part of the Artist API, which also provides support for text. In fact, everything drawn using Matplotlib is part of the artists module. It’s just a different level of access for drawing shapes compared to plots.

Drawing

There are multiple ways to write Matplotlib code¹. Whilst I’m using Pyplot in the demonstrations below, the usage is essentially the same. The differences are in how the figure is initialised.

Drawing is a matter of adding the patch to the current figure’s axes, which using Pyplot looks something like this:

import matplotlib.pyplot as plt

plt.axes()

circle = plt.Circle((0, 0), radius=0.75, fc='y')
plt.gca().add_patch(circle)

plt.axis('scaled')
plt.show()

gca() returns the current Axis instance. Setting the axis to “scaled” ensures that you can see the added shape properly. This should give you something like Figure 2².

Rectangles

rectangle = plt.Rectangle((10, 10), 100, 100, fc='r')
plt.gca().add_patch(rectangle)

rectangle is a Rectangle patch. It accepts a tuple of the bottom left hand corner, followed by a width and a height.

kwargs of either ec or fc set the edge or face colours respectively. In this case, it gives us a red rectangle without a border. Various others are also supported, as this is just a subclass of Patch.

Circles

circle = plt.Circle((0, 0), 0.75, fc='y')
plt.gca().add_patch(circle)

circle is a Circle patch. It accepts a tuple of the centre point, and then the radius.

The argument of fc gives us a yellow circle, without a border.

Lines

line = plt.Line2D((2, 8), (6, 6), lw=2.5)
plt.gca().add_line(line)

A basic line is a Line2D instance. Note that it’s an Artist itself and so its not added as a patch. The first tuple gives the x positions of the line, the second gives the y positions. lw specifies the line width. Much like lines that are part of plots in Matplotlib, the line has a lot of configurable styles, such as the following:

dotted_line = plt.Line2D((2, 8), (4, 4), lw=5., 
                         ls='-.', marker='.', 
                         markersize=50, 
                         markerfacecolor='r', 
                         markeredgecolor='r', 
                         alpha=0.5)
plt.gca().add_line(dotted_line)

which gives the lower line in Figure 3. ls defines the line style and marker gives the start and end points.

Note: If you can’t see the lines and only the end markers: There’s a Bug in WebKit which stops you seeing straight lines. You probably can’t see the plot grid lines, either.

Polygons

Polygons are just a series of points connected by lines — allowing you to draw complex shapes. The Polygon patch expects an Nx2 array of points.

points = [[2, 1], [8, 1], [8, 4]]
polygon = plt.Polygon(points)

Polygons are also a nice way to implement a multi-step line, this can be done by tuning the Polygon constructor somewhat:

points = [[2, 4], [2, 8], [4, 6], [6, 8]]
line = plt.Polygon(points, closed=None, fill=None, edgecolor='r')

This gives the red line in Figure 4. closed stops Matplotlib drawing a line between the first and last lines. fill is the colour that goes inside the shape, setting this to None removes it and the edgecolor gives the line it’s colour.

Animation

The interest here is to move certain shapes around, and in the case of something like a line (which could, for example, represent a path) update its state. Here, I’m going to get a ball to orbit around a central point at a set distance away from it:

import numpy as np
from matplotlib import pyplot as plt
from matplotlib import animation

fig = plt.figure()
fig.set_dpi(100)
fig.set_size_inches(7, 6.5)

ax = plt.axes(xlim=(0, 10), ylim=(0, 10))
patch = plt.Circle((5, -5), 0.75, fc='y')

def init():
    patch.center = (5, 5)
    ax.add_patch(patch)
    return patch,

def animate(i):
    x, y = patch.center
    x = 5 + 3 * np.sin(np.radians(i))
    y = 5 + 3 * np.cos(np.radians(i))
    patch.center = (x, y)
    return patch,

anim = animation.FuncAnimation(fig, animate, 
                               init_func=init, 
                               frames=360, 
                               interval=20,
                               blit=True)

plt.show()

To do this, I’m just using the equation for a point on a circle (but with the sine/ cosine flipped from the typical — this just means it goes around in clockwise), and using the animate function’s i argument to help compute it. This works because I’ve got 360 frames.

The init() function serves to setup the plot for animating, whilst the animate function returns the new position of the object. Setting blit=True ensures that only the portions of the image which have changed are updated. This helps hugely with performance. The purpose of returning patch, from both init() and animate() is to tell the animation function which artists are changing. Both of these except a tuple (as you can be animating multiple different artists.) The Circle is initially created off screen as we need to initialise it before animating. Without initialising off screen, blitting causes a bit of an artifact.

And so, this (with the addition of the section below) produces the video in Figure 5 below:

Jake Vanderplas’ notes on Matplotlib were invaluable in figuring this section out. Notably in blitting³. But generally, simple Artist animation is a bit thin on the ground. Hopefully this helps with that somewhat.

Output

I initially wanted to be able to export in two formats, one as an H.264 video, like the one above and as an animated gif. My initial assumption was that a gif would most likely have less of an overhead than that of a video and it would avoid browser inconsistencies.

To solve the video export, Matplotlib comes with support for exporting video sequences from an animation using the save method on Animate. It pipes out support for video to ffmpeg, but video formats are somewhat fickle and so you need to add a few more flags to get it to render correctly.

anim.save('animation.mp4', fps=30, 
          extra_args=['-vcodec', 'h264', 
                      '-pix-fmt', 'yuv420p'])

This saves an H.264 file to ‘animation.mp4’ in a format supported by QuickTime⁴. In Jake Vanderplas’ animation tutorial he doesn’t specify a pixel format and it works fine. I suspect this might be something to do with ffmpeg defaults. You’ll also need to be using Matplotlib version 1.2.0 or greater (I had issues with 1.1.0.)

Sadly, Matplotlib doesn’t support producing gifs on it’s own, but this was only the first of a few issues. We can convert the image we’ve just produced using ffmpeg and stitch it back together as a gif using ImageMagick:

ffmpeg -i animation.mp4 -r 10 output%05d.png

convert output*.png output.gif

Here, ffmpeg converts the video file from before into a series of PNG files, then we use ImageMagick’s convert command to push these back together into a gif. On its own, this ended up with a 4MB file. Even the video was only 83KB and so this isn’t so great. After attempting to compress the final output gif using ImageMagick (they have a huge article on Animation Optimisation), I turned to compressing each input file using ImageOptim. This ended up giving me a final image size of 8.0MB (and took a good hour.) Worse than I started with.

With this, I concluded that a gif isn’t that great of an option. I’d like to see animated SVG support for Matplotlib, but I’d wonder if this required moving to something which was stylised using a Document-Object-Model (DOM). Michael Droettboom touched on this in his “Matplotlib Lessons Learned” post. Even for something much more complex than these contrived examples, using JavaScript to animate the SVG is much better option. But for now, a video is the best way.

And that’s about it. One of the advantages of having the Matplotlib provided grid is that the numbers are relatively easy to determine — for my purposes this means hooking the view to the calculations.

It’s quite fantastic to see how far Digpen has come from where it started. Way back when in 2011, we had 80 people in a classroom at Plymouth University, to now having a full day of talks, split across two tracks, covering a wide selection of web topics and all at a lovely venue. Sophie put much of it down to the community itself — this is certainly true, it’s a diverse group of people far more happy to put more into it than they take out — but without her (and Andy’s steering) behind it, it wouldn’t be what it is today.

The talks started off with Matt Connelly talking about using some data analysis techniques to trace where Iteracy’s clients have come from. This was rather fascinating. The root message here was to suggest that whilst certain smaller clients may not seem so worthwhile at the time, without looking at the data you don’t know where this project will lead — it could well be a much larger project.

Chris then went on to talk about the Moby Dick Big Read, and the technical (and human) challenges that occur when a project relies on a large amount of people (none of whom technical) and suddenly becomes rather popular¹. This was followed by a break, after which we split into two tracks.

I’m not usually much of a fan of conferences with different tracks. FOSDEM or ORGCon for example, always felt like you were missing out on something. But this was done well. In fact, the track I followed hah a bit of a dConstruct vibe to it. But, the other track quite successfully seemed to fill in the more technical side (from those I talked to after.)

Will talked about making time for projects outside of the day job, of planning out long term ones and making it all fit in. He emphasised planning out as much as possible, splitting big jobs into managable ones and allowing bits of projects to slot in where they can. He also referenced Merlin Mann’s ideas of deliberately planning and making time for everything. Of keeping the devices tucked away when it’s most appropriate and planning blocks of time to keep us present at all of the things that matter.

Following this, Dan Goodwin talked about not reading and not following everything. Because it’s impossible, and so we shouldn’t feel bad about it. As long, of course, as we are doing something to keep current. After this was lunch, and a chance to explore the Eden Project. Even in a low-season like it is at the moment, it’s a fanastic place to explore.

After lunch, Stuart Marsh talked about how, and the motivations behind building his service, Issue Pop. Much like Matt’s talk at the start of the day, it’s great to see people sharing this side of projects and their businesses. Whilst we’re happy to write about or stand up and talk about the design or technical sides of our businesses, it’s still a little taboo to talk about the actual business side. After this, Dan gave a nice and accessible introduction to Design for Developers. If you’re in the same boat, I suggest picking up a copy of Design for Hackers, it’ll cover much of what he spoke about and the bits he didn’t quite have time to fit in, too.

To close off the end of the day was Jo and Stephen talking about maintaining our creativity. They mostly focused around design and illustration, but everything transfers well to programming too². Their tips and stories centred around getting away from everything, and working on other things away from work — especially in other mediums. My main take away from this, was a reminder that if it isn’t fun, what’s the point? And that we can, with a bit of planning, keep our creativity, even at the hardest of times.

The whole day was wonderful, especially for catching up with all of the friends I’ve made over the years over Devon and Cornwall. The last few weeks have been damn hard, so it also worked as a perfectly timed burnout antidote.

It’s in no-way perfect, but it’s a significant improvement from my orignal version.

Like most of my projects, it’s also up on GitHub.

Posts

There’s a few posts which I’ve finally been able to post. Most of these had been sat around as drafts, others needed a bit of completion. But, without further ado, my last few months:

• NSConf Mini: Developers vs. Designers
• dConstruct 2012
• Young Rewired State 2012
• Finishing at Rokk Media

I’m now a good few weeks into the final-year of my degree and so my focus has (necessarily) shifted quite a bit. My focus from now on will be mostly centred around Python, Qt (using PyQt and some OpenGL), Robotics, GPU Computation using CUDA and various AI topics. I already have a few drafts relating to these in the works (I find it a great way to learn). I will also be occasionally posting about my degree project — a Quadrotor platform, Simulator and associated mapping algorithms. It’s going to be an interesting year.

A Colophon

This version is based upon Hakyll. It’s a static site generator written in Haskell. Before this, I tried out multiple others, from Hyde to Mynt, and the original, Jekyll. But each had issues in one sense or another and didn’t work all that well for me. Obviously, your requirements will vary.

But, the key part of Hakyll is it’s use of pandoc. This is a document conversion tool (also written in Haskell) that I’ve since started using for generating documents for print. It also had the Markdown extensions that I wanted (footnotes, citations, LaTeX maths support, tables, etc.)

The design itself is responsive¹ — albeit, not tested absolutely everywhere — but not mobile first (because I’d written it desktop first), I’ll probably fix this at some point.

It uses TypeKit to provide “Proxima Nova” for headings, and “Adelle” for the body text. The icons are font-icons from Pictos Server.

Syntax highlighting is provided in <pre> and <code> blocks using Pygments. Mathematics symbols are through MathJax.

It’s deployed by pushing a Git repository to a VPS hosted with Prgmr.com, which I’ve used for several years now.

John and I went up horrifically early in the morning and got in just after Matt Gemmell’s talk started.

Matt Gemmell talked about crafting applications. Focusing on experiences over features, locking to the core idea and apply this to people’s needs and using people’s emotions to help improve the experience.

Jaime Newbury talked about problem solving in projects and what to do (including how to avoid) problems that (always) occur during projects. The core message was communication. Typically, either the client or the service provider fails to communicate properly on one side, causing issues to either occur quite quickly, or slowly build up over time causing misery for everyone involved.

Communication -> Understanding -> Confidence -> Trust

But also, to have standards. Firing client is okay, and it’s what will differentiate you.

Cathy Shire talked about both the frameworks that we use and the interactions between designers and developers on iOS and Mac projects.

I have a really hard time seperating the design of the product with the application. To me, there the same thing.

She advocated putting designers and developers at the same cluster of tables (as had been done at Sofa). There’s always design work, so designers won’t get bored. At Sofa, tables were organised per project, this involved a designer and developer in close proximity. If the developer can access the designer (and vice-versa), queries and assumptions can easily be talked through, saving issues later on — and avoiding the divide.

On the side of code: Go for adaptability and resiliance, not reuse. The frameworks (say, Cocoa, et. al.) provide the reuse for us. Because of this, we should build frameworks for specific jobs (as Cocoa does for seperable parts.)

Dave Wiskus gave us a story (a rather entertaining one, actually, fitting for just after lunch.) He stated that “Design is what happens when you create and care at the same time”.

Josh Clark talked about buttons. They’re a hack, they work, but we can often do better. Big screens want for big gestures. Whilst your application can be featureful, your interface shouldn’t be. UI is a social convention, and it varies from place to place.

If the content can be navigable, let it be so. Abstractions need to be learned, so if you can borrow from other areas and keep it obvious, you are improving the experience for the user. But, if you’re going to take the effort to implement an stylistic interface, you really should be taking into account all of the other aspects of it.

Importantly, remember what it was like when you were getting where you are today. What was it like when you started? Hard? Well, help make it easier for everyone.

After this, most of us went for a curry, followed by a few pub visits. It was great spending a bunch of time talking to different iOS and Mac developers — something you don’t get in Devon very often.

In the process of this I ended up finally talking to Alasdair Allan, briefly catching up with Simon Whitaker, and also meeting; Duncan Lowrie, Torbjørn Vik Lunde, a few people from Black Pixel and several others. So, all in all, it was a rather good event.

After picking up my conference pass on Thursday, I met up with Paul Adam Davis for burgers. From there I caught Andrew and Vero of Alfred¹ along with David Thompson. We then had some (rather nice) cocktails before heading off to the pre-party, where I ended up talking with people from Aral Balkan to Tantek Çelik and many more nice people too.

Talks

Ben Hammersley talked about beauty being everywhere, but something that we need to fight for. He cited the web as a good example of this, from what was once a horrible mess of marquee tags has propped up both a social and cultural evolution — art pieces to well organised centres of knowledge. But, also important: We’re the first generation to see exponential growth, and as a species, we don’t quite know how to deal with it. But, we’ll work it out over time.

Jenn Lukas talked about learning. There’s tons of stuff that we’d like to learn. But, more specifically, people from all different corners of the world, with various different backgrounds are interested in learning to code. Putting together a shared curriculum gives us a far better way of visualising progress, and hell, excitement. More importantly, though, just share what you know.

Scott Jensen talked about “default thinking” — where we use a new product or service in exactly the same way as we used the last new product or service until we stumble across the best way to use it.

And then, he announced that he’d: “work[ed] on the Newton”. There was a huge round of applaus.

His core direction was, however pointed towards “apps”. This is something that fits quite closely to my own thoughts &mash; “Apps aren’t needed for everything — we need a better interface model.” Currently, we’ve fallen onto the wrong side of a “Value : Pain” ratio. There are far too many individual (and pointless) applications available to us that it causes us some form of pain (be it findability, terrible user experiences, etc.)

How about we flip the app idea into the devices themselves? Instead the application will react with it’s environment, rather than the other way around². This fits well into the Spimes concept³. Instead of what we currently have, we can move towards a “just in time interaction” model. We’ll see stuff available when we want to interact with it, but it won’t pester us in general. So far, we’ve succeeded at location-based reminders. We can do much better.

The mobile web is a fantastic thing, but we need to cut down the pain of accessing resources and processing the information around us. And so, a model of more internet and less web would be quite fantastic (and so, more distributed, none of this single web property everyone uses.)

Ariel Waldman talked about her work with NASA, design and pushing of projects such as Space Hack and Science Hack Day.

“When I look at blackholes, I just see massive hackspaces” — Ariel Waldman.

Lauren Beukes talked about the role fiction and storytelling has in connecting to other people and understanding another persons’ perspective and how impressively (and indeed, scarily) real dystopian science fiction can become reality.

Jason Scott talked about his work with the Internet Archive and his opinions on the handling of users’ data. He stated that it’s a crime that we treat people’s data with such little respect and that we shouldn’t (paraphrased) “just delete things because you’re done with it — someone probably wants it.”.

We should store things — there’s always a sideways value to things.

And, if you’re running anything you have both a trust and a responsibility towards that data. You should certainly provide export capabilities.

Tom Armitage talked about toys. Toys make a good caracature of real-life things — and they can be anything. But they also work as a definition of a craft.

Utility and purpose is not the same thing. Something can be useful without serving a purpose. Or the use might evolve from play. Joan Erikson said: “The opposite of play is obey, not work”. You can indeed play for work."

Making is analogous to playing.

James Burke talked about how the past can help us work with the future. Because we have nowhere else to look.

Everything is, and will continue to be multi-disciplinary and because of this, our education system need to adapt. But similarly, everything is an interconnected web — both concepts and people.

The growth of nano-technology will undermine all of our current theories and philosophies as it’s all based upon scarcity. What happens when something that suddenly breaks can either fix itself, or we can print our own part inside our home?

Humanities’ typical outlook of “business and usual” is analogous to falling off a sky-scraper — it’s going fine until it isn’t, then you realise you’re screwed.

A clear message of the whole event was in moving away from multiple web silos and instead more to a model much more internet like. Collaborating together, distributed around and without the risks we currently have with both our data and our futures with the way we’re currently using the web.

It was hosted under the wing of i-DAT, the Plymouth University Institute for Art and Technology who provided rooms and other resources. We started out in the Dome (officially the Immersive Vision Theatre), with an introduction by iDAT’s Director, Mike Phillips. After talking through what iDAT did, Chris introduced what the week would be about and then we moved on to what would be are base for the rest of the week, one of the labs in Babbage.

For mentors, we had a pretty nice mix. Paul had come along with his brother, Gordon (who’s now somewhat internet-famous for his lower level work with the Raspberry Pi), and also the drop ins of two lecturers, Shirley Atkinson, one of Paul and I’s and then Chris Saunders from iDAT. And of course, Chris and myself.

We opened with a short brainstorm of resources we could work with. The goal of Rewired State is to do smart things with data and this was no different. I also made a suggestion of talking about some of the projects that we (the mentors) had worked on to try and give some potential inspiration for what could be done.

For the rest of the day, they split up into teams and started to formulate project ideas. By the end of day one, we had four distinct projects on the go.

Over the next few days, projects gradually developed; going from mere figments of imagination on day one, to most of a working product by the Thursday.

Our projects were:

• TwitTone by Sam Wray and Simon Barnes.
• Satellite Tracker by Owen Hallet and Ryan Hockett
• ODPlayground by Daniel Hart and Michael Maddocks

And on the Friday, we headed up to Birmingham for the Festival of Code.

The Festival of Code was hosted at Birmingham’s Custard Factory, a rather neat place¹ which dubs itself as a “revolutionary new arts and media quarter”. It’s filled with small units, as well as a nightclub, bar, café and other larger areas that were needed to host the Festival of Code. We arrived, registered and positioned ourselves in one of the lower room, as the teams worked on finishing off their projects ready to demonstrate the next day.

Whilst that was underway, Chris and I grabbed some lunch. We also met up with Neil Ford, Adam McGreggor and, briefly, Andy Piper, who are always nice to see. After this, we explored the venue, scoping out where everything would be.

After what must have been the biggest delivery of pizza I’d ever seen, then a set of talks by various different people, a dodgy night’s sleep on the Old Theatre’s floor was had. Saturday was judging day. As a group, we filed into Theatre 2 and then sat and watched a fantastic set of demos of projects in our category. This went on for the rest of the day (there were many projects), up until the main event in the evening.

The main event was huge. It bought everyone together to see the projects who had made it in to the final stages. One of our projects, TwitTone also made it into the final!² Then was the judging, followed by prize giving.

Seeing some of the many projects that had made it into the final was incredible. Even after going to hack days myself, I will be continually amazed by what people can do in a short period of time (very well polished stuff, too) on something that catches their eye. It’s even better when it’s young people.

Emma, the rest of the people from Rewired State and the many, many volunteers did a fantastic job, making it a very good week for everyone involved.

Want to read more? There’s a set of photos I took on Flickr and the YRS iDAT 2012 site. Some of the code made it’s way to GitHub.

There’s a lot to be said for being able to write code everyday for just over a year. But, on top of this, get advice, support and different constraints from what I’ve previously experienced on my own projects and those through University.

There are often pinacle points in your life where you go “yes, this was damn important and influential”. This was one of them.

So, thanks Adam, Tristan and the other people at Rokk Media. It was a great year.

Projects

• Bat Finder: Since the FSC Hack Day, I’ve been learning Core Audio and starting to understand audio analysis and visualisation. It’s hard stuff. It’ll be a while before I have anything significant, I suspect.
• Predict the Sky: The end result of the NASA Space Apps Challenge at the Met Office back in April. We’re not that far from a testable end product.
• RepRap: Just over a month ago, I bought a Prusa Mendel RepRap Kit. I’ve since been slowly assembling and testing it. It’s intended to be for robotics prototyping, but I’m sure I’ll end up with endless printed trinkets along the way, too.
• Quadcopter: This is intended to be the culumation of several of these projects and things I’m learning. It needs: electronics, control code¹ and then I can look into swarming behaviour and mapping.
• Environmental Monitoring: Energy usage. Common behaviours. What can we discover by logging everything over an academic year? This is a little project for October onwards.

Learning

• Maths (matrices, compound event probability, calculus): All as a foundation to the rest of the stuff I’m doing. It’s often hard, but amazing².
• Statistics: Udacity’s “Intro to Statistics” starts tomorrow. There’s data everywhere. We’re producing more at an exponential rate. The power is in those who can do stuff with it.
• AI: As I go into my final year³, this is everything. My current overall objective is to learn as much is humany possible to make next year go smoothly. I’ve been doing AI Class on-and-off over the past month.
• iOS 6 & Mountain Lion Additions: I’ve spent the last year doing this. I still find it a fantastic platform to develop for. There is a ton of stuff added at the developer level. Lots of improved things, as well as new APIs to learn.

So, yes. I like to keep myself busy. Some of this (the bat detection stuff, certainly) I do intend to document here (outside of textbooks, this kind of programming isn’t well written about). But most of that will come with time. Pester me if something is specifically interesting. Otherwise I assume it’s all easy to learn stuff.

1. Whilst quadcopters are fantastically agile machines (they can be anywhere in a given space), they have no inherant stability. Just getting one to hover is the first challenge.↩
2. I can recommend “Understanding Pure Mathematics” by Sadler and Thorning (which covers basic algebra, matrices and some calculus, etc.), which was kindly lent to me by a friend. But also Khan Academy.↩
3. Holy shit. I’ve nearly finished my degree.↩

Set in the picturesque Devon countryside, a bunch of us turned up not quite knowing what to expect. For most, it’d been quite a while since Geography lessions in school, let alone any fieldwork.

The hackday had been organised by Harriet White of the FSC, Ant Miller of BBC R&D, and John Bevan of Mozilla. But we also had people from elsewhere in the BBC, a few associated with Rewired State, FSC tutors on hand and designers and developers of various afflictions. We were also lucky to have a few people from the Bristol Hackspace, too.

Bat Detection

Mike and I worked on a low-cost bat detector. Mike had picked up an off-the-shelf bat detector and had been experimenting with outputting typical British bat tones (in the 40kHz to 110Khz range) using a multitude of different devices. In the end, he’d found that it was possible with expensive external sound cards and really cheap internal sound cards (we concluded that manufacturers are unlikely to ensure output is kept to a sensible range at the low end, but unlike the expensive stuff it certainly wouldn’t be optimised for being used like it.)

This allowed us to simulate bat calls indoors, at any time. For us, this was vital. But has the additional effect of being mightily useful in the classroom. On a weekend of fieldwork, there would not be much chance for evenings spent hunting around for bats; a simulator could provide a demonstration of what it’ll be like using bat detectors and what you would be likely to discover.

After experimenting with a few different bat detectors, we talked about what we could do to make bat finding more accessible. In the morning of the first day, David Rogers of the Priory School in Portsmouth had talked about student devices and what he (and other teachers) had been doing to use them to aid learning. Whilst the BYOD model seems slightly problematic for our case¹, a universal bit of hardware seemed a good place to start.

Off the shelf bat detectors use frequency shifting to take the bat’s ultrasonic clicks and squeaks and turn it into human audible frequencies. In the UK, bats are typically somewhere around 45Khz. (Human hearing is somewhere between 20Hz-20Khz.) Fortunately, doing this is quite typical (its how transistor radios work) and relatively cheap to do. So, with this in mind, we came up with the idea of a Square like dongle which is inserted into the audio input of a mobile device. The device can then “listen” in to the audio signal and provide some sort of output.

Mike spent the rest weekend working on the circuit, and quite quickly got something working. He then iterated over it to attempt to reduce the amount of components, and try different microphones to see which was best. I’m sure he’ll blog about it soon.

In the mean time, I looked into how we could analyse and present the audio data as it came in, and what the audio range of the iPhone² was like. I didn’t go much further with looking at the audio range, other than concluding that it was at least slightly outside of standard Human hearing range, but seemingly not up to ultrasonic.

I quickly discovered that audio programming is hard. Or at least, low-level audio programming is. Exploring the lower-level Core Audio APIs was a little beyond hacking at a weekend. I did however recall reading about Novacaine and a few other audio recording projects which kickstarted pulling in audio data.

I was quite surprised to find that graphing the data was also not easy. With libraries for most things, you assume it’ll be easy to put together a time-series graph of live data. Whilst the likes of Core Plot exist, it’s not as elegant as it could be. Subsequently I’ve spent last weekend working on something to solve it. It’s common for bat detectors to use frequency-histograms to show ‘loud’ points in the past over various frequencies at once, as this helps with detection, too. So I also need to work out how to do that. Graphing additionally comes with the advantage of being far more accessible than an audio output. We’re already shifting sounds that we, as humans are unable to hear, so why do we limit it to outputting just human audible sound?

Handily, Novacaine is extracted from a few audio analysis projects which in the end I used to demonstrate what we’d found. On the concluding part on the Sunday, the attendants, organisers and judges came around to each lab to see what we’d done. After Mike had run through the basics of bat detection, and the circuit that he’d put together, we demonstrated what could be done using an iPad and oScope. You could clearly see the bat calls on the oscilloscope display, as played through a cheap netbook and the speakers in the room.

It also turned to demonstrate the phenomena of presbycusis or age-related hearing loss, as the room speakers spread across the audial-range of the bat recordings more than on the netbook did on its own. People under the age of ~35 were unable to hear this noise, but those below could. (It’s like a very high-pitched, uncomfortable drone.) We also discovered that modern (especially cheap) digital electronics are remarkably noisy in the ultrasonic range. Passing one of the bat detectors over certain devices in various different ranges lead to a lot of background noise.

Winning an Award

We were in Lab 1 and went first. After we’d done our demonstration, we went through the other labs to see what had been done. Everyone did something fantastic. From 3D printed representations of beach profiles, to Arduino powered data logging and applications for collecting data in the field, to tools for anaylising that data back in the classroom. Everyone attempted to solve — with a sane use of technology — the problems that the team have at the FSC.

On the blog, Ant has written up about all of the projects and the winners. Which you should go and read.

Done that? Good. Well, there were four awards. And we won one of them!

We won the “People’s Choice Award”, awarded by putting stones into tubs after we’d all looked at the different projects. It was extremely satisfying to gain the approval of all of the other people there. It was fantastic to see so many people interested in what we had been doing.

Documeting it All

Before and during the weekend we had Rowan Stanfield and others documenting the event. I’m pretty sure I haven’t seen it done to this much detail before. Looking back it was quite a marvellous idea. The whole weekend is laid out for us to look back onto, as well as for others to see what was done. It provides a fantastic documented legacy of the whole event.

You can see all of this on the Fieldwork Hackday site.

Result Showcase Event

On Wednesday there is a wrap up/project show off event up in London at the Mozilla Space. I’ll be there presenting the bat stuff (with hardware, too!)

I understand it’s going to be streamed — as well as showing off the video that was produced of the event. I’ll write up something else after the event, probably with more pictures.

1. It’s hard plan out where you dedicate resources to an almost unlimited amount of possible devices. Especially on a project like this. But, more importantly, it’s unfair to over emphasise finances where we want people learning.↩
2. Given I’ve spent the last year doing iOS development it was the right thing to start on. But, there’s absolutely no reason why it wouldn’t work elsewhere too.↩

The group I was in worked on a project which we dubbed ‘Predict the Sky’. The idea was to take astronomical passovers and combine that with weather data to provide a useful indication of what you would be able to see in the night sky and it worked remarkably well. We ended up implementing a web service, iOS & Android applications and began on a web frontend. At the end we demonstrated it using a video walkthrough of the two applications and sections of the design process.

We had a fantastic team (consisting of: Tom Bell, Sophie Dennis, Rob Glover, Emma Hibling, Stephen Haddad, Angela Relle & Adam Retter.), who were all a pleasure to work with. Quite a few people commented on both how organised and on the amount we achieved in the time. I couldn’t have had a better team of people to work with.

The other teams & challenges were pretty fantastic too. The winners of the event where Growers Nation and We Love Data. The former aims to crowdsource crop growing information, combined with climate and soil data to give the best indication on when crops should be planted. We Love Data aimed to give a bunch of demonstrations on what could be done with data “in the real world” using a bunch of Arduino’s and related hardware. They produced a rather impressive wave machine using the stream running through the “Street” in the middle in the Met Office building that would activate and push through a floating “ISS Duck” when the International Space Station would pass over. It worked rather well on the Saturday night. They also produced a clever 3D-printed pollen grain which would colour according to Met Office-provided pollen counts. The idea behind this was to provide smart, internet connected objects which wouldn’t necessarily be connected to a computer. These kind of products (much like the wider Internet of Things concept) have the power to open up data (like pollen counts) to those who wouldn’t want a ‘widget’ or application to handle it on a more general purpose computer. Dougie Kinnear, one of the team members produced a rather comprehensive write up.

Over the weekend we got impressively far through the project. But not quite enough to release it. On the iOS side (the bit I worked on) I got the stage where we just needed to wire up the network request to the view. It looks like we’ll be doing some sort of workshop-like session in the next few weeks to iron out the rest and figure out where we see it in the future.

However. If you’d like to look it at now. It’s all on GitHub. Adam also wrote up what happened.

Cory Doctorow: The Coming War on General Purpose Computing

This was about the progression towards specific computers - not ones which can crunch any numbers that you throw at it, but tailored and locked down to a specific purpose.

• Original ”early” DRM -> Easy to break. Broken because it was more convenient with it removed.
• The early information age was very much a ”charge for all the things” charging module. Skipping, pausing, etc.
• DRM is exponentially problematic, one use opens up many more problems and as you solve those the problems just get bigger.
• Complete failure of the UN’s (WIPO) copyright laws.
  • they posed unrealistic demands upon reality.
  • it’s a good example of a worldwide treaty which solves a of the wrong problems.
• Law makers represent populations of people, not subject areas. They use heuristics to balance good rules for the population.
• But these heuristics utterly fail for technology.
• General Purpose Computing is much like the wheel is to the car.
• As control is aimed at General Purpose Computing, appliances look like a solution.
• Unfortunately, this opens up more issues: SOPA, Human Rights, etc.
• Copyright however is usually considered less economically important (certainly in comparison to food, water, etc), even though computers now essentially make up everything around us.
• Every big industry will turn to the easy solution of locking down General Purpose Computing.
• Thus, the fight will only get harder.

This talk was an updated version of a previous one, and also an article on BoingBoing.

Wendy Seltzer: Organising for the Open Net

• With SOPA/PIPA, representatives finally saw the power of the Internet.
• There is a long history of stupid, naïve attempts at legislation in the States (HEOA, COICA.)
• You cannot control the Internet without fundamentally breaking it.
  • the Internet would not have grown to be what it is with significant control.
• And, how can it be right that one Government Department can prop up a dying business model?
  • whilst one part of the US Government was praising the freedom of speech aspect, others were going in the opposite direction.
• The power of the hivemind for real world events cannot be understated. Even the effect that changing a Twitter avatar has is greater than one would expect. (Even if it does appear to be “slactivism”)
• The “white blood cell”/vaccine metaphor fits quite well. But, can following such a model actually make us more resilient against future issues?
• But an immune system can also kill itself if it overreacts. We (as the internet), have to be careful to not cry wolf at the next piece of legislation to rear its head.
• Motivation to keep fighting can and will come from the very specific niche interest that makes the general purpose Internet amazing.

Panel: Is all this data doing us any good?

This session was a panel, with Chris Taggart of OpenCorporates, Rufus Pollock (of Open Knowledge Foundation) and Heather Brooke (who worked on uncovering the MP’s expenses scandal). It was split into questions, but some notes ended up merged into others.

Who are ”we” empowering?

• Only about 1% of all Government data (in the UK) is being opened up.
• That is mostly because it is sold, and a business model has been built around it.
• Stops Innovation. “If your business model doesn’t match, you can’t play.”
• Closed data is growing exponentially, far faster than that which is being open.
• We’re empowering everyone. But, the small guy has far more power even with smaller resources.
• The American journalism model relies upon public records. In the UK, we don’t have this. Unfortunately, this leads to a patronage network - making it harder for new people to get in.

Value

• Governments may very well agree with opening up data, but it is expensive (it needs to be organised, packaged, etc.)
• Government need to understand the bigger social value other than the direct business profit.
• However; data without tools isn’t useful.
  • tools, analytics, etc is a resource problem.
• “Pirate” Data Sets are good and bad.
  • The MPs expenses leak was good - it raised public awareness of an ongoing issue.
  • The Wikileaks “cablegate” wasn’t good - it hurt government’s opinions on transparency (not everything is appropriate open.)
• Journalists need tools and education to be able to utilise data for the “common man”. Just opening up stuff isn’t useful.

Opening Up Data

• It’s a politcal, rather than legal issue.
  • even though it might well be against usage terms and conditions, organisations are often happy to ignore them to see people doing good things with it.
• The ease of getting data opened is somewhat because someone isn’t fighting the other way (unlike copyright, for example.)
• Do it, appologise later. “Generally” not much has happened to people doing precisely that.
• A good way forward would be “open by default” - no more Crown Copyright, for example.
• But, people need to ask for data that they wish to use - organisations may just assume people aren’t interested otherwise.
• And, uncovering something like corruption because of open data wouldn’t hurt at all.

Ross Anderson: How Secure is the Anonymisation of Open Data?

This session was given by Ross Anderson, a Security Professor at Cambridge Computer Lab. It mostly focused upon medical data, and the information than can be inferred from disparate data sets.

• Back in 1979, it was shown that you can find more data than anyone had previously expected in US Census Data.
• In the ’90’s, it was shown that a State Senator could be identified through health data.
• The same happened in the UK with NHS data.
• Generally, if you know enough information about a place (e.g. Cambridge Computer Lab.) you can easily glean a lot from publicly available data (for example, there is only one female professor, and the University publish average salary statistics.)
• It is very hard to stop data leakage through inference unless you know all of the cases where it can be “attacked”.
• The more data that is opened up (inc. social network data), the more patterns that can be seen.

Mozilla: Do Not Track

• Aims to give users a universal, simple opt-out from advertising based data collection.
• It is being standardised, by consensus by the W3C (from a mix of privacy advocates, advertisers and governments.)
• Implemented as an HTTP Header, and an accompanying JSON status file.
• In general, the policies define the levels of sharing between first and third parties on a given website (e.g.: a Twitter “Tweet” button is a third party to the New York Time’s site, until you click on it.)
• It also defines “outsource” providers for analytics and CDNs.

My initial thought with this was about it’s “opt-out” nature and requirement that organisations implement it at their end. The well-behaving organisations are likely happy to be involved and implement it - but they’re probably already not doing crappy things with data collected from our browsing sessions.

You can read more about Mozilla’s Do Not Track Project here.

The Upcoming Data Protection Act Changes

• The original, “business friendly” Data Protection Act will be (in about four years) replaced.
• In general, this will balance out the laws throughout the EU.
• Industry should see less red-tape.
• Notably however, instead of two sets of data being considered seperate (where one could be identified from the merging of the two), this would no longer be allowed. [See "How Secure is the Anonymisation of Open Data?, above.]
• What defines “personal data” is being redefined (e.g.: IP addresses, cookies, etc.). Some of this is still undefined.
• It is also possible that there will be a clause requiring individuals to request data from companies that is stored about them.
• And it is also possible that there will be a clause requiring companies to comply with user’s requests to remove data stored about them (this may cause issues for search engines, social networks, etc.)

Laurence Lessig: Recognizing The Fight We’re In

Lessig is a fantastic speaker. This talk was an outcry to release how important the fight for which the ORG stands for is. It spreads from the model of spectrum allocation for managing radio standards to corruption in modern politics.

• Spectrum allocation (and property) only works on big, wide broadcasts, not the small scale usage we generally see now (e.g.: wifi, bluetooth, etc.)
• Why can’t the TCP/IP model work?
  • defined as scalable usage with users, rather than applying scarcity.
• It stays as property because of lobbyist.
  • We need to rememeber to askt he question: “Who makes money from this?”.
• Poor broadbad, copyright extensions & SOPA insanity is all driven by greed.
• Fundamental copyright is necessary. However, it needs to benefit the creators & authors - not publishers.
• Locking up academic information in exclusivity is an ethical problem.
  • How can it be right that publically funded research makes money for private publishing organisations?
• The state of US politics is a kind of corruption - relative to the baseline on which it was founded.
• Most people do not vote because they have no belief in the system.
  • it is thought that corporate interests rule the way and that voting alone is pointless.
• The growth of citizen politics is leading to an international anti-corruption movement. This is a fantastic thing.

You can watch the video of Lessig’s talk here.

In fact, the whole day was excellent. An afternoon of interesting, detailed and well choreographed sessions. You can find a collection of them over on Lanyrd. Most have slides and there’s videos, too.

The slides are embedded below, but you can also find them on Speaker Deck. There’s a repository over on GitHub with the rest of the resources.

I gather I also agreed to do a follow on session at a future one. So, with that in mind, shout if you have something in mind that you’d like covered.

After debunking some common myths, the book delves into pushing code onto a jailbroken device. It was quite an eye opener to see how simple it was (this isn’t something I’ve done since back on iPhone OS 1.3.3, or so.) If you can compile something using gcc, you can just about as easily (and quickly) push something onto a device.

Related to the Path fiasco, the next fuller example (in Chapter 2) is about pushing the Address Book over the network. I found that quite amusing.

The book then descends into exploiting the filesystem, and other common attack vectors. This is followed by sections on manipulating the Objective-C runtime and examples in applications which are the time of writing claimed to be “secure”, but suffered from simple to discover flaws.

The second half of the book delves into advice on writing secure applications. Its “now you know what you can do, here’s how to engineer around it” style works fantastically and provides the most value - especially if you’re building something which is security conscious. Notable here was the chapter on encryption. It covered implementing SSL and flaws relating to it, as well as delving into using public-key encryption along with SSL when passing data around.

After this, the book delves into ways to obfuscate methods and protect the data the application is working with. For example, providing traps which when executed would erase any useful encryption keys, or phone home (passing logs and/or GPS coordinates) to help mitigate any knock-on effects of a breached application. Some of these security holes are due to the reflective nature of Objective-C, which allows you to modify the runtime as it is executing - catching tampering attempts, or placing honey traps for attackers can be used as another line of defence.

But, more importatly, the book aims to bring across a fundamental of security and penetration testing: You need to think and act like an attacker to see potential flaws and attacks. For this it is organised well, and because of that, it’s a great position to leap off from.

O’Reilly provided a copy of the book for to review for free, as part of their Bloggers Programme.

Security is a numbers game. Nothing is or will ever be perfectly secure¹, but the whole notion of attempting to protect users seems to have been entirely ignored. In most cases (Path, for example) this is just blind ignorance, but others seem to be actively avoiding such simple things as hashing.

I’m not sure which is worse. Is it worse that developers simply did not consider encrypting the data they were passing around, or another developer actively decided against it? At least if users know that their data is not being encrypted along the wire, and is being stored in plain text they can make a decision about whether or not to use it². I suspect the bigger issue is education. The open nature of the Web is a fantastic thing and to some extent this is transferring to mobile development, too.

But this means that developers do not have a well rounded conceptial understanding of what they are doing. Developers like Matt Gemmell shouldn’t have to write articles like: “Hashing for privacy in social apps” and receive responses from other developers.

On iOS, the Address Book API calls have almost entirely been pulled over from the Mac. Only the picker tool is Cocoa, the rest is Core Foundation (C). I suspect the lack of user permissions is because of this, rather than any malicious intent. Regardless, it’s against the developer agreement.

I certainly don’t think that a CS degree should be a prerequisite for building any software; but somewhere we, as developers need to build up interest into. Just throwing in a little bit of hashing and salting at the end of a project cannot be a valid security standpoint to take for software that is used by a wide variety of users.

1. In iOS’s case, whilst the filesystem is encrypted, it’s still like keeping your house keys under your doormat. You cannot unlock the filesystem without also shipping the keys with the device. ↩
2. But I don’t think this is a good excuse. As app developers it is our, as much as the platform providers’, responsibility to protect our users. ↩

The first session I went to was titled “Why Java for Linux applications?”. It was interesting to see someone’s approach to implementing something that needed to be cross-platform, but I find both the development environment and the eventual product of Swing hard to take as a recommendation. The next session was “Debian packaging for beginners”. I’ve done some repackaging before, but nothing completely from scratch. It was interesting to see the process done, it’s not as complex as it first makes out. After these, I headed over to the GNUstep Devroom.

Since spending a large amount of time developing for iOS, and some for the Mac, I’m obviously heavily invested in Objective-C and Cocoa. The talk on the newer features of Obj-C, by David Chisnall was especially interesting. Support for the likes of Automatic Reference Counting and blocks is far more comprehensive than I expected, and in some cases far better than Apple’s own implentation. I’ll be interested to see how the Étoilé project and others progress.

On the other hand, the final talk of the day, QuantumSTEP was less impressive. It provides an implementation of Objective-C and Cocoa for the GTA05 Open Source Phone platform. It seemed to be a little too much about slowly chasing up iOS, which itself is moving rather fast. It was interesting, though.

On the Sunday, I primarily spent time in the Virtualisation & Cloud Devroom. I had previously heard about oVirt, but without an official release hadn’t looked too deeply into it. The introduction talk, “Virtualization Management the oVirt way” and the subsequent talks outlined the architecture that was being implemented.

oVirt is intended to be an open source version of VMware’s vSphere product, but using KVM. The goal here is to provide a managed environment for running VMs, either for servers or “virtual desktops” for end users. This arrangement provides failover and general management on the server, rather than at the application level - something more common with traditional, non-web application infrastructure.

Unfortunately, it seemed a brittle solution to the problem. I’m not a fan of projects that are made up of many complex interconnecting parts to form a whole. At least not when they depend on a core central component. There also seemed to be a little confusion in the Q&A about what would happen if you upgraded the frontend (nothing bad, as it happens), which was a little disconcerting. But, at the time, it hadn’t yet been released (this happened on the 9th Feb). I shall likely be giving it ago at some point.

Other than sessions themselves, I met up with Bob on the Friday for an eventful, but good evening at the FOSDEM Beer Event. There, I also met Kostas Georgiou & Elaine McLeod from OpenGamma, who are working on an open platform for the financial services industry. Interesting stuff.

On Sunday afternoon, I met up with Goedele, which was good fun. After grabbing some lunch, we went to this odd bar which serves beer in skulls. The music was good, too.

Belgium was exceptionally cold for much of the time. When Ben and I arrived, it’d already been snowing for quite a while. I gather on the Friday night it got down to somewhere around -12°.

Belgium is a lovely country, which topped off an excellent weekend.

It depends on OmniAuth (~> 1.0), rpam-ruby19 and the PAM headers (that’s the libpam0g-dev package on Debian and Ubuntu.)

It has only been tested on Debian 6.0 using Ruby 1.9.3-p0 (but there’s no reason why it won’t work elsewhere.)

The project is on GitHub. Log an issue if something doesn’t work as you expect.

Usage

Include provider type:

use Rack::Session::Cookie
use OmniAuth::Strategies::PAM

Implement the callback (as in the OmniAuth documentation), and then navigate to: /auth/pam.

It uses the authenticated user as the UID, as on a Linux system this would be unique.

Supporting Ruby 1.8

There is an older gem available for Ruby 1.8 for supporting PAM. The syntax is slightly different, but only a small change if you wanted it.

Instead of including rpam-ruby19 instead use rpam and change the implementation of callback_response in lib/omniauth/strategies/pam.rb to:

def callback_phase
    unless authpam(request['username'], request['password'])
        return fail!(:invalid_credentials)
    end

    super
end

You will also need to add include Rpam beneath include OmniAuth::Strategy.

As it’s only small (the whole thing is tiny as it is) I figured it’d be best to document the difference, rather than aim to support two different gems.

Apache & PHP are included with Lion by default, but various parts are disabled. This will show you how to enable them without breaking the local install.

Configuration Files

The Apache config files are located in /etc/apache2 (the standard place.)

The most important part of this directory is httpd.conf. This is the main configuration file. In the users/ subdirectory are the configurations for local users (accessible through http://localhost/~<username>.)

The PHP configuration file (php5.conf) is held under other/.

The global virtual-hosts config file is held under extra/httpd-vhosts.conf. Although, by default this is commented out in httpd.conf.

The rest is mostly Apple specifics, including some of the tools included with Lion Server, and other areas of the config you are unlikely to need to change.

Permissions

By default, the main httpd.conf file is set to only be readable by all (444.) I would assume this is so users do not inadvertantly break the defaults.

To change it to be writeable by it’s own user (root), change it to 644 (readable by all, readable by it’s owner.) like so:

chmod 644 httpd.conf

You can then edit it using sudo, in your favourite editor.

Enabling PHP

Apple have always shipped with PHP disabled by default (even in Lion Server, you need to select a checkbox to specifically enable it.)

In httpd.conf, it is located somewhere around line 111, towards the end of the other LoadModule statements. This line is commented out. You need to remove the hash to it looks something like this:

LoadModule alias_module libexec/apache2/mod_alias.so
LoadModule rewrite_module libexec/apache2/mod_rewrite.so
LoadModule php5_module libexec/apache2/libphp5.so                                    

#Apple specific modules
LoadModule apple_userdir_module libexec/apache2/mod_userdir_apple.so

After doing this, you will need to restart Apache. You can do that from System Preferences/Sharing, or like so:

sudo apachectl restart

Virtual Hosts

For local development (for PHP this isn’t so often), I usually add a virtual host for the project I’m working on, and then adjust /etc/hosts to give it a usable domain.

You’ll need to edit httpd.conf again. There is an include for httpd-vhosts.conf quite a way in, somewhere around line 623.

You’ll find you’ll want to remove most of the default example content from extras/httpd-vhost.conf (by default accessing anything will give you a 403: Forbidden error.)

From there, each project/application/etc will need a VirtualHost block configured for it. This allows Apache to respond to a given domain.

The logging entries inside the block are optional, but recommended. Console.app is useful to keep an eye on the logs (it will automatically refresh when it changes.)

<VirtualHost *:80>
	DocumentRoot /path/to/files
	ServerName project.example.com

	ErrorLog /path/to/files/logs/error.log
	LogLevel warn
</VirtualHost>

To get the domain working, you need to edit /etc/hosts and add a line something like this (below the comments, before the rest.):

127.0.0.1	project.example.com

Now, you should be able to navigate to that domain and access it.

This is probably the most elegant way of running PHP applications locally. It keeps the already present tools, but makes them work as expected - which is far nicer than hacking other tools and configurations in place.

Mobile First & Emotional Design

These were probably the shortest two that I read, but both jam packed with some quite useful information. With Mobile First, some of it was already preaching to the converted, but the examples and approaches were useful to see.

Emotional Design, however, made me think. It’s about optimising designs for the way human emotions operate. And, it works very well.

Much of the book outlines case studies, some from Aarron’s work with MailChimp, and others from elsewhere. It does a very good job of making its case - something that I’ll think about in the future.

A PhD is Not Enough

I read this after spending a good chunk of time reading Matt Might’s blog. It’s a book about academic careers and attempts to reveal exactly what that entails.

Like most books on the same subject, it’s US centric, but still valuable. If the thought of academic is on the horizon, I’d suggest you give it a read.

The Passionate Programmer

This is about programming careers. I think it’s well targeted at those soon to be leaving University or in University studying something programming related.

I find what most people say on careers to be simply offensive, this is good.

A big pain point I see in people is seeing what they’re being taught at University to be the be-all and end-all of what they need to know to get a job. For some, this is obvious, but for others not so. The book makes a good point of suggesting that you firstly should be problem, not tool orientated, and that also you should be pushing out into other technologies. Especially new ones.

For the former, this means pushing far out of your comfort zone. Used to Java? Learn Ruby, or Objective-C. Or, even better go functional and learn something like Haskell. But, more important is to remember what you’re working on now is just the thing of the moment. If you keep that view, you can keep ploughing forward. And anyway, if you learn around your favourite technology and methodologies enough, you are still improving on the original one.

It is a little dry at times, and somewhat fear driven (there’s lots about the risk of having your job outsourced), but overall it’s a good book. It has some nice personal case studies, too.

The Macintosh Way

This wasn’t what I expected. It’s a book about applying the way the Macintosh was created to other companies. Some things are a little dated (it was published in 1990.)

It doesn’t contain a gushing history, but a a slightly rose tinted, but critical look at how the Macintosh was crafted (and maintained after Jobs left.)

The point of the book is how to apply this to other companies. Beware of the dating chapter. It’s painful.

Core Data for iOS

This has got to be the first book that has made me enjoy dealing with data. Up until this point, I had avoided Core Data (it is a complex framework.) But, I got this as a Christmas Present.

For iOS, Core Data is essentially a wrapper and set of tools for dealing with SQLite. I’ve not finished reading it yet, but so far this has given a enjoyable discussion of the way Core Data works and how to go about using it. It has some nice tips on performance, too - especially important if you are handling large data sets.

The information is not limited to just iOS, either. Whilst this is the primary focus, it does cover enough if you are looking into the Mac, too.

I did read a lot of other things, too (mostly catching up.) You can find those over on Pinboard.

Unfortunately, these clash a if you attempt to install and use the database daemon.

Install & Configure

To start with, install Postgres from Homebrew:

brew install postgres

Then (as told) initialise the database:

initdb /usr/local/var/postgres

But ignore the rest (as in, don’t add the LaunchAgent).

Next, it is necessary to adjust your $PATH slightly. On OS X, this is stored in /etc/paths. You need to push /usr/local/bin to be searched before /usr/bin. This will ensure that the homebrew compiled binaries are used instead of the system ones.

Control

The LaunchAgent step that was skipped is so that you can control Postgres yourself. If this is setup, it will interfere with the pg_ctl command. Usually, the LaunchAgent would be used to manage it all for you.

Starting/Stopping Postgres

pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start

pg_ctl -D /usr/local/var/postgres stop -s -m fast

These commands will start and stop Postgres, respectively. The -m fast flag on the stop command ignores the presence of other clients as it goes down.

Aliases

As these are a little long, I have these aliased to pgstart and pgstop. Done like this:

export PGDATA='/usr/local/var/postgres'
alias pgstart='pg_ctl -l $PGDATA/server.log start'
alias pgstop='pg_ctl stop -m fast'

So, there you go. That’s how to setup Postgres properly on Lion. My next challenge is to make libpq (the C library for Postgres) up and running.

Most of the calls are just C, but the example (linked at the bottom) shows all of the examples drawn inside a UIView on iOS. The rest of this assume you know what I have just written, and what CGRectMake() does.

Requirements

You need to add the QuartzCore framework to your project in Xcode. Then you need to import the headers in relevant files. All that is required for the examples here is: #import <QuartzCore/QuartzCore.h>.

A Bit of Theory

Quartz works similar to a pen on paper, especially for lines and rectangles. Much like picking the right type of pen before making marks on paper, Quartz works in a similar way. Changing the attributes of lines (the pen) is done just before drawing. Drawings also layer in a similar manner. If you tell Quartz to draw over a point that you have already drawn on, it will be layered on top of it.

// get the initial context
CGContextRef context = UIGraphicsGetCurrentContext();

// save the current state, as we'll overwrite this
CGContextSaveGState(context);

// draw stuff

// do the actual drawing
CGContextStrokePath(context);

// restore the state back after drawing on it.
CGContextRestoreGState(context);

The rest of these examples assume the above. This gets the current drawing context (in this case, the UIView we are going to draw onto, but this could be almost anything, including a PDF). It saves the current state, as this is what we’ll be accessing.

After the drawing statements, we proceed to do the actual drawing, and then save it back to the context we were working with. This then renders the drawing on the relevant view.

Lines

The simplest primitive to draw is a line. These (and arcs) make up the majority of work done with Quartz. It also fits the best within the real-world drawing metaphor.

First, you need to position the pen, then you can draw a line to the next coordinate:

// move the pen to the starting point
CGContextMoveToPoint(context, 10, 10);

// draw a line to another point
CGContextAddLineToPoint(context, 290, 10);

This starts at position 10, 10 and draws a line parallel to the top of the view to the other side. It looks like this:

A line drawn on a UIView

The pixels are relevant to the view they are created in. So, drawing to 100, 100 would draw a line at a 45° angle pointing down towards the middle of the view.

Styling

• By default, a line is 1px thick. You can change this by setting the second parameter of: CGContextSetLineWidth(context, 5).
• To set the colour: CGContextSetRGBStrokeColor(context, 1, 0, 0, 1);. Where the numbers are RGBA values.
• By default, the stroke straddles both sides of the line. If you’re positioning elements with a 5px stroke, you will want to offset the position by 2.5px.
• Each call to CGContextStrokePath(context); commits the drawing to the canvas. If you want to set different styling for different elements, you need to commit it first. Then ‘pick up the pen’ with the next element.

Rectangles

For shapes, the position is based upon a CGRect. This makes it handy for colouring the position of elements if you are drawing/comparing more complex shapes. This also means that it doesn’t use the notion of a pen.

CGContextAddRect(context, CGRectMake(10, 20, 280, 30));

This would draw a rectangle just below the line, 30px high. Rectangles are rendered from their origin point (top left corner) and drawn out from there.

A box drawn on a UIView

Circles

Circles (also called arcs and ellipses) are more complicated. There are two ways to define a complete circle, and various other ways to define different types of arc — including Bézier curves. The snippet below just describes circles, however:

The simplest way to define a circle is placing one inside a CGRect. The circle will be drawn to fit the best it can inside the rectangle.

CGContextAddEllipseInRect(context, CGRectMake(50, 70, 200, 200));

This will draw a circle with 200px in diameter below the other elements.

A circle drawn on a UIView

Memory Management

Assuming you are not using ARC, the general rules apply from Core Foundation:

• If you call a function that has ‘Create’ or ‘Copy’ in the name, it’s yours.
• If you get an object from elsewhere you don’t own it, unless you explicitly retain it.
• And to release you should call CFRelease.

Example Project

You can download an example project here. It contains the examples above, but put all into a single view. The object of importance is PrimitivesView. This provides the drawing, the rest is just a simple FlipSideViewController based project using Storyboards and ARC (for ease.) The repository is also on GitHub.

“Asking computer nerds to design social software is a little bit like hiring a Mormon bartender. Our industry abounds in people for whom social interaction has always been more of a puzzle to be reverse-engineered than a good time to be had, and the result is these vaguely Martian protocols.”

Attempting to apply a graph to people works in the first instance. But, whilst we are all connected, we are connected to each other in mind bogglingly different ways. And, people care about that.

“Social networks exist to sell you crap. The icky feeling you get when your friend starts to talk to you about Amway, or when you spot someone passing out business cards at a birthday party, is the entire driving force behind a site like Facebook.”

And:

“Now tell me one bit of original culture that’s ever come out of Facebook.”

Facebook and Google+ are passing phases. Hopefully the end of this phase will hurry up. Pictures of kittens might get tedious, but I can’t imagine a web without them. And anyway, “Liking” is “shit work”..

To me, a more important problem to solve is that of identity. A lot of smart people were working on things like OpenID (and then they went to work for Google & Facebook. sigh.) Whilst I think OpenID had its flaws, every design can be iterated on. The growth of “Sign in with [Twitter/Facebook/Google/etc.]” makes me sad.

I talked about this briefly at the weekend with Ben. We didn’t come to much of a solution, other than thinking that something like ssh-agent might work.

As humans, we are too ready and willing to give up the one thing we truly defines us: Our Identity.

UrbanScraper

It lead from a conversation with someone where I ended up having to reference some stuff a few times, I have a terrible memory for abbreviations and acronyms. I wanted to be able to find the definitions using Alfred, but Urban Dictionary didn’t have an API.

Apparently, there used to be an API (albeit, SOAP), and there appeared to be other stuff around, but I figured I’d take a hack at it myself.

UrbanScraper scrapes the definitions (on demand) from Urban Dictionary and outputs simple JSON for you to use. It mostly uses XPath.

It’s not exactly speedy (XPath isn’t terribly fast), but it does the job with very little code. I’ve put it up on GitHub. For the time being, it’s hosted up on Heroku.

For the love of god, please don’t use this as an example of how to build Sinatra/Ruby apps, how to do Screen Scraping, or how to build RESTful Web Services. It was hacked together pretty quickly.

The Alfred Extension

You can download the extension here. Obviously, you’ll need Alfred and the Powerpack to use it. But damn, is it worth it.

It’s setup with the keyword urban. Passing something like “zomg” will give you a Growl notification containing: “zOMG is a variant of the all-to-popular acronym”OMG“, meaning”Oh My God“. The”z" was originally a mistake while attempting to hit the shift…".

It (like UrbanScraper), just grabs the first result. So it might not always be very good.

Shout if anything breaks.

Being Responsive

Responsive Design is all about providing one singular web experience that is optimal for many different devices. With the growth of what some call the “mobile web”, you’ve probably come across some horrible web experiences.

I believe that we should present an optimal experience for as many browsers, in as many situations, as possible.

Jacks uses a fluid grid which is tamed with the use of media queries. It’s very much based around the work done by Ethan Marcotte, in his book, “Responsive Web Design”. If you’re interested in this at all, I urge you to go ahead and read it. It does a much better explanation than I could ever aim to do.

Jacks Gives

So, what does this actually do? Jacks provides you a bunch of presets to work off of.

These presets include:

• Some default libraries:
  • Eric Meyer’s CSS Reset
  • jQuery
  • LESS CSS
• An example HTML page to start off with.
• Some basic CSS styles for:
  • Typography
  • Colours
  • Common Effects (shadows, rounded corners, etc)
  • Flexible Media Blocks (aka, max-width: 100%)
  • Media Queries for common display sizes.

It uses LESS to make CSS easier and more maintainable.

Jacks uses media queries to tame the fluid grid. The default layout is assumed to be approximately 960px (which you’ve probably already been doing), then it provides increasingly smaller steps until a much larger block for handling large browser windows (>1200px).

I call this the upside-down triangle on a shelf method. I find it works quite well.

GitHub

Like most of my projects, it’s up on GitHub. It’s licensed under the MIT license. I intend to improve and adjust this as I go.

I encourage you to fork and add your own changes. Do a pull request if you want it to be pushed up into the main repository.

It’s been used in one production project so far. I’d certainly be interested in hearing if it’s used elsewhere!

Gitosis is in Debian’s package manager, but I wasn’t too keen on the Debian provided configuration, so here’s a few steps to sanitise it:

First, install it. You’ll obviously need the dependencies.

$ sudo apt-get install gitosis

Next, we’ll change a few settings. You might find the documentation useful for this, you’ll find that under: /usr/share/doc/gitosis.

Gitosis stores is config in a Git repository. This also means that it doesn’t have a configuration until you initialise it. Now, I’d rather use git as the user rather than gitosis. So first, create a new user:

$ sudo adduser \
    --system \
    --shell /bin/sh \
    --gecos 'Git' \
    --group \
    --disabled-password \
    --home /home/git \
    git

You can also remove the gitosis user: sudo userdel gitosis.

Next, you’ll need your public key somewhere. Then, setup the admin repository:

$ sudo -H -u git gitosis-init < <path to your public key>

Now, on your local machine, you can pull down the newly initialised Gitosis admin repository.

$ git clone git@server:gitosis-admin.git

The gitosis-admin directory contains the main config file (gitosis.conf) and the public keys of all of the users able to access repositories.

A few notes:

• The members line in the config file needs to match up with a user in keydir. But without the .pub extension.
• Groups are used to split up users and/or projects. You define a group, which holds a repository and the users with that permission.
• You can set read/write or read only privileges on groups. But, you’ll need to create a new group to define the different privileges for different users.
• Members are separated by spaces.
• A group can also be used to define members, which then can be assigned to another group. (You access the members as a variable by prefixing @).
• When you define a repository in Gitosis, it will be created on the server when you push it.

You’ll find Pro Git probably gives you a better explanation of some of the other features. But you should (after this) be able to setup Gitosis on Debian quite nicely.

In my own projects, and some of those I’ve been working on at work, I’ve wanted to be able to use QuickDialog to make development faster. Unfortunately, it requires Automatic Reference Counting, and I usually work on the current stable version of iOS.

Whilst I do think ARC is a good thing, I wanted to use QuickDialog now. So, I forked it and adjusted it where needed to make it compile under iOS 4.3 and the Xcode 4.1 toolchain without errors.

It’s not completely tested and I don’t intend to closely track the original branch, nor actively maintain it once iOS 5 is released. But, it’s there and working if you want it.

Note: In the future, I intend to push changes back into the original QuickDialog (especially when I find I want different controls.)

You can get it over on GitHub.

This fixes the crashing on load bug which ended up falling through testing (if you saw it, you’d have thought I didn’t test it, but I did, but obviously not well enough).

For the those who are interested, it was a pointer to integer comparison that caused the problem. Oddly, it was not caught by GCC/LLVM. Unfortunately, that’s one of the disadvantages of an unmanaged memory model.

But anyway. I’m pleased it is finally in the store.

Where’s Next? takes the places you need to visit and according to where you are now tells you the quickest way to visit them.

I’ve spent the last month or two slowly polishing off and fixing bugs on the project I started before the start of this year with the intent of learning iOS development.

I could have released (and “finished”) it a long time ago, but being a perfectionist, it had to be right before I submitted it. With it’s limited feature set, I now think it is ready for release. It’s said that you should be embarrassed with your 1.0 and I am. It only implements the core idea, but this means that there are many possibilities for the future.

With all projects, this wouldn’t have happened without those who I leaned on:

• Without Dr Nick Outram’s iPhone Programming Course, I’d probably still be trying to master Cocoa and not releasing my own apps and having a job doing it.
• The icon was produced by Ulrik Stoch Jensen, a young icon designer from Denmark who I found through Dribbble. He’s fantastic to work with and even put up with my backseat designing.
• Those who helped test: Chris Hunt, Luke Davies and Georgie Aggett and provided valuable feedback.
• Those on Forrst who provided me some confidence in the site design.
• and many others who I’ve spent hours and hours talking to about it.

Thanks guys, you’re awesome.

You can you read more at the App’s website here. Or go directly to iTunes.

If you like it, please mention it to others, it’d be greatly appreciated!

My first week has been rather good. The focus so far has been figuring out a few things in iOS on some projects which I hadn’t yet come across (including some sensor stuff and 2D game programming, which is rather interesting). I’ve also pitched in with a couple of web projects.

I’m currently commuting from Plymouth (via Newton Abbot and then onto Exeter St. Thomas) to the office each day. Whilst it does work, it’s unfortunately quite exhausting, so the intention is to move to Exeter in the next few weeks (any tips would be awesome!).

So far, all is going well. They’re a great bunch of people to work for, so it looks like I’m about to have a pretty good placement year.

Good

• The University could become a leader in “actual, real, commercial products”.
• In turn, the University could have more income streams, which could put fees down, or improve resources.

Bad

• Unprofitable but valuable research could be scrapped.
• The focus for new research projects could be towards quick profits and not long term innovation. So, frivolous, but profitable research could become the mainstay of research groups.
• Potential to take the very best of the Universities’ lecturers away from teaching and out into external ventures.
• Potential to remove the option for students in accessing the research groups for final year, or Masters projects.
• By pushing the IP out into other ventures, the University will no longer be able to take direct credit for the achievements of various projects.
• The research groups of the University can be a pulling force for new students, by pushing out the interests might stop that happening.

It looks like we’re stuck with it. But let’s hope it’s not contagious. The sight of a country of profit driven research feels pretty grim, to me. On top of that, let’s hope the good points come to be true, rather than the possible downsides. That will come in the future.

Once again, it was a great event, with a panel/session on Agile methodologies, a focus on two possible projects (website and store), and once again a set of lightening talks by various people. Then there was a BBQ down on the beach (though, I didn’t go to that bit.)

I also took photos this time. The lighting was much better at this venue than in Exeter, but there still was a lot of blurry shots. They certainly could have been better (shooting these things with a 50mm is quite an impractical pain…), but I’m not too displeased with them.

Anyway; you can find them here on Flickr, as always.

The conference overall was pretty good, although it did feel like some of the people were pitching a little bit too much. Given the mostly technical community, it would be nice to hear more about the decisions made behind projects - stuff that’s useful to all.

You can see the rest of the images here on Flickr.

All of these were shot on a Canon 400D, with a 50mm lens. They were shot at ISO1600. It was quite a dark room, so many of the shots taken didn’t work out too well. In other news, if someone wants me to build something for them, I could probably get a better lens.

Introduction

ListView’s are the solution to most data problems on Android (much like UITableView is used extensively in iOS). However, they are a little undocumented. The rest of this article should allow you to go from having a basic ListView, to one much more complex and useful.

The Project

This is implemented using Android 2.2, but it will work with more recent versions. These are the project settings used:

Project name: HelloListView
Build target: Android 2.2
Application name: CustomListViews
Package name: org.example.hellolistview
Create Activity: HelloListView
Min SDK Version: 8

Prerequisites

Firstly, I assume you have tried building a basic ListView (This tutorial is good for that). Secondly, I assume your data source is an ArrayList, containing objects for each element of data.

Once you have followed the basics of this guide, you will find that you can use the latter sections as you wish.

At 2010’s Google IO, there was an hour long session which talked about ListView’s extensively, you may wish to watch that first. You’ll find it here.

Building the Foundations

This starts with building a basic view which is backed onto an ArrayList. You can expand on the complexity from here.

Extending ListActivity

The first step is to subclass ListActivity instead of Activity. This provides us with some functionality specific to lists. Of note, if we have no data we can easily provide an alternative.

public class HelloListView extends ListActivity {
    // ....
}

The View XML (main.xml, list_item.xml)

main.xml

<?xml version="1.0" encoding="utf-8"?>

<LinearLayout 
	xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent">

    <ListView 
    	android:id="@android:id/list"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content" />

    <TextView
    	android:id="@android:id/empty"
    	android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="@string/empty" />
</LinearLayout>

This defines the main view. It contains a layout which fills the screen, which in turn contains two subviews. The ListView defines the view, and the TextView is the backup which is called by ListActivity when there is no data to display.

list_item.xml

<?xml version="1.0" encoding="utf-8"?>

<LinearLayout 
	xmlns:android="http://schemas.android.com/apk/res/android"
	android:layout_width="wrap_content" 
	android:layout_height="wrap_content" 
	android:background="#000000">
	<TextView 
		android:layout_width="wrap_content"
		android:layout_height="wrap_content" 
		android:id="@+id/text">
	</TextView>
</LinearLayout>

This defines a very basic view. A screenshot is shown below. You do need to provide android:layout_width and android:layout_height declarations for each, otherwise it will not render.

Data Source and Adapter

To display what is shown in the screenshot above, the following code is used. It’s not necessarily the most concise, but you should find it simple to follow.

HelloListView.java

public class HelloListView extends ListActivity {
	// define the data source
	private ArrayList<String> data;

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        // setup the data source
        this.data = new ArrayList<String>();

        // add some objects into the array list
        this.data.add("List Item 1");
        this.data.add("List Item 2");
        this.data.add("List Item 3");

        // use main.xml for the layout
        setContentView(R.layout.main);

        // setup the data adaptor
        ArrayAdapter<String> adapter = new ArrayAdapter<String>(this, R.layout.list_item, R.id.text, this.data);

        // specify the list adaptor
        setListAdapter(adapter);
    }
}

Here, we are creating the ArrayList holding our data set. Then we are adding the three elements we wish to display.

After this, we are setting up the ArrayAdapter to bridge the ListView to the dataset, and the specific item in the list.

The ArrayAdaptor translates the objects given to it (the last parameter) using a .toString(). This places the value inside the element with id @+id/text inside the list_item.xml file.

Displaying Custom Objects

For the rest of this article, the ListItem class is going to be used to display inside the ListView. This contains two members, title and subtitle. This is enough to show off using a custom adaptor.

ListItem.java

/*
* Defines a simple object to be displayed in a list view.
*/
package org.example.HelloListView;

public class ListItem {
	public String title;
	public String subTitle;

	// default constructor
	public ListItem() {
		this("Title", "Subtitle");
	}

	// main constructor
	public ListItem(String title, String subTitle) {
		super();
		this.title = title;
		this.subTitle = subTitle;
	}

	// String representation
	public String toString() {
		return this.title + " : " + this.subTitle;
	}
}

There are also a few changes to be made to the rest of the project to get this to work.

HelloListView.java

// setup the data source
this.data = new ArrayList<ListItem>();

// add some objects into the array list
ListItem item = new ListItem("Hello", "Nick");
this.data.add(item);

You will also need to change any references to ArrayList<String> to ArrayList<ListItem>.

By default, this will still display in the current incarnation. This is because we’re simply outputting a string representation of the object. To display more complex information, the SimpleAdaptor class can render checkable objects (like a CheckBox), Strings and Images. Anything more complicated would require building a custom data adaptor.

Complex ListViews with SimpleAdapter

SimpleAdapter can be used for building more complex ListViews. Compared to the ArrayAdapter class, SimpleAdaptor takes a few more arguments to map more data to more views.

Unfortunately, SimpleAdapter requires a collection of Maps to define the data. There are two ways to put together the objects wanted, the first is to create all new objects, and the second is to build a simple wrapper around Map on our original ListItem class. The former is shown below:

Creating new Objects

HelloListView.java

public class HelloListView extends ListActivity {
	// define the data source
	private ArrayList<Map> data;

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        // setup the data source
        this.data = new ArrayList<Map>();

        // add some objects into the array list
        Map m = new HashMap();
        m.put("title", "Hello");
        m.put("subtitle", "Nick");

        this.data.add(m);

        // use main.xml for the layout
        setContentView(R.layout.main);

        // setup the data adaptor
        String[] from = {"title", "subtitle"};
        int[] to = {R.id.title, R.id.subtitle};
        SimpleAdapter adapter = new SimpleAdapter(this, (List<? extends Map<String, ?>>) this.data, R.layout.list_item, from, to);

        // specify the list adaptor
        setListAdapter(adapter);
    }
}

list_item.xml

<?xml version="1.0" encoding="utf-8"?>

<LinearLayout 
	xmlns:android="http://schemas.android.com/apk/res/android"
	android:layout_width="wrap_content" 
	android:layout_height="wrap_content" 
	android:background="#000000">

	<TextView 
		android:layout_width="wrap_content"
		android:layout_height="wrap_content" 
		android:id="@+id/title">
	</TextView>

	<TextView 
		android:layout_width="wrap_content"
		android:layout_height="wrap_content" 
		android:id="@+id/subtitle">
	</TextView>

</LinearLayout>

Here, we are creating a new object (which is a Map), and placing this into our collection. This is then passed into SimpleAdapter.

Extending Map

By extending Map, we can adjust our ListItem class to appear to be a Map. Here, we will use the members of the class as the key, and their values, as the values.

ListItem.java

public class ListItem implements Map<String, String> {
	public String title;
	public String subTitle;

	// default constructor
	public ListItem() {
		this("Title", "Subtitle");
	}

	// main constructor
	public ListItem(String title, String subTitle) {
		super();
		this.title = title;
		this.subTitle = subTitle;
	}

	// String representation
	public String toString() {
		return this.title + " : " + this.subTitle;
	}

	// Map interface classes

	// return a count of our members
	public int size() {
		return 2;
	}

	// set the values of the object to null
	public void clear() {
		this.title = null;
		this.subTitle = null;
	}

	// return all of the values as a collection
	public ArrayList<String> values() {
		ArrayList<String> list = new ArrayList<String>();

		list.add(title);
		list.add(subTitle);

		return list;
	}

	// if the values of the members are null, return true
	public boolean isEmpty() {
		if ((this.title == null) && (this.subTitle == null)) {
			return true;
		} else {
			return false;
		}
	}

	// return a set of the members
	public Set<String> keySet() {
		Set<String> s = new HashSet<String>();

		s.add("title");
		s.add("subTitle");

		return s;
	}

	// return a set of the member values
	public Set entrySet() {
		Set<String> s = new HashSet<String>();

		s.add(this.title);
		s.add(this.subTitle);

		return s;
	}

	// return the value of the given key
	public String get(Object key) {
		if (key.equals("title")) {
			return this.title;
		}
		if (key.equals("subTitle")) {
			return this.subTitle;
		}
		// if we can't return a value, throw the exception
		throw new ClassCastException();
	}

	// set the value of a given key
	public String put(String key, String value) {
		if (key.equals("title")) {
			this.title = value;
		}
		if (key.equals("subTitle")) {
			this.subTitle = value;
		}
		return value;
	}

	// remove a key (nullify)
	public String remove(Object key) {
		String value = null;
		if (key.equals("title")) {
			value = this.title;
			this.title = null;
		}
		if (key.equals("subTitle")) {
			value = this.subTitle;
			this.subTitle = null;
		}
		return value;
	}

	// return boolean if we have a member
	public boolean containsKey(Object key) {
		if (key.equals("title")) {
			return true;
		}
		if (key.equals("subTitle")) {
			return true;
		}
		return false;
	}

	// return boolean if we have a member's value
	public boolean containsValue(Object value) {
		if (value.equals(this.title)) {
			return true;
		}
		if (value.equals(this.subTitle)) {
			return true;
		}
		return false;
	}

	// set the values of this map to that of another
	public void putAll(Map<? extends String, ? extends String> arg0) {
		// we only need the stub.
	}

}

This implements all of the methods required by the Map interface. You may not need all of them to support SimpleAdapter. I’d suggest subclassing the class above, and making this abstract. Then you can implement just what you need.

In HelloListView.java, the original object, then map declarations can then be replaced with:

// add some objects into the array list
ListItem item = new ListItem("Hello", "Nick");

this.data.add(item);

The view will then look like the previous screenshot.

Creating a Custom Data Adaptor

ArrayAdapter provides a simple way to add an array of strings to a ListView. SimpleAdapter provides a way to specify a more complex object (mostly containing strings) and place those into a ListView.

However, if you want to do anything more complicated you need to roll your own Adaptor. The Data Adaptor provides the link between the data and the View. It implements the methods of BaseAdapter to provide what is needed by the ListView. Here, we are assuming that you wish to stick with XML for layout (it’s the suggested way). If you wish to do it just in code, here’s an example.

For this section, we are starting again with the basic list view implemented earlier.

HelloListView.java

public class HelloListView extends ListActivity {
	private ArrayList<ListItem> data;
    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);

        // setup the data source
        this.data = new ArrayList<ListItem>();

        // create some objects
        ListItem item1 = new ListItem("Title", "Subtitle");

        // add them into the array list
        this.data.add(item1);

        // use main.xml for the layout
        setContentView(R.layout.main);

        // setup the data adaptor
        CustomAdapter adapter = new CustomAdapter(this, R.layout.list_item, this.data);

        // specify the list adaptor
        setListAdapter(adapter);
    }
}

CustomAdapter.java

public class CustomAdapter extends BaseAdapter {
	// store the context (as an inflated layout)
	private LayoutInflater inflater;
	// store the resource (typically list_item.xml)
	private int resource;
	// store (a reference to) the data
	private ArrayList<ListItem> data;

	/**
	 * Default constructor. Creates the new Adaptor object to
	 * provide a ListView with data.
	 * @param context
	 * @param resource
	 * @param data
	 */
	public CustomAdapter(Context context, int resource, ArrayList<ListItem> data) {
		this.inflater = (LayoutInflater) context.getSystemService(Context.LAYOUT_INFLATER_SERVICE);
		this.resource = resource;
		this.data = data;
	}

	/**
	 * Return the size of the data set.
	 */
	public int getCount() {
		return this.data.size();
	}

	/**
	 * Return an object in the data set.
	 */
	public Object getItem(int position) {
		return this.data.get(position);
	}

	/**
	 * Return the position provided.
	 */
	public long getItemId(int position) {
		return position;
	}

	/**
	 * Return a generated view for a position.
	 */
	public View getView(int position, View convertView, ViewGroup parent) {
		// reuse a given view, or inflate a new one from the xml
		View view;

		if (convertView == null) {
			view = this.inflater.inflate(resource, parent, false);
		} else {
			view = convertView;
		}

		// bind the data to the view object
		return this.bindData(view, position);
	}

	/**
	 * Bind the provided data to the view.
	 * This is the only method not required by base adapter.
	 */
	public View bindData(View view, int position) {
		// make sure it's worth drawing the view
		if (this.data.get(position) == null) {
			return view;
		}

		// pull out the object
		ListItem item = this.data.get(position);

		// extract the view object
		View viewElement = view.findViewById(R.id.title);
		// cast to the correct type
		TextView tv = (TextView)viewElement;
		// set the value
		tv.setText(item.title);

		viewElement = view.findViewById(R.id.subTitle);
		tv = (TextView)viewElement;
		tv.setText(item.subTitle);

		// return the final view object
		return view;
	}
}

list_item.xml

<?xml version="1.0" encoding="utf-8"?>

<LinearLayout 
   	xmlns:android="http://schemas.android.com/apk/res/android"
   	android:layout_width="wrap_content" 
   	android:layout_height="wrap_content" 
   	android:background="#000000">
   	<TextView 
   		android:layout_width="wrap_content"
   		android:layout_height="wrap_content" 
   		android:id="@+id/title">
   	</TextView>
   	<TextView 
   		android:layout_width="wrap_content"
   		android:layout_height="wrap_content" 
   		android:id="@+id/subTitle">
   	</TextView>
</LinearLayout>

The same source as above was used for the ListItem class.

The bindData method is where the most customisation will be required. This extracts the given view objects (the XML TextView, in this case), and binds the value of the members to it. Here, we have just used TextViews, but something similar would be used for other parts of the view.

Handling Events

Single Taps

The most obvious case for handling events is handling when a user taps (or clicks) a row. To do this, you define the onItemClick() method, and then inside this you can extract the original object back out to do stuff with it.

    ListView lv = getListView();
    lv.setTextFilterEnabled(true);

    lv.setOnItemClickListener(new OnItemClickListener() {
      public void onItemClick(AdapterView<?> parent, View view, int position, long id) {
    	  // When clicked, show a toast with the TextView text
    	  Toast.makeText(getApplicationContext(), parent.getItemAtPosition(position).toString(),
            Toast.LENGTH_SHORT).show();
      }
    });

This defines the on onItemClick() method for handling the event. When a user taps on the item, it prints out the string representation of the object. It uses “Toast”, Android’s discrete notifications class.

The important call is parent.getItemAtPosition(position). This extracts the selected object from the ListAdapter.

Long Taps

“long taps” are perceived to be the equivalent to right-clicks on the desktop. On a list item, you would take this to mean a desire for more information about an item.

Implementing LongClick is much the same as normal clicks (taps). The difference is merely in the method calls which are defined:

lv.setOnItemLongClickListener(new OnItemLongClickListener() {
    public boolean onItemLongClick(AdapterView<?> parent, View view, int position, long id) {
      	  // When clicked, show a toast with the TextView text
      	  Toast.makeText(getApplicationContext(), "You long clicked on: " + parent.getItemAtPosition(position).toString(),
              Toast.LENGTH_SHORT).show();

      	  return true;
    }
});

Adjusting Data in the View

This is a bit of a hack. The intention here is to show how to add and notify the adaptor of changes, rather than suggest a good way to go about doing it.

Adding New Data

The simplest way to demonstrate this is to make the list item duplicate itself on tap (or click). Add the following before the Toast declarations, and the data model will be updated:

// on press, duplicate the object
ListItem item = (ListItem)parent.getItemAtPosition(position);
ListItem newItem = new ListItem(item.title, item.subTitle);
data.add(newItem);

The next step is to notify the ListView that the data is invalid. This will reload the data from the data model, and the new object will appear.

// get the adaptor
SimpleAdapter adapter = (SimpleAdapter)parent.getAdapter();
adapter.notifyDataSetChanged();

The same goes for handling editing and deleting functions on the underlying data. You just need to make sure you keep adjust the data set, and keep the adapter informed - then the most recent data is both saved and displayed.

Code

For each of the sections in this article, I have put together a set of examples. They are Zip archives of the Eclipse projects which were created whilst I was writing this.

The projects are targeted at Android 2.2, and were used with Eclipse 3.5.2 (Helios). The code is licensed under the MIT license.

You can find the code here.

Further Reading

To move along from here, I would suggest reading Chapter 9 (Putting SQL to Work) of Hello, Android (3rd Edition) by Ed Burnette. This provides a basic introduction to ListViews, but more importantly talks about hooking up a ListView to SQLite.

As mentioned earlier, the basic ListView tutorial and the ListActivity Class Reference should also be of use to you.

Conclusion

Android’s ListView is pretty powerful, unfortunately, the documentation isn’t go great. Hopefully this will give people new to Android a kick-start in using the ListView.

Thanks to Paul Hallet for reviewing this before posting.

I think the Arduino Cookbook will fill the void between books for which their intention is to cover lots of programming fundamentals, and books which only cover the Arduino boards with a cursory mention.

Like most of O’Reilly’s cookbooks, they intend to provide a group of solutions to common problems. Sometimes they feel a little filled out; the thought that this recipe and solution are only in there to fill a gap often comes to mind. With my limited Arduino experience, I didn’t see this as much as with others, as a somewhat experienced programmer, the programming Chapters did feel a little bit long, but what is there does cover issues where you’d rather have an instant answer than not understand why something is the case.

Overall, I was impressed by what it covered. Notably, serial communication, sensors, driving motors, I2C communication and networking stand out. These are all required for the few projects I have in mind.

If you have any experience with lower-level microcontrollers (I have a little bit of experience with C on Microchips’ PIC24 chip), you’ll find some more advanced stuff like interrupts and timers towards the end. This is a welcome addition.

The appendixes should be rather useful for most; they cover a basic introduction to electronics, but, as the author says himself, it’s not a replacement for a proper book on the subject.

The author suggests that it would be good for a variety of readers, personally, I think it will slot rather nicely into the likes of most programmers - especially those who wish to add something physical to their projects.

If you are looking for something to help complete a project, or just get started with the Arduino quickly this is probably the best bet. You’ll find plenty in here to get you going.

In the spirit of transparency, this review was written for the O’Reilly Blogger Review Program, because of that, they gave me a copy of the ebook for free.

It was my first time speaking in front of more than about ten people. And as I quipped on Twitter just after, “OH GOD, THATS A LOT OF PEOPLE”. I did however enjoy it enough that I would likely agree to do it again.

Afterwards a couple of people came up to ask me about the kinds of people in TermiSoc, and many more mentioned that they’d wanted to come to one of our Monday Workspace evenings.

As fluffy as it sounds, I hope that we can stay connected with the rest of the South West Web community, they’re all a great bunch.

The general complaint was that it just wasn’t long enough, which I totally understand, I would like to have talked to many more people.

The next event is apparently going to take place in Exeter, so see you all there!

Note: You’ll need to follow these steps if you are installing via the gem or doing it manually

Prerequisites

Firstly; you’ll need to install the libaio-dev package, as instant client relies upon it.

sudo apt-get install libaio-dev

Oracle Instant Client

After this, you’ll need to pick up the Instant Client Basic and Instant Client SDK packages from Oracle’s site. Note: You’ll need to register to access these.

Jump into /opt/oracle and extract them.

You will then end up with a folder such as instant-client_11_2 (the SDK will end up in the same folder).

Inside this folder, you will want to create a symlink to the current version of the libclntsh.so.* library:

sudo ln -s libclntsh.so.11.1 libclntsh.so

Add to your PATH

To tell Ruby where to find the instant client libraries, you need to add the newly setup folder into your PATH. You can do this by doing something similar to the following:

export LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2

If you don’t add the libraries to your PATH, Ruby will not be able to access them. If you don’t add them to something like your .bashrc, they will be forgotten on reboot and when using the OCI8 gem, Ruby will complain at you.

Install the gem

Next you need to install the Ruby library itself. You can find out about compiling it yourself here.

To install you will need superuser access, however sudo will not pass in the library location, to get around this we can deliberately pass in the library path to the gem installer.

sudo env LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2 gem install ruby-oci8

At this point, everything should be working. Shared Library errors are generally caused by an issue with Oracle’s instant client, especially when the PATH has been reset. This article was based on a few others, you can read those here.

Introduction

This first session was intended to explain the overall plan, how I intended to approach the prior sessions, but to overall drum up some enthusiasm about what we should build.

Overall, we ended up with a set of ideas ranging from games, to Arduino projects, to more complicated stuff using open data sets, algorithms and natural language processing.

The plan so far is to target mobile devices, add in location services and then add some element of gaming to it. This means that I will likely run some sessions covering things from HTML5/CSS & JavaScript to Python for handling the backend. If I get time, I’ll likely cover some Google App Engine stuff, too.

It really is however based upon the idea itself. Something that will be talked about on the mailing list, over the next week or so.

If you’re reading this and not part of TermiSoc, feel free to give me a shout. I don’t wish to run a closed event, but I also hope that people will show up to each of the prior sessions. Then we can be sure we’ll hit the ground running.

Refresh: HTML5 & The New JavaScript APIs

This second session was to go over some of the new HTML5 technologies which are at our disposal. It worked out as a not too bad ~40 minute talk, although, I think I could have gone into more detail in certain areas, especially where using it now is concerned.

We seem to be on track so far, but the idea is still an issue. Give me a shout if you have any ideas!

Version Control with Git

The idea of this session was to introduce the concept of version control, but also to introduce Git as a tool itself. Instead of relying on a set of slides, the idea of the evening was to explain things at everyones pace.

As this is not something that’s covered on most people’s degree courses, it’s relevant to all.

Here is the Guide to Git which I had written beforehand. It was mostly put together through a series of notes I already had, and then formatted. Some sections could be incorrect, so use with caution, and .

This session went rather well, with people saying nice things about it afterwards. At the Unconference, Ben Masters of Fuel said that it had been useful to him. Because of this, I will likely turn it into it’s own blog post soon.

Getting Started with Python

This session was intended to be a basic introduction to Python. It’s the language I intend to use in projects for TermiSoc from this point onwards. In the session, I talked a little about the language itself (thus the slides), and then moved on to the guide I had prepared.

You will find the slides below and the guide linked here.

The next few sessions intend to expand upon where we started here, going over some of the nice build in data structures, using libraries and moving on to some web application stuff. I also intend to start talking about what we’d like to build.

Note: Please use any of the PDFs or slides linked above and modify them; but please don’t sell them, and make sure that you link back. If you want a more formal license, it’s CC BY-NC-SA. Also; please give me a shout if you’re using them somewhere, I’d love to know how they can be improved.

However, after reading various bits of writing on the subject, from Chapter 5 of Java and XML to this O’Reilly OnJava article on Simple XML Parsing with SAX and DOM, as suggested by Chris, it still didn’t cut the ultra simplicity I wanted to Just Get the Damn Thing Done™.

Sample XML File

For this example, I’m just going to show you how to deal with a single element inside an XML document. Obviously, in the real world, it wouldn’t be this simple, but it should be enough to provide the understanding you need.

<?xml version="1.0"?>
<people>
	<person>
		<age>50</age>
	</person>
</people>

Opening the File

The block below opens up the file, parses it’s contents, then closes it back up. It does this by opening up the file in a stream, then passing this stream into the SAX parser.

import java.io.*;
import org.xml.sax.*;
import org.xml.sax.helpers.*;

public class SAXClient {
    public static void main(String[] args) {
        try {
            // specify the SAXParser
            XMLReader parser = XMLReaderFactory.createXMLReader(
                "com.sun.org.apache.xerces.internal.parsers.SAXParser"
            );
            // setup the handler
            ContentHandler handler = new Handler();
            parser.setContentHandler(handler);
            // open the file
            FileInputStream in = new FileInputStream("file.xml");
            InputSource source = new InputSource(in);
            // parse the data
            parser.parse(source);
            // print an empty line under the data
            System.out.println();
            // close the file
            in.close();
        }
        catch (Exception e) {
            System.err.println(e); 
        }
    }
}

Handling the Content

We implement a class which extends the DefaultHandler, which handles what happens when it reaches each part of the XML document.

When the handler reaches the start of the element, a flag is set to true. When it reaches the end of the tag, this flag is set to false. When it is inside the tag, the contents is printed out.

import org.xml.sax.*;
import org.xml.sax.helpers.DefaultHandler;

public class Handler extends DefaultHandler {
    private boolean inAge = false;

    public void startElement(String namespaceURI, String localName, String qualifiedName, Attributes atts) throws SAXException {
        if (localName.equals("age")) inAge = true;
    }

    public void endElement(String namespaceURI, String localName, String qualifiedName) throws SAXException {
        if (localName.equals("age")) inAge = false;
    }

    public void characters(char[] ch, int start, int length) throws SAXException {
        if (inAge) {
            for (int i = start; i < start+length; i++) {
                System.out.print(ch[i]); 
            }
        }
    }
}

To deal with the output, you’ll need to implement some form of data structure to hold the data. A good tip that can be provided here is that the parser will follow from the top to the bottom when navigating your XML structure. This can be taken advantage of when dealing with the pending data.

I think competitions are a great way to stretch peoples’ minds on problems they would not otherwise encounter. There are lots of things to learn across many different fields in Computer Science, from data handling algorithms to neural networks in AI and Physics in games.

The hope is to over time produce a site where you can find open and available competitions, covering a variety of different subject areas.

Obviously, I cannot do this alone, so, if you hear of any upcoming competitions, please submit it here.

It was certainly an interesting talk, he covered the various standards (ISO 27001:2005), and pointed out a few case studies, and points often over looked. This was then followed up with a demonstration, and “practical” showing lock picking.

Points

• Electronic door access systems often just control the door latch, rather than the deadbolt itself.
  • this means that typically, it is possible to circumvent the latch using a piece of plastic. Most systems will also not log door openings, leaving such entrances undetected.
• Entry systems that use PIN code systems should be changed regularly, otherwise it will eventually be obvious to a casual visitor what the code is.
• There is usually a simple to identify weakest link in a Physical system, much like any other system.
  • an example given was an industrial fridge in student accommodation in Austria. To open the whole fridge it was secured by a much simpler lock than the individual compartments.
• People generally disclose more information over an internal phone system.
  • this is most likely because they assume you are trusted for using it.
• In a report in 2008, the Financial Services Authority found that 10/39 small and large companies in the financial industry had basic lapses of security.
  • in most cases they had implemented solutions like CCTV, and PIN code access on doors, but simply left those doors open, for example.
• Other examples highlighted oversights such as:
  • access to server rooms with visitor passes
  • keypad entry systems, but leaving main doors open
  • in companies where they had been implemented, 8/10 of the employees questioned had no idea, or only a cursory understanding of a clear desk policy.
• Another example given was of an Airport in Rome which had no security overnight. This meant that someone was able to walk through the airport, potentially planting banned objects.

In Conclusion

Overall, it was a good talk. The examples showed where people often overlook potential security policies, and the relevance of physical penetration testing. On display where Practical Lock Picking: A Physical Penetration Tester’s Training Guide and Unauthorised Access: Physical Penetration Testing for IT Security Teams which should hopefully stir on my initial interest further.

The talk was hosted by Lawrence Stevenson, who I assume holds a job title like “HE Evangelist”, or such like.

The majority of the talk focused on covering the usage of mobile technologies in HE. From mobile optimised web apps which would tell you about the status of washing machine in halls (such as at the University of Iowa), to iPad’s in class being used by students to find out information during lectures.

Figures

Some interesting figures were given out during the presentation.

• Since 2009, mobile device shipments have declined 10% year on year.
• But, smartphones grew 20%, and now account for 17% of mobile devices.
• Smartphones are projected to take 37% of the market by 2012.
• In US HE, smartphone use has gone from 1.28% in 2005, to 51.2% in 2009.
• Graduate skill sets have changed from a dominance on physical skills to skills relating to IT (intellectual). In the last 50 years, this has changed from less than 40% to over 65%, today.

Other Points

Some other points made were:

• “Mobile computing is about producing context and engagement”.
• “We’ve gone from what is your mobile strategy to where is your mobile strategy”.
• Slowness in HE integration can route back to lecturers not wanting their jobs to change. Links to Darwin were made on that point.
• There isn’t an education discount for the iPad because the demand is so high.

In Conclusion

Unfortunately, towards the end we didn’t get a chance to catch him and ask a few questions. He was swamped with people asking about iOS SDK related things, those which are widely reported too.

All in all, it was a good book. It’s laid out as a dialogue between the two, and in general split into chapters at country borders.

I think it is a book where, through the eyes of the authors, you get an idea of what travelling through many different places is like. For me at least; it’s spurred me on to other books, like Ted Simon’s Jupiter’s Travels, the story of another adventure on a Motorbike.

In the process of tweeting about it, I did the (probably stupid) thing of reading the reviews from Amazon. All-in-all, they were pretty awful. Most of them were centred around them being well known elsewhere, or the ghost writing. Others on the convoy they travelled with.

I can certainly agree with some of it, the travelling does feel a little bit over done for what they talked about in both the series and the book. Mostly, the book replays the series, not adding much to the original story.

I think this is a shame. Whilst some come would come to a book to hear about the adventure, I hoped it would come with the hope that it would give a far greater depth. There’s no answer to the ‘Why?’. Why did they pick this route? Why the specific bikes? What about the choice of the rest of the equipment? None of these were answered.

Another point made was the focus on the timetable. Whilst it’s quite easy to plan traveling around somewhere like Europe, travelling through Africa is unpredictable - almost entirely because of the road surfaces. Setting up commitments to meet with Aid Agencies, or ferries seems to me to have been a mistake. Though; I suspect some of this comes from their status, and the filming.

All of these things aside though, it’s a good read. I think it’s a good springboard onto other books, like Jupiter’s Travels, or classics like Thesiger’s Arabian Sands (which I must read).

So I made this.

Features

• border-radius
• box-shadow
• Odd/Even Selectors
• Columns (WebKit only)
• Text Stroke (WebKit only)
• text-shadow
• opacity
• display-inline

Where to Find it

You can find the source on GitHub, a download here. And the issues page here, where you can suggest new features.

Contributions are very welcome, it’s certainly not complete.

I do have a like of redoing some aspect of this every 6 months. I think it’s a cross between me changing as a person, and my desire to have this as a reflection of that.

This is a little app written in Sinatra, utilising Markdown for the markup (handled by Maruku), backed up by MySQL. It’s hosted on Apache/Passenger at Prgmr.com.

The code is also on GitHub. Including most of the content.

Plans

• Open Source the underlying “blogging engine”, as a project called “Sinba”.
• Use it to dump notes as I work on projects.
• Document a few things released, or nearing release.

Future

Over the Summer I have (through my Dad) scored a two month Sysadmin Job at ADNEC, in Abu Dhabi. Around this time last year, I spent two days doing some work experience. Which, apart from being brilliant (there’s nowhere you can find a better mix of people than abroad) and educational, was very useful to me.

For the evenings, I have a few things rolled up my sleeve. As usual, there’s a lot of things I’d like to learn, from iPhone OS Programming, to some C stuff. But, mostly, I want to go into September with at least a few simple projects up my sleeve, and something bid nearly there.

Ready? Let’s Begin.

For a project I wanted to use SQLite for dealing with it’s data. I quite readily found System.Data.SQLite. But, then I needed to figure out how ADO.NET worked. This has resulted in these short notes, mostly compiled from O’Reilly’s out of print, ADO.NET in a Nutshell.

Most of the rest of these notes are general to ADO.NET, but with a few System.Data.SQLite specifics.

Prerequisites

• You’ll need the System.Data.SQLite.dll file.
• You’ll need to reference it in the VS Solution.

• You’ll need to include the namespace at the top of your code.

  using System.Data.SQLite;

Connection String

The first stage is to connect to the DB and to do that you’ll create a “Connection” using a ConnectionString. An example is below:

// initialise the database
SQLiteConnection con = new SQLiteConnection("Data Source=../../Database.sqlite");

Opening

This simply opens the database connection. (See Closing for, ..closing).

con.Open;

You can test the status of the connection by writing:

Console.WriteLine("Connection is " + con.State.ToString());

Queries and NonQueries

ADO.NET differentiates between “Queries” and “NonQueries”. The difference is dependant on the returned data.

A “NonQuery” is an SQL statement such as “UPDATE”, “DELETE” and “INSERT”, as they do not return data. You will however get a count of how many rows were effected on execution.

Command String

NonQueries

// define our SQL.
string SQL = "UPDATE people SET name='Someone Else' WHERE id=1";

// Create the Command
SQLiteCommand cmd = new SQLiteCommand(SQL, con);

// Open the connection (if you haven't already).
con.Open();

// execute our command.
int rowsAffected = cmd.ExecuteNonQuery();

First we define our SQL query (which is just a string), then instantiate the SQLiteCommand object, and finally run our command.

Returning a Single Result

The ExecuteScalar() method returns a single value. This would be used to return the result of a calculation. Such as requesting a COUNT.

cmd.ExecuteScalar();

The returned value is an Object of the result.

Example:

string SQL = "SELECT id FROM people WHERE id='1';";

SQLiteCommand cmd = new SQLiteCommand(SQL, con);

con.Open();

object result = cmd.ExecuteScalar();
int convert = Convert.ToInt16(result);

con.Close();
Console.WriteLine(convert.ToString() + " rows.");

Creating Tables

Creating tables is done in a similar way to the above, but with just a different SQL command. (So, I won’t mention it here.)

Performing SELECTs / Using DataReader

The DataReader class provides the methods to iterate over rows in a database. So, this is the function to use to perform SELECT statements that return more than one record.

Example:

string SQL = "SELECT ContactName FROM Customers";

// Create ADO.NET objects.
SQLiteConnection con = new SQLiteConnection("Data Source=../../Database.sqlite");
SQLiteCommand cmd = new SqlCommand(SQL, con);
SQLiteDataReader reader = null;

// Execute the command.
try
{
    con.Open();
    reader = cmd.ExecuteReader();

    // Iterate over the results.
    while (reader.Read())
    {
        lstNames.Items.Add(reader["ContactName"]);
    }
}
catch (Exception err)
{
    MessageBox.Show(err.ToString());
}
finally
{
    if (reader != null) reader.Close();
    con.Close();
}

You can also run more than one SELECT query, split with a semicolon, like this:

string SQL = "SELECT * FROM Customers; SELECT * FROM Orders;"

To differentiate between the two result sets, you need to apply a reader.NextResult();, like below:

while (reader.Read())
{
    // (Process the category rows here.)
}

reader.NextResult();

while (reader.Read())
{
    // (Process the product rows here.)
}

Note: Most of these code examples are from Chapter/Section 5.2.

Conclusion

This has hopefully provided you a basic introduction to using the System.Data.SQLite and ADO.NET. Once you understand how ADO.NET works, System.Data.SQLite drops straight in. It follows all of the usual ADO.NET conventions.

Recently I’ve been doing quite a few REST API building stuffs with Ruby, Sinatra and trying to pick up ActiveRecord to use with Rails. I was however getting quite a lot of problems with Gems loading, but not loading completely.

If when running gem check --alien, you get something similar to below:

rails-1.2.6 has 2 problems
	/Library/Ruby/Gems/1.8/specifications/rails-1.2.6.gemspec:
	Spec file doesn't exist for installed gem

	/Library/Ruby/Gems/1.8/cache/rails-1.2.6.gem:
	missing gem file /Library/Ruby/Gems/1.8/cache/rails-1.2.6.gem

Clear out all of the gems located in the following directories. You will need to similarly remove anything in the bin/, cache/, doc/, gems/ and specifications/ directories.

/Library/Ruby/Gems/1.8
/Users/<username>/.gem/ruby/1.8
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8

(These directories can be found by entering gem environment).

You will then need to reinstall all of your gems. You may wish to issue a gem check on each of them to ensure it’s all good.

The thing is, I’ve never seen the actual point of tablets. They are generally not convenient enough to make them worthwhile, the wrong size, too limited and have an awful input mechanism.

There was an interesting discussion on Hacker News earlier today about tablets in medicine. The article itself suggested that tablets would be a great way for doctors to enter information about patients - forgetting of course that paper has worked wonderfully since pre-Nightingale and such an adaption would be an uphill struggle. Anyway, I digress.

From my rather biased perspective, I think Apple has done something interesting with the “iPad”.

The interesting bit here, at least, I think, is their showing off of iWork on the device. The biggest issue with such a device is text input. Being just a large screen, there’s not the ability to easily enter text. The iPhone isn’t a bad, if a little small, but still a pretty good mobile keypad, the iPad is similar, although on a larger scale.

What I can’t see though is the gap between the iPhone and the MacBook models. Therefore, I don’t think it’s revolutionary, at all.

I find reading to be quite possible on both my iPhone and MacBook Pro, leading me to not need something to fit in that gap. At least, not with just a LCD screen. e-Ink would be something different entirely, reading of an LCD screen is tiring on the eyes.

I can’t see why you would use such a device for writing a document when a laptop is far nicer and why you would use such a device for watching TV, recorded or otherwise.

What I do think is clever though is the price point. It’s not cheap enough that I would buy one - not least at the 3G end of the scale, but priced cleverly enough to provide another option for people looking for a new machine.

Maybe the revolutionary aspect is producing a excellent, portable device?

I intend to see it pan out, I can’t see me buying one, but it’s interesting, none-the-less.

It has certainly been a experience, but something I thought would be overly more strenuous and complex. Especially after the first few weeks. Unfortunately, it has not been like what I had hoped. I certainly don’t find the amount of work I’m given hard enough - or even to the level of other’s at different Universities and the topics covered never seem to go into the depth I’d like. All in all, it either feels like I’m not learning anything or that I am rather having to teach myself.

Arguably this is a failing of the University itself and speaking with others the first year doesn’t appear to change, and some have stated that the second year is not much better.

What I do want to talk about though are ideas. Ideas are those amazing things you dream up doing and in most people’s cases, don’t get the chance to execute them. I used to have a constant flow of such things. Possible projects flying out of my ears just waiting to be executed.

Since getting to Plymouth though, that all seems to have changed. I have ended up having passing possibles, rather than something that sticks for any reasonable amount of time.

Quite possibly this down to not specifically having any problems to solve, or indeed too many options to solve it, neither of them which are worth spending time on.

As an example, I have been meaning to launch something of substance on Heroku for a while. Could do with patching up my Ruby knowledge by writing some client-side apps, rather than simple ideas executed with Sinatra and then abandoned. Similarly, I haven’t used C for a while, would like to learn some Objective-C for Mac development (and Cocoa, obviously) but have no reasons to push ahead with it. No project idea generally means that I merely read and plan about either a project, or something new (like a framework) and not end up with something at the end.

As part of my course, I am supposed to be learning C#, half of my problem here is that I don’t use Windows and therefore I don’t have a need to build anything there. Although, fortunately I seem to be able to pick it up quite easily.

So, instead of spending my time working on something specific I am slowly reading through a mountain of books. To run off a list off the top of my head:

• Learning Ruby
• Learning C# 2005
• Beginning Linux Programming
• RESTful Web Services
• Learn C on the Mac
• and finally: Cocoa Programming for Mac OS X

I do also have a couple of UK-centric business books which I’m slowly attacking, although there is not much point (bar the background reading) to learn about that when I have nothing to execute!

Of course this “lack of ideas” could instead be covering another problem. Instead I may have lost the passion I used to have for simple things that felt worthwhile building. Whether that is a good or bad thing, I’m not sure. Possibly better in the sense that I end up not starting and then stopping projects, but not so great in the sense that in four months I have produced nothing.

Retrospect is a wonderful thing and the last few weeks can be looked back upon with the odd test, only three assignments, and a pretty dodgy sleep pattern for me to look back upon. Don’t get me wrong, University is worth it’s while, however when you feel you are getting nowhere (because it’s too simple), it’s easy to wonder if being somewhere is worth it’s time.

So now, my general week consists of 5/6 lecture slots, sinking my way through some quite heavy books, eventually learning the Guitar and spending just a little too much money. That’s quite a far cry from the last two years. Maybe I shouldn’t complain?

From now on, this is a Git repository made up of Markdown files and a sprinkling of HTML. This comes with quite a few benefits, those of which I hope to explain below.

Versioning & Backup

As it is powered primarily by Git, each post has version tracking without any extra effort. This was recently added to Wordpress, instead of a clean solution, this just turned to make the database even more of a mess than it already was.

Secondly, due to the distributed nature of Git, I always have two copies. One is served up at GitHub, and the other is my working copy stored locally on my main machine. On top of this, other people can fork it, adding more backups.

Security & Access

Without the intention of using this primarily as a vehicle for moaning about Wordpress, this is another of it’s issues. Certainly the two most important issues are security and access to the content.

As is typical with any large dynamic project, it’s going to have security issues. This means that new versions of Wordpress are routinely rolled out, this means that you need to keep up-to-date with the latest build to keep your writings safe.

Next, is access of the posts themselves. They’re stored in a MySQL database. This both means that accessing the individual posts requires accessing the database, rather than just opening a file. Opening a flat HTML file will always scale better than a solution which involves reloading the same information from a database on each connection (presuming no caching is used).

The Source

You can find the source on GitHub. Please feel free to reuse any parts which are not posts without attribution. The posts themselves are licensed under a Creative Commons Attribution-Non-Commercial-Share Alike license. You can read more about that in the README.

On the local machine

1. run: ssh-keygen -t rsa
2. Accept default path.
3. Enter a passphrase. (terminal can save this in Keychain)
4. Finder > Go > Type: “.ssh”
5. Copy id_rsa.pub

On the remote machine

1. run: touch .ssh/authorized_keys
2. Edit the file: .ssh/authorised_keys
3. Paste the contents of the id_rsa.pub file into .ssh/authorized_keys

Permissions Check

• The authorized_keys file needs to be rw for the user.
• That is chmod 600 to change, if needed.

Logging in

• Login as usual from the Terminal.
• When asked for the password to the ssh key, this is the passphrase mentioned earlier.

Once again, thank you, Peter for recording it for me, and the mention.

Yesterday my new MacBook Pro arrived. It’s a 2.66Ghz model with a Matte screen. It really is wonderful. It’s certainly the fastest, and most custom machine I’ve had. Coupled with the Apple Student Discount, it does end up working affordable.

Last night I tested out the power of this machine. There’s various things which I couldn’t do which I now can. Today I took some photos (ones of the Matte screen are rather thin on the ground).

Flash

I can play Flash. Especially HD flash, I rather liked watching the HD shorts on Vimeo but this had to come to a halt a while back as I wasn’t able to use it. It got to the point where I couldn’t even use YouTube, so it’s quite a difference.

Last night I watched some iPlayer content in HD - the difference in it not stuttering is quite different.

Interestingly; Flash in Snow Leopard (could have also been before) is now a separate process. So Flash itself uses ~170% CPU, whilst Safari uses <10%.

Development

Development should be faster. I was having lag of minutes at times using TextMate, especially as I typically develop server side using ExpanDrive to aid with using SFTP. This also meant quite a few things have fallen behind where I’d like them, or I haven’t put the effort in where I should. This really was a reflection of my main development environment rather than me, a pain, but now that bottleneck is gone.

So now, hopefully various things will be given a kick up the backside into getting started, finished or otherwise.

In Conclusion

All in all, everything is great. Things are slotting together ready for my move over to Plymouth for University. So if you’re around, pop me an email and visit!

I was rather surprised to hear myself mentioned, so a huge hat tip (presumes a huge hat, I guess…) to Peter for doing that.

Thanks!

To do this you must use the command “visudo”. Whilst you do not need to use “vi/vim” to do the actual editing (as it will use your preferred editor as listed in .bashrc), you will not be able to save changes.

From here you will see the config file. Under “# User alias specification” you will want to list the users required to access. You can comma separate values here.

# User alias specification
User_Alias STAFF nickcharlton, otheruser

Next, although optional, it is possible to specify the applications that the user can run.

# Cmnd alias specification
Cmnd_Alias DEB = /usr/bin/apt-get

Similarly, this section can be comma separated. On a system where it’s users can be trusted at a higher level, it’s not important to drill down tightly on these.

Next, you need to allow access under the “# User privilege specification”. Where no commands have been specified it is appropriate to simply duplicate that of the root user.

# User privilege specification
root ALL=(ALL) ALL
MAINTAINERS ALL = DEB
STAFF ALL-(ALL) ALL

And that’s it. Another simple task, which at first can seem a little daunting.

Sources: NewbieDoc: Configuring Sudo to Run Programs as Another User

Edit the file under “/etc/motd”.

Change this to something informative such as the below:

========================================
=         Welcome to Hitchcock         =
=  IP: 10.10.10.10                     =
=  Hostname: hitchcock.example.com     =
=  OS: Debian 5.01/Lenny               =
========================================

Just a short one this, but useful to know.

Points

A couple of things to note before going ahead and using SQLite are that:

• The directory and the database need to be writable from the web server.
• As it is simply a flatfile, this should be stored where it cannot be accessed by browsers.
• Where an “auto_increment” field is used in MySQL, it should be created by specifying “INTEGER PRIMARY KEY” when configuring the table. This is further explained later.

Opening a SQLite DB

Opening an SQLite Database is quite simple. The “sqlite_open” function is assigned to a variable which is then used to indicate that database later on in queries. If the database does not already exist then it will be created upon opening.

The file extension need not be .db, it may be anything, or even have none. The directory which is referenced should be both outside of the web root and writable by the server user. On Debian Etch with Apache, the user is “www-data”.

$db = sqlite_open("../db/name.db");

Creating a table

Before data can be written or read to a Database, a table needs to be created to hold it. The the line below runs a CREATE query which describes the table which will be created. Whilst I do not wish to explain how to use SQL in this article, the following creates a table with two fields, one (called “id”) which automatically increments in regards to it’s value and is set as a primary key and the second, called “name” containing text of up to 255 characters. The table itself is called “example”. The SQLite Documentation provides a good guide behind the syntax.

sqlite_query($db, "CREATE TABLE example (id INTEGER PRIMARY KEY, name CHAR(255))");

Inserting data

To insert data, the “sqlite_query” function is used which specifies the database in the first part, and then the query behind it.

The query in this example inserts the name “Nick” into the field “name” in the table “example”. As the field “id” auto increments, it is not necessary to specify a value for it.

sqlite_query($db, "INSERT INTO example (name) VALUES ('Nick')");

Querying data

The following line would fetch all of the data in the table “example” and display it as a “printed array”.

$result = sqlite_query($db, "SELECT * FROM table");
while ($row = sqlite_fetch_array($result)) {
	echo "<pre>";
	print_r($row);
	echo "</pre>";
}

On an alternative note, such a method can be used to see the contents of a query and check the names of fields, at least in development.

Further Reading

Whilst this article focuses on the procedural method built into PHP 5 (which incidentally is limited to SQLite 2), SQLite can also be accessed through PDO and through the Object Orientated Method.

• PHP Manual: SQLite
• Object Orientated SQLite, Devshed
• SQLite 3 with PDO, Zend DevZone

To to this I used a SanDisk 4GB SDHC card which can be bought for around £6/$9. Better performance could be gained from using a faster card. However, for the most part this card is quite acceptable.

Note: Solid state memory (such as SD cards) generally has a limited amount of writes that can be possibly made to it. This means that the card used will not last forever.

Step 1

The first step is to prepare the tools you need and boot from the installation media.

• Ubuntu 9.04 Desktop ISO image
• A CD or thumb drive (to install the image from)
• SanDisk 4GB SDHC Card
• A machine to try it on

Step 2

I chose to first boot into the live environment and run the installer from there. Whilst I have had bad experiences using CD drives for such a procedure, booting off a thumb drive is quite acceptable.

Next, start the install.

Step 3

When you reach the partitioning stage you will need to select the SD card, rather than the hard drive. On my system (an HP (2133) Mini) this appeared as “/dev/sdc” as a SCSI device.

The naming of the device will vary per system, so could show up as hdc (if it is on an IDE controller) or in another manner.

Finally

Once that is finished all that is required is to boot the system. You could set this in the bios to boot first, in which case it’d boot if an external drive is inserted first, or pick at boot time.

Some Notes

• Overall, Ubuntu 9.04 used 2GB (or 50%) of my 2GB drive.
• Issues: When suspending I noticed that it would not be able to mount the disk, and thus fail.

Whilst this does not reflect a model of real world usage, it can aid in the tweaking and performance improvement of the Web Server itself.

This tool is of course aimed at testing Apache servers, however it can be used on others.

Usage

Usage is rather simple. At it’s core, the amount of requests to complete and the url to test are required. On top of this you may inform the application to carry out more than one request at a time, as show in the second example.

   $ ab -n 100 http://domain/

   $ ab -n 100  -c 3 http://domain/

When this is run the specified web page is downloaded and the time taken for it to happen is measured.

Some Results

These are the results I gained from running this against the server behind this, testing using 3 concurrent connections and 100 requests.

$ ab -n 100 -c 3 http://nickcharlton.net/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking nickcharlton.net (be patient).....done


Server Software:        Apache/2.2.3
Server Hostname:        nickcharlton.net
Server Port:            80

Document Path:          /
Document Length:        34350 bytes

Concurrency Level:      3
Time taken for tests:   13.371 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      3471900 bytes
HTML transferred:       3435000 bytes
Requests per second:    7.48 [#/sec] (mean)
Time per request:       401.120 [ms] (mean)
Time per request:       133.707 [ms] (mean, across all concurrent requests)
Transfer rate:          253.58 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       27  135 160.8     87    1295
Processing:    65  257 562.0    140    5108
Waiting:       31  149 169.3    105    1297
Total:        104  392 617.4    254    5483

Percentage of the requests served within a certain time (ms)
  50%    254
  66%    317
  75%    348
  80%    510
  90%    776
  95%    920
  98%   2311
  99%   5483
 100%   5483 (longest request)

Limitations

ApacheBench does not however give you a figure to suggest how many requests a server may complete, or reflect the usage which will be shown with a real set of users. This is because any given page or application may consist of many requests

What to do with the Results?

The results which you gain are specifically useful in tweaking settings regarding the web server itself. Whilst ApacheBench cannot directly tell you what needs tweaking, once you start changing settings you can realise what is best for your server.

My thanks go out to Ante for helping me improve this post, clearing up mistakes and ensuring that the right information was being given. This post has been slightly edited since it’s original posting.

nslookup

nslookup is used to pull up the basic information associated with a domain. Using nslookup is as simple as providing the following arguments:

$ nslookup nickcharlton.org.uk

A likely response could be similar to the following:

Server:		208.67.222.222
Address:	208.67.222.222#53

Non-authoritative answer:
Name:	nickcharlton.org.uk
Address: 78.86.198.208

This response tells you the nameserver which was queried, in this case one of those of OpenDNS which I use at home and the response from the name server when it was queried. The last IP is that of my server.

dig

Dig is a most useful tool, it officially stands for Domain Information Gopher and allows you to pull the data from the name server just as your machine would in taking a request. This allows you to pull the public records for the domain.

$ dig nickcharlton.org.uk

<<>> DiG 9.4.2-P2 <<>> nickcharlton.org.uk
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4635
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;nickcharlton.org.uk.		IN	A

;; ANSWER SECTION:
nickcharlton.org.uk.	603718	IN	A	78.86.198.208

;; Query time: 19 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Sat Feb 21 03:49:25 2009
;; MSG SIZE  rcvd: 53

What can be taken from here is the A record for the domain. That is the record which takes a domain and points it to the IP of the remote machine.

dig (with @IP/Domain)

Whilst not strictly a tool within it’s own right, it is different from it’s basic no arguments call. This allows you to pull records from a specific server, which is useful for testing the records from a new server before it is live. An example is provided below, and would be used if pulling records from a local server:

$ dig @localhost nickcharlton.org.uk

And it’s response:

; <<>> DiG 9.4.2-P2 <<>> @kubrick.nickcharlton.org.uk nickcharlton.org.uk
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63435
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;nickcharlton.org.uk.		IN	A

;; ANSWER SECTION:
nickcharlton.org.uk.	604800	IN	A	78.86.198.208

;; AUTHORITY SECTION:
nickcharlton.org.uk.	604800	IN	NS	ns1.nickcharlton.org.uk.
nickcharlton.org.uk.	604800	IN	NS	ns2.nickcharlton.org.uk.
;; ADDITIONAL SECTION:
ns1.nickcharlton.org.uk. 10800	IN	A	92.243.13.80
ns2.nickcharlton.org.uk. 10800	IN	A	78.86.198.208
;; Query time: 27 msec
;; SERVER: 92.243.13.80#53(92.243.13.80)
;; WHEN: Sat Feb 21 03:50:44 2009
;; MSG SIZE  rcvd: 121

Note the increase in the details of the records. I’m not entirely sure why this happens, so if you could enlighten me, please do. (Also note that the appropriate call I used was towards my server directly, and not local host, as I don’t have a DNS server here.)

rndc

This tool allows you to deal with your current DNS settings. The most commonly use argument of this is “reload” and that allows you to flush, or reload your DNS settings so that they are all fresh.

$ rndc reload

host

Host translates IP’s from domains and vice versa. This is what happens when you query a domain, you get an IP response. It can also be used backwards.

$ host nickcharlton.org.uk

nickcharlton.org.uk has address 78.86.198.208
nickcharlton.org.uk mail is handled by 20 fb.mail.gandi.net.
nickcharlton.org.uk mail is handled by 10 spool.mail.gandi.net.

whois

Last there is whois. This is obviously the most well known of the lot and provides ownership information for the domain. I will not provide an example here, as it will be very long, but this is simply used as below:

$ whois nickcharlton.org.uk

The process for doing this is quite easy and logical, but not so easy to remember.

hdiutil convert -format UDZO Source.sparseimage -o Output.dmg

The process will of course take quite a while, but you will be provided with a simple “..” style progress bar.

hdutil is also SMP aware, so it can use more than one CPU. This not only faster, but will also hammer your machine. With this reason in mind, I’m not sure if it would be wise to run it with a low process number, if you are planning on doing a lot at the same time, maybe this would be a good idea.

To install it under Debian, use:

apt-get install lm-sensors

Although there are other pre-requisites, a default install of Etch will work quite nicely.

Next, you need to load the i2c-dev module. This allows you to access some of the chips on your motherboard which are hooked up to temperature sensors.

modprobe i2c-dev

Now you need to detect the sensors on your system using the wizard in the following command. I just stack to the defaults.

sensors-detect

At the end of this you will be told a list of devices, these need to be loaded too, as an example:

modprobe -a i2c-viapro i2c-isa eeprom w83627hf

By loading the:

sensors

app you will be thrown at with a list of devices, it will also tell you some voltages, however for me the CPU temp, fan sped, and system temperature.

Hard Disk temperature can also be watched, this is slightly easier however as you simply need to install hddtemp and specify the drives.

apt-get install hddtemp

hddtemp /dev/hd?

if your system sees your drives as SCSI, or you have SCSI drives that is…

Produced in aid of: http://www.debian-administration.org/articles/327