Hack Manchester

Last month I attended Hack Manchester – a 24 coding event as part of the Manchester Science Festival, held at MOSI. Having only arranged to team up with Mike, we ended up joining two guys Shaf introduced us to, his colleagues from the BBC, by the names of Jack and Tom. The four of us formed a team, and after browsing the challenges set, we liked the idea of Intechnica‘s Bacon Number problem the most, but rather than just solve the Bacon Number problem, we derived the challenge and set off to build a tool to find the film set with the largest birthday party (most common birthday per film, among actors in the same film).

We decided the data provided was too poorly formatted, and because any alternatives (such as the Open Movie Database) required sign-up and prior approval, we ended up scraping IMDB for the actor birthday data. I wrote a Python script using Beautiful Soup which worked really well, it hit every day page on IMDB and stored each of the actor’s names in a MongoDB collection in the following format:

{
    birthday: '27-10',
    name: 'John Cleese'
}

I ran the script for the 1st Jan page, to see that it worked – and I had a number of records stored, all with ’01-01′ as the date, so it seemed to be working ok. I wrapped a loop around it to hit every day of every month. I started running the script, with no idea how long it would take. I quit it quite early and added a print statement on every new month for indication as to where it was up to. Just how I used to solve most of the number crunching Project Euler problems! I watched it run, and it seemed to take about a minute and a half per month, so it took about 20 mins to run in total (it also crashed out at one point when it got a 500 error from IMDB – I deleted all from the collection from May (incomplete) and ran it from there again, in order not to get duplicates, or miss any out!). Also, I should point out that we were having to run off tethering from my phone, because the Hack Manchester wifi only had 100 IPs to dish out (not ideal for a hackathon with 250 geeks with ~4 devices each!) – a real shame as I’m sure the organisers did all they could to reassure the providers that they would need a lot of connected devices. Quite a lot of data ran through my phone that night – hundreds of hits at IMDB, various packages (such as Beautiful Soup, the Mongo libraries, the IMDB text file data, etc.)

I sanity-checked this data, by looking at the number of records held in each of the dates in the collection. I noticed that 1 January had a significant number more than all the other dates. I assumed I had left the data in from when I initially ran it on 1st Jan to test the script – although it was more than 2, even 3 (actually about 10) times all the other days. I deleted Jan 1st and ran it on that day again, and got the same number. I looked at the IMDB page for 1st Jan and there were genuinely a lot more than for any other day. I asked around my team mates for an idea – someone suggested that people aim for 1st January as a birth date, but I said it’s not distributed among nearby dates, and that didn’t really make sense anyway. Of course (you probably already deduced this – please excuse us – we were tired), it was that 1st January would have been the default value if no date was entered, or maybe this list included actors without a birth date given.

I committed this code at around 1:45am, and about 45 minutes later, while browsing the team’s work on github, I noticed the commit times for some files. The times, given in a friendly time-relative human-readable way were:

30 minutes ago
in 16 minutes
27 minutes ago
an hour ago
5 hours ago

What’s that? I committed the file … in 16 minutes? As in, in the future? How is that so? Well of course, Hack Manchester happened overnight on the day in the year when daylight saving reverts back and we move from BST to GMT, and this happens at 2am, when it goes back to 1am. So every ‘time’ between 01:00 and 01:59 happened twice. I thought this was rather amusing :)

We then had a searchable database of actors and their birthday. Jack whipped up a Twitter Bootstrap web interface, in to which I added some PHP code (using the PHP MongoDB library) to display a list of actors with a given birthday, or show a given actor’s birthday. At this point we have no movies stored, so we had limited functionality.

Meanwhile, Mike had been writing a bunch of PHP classes containing methods for looking up the data. He’d also started writing a Ruby script to extract film-actor data from some text files he found somewhere. He’d had real trouble extracting out the data in a way it would be useful to us. It was tab-separated and had referenced films by random alphanumeric IDs rather than film names, and also contained a ridiculous number of porno films. Later on, Jack and I adapted this code to try to get it to insert the data in to our existing MongoDB collection. It was quite fiddly, and we weren’t really sure how accurately the data was being collated, but worth a try!

At this point we had a discussion about how we would store the data. Someone suggested:

We need another table to store the films, and another to store the film-actor relations

Erm, that’s not how Mongo works. I’m no expert, and my solution may not have been the best, or Mongo-est, but I know you can store lists as values (making multiple ‘tables’, or collections or whatever, unnecessary), so I suggested we would be fine to add a ‘movies’ field to the existing actor storage, which would be a list of films they’d been in, e.g:

{
    birthday: '18-12',
    name: 'Brad Pitt',
    movies: ['Fight Club', 'Inglourious Basterds']
}

We managed to figure out how to add the movie field to an actor, and how to append a movie to list already containing one, and we wrote this in to the script and let it run. We left in a print statement to see what was happening, which obviously slowed the process down a lot. Think about how many movies there are, and think about how many actors there are. Now think about how many instances of an actor being in a movie. That’s a lot. It took bloody ages. And didn’t seem to work. We were out of time by this point (in fact time was almost up when the program started running). Doing this properly we’d have tested it better, and ensured all data was being entered correctly. We were just having a bash at getting it to work.

All of us completely exhausted, we awaited the event closing and awards ceremony. Mike and I had stayed in the museum all night – each attempting a short nap on a couple of occasions, rather unsuccessfully in a room full of geeks bashing away at their respective keyboards. Tom had a prior engagement, so he shot off early evening, and Jack headed home later on due to problems with the wifi, and worked on setting us up an amazon instance to host the project from home.

Among the hackers were many friends of mine – including a team consisting of Michael Heap and Tim Hastings; an MMU team with Farkie; a Manchester Girl Geeks team; a couple of Laterooms team including Mark/Kirsty, Jim & Andy; a Thoughtworks team with Daley, and so on. I had plenty of people to chat to while taking breaks (I drank a lot of coffee) – and met a bunch of new people too.

It came to the closing and the winners of each category was named, and had a chance to give a short demo of their project. Some amazing stuff went on show – it was great to see so much innovation from so many teams. By chance, no-one else had chosen the Bacon Number challenge, so we won by default! A bit lame, I know, but the way I see it is that we weren’t so awful that they decided to withdraw! I count that as a win. And what was the prize? A brand new 512MB Raspberry Pi each! Can’t complain! Huge thanks to Intechnica for the prizes :)

Also a great big thanks to Gemma and Sean for putting the event on. It was fantastic! I will definitely enter events like this in future, even without a team – you can always group up with people and get something done. I was worried about working with people who used different languages or frameworks and that we wouldn’t be able to get things done, but we pooled ideas and skills together and managed to build some cool stuff! Also thanks to MOSI for the use of the space (all through the night!) during the science festival.

The code from our hack is available at github – it may or may not get updated/fixed in future, but at the time of writing was as we left at the end of the event

Also check out Farkie’s blog post on the Magma Digital blog – Hack Manchester 2012

Manchester Raspberry Jam Continued

I’ve now hosted four Raspberry Jams in Manchester. I posted about the first and second, here’s a summary of what went on at III & IV.

The August Manchester Raspberry Jam kicked off when Kat opened up the Madlab and I gave an opening talk about what had been going on in the news in the Raspberry Pi community, featuring announcements like Raspbian and the Gertboard, and showed examples of what people have been doing with their Pis – including Freaky Clown ruinning Metasploit (a pen testing tool) to the Pi and my personal favourite, Pi in the Sky – the first Raspberry Pi to visit near space. I then got people to write down what they wanted to learn, or what they needed help with, and then what they would be able to help others with, and tried to pair or group people together to work on things.

Kat and Alex got straight on with playing Quake on the big TV, which was pretty cool. Meanwhile visiting newcomer Simon – who was shocked when the taxi driver who shuttled him from the station said he’d never heard of the supposedly world famous Madlab – got out his lego-built Pi/breadboard stand and demonstrated his Scratch/Python/GPIO hybrid project which involved a program built in Scratch which would run through the traffic lights sequence on screen when the space bar was pressed, and also feed the same sequence through to the LEDs on his breadboard. Amazing stuff  – if you’re interested in this sort of thing drop him a line on twitter, or take a look at the questions he’s posted to the Raspberry Pi forums to see if you can help.

We also had Paul helping his daughter built a game in Scratch, we saw a Motorola Atrix dock powered by the Pi, we even VNC’d in to a Pi using an Android app on my mobile phone and then from my Nexus 7 tablet! Around 5pm once people had helped my carry my 3 monitors, TV, laptop, netbook and bag full of cables back to my flat, we headed to the pub where we stayed for several hours, and where Simon kept his pink lego Pi kit out on the table. During this time we witnessed HacMan move in to their new hackspace premises just a couple of doors down from Madlab. The phrase “How many geeks does it take to network a hackspace?” was brought up and probably tweeted, as we watched them run a patch cable across the outside wall from Madlab over the café next door. The following day I got my Pi out at home (I never get to play at the Jam – I just end up running around making sure everything’s running smoothly, and in any case all my kit gets handed out and all the screens tend to be in use) and installed OpenElec on a spare SD Card (keeping Raspbian on another), which runs XBMC – a fantastic media centre which I easily managed to use to stream high definition video over the local network from my NAS. I tested it out with X-Men: First Class (720p) which played no problem – looks great on monitor or big TV! I’ve since installed the XBMC Android app on my phone and tablet and I can select something to watch on either device and tell it to play on the TV, then use as a remote control. The future is here.

A month later and we’re back in Madlab for Manchester Raspberry Jam IV – again I begin with a recap of the news: Revision 2.0, Made In the UK and Make Your Own PiOS, and move on to skill share pairing! As usual we get a couple of people who need help setting up their SD card so I volunteered to show them get them up and running, and everybody else just picks something and starts hacking! Simon fires up his pink lego kit again (and eventually ends up blowing it up somehow…), and a newcomer called Ian started work on a project which would allow the Pi to work as Time Machine to back up his Mac!

Later on in the morning a 10 year old kid came up to me and asked if I could teach him some Python (I’d put that up as a sharable skill on the wall) – I said I’d love to, and asked what he wanted to learn, and he said anything to get him started. His Dad sat next to us an observed, saying it reminded him of learning BASIC when he was at school. I started with printing a string, then storing a string in a variable, printing it from its stored value, then storing integers. We then looked at the difference between assignment (x = 1) and comparison (x == 1), and the first idea that came to mind for implementing some of this in a program was checking an entered PIN number against a set value. While showing him the syntax of things like if statements, I let the kid type away, letting him make mistakes if I thought he would learn from the error messages, sometimes asking him what he thought it would do (he would often spot a flaw immediately), sometimes just reminding him to go on to a new line or whatever. But he was doing great. Whenever we completed adding a new feature, I’d ask him what we could do next, so at this point he’d say, “Why don’t we give them 3 chances to get the PIN right?” and we’d look at how to implement that. We’d start by copying the code that asked for the PIN the first time, and sticking it in the ‘else’, so the user would be prompted a second time if (and only if) they got it wrong the first time.

Once we had that working, I introduced the while loop, and explained a simple use case whereby it would count the number of attempts made, and keep running over the same code in the loop until the number of attempts reached 3. Then while adding feedback “Your card is locked” if they were incorrect 3 times, I noted that we could make the number of tries a variable, so that we could change it once at the top and not everywhere in the code. He got that. Then we made it tell you how many attempts you had remaining, by subtracting the number of attempts from the number of attempts allowed. And then he said he thought if he learned any more he might forget what I’d taught him, so he wanted to stop and practise that at home. I emailed the file to his Dad and suggested he might want to try adding something that calculates the balance if they withdrew money.

We finished with a short presentation from a young lad called Barney, who had built a Morse code device with his Dad – you type in a word on the keyboard and the LED on the breadboard flashed according to the Morse codes for the letters in that word. Brilliant! Then we cleared up (and hauled the vast array of monitors to my flat again) and went on to the pub. Everyone always has great fun at the Jams, and everyone’s got something different they want to get out of it. Some just come and take notes and listen to people, some bring their kids to get them excited about computers, some bring crazy contraptions they hope to get working on the Pi with the support of the group, some – like me – just like being there! It’s a great atmosphere and it’s fantastic to see kids and adults playing away, learning and having fun! I don’t generally get much chance to do much myself, but it’s always nice to pass on some python knowledge and help out where possible – and to witness amazing things happening all around me! The Pi is a wonder and we all thank the foundation for all their work in getting it to where it is – and look forward to see where it goes next! Thanks again also to Madlab for hosting the events. I really don’t know how we’d do this if the space wasn’t available.

Manchester Raspberry Jam

I’ve now run two Raspberry Pi events in Manchester, affectionately known as the Raspberry Jam. The first in June, which was the first Raspberry Jam in the UK, and which featured on the Raspberry Pi Foundation’s website where we gained recognition for getting people together to share ideas, demonstrate what we’ve been doing with the Pi, and getting kids interested in building games and writing code as well as inspiring people all over the UK (and the world) to set up their own groups. Continuing the success of the Jam, our second event took place a month later in July, which was equally enjoyable. The July event took place on the same day as the first Cambridge Raspberry Jam – a huge event attended by 300 people and hosted by the Raspberry Pi Foundation at Cambridge University. A number of the attendees from the first Manchester Jam went along to the Cambridge for this, so we look forward to hearing from them about what went on there.

I opened the first event with an introductory talk about my experience with the various computer systems in use in education as I progressed through school. This covered me playing on Granny’s Garden on the BBC Micro and word processing on an Acorn Computer before moving on to Windows 95 PCs, battling with this ‘Internet‘ thing we’d begun to hear about – wondering what the word ‘Yahoo‘ that often appeared was supposed to mean, and my first delivered tech solution at age 11 involving a series of coloured floppy disks.

I kicked things off by initialising a barcamp-style talks grid, and encouraged people to give talks or demos – we had a Python tutorial, talks on Linux for beginners and 3D printing a RasPi case, and a demo of a Raspberry Pi / Arduino hybrid implementation of the game ‘Operation‘ using a live-size manikin! Talks were optional and there was plenty else going on alongside this – with people demonstrating the likes of XBMC on the Pi, chatter over the distributions in use, people learning Python, and most notably Dan Hett from Manchester Game Jam teaching 13 year old girl geek Amy how to get started building games using Scratch – before long she had drawn up some characters for her game, implemented them as sprites in her Pacman clone and started using the drag-and-drop constructs in Scratch to control the actions in the game – it was really impressive how she was thinking like a professional developer – understanding the problems that arose with laying out the behaviour of the objects and their interactions during gameplay. Not only did she understand what these problems were and why they were problems, but she actually thought hard about how she could overcome them, without giving up on it being too difficult. Within a couple of hours or so she had built a Pacman clone with her own drawings as characters and built another game involving flying bats – which I noticed she had instructed to keep track of high scores!

Amy showed me how she had done all this in Scratch (I have not used this before) and it looked to me like lines of programming code – just with each construct in different coloured blocks. I told her if she could read that she would find Python no problem so I asked her if she wanted to watch the Python tutorial – she watched it quietly for about 15 mins, then walked off when the adults started debating over Python 2.7 vs. Python 3.2, and just went back to her Pi and started typing away. She’d remembered all the things shown in the tutorial, and tried to recreate the examples shown. With a little help from me (syntax only) she managed to build a command line based ‘game’ making use of user input and randomly generated numbers. These kids have serious potential. When given the opportunity, they make awesome stuff.

Once we packed everything away I got together with the guys from the Blackpool Linux User Group and we began recording for the Full Circle Podcast – generally covering the events of the day, discussing the potential of the Pi and the future of such events.

The second event was much like the first, except I introduced an idea I saw at another event I attended at MadLab recently – LAMP & Beyond – run by Jeremy of PHPNW / Magma Digital, whereby attendees were asked to contribute post-it notes of skills they had to share and skills they wanted to attain – with the intention of matching groups of people up with a person to lead them in learning a new skill, be it language, development tool, source control package or the latest hipster technology! This worked really well and everyone got a lot out of it. With the Raspberry Jam I hoped this would allow beginners to get started, others to share ideas and help each other with a leg-up without having to figure everything out for themselves and overcome initial issues and get on with something cool. A few people wanted to learn Python so I tutored a few people through some basic constructs of the language compared to what they’re used to in their own languages. Some people needed an OS image loaded on to their SD cards so we had a few people helping them with that, others were making games again (Amy returned to continue with her projects) and some others had the Pi running on an old serial terminal! Towards the end I finally got chance to do something for myself – I got my Pi talking to a breadboard using the PyPi GPIO library! We figured out which pins were which and hooked up an LED with a resistor and controlled its on/off status with a Python command – followed by a few functions (let_there_be_light() and such) which could be called to turn the light on or have it flash. A pretty cool start to external hardware controlled by the Pi. The interesting thing about this ability is that it would be possible to program the Pi to do something and have it do it without use of an external display – just listen to events happen and do things according to the state of those inputs – anything from controlling water sprinkler systems based on soil moisture to … well, anything!

I think it’s fair to say that everyone who comes to one of these events finds something out about the Pi they didn’t know before – the skills sharing nature of the Jam is to be praised and I would encourage anyone of any age or experience to attend one – or start their own!

Update: a few pictures from Jams I & II:

Photos courtesy of Les Pounder [flickr] and Dan Hett

ASCII Bar Charts for Quick & Easy Visualisation

So you have some data. Let’s say it’s a record of the number of instances of some things. Let’s say it’s the number of movies you own, grouped by the year they were released.

Let’s say you have those data in the form of a dictionary in Python, like so:

years = {2000: 2, 2001: 9, 2002: 10, 2003: 9, 2004: 14, 2005: 11, 2006: 8, 2007: 10, 2008: 14, 2009: 19, 2010: 16, 2011: 17}

The following loop will print out an ASCII bar chart for a quick & easy visualisation of these data:

Which looks like this:

Note I used the ‘pipe’ character in this example. First I used ‘o’, which worked well, but I tried a few others ('O','x','X','*','@',':','/','#','[]','+','-','=','_',':)',…) and liked this the most.

That’s the end of what I wanted the blog post to show, but I may as well throw in how I got my data in this case. I have movies saved in a folder, and by convention I name them with the year in brackets at the end so I used glob to loop through the files in this folder, extract the year, and increment the counter in my years dictionary. I have another blog post in draft about using glob to edit filenames in batch. Coming soon.

PHP 5.4 Released

So the other day, Rasmus tweeted that PHP 5.4 was fully released (following several release candidates):

 

There are some great additions, the highlights (other than a huge increase in speed, apparently) being square bracket notation for arrays, array dereferencing and the ability to use traits.

I’m quite excited (sadly) about the use of square brackets to initialise an array, and to be able to code up their contents in this way:

<?php

// PHP 5.3
$arr = array();
$arr2 = array('a','b','c');
$arr3 = array('a' => 2, 'b' => 4, 'c' => 8);
$arr4 = array(array('abc','def'), array(2,4,8), 16);

// PHP 5.4
$arr = [];
$arr2 = ['a','b','c'];
$arr3 = ['a' => 2, 'b' => 4, 'c' => 8];
$arr4 = [['abc','def'], [2,4,8], 16]];

This is similar to the way we do lists in Python.

Array dereferencing means we can now access particular elements within an array upon creating it:

<?php

function makeArray() {
    return [1,2,3];
}

// The PHP 5.3 way
$arr = makeArray();
echo $arr[2];

// The PHP 5.4 way
echo makeArray()[2]; // returns 3

This works the same way with Objects:

There’s now something called Traits, which is a concept brought in to PHP 5.4:

This allows a trait to be reused by any objects which refer to it in this way. It’s to save on copy and pasting blocks of code. The compiler now does that for us!

Also, up to and including PHP 5.3 we could attempt to echo an array without a notice given, but the word ‘Array’ would be echoed instead of any of the array’s content. Now in PHP 5.4 a notice is given:

Although it does still echo the word ‘Array’.

Here’s a great video (the keynote at PHPUK Conference in London) of Rasmus talking about the PHP project from the beginning, and about PHP 5.4: