Planet Linux Australia

Syndicate content
Planet Linux Australia -
Updated: 57 min 35 sec ago

Linux Users of Victoria (LUV) Announce: LUV Beginners August Meeting : MythTV

Wed, 2014-08-13 22:29
Start: Aug 16 2014 12:30 End: Aug 16 2014 16:30 Start: Aug 16 2014 12:30 End: Aug 16 2014 16:30 Location: 

RMIT Building 91, 110 Victoria Street, Carlton South


MythTV is a free and open source home entertainment application with a simplified "10-foot user interface" design for the living-room TV, and turns a computer with the necessary hardware into a network streaming digital video recorder, a digital multimedia home entertainment system, or home theatre personal computer. It runs on various operating systems, primarily Linux, Mac OS X and FreeBSD.

This introduction to MythTV with live examples, will be presented by LUV Committee member Deb Henry.

LUV would like to acknowledge Red Hat for their help in obtaining the Buzzard Lecture Theatre venue and VPAC for hosting, and BENK Open Systems for their financial support of the Beginners Workshops

Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

August 16, 2014 - 12:30

Michael Still: More turning

Wed, 2014-08-13 17:27
Some more pens, and then I went back to bowls for a bit.

The attraction of pens is that I can churn out a pen in about 30 minutes, whereas a bowl can take twice that. Therefore when I have a small chance to play in the garage I'll do a pen, whereas when I have more time I might do a bowl.


Tags for this post: wood turning 20140718-woodturning photo


Michael Still: I've been making pens

Wed, 2014-08-13 16:27
So, I've been making pens for the last few months. Here are some examples. The first two are my first "commission", in that my brother in law asked for some pens for a friend's farewell gift.


Tags for this post: wood turning 20140628-woodturning photo


Michael Still: Thoughts from the PTL

Wed, 2014-08-13 11:27
I sent this through to the openstack-dev mailing list (you can see the thread here), but I want to put it here as well for people who don't actively follow the mailing list.

First off, thanks for electing me as the Nova PTL for Juno. I find the outcome of the election both flattering and daunting. I'd like to thank Dan and John for running as PTL candidates as well -- I strongly believe that a solid democratic process is part of what makes OpenStack so successful, and that isn't possible without people being will to stand up during the election cycle. I'm hoping to send out regular emails to this list with my thoughts about our current position in the release process. Its early in the cycle, so the ideas here aren't fully formed yet -- however I'd rather get feedback early and often, in case I'm off on the wrong path. What am I thinking about at the moment? The following things: * a mid cycle meetup. I think the Icehouse meetup was a great success, and I'd like to see us do this again in Juno. I'd also like to get the location and venue nailed down as early as possible, so that people who have complex travel approval processes have a chance to get travel sorted out. I think its pretty much a foregone conclusion this meetup will be somewhere in the continental US. If you're interested in hosting a meetup in approximately August, please mail me privately so we can chat. * specs review. The new blueprint process is a work of genius, and I think its already working better than what we've had in previous releases. However, there are a lot of blueprints there in review, and we need to focus on making sure these get looked at sooner rather than later. I'd especially like to encourage operators to take a look at blueprints relevant to their interests. Phil Day from HP has been doing a really good job at this, and I'd like to see more of it. * I promised to look at mentoring newcomers. The first step there is working out how to identify what newcomers to mentor, and who mentors them. There's not a lot of point in mentoring someone who writes a single drive by patch, so working out who to invest in isn't as obvious as it might seem at first. Discussing this process for identifying mentoring targets is a good candidate for a summit session, so have a ponder. However, if you have ideas let's get talking about them now instead of waiting for the summit. * summit session proposals. The deadline for proposing summit sessions for Nova is April 20, which means we only have a little under a week to get that done. So, if you're sitting on a summit session proposal, now is the time to get it in. * business as usual. We also need to find the time for bug fix code review, blueprint implementation code review, bug triage and so forth. Personally, I'm going to focus on bug fix code review more than I have in the past. I'd like to see cores spend 50% of their code review time reviewing bug fixes, to make the Juno release as solid as possible. However, I don't intend to enforce that, its just me asking real nice. Thanks for taking the time to read this email, and please do let me know if you think this sort of communication is useful.

Tags for this post: openstack juno ptl nova

Related posts: Juno Nova PTL Candidacy; Review priorities as we approach juno-3; Havana Nova PTL elections; Expectations of core reviewers; Juno nova mid-cycle meetup summary: ironic; Merged in Havana: fixed ip listing for single hosts


Michael Still: Juno Nova PTL Candidacy

Wed, 2014-08-13 11:27
This is a repost of an email to the openstack-dev list, which is mostly here for historical reasons.

Hi. I would like to run for the OpenStack Compute PTL position as well. I have been an active nova developer since late 2011, and have been a core reviewer for quite a while. I am currently serving on the Technical Committee, where I have recently been spending my time liaising with the board about how to define what software should be able to use the OpenStack trade mark. I've also served on the vulnerability management team, and as nova bug czar in the past. I have extensive experience running Open Source community groups, having served on the TC, been the Director for 2013, as well as serving on the boards of various community groups over the years. In Icehouse I hired a team of nine software engineers who are all working 100% on OpenStack at Rackspace Australia, developed and deployed the turbo hipster third party CI system along with Joshua Hesketh, as well as writing nova code. I recognize that if I am successful I will need to rearrange my work responsibilities, and my management is supportive of that. The future -------------- To be honest, I've thought for a while that the PTL role in OpenStack is poorly named. Specifically, its the T that bothers me. Sure, we need strong technical direction for our programs, but putting it in the title raises technical direction above the other aspects of the job. Compute at the moment is in an interesting position -- we're actually pretty good on technical direction and we're doing interesting things. What we're not doing well on is the social aspects of the PTL role. When I first started hacking on nova I came from an operations background where I hadn't written open source code in quite a while. I feel like I'm reasonably smart, but nova was certainly the largest python project I'd ever seen. I submitted my first patch, and it was rejected -- as it should have been. However, Vishy then took the time to sit down with me and chat about what needed to change, and how to improve the patch. That's really why I'm still involved with OpenStack, Vishy took an interest and was always happy to chat. I'm told by others that they have had similar experiences. I think that's what compute is lacking at the moment. For the last few cycles we're focused on the technical, and now the social aspects are our biggest problem. I think this is a pendulum, and perhaps in a release or two we'll swing back to needing to re-emphasise on technical aspects, but for now we're doing poorly on social things. Some examples: - we're not keeping up with code reviews because we're reviewing the wrong things. We have a high volume of patches which are unlikely to ever land, but we just reject them. So far in the Icehouse cycle we've seen 2,334 patchsets proposed, of which we approved 1,233. Along the way, we needed to review 11,747 revisions. We don't spend enough time working with the proposers to improve the quality of their code so that it will land. Specifically, whilst review comments in gerrit are helpful, we need to identify up and coming contributors and help them build a relationship with a mentor outside gerrit. We can reduce the number of reviews we need to do by improving the quality of initial proposals. - we're not keeping up with bug triage, or worse actually closing bugs. I think part of this is that people want to land their features, but part of it is also that closing bugs is super frustrating at the moment. It can take hours (or days) to replicate and then diagnose a bug. You propose a fix, and then it takes weeks to get reviewed. I'd like to see us tweak the code review process to prioritise bug fixes over new features for the Juno cycle. We should still land features, but we should obsessively track review latency for bug fixes. Compute fails if we're not producing reliable production grade code. - I'd like to see us focus more on consensus building. We're a team after all, and when we argue about solely the technical aspects of a problem we ignore the fact that we're teaching the people involved a behaviour that will continue on. Ultimately if we're not a welcoming project that people want to code on, we'll run out of developers. I personally want to be working on compute in five years, and I want the compute of the future to be a vibrant, friendly, supportive place. We get there by modelling the behaviour we want to see in the future. So, some specific actions I think we should take: - when we reject a review from a relatively new contributor, we should try and pair them up with a more experienced developer to get some coaching. That experienced dev should take point on code reviews for the new person so that they receive low-latency feedback as they learn. Once the experienced dev is ok with a review, nova-core can pile on to actually get the code approved. This will reduce the workload for nova-core (we're only reviewing things which are of a known good standard), while improving the experience for new contributors. - we should obsessively track review performance for bug fixes, and prioritise them where possible. Let's not ignore features, but let's agree that each core should spend at least 50% of their review time reviewing bug fixes. - we should work on consensus building, and tracking the progress of large blueprints. We should not wait until the end of the cycle to re-assess the v3 API and discover we have concerns. We should be talking about progress in the weekly meetings and making sure we're all on the same page. Let's reduce the level of surprise. This also flows into being clearer about the types of patches we don't want to see proposed -- for example, if we think that patches that only change whitespace are a bad idea, then let's document that somewhere so people know before they put a lot of effort in. Thanks for taking the time to read this email!

Tags for this post: openstack juno ptl nova election

Related posts: Havana Nova PTL elections; Review priorities as we approach juno-3; Thoughts from the PTL; Expectations of core reviewers; Juno nova mid-cycle meetup summary: ironic; Merged in Havana: fixed ip listing for single hosts


Michael Still: Expectations of core reviewers

Wed, 2014-08-13 11:27
One of the action items from the nova midcycle was that I was asked to make nova's expectations of core reviews more clear. This blog post is an attempt at that.

Nova expects a minimum level of sustained code reviews from cores. In the past this has been generally held to be in the order of two code reviews a day, which is a pretty low bar compared to the review workload of many cores. I feel that existing cores understand this requirement well, and I am mostly stating it here for completeness.

Additionally, there is increasing levels of concern that cores need to be on the same page about the criteria we hold code to, as well as the overall direction of nova. While the weekly meetings help here, it was agreed that summit attendance is really important to cores. Its the way we decide where we're going for the next cycle, as well as a chance to make sure that people are all pulling in the same direction and trust each other.

There is also a strong preference for midcycle meetup attendance, although I understand that can sometimes be hard to arrange. My stance is that I'd like core's to try to attend, but understand that sometimes people will miss one. In response to the increasing importance of midcycles over time, I commit to trying to get the dates for these events announced further in advance.

Given that we consider these physical events so important, I'd like people to let me know if they have travel funding issues. I can then approach the Foundation about funding travel if that is required.

Tags for this post: openstack juno ptl nova

Related posts: Juno Nova PTL Candidacy; Review priorities as we approach juno-3; Thoughts from the PTL; Havana Nova PTL elections; Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: scheduler


Michael Still: Juno TC Candidacy

Wed, 2014-08-13 11:27
Another email archived for historical reasons.

I'd also like to announce my TC candidacy. I am currently a member of the TC, and I would like to continue to serve. I first started hacking on Nova during the Diablo release, with my first code contributions appearing in the Essex release. Since then I've hacked mostly on Nova and Oslo, although I have also contributed to many other projects as my travels have required. For example, I've tried hard to keep various projects in sync with their imports of parts of Oslo I maintain. I work full time on OpenStack at Rackspace, leading a team of developers who work solely on upstream open source OpenStack. I am a Nova and Oslo core reviewer and the Nova PTL. I have been serving on the TC for the last year, and in the Icehouse release started acting as the liaison for the board "defcore" committee along with Anne Gentle. "defcore" is the board effort to define what parts of OpenStack we require vendors to ship in order to be able to use the OpenStack trade mark, so it involves both the board and the TC. That liaison relationship is very new and only starting to be effective now, so I'd like to keep working on that if you're willing to allow it.

Tags for this post: openstack juno tc election

Related posts: Juno Nova PTL Candidacy; Havana Nova PTL elections


Andrew Pollock: [life] Day 195: Kindergarten, tennis, play date

Tue, 2014-08-12 21:25

Today should have have been more focused on my real estate licence training than it was, but live intruded.

Zoe had a good sleep, almost bang on 11 hours. She was happy but very congested when she woke up. Today was pajama day at Kindergarten, and she was very excited. As I result, I got her to Kindergarten quite quickly.

After I got home, I gave Zoe's bunk bed drawers another coat of sealant and by the time I was done with that, I pretty much had to jump on a bus to the city for a lunch meeting. I got a tiny bit of work done on my next course assessment on the bus.

After my lunch meeting I jumped in a taxi back home to meet with my Thermomix Group Leader to give her my application to become a consultant and then it was time to pick up Zoe from Kindergarten.

I almost got to Kindergarten before I realised that in my haste, I hadn't grabbed a change of clothes for Zoe to do tennis in. Fortunately I'd repacked her spare clothes for Kindergarten that morning, so after some brief dithering, I decided to run with that set of clothes. It worked out okay.

I think with tennis, it's a battle between being hungry after Kindergarten and tired after Kindergarten. Zoe did a better job of being focused, but still was ready to pack it in a little bit before the end of class. Still fighting off a cold can't have helped either. Next week I'll try and remember to bring a quick snack for her to eat between Kindergarten and tennis.

Zoe was desperate for Megan to come over for a play date after tennis. Jason had some stuff to do first, so we came home, and Zoe watched a bit of TV, and then Jason dropped Megan off and dashed off to Bunnings for a bit.

I ended up having dinner ready by the time he was due back, so I suggested they all stay for dinner, which they did, and then they went home afterwards.

Zoe was pretty tired, so I got her to bed a bit early.

Silvia Pfeiffer: Progress with

Tue, 2014-08-12 16:59

At the end of July, I gave a presentation about WebRTC and at the WDCNZ Web Dev Conference in beautiful Wellington, NZ.

Putting that talk together reminded me about how far we have come in the last year both with the progress of WebRTC, its standards and browser implementations, as well as with our own small team at NICTA and our WebRTC toolbox.

One of the most exciting opportunities is still under-exploited: the data channel. When I talked about the above slide and pointed out Bananabread, PeerCDN, Copay, PubNub and also later WebTorrent, that’s where I really started to get Web Developers excited about WebRTC. They can totally see the shift in paradigm to peer-to-peer applications away from the Server-based architecture of the current Web.

Many were also excited to learn more about, our own npm nodules based approach to a JavaScript API for WebRTC.

We believe that the World of JavaScript has reached a critical stage where we can no longer code by copy-and-paste of JavaScript snippets from all over the Web universe. We need a more structured module reuse approach to JavaScript. Node with JavaScript on the back end really only motivated this development. However, we’ve needed it for a long time on the front end, too. One big library (jquery anyone?) that does everything that anyone could ever need on the front-end isn’t going to work any longer with the amount of functionality that we now expect Web applications to support. Just look at the insane growth of npm compared to other module collections:

Packages per day across popular platforms (Shamelessly copied from:

For those that – like myself – found it difficult to understand how to tap into the sheer power of npm modules as a font end developer, simply use browserify. npm modules are prepared following the CommonJS module definition spec. Browserify works natively with that and “compiles” all the dependencies of a npm modules into a single bundle.js file that you can use on the front end through a script tag as you would in plain HTML. You can learn more about browserify and module definitions and how to use browserify.

For those of you not quite ready to dive in with browserify we have prepared prepared the rtc module, which exposes the most commonly used packages of through an “RTC” object from a browserified JavaScript file. You can also directly download the JavaScript file from GitHub.

Using rtc JS library

So, I hope you enjoy and I hope you enjoy my slides and large collection of interesting links inside the deck, and of course: enjoy WebRTC! Thanks to Damon, JEeff, Cathy, Pete and Nathan – you’re an awesome team!

On a side note, I was really excited to meet the author of browserify, James Halliday (@substack) at WDCNZ, whose talk on “building your own tools” seemed to take me back to the times where everything was done on the command-line. I think James is using Node and the Web in a way that would appeal to a Linux Kernel developer. Fascinating!!

Peter Lieverdink: iPhotography Widget

Tue, 2014-08-12 13:26

A while ago, I was contacted by MobileZap, a reseller of mobile phone accessories and asked if I was interested in reviewing an iPhone zoom lens attachment. Unfortunately the widget only attached to the iPhone 5S - mine's a 5c - so I wasn't able to. I also mostly do wide angle photography (landscapes) so a zoom lens would be sort of wasted on me anyway.

When browsing the website, I did stumble across the olloclip wide-angle/fisheye/macro lens kit, which piqued my interest. When I mentioned this, they sent me one and asked me to write a blog about my experiences using it. Happily, I was about to go on a road trip past some very large holes in the ground where it would come it very handly indeed!

The (to give it its full and proper name)  olloclip iPhone 5S / 5 Fisheye, Wide-angle, Macro Lens Kit comes in a fully recyclable plastic and paper package. It includes the phone adapter with lenses, an insert to make the adapter fit iPods and a fabric pouch to keep the lenses free of scratches when not in use. The pouch also doubles as a lens cloth.


First Light: Royal Park

I've been taking an image of Melbourne's CBD once a day (when I'm in the country) from the same spot in Royal Park for close to a year, so I thought I'd start by using the olloclip for the same image:

iPhone 5c standard.


iPhone 5c with olloclip wide angle lens.


iPhone 5c with the olloclip fisheye lens.


Oops, it turns out the lens kit doesn't really fit the iPhone 5c! The adapter is made for the 5s model and the rounded edge on the 5c means it doesn't slide all the way on, so the lens and the camera don't quite align. Mind you, a little bit of image editing to trim this image still results in something useable for blogs and twitter :-)

To give you an idea of the field of view of each of the lens adapters, I've stacked the three images on top of each other at the approximate same size:

Relative sizes of the fields of view of the olloclip wide anfle and fish eye lenses.


Big Things: Road Trip

The main reason I agreed to review this lens kit was to play on the road trip, which was through the south eastern USA, from Los Angeles to Austin. Happily, that included a few choice large holes in the ground subjects for wide angle photography, as well as a friend with a iPhone 5S, on which the olloclip fit just fine.

Cathedral, Sedona, AZ. iPhone 5S with olloclip fish-eye lens.


Barringer Crater, Flagstaff, AZ. iPhone 5S with olloclip fish-eye lens. 


Grand Canyon South Rim, Desert View, AZ. iPhone 5c with olloclip wide angle lens.


As you can see, the olloclip fits just fine on the iPhone 5S - there is no assymmetric distortion like there was in the iPhone 5c fish-eye image.


Small Things: Macro

The wide angle lens consists of two lenses, the top one of which you can unscrew and remove to make a macro lens. I didn't really have anything to take photos of, until a house move left me with a large pile of small change to sort through.

It turns out that some old Australian coins have minting errors or oversights, which make them sought after by collectors. Specifically, the some of the 2 cent coins are missing the designer's initials (S.D.).

Here was a lovely way to try out the macro lens. It works fine as a magnifying glass, too!

Australian 2 cent piece with 'SD' initials (just left of the lizard's toe), iPhone 5c with olloclip macro lens. 


 Australian 2 cent piece without initials, iPhone 5c with olloclip macro lens.


Australian 1917 penny, iPhone 5c with olloclip macro lens.



The macro attachment has come in incredibly handy for close-up images of items to put on eBay. The fact that it doesn't quite fit the iPhone 5c has not been a hindrance, in the way it was for the fish-eye lens.

All in all, I've found the olloclip to be a nifty little attachment and nice to have handy.


Full disclosure

I am not affiliated with olloclip or MobileZap. MobileZap provided me with a free olloclip to review.

Tags: AdvertorialphotographyiPhoneOlloclipiPhotography

Andrew Pollock: [life] Day 194: Kindergarten, random household stuff

Mon, 2014-08-11 21:25

I had grand plans of going for a run this morning before I got stuck into my day. The fact that I don't even remember my alarm going off, or turning it off and then waking up an hour later shows that I'm still not quite over my cold. I did feel better for the sleep in though.

After I got started, I applied the last coat of sealant to the drawers under Zoe's bed (that's a story for another time) and then decided to get stuck into cleaning the front balcony, which looks a bit like a craft shop exploded on it.

I stopped for lunch and then did some more mopping before it was time to pick up Zoe from Kindergarten.

After I picked her up, we popped over to Bunnings to grab some more sealant and a drop cloth, and also dropped past Overflow. I found the sort of cat water thing that I've been wanting for Smudge, and Zoe got all excited about a number 4 sparkler.

I'm also trying to track down a plush soccer ball-sized ball for kicking around the garage downstairs, without luck. We dropped into K Mart and couldn't find one, but Zoe wanted every doll in sight. We bumped into one of Zoe's Kindergarten teachers on the way out of the shopping centre. Zoe's very excited about pajama day tomorrow.

By the time we got home, it was pretty much time to start dinner. I finished off mopping the balcony first, while Zoe watched some TV.

Zoe had a really good dinner tonight. She tends to do pretty well with my Meatless Monday dinners. The zucchini fritters are always popular, and go well in her lunch box the next day too.

She's still a bit congested, but her cough is definitely improving.

Michael Still: devpi as a pip cache

Mon, 2014-08-11 14:27
Richard Jones introduced me to devpi as a pip cache, and its pretty cool. He's written up a guide on his blog at The basic idea is that it caches on demand, so you can have one of these for your home network and avoid downloading things over and over. Even better, it your machines share a linux ABI (think same Ubuntu release or whatever), then you can upload wheels and avoid all those repeated compiles.

Tags for this post: openstack pip cache devpi wheel

Related posts: Faster pip installs; Water, wheels, tyres (tires?) and computers; Wanted: a rear wheel bike computer; Dirty wheel rims


Richard Jones: Review: Lennart Regebro's "Porting to Python 3"

Mon, 2014-08-11 14:25

Lennart Regebro has provided me with a free copy of his self-published book "Porting to Python 3" so that I may review it. He has considerable experience in porting code from Python 2 to Python 3, and it shows in much of his advice and examples. Regebro has assembled an excellent cast of helpers: the technical reviewer Martin von Löwis also has much expertise with Python 3 (he implemented the first port of the popular Django web framework). Brett Canon's introduction also provides a good starting point for someone new to the story of Python 3.

The book's structure is well thought-out. The first chapter immediately invites the reader to make sure they really need to port their code, and what considerations might be taken into account when deciding to do so. The book even has advice, backed up with useful information, for those who are currently unable to port their code but may start preparing for doing so in the future.

The second chapter discusses strategies for moving to Python 3 (view it online). These are presented clearly with one section per strategy. Within each section there are clear references to the other, more detailed chapters of the book which may be used to implement each strategy.

Once you've decided which strategy to apply you can focus on the pertinent remaining chapters. These follow a pretty logical progression:

  • firstly preparing the way by making your code more modern,
  • introducing the 2to3 tool for automatically converting code to Python 3 and the various options for how to use it,
  • talking through some common migration problems and some simple solutions to them,
  • presenting some modern Python (2.6+) idioms that your code may use in Python 2 and 3, and
  • supporting Python 2 and 3 without using the 2to3 tool.

The final two chapters are for more limited audiences and cover migrating C extensions and writing custom 2to3 tool fixers. Even though these are topics which most readers won't need to worry about they are still covered in good detail with minimum fuss. It is in this final chapter that Regebro's experience with porting gives weight to the advice and examples he presents.

Finally there's a couple of pretty comprehensive appendices covering over the main language incompatibilities and library changes which makes it a good reference.

As previously mentioned, the book is very well-organised giving the reader an easy path through the material depending on their situation. Throughout the book there are many pertinent and clear references to software written by others. The book suffers from some typesetting and grammatical errors, though these are minor and none could cause any confusion regarding the content.


There's a lot of information out there for porting from Python 2 to 3, but Regebro has produced a concise, well-organised and complete reference for doing so. It's not skimped on any detail though; the book covers all areas related to moving to Python 3. It's not just about porting code; it's also a handy book for any programmer who's grown up with Python 2 (or 1!) and is looking to move to Python 3.

Richard Jones: devpi quick-start

Mon, 2014-08-11 14:25

(devpi is the caching proxy for PyPI which does a bunch of other things too but mostly just speeds up "pip install" and isolates you from network issues - or complete lack of connectivity)

I gave a 25-minute introduction to devpi at PyCon AU and since then have been requested to put together a quick-start guide:

devpi in 60 seconds

Step 1: create a virtualenv

mkvirtualenv devpi

Step 2: install devpi

pip install devpi

Step 3: run devpi

devpi-server --start

Step 4: use devpi (noting the URL from the previous command output)

devpi use --set-cfg [URL]/root/pypi

Step 5: profit!

pip install yarg

The first installation will call out to the Internet but subsequent installs will use the cached version.


Richard Jones: Using Solaris "priv" with Fabric

Mon, 2014-08-11 14:25

I needed support for "priv" instead of Fabric's built-in "sudo" support. I went through a number of (sometimes quite horrific) iterations before I settled on this relatively simple solution:

import contextlib @contextlib.contextmanager def priv(user): '''Context manager to cause all run()'ed operations to be priv('user')'ed. Replaces with the priv command for the duration of the context. ''' save_shell = = 'priv su - %s -c' % user yield = save_shell

This is then used in a fabfile like so:

with priv('remote_user'): run('do some remote command as remote_user') run('another remote command as remote_user')

Richard Jones: Tips for Testing Twisted

Mon, 2014-08-11 14:25

This post contains some tips for testing Twisted applications. Note that I'm not using trial, the official Twisted test runner, for a variety of reasons.

Twisted Deferreds in Testing

You could use trial, but if you need to use another testing tool (nose, or behave) then you'll need to do a little more work.

To let your deferreds run (and make sure they do finish) wrap your test functions in the "@deferred" decorator from nose.twistedtools:


This will run the reactor for 1 second before declaring it hung.

If you want to see deferreds that are not behaving you will need to enable logging of Twisted events if they're not already enabled:

defer.setDebugging(1) observer = log.PythonLoggingObserver() observer.start()

and then patch nose.twistedtools as follows:

--- original/nose/ +++ installed/nose/ @@ -39,12 +39,19 @@ _twisted_thread = None +_deferreds = [] + def threaded_reactor(): """ Start the Twisted reactor in a separate thread, if not already done. Returns the reactor. The thread will automatically be destroyed when all the tests are done. """ + from twisted.internet import defer + def __init__(self): + _deferreds.append(self) + defer.DebugInfo.__init__ = __init__ + global _twisted_thread try: from twisted.internet import reactor @@ -135,6 +142,8 @@ def errback(failure): # Retrieve and save full exception info try: + if issubclass(failure.type, StandardError): + print failure failure.raiseException() except: q.put(sys.exc_info()) @@ -156,6 +165,9 @@ try: error = q.get(timeout=timeout) except Empty: + for d in _deferreds: + if hasattr(d, 'creator') and not hasattr(d, 'invoker'): + print d._getDebugTracebacks() raise TimeExpired("timeout expired before end of test (%f s.)" % timeout) # Re-raise all exceptions

(I tried a couple of times to figure out how trial does the above but failed miserably)

This will result in a display of all the deferreds still pending when the deferred decorator timeout fires. Note that the deferred decorator introduces a deferred that will not have terminated.

One outstanding problem I've not solved is that inlineCallbacks aren't displayed usefully in the failure debug output - you get told that it is some inline callback that is still hanging, but not where it was created.

Twisted Web and Selenium

Testing Twisted web using Selenium is made more complicated by the need to allow the Twisted reactor to run at the same time as the selenium webdriver is trying to poke at it.

In other applications I've tested like this I just run up the server being tested in a separate thread but I can't do that in the case of Twisted.

To make this work any selenium operation that will cause a Twisted reactor event must be deferred to a thread, hence the liberal sprinkling of deferToThread throughout this code. Some sections do not touch the reactor (for example, filling out a form once the form page has been loaded doesn't), so it can be straight selenium calls.

Note also that the pages are generated in a server dynamically using deferreds so I use a "done" tag marker at the end to indicate to the tests that the page has been fully generated. Selenium's timeout on element access allows me to cause the test to fail if the done marker does not appear.

For example this step function directly from a behave feature implementation:

@then('the message should be sent') @deferred(1) @inlineCallbacks def step(context): # wait until we're all done context.browser.implicitly_wait(1) yield threads.deferToThread(lambda: get_element(context.browser, id="done")) # there should be a result container results = yield threads.deferToThread(lambda: find_elements(context.browser, id="result")) result = results[-1] status = yield threads.deferToThread(lambda: get_element(result, id="status")) # the status could be a number of values depending on synchronisiation of # the tests / runner assert status.text in 'created new accepted done'.split() message = context.db.messages.values()[0] if message.status != message.STATUS_DONE: yield esme._wait_until_done(context)

More tips as I happen to think of them or discover them :-)

Richard Jones: Python Game Programming Challenge (PyWeek) #17 is coming!

Mon, 2014-08-11 14:25

The 17th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 1st to the 8th of September.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

Check out the help page for how to compete and the growing resources message board post.

Richard Jones: Python Game Programming Challenge (PyWeek) #16 is coming!

Mon, 2014-08-11 14:25

The 16th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 14th to the 21st of April.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

Check out the help page for how to compete and the growing resources message board post.

Richard Jones: Python Game Programming Challenge (PyWeek) #15 is coming!

Mon, 2014-08-11 14:25

The 15th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 9th to the 16th of September.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

Check out the help page for how to compete and the growing resources message board post.

Richard Jones: Python Game Programming Challenge (PyWeek) #14 is coming!

Mon, 2014-08-11 14:25

The 14th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 6th to the 13th of May.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

If you're in the US and can make it I'm co-presenting a 3 hour pygame tutorial at PyCon in March.