Planet Linux Australia

Syndicate content
Planet Linux Australia -
Updated: 1 hour 35 min ago

Andre Pang: Movable Type's Export File Format

Tue, 2014-07-08 04:26

Here are a short list of things that possess more elegance than Movable Type’s export file format:

  • XML,
  • SMTP,
  • the C string API,
  • the C multibyte string API (mbsinit, wcrtomb, mbsnrtowcs, etc),
  • the C++ grammar specification,
  • C++ template error messages,
  • the BIND zone file format,
  • Bourne shell parameter expansion involving spaces,
  • PHP,
  • CSV,
  • GNU libtool,
  • wGetGUI,
  • POSIX regular expressions,
  • MPEG-7,
  • the mplayer code base,
  • the Cisco VPN client,
  • the ld(1) manpage on the UNIX system of your choice,
  • the sudoers(5) manpage,
  • Makefiles generated by GNU autogoats,
  • Eric S. Raymond,
  • ICCCM,
  • pretty much everything.

Feel free to extend this list in the comments.

Andre Pang: Merry Christmas!

Tue, 2014-07-08 04:26

It’s been an introspective journey this year. But at least I have a new web site! Woo baby!

Here’s a public thank you to all my family and friends for always being so supportive, and all my workmates at Rising Sun Research and Rising Sun Pictures for a wonderful working environment and teaching me a ton about software development (as well as contributing so much to some awesome movies).

2007 is looking good already. See some of you kids at in January! (You are going, right? If not, why not?)

Merry Christmas everyone, and have a safe, relaxing and happy end-of-year holidays to bring in 2007!

Andre Pang: For the Mac Vim lovers

Tue, 2014-07-08 04:26

Do you like Mac OS X?

Do you like… Vim?

If so, your prayers may just have been answered: see the Vi Input Manager Plugin by Jason Corso. Vi-style key bindings in any Mac OS X text input area? Schyeah baby. As Jason says:

Right now, you should be thinking — “you mean the editor in XCode will behave like Vi?” Answer: Yes.

It’s open source too. Nice work Jason; let the hacking begin!

Andre Pang: Mac OS X Software List Updated

Tue, 2014-07-08 04:26
I've finally updated my Mac OS X software list to be Leopard-aware, for those of you new to Apple's shiny little operating system. Spotting the changes between the older version and newer one is left as an exercise for the reader :-). (Boy I'm glad to have TextExtras working with garbage-collected applications on Leopard!)

Andre Pang: Mac OS X Software for the Uninitiated

Tue, 2014-07-08 04:26

I have a lot of friends who’ve switched to Mac OS X from both Windows and Linux in the past few years. I think it’s a good computing platform (duh, otherwise I wouldn’t be using it), but of course it can take a while to find all those handy little bits of software that make life just a bit easier.

So, since I’m a lazy bastard and got sick of regurgitating my list of Mac OS X software to switcher friends in the past few years, I finally made a Mac OS X Resources page with a list of software that I personally use and think kicks ass. There’s also a (small) collection of hints and tips, including some coding tips for those moving across from Linux. (I’m aware that the coding tips really are quite sparse — I’ll hopefully find some time to expand that in the future.) I hope the resources page is useful for someone else out there: if you do find it useful, a very simple one-line email saying thanks is always appreciated! As Larry Wall would say, have the appropriate amount of fun.

Andre Pang: Mac Developer Roundtable #11

Tue, 2014-07-08 04:26

The Mac Developer Network features an excellent series of podcasts aimed at both veteran Mac developers and those new to the platform who are interested in developing for the Mac. If you’re a current Mac coder and haven’t seen them yet, be sure to check them out. I’ve been listening to the podcasts for a long time, and they’re always both informative and entertaining. (Infotainment, baby.)

Well, in yet another case of “Wow, do I really sound like that?”, I became a guest on The Mac Developer Roundtable episode #11, along with Marcus Zarra, Jonathan Dann, Bill Dudney, and our always-eloquent and delightfully British host, Scotty. The primary topic was Xcode 3.1, but we also chatted about the iPhone NDA (c’mon Apple, lift it already!) and… Fortran. I think I even managed to sneak in the words “Haskell” and “Visual Studio” in there, which no doubt left the other show guests questioning my sanity. I do look forward to Fortran support in Xcode 4.0.

It was actually a small miracle that I managed to be on the show at all. Not only was the podcast recording scheduled at the ungodly time of 4am on a Saturday morning in Australian east-coast time, but I was also in transit from Sydney to the amazing alpine village of Dinner Plain the day before the recording took place. While Dinner Plain is a truly extraordinary village that boasts magnificent ski lodges and some of the best restaurants I’ve ever had the pleasure of eating at, it’s also rather… rural. The resident population is somewhere around 100, the supermarket doesn’t even sell a wine bottle opener that doesn’t suck, and Vodafone has zero phone reception there. So, it was to my great surprise that I could get ADSL hooked up to the lodge there, which was done an entire two days before the recording. Of course, since no ADSL installation ever goes smoothly, I was on the phone to iPrimus tech support1 at 10pm on Friday night, 6 hours before the recording was due to start. All that effort for the privilege of being able to drag my sleepy ass out of bed a few hours later, for the joy of talking to other Mac geeks about our beloved profession. But, I gotta say, being able to hold an international conference call over the Intertubes from a tiny little village at 4am in the morning, when snow is falling all around you… I do love technology.

Of course, since I haven’t actually listened to the episode yet, maybe it’s all a load of bollocks and I sound like a retarded hobbit on speed. Hopefully not, though. Enjoy!

1 Hey, I like Internode and Westnet as much as every other Australian tech geeks, but they didn’t service that area, unfortunately.

Andre Pang: The Designers of Lua on Concurrency

Tue, 2014-07-08 04:26

I came across the following quote about threads while reading The Evolution of Lua:

… we did not (and still do not) believe in the standard multithreading model, which is preemptive concurrency with shared memory: we still think that no one can write correct programs in a language where ‘a=a+1’ is not deterministic.

That is quite possibly the best one-line summary I have read about why shared-state concurrency is evil. Just say no!

Andre Pang: LittleSnapper and Mac Development Talky Talk

Tue, 2014-07-08 04:26

Four little announcements, all of them Mac-related:

First, myself and my comrades at Realmac Software are very proud to announce the release of LittleSnapper 1.0, our swiss-army-knife picture, screenshot and website organisation utility thingamijiggo. We’ve all worked hard on this for the past few months and sweated over a ton of details to try to make it a polished user experience and be a joy to use; we hope you agree. (You would not believe how long we spent figuring out how the blur and highlighting tools should work before they became their final incarnations, or how much pain was involved when we decided to add FTP and SFTP1 support late in the development cycle.) If you’re a Mac user, give it a whirl; it’s a hard program to describe because it has a lot of different workflows, but between the quick annotation tools, easy Web sharing with QuickSnapper/Flickr/SFTP1, website DOM snapping, and the iPhoto-like forget-about-what-folder-you-need-to-put-your-picture-in snapshot management, I’m sure you’ll find something useful for you in there. Hopefully our hard work can make life just a little easier for you!

1 FTP must die.

I blogged earlier that I was speaking at MacDev 2009 in April, but didn’t mention exactly what I was talking about. Well, the talk abstract’s up now:

One reason for Mac OS X’s success is Objective-C, combining the dynamism of a scripting language with the performance of a compiled language. However, how does Objective-C work its magic and what principles is it based upon? In this session, we explore the inner workings of the Objective-C runtime, and see how a little knowledge about programming language foundations—such as lambda calculus and type theory—can go a long way to tackling difficult topics in Cocoa such as error handling and concurrency. We’ll cover a broad range of areas such as garbage collection, blocks, and data structure design, with a focus on practical tips and techniques that can immediately improve your own code’s quality and maintainability.

So, two sections: first, low-level hackery of the Objective-C runtime. Second, a different kind of low-level hackery, and one that’s arguably far more important: understanding the essence of computation and programming languages, and why I fell in love with both Haskell & Objective-C, two languages at completely opposite ends of the planet.

I’d like to point out that while the MacDev registration fee seems fairly expensive at £399, keep in mind that covers your accommodation and also meals, which easily covers £100-£150. Scotty’s done a lot of organising so that you don’t have to. There’s also a Christmas special on at the moment where a few applications are included in the registration price; check the MacDev 2009 website for details.

If you’re an imsoniac and are having trouble sleeping, you’ll hopefully enjoy a recent Late Night Cocoa episode where I talk to Scotty about Garbage Collection. (Actually, you probably won’t enjoy it so much after you find out exactly how -retain & -release are implemented under-the-hood. The words CFBag and “lock” should hopefully scare you enough.) It’s a bit of a long episode at over an hour and a quarter long, but next time I’ll say “um” a bit less which should shorten it to about half an hour. Have fun. And use GC! (LittleSnapper and RapidWeaver both aren’t GC yet, but you bet your ass they will be for the next major versions.)

I’ve had a pretty long exodus away from the fp-syd user group since I was off getting drunk overseas for about four months. That, of course, meant that somehow my brain was rather misplaced when I arrived back in Sydney, so I decided to give a talk at fp-syd upon my return… on the same day that LittleSnapper 1.0 was due to be released, leaving pretty much no margin for error. Oops. I’ll glad to say that the gusto prevailed, and that both the talk seemed to go OK (well, I wasn’t booed off the stage anyway), and LittleSnapper was released on time. (Just; thanks Alan and Danny!) My talk there was similar to the one I gave at Galois in Portland earlier this year: a whirlwind tour of the Objective-C programming language and Mac OS X technologies for a functional programming audience. In particular:

  • basics of the runtime system,
  • higher-order messaging and its analogy to higher-order functions in functional languages,
  • some details on the engineering marvel that is the Objective-C garbage collector, and
  • (updated!) information on Blocks, LLVM and Clang, and a wee tiny bit of info on Grand Central Dispatch and OpenCL.

I’ve updated the talk with a few extra slides, since Apple have made a little more information to the public now. (In particular, brief information on Blocks, Grand Central Dispatch and OpenCL.) Enjoy all!

Andre Pang: Linux Audio and the Paradox of Choice

Tue, 2014-07-08 04:26

Mike Melanson—a primary author of the Linux Flash plugin, xine, ffmpeg, and a general crazy-good multimedia hacker—on the state of Linux Audio APIs:

There are 2 primary methods of sending audio data to a DAC under Linux: OSS and ALSA. OSS came first; ALSA supplanted OSS. Despite this, and as stated above, there are numerous different ways to do the DAC send. There are libraries and frameworks that live at a higher level than OSS and ALSA. In the end, they all just send the data out through OSS or ALSA.

The zaniest part is that some of these higher level libraries can call each other, sometimes in a circular manner. Library A supports sending audio through both OSS and ALSA, and library B does the same. But then library A also has a wrapper to send audio through library B and vice versa. For that matter, OSS and ALSA both have emulation layers for each other. I took the time to map out all of the various libraries that I know of that operate on Linux and are capable of nudging that PCM data out to a DAC:

Barry Schwartz would be shaking his head, methinks. And yes, I’m well aware of efforts to unify this mess. That doesn’t excuse that this jungle has been the state of Linux audio for the past ten years. I love the comments too: instead of admitting how dumbass this is, they give suggestions for using even more APIs (“try KDE4’s Phonon! That’ll fix everything!”)… totally missing the amusing irony, and also missing the point that Mike needs something that works on as many Linux distributions as possible.

Andre Pang: Justice Kirby on Code as Law

Tue, 2014-07-08 04:26

I found this short article about law and code on builderAU rather interesting. The money quotes are the first and last paragraphs:

Technology has outpaced the legal system’s ability to regulate its use in matters of privacy and fair use rights, said Kirby… “We are moving to a point in the world where more and more law will be expressed in its effective way, not in terms of statutes solidly enacted by the parliament… but in the technology itself—code,” said Kirby.

I think that’s a great quote, and it shows that Justice Kirby has a pretty solid understanding of what code is, how it interacts with law, and that the USA’s Digital Millennium Copyright Act (DMCA)—and Australia’s Digital Agenda Act—are dangerous things. (I do think that the builderAU article emphasis on Google and Yahoo being the two culprits seem odd, although it’s hard to say this without listening to Kirby’s original speech.) I’ve always been a fan of Justice Kirby, and it’s nice to know that somebody-on-high understands that code-as-law is a problem, and it’s a complex one.

Andre Pang: John Romero on Apple and NeXT

Tue, 2014-07-08 04:26

John Romero celebrates the ten-year anniversary of when Apple bought over NeXT:

We were building fat binaries of the tools for all 3 processors in the office - one .app file that had code for all 3 processors in it and executed the right code depending on which machine you ran it on. All our data was stored on a Novell 3.11 server and we constantly used the NeXTSTEP Novell gateway object to transparently copy our files to and from the server as if it was a local NTFS drive. This was back in 1993!

Indeed: even the most modern Windows and Linux development environments today still feel far inferior to the NeXT platforms in many ways. I’m thankful that the superb technology from NeXT lives on to this day: when anyone argues that the object-oriented paradigm has failed, Cocoa is living proof that it can work, and deliver an order-of-magnitude improvement in development productivity. It’s humbling to think that most of the libraries and frameworks that Mac OS X developers take for grated today existed a dozen years ago, before even Windows 95 was around. If you’re not convinced, watch Steve Jobs’s demo of NeXTSTEP version 3 running on a 50MHz cube — in 1991. Windows Presentation Foundation, Glade/GTK+ and Qt Designer look like stone-age tools compared to what NeXT had fifteen years ago.

Andre Pang: Jesus on E's MP4s

Tue, 2014-07-08 04:26

For all the oldskool Amiga demoscene folks out there, I’ve weirdly had a bit of nostalgia for the classic Jesus on E’s demo from 1992. It was was understandably not featured on Mindcandy Volume 2, although there are videos of it on the Web floating around. It’s somewhat amusing that the MPEG-4 video is around 120MB when the original version fit on two 880k disks.

So, I chopped up the MPEG-4 videos I found floating around on the Web, and exported the soundtrack to MPEG-4 audio files so I could throw them onto my iPod. The tracks are available at:

If you’re into oldskool techno and rave tracks from the ~1992 era, you can’t beat this stuff. (And if you don’t like oldskool techno and rave tracks, the soundtrack will probably send you completely insane). Have the appropriate amount of fun!

Andre Pang: Add Cmd-1 and Cmd-2 back to iTunes 7

Tue, 2014-07-08 04:26

iTunes 7.0 removed the Cmd-1 and Cmd-2 shortcuts to access the iTunes window and the equaliser, for whatever reason. You can add them back in via the Keyboard preference in System Preferences:

  • launch System Preferences,
  • go to the Keyboard & Mouse preference,
  • click on the Keyboard Shortcuts tab,
  • hit the + button, pick iTunes as the application, type in “Show Equalizer” as the menu title, and use Cmd-2 for the keyboard shortcut.
  • hit the + button, pick iTunes as the application, type in “Hide Equalizer” as the menu title, and use Cmd-2 for the keyboard shortcut.
  • hit the + button, pick iTunes as the application, type in “iTunes” as the menu title, and use Cmd-1 for the keyboard shortcut.


Andre Pang: It Gets Better

Tue, 2014-07-08 04:26

Pixar’s latest short film. I’m so proud and honoured to be working with such an amazing group of people.

Andre Pang: Hacking Your iPhone Headphones

Tue, 2014-07-08 04:26

Perhaps the most annoying feature of the iPhone is its recessed headphones jack:

This miserable little design decision has spawned an entire bloody industry of headphones adapters just so that you can use your own preferred set of headphones with the thing (though admittedly some headphones adapters are way cooler than others). Me, I preferred a slightly cheaper hack. And by hack, I mean hacksaw.


  1. G’day, where’s the cheese?Get a kitchen knife.
  2. Use kitchen knife obtained from step 1 to cut off the end of the rubber sheath at the end of your headphones plug.

Et voilà! Headphones that fit in rather nicely to that stupid jack. It also seems that a few other people have done this as well, but they applied slightly more rigourous methods than me (i.e. they used one of those “proper knives” rather than, say, a kitchen knife).

It's entirely stupid that we have to do this in the first place, but on the bright side, it does solve one of the only major annoyances I've had with the thing, which elevates the iPhone from being "pretty damn good" to "near perfect" in my eyes. One hopes that Apple won't be repeating this particular design decision for their next iPhone revision.

Andre Pang: iPhone: Currency Converter

Tue, 2014-07-08 04:26

A small tip for l’iPhone digerati (aw haw haw haw!): if you, like me, like to look up currency rates, forget about all this Web browser and Web application malarkey. Use the Stocks application instead:

  1. go to the Stocks application,
  2. add a new stock,
  3. use a stock name of AUDUSD=X for the Australian to US dollar, USDGBP=X for USD to the British Pound, etc. (Use the Yahoo! finance page if you don’t know the three-letter currency codes.)

Since a picture is worth a thousand words, here you go:


Andre Pang: Neverwinter Nights 2 Hints and Tips

Tue, 2014-07-08 04:26

I’ve added a games section to the site. There’s some small cheat… uhh, tricks, that you may find handy, some information on crafting items, and also a small patch to remove the XP penalty for multiclassing if you think it’s stupid (which I do). Have fun!

Andre Pang: Interview with Marshall Kirk McKusick

Tue, 2014-07-08 04:26

A website named Neat Little Mac Apps is not the kind of place you’d expect to find an interview with a operating systems and filesystems hacker. Nevertheless, one of their podcasts was just that: an interview with UNIX and BSD legend Marshall Kirk McKusick. (He has his own Wikipedia page; he must be famous!)

There’s some great stuff in there, including the origin of the BSD daemon (Pixar, would you believe? Or, well, Lucasarts at the time…), and a great story about how a bug was introduced into the 4.2 BSD version of the pervasive UNIX diff utility. Marshall’s full of energy, and it’s a great interview; it’s a little amusing to see the stark contrast between the interviewer and McKusick, both of whom have rather different definitions of what constitutes an operating system.

Andre Pang: In-ear Headphones Adventures

Tue, 2014-07-08 04:26

I’ve been on a bit of a headphones splurge lately: anyone who knows me will know that I like my music, and I love my gadgets. I like in-ear headphones for travelling since they’re small, unobtrusive, yet can still give excellent sound quality. While nothing beats a nice pair of full-head cans for comfortable listening, they’re a little bit awkward to tote around. Interestingly, I’ve found that my experiences with in-ear headphones have been quite different from most of the reviews that I’ve read on the ‘net about them, so I thought I’d post my experiences here to bore everyone. The executive summary: make sure that whatever in-ear phones you buy, wherever you buy them, be prepared to bear the cost if you don’t like them, or make sure you can return them.

Note: For comparative purposes, my preferred non-in-ear headphones are the Sennheiser PX-100s (USD$60), IMHO one of the best open earphones in its price range. I don’t consider myself to be an audiophile at all: I think I know sound reasonably well, but I’m not fussy as long as the general reproduction is good. A general guideline is that if I state that something lacks bass compared to something else, you’ll be able to hear the difference unless you’re totally deaf, but I’m not fussy about whether the bass is ‘soft and warm’ and all that other audiophile crap. For serious listening/monitoring, I currently use Beyerdynamic DT 531s, and previously used AKG-141s. The Sennheiser PX-100 nor any of the headphones below are any real comparison to these beasts: the 531s and 141s are meant for professional monitoring with serious comfort; it’s a whole different target market.

So, here are some in-ear headphones that I’ve tried over the years:

  • Apple iPod earbuds: Mediocre sound quality, trendy white, comes with iPod. Stays in the ear reasonably well. Not enough bass for my liking, but that’s no surprise considering they come for free. That said, they’re a hell of a lot better than the earbuds that come with most MP3/CD players; the Sony PSP also came with trendy-white headphones, and they sucked a lot more than the iPod earbuds.

  • Griffin Earjams (USD$15): Designed as an attachment to the iPod earbuds. This doobie significantly changes the output of the earbuds so that you now get a good amount of bass, with very little treble. Not good enough: I like bass in my doof-doof music, but I don’t want to hear just the doof-doof-doof, you know? If you’re tempted to get these, I’d suggest trying Koss’s The Plug or Spark Plug in-ear phones instead, which are just as cheap, have just as much (if not more bass) reproduction, and don’t suffer so badly on the treble end.

  • Apple in-ear Headphones (USD$40): I really liked these. About the only thing I don’t like about them is that they don’t fit in my ear well enough: they’re fine for a couple of minutes, but after about 10 minutes I realise that they’re falling out a bit and I have to wiggle them in again. I don’t find this too much of an annoyance, but no doubt other people will. If you plan to do exercise with your in-ear phones, you will definitely want to give these a very good break in and find out whether they come loose during your gymming/running/skiing/breakdancing/whatnot.

    I’m fairly sure the loose fitting is due to a different design they use: the actual earpiece/driver seems to be much bigger than the other in-ear headphones I’ve tried here, and I guess they’re designed to be used further away from the ear rather than being stuck deep into your ear canal, as with the Sony and Shures I tried (and no doubt like the Etymotics in-ear phones too). The nice thing about this design is that you don’t have to stick them really, really deep, which is the main reason I like them. I find the Sony and the Shures want to be too deep for my comfort level; see below.

  • Sony MDR-EX71SL (USD$49): Available in white or black. After much internal debate and agony, I got the white model. (Trendy or street-wise? Trendy or street-wise? Trendy it is!) This was the first model I bought after the Apple in-ear headphones, expecting to have seriously better sound. After all, everyone knows that for audio, more expensive always means better—especially those gold-plated, deoxygenated, shiny-looking, made-from-unobtanium Monster Cables. Anyway, I was fairly disappointed with their sound quality: less bass than the Apple’s iPod earbuds. Well, OK, let’s be honest: I’m sure they do sound really good if not far better, but I’ll never find out, because I really don’t want to shove them in my ear that far. Unlike the Koss Plug series, these Sonys don’t come with foam earpieces that can be inserted at a comfortable distance, so I think you have to put the earpiece far deeper into your ear than I did. In the very unlikely scenario that I did insert them as far as they were meant to go, they sounded pretty crap indeed.

  • Koss Spark Plug (USD$20): The sequel to Koss’s suspiciously cheap older model, which was simply named “The Plug”. The Spark Plug’s just as cheap as The Plug, but supposedly better (or at least different). For $20, I don’t think you can complain a whole lot: they give you a pretty good amount of bass (comparable to the Griffin Earjams), but they’re definitely lacking at the high end, though not as much as the Earjams are. I find the foam earpieces that come with these fit my ear pretty well, don’t have to be put in that deep to deliver that doof-doof bass I’m looking for, and they’re also reasonably comfortable. They’re not quite as comfortable as Apple’s in-ear headphones, but still, better than I’d expect for a piece of foam stuck into my ear. If your sound-playing device has an equaliser, you can likely correct for the Spark Plug’s lack of treble response by putting it on an Eq setting that boosts the treble a bit: I found the Spark Plug to be sound pretty good on iPod’s Treble Boost or Electronic Eqs. A good, safe (and pretty cheap) buy.

  • Shure E3c/E4c (USD$179 and USD$299): So, I found the Apple Store stocking one of the most acclaimed in-ear phones I’ve ever read about, which would be the Shure E3c. (Note: the ‘c’ in E3c stands for consumer, and basically translates to “trendy iPod white”. The Shure E3 is exactly the same, but is black, and therefore faster.) Additionally, they also had the Shure E4c in stock, which are far more pricey than the E3c (and therefore better). Unfortunately, I put both of these headphones into the same category as the Sony MDR-EX71SLs: I’m sure they’re as absolutely awesome as all the glowing reviews say, but I’m just a wimp and refuse to shove those damn things that far into my brain. Oh yeah, on a not-very-relevant note, Shure changed the packaging between the E3c and the E4c. The E3c had a (gasp) easy-to-open plastic pack, but apparently Shure have since been attending the American School of Plastic Packaging, and used them evil blister packs instead.

  • Griffin EarThumps (USD$20): Released at the start of 2006, I managed to find a pair of EarThumps calling me at my local AppleCentre, and I like them alot. They’re pretty cheap (they’re sure as hell not in the Shure E3c/E4c price range), and they sound great. The bass production is nearly as good as Koss’s Plug series, and they actually manage to produce pretty decent treble too, so you won’t have to play around with your graphic equaliser too much to make these sound decent. Additionally, they appear to have the larger drivers that Apple used for their in-ear headphones: this means that you don’t have to shove them so far in your ear. In fact, these were even easier to get into your ear than Apple’s in-ear headphones, and were much less annoying to put in than the Koss ‘phones. They’re also reasonably comfortable and stay in your ear quite well, and come in both black and white models if colour-matching your iPod is an important thing to you.

For on-the-road listening, my current preferred earphones are the Griffin EarThumps: small enough to fit next to an iPod nano in one of those evilly cute iPod socks, with good enough sound and a good comfort level. I was previously using Koss’s Spark Plug, and before that, Apple’s in-ear headphones. The EarThumps are a set of earphones that I can heartily recommend to friends, since they’re reasonably cheap, and more importantly, I know that they won’t have to shove some piece of equipment halfway inside their brain to get any decent sound out of it.

Andre Pang: Immutability and Blocks, Lambdas and Closures [UPDATE x2]

Tue, 2014-07-08 04:26

I recently ran into some “interesting” behaviour when using lambda in Python. Perhaps it’s only interesting because I learnt lambdas from a functional language background, where I expect that they work a particular way, and the rest of the world that learnt lambdas through Python, Ruby or JavaScript disagree. (Shouts out to you Objective-C developers who are discovering the wonders of blocks, too.) Nevertheless, I thought this would be blog-worthy. Here’s some Python code that shows the behaviour that I found on Stack Overflow:

Since I succumb to reading source code in blog posts by interpreting them as “blah”1, a high-level overview of what that code does is:

  1. iterate over a list of strings,
  2. create a new list of functions that prints out the strings, and then
  3. call those functions, which prints the strings.

Simple, eh? Prints “do”, then “re”, then “mi”, eh? Wrong. It prints out “mi”, then “mi”, then “mi”. Ka-what?

(I’d like to stress that this isn’t a theoretical example. I hit this problem in production code, and boy, it was lots of fun to debug. I hit the solution right away thanks to the wonders of Google and Stack Overflow, but it took me a long time to figure out that something was going wrong at that particular point in the code, and not somewhere else in my logic.)

The second answer to the Stack Overflow question is the clearest exposition of the problem, with a rather clever solution too. I won’t repeat it here since you all know how to follow links. However, while that answer explains the problem, there’s a deeper issue. The inconceivable Manuel Chakravarty provides a far more insightful answer when I emailed him to express my surprise at Python’s lambda semantics:

This is a very awkward semantics for lambdas. It is also probably almost impossible to have a reasonable semantics for lambdas in a language, such as Python.

The behaviour that the person on SO, and I guess you, found surprising is that the contents of the free variables of the lambdas body could change between the point in time where the closure for the lambda was created and when that closure was finally executed. The obvious solution is to put a copy of the value of the variable (instead of a pointer to the original variable) into the closure.

But how about a lambda where a free variable refers to a 100MB object graph? Do you want that to be deep copied by default? If not, you can get the same problem again.

So, the real issue here is the interaction between mutable storage and closures. Sometimes you want the free variables to be copied (so you get their value at closure creation time) and sometimes you don’t want them copied (so you get their value at closure execution time or simply because the value is big and you don’t want to copy it).

And, indeed, since I love being categorised as a massive Apple fanboy, I found the same surprising behaviour with Apple’s blocks semantics in C, too:

You can see the Gist page for this sample code to see how to work around the problem in Objective-C (basically: copy the block), and also to see what it’d look like in Haskell (with the correct behaviour).

In Python, the Stack Overflow solution that I linked to has an extremely concise way of giving the programmer the option to either copy the value or simply maintain a reference to it, and the syntax is clear enough—once you understand what on Earth what the problem is, that is. I don’t understand Ruby or JavaScript well enough to comment on how they’d capture the immediate value for lambdas or whether they considered this design problem. C++0x will, unsurprisingly, give programmers full control over lambda behaviour that will no doubt confuse the hell out of people. (See the C++0x language draft, section 5.1.2 on page 91.)

In his usual incredibly didactic manner, Manuel then went on to explain something else insightful:

I believe there is a deeper issue here. Copying features of FP languages is the hip thing in language design these days. That’s fine, but many of the high-powered FP language features derive their convenience from being unspecific, or at least unconventional, about the execution time of a piece of code. Lambdas delay code execution, laziness provides demand-dependent code execution plus memoisation, continuations capture closures including their environment (ie, the stack), etc. Another instance of that problem was highlighted by Joe Duffy in his STM retrospective.

I would say, mutability and flexible control flow are fundamentally at odds in language design.

Indeed, I’ve been doing some language exploration again lately as the lack of static typing in Python is really beginning to bug me, and almost all the modern languages that attempt to pull functional programming concepts into object-oriented land seem like a complete Frankenstein, partially due to mutability. Language designers, please, this is 2011: multicore computing is the norm now, whether we like it or not. If you’re going to make an imperative language—and that includes all your OO languages—I’ll paraphrase Tim Sweeney: in a concurrent world, mutable is the wrong default! I’d love a C++ or Objective-C where all variables are const by default.

One take-away point from all this is to try to keep your language semantics simple. I love Dan Ingall’s quote from Design Principles Behind Smalltalk: “if a system is to serve the creative spirit, it must be entirely comprehensible to a single individual”. I love Objective-C partially because its message-passing semantics are straightforward, and its runtime has a amazingly compact API and implementation considering how powerful it is. I’ve been using Python for a while now, and I still don’t really know the truly nitty-gritty details about subtle object behaviours (e.g. class variables, multiple inheritance). And I mostly agree with Guido’s assertion that Python should not have included lambda nor reduce, given what Python’s goals are. After discovering this quirk about them, I’m still using the lambda in production code because the code savings does justify the complexity, but you bet your ass there’s a big comment there saying “warning, pretentous code trickery be here!”

1. See point 13 of Knuth et al.’s Mathematical Writing report.

UPDATE: There’s a lot more subtlety at play here than I first realised, and a couple of statements I’ve made above are incorrect. Please see the comments if you want to really figure out what’s going on: I’d summarise the issues, but the interaction between various language semantics are extremely subtle and I fear I’d simply get it wrong again. Thank you to all the commenters for both correcting me and adding a lot of value to this post. (I like this Internet thing! Other people do my work for me!)

Update #2

I’ve been overwhelmed by the comments, in both the workload sense and in the pleasantly-surprised-that-this-provoked-some-discussion sense. Boy, did I get skooled in a couple of areas. I’ve had a couple of requests to try to summarise the issues here, so I’ll do my best to do so.

Retrospective: Python

It’s clear that my misunderstanding of Python’s scoping/namespace rules is the underlying cause of the problem: in Python, variables declared in for/while/if statements will be declared in the compound block’s existing scope, and not create a new scope. So in my example above, using a lambda inside the for loop creates a closure that references the variable m, but m’s value has changed by the end of the for loop to “mi”, which is why it prints “mi, mi, mi”. I’d prefer to link to the official Python documentation about this here rather than writing my own words that may be incorrect, but I can’t actually find anything in the official documentation that authoritatively defines this. I can find a lot of blog posts warning about it—just Google for “Python for while if scoping” to see a few—and I’ve perused the entire chapter on Python’s compound statements, but I just can’t find it. Please let me know in the comments if you do find a link, in which case I’ll retract half this paragraph and stand corrected, and also a little shorter.

I stand by my assertion that Python’s for/while/if scoping is slightly surprising, and for some particular scenarios—like this—it can cause some problems that are very difficult to debug. You may call me a dumbass for bringing assumptions about one language to another, and I will accept my dumbassery award. I will happily admit that this semantics has advantages, such as being able to access the last value assigned in a for loop, or not requiring definitions of variables before executing an if statement that assigns to those variables and using it later in the same scope. All language design decisions have advantages and disadvantages, and I respect Python’s choice here. However, I’ve been using Python for a few years, consider myself to be at least a somewhat competent programmer, and only just found out about this behaviour. I’m surprised 90% of my code actually works as intended given these semantics. In my defence, this behaviour was not mentioned at all in the excellent Python tutorials, and, as mentioned above, I can’t a reference for it in the official Python documentation. I’d expect that this behaviour is enough of a difference vs other languages to at least be mentioned. You may disagree with me and regard this as a minor issue that only shows up when you do crazy foo like use lambda inside a for loop, in which case I’ll shrug my shoulders and go drink another beer.

I’d be interested to see if anyone can come up an equivalent for the “Closures and lexical closures” example at, given another Python scoping rule that assignment to a variable automatically makes it a local variable. (Thus, the necessity for Python’s global keyword.) I’m guessing that you can create the createAdder closure example there with Python’s lambdas, but my brain is pretty bugged out today so I can’t find an equivalent for it right now. You can simply write a callable class to do that and instantiate an object, of course, which I do think is about 1000x clearer. There’s no point using closures when the culture understands objects a ton better, and the resulting code is more maintainable.

Python summary: understand how scoping in for/while/if blocks work, otherwise you’ll run into problems that can cost you hours, and get skooled publicly on the Internet for all your comrades to laugh at. Even with all the language design decisions that I consider weird, I still respect and like Python, and I feel that Guido’s answer to the stuff I was attempting would be “don’t do that”. Writing a callable class in Python is far less tricky than using closures, because a billion times more people understand their semantics. It’s always a design question of whether the extra trickiness is more maintainable or not.

Retrospective: Blocks in C

My C code with blocks failed for a completely different reason unrelated to the Python version, and this was a total beginner’s error with blocks, for which I’m slightly embarrassed. The block was being stack-allocated, so upon exit of the for loop that assigns the function list, the pointers to the blocks are effectively invalid. I was a little unlucky that the program didn’t crash. The correct solution is to perform a Block_copy, in which case things work as expected.

Retrospective: Closures

Not all closures are the same; or, rather, closures are closures, but their semantics can differ from language to language due to many different language design decisions—such as how one chooses to define the lexical environment. Wikipedia’s article on closures has an excellent section on differences in closure semantics.

Retrospective: Mutability

I stand by all my assertions about mutability. This is where the Haskell tribe will nod their collective heads, and all the anti-Haskell tribes think I’m an idiot. Look, I use a lot of languages, and I love and hate many things about each of them, Haskell included. I fought against Haskell for years and hated it until I finally realised that one of its massive benefits is that things bloody well work an unbelievable amount of the time once your code compiles. Don’t underestimate how much of a revelation this is, because that’s the point where the language’s beauty, elegance and crazy type system fade into the background and, for the first time, you see one gigantic pragmatic advantage of Haskell.

One of the things that Haskell does to achieve this is the severe restriction on making things immutable. Apart from the lovely checkbox reason that you can write concurrent-safe algorithms with far less fear, I truly believe that this makes for generally more maintainable code. You can read code and think once about what value a variable holds, rather than keep it in the back of your mind all the time. The human mind is better at keeping track of multiple names, rather than a single name with different states.

The interaction of state and control flow is perhaps the most complex thing to reason about in programming—think concurrency, re-entrancy, disruptive control flow such as longjmp, exceptions, co-routines—and mutability complicates that by an order of magnitude. The subtle difference in behaviour between all the languages discussed in the comments is exemplar that “well-understood” concepts such as lexical scoping, for loops and closures can produce a result that many people still don’t expect; at least for this simple example, these issues would have been avoided altogether if mutability was disallowed. Of course mutability has its place. I’m just advocating that we should restrict it where possible, and at least a smattering of other languages—and hopefully everyone who has to deal with thread-safe code—agrees with me.


I’d truly like to thank everyone who added their voice and spent the time to comment on this post. It’s been highly enlightening, humbling, and has renewed my interest in discussing programming languages again after a long time away from it. And hey, I’m blogging again. (Though perhaps after this post, you may not think that two of those things are good things.) It’s always nice when you learn something new, which I wouldn’t have if not for the excellent peer review. Science: it works, bitches!