Planet Linux Australia

Syndicate content
Planet Linux Australia - http://planet.linux.org.au
Updated: 32 min 55 sec ago

Andre Pang: Mac OS X Software for the Uninitiated

Tue, 2014-07-08 04:26

I have a lot of friends who’ve switched to Mac OS X from both Windows and Linux in the past few years. I think it’s a good computing platform (duh, otherwise I wouldn’t be using it), but of course it can take a while to find all those handy little bits of software that make life just a bit easier.

So, since I’m a lazy bastard and got sick of regurgitating my list of Mac OS X software to switcher friends in the past few years, I finally made a Mac OS X Resources page with a list of software that I personally use and think kicks ass. There’s also a (small) collection of hints and tips, including some coding tips for those moving across from Linux. (I’m aware that the coding tips really are quite sparse — I’ll hopefully find some time to expand that in the future.) I hope the resources page is useful for someone else out there: if you do find it useful, a very simple one-line email saying thanks is always appreciated! As Larry Wall would say, have the appropriate amount of fun.

Andre Pang: Mac Developer Roundtable #11

Tue, 2014-07-08 04:26

The Mac Developer Network features an excellent series of podcasts aimed at both veteran Mac developers and those new to the platform who are interested in developing for the Mac. If you’re a current Mac coder and haven’t seen them yet, be sure to check them out. I’ve been listening to the podcasts for a long time, and they’re always both informative and entertaining. (Infotainment, baby.)

Well, in yet another case of “Wow, do I really sound like that?”, I became a guest on The Mac Developer Roundtable episode #11, along with Marcus Zarra, Jonathan Dann, Bill Dudney, and our always-eloquent and delightfully British host, Scotty. The primary topic was Xcode 3.1, but we also chatted about the iPhone NDA (c’mon Apple, lift it already!) and… Fortran. I think I even managed to sneak in the words “Haskell” and “Visual Studio” in there, which no doubt left the other show guests questioning my sanity. I do look forward to Fortran support in Xcode 4.0.

It was actually a small miracle that I managed to be on the show at all. Not only was the podcast recording scheduled at the ungodly time of 4am on a Saturday morning in Australian east-coast time, but I was also in transit from Sydney to the amazing alpine village of Dinner Plain the day before the recording took place. While Dinner Plain is a truly extraordinary village that boasts magnificent ski lodges and some of the best restaurants I’ve ever had the pleasure of eating at, it’s also rather… rural. The resident population is somewhere around 100, the supermarket doesn’t even sell a wine bottle opener that doesn’t suck, and Vodafone has zero phone reception there. So, it was to my great surprise that I could get ADSL hooked up to the lodge there, which was done an entire two days before the recording. Of course, since no ADSL installation ever goes smoothly, I was on the phone to iPrimus tech support1 at 10pm on Friday night, 6 hours before the recording was due to start. All that effort for the privilege of being able to drag my sleepy ass out of bed a few hours later, for the joy of talking to other Mac geeks about our beloved profession. But, I gotta say, being able to hold an international conference call over the Intertubes from a tiny little village at 4am in the morning, when snow is falling all around you… I do love technology.

Of course, since I haven’t actually listened to the episode yet, maybe it’s all a load of bollocks and I sound like a retarded hobbit on speed. Hopefully not, though. Enjoy!

1 Hey, I like Internode and Westnet as much as every other Australian tech geeks, but they didn’t service that area, unfortunately.

Andre Pang: The Designers of Lua on Concurrency

Tue, 2014-07-08 04:26

I came across the following quote about threads while reading The Evolution of Lua:

… we did not (and still do not) believe in the standard multithreading model, which is preemptive concurrency with shared memory: we still think that no one can write correct programs in a language where ‘a=a+1’ is not deterministic.

That is quite possibly the best one-line summary I have read about why shared-state concurrency is evil. Just say no!

Andre Pang: LittleSnapper and Mac Development Talky Talk

Tue, 2014-07-08 04:26

Four little announcements, all of them Mac-related:



First, myself and my comrades at Realmac Software are very proud to announce the release of LittleSnapper 1.0, our swiss-army-knife picture, screenshot and website organisation utility thingamijiggo. We’ve all worked hard on this for the past few months and sweated over a ton of details to try to make it a polished user experience and be a joy to use; we hope you agree. (You would not believe how long we spent figuring out how the blur and highlighting tools should work before they became their final incarnations, or how much pain was involved when we decided to add FTP and SFTP1 support late in the development cycle.) If you’re a Mac user, give it a whirl; it’s a hard program to describe because it has a lot of different workflows, but between the quick annotation tools, easy Web sharing with QuickSnapper/Flickr/SFTP1, website DOM snapping, and the iPhoto-like forget-about-what-folder-you-need-to-put-your-picture-in snapshot management, I’m sure you’ll find something useful for you in there. Hopefully our hard work can make life just a little easier for you!

1 FTP must die.



I blogged earlier that I was speaking at MacDev 2009 in April, but didn’t mention exactly what I was talking about. Well, the talk abstract’s up now:

One reason for Mac OS X’s success is Objective-C, combining the dynamism of a scripting language with the performance of a compiled language. However, how does Objective-C work its magic and what principles is it based upon? In this session, we explore the inner workings of the Objective-C runtime, and see how a little knowledge about programming language foundations—such as lambda calculus and type theory—can go a long way to tackling difficult topics in Cocoa such as error handling and concurrency. We’ll cover a broad range of areas such as garbage collection, blocks, and data structure design, with a focus on practical tips and techniques that can immediately improve your own code’s quality and maintainability.

So, two sections: first, low-level hackery of the Objective-C runtime. Second, a different kind of low-level hackery, and one that’s arguably far more important: understanding the essence of computation and programming languages, and why I fell in love with both Haskell & Objective-C, two languages at completely opposite ends of the planet.

I’d like to point out that while the MacDev registration fee seems fairly expensive at £399, keep in mind that covers your accommodation and also meals, which easily covers £100-£150. Scotty’s done a lot of organising so that you don’t have to. There’s also a Christmas special on at the moment where a few applications are included in the registration price; check the MacDev 2009 website for details.



If you’re an imsoniac and are having trouble sleeping, you’ll hopefully enjoy a recent Late Night Cocoa episode where I talk to Scotty about Garbage Collection. (Actually, you probably won’t enjoy it so much after you find out exactly how -retain & -release are implemented under-the-hood. The words CFBag and “lock” should hopefully scare you enough.) It’s a bit of a long episode at over an hour and a quarter long, but next time I’ll say “um” a bit less which should shorten it to about half an hour. Have fun. And use GC! (LittleSnapper and RapidWeaver both aren’t GC yet, but you bet your ass they will be for the next major versions.)



I’ve had a pretty long exodus away from the fp-syd user group since I was off getting drunk overseas for about four months. That, of course, meant that somehow my brain was rather misplaced when I arrived back in Sydney, so I decided to give a talk at fp-syd upon my return… on the same day that LittleSnapper 1.0 was due to be released, leaving pretty much no margin for error. Oops. I’ll glad to say that the gusto prevailed, and that both the talk seemed to go OK (well, I wasn’t booed off the stage anyway), and LittleSnapper was released on time. (Just; thanks Alan and Danny!) My talk there was similar to the one I gave at Galois in Portland earlier this year: a whirlwind tour of the Objective-C programming language and Mac OS X technologies for a functional programming audience. In particular:

  • basics of the runtime system,
  • higher-order messaging and its analogy to higher-order functions in functional languages,
  • some details on the engineering marvel that is the Objective-C garbage collector, and
  • (updated!) information on Blocks, LLVM and Clang, and a wee tiny bit of info on Grand Central Dispatch and OpenCL.

I’ve updated the talk with a few extra slides, since Apple have made a little more information to the public now. (In particular, brief information on Blocks, Grand Central Dispatch and OpenCL.) Enjoy all!

Andre Pang: Linux Audio and the Paradox of Choice

Tue, 2014-07-08 04:26

Mike Melanson—a primary author of the Linux Flash plugin, xine, ffmpeg, and a general crazy-good multimedia hacker—on the state of Linux Audio APIs:

There are 2 primary methods of sending audio data to a DAC under Linux: OSS and ALSA. OSS came first; ALSA supplanted OSS. Despite this, and as stated above, there are numerous different ways to do the DAC send. There are libraries and frameworks that live at a higher level than OSS and ALSA. In the end, they all just send the data out through OSS or ALSA.

The zaniest part is that some of these higher level libraries can call each other, sometimes in a circular manner. Library A supports sending audio through both OSS and ALSA, and library B does the same. But then library A also has a wrapper to send audio through library B and vice versa. For that matter, OSS and ALSA both have emulation layers for each other. I took the time to map out all of the various libraries that I know of that operate on Linux and are capable of nudging that PCM data out to a DAC:

Barry Schwartz would be shaking his head, methinks. And yes, I’m well aware of efforts to unify this mess. That doesn’t excuse that this jungle has been the state of Linux audio for the past ten years. I love the comments too: instead of admitting how dumbass this is, they give suggestions for using even more APIs (“try KDE4’s Phonon! That’ll fix everything!”)… totally missing the amusing irony, and also missing the point that Mike needs something that works on as many Linux distributions as possible.

Andre Pang: Justice Kirby on Code as Law

Tue, 2014-07-08 04:26

I found this short article about law and code on builderAU rather interesting. The money quotes are the first and last paragraphs:

Technology has outpaced the legal system’s ability to regulate its use in matters of privacy and fair use rights, said Kirby… “We are moving to a point in the world where more and more law will be expressed in its effective way, not in terms of statutes solidly enacted by the parliament… but in the technology itself—code,” said Kirby.

I think that’s a great quote, and it shows that Justice Kirby has a pretty solid understanding of what code is, how it interacts with law, and that the USA’s Digital Millennium Copyright Act (DMCA)—and Australia’s Digital Agenda Act—are dangerous things. (I do think that the builderAU article emphasis on Google and Yahoo being the two culprits seem odd, although it’s hard to say this without listening to Kirby’s original speech.) I’ve always been a fan of Justice Kirby, and it’s nice to know that somebody-on-high understands that code-as-law is a problem, and it’s a complex one.

Andre Pang: John Romero on Apple and NeXT

Tue, 2014-07-08 04:26

John Romero celebrates the ten-year anniversary of when Apple bought over NeXT:

We were building fat binaries of the tools for all 3 processors in the office - one .app file that had code for all 3 processors in it and executed the right code depending on which machine you ran it on. All our data was stored on a Novell 3.11 server and we constantly used the NeXTSTEP Novell gateway object to transparently copy our files to and from the server as if it was a local NTFS drive. This was back in 1993!

Indeed: even the most modern Windows and Linux development environments today still feel far inferior to the NeXT platforms in many ways. I’m thankful that the superb technology from NeXT lives on to this day: when anyone argues that the object-oriented paradigm has failed, Cocoa is living proof that it can work, and deliver an order-of-magnitude improvement in development productivity. It’s humbling to think that most of the libraries and frameworks that Mac OS X developers take for grated today existed a dozen years ago, before even Windows 95 was around. If you’re not convinced, watch Steve Jobs’s demo of NeXTSTEP version 3 running on a 50MHz cube — in 1991. Windows Presentation Foundation, Glade/GTK+ and Qt Designer look like stone-age tools compared to what NeXT had fifteen years ago.

Andre Pang: Jesus on E's MP4s

Tue, 2014-07-08 04:26

For all the oldskool Amiga demoscene folks out there, I’ve weirdly had a bit of nostalgia for the classic Jesus on E’s demo from 1992. It was was understandably not featured on Mindcandy Volume 2, although there are videos of it on the Web floating around. It’s somewhat amusing that the MPEG-4 video is around 120MB when the original version fit on two 880k disks.

So, I chopped up the MPEG-4 videos I found floating around on the Web, and exported the soundtrack to MPEG-4 audio files so I could throw them onto my iPod. The tracks are available at:

If you’re into oldskool techno and rave tracks from the ~1992 era, you can’t beat this stuff. (And if you don’t like oldskool techno and rave tracks, the soundtrack will probably send you completely insane). Have the appropriate amount of fun!

Andre Pang: Add Cmd-1 and Cmd-2 back to iTunes 7

Tue, 2014-07-08 04:26

iTunes 7.0 removed the Cmd-1 and Cmd-2 shortcuts to access the iTunes window and the equaliser, for whatever reason. You can add them back in via the Keyboard preference in System Preferences:

  • launch System Preferences,
  • go to the Keyboard & Mouse preference,
  • click on the Keyboard Shortcuts tab,
  • hit the + button, pick iTunes as the application, type in “Show Equalizer” as the menu title, and use Cmd-2 for the keyboard shortcut.
  • hit the + button, pick iTunes as the application, type in “Hide Equalizer” as the menu title, and use Cmd-2 for the keyboard shortcut.
  • hit the + button, pick iTunes as the application, type in “iTunes” as the menu title, and use Cmd-1 for the keyboard shortcut.

Done.

Andre Pang: It Gets Better

Tue, 2014-07-08 04:26

Pixar’s latest short film. I’m so proud and honoured to be working with such an amazing group of people.

Andre Pang: Hacking Your iPhone Headphones

Tue, 2014-07-08 04:26

Perhaps the most annoying feature of the iPhone is its recessed headphones jack:

This miserable little design decision has spawned an entire bloody industry of headphones adapters just so that you can use your own preferred set of headphones with the thing (though admittedly some headphones adapters are way cooler than others). Me, I preferred a slightly cheaper hack. And by hack, I mean hacksaw.

Process:

  1. G’day, where’s the cheese?Get a kitchen knife.
  2. Use kitchen knife obtained from step 1 to cut off the end of the rubber sheath at the end of your headphones plug.

Et voilà! Headphones that fit in rather nicely to that stupid jack. It also seems that a few other people have done this as well, but they applied slightly more rigourous methods than me (i.e. they used one of those “proper knives” rather than, say, a kitchen knife).





It's entirely stupid that we have to do this in the first place, but on the bright side, it does solve one of the only major annoyances I've had with the thing, which elevates the iPhone from being "pretty damn good" to "near perfect" in my eyes. One hopes that Apple won't be repeating this particular design decision for their next iPhone revision.

Andre Pang: iPhone: Currency Converter

Tue, 2014-07-08 04:26

A small tip for l’iPhone digerati (aw haw haw haw!): if you, like me, like to look up currency rates, forget about all this Web browser and Web application malarkey. Use the Stocks application instead:

  1. go to the Stocks application,
  2. add a new stock,
  3. use a stock name of AUDUSD=X for the Australian to US dollar, USDGBP=X for USD to the British Pound, etc. (Use the Yahoo! finance page if you don’t know the three-letter currency codes.)

Since a picture is worth a thousand words, here you go:

Pretty!

Andre Pang: Neverwinter Nights 2 Hints and Tips

Tue, 2014-07-08 04:26

I’ve added a games section to the site. There’s some small cheat… uhh, tricks, that you may find handy, some information on crafting items, and also a small patch to remove the XP penalty for multiclassing if you think it’s stupid (which I do). Have fun!

Andre Pang: Interview with Marshall Kirk McKusick

Tue, 2014-07-08 04:26

A website named Neat Little Mac Apps is not the kind of place you’d expect to find an interview with a operating systems and filesystems hacker. Nevertheless, one of their podcasts was just that: an interview with UNIX and BSD legend Marshall Kirk McKusick. (He has his own Wikipedia page; he must be famous!)

There’s some great stuff in there, including the origin of the BSD daemon (Pixar, would you believe? Or, well, Lucasarts at the time…), and a great story about how a bug was introduced into the 4.2 BSD version of the pervasive UNIX diff utility. Marshall’s full of energy, and it’s a great interview; it’s a little amusing to see the stark contrast between the interviewer and McKusick, both of whom have rather different definitions of what constitutes an operating system.

Andre Pang: In-ear Headphones Adventures

Tue, 2014-07-08 04:26

I’ve been on a bit of a headphones splurge lately: anyone who knows me will know that I like my music, and I love my gadgets. I like in-ear headphones for travelling since they’re small, unobtrusive, yet can still give excellent sound quality. While nothing beats a nice pair of full-head cans for comfortable listening, they’re a little bit awkward to tote around. Interestingly, I’ve found that my experiences with in-ear headphones have been quite different from most of the reviews that I’ve read on the ‘net about them, so I thought I’d post my experiences here to bore everyone. The executive summary: make sure that whatever in-ear phones you buy, wherever you buy them, be prepared to bear the cost if you don’t like them, or make sure you can return them.

Note: For comparative purposes, my preferred non-in-ear headphones are the Sennheiser PX-100s (USD$60), IMHO one of the best open earphones in its price range. I don’t consider myself to be an audiophile at all: I think I know sound reasonably well, but I’m not fussy as long as the general reproduction is good. A general guideline is that if I state that something lacks bass compared to something else, you’ll be able to hear the difference unless you’re totally deaf, but I’m not fussy about whether the bass is ‘soft and warm’ and all that other audiophile crap. For serious listening/monitoring, I currently use Beyerdynamic DT 531s, and previously used AKG-141s. The Sennheiser PX-100 nor any of the headphones below are any real comparison to these beasts: the 531s and 141s are meant for professional monitoring with serious comfort; it’s a whole different target market.

So, here are some in-ear headphones that I’ve tried over the years:

  • Apple iPod earbuds: Mediocre sound quality, trendy white, comes with iPod. Stays in the ear reasonably well. Not enough bass for my liking, but that’s no surprise considering they come for free. That said, they’re a hell of a lot better than the earbuds that come with most MP3/CD players; the Sony PSP also came with trendy-white headphones, and they sucked a lot more than the iPod earbuds.

  • Griffin Earjams (USD$15): Designed as an attachment to the iPod earbuds. This doobie significantly changes the output of the earbuds so that you now get a good amount of bass, with very little treble. Not good enough: I like bass in my doof-doof music, but I don’t want to hear just the doof-doof-doof, you know? If you’re tempted to get these, I’d suggest trying Koss’s The Plug or Spark Plug in-ear phones instead, which are just as cheap, have just as much (if not more bass) reproduction, and don’t suffer so badly on the treble end.

  • Apple in-ear Headphones (USD$40): I really liked these. About the only thing I don’t like about them is that they don’t fit in my ear well enough: they’re fine for a couple of minutes, but after about 10 minutes I realise that they’re falling out a bit and I have to wiggle them in again. I don’t find this too much of an annoyance, but no doubt other people will. If you plan to do exercise with your in-ear phones, you will definitely want to give these a very good break in and find out whether they come loose during your gymming/running/skiing/breakdancing/whatnot.

    I’m fairly sure the loose fitting is due to a different design they use: the actual earpiece/driver seems to be much bigger than the other in-ear headphones I’ve tried here, and I guess they’re designed to be used further away from the ear rather than being stuck deep into your ear canal, as with the Sony and Shures I tried (and no doubt like the Etymotics in-ear phones too). The nice thing about this design is that you don’t have to stick them really, really deep, which is the main reason I like them. I find the Sony and the Shures want to be too deep for my comfort level; see below.

  • Sony MDR-EX71SL (USD$49): Available in white or black. After much internal debate and agony, I got the white model. (Trendy or street-wise? Trendy or street-wise? Trendy it is!) This was the first model I bought after the Apple in-ear headphones, expecting to have seriously better sound. After all, everyone knows that for audio, more expensive always means better—especially those gold-plated, deoxygenated, shiny-looking, made-from-unobtanium Monster Cables. Anyway, I was fairly disappointed with their sound quality: less bass than the Apple’s iPod earbuds. Well, OK, let’s be honest: I’m sure they do sound really good if not far better, but I’ll never find out, because I really don’t want to shove them in my ear that far. Unlike the Koss Plug series, these Sonys don’t come with foam earpieces that can be inserted at a comfortable distance, so I think you have to put the earpiece far deeper into your ear than I did. In the very unlikely scenario that I did insert them as far as they were meant to go, they sounded pretty crap indeed.

  • Koss Spark Plug (USD$20): The sequel to Koss’s suspiciously cheap older model, which was simply named “The Plug”. The Spark Plug’s just as cheap as The Plug, but supposedly better (or at least different). For $20, I don’t think you can complain a whole lot: they give you a pretty good amount of bass (comparable to the Griffin Earjams), but they’re definitely lacking at the high end, though not as much as the Earjams are. I find the foam earpieces that come with these fit my ear pretty well, don’t have to be put in that deep to deliver that doof-doof bass I’m looking for, and they’re also reasonably comfortable. They’re not quite as comfortable as Apple’s in-ear headphones, but still, better than I’d expect for a piece of foam stuck into my ear. If your sound-playing device has an equaliser, you can likely correct for the Spark Plug’s lack of treble response by putting it on an Eq setting that boosts the treble a bit: I found the Spark Plug to be sound pretty good on iPod’s Treble Boost or Electronic Eqs. A good, safe (and pretty cheap) buy.

  • Shure E3c/E4c (USD$179 and USD$299): So, I found the Apple Store stocking one of the most acclaimed in-ear phones I’ve ever read about, which would be the Shure E3c. (Note: the ‘c’ in E3c stands for consumer, and basically translates to “trendy iPod white”. The Shure E3 is exactly the same, but is black, and therefore faster.) Additionally, they also had the Shure E4c in stock, which are far more pricey than the E3c (and therefore better). Unfortunately, I put both of these headphones into the same category as the Sony MDR-EX71SLs: I’m sure they’re as absolutely awesome as all the glowing reviews say, but I’m just a wimp and refuse to shove those damn things that far into my brain. Oh yeah, on a not-very-relevant note, Shure changed the packaging between the E3c and the E4c. The E3c had a (gasp) easy-to-open plastic pack, but apparently Shure have since been attending the American School of Plastic Packaging, and used them evil blister packs instead.

  • Griffin EarThumps (USD$20): Released at the start of 2006, I managed to find a pair of EarThumps calling me at my local AppleCentre, and I like them alot. They’re pretty cheap (they’re sure as hell not in the Shure E3c/E4c price range), and they sound great. The bass production is nearly as good as Koss’s Plug series, and they actually manage to produce pretty decent treble too, so you won’t have to play around with your graphic equaliser too much to make these sound decent. Additionally, they appear to have the larger drivers that Apple used for their in-ear headphones: this means that you don’t have to shove them so far in your ear. In fact, these were even easier to get into your ear than Apple’s in-ear headphones, and were much less annoying to put in than the Koss ‘phones. They’re also reasonably comfortable and stay in your ear quite well, and come in both black and white models if colour-matching your iPod is an important thing to you.

For on-the-road listening, my current preferred earphones are the Griffin EarThumps: small enough to fit next to an iPod nano in one of those evilly cute iPod socks, with good enough sound and a good comfort level. I was previously using Koss’s Spark Plug, and before that, Apple’s in-ear headphones. The EarThumps are a set of earphones that I can heartily recommend to friends, since they’re reasonably cheap, and more importantly, I know that they won’t have to shove some piece of equipment halfway inside their brain to get any decent sound out of it.

Andre Pang: Immutability and Blocks, Lambdas and Closures [UPDATE x2]

Tue, 2014-07-08 04:26

I recently ran into some “interesting” behaviour when using lambda in Python. Perhaps it’s only interesting because I learnt lambdas from a functional language background, where I expect that they work a particular way, and the rest of the world that learnt lambdas through Python, Ruby or JavaScript disagree. (Shouts out to you Objective-C developers who are discovering the wonders of blocks, too.) Nevertheless, I thought this would be blog-worthy. Here’s some Python code that shows the behaviour that I found on Stack Overflow:

Since I succumb to reading source code in blog posts by interpreting them as “blah”1, a high-level overview of what that code does is:

  1. iterate over a list of strings,
  2. create a new list of functions that prints out the strings, and then
  3. call those functions, which prints the strings.

Simple, eh? Prints “do”, then “re”, then “mi”, eh? Wrong. It prints out “mi”, then “mi”, then “mi”. Ka-what?

(I’d like to stress that this isn’t a theoretical example. I hit this problem in production code, and boy, it was lots of fun to debug. I hit the solution right away thanks to the wonders of Google and Stack Overflow, but it took me a long time to figure out that something was going wrong at that particular point in the code, and not somewhere else in my logic.)

The second answer to the Stack Overflow question is the clearest exposition of the problem, with a rather clever solution too. I won’t repeat it here since you all know how to follow links. However, while that answer explains the problem, there’s a deeper issue. The inconceivable Manuel Chakravarty provides a far more insightful answer when I emailed him to express my surprise at Python’s lambda semantics:

This is a very awkward semantics for lambdas. It is also probably almost impossible to have a reasonable semantics for lambdas in a language, such as Python.

The behaviour that the person on SO, and I guess you, found surprising is that the contents of the free variables of the lambdas body could change between the point in time where the closure for the lambda was created and when that closure was finally executed. The obvious solution is to put a copy of the value of the variable (instead of a pointer to the original variable) into the closure.

But how about a lambda where a free variable refers to a 100MB object graph? Do you want that to be deep copied by default? If not, you can get the same problem again.

So, the real issue here is the interaction between mutable storage and closures. Sometimes you want the free variables to be copied (so you get their value at closure creation time) and sometimes you don’t want them copied (so you get their value at closure execution time or simply because the value is big and you don’t want to copy it).

And, indeed, since I love being categorised as a massive Apple fanboy, I found the same surprising behaviour with Apple’s blocks semantics in C, too:

You can see the Gist page for this sample code to see how to work around the problem in Objective-C (basically: copy the block), and also to see what it’d look like in Haskell (with the correct behaviour).

In Python, the Stack Overflow solution that I linked to has an extremely concise way of giving the programmer the option to either copy the value or simply maintain a reference to it, and the syntax is clear enough—once you understand what on Earth what the problem is, that is. I don’t understand Ruby or JavaScript well enough to comment on how they’d capture the immediate value for lambdas or whether they considered this design problem. C++0x will, unsurprisingly, give programmers full control over lambda behaviour that will no doubt confuse the hell out of people. (See the C++0x language draft, section 5.1.2 on page 91.)

In his usual incredibly didactic manner, Manuel then went on to explain something else insightful:

I believe there is a deeper issue here. Copying features of FP languages is the hip thing in language design these days. That’s fine, but many of the high-powered FP language features derive their convenience from being unspecific, or at least unconventional, about the execution time of a piece of code. Lambdas delay code execution, laziness provides demand-dependent code execution plus memoisation, continuations capture closures including their environment (ie, the stack), etc. Another instance of that problem was highlighted by Joe Duffy in his STM retrospective.

I would say, mutability and flexible control flow are fundamentally at odds in language design.

Indeed, I’ve been doing some language exploration again lately as the lack of static typing in Python is really beginning to bug me, and almost all the modern languages that attempt to pull functional programming concepts into object-oriented land seem like a complete Frankenstein, partially due to mutability. Language designers, please, this is 2011: multicore computing is the norm now, whether we like it or not. If you’re going to make an imperative language—and that includes all your OO languages—I’ll paraphrase Tim Sweeney: in a concurrent world, mutable is the wrong default! I’d love a C++ or Objective-C where all variables are const by default.

One take-away point from all this is to try to keep your language semantics simple. I love Dan Ingall’s quote from Design Principles Behind Smalltalk: “if a system is to serve the creative spirit, it must be entirely comprehensible to a single individual”. I love Objective-C partially because its message-passing semantics are straightforward, and its runtime has a amazingly compact API and implementation considering how powerful it is. I’ve been using Python for a while now, and I still don’t really know the truly nitty-gritty details about subtle object behaviours (e.g. class variables, multiple inheritance). And I mostly agree with Guido’s assertion that Python should not have included lambda nor reduce, given what Python’s goals are. After discovering this quirk about them, I’m still using the lambda in production code because the code savings does justify the complexity, but you bet your ass there’s a big comment there saying “warning, pretentous code trickery be here!”

1. See point 13 of Knuth et al.’s Mathematical Writing report.

UPDATE: There’s a lot more subtlety at play here than I first realised, and a couple of statements I’ve made above are incorrect. Please see the comments if you want to really figure out what’s going on: I’d summarise the issues, but the interaction between various language semantics are extremely subtle and I fear I’d simply get it wrong again. Thank you to all the commenters for both correcting me and adding a lot of value to this post. (I like this Internet thing! Other people do my work for me!)

Update #2

I’ve been overwhelmed by the comments, in both the workload sense and in the pleasantly-surprised-that-this-provoked-some-discussion sense. Boy, did I get skooled in a couple of areas. I’ve had a couple of requests to try to summarise the issues here, so I’ll do my best to do so.

Retrospective: Python

It’s clear that my misunderstanding of Python’s scoping/namespace rules is the underlying cause of the problem: in Python, variables declared in for/while/if statements will be declared in the compound block’s existing scope, and not create a new scope. So in my example above, using a lambda inside the for loop creates a closure that references the variable m, but m’s value has changed by the end of the for loop to “mi”, which is why it prints “mi, mi, mi”. I’d prefer to link to the official Python documentation about this here rather than writing my own words that may be incorrect, but I can’t actually find anything in the official documentation that authoritatively defines this. I can find a lot of blog posts warning about it—just Google for “Python for while if scoping” to see a few—and I’ve perused the entire chapter on Python’s compound statements, but I just can’t find it. Please let me know in the comments if you do find a link, in which case I’ll retract half this paragraph and stand corrected, and also a little shorter.

I stand by my assertion that Python’s for/while/if scoping is slightly surprising, and for some particular scenarios—like this—it can cause some problems that are very difficult to debug. You may call me a dumbass for bringing assumptions about one language to another, and I will accept my dumbassery award. I will happily admit that this semantics has advantages, such as being able to access the last value assigned in a for loop, or not requiring definitions of variables before executing an if statement that assigns to those variables and using it later in the same scope. All language design decisions have advantages and disadvantages, and I respect Python’s choice here. However, I’ve been using Python for a few years, consider myself to be at least a somewhat competent programmer, and only just found out about this behaviour. I’m surprised 90% of my code actually works as intended given these semantics. In my defence, this behaviour was not mentioned at all in the excellent Python tutorials, and, as mentioned above, I can’t a reference for it in the official Python documentation. I’d expect that this behaviour is enough of a difference vs other languages to at least be mentioned. You may disagree with me and regard this as a minor issue that only shows up when you do crazy foo like use lambda inside a for loop, in which case I’ll shrug my shoulders and go drink another beer.

I’d be interested to see if anyone can come up an equivalent for the “Closures and lexical closures” example at http://c2.com/cgi/wiki?ScopeAndClosures, given another Python scoping rule that assignment to a variable automatically makes it a local variable. (Thus, the necessity for Python’s global keyword.) I’m guessing that you can create the createAdder closure example there with Python’s lambdas, but my brain is pretty bugged out today so I can’t find an equivalent for it right now. You can simply write a callable class to do that and instantiate an object, of course, which I do think is about 1000x clearer. There’s no point using closures when the culture understands objects a ton better, and the resulting code is more maintainable.

Python summary: understand how scoping in for/while/if blocks work, otherwise you’ll run into problems that can cost you hours, and get skooled publicly on the Internet for all your comrades to laugh at. Even with all the language design decisions that I consider weird, I still respect and like Python, and I feel that Guido’s answer to the stuff I was attempting would be “don’t do that”. Writing a callable class in Python is far less tricky than using closures, because a billion times more people understand their semantics. It’s always a design question of whether the extra trickiness is more maintainable or not.

Retrospective: Blocks in C

My C code with blocks failed for a completely different reason unrelated to the Python version, and this was a total beginner’s error with blocks, for which I’m slightly embarrassed. The block was being stack-allocated, so upon exit of the for loop that assigns the function list, the pointers to the blocks are effectively invalid. I was a little unlucky that the program didn’t crash. The correct solution is to perform a Block_copy, in which case things work as expected.

Retrospective: Closures

Not all closures are the same; or, rather, closures are closures, but their semantics can differ from language to language due to many different language design decisions—such as how one chooses to define the lexical environment. Wikipedia’s article on closures has an excellent section on differences in closure semantics.

Retrospective: Mutability

I stand by all my assertions about mutability. This is where the Haskell tribe will nod their collective heads, and all the anti-Haskell tribes think I’m an idiot. Look, I use a lot of languages, and I love and hate many things about each of them, Haskell included. I fought against Haskell for years and hated it until I finally realised that one of its massive benefits is that things bloody well work an unbelievable amount of the time once your code compiles. Don’t underestimate how much of a revelation this is, because that’s the point where the language’s beauty, elegance and crazy type system fade into the background and, for the first time, you see one gigantic pragmatic advantage of Haskell.

One of the things that Haskell does to achieve this is the severe restriction on making things immutable. Apart from the lovely checkbox reason that you can write concurrent-safe algorithms with far less fear, I truly believe that this makes for generally more maintainable code. You can read code and think once about what value a variable holds, rather than keep it in the back of your mind all the time. The human mind is better at keeping track of multiple names, rather than a single name with different states.

The interaction of state and control flow is perhaps the most complex thing to reason about in programming—think concurrency, re-entrancy, disruptive control flow such as longjmp, exceptions, co-routines—and mutability complicates that by an order of magnitude. The subtle difference in behaviour between all the languages discussed in the comments is exemplar that “well-understood” concepts such as lexical scoping, for loops and closures can produce a result that many people still don’t expect; at least for this simple example, these issues would have been avoided altogether if mutability was disallowed. Of course mutability has its place. I’m just advocating that we should restrict it where possible, and at least a smattering of other languages—and hopefully everyone who has to deal with thread-safe code—agrees with me.

Closing

I’d truly like to thank everyone who added their voice and spent the time to comment on this post. It’s been highly enlightening, humbling, and has renewed my interest in discussing programming languages again after a long time away from it. And hey, I’m blogging again. (Though perhaps after this post, you may not think that two of those things are good things.) It’s always nice when you learn something new, which I wouldn’t have if not for the excellent peer review. Science: it works, bitches!

Andre Pang: Google Talk and Jabber vs the Rest

Tue, 2014-07-08 04:26

I have a lot of friends on the MSN Messenger chat network that I quite enjoy talking to, but unfortunately my favourite instant messaging program — iChat on Mac OS X — can’t talk to MSN. (Linux people, keep reading, I promise this post will get more relevant to you despite my reference to iChat.) iChat, can, however, talk to Jabber servers. I’ve known for a while that many Jabber servers have the capability to bridge networks, so that you can talk to people on all the other chatting networks (ICQ, Yahoo!, AIM, MSN, and Google Talk) by simply logging into a single Jabber server. However, I never bothered configuring iChat to do this until today.

I’m very happy to say that one of the first guides I found on Google about how to set up iChat to talk to MSN was on the Jabber Australia site, which is also one of the most professional Web sites I’ve seen for a free, open service. The process was quite painless and nearly completely Web-based: the Jabber Australia site imports all your current MSN contacts into your new Jabber contacts list, so you don’t have to re-type them all in. The Jabber<->MSN bridge even sends across all your buddies’ icons, so I can see iChat proudly displaying pictures of all my mates. Very schmick.

Step the second: I also have friends on Google Talk, and since Google opened up their servers to Jabber federation in January 2006, that means I also get to chat with all my friends who have Gmail accounts and never bother logging into the IM networks. (Muahaha, and they thought they could avoid me!) Now I’m finally gotten into the 2000 era and have one chat client that can talk to any of MSN, ICQ, Yahoo!, Google Talk, and AIM; woohoo, clever me.

However, the story’s much bigger than just simple chat network interoperability. Google’s move into the IM market by unleashing Google Talk might have seemed rather underwhelming when it was first announced: after all, it was just like Skype… but Google voice chat was Windows-only when Skype is Windows/Mac/Linux, and oh yeah, it also had, like, none of Skype’s user base. However, Google Talk has one massive weapon behind it: open standards. In the past few months, Google has done two very significant things:

  • They’ve opened up their servers so that they can interoperate with other Jabber servers.
  • Published their voice-call protocol as an open standard, and even provided third-party developers with a library (libjingle) that can be integrated into any IM client.

The second point is huge: in one move, Google has brought voice capability to the entire Jabber federation chat network. (And, if you haven’t used Google Talk, the voice quality is damn good: better than Skype, and possibly on par or better than iChat AV.) The implication of this is that there’s going to be a big split in the short-term between the official ICQ/Yahoo!/AIM/MSN clients and everyone else (i.e. Trillian, Gaim, AdiumX, etc). The official clients will, of course, only work with their own network since they want to lock you in, but every other IM client that doesn’t currently support voice chatting — which means everybody except for Skype and iChat AV — is very, very likely to be putting Google’s voice protocols into their own chat clients. Look a couple of years ahead, and I think you’ll find that every IM chat client is going to have voice support, and that they’ll be divided into two camps: the ones that support Google’s voice protocol because it’s an open standard, and everybody else.

The thing is, right now, that “everybody else” is really only one other group: Skype. There’s also iChat AV, but that’s small fry compared to Skype, and since Apple piggybacks off the AIM network right now, they don’t have a large interest in locking customers into one particular network. (It’d be relatively easy for Apple to migrate all their .Mac accounts over to a Jabber-federated network just by throwing a couple of Jabber servers up for their .Mac users and publishing a new version of iChat that talked to that.) This means that it’s more-or-less going to be Skype vs Google Talk in the next coming years, and while Skype has absolutely huge mindshare right now, I don’t think they have a hope of holding out, because they’re the only damn network right now that absolutely requires you to use their own client. The one killer advantage that Skype has compared to Google Talk is that you can use Skype call-out and call-in to do phone calls, but once Google gets SIP support into Jingle, Google Talk will have that capability as well. Unless Skype do something radical, they’re going to be extinct in a few years as developers start pushing Jingle support into every single IM client.

Heh, not a bad situation at all: in one move, Google’s not only guaranteed some measure of success for themselves in the IM market, but they’ve also made the world a slightly better place by giving users client choice and software choice thanks to open standards. One voice protocol to rule them all, and in the darkness bind them…

Andre Pang: Thoughts to Kathy Sierra

Tue, 2014-07-08 04:26

For those of you who were fortunate enough to see the magnificent Kathy Sierra keynote at Linux.conf.au this year but don’t read her blog, she’s received death threats and sex threats from some anonymous bloggers and comments. It was serious enough that she cancelled a presentation at an upcoming conference, and the police have been informed.

Wikipedia has some updated information on her harassment. Dave Winer, in a remarkably objective post, reckons it’s just a bunch of trolls and that those kind of death threats are nothing new. I think it’s a little too early to tell yet exactly what the hell is going on, but even if it is “just some trolls”, it’s still outrageous behaviour. Be sure to also read her update on the situation if you’re checking out the other links.

Lesson learned: don’t start a Web site that encourages abusive behaviour unless you’re prepared to deal with the consequences. In fact, just don’t start a Web site that encourages abusive behaviour at all. As Kathy herself says, angry and negative people can be bad for you. I wonder whether it was that article that triggered off some power-hungry kid’s frontal lobe.

Godspeed, Kathy. The world needs more people like you. Hopefully whoever made those comments will be punished—and redeem themselves.

Andre Pang: Gmail Losing Emails?

Tue, 2014-07-08 04:26

Like a lot of other folks, I’d switched over to Gmail my primary email account. Their Web interface is great, and Gmail’s spam filtering is possibly the best I’d ever seen. However, there was one rather small problem, where by “rather small problem” I mean “kinda huge problem”: I was losing emails. As in, I know the person had sent me emails, and I never received them. Yep, I checked the spam folder. The emails never showed up1.

I was willing to forgive this once or twice, seeing as how complicated and fragile all these SMTP shenanigans is. (Seriously, SMTP has to contend with FTP for the Most Stupid Protocol Ever Award.) However, after Gmail lost emails three or four times and I received the emails successfully at my other, non-Gmail accounts, I couldn’t ignore the problem any longer. The last straw was when one of my friends forwarded me an email three times from another Gmail account and none of the mails came through.

I’ve since switched to Pobox.com and have been a happy chappy since. Pobox’s spam filtering isn’t quite as good as Gmail’s, but it’s good enough, and their once-per-day spam report where you can simply click on a false positive spam to whitelist and retrieve it is just brilliant. That feature alone is worth the $20/year. If they had an email forwarding address where I could send them the rest of my spams to improve their overall spam training, Pobox would be perfect. If you’re looking for a powerful, reliable email forwarding service, I can recommend Pobox without hesitation.

So, here’s a question: has anybody else out there lost emails with Gmail? Surely it must’ve happened to somebody besides me. (Oh, and if you read this, work for Google and would like to figure out what’s going on, drop me a mail: I have the Message-IDs of at least a couple of the emails that were lost. Maybe we can all work this out.)

1 The first time I experienced any email lossage with Gmail was, very unfortunately, a business-related mail: one of our RapidWeaver customers had a Gmail address, and despite me sending two or three emails to him, he claimed never to have never received them. Of course, said customer thought that our customer support was rather lacking when he never received an email within a month, despite me sending one off within 14 hours of seeing the problem. A small drama ensued, threats were being made to post about our lack of customer care to the front page of Digg, etc etc. Thankfully everything was sorted out at the very last minute and unhappy people were made happy again, but geez, it’s a nice reminder of how things can go wrong very fast when communication channels break down.

Andre Pang: git-svn & svn:externals

Tue, 2014-07-08 04:26

I’ve written before about git-svn and why I use it, but a major stumbling block with git-svn has been been a lack of support for svn:externals. If your project’s small and you have full control over the repository, you may be fortunate enough to not have any svn:externals definitions, or perhaps you can restructure your repository so you don’t need them anymore and live in git and Subversion interoperability bliss.

However, many projects absolutely require svn:externals, and once you start having common libraries and frameworks that are shared amongst multiple projects, it becomes very difficult to avoid svn:externals. What to do for the git-svn user?

If you Google around, it’s easy enough to find solutions out there, such as git-me-up, step-by-step tutorials, explanations about using git submodules, and an overview of all the different ways you can integrate the two things nicely. However, I didn’t like any of those solutions: either they required too much effort, were too fragile and could break easily if you did something wrong with your git configuration, or were simply too complex for such a seemingly simple problem. (Ah, I do like dismissing entire classes of solutions by wand-having them as over-engineering.)

So, in the great spirit of scratching your own itch, here’s my own damn solution:

git-svn-clone-externals

This is a very simple shell script to make git-svn clone your svn:externals definitions. Place the script in a directory where you have one or more svn:externals definitions, run it, and it will:

  • git svn clone each external into a .git_externals/ directory.
  • symlink the cloned repository in .git_externals/ to the proper directory name.
  • add the symlink and .git_externals/ to the .git/info/excludes/ file, so that you’re not pestered about it when performing a git status.

That’s pretty much about it. Low-tech and cheap and cheery, but I couldn’t find anything else like it after extensive Googling, so hopefully some other people out there with low-tech minds like mine will find this useful.

You could certainly make the script a lot more complex and do things such as share svn:externals repositories between different git repositories, traverse through the entire git repository to detect svn:externals definitions instead of having to place the script in the correct directory, etc… but this works, it’s simple, and it does just the one thing, unlike a lot of other git/svn integration scripts that I’ve found. I absolutely do welcome those features, but I figured I’d push this out since it works for me and is probably useful for others.

The source is on github.com at http://github.com/andrep/git-svn-clone-externals/tree/master. Have fun subverting your Subversion overlords!