Planet Linux Australia

Syndicate content
Planet Linux Australia -
Updated: 40 min 1 sec ago

Rod Whitby: Syncing Treo650 with Google Contacts using CompanionLink for Google

Tue, 2016-09-06 13:02

In preparation for a possible move from my Treo 650 to the new Palm Pre, I’ve decided to try and synchronise my contacts between the Treo and Google Contacts.

So I’m evaluating CompanionLink for Google as a possible tool to achieve this. Another option might be GooSync.

I tried syncing some sample contacts in both directions, with the following results:

  1. Google Contacts to PalmOS Contacts
  2. Syncing a new contact from Google Contacts to PalmOS Contacts results in the following fields being synched correctly:

    • Name
    • Title
    • Company
    • Home Phone
    • Work Phone
    • Mobile Phone
    • E-mail (synched with the Work Email field in Google Contacts)
    • Fax (synched with the Work Fax field in Google Contacts)
    • Home Address
    • Work Address
    • Other Address
    • Notes

    The following Google Contacts fields are not synched:

    • Home Email
    • Other Email
    • Home Fax
    • Pager
    • Other Phone
    • IM Fields (Google Talk, Skype, Jabber, etc)
    • Custom Field

  3. PalmOS Contacts to Google Contacts
  4. Syncing a new contact from PalmOS Contacts to Google Contacts results in the following fields being synched correctly:

    • Name
    • Title
    • Company
    • Work Email (synched with the first E-mail field in PalmOS Contacts)
    • Home Phone
    • Work Phone
    • Mobile Phone
    • Home Address
    • Work Address
    • Other Address
    • Google Talk (synched with the IM field in PalmOS Contacts)
    • Notes (synched with a combination of the Custom and Note fields in PalmOS Contacts)

    The following PalmOS Contacts fields are not synched:

    • Secondary E-mail entries
    • Other Phone
    • AIM
    • MSN
    • Web site

I then tried duplicating contacts to see if I could determine the primary synchronisation key. Duplicating a contact in PalmOS Contacts and then synchronising did not result in a duplicated contact in Google Contacts. However, changing the E-Mail field in the duplicated contact in PalmOS Contacts was enough to cause it to be created as a second separate record in Google Contacts. So it seems that the PalmOS E-Mail field (which syncs with the Google Work Email field) is the primary key.
Interestingly, even while the PalmOS Contacts HotSync conduit is set up to sync with Google Contacts, the syncing with the Palm Desktop still happens. Indeed, the deletion of a record in the Palm Desktop is reflected in PalmOS Contacts on each sync, but it seems does not trigger a corresponding deletion in Google Contacts (perhaps there is some QuickSync vs SlowSync thing happening here). Modifying the record in Google Contacts which had already been deleted in PalmOS Contacts (through the Palm Desktop) did cause it to be reinstated in PalmOS Contacts on the next sync.
Adding a new E-Mail field to the PalmOS Contacts record (before the existing field) causes that new field to be the one that is synched with the Google Contacts Work Email field. So it seems that synchronisation happens between the first E-Mail field in PalmOS Contacts and the Work Email field in Google Contacts, and that only one Email address is ever synchronised between the two. If there is no Work Email field in Google Contacts, then all E-Mail fields in PalmOS Contacts are deleted. Additional Email fields in Google Contacts are not replicated in PalmOS Contacts. If an additional E-Mail field is added to PalmOS Contacts, then synchronisation exits with an error on the first attempt (“Can have at most one primary email address, found 2″), and prevents other fields being synchronisedbut then succeeds on the second attempt.  As long as a Work Email field is synching properly, other non-synching Email fields on both sides are retained (but not synched, even though other non-Email fields are synched).

Rod Whitby: Five new NSLU2 firmware releases in five days

Tue, 2016-09-06 13:02

In the last five days, we have made five new NSLU2 firmware releases:

2007-12-31 – Unslung 6.10-beta Release
2007-12-30 – SlugOS 4.8-beta Release
2007-12-29 – OpenWrt/NSLU2 Kamikaze 7.09 Release
2007-12-28 – Angstrom/NSLU2 2007.12 Release
2007-12-27 – Debian/NSLU2 Stable 4.0r2 Release

All of these new releases are available at

See for
an explanation of the pros and cons of each different firmware
distribution, and the installable packages available for each.

Thanks to everyone in the NSLU2-Linux, OpenWrt, Angstrom, OpenEmbedded
and Debian projects who contributed to these releases.

Remember, if you find any of the firmware or packages that the
NLSU2-Linux project provides useful, feel free to make a donation to
the project at

We are currently in need of about $500 to buy a RAID controller card
and some disks for our autobuild machine to support all this new
firmware with up-to-date package feeds …

Rod Whitby: The Definitive Analysis of Palm Pre App Install Limits and the Palm App Catalog Hanging

Tue, 2016-09-06 13:02

After the Preware 0.9.4 release, which included Applications, Themes, and Patches, and offers over 670 homebrew packages for installation, we (  started getting reports of the Palm App Catalog “hanging” the user interface for 30 seconds or more when the installation of a new application is initiated, but only when the Package Manager Service (the service which does all the Linux-level work for the Preware front-end) was installed.

After some analysis, I found that disabling all the feeds in Preware reduced the “hang” from more than 30 seconds to less than a second.

Looking through the output of ‘dbus-util –capture’ showed that the “hang” was during the call to the queryInstallCapacity method of the com.palm.appinstaller service, the routine that the Palm App Catalog application uses to determine whether there is enough space to install a new application from the Palm App Catalog.  Yes, this is the method which is at the heart of the dreaded “Too many applications” errors that people are seeing when they have a number of homebrew applications installed and try to install a new application from the Palm App Catalog.

Watching the output of ‘ps -ef’ during calls to this method (you can call it manually using luna-send) showed that palm was calling “ipkg -o /var list”.  Curious.  Why would you want to know the *complete* list of all *available* applications when determining whether there is room to install one known new application.  I suspect that Palm should be calling “ipkg -o /var list_installed” instead (which just lists the installed applications).  Note that Palm doesn’t use feeds the way that Preware does, so for Palm’s official use of ipkg, list and list_installed would return the same thing in their testing, but list_installed is definitely what they should be using to determine the size of installed applications.

The plot thickens when you use strace (which Palm conveniently includes in the official firmware image) on the running LunaSysMgr process.

It seems that LunaSysMgr calls “ipkg -o /var list” to get a list of installed packages (the probably intend to just get the list of installed packages, but when you have Preware installed and have feed configuration files in /var/etc/ipkg/*.conf, it actually returns a list of all *available packages).

LunaSysMgr then does an execve of “/usr/bin/du -s /var/usr/palm/applications/package” for each package in that list.  (BTW Palm, you seem to have a bug in the logic of that code, cause it’s running du on random garbage strings after the end of the real package list)

Yes, that’s right.  A call to queryInstallCapacity spawns a new program (“du”) once for each package returned by “ipkg -o /var list”.  No wonder the UI hangs for 30 seconds or more!

A single “du -s /var/usr/palm/applications/*” would be a far more efficient way to get exactly the same information, but again, Palm would not see this difference in testing because they do not support the third-party Preware application usage of ipkg feeds.

You can imagine that this behaviour is probably related to the app install limit that many people are experiencing too.  Unfortunately, I’ll have to reduce my /var partition size down from it’s current 2GB size (courtesy of the WebOS Internals Meta-Doctor) to be able to investigate this one.

Now the developers need to develop a new method of installing homebrew applications so that this bug in Palm’s appInstaller service is not triggered.

In the meantime, the work-around is to go into the Preware Preferences screen, hit the “Feeds” button in the top-right corner, and disable all the feeds while you use the Palm App Catalog application in another card.  No need to exit the Feeds screen, just toggle all the button to “Off”, and then toggle them back to “On” when you’re finished with the App Catalog.

For the solution to this problem, see Update #2, below.

I’ve created a thread on PreCentral where this issue can be discussed.  As I uncover more information, I’ll publish my finding here.

Update #1: I’ve now webOS Doctored my Pre in the name of science, and have tested the limits of installing applications.

If you run “du -s /var/usr/palm/applications/*”, and add up all the numbers in the first column, then as soon as you hit the 62367 1K blocks limit of the addition of the sizes reported by that “du” command and the size of the app you with to install, you will get the dreaded “Sorry, Not Enough Memory” error from the Palm App Catalog application (and any other installer, like fileCoaster or PreLoad, which uses the palm appInstaller API).  It doesn’t matter whether you have 192MB free in your /var partition, it will max out at just under 64MB of application usage.

Update #2: I have now created a Linux Application called “Fair Dinkum App Limit” (org.webosinternals.fairdinkum), which removes both the “hang” and the arbitrary application limit.  You can find it in Preware.  Just install it (no need to even run anything – if it is installed, it’s working), and you’re ready to install more applications than you can poke a stick at …

Fair Dinkum App Limit works by simply putting a couple of wrapper scripts in /usr/local/bin, which returns a size of zero when du is called, and returns the output of “ipkg -o /var list_installed” when “ipkg -o /var list” is called.  In the future, the wrappers will be made much more sophisticated than they are right now to prevent operation outside of the specific cases where they need to fool LunaSysMgr, and to also include a safety buffer so that users do not fill the /var partition.  This is a tactical response to a problem that people using homebrew applications are experiencing.  Hopefully, Palm will provide the long term solution for limits on application installation in a release in the near future.

Notes for Palm, if you happen to read this:

1) We fully appreciate that the usage of the ipkg database in /var for homebrew applications is a choice that the homebrew community has made, and is not supported by Palm.

2) We fully agree that the use of “ipkg -o /var list” instead of “ipkg -o /var list_installed” would work perfectly fine for the way that Palm is officially using the ipkg database in /var/usr/lib/ipkg, but we contend that the “list” and “list_installed” commands have clear intended usage, and the one to find the list of installed applications for checking available disk space should be “list_installed”.

3) We fully realise that the initial version of the FairDinkum scripts are unsafe.  Returning a zero value for du is a temporary solution while we work out how to achieve the same result safely.  The intention is to only return false values when du is being called from LunaSysMgr, and to make sure that a safety buffer is kept so that users do not fill the /var partition.

4) I would be very happy to discuss these issues with anyone at Palm (Chuq and Google both have my email address), and would hope that we can together architect a solution for supporting homebrew application installation which does not require these work-arounds.

5) We have designed these work-arounds in a way which does not collide with Palm OTA Updates, and in a way that we can update them automatically using the Preware installer application, and in a way that we can cause them to self-remove when Palm releases a long term solution.

Update #3:

It seems that there is yet another limit on application installation imposed by LunaSysMgr.  Once the used space on /var crosses around 60%, LunaSysMgr will once again refuse to install applications.

I’m going to need to webOS Doctor my Pre yet again (to reallocate 2GB to /var) to determine whether this limit is a fixed percentage, or a fixed number of blocks.

Update #4:

The limit is 102400 free blocks.  Mystery solved.  That also means the Fair Dinkum App Limit cannot cause your /var to be filled.

Update #5:

Thanks to Carl Thompson, an improved version of Fair Dinkum App Limit which does not alter the operation of ‘du’ has been released.

Rod Whitby: Replacing dropbear with openssh

Tue, 2016-09-06 13:02

I prefer to use OpenSSH rather than Dropbear on my devices.  The main reason is to get sftp support (which is required by sshfs).  Another reason is to get better support for agent forwarding (which is essential for bouncing from one machine to another without leaving your private keys all over the internet).

To do this on OpenMoko (or any other OpenEmbedded-based distribution for that matter, for instance SlugOS or Angstrom):

  1. Edit /etc/init.d/dropbear by replacing “DROPBEAR_PORT=22” with “DROPBEAR_PORT=2222” (or any other unused port).
  2. Run “ipkg install -force-depends openssh” to install openssh.
  3. Make sure you have set a root password before rebooting (use “passwd” to set it).
  4. Reboot (dropbear will restart on the new port, and openssh will start on the normal ssh port).
  5. Check that openssh is now serving on port 22 by logging into the device over ssh.
  6. Run “ipkg remove -force-depends dropbear” to remove dropbear.
  7. Then run “ipkg install openssh-sftp” to install support for the sftp protocol which sshfs uses.

Rod Whitby: Palm Pre GPS doesn’t like my hemisphere

Tue, 2016-09-06 13:02

It seems the Palm Pre GPS was never tested in the southern hemisphere – my new Pre’s GPS reports Lat: 394.6, Long: 138.6

24926.609       PUB     call    460             :1.26   com.palm.location       //getCurrentPosition  «string=“{}”, string=“com.sfmpllc.sendmylocation 1058”» 24926.641       PRV     call    238             :1.68  /com/palm/phone/tel_getradiotype      «» 24926.661       PRV     return  238     0.020   :1.26   :1.68           «string=“success”, string=“CDMA”» 24926.751       PRV     call    239             :1.68  /com/palm/phone/tel_getbsinfo «» 24926.786       PUB     call    461             com.palm.luna   org.freedesktop.DBus    /org/freedesktop/DBus/AddMatch        «string=“interface=org.freedesktop.DBus,member=NameOwnerChanged,arg0=com.palm.location”» 24926.920       PUB     return  460             com.palm.location       com.palm.luna        «string=“{“errorCode”:0,”timestamp”:1.254820510841E12,”latitude”:394.593215,”longitude”:138.681593,”horizAccuracy”:150,”heading”:0,”velocity”:0,”altitude”:0,”vertAccuracy”:0}”»

24926.609       PUB     call    460             :1.26   com.palm.location       //getCurrentPosition  «string=“{}”, string=“com.sfmpllc.sendmylocation 1058”»

24926.920       PUB     return  460             com.palm.location       com.palm.luna        «string=“{“errorCode”:0,”timestamp”:1.254820510841E12,”latitude”:394.xxxxxx,”longitude”:138.xxxxxx,”horizAccuracy”:150,”heading”:0,”velocity”:0,”altitude”:0,”vertAccuracy”:0}”»

The latitude value should be 34.6 degrees South (or -34.6 degrees).

That would explain why Google Maps isn’t working.

Now I need to work out how to replace the Coordinates java class in /usr/lib/luna/java/location.jar, so that the getLatitude method returns a number between -90 and +90 …

I wonder how many WebOS applications will then barf on a negative latitude value …

The PreCentral thread has more information on other GPS tweaks.

Rod Whitby: Connecting a Treo650 to a Freecom DataTank 2

Tue, 2016-09-06 13:02
  1. Install bluez2-utils from Optware
  2. Install the following kernel modules: bluetooth, hci_usb, l2cap, bnep, rfcomm, hidp
  3. Create /dev/rfcomm0 as follows:
    mknod /dev/rfcomm0 c 216 0
  4. Enable routing from ppp0 to eth1 (don’t do this if you use ppp for your internet connection):
    /etc/init.d # diff -u routing.orig routing --- routing.orig Sat Mar 22 18:57:23 2008 +++ routing Sat Mar 22 15:14:29 2008 @@ -37,6 +37,7 @@ # lo & eth0 always accepted (also if WAN port IP not set) /sbin/iptables -A INPUT -i $INIF -j ACCEPT + /sbin/iptables -A INPUT -i ppp0 -j ACCEPT /sbin/iptables -A INPUT -i lo -j ACCEPT # get IP address from WAN port @@ -150,6 +151,7 @@ /sbin/iptables -A FORWARD -j TCPMSS -o $EXIF --clamp-mss-to-pmtu -p tcp --tcp-flags SYN,RST SYN fi /sbin/iptables -A FORWARD -i $EXIF -o $INIF -m state --state ESTABLISHED,RELATED -j ACCEPT + /sbin/iptables -A FORWARD -i $EXIF -o ppp0 -m state --state ESTABLISHED,RELATED -j ACCEPT grep -q ^proxy_server=checked /etc/master.conf @@ -163,6 +165,7 @@ fi else /sbin/iptables -A FORWARD -s $SUBNET -i $INIF -j ACCEPT + /sbin/iptables -A FORWARD -s $SUBNET -i ppp0 -j ACCEPT fi # port forwarding

  5. Edit /etc/ppp/options as follows:

    ms-dns 192.168.1.ZZ

    (edit the last two lines to suit your network topology, the first IP address
    is your gateway device, the second IP address will be assigned to the client,
    and the third IP address is your DNS server)
  6. Add the following line to /etc/dnsmasq.conf:

Rod Whitby: The WebOS Internals Meta-Doctor

Tue, 2016-09-06 13:02

Palm supplies Palm Pre owners with this wonderful recovery tool called the webOS Doctor.  Part of the charter of the WebOS Internals project is to ensure that anything we (or anyone following instructions we publish or installing packages we develop) do can always be reverted using the webOS Doctor.

Usually, a Palm Pre is required to be activated on the Sprint network before it can be used.  This is not possible for a Palm Pre in Australia.

So we need to allow the Pre to be used without activation, and there are a number of information sources showing how this can be done.  There are also some dubious sites that redistribute modified versions of the webOS Doctor (which is a clear violation of copyright law, since it contains proprietary Palm software).  Note that WebOS Internals is always careful to comply with all copyright laws (copyright law is the foundation upon which open source licenses are based).

So we need a way for a Pre owner (who has the right to use the webOS Doctor on their own Pre) to modify the webOS Doctor that is specific to their particular version of the Palm Pre before using it to flash that modified firmware onto their Pre.

That’s where the WebOS Internals “Meta-Doctor” comes into play.

I have created a tool which will download, unpack, patch, and repack a webOS Doctor image, applying a number of transformations along the way to:

  1. Bypass the need for activation
  2. Enable Palm Profile access
  3. Set developer mode on by default
  4. Increase the size of the /var partition to 2GB

You can find this tool in the WebOS Internals source code repository at

Do not redistribute modified versions of the webOS Doctor created using this tool – it is for end-user use only.

I’ve created a forum thread on PreCentral for discussion about this tool.

Rod Whitby: Setting the OpenMoko timezone

Tue, 2016-09-06 13:02

If you want to set the timezone on your phone correctly, do the following:

  1. ipkg install tzdata
  2. ipkg install your desired tzdata-* packages.  For instance, I use “tzdata-australia“.
  3. Enable your desired timezone by symlinking it to “/etc/localtime“.  Adjust the following example command line for your locality.
    • ln -s /usr/share/zoneinfo/Australia/Adelaide /etc/localtime
  4. The “date” command should now show the correct time for your timezone.  If it is not correct, then install the “ntpclient” package, and use it to set your clock.

Note that this technique should work on any OpenEmbedded-based Linux distribution.

    Rod Whitby: The Palm Pre lands in Australia

    Tue, 2016-09-06 13:02

    Thanks to the generosity of the PreCentral and WebOS Internals communities, I am now the proud owner of a Palm Pre.

    There is just one catch  – since I live in Australia (which uses a proper cellular communications standard), the CDMA Palm Pre that I was able to import from the USA will never work as a phone here (yes, I knew this before I purchased it).  I plan to also purchase an unlocked GSM/UMTS Pre when one becomes available (maybe I’ll buy a German one and then swap the two keyboards).

    After founding the WebOS Internals project, and using the Pre Emulator in the development of Preware, it is great to have a real device to get the full Pre experience.

    If you want to keep up to date with the activities of the WebOS Internals group, just follow @webosinternals on Twitter.  You can also find a list of articles about WebOS Internals in our Press_Room.  We hang out in the #webos-internals IRC channel on Freenode, and have a webos-internals-announce Google group.

    I still use my trusty old Treo 650 as my daily phone, which allows me to not have to worry about reflashing the Pre to try out things, as I don’t keep any real personal data on it.

    I guess this also closes the OpenMoko chapter of my open source development activities.  I was involved with OpenMoko from the very start, but always said that an OpenMoko device with a hardware keyboard was my preferred form factor, and the Pre seems to satisfy that personal hardware form factor preference whilst still being open enough on the software side to attract my interest.  I wish those who are continuing the OpenMoko path the best of success.

    I’ll document my experiences with the Pre in subsequent posts …

    Binh Nguyen: Diplomacy, Russia Vs USA, and More

    Mon, 2016-09-05 02:20
    Over and over again, the US and Russia (and other countries) seem to get in each others way. The irony is that while there are many out there who believe that they can come to an 'agreement' of sorts the more I look the more difficult I find this proposition to occur. - first, let's introduce ourselves to modern diplomacy though some videos Diplomacy

    Ben Martin: Houndbot rolling stock upgrade

    Sun, 2016-09-04 16:16
    After getting Terry the robot to navigate around inside with multiple Kinects as depth sensors I have now turned my attention to outdoor navigation using two cameras as sensors. The cameras are from a PS4 eye which I hacked to be able to connect to a normal machine. The robot originally used 5.4 inch wheels which were run with foam inside them. This sort of arrangement can be seen in many builds in the Radio Controlled (RC) world and worked well when the robot was simple and fairly light. Now that it is well over 10kg the same RC style build doesn't necessarily still work. Foam compresses a bit to easily.

    I have upgraded to 12 inch wheels with air tube tires. This jump seemed a bit risky, would the new setup overwhelm the robot? Once I modified the wheels and came up with an initial mounting scheme to test I think the 12 inch is closer to what the robot naturally wants to have. This should boost the maximum speed of the machine to around 20km/h which is probably as much as you might want on something autonomous. For example, if your robot can out run you things get interesting.

    I had to get the wheels attached in order to work out clearances for the suspension upgrade. While the original suspension worked great for a robot that you only add 1-2kg to, with an itx case, two batteries, a fused power supply etc things seem to have added up to too much weight for the springs to counter.

    I now have some new small 'coil overs' in hand which are taken from mini mountain bike suspension. They are too heavy for what I am using, with around 600lb/inch compression. I have in mind some places that use coil overs in between the RC ones and the push bike ones which I may end up using. Also with slightly higher travel distance.

    As the photo reveals, I don't actually have the new suspension attached yet. I'm thinking about a setup based around two bearing mounts from sparkfun. I'd order from servocity but sfe has cheaper intl shipping :o Anyway, two bearing mounts at the top, two at the bottom and a steel shaft that is 8mm in the middle and 1/4 inch (6.35mm) on the edges. Creating the shafts like that, with the 8mm part just the right length will trap the shaft between the two bearing mounts for me. I might tack weld on either side of the coil over mounts so there is no side to side movement of the suspension.

    Yes, hubs and clamping collars were by first thought for the build and would be nice, but a reasonable result for a manageable price is also a factor.

    Danielle Madeley: Websockets + on the ESP8266 w/ Micropython

    Sun, 2016-09-04 01:01

    I recently learned about the ESP8266 while at Pycon AU. It’s pretty nifty: it’s tiny, it has wifi, a reasonable amount of RAM (for a microcontroller) oh, and it can run Python. Specifically Micropython. Anyway I purchased a couple from Adafruit (specifically this one) and installed the Micropython UNIX port on my computer (be aware with the cheaper ESP8266 boards, they might not be very reflashable, or so I’ve been told, spend the extra money for one with decent flash).

    The first thing you learn is that the ports are all surprisingly different in terms of what functionality they support, and the docs don’t make it clear like they do for CPython. I learned the hard way there is a set of docs per port, which maybe is why you the method you’re looking for isn’t there.

    The other thing is that even though you’re getting to write in Python, and it has many Pythonic abstractions, many of those abstractions are based around POSIX and leak heavily on microcontrollers. Still a number of them look implementable without actually reinventing UNIX (probably).

    The biggest problem at the moment is there’s no “platform independent” way to do asynchronous IO. On the microcontroller you can set top-half interrupt handlers for IO events (no malloc here, yay!), gate the CPU, and then execute bottom halfs from the main loop. However that’s not going to work on UNIX. Or you can use select, but that’s not available on the ESP8266 (yet). Micropython does support Python 3.5 asyncio coroutines, so hopefully the port of asyncio to the ESP8266 happens soon. I’d be so especially ecstatic if I could do await pin.trigger(Pin.FALLING).

    There’s a few other things that could really help make it feel like Python. Why isn’t disabling interrupts a context manager/decorator. It’s great that you can try/finally your interrupt code, but the with keyword is so much more Pythonic. Perhaps this is because the code is being written by microprocessor people… which is why they’re so into protocols like MQTT for talking to their devices.

    Don’t get me wrong, MQTT is a great protocol that you can cram onto all sorts of devices, with all sorts of crappy PHYs, but I have wifi, and working SSL. I want to do something more web 2.0. Something like websockets. In fact, I want to take another service’s REST API and websockets, and deliver that information to my device… I could build a HTTP server + MQTT broker, but that sounds like a pain. Maybe I can just build a web server with and connect to that directly from the device?!

    The ESP8266 already has some very basic websocket support for its WebREPL, but that’s not very featureful and seems to only implement half of the spec. If we’re going to have Python on a device, maybe we can have something that looks like the great websockets module. Turns out we can! is a little harder, it requires a handshake which is not documented (I reversed it in the end), and decoding a HTTP payload, which is not very clearly documented (had to read the source). It’s not the most efficient protocol out there, but the chip is more than fast enough to deal with it. Also fun times, it turns out there’s no platform independent way to return from waiting for IO. Basically it turned out there were a lot of yaks to shave.

    Where it all comes into its own though is the ability to write what is pretty much everyday, beautiful Python however, it’s worth it over Arduino sketches or whatever else takes your fancy.

    uwebsockets/usocketio on Github.

    Linux Users of Victoria (LUV) Announce: LUV Main September 2016 Meeting: Spartan / The Future is Awesome

    Sat, 2016-09-03 23:03
    Start: Sep 6 2016 18:30 End: Sep 6 2016 20:30 Start: Sep 6 2016 18:30 End: Sep 6 2016 20:30 Location: 

    6th Floor, 200 Victoria St. Carlton VIC 3053



    • Lev Lafayette, Spartan: A Linux HPC/Cloud Hybrid
    • Paul Fenwick, The Future is Awesome (and what you can do about it)

    200 Victoria St. Carlton VIC 3053

    Late arrivals, please call (0490) 049 589 for access to the venue.

    Before and/or after each meeting those who are interested are welcome to join other members for dinner. We are open to suggestions for a good place to eat near our venue. Maria's on Peel Street in North Melbourne is currently the most popular place to eat after meetings.

    LUV would like to acknowledge Red Hat and Infoxchange for their help in obtaining the meeting venues.

    Linux Users of Victoria Inc. is an incorporated association, registration number A0040056C.

    September 6, 2016 - 18:30

    read more

    Chris Smart: Configuring QEMU bridge helper after “access denied by acl file” error

    Wed, 2016-08-31 21:03

    QEMU has a neat bridge-helper utility which allows a non-root user to easily connect a virtual machine to a bridged interface. In Fedora at least, qemu-bridge-helper runs as setuid (any user can run as root) and privileges are immediately dropped to cap_net_admin. It also has a simple white/blacklist ACL mechanism in place which limits connections to virbr0, libvirt’s local area network.

    That’s all great, but often you actually want a guest to be a part of your real network. This means it must connect to a bridged interface (often br0) on a physical network device.

    If your user tries to kick up such a QEMU guest while specifying bridge,br=br0, something like this (although probably also with a disk or kernel and initramfs):

    qemu-system-x86_64 \
    -machine accel=kvm \
    -cpu host \
    -netdev bridge,br=br0,id=net0 \
    -device virtio-net-pci,netdev=net0

    You may run into the following error:

    access denied by acl file
    qemu-system-ppc64: -netdev bridge,br=br0,id=net0: bridge helper failed

    As mentioned above, this is the QEMU bridge config file /etc/qemu/bridge.conf restricting bridged interfaces to virbr0 for all users by default. So how to make this work more nicely?

    One way is to simply edit the main config file and change virbr0 to all, however that’s not particularly fine-grained.

    Instead, we could create a new config file for the user which specifies any (or all) bridge devices that this user is permitted to connect guests to. This way all other users are restricted to virbr0 while your user can connect to other bridges.

    This doesn’t have to be a user, it could also be a group (just substitute ${USER} for the group, below), and you can also add multiple files.

    Instead of allow you can use deny in the same way to prevent a user or group from attaching to any or all bridges.

    So, let’s create a new file for our user and give them access to all interfaces (requires sudo):

    echo "allow all" | sudo tee /etc/qemu/${USER}.conf
    echo "include /etc/qemu/${USER}.conf" | sudo tee --append /etc/qemu/bridge.conf
    sudo chown root:${USER} /etc/qemu/${USER}.conf
    sudo chmod 640 /etc/qemu/${USER}.conf

    This user should now be able to successfully kick up the guest connected to br0.
    qemu-system-x86_64 \
    -machine accel=kvm \
    -cpu host \
    -netdev bridge,br=br0,id=net0 \
    -device virtio-net-pci,netdev=net0

    Russell Coker: Monitoring of Monitoring

    Mon, 2016-08-29 13:02

    I was recently asked to get data from a computer that controlled security cameras after a crime had been committed. Due to the potential issues I refused to collect the computer and insisted on performing the work at the office of the company in question. Hard drives are vulnerable to damage from vibration and there is always a risk involved in moving hard drives or systems containing them. A hard drive with evidence of a crime provides additional potential complications. So I wanted to stay within view of the man who commissioned the work just so there could be no misunderstanding.

    The system had a single IDE disk. The fact that it had an IDE disk is an indication of the age of the system. One of the benefits of SATA over IDE is that swapping disks is much easier, SATA is designed for hot-swap and even systems that don’t support hot-swap will have less risk of mechanical damage when changing disks if SATA is used instead of IDE. For an appliance type system where a disk might be expected to be changed by someone who’s not a sysadmin SATA provides more benefits over IDE than for some other use cases.

    I connected the IDE disk to a USB-IDE device so I could read it from my laptop. But the disk just made repeated buzzing sounds while failing to spin up. This is an indication that the drive was probably experiencing “stiction” which is where the heads stick to the platters and the drive motor isn’t strong enough to pull them off. In some cases hitting a drive will get it working again, but I’m certainly not going to hit a drive that might be subject to legal action! I recommended referring the drive to a data recovery company.

    The probability of getting useful data from the disk in question seems very low. It could be that the drive had stiction for months or years. If the drive is recovered it might turn out to have data from years ago and not the recent data that is desired. It is possible that the drive only got stiction after being turned off, but I’ll probably never know.

    Doing it Properly

    Ever since RAID was introduced there was never an excuse for having a single disk on it’s own with important data. Linux Software RAID didn’t support online rebuild when 10G was a large disk. But since the late 90’s it has worked well and there’s no reason not to use it. The probability of a single IDE disk surviving long enough on it’s own to capture useful security data is not particularly good.

    Even with 2 disks in a RAID-1 configuration there is a chance of data loss. Many years ago I ran a server at my parents’ house with 2 disks in a RAID-1 and both disks had errors on one hot summer. I wrote a program that’s like ddrescue but which would read from the second disk if the first gave a read error and ended up not losing any important data AFAIK. BTRFS has some potential benefits for recovering from such situations but I don’t recommend deploying BTRFS in embedded systems any time soon.

    Monitoring is a requirement for reliable operation. For desktop systems you can get by without specific monitoring, but that is because you are effectively relying on the user monitoring it themself. Since I started using mon (which is very easy to setup) I’ve had it notify me of some problems with my laptop that I wouldn’t have otherwise noticed. I think that ideally for desktop systems you should have monitoring of disk space, temperature, and certain critical daemons that need to be running but which the user wouldn’t immediately notice if they crashed (such as cron and syslogd).

    There are some companies that provide 3G SIMs for embedded/IoT applications with rates that are significantly cheaper than any of the usual phone/tablet plans if you use small amounts of data or SMS. For a reliable CCTV system the best thing to do would be to have a monitoring contract and have the monitoring system trigger an event if there’s a problem with the hard drive etc and also if the system fails to send a “I’m OK” message for a certain period of time.

    I don’t know if people are selling CCTV systems without monitoring to compete on price or if companies are cancelling monitoring contracts to save money. But whichever is happening it’s significantly reducing the value derived from monitoring.

    Related posts:

    1. Health and Status Monitoring via Smart Phone Health Monitoring Eric Topol gave an interesting TED talk about...
    2. Planning Servers for Failure Sometimes computers fail. If you run enough computers then you...
    3. Shelf-life of Hardware Recently I’ve been having some problems with hardware dying. Having...

    Steven Hanley: [mtb/events] Oxfam Trailwalker - Sydney 2016 - ARNuts

    Mon, 2016-08-29 13:00

    A great day out on the trail with friends (fullsize)
    Though it did not really hit me in the lead up or during the event until half way that it was yet another 100km and these are indeed somewhat tough to get through. The day out in the bush with my friends Alex, David and Julie was awesome.

    As I say in the short report with the photos linked below, Oxfam is a great charity and that they have these trailwalker events in many places around the world to fundraise and get people to enjoy some quality outdoor time is pretty awesome. This is a hard course, that it took us 14h30m to get through it shows that but it sure is pretty, amazing native flowers, views (water ways and bush) and that it can get in to Manly with hardly realising you are in the middle of the biggest city in Australia is awesome.

    My words and photos are online in my Oxfam Trailalker - Sydney 2016 - ARnuts gallery. What a fun day out!.

    Craig Sanders: fakecloud

    Sun, 2016-08-28 15:02

    I wrote my first Mojolicious web app yesterday, a cloud-init meta-data server to enable running pre-built VM images (e.g. as provided by debian, ubuntu, etc) without having to install and manage a complete, full-featured cloud environment like openstack.

    I hacked up something similar several years ago when I was regularly building VM images at home for openstack at work, with just plain-text files served by apache, but that had pretty-much everything hard-coded. fakecloud does a lot more and allows per-VM customisation of user-data (using the IP address of the requesting host). Not bad for a day’s hacking with a new web framework.

    fakecloud is a post from: Errata

    Chris Smart: Live migrating Btrfs from RAID 5/6 to RAID 10

    Fri, 2016-08-26 13:03

    Recently it was discovered that the RAID 5/6 implementation in Btrfs is broken, due to the fact that can miscalculate parity (which is rather important in RAID 5 and RAID 6).

    So what to do with an existing setup that’s running native Btfs RAID 5/6?

    Well, fortunately, this issue doesn’t affect non-parity based RAID levels such as 1 and 0 (and combinations thereof) and it also doesn’t affect a Btrfs filesystem that’s sitting on top of a standard Linux Software RAID (md) device.

    So if down-time isn’t a problem, we could re-create the RAID 5/6 array using md and put Btrfs back on top and restore our data… or, thanks to Btrfs itself, we can live migrate it to RAID 10!

    A few caveats though. When using RAID 10, space efficiency is reduced to 50% of your drives, no matter how many you have (this is because it’s mirrored). By comparison, with RAID 5 you lose a single drive in space, with RAID 6 it’s two, no-matter how many drives you have.

    This is important to note, because a RAID 5 setup with 4 drives that is using more than 2/3rds of the total space will be too big to fit on RAID 10. Btrfs also needs space for System, Metadata and Reserves so I can’t say for sure how much space you will need for the migration, but I expect considerably more than 50%. In such cases, you may need to add more drives to the Btrfs array first, before the migration begins.

    So, you will need:

    • At least 4 drives
    • An even number of drives (unless you keep one as a spare)
    • Data in use that is much less than 50% of the total provided by all drives (number of disks / 2)

    Of course, you’ll have a good, tested, reliable backup or two before you start this. Right? Good.

    Plug any new disks in and partition or luksFormat them if necessary. We will assume your new drive is /dev/sdg, you’re using dm-crypt and that Btrfs is mounted at /mnt. Substitute these for your actual settings.
    cryptsetup luksFormat /dev/sdg
    UUID="$(cryptsetup luksUUID /dev/sdg)"
    echo "luks-${UUID} UUID=${UUID} none" >> /etc/crypttab
    cryptsetup luksOpen luks-${UUID} /dev/sdg
    btrfs device add /dev/mapper/luks-${UUID} /mnt

    The migration is going to take a long time, so best to run this in a tmux or screen session.

    time btrfs balance /mnt
    time btrfs balance start -dconvert=raid10 -mconvert=raid10 /mnt

    After this completes, check that everything has been migrated to RAID 10.
    btrfs fi df /srv/data/
    Data, RAID10: total=2.19TiB, used=2.18TiB
    System, RAID10: total=96.00MiB, used=240.00KiB
    Metadata, RAID10: total=7.22GiB, used=5.40GiB
    GlobalReserve, single: total=512.00MiB, used=0.00B

    If you still see some RAID 5/6 entries, run the same migrate command and then check that everything has migrated successfully.

    Now while we’re at it, let’s defragment everything.
    time btrfs filesystem defragment /srv/data/ # this defrags the metadata
    time btrfs filesystem defragment -r /srv/data/ # this defrags data

    For good measure, let’s rebalance again without the migration (this will also take a while).
    time btrfs fi balance start --full-balance /srv/data/

    Francois Marier: Debugging gnome-session problems on Ubuntu 14.04

    Thu, 2016-08-25 15:02

    After upgrading an Ubuntu 14.04 ("trusty") machine to the latest 16.04 Hardware Enablement packages, I ran into login problems. I could log into my user account and see the GNOME desktop for a split second before getting thrown back into the LightDM login manager.

    The solution I found was to install this missing package:

    apt install libwayland-egl1-mesa-lts-xenial Looking for clues in the logs

    The first place I looked was the log file for the login manager (/var/log/lightdm/lightdm.log) where I found the following:

    DEBUG: Session pid=12743: Running command /usr/sbin/lightdm-session gnome-session --session=gnome DEBUG: Creating shared data directory /var/lib/lightdm-data/username DEBUG: Session pid=12743: Logging to .xsession-errors

    This told me that the login manager runs the gnome-session command and gets it to create a session of type gnome. That command line is defined in /usr/share/xsessions/gnome.desktop (look for Exec=):

    [Desktop Entry] Name=GNOME Comment=This session logs you into GNOME Exec=gnome-session --session=gnome TryExec=gnome-shell X-LightDM-DesktopName=GNOME

    I couldn't see anything unexpected there, but it did point to another log file (~/.xsession-errors) which contained the following:

    Script for ibus started at run_im. Script for auto started at run_im. Script for default started at run_im. init: Le processus gnome-session (GNOME) main (11946) s'est achevé avec l'état 1 init: Déconnecté du bus D-Bus notifié init: Le processus logrotate main (11831) a été tué par le signal TERM init: Le processus update-notifier-crash (/var/crash/_usr_bin_unattended-upgrade.0.crash) main (11908) a été tué par le signal TERM

    Seaching for French error messages isn't as useful as searching for English ones, so I took a look at /var/log/syslog and found this:

    gnome-session[4134]: WARNING: App 'gnome-shell.desktop' exited with code 127 gnome-session[4134]: WARNING: App 'gnome-shell.desktop' exited with code 127 gnome-session[4134]: WARNING: App 'gnome-shell.desktop' respawning too quickly gnome-session[4134]: CRITICAL: We failed, but the fail whale is dead. Sorry....

    It looks like gnome-session is executing gnome-shell and that this last command is terminating prematurely. This would explain why gnome-session exits immediately after login.

    Increasing the amount of logging

    In order to get more verbose debugging information out of gnome-session, I created a new type of session (GNOME debug) by copying the regular GNOME session:

    cp /usr/share/xsessions/gnome.desktop /usr/share/xsessions/gnome-debug.desktop

    and then adding --debug to the command line inside gnome-debug.desktop:

    [Desktop Entry] Name=GNOME debug Comment=This session logs you into GNOME debug Exec=gnome-session --debug --session=gnome TryExec=gnome-shell X-LightDM-DesktopName=GNOME debug

    After restarting LightDM (service lightdm restart), I clicked the GNOME logo next to the password field and chose GNOME debug before trying to login again.

    This time, I had a lot more information in ~/.xsession-errors:

    gnome-session[12878]: DEBUG(+): GsmAutostartApp: starting gnome-shell.desktop: command=/usr/bin/gnome-shell startup-id=10d41f1f5c81914ec61471971137183000000128780000 gnome-session[12878]: DEBUG(+): GsmAutostartApp: started pid:13121 ... /usr/bin/gnome-shell: error while loading shared libraries: cannot open shared object file: No such file or directory gnome-session[12878]: DEBUG(+): GsmAutostartApp: (pid:13121) done (status:127) gnome-session[12878]: WARNING: App 'gnome-shell.desktop' exited with code 127

    which suggests that gnome-shell won't start because of a missing library.

    Finding the missing library

    To find the missing library, I used the apt-file command:

    apt-file update apt-file search

    and found that this file is provided by the following packages:

    • libhybris
    • libwayland-egl1-mesa
    • libwayland-egl1-mesa-dbg
    • libwayland-egl1-mesa-lts-utopic
    • libwayland-egl1-mesa-lts-vivid
    • libwayland-egl1-mesa-lts-wily
    • libwayland-egl1-mesa-lts-xenial

    Since I installed the LTS Enablement stack, the package I needed to install to fix this was libwayland-egl1-mesa-lts-xenial.

    I filed a bug for this on Launchpad.

    Maxim Zakharov: Small fix for AMP WordPress plugin

    Wed, 2016-08-24 11:03

    If you use AMP plugin for WordPress to make AMP (Accelerated Mobile Pages) version of your posts and have some troubles validating them on AMP validator, you may try this fix for AMP plugin to make those pages valid.