Planet Linux Australia

Syndicate content
Planet Linux Australia - http://planet.linux.org.au
Updated: 19 min 25 sec ago

Steven Hanley: [mtb] Hume and Hovell Ride Albury to Canberra 2012

Wed, 2015-11-18 07:25


A creek crossing on day 1 (fullsize)

This was a really fun ride, 3 days riding from Albury to Canberra on the Hume and Hovell track, a bunch of ARNuts and others, stopping overnight in Tumbarumba and then Tumut.

It was also not long after the Greenedge Call Me Maybe video came out so many of us spent a fiar proportion of the ride posing for photos and some videos miming the actions. I have never tried to edit the videos into anything together however you can see the poses in many of the photos.

The ride itself has a good variety of terrain, great views in places, confusing areas where it is difficult to follow the track and we all had fun. Photos and some words are online on my Hume and Hovell track ride 2012 page.

Binh Nguyen: Middle Eastern/African/Asian Background, NSA Whistleblowers, and More

Wed, 2015-11-18 02:22
- whenever you take a on a new job you feel naive (the following are all publicly available videos/documentaries often from well known media outlets). Despite what is being said by a lot of people in the public spotlight I don't believe that there is a way to acheive victory in a timely fashion. Kids of primary school age are being trained to hate the West, to learn how to use weapons, to become

Gary Pendergast: Replacing Rdio

Tue, 2015-11-17 23:26

I guess we’ve all heard of the impending demise of Rdio.

As one of the 500k subscribers with good taste in their streaming apps, it’s now time to consider a replacement. Here are my criteria – some of them may vary for you, but it’ll hopefully give you an idea for how you can choose, too.

Must Have
  • Offline sync to mobile (I listen to music when I’m flying a lot)
  • Ability to play from my Mac (I listen when I’m working)
  • Ability to play on Sonos (the rest of my house)
  • Family accounts
Should Have
  • Desktop App (I kill my browser pretty regularly, I don’t want that to interfere with my music)
Nice To Have
  • Android Auto support (I don’t have an Android Auto device, but I’m likely to buy one in the near future)
  • Account sharing instead of family accounts (it’s cheaper, and my wife and I mostly don’t use the account in different locations at the same time)

Given that the death of Rdio was most likely due to its lack of market share, I decided to only go with major players – this quickly narrowed it down to Google Play Music, Apple Music, and Spotify.

Google Play Music

Out of the box, Google Play Music does okay – it has an excellent selection of music, the mobile app isn’t terrible, and it works on Sonos. YouTube Red is supposed to be pretty nice, too, but it’s currently not available in Australia.

It falls down heavily when using it on my desktop, though. There’s a Chrome extension to hook into my keyboard media buttons, or there are third party apps available, none of which are very good.

Finally, it becomes completely unusable to share with my wife – I obviously can’t sign into my Google account on her phone, and Google still don’t have family accounts (though they have been announce as “coming soon”).

Apple Music

I’ve never had a good relationship with iTunes – it’s always been a clunky beast, and my recent experiments seem to indicate that not much has changed, except for a re-skin of some of the UI. It feels really hacked together. It is a native app, though, so it wins points by not being associated with my browser.

The family account was super janky to setup, I found the UI kept dying on me. Eventually I got through, however, and I hopefully will never need to touch that again (famous last words…).

On the bright side, the Apple Music app for Android is really nice, despite being a recent beta release. There’s no word on if it supports Android Auto, but that’s not an immediate requirement for me, so I’m happy to let it go.

Spotify

Spotify’s biggest benefit is that it’s not attached to a personal account. Unlike with Google or Apple, my wife and I could share the same account, without needing to share our personal logins. It’s cheating the system slightly, but it’d save us $6/month, so I’m not too concerned about it.

Spotify’s apps have been severely ugly in the past, but the good news is that the Android app is much more useable now. Unfortunately, I was unable to try out the OSX app, because the downloader was broken. The web app requires Adobe Flash, which is a total non-starter.

Conclusion

In the end, I chose Apple Music, for two reasons. One, it was the only one with a desktop app that actually worked. And two, it’s the only service that I can play Taylor Swift’s 1989 on. If the other services can’t get their act together enough to negotiate for a popular album to be on their service, then I’m concerned about their future ability to do so.

I may end up needing to re-evaluate this decision, particularly if the Sonos support doesn’t happen before Rdio finally closes it’s doors (I’m maintaining my Rdio account just for that). But for now, this will do.

Sridhar Dhanapalan: Twitter posts: 2015-11-09 to 2015-11-15

Mon, 2015-11-16 22:27

Steven Hanley: [mtb/events] Australian Single Speed Nationals 2012 - Beechworth, Bushranger themed (sort of)

Mon, 2015-11-16 22:25


Posing with Jeebus (fullsize)

Wow right now I am finding it surprising I have not been to the single speed nationals since 2012, (un) organised every year by a group of locals somewhere, sort of overseen by Australian Recreational Singlespeed Enthusiasts (ARSE). The 2010 Canberra SSNATS event in Majura Pines was heaps of fun, organised by Canberra One Gear Society (COGS).

In 2013 the event was in Cairns and though it sounded fun I decided it was just a bit too far to head up there for the event, in 2014 I tried and tried to talk friends into heading up for the weekend in Dungog NSW, however few of my Canberra friends were keen and I did something else that weekend (softie that I am).

I am still hoping the road trip to Wombat State Forest in Victoria will go ahead for the 2015 event. There was a ANZAC event in Rotortua over easter however I skipped that. This however is all getting off the topic of 2012.

I made it down to 2012, camping with McCook and having a fantastic weekend of mtb riding with the crew in Beechworth. The rather important aspect of beer was sorted that weekend and Bridge Road Brewerers in that town and they are possibly my favourite brewer in Australia.

The Beechworth mtb park is a great mix of interesting technical stuff and fun all in native bush, there were other ride options as can be seen in my gallery also. Photos and words from the 2012 Australian Single Speed Nationals are online in the link.

Erik de Castro Lopo: Forgive me Curry and Howard for I have Sinned.

Mon, 2015-11-16 22:22

Forgive me Curry and Howard for I have sinned.

For the last several weeks, I have been writing C++ code. I've been doing some experimentation in the area of real-time audio Digital Signal Processing experiments, C++ actually is better than Haskell.

Haskell is simply not a good fit here because I need:

  • To be able to guarantee (by inspection) that there is zero memory allocation/de-allocation in the real-time inner processing loop.
  • Things like IIR filters are inherently stateful, with their internal state being updated on every input sample.

There is however one good thing about coding C++; I am constantly reminded of all the sage advice about C++ I got from my friend Peter Miller who passed away a bit over a year ago.

Here is an example of the code I'm writing:

class iir2_base { public : // An abstract base class for 2nd order IIR filters. iir2_base () ; // Virtual destructor does nothing. virtual ~iir2_base () { } inline double process (double in) { unsigned minus2 = (minus1 + 1) & 1 ; double out = b0 * in + b1 * x [minus1] + b2 * x [minus2] - a1 * y [minus1] - a2 * y [minus2] ; minus1 = minus2 ; x [minus1] = in ; y [minus1] = out ; return out ; } protected : // iir2_base internal state (all statically allocated). double b0, b1, b2 ; double a1, a2 ; double x [2], y [2] ; unsigned minus1 ; private : // Disable copy constructor etc. iir2_base (const iir2_base &) ; iir2_base & operator = (const iir2_base &) ; } ;

Sam Watkins: sswam

Mon, 2015-11-16 13:29

TLDR: Division by zero is not as scary as it’s made out to be:

a/0 = b ⟺ a = 0

Division is multiplication, backwards. These two equations are exactly equivalent, by definition:

a/c = b

a = b×c

It’s easy to understand division by zero if we look at the equivalent multiplication.

a/0 = b

a = b×0

For any real number b:

a = b×0 = 0

a = 0

There are two cases with division by zero:

If a = 0, then a/0 = b is unconstrained, any real number b satisfies the equation. You can discard such an equation which does not constraint the result.

If a ≠ 0 then a/0 = b is contradictory. There is no real number b which satisfies that equation. This is still useful to know; “there is no answer” can be a sort of meta-answer. For example if trying to solve a system of equations of static forces, “there is no answer” might mean you need to consider a different design for your bridge!

There is no need to consider advanced concepts such as limits in order to fully understand division.

In short, a/0 = b is true if and only if a = 0.

If you see such an equation a/0 = b, you may simplify it to a = 0.

a/0 = b ⟺ a = b×0 ⟺ a = 0

a/0 = b ⟺ a = 0

I posted this here about a year ago: http://matheducators.stackexchange.com/a/5667/3287



David Rowe: Give Us Our Daily Bread

Mon, 2015-11-16 10:29

Last week I visited a modern Australian farm on the Eyre Peninsula of South Australia, about 500km from where I live in Adelaide.

This farm has been in one family for several generations, and has steadily grown to 8000 acres (3200 hectares). This same area was previously farmed by 7 families, and now provides a livelihood for just one. This tells me that modern agriculture is super efficient, and explains why food (and calories) are super cheap for us here in the affluent Western world.

This is both good and bad. Given the right political conditions, science and technology enables us to feed the world. We don’t need to be hungry and can use those excess calories for other purposes. The jobs lost in one industry migrate to others. This farming family, for example, has spawned a variety of professionals that have left the family farm and done good things for the world.

It also brings diseases of affluence. Our poor bodies are not evolved to deal with an excess of food. We are evolved to be hunter-gatherers, constantly on the look out for the next calorie. Historically we haven’t had enough. So we are hard wired to eat too much. Hence the rise of heart disease and diabetes.

Breathtaking Array of Skills

I was impressed by the diverse array of skills required to run the farm. Business, animal husbandry, mechanical, agricultural science. The increased mechanisation means computers everywhere and I imagine robotics is on the horizon. During our visit they were measuring the moisture content of the crop to determine the best time to harvest. They even have an animal “retirement village” – they care for several old working dogs who had kept foxes away from the sheep for years.

Unlike many jobs, they don’t know what their yields and hence income will be from year to year. That’s a lot of risk in your annual income.

Overall, It takes about 12 years to learn the skills needed to run a modern farm.

This farm produces 3,500 tonnes of wheat per year. Based on 13680 kJ/kg of wheat, and a person needing 8700 kJ/day, that’s enough to feed 15,000 people every year. From the work of one family farm. Wow.

Organic Farming

I asked them about organic farming. The bottom line is the yields would be halved. So double the prices for everything we eat. That may be fine if you are a rich Westerner but that is the line between life and death for someone in the developing world. Alternatively, it means using twice the land under cultivation for the same amount of food. Organic means starving poor people and goodbye rain forests.

Their use of pesticides is strictly monitored and all residues must be removed. They have modern, scientific methods of erosion control to manage the soil, and techniques to naturally fix nitrogen. Sustainability is being addressed right now by modern, scientific, farming.

In my opinion the organic food movement is a more about scientific illiteracy and marketing than health.

Wind Farming

On a nearby hill was a 75MW wind farm, part of many that have sprung up in South Australia over the past decade. I am quite proud that South Australia now averages 30% wind power. We are about to close down our last remaining coal power station.

In this case, the lucky farmer that owns the land leased for the wind turbines receives $100k per year in passive income. K-ching K-ching as the turbines rotate.

It’s incredible to think that for years there have been “rivers of energy” flowing over those hills. It took technology and the right economic conditions to reach up and pluck that energy out of the sky.

Michael Still: Mount Stranger one last time

Mon, 2015-11-16 07:27
This is the last walk in this series, which was just a pass through now that the rain has stopped to make sure that we hadn't left any markers or trash lying around after the Scout orienteering a week ago. This area has really grown on me -- I think most people stick to the path down by the river, whereas this whole area has nice terrain, plenty of gates through fences and is just fun to explore. I'm so lucky to have this so close to home.



Interactive map for this route.



Tags for this post: blog canberra bushwalk



Comment

Chris Smart: Btrfs RAID 6 on dm-crypt on Fedora 23

Sun, 2015-11-15 21:29

I’m building a NAS and given the spare drives I have at the moment, thought I’d have a play with Btrfs. Apparently RAID 6 is relatively safe now, so why not put it through its paces? As Btrfs doesn’t support encryption, I will need to build it on top of dm-crypt.

Boot drive:

  • /dev/sda

Data drives:

  • /dev/sdb
  • /dev/sdc
  • /dev/sdd
  • /dev/sde
  • /dev/sdf

I installed Fedora 23 Server onto /dev/sda and just went from there, opening a shell.

# Setup dm-crypt on each data drive

# and populate the crypttab file.

for x in b c d e f ; do

  cryptsetup luksFormat /dev/sd${x}

  UUID="$(cryptsetup luksUUID /dev/sd${x})"

  echo "luks-${UUID} UUID=${UUID} none" >> /etc/crypttab

done

 

# Rebuild the initial ramdisk with crypt support

echo "dracutmodules+=crypt" >> /etc/dracut.conf.d/crypt.conf

dracut -fv

 

# Verify that it now has my crypttab

lsinitrd /boot/initramfs-$(uname -r).img |grep crypttab

 

# Reboot and verify initramfs prompts to unlock the devices

reboot

 

# After boot, verify devices exist

ls -l /dev/mapper/luks*

OK, so now I have a bunch of encrypted disks, it’s time to put btrfs into action (note the label, btrfs_data):

# Get LUKS UUIDs and create btrfs raid filesystem

for x in b c d e f ; do

  DEVICES="${DEVICES} $(cryptsetup luksUUID /dev/sd${x}\

    |sed 's|^|/dev/mapper/luks-|g')"

done

mkfs.btrfs -L btrfs_data -m raid6 -d raid6 ${DEVICES}'

See all our current btrfs volumes:

btrfs fi show

Get the UUID of the filesystem so that we can create an entry in fstab, using the label we created before:

UUID=$(btrfs fi show btrfs_data |awk '{print $4}')

echo "UUID=${UUID} /mnt/btrfs_data btrfs noatime,subvolid=0 0 0"\

  >> /etc/fstab

Now, let’s create the mountpoint and mount the device:

mkdir /mnt/btrfs_data

mount -a

Check data usage:

btrfs filesystem df /mnt/btrfs_data/

This has mounted the root of the filesystem to /mnt/btrfs_data, however we can also create subvolumes. Let’s create one called “share” for shared network data:

btrfs subvolume create /mnt/btrfs_data/share

You can mount this specific volume directly, let’s add it to fstab:

echo "UUID=${UUID} /mnt/btrfs_share btrfs noatime,subvol=share 0 0"\

  >> /etc/fstab

mkdir /mnt/btrfs_share

mount -a

You can list and delete subvolumes:

btrfs subvolume list -p /mnt/btrfs_data/

btrfs subvolume delete /mnt/btrfs_data/share

Now I plugged in a few backup drives and started rsyncing a few TB across to the device. It seemed to work well!

There are lots of other things you can play with, like snapshots, compression, defragment, scrub (use checksums to repair corrupt data), rebalance (re-allocates blocks across devices) etc. You can even convert existing file systems with btrfs-convert command, and use rebalance to change the RAID level. Neat!

Then I thought I’d try the rebalance command just to see how that works with a RAID device. Given it’s a large device, I kicked it off and went to do something else. I returned to an unwakeable machine… hard-resetting, journalctl -b -1 told me this sad story:



Nov 14 06:03:42 localhost.localdomain kernel: ------------[ cut here ]------------

Nov 14 06:03:42 localhost.localdomain kernel: kernel BUG at fs/btrfs/extent-tree.c:1833!

Nov 14 06:03:42 localhost.localdomain kernel: invalid opcode: 0000 [#1] SMP

Nov 14 06:03:42 localhost.localdomain kernel: Modules linked in: fuse joydev synaptics_usb uas usb_storage rfcomm cmac nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtab

Nov 14 06:03:42 localhost.localdomain kernel: snd_soc_core snd_hda_codec rfkill snd_compress snd_hda_core snd_pcm_dmaengine ac97_bus snd_hwdep snd_seq snd_seq_device snd_pcm mei_me dw_dmac i2c_designware_platform snd_timer snd_soc_sst_a

Nov 14 06:03:42 localhost.localdomain kernel: CPU: 0 PID: 6274 Comm: btrfs Not tainted 4.2.5-300.fc23.x86_64 #1

Nov 14 06:03:42 localhost.localdomain kernel: Hardware name: Gigabyte Technology Co., Ltd. Z97N-WIFI/Z97N-WIFI, BIOS F5 12/08/2014

Nov 14 06:03:42 localhost.localdomain kernel: task: ffff88006fd69d80 ti: ffff88000e344000 task.ti: ffff88000e344000

Nov 14 06:03:42 localhost.localdomain kernel: RIP: 0010:[] [] insert_inline_extent_backref+0xe7/0xf0 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: RSP: 0018:ffff88000e3476a8 EFLAGS: 00010293

Nov 14 06:03:42 localhost.localdomain kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000

Nov 14 06:03:42 localhost.localdomain kernel: RDX: ffff880000000000 RSI: 0000000000000001 RDI: 0000000000000000

Nov 14 06:03:42 localhost.localdomain kernel: RBP: ffff88000e347728 R08: 0000000000004000 R09: ffff88000e3475a0

Nov 14 06:03:42 localhost.localdomain kernel: R10: 0000000000000000 R11: 0000000000000002 R12: ffff88021522f000

Nov 14 06:03:42 localhost.localdomain kernel: R13: ffff88013f868480 R14: 0000000000000000 R15: 0000000000000000

Nov 14 06:03:42 localhost.localdomain kernel: FS: 00007f66268a08c0(0000) GS:ffff88021fa00000(0000) knlGS:0000000000000000

Nov 14 06:03:42 localhost.localdomain kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

Nov 14 06:03:42 localhost.localdomain kernel: CR2: 000055a79c7e6fd0 CR3: 00000000576ce000 CR4: 00000000001406f0

Nov 14 06:03:42 localhost.localdomain kernel: Stack:

Nov 14 06:03:42 localhost.localdomain kernel: 0000000000000000 0000000000000005 0000000000000001 0000000000000000

Nov 14 06:03:42 localhost.localdomain kernel: 0000000000000001 ffffffff81200176 0000000000270026 ffffffffa0925d4a

Nov 14 06:03:42 localhost.localdomain kernel: 0000000000002158 00000000a7c0ba4c ffff88021522d800 0000000000000000

Nov 14 06:03:42 localhost.localdomain kernel: Call Trace:

Nov 14 06:03:42 localhost.localdomain kernel: [] ? kmem_cache_alloc+0x1d6/0x210

Nov 14 06:03:42 localhost.localdomain kernel: [] ? btrfs_alloc_path+0x1a/0x20 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] __btrfs_inc_extent_ref.isra.52+0xa9/0x270 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] __btrfs_run_delayed_refs+0xc84/0x1080 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_run_delayed_refs.part.73+0x74/0x270 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] ? btrfs_release_path+0x2b/0xa0 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_run_delayed_refs+0x15/0x20 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_commit_transaction+0x56/0xad0 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] prepare_to_merge+0x1fe/0x210 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] relocate_block_group+0x25e/0x6b0 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_relocate_block_group+0x1ca/0x2c0 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_relocate_chunk.isra.39+0x3e/0xb0 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_balance+0x9c4/0xf80 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_ioctl_balance+0x3c4/0x3d0 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_ioctl+0x541/0x2750 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: [] ? lru_cache_add+0x1c/0x50

Nov 14 06:03:42 localhost.localdomain kernel: [] ? lru_cache_add_active_or_unevictable+0x32/0xd0

Nov 14 06:03:42 localhost.localdomain kernel: [] ? handle_mm_fault+0xc8a/0x17d0

Nov 14 06:03:42 localhost.localdomain kernel: [] ? cp_new_stat+0xb3/0x190

Nov 14 06:03:42 localhost.localdomain kernel: [] do_vfs_ioctl+0x295/0x470

Nov 14 06:03:42 localhost.localdomain kernel: [] ? selinux_file_ioctl+0x4d/0xc0

Nov 14 06:03:42 localhost.localdomain kernel: [] SyS_ioctl+0x79/0x90

Nov 14 06:03:42 localhost.localdomain kernel: [] ? do_page_fault+0x2f/0x80

Nov 14 06:03:42 localhost.localdomain kernel: [] entry_SYSCALL_64_fastpath+0x12/0x71

Nov 14 06:03:42 localhost.localdomain kernel: Code: 10 49 89 d9 48 8b 55 c0 4c 89 7c 24 10 4c 89 f1 4c 89 ee 4c 89 e7 89 44 24 08 48 8b 45 20 48 89 04 24 e8 5d d5 ff ff 31 c0 eb ac <0f> 0b e8 92 b7 76 e0 66 90 0f 1f 44 00 00 55 48 89 e5

Nov 14 06:03:42 localhost.localdomain kernel: RIP [] insert_inline_extent_backref+0xe7/0xf0 [btrfs]

Nov 14 06:03:42 localhost.localdomain kernel: RSP

Nov 14 06:03:42 localhost.localdomain kernel: ---[ end trace 63b75c57d2feac56 ]---

Bummer!

Looks like rebalance has a major bug at the moment. I did a search and others have the same problem, looks like I’m hitting this bug. I’ve reported it on Fedora Bugzilla.

Anyway, so I won’t do a rebalance at the moment, but other than that, btrfs seems pretty neat. I will make sure I keep my backups up-to-date though, just in case…

Steven Hanley: [mtb] Around the K 2013 - Cold morning and night lap of Kosci

Sun, 2015-11-15 11:25


The awesome open views heading toward Kiandra (fullsize)

Like the other Round the K galleries, another great day out on road bikes, this was the first time I had made it all the way around the loop too. The photo I am using to the left here is a great example of the open alpine regions neat Kiandra, those who have only done the Jindabyne - Cabramurra section have missed out on this bit of riding.

Gallery from the day is online Around The K 2012 gallery and as I said in the last few links to Round the K, bring on the next one in a few weeks. I am as this appears out competing in Triple Tri in pairs though so wrote the post ahead of time and am letting it appear during the day, unlikely that it matters as I doubt I have many readers.

And I have just noticed as I went to do an entry for Monday 2015-11-16 that I had in fact already posted the link and a photo for the post today. Oh well laziness is an artform so it is staying here.

Steven Hanley: [mtb] Blue Mountains Six foot/TNF100 scouting trip Feb 2014

Sat, 2015-11-14 22:25


Jane loving the trail run down to Coxs river (fullsize)

Oops I realised I forgot to link to this one in my reverse posting of all these adventures, this was a weekend Jane and I headed up to the Blue Mountains for some running and to scout out the Six Foot course (made Jane more comfortable on the course (and as she finished 2nd in the race it probably helped)) and for me we were able to do the climb up Furber Steps (and a nice run along Federal pass including the giant stair case descent).

Though I did get to climb the steps in the Mt Solitary Ultra I had not at that time planned to do that race so I was happy to see them for TNF100 prep. We had a good weekend up there and it was nice to have a relaxed run to the river and back, we managed to see a number of people out for a Fat Arse run on the course too. My gallery from my Blue Mountains weekend in Feb 2014 is online here, thanks for the company Jane, hope to see you back on the trails soon.

Francois Marier: How Tracking Protection works in Firefox

Sat, 2015-11-14 07:42

Firefox 42, which was released last week, introduced a new feature in its Private Browsing mode: tracking protection.

If you are interested in how this list is put together and then used in Firefox, this post is for you.

Safe Browsing lists

There are many possible ways to download URL lists to the browser and check against that list before loading anything. One of those is already implemented as part of our malware and phishing protection. It uses the Safe Browsing v2.2 protocol.

In a nutshell, the way that this works is that each URL on the block list is hashed (using SHA-256) and then that list of hashes is downloaded by Firefox and stored into a data structure on disk:

  • ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozstd-track* on Linux
  • ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozstd-track* on Mac
  • C:\Users\XXXX\AppData\Local\mozilla\firefox\profiles\XXXX\safebrowsing\mozstd-track* on Windows

This sbdbdump script can be used to extract the hashes contained in these files and will output something like this:

$ ~/sbdbdump/dump.py -v . - Reading sbstore: mozstd-track-digest256 [mozstd-track-digest256] magic 1231AF3B Version 3 NumAddChunk: 1 NumSubChunk: 0 NumAddPrefix: 0 NumSubPrefix: 0 NumAddComplete: 1696 NumSubComplete: 0 [mozstd-track-digest256] AddChunks: 1445465225 [mozstd-track-digest256] SubChunks: ... [mozstd-track-digest256] addComplete[chunk:1445465225] e48768b0ce59561e5bc141a52061dd45524e75b66cad7d59dd92e4307625bdc5 ... [mozstd-track-digest256] MD5: 81a8becb0903de19351427b24921a772

The name of the blocklist being dumped here (mozstd-track-digest256) is set in the urlclassifier.trackingTable preference which you can find in about:config. The most important part of the output shown above is the addComplete line which contains a hash that we will see again in a later section.

List lookups

Once it's time to load a resource, Firefox hashes the URL, as well as a few variations of it, and then looks for it in the local lists.

If there's no match, then the load proceeds. If there's a match, then we do an additional check against a pairwise allowlist.

The pairwise allowlist (hardcoded in the urlclassifier.trackingWhitelistTable pref) is designed to encode what we call "entity relationships". The list groups related domains together for the purpose of checking whether a load is first or third party (e.g. twitter.com and twimg.com both belong to the same entity).

Entries on this list (named mozstd-trackwhite-digest256) look like this:

twitter.com/?resource=twimg.com

which translates to "if you're on the twitter.com site, then don't block resources from twimg.com.

If there's a match on the second list, we don't block the load. It's only when we get a match on the first list and not the second one that we go ahead and cancel the network load.

If you visit our test page, you will see tracking protection in action with a shield icon in the URL bar. Opening the developer tool console will expose the URL of the resource that was blocked:

The resource at "https://trackertest.org/tracker.js" was blocked because tracking protection is enabled.

Creating the lists

The blocklist is created by Disconnect according to their definition of tracking.

The Disconnect list is on their Github page, but the copy we use in Firefox is the copy we have in our own repository. Similarly the Disconnect entity list is from here but our copy is in our repository. Should you wish to be notified of any changes to the lists, you can simply subscribe to this Atom feed.

To convert this JSON-formatted list into the binary format needed by the Safe Browsing code, we run a custom list generation script whenever the list changes on GitHub.

If you run that script locally using the same configuration as our server stack, you can see the conversion from the original list to the binary hashes.

Here's a sample entry from the mozstd-track-digest256.log file:

[m] twimg.com >> twimg.com/ [canonicalized] twimg.com/ [hash] e48768b0ce59561e5bc141a52061dd45524e75b66cad7d59dd92e4307625bdc5

and one from mozstd-trackwhite-digest256.log:

[entity] Twitter >> (canonicalized) twitter.com/?resource=twimg.com, hash a8e9e3456f46dbe49551c7da3860f64393d8f9d96f42b5ae86927722467577df

This in combination with the sbdbdump script mentioned earlier, will allow you to audit the contents of the local lists.

Serving the lists

The way that the binary lists are served to Firefox is through a custom server component written by Mozilla: shavar.

Every hour, Firefox requests updates from shavar.services.mozilla.com. If new data is available, then the whole list is downloaded again. Otherwise, all it receives in return is an empty 204 response.

To replicate how Firefox downloads the list, you can use this download script to ask the server for a copy of the full TP list:

$ ./download-list.py n:3600 i:mozstd-track-digest256 u:tracking-protection.cdn.mozilla.net/mozstd-track-digest256/1445465225

and then follow the URL redirection to get the actual list payload from the CDN:

$ wget https://tracking-protection.cdn.mozilla.net/mozstd-track-digest256/1445465225

Once you've downloaded that binary file, you can examine its content using this extractor script:

$ ./redirect-response-extractor.py 1445465225 Parsing a 54294-byte response file Processing control line... Add chunk 1445465225 contains 54272 bytes of 32-byte hashes Found 1696 prefixes in 54272 bytes

and dump all of the hashes it contains using the --verbose argument:

$ ./redirect-response-extractor.py 1445465225 Parsing a 54294-byte response file Processing control line... Add chunk 1445465225 contains 54272 bytes of 32-byte hashes 35e032660edb921c0c0ce59bfa289dc5a84c71b99584b359d74d6b03d00de66f 532239bcc9edf7681023070798bee5ec5e4a6bc7c0bb68e1e8e9099e45fdff94 52c058e95fc8d0e51bb9dd4b72f1364aa471157475a8435daa71e8e1c9533615 ... e48768b0ce59561e5bc141a52061dd45524e75b66cad7d59dd92e4307625bdc5 ... 8a565d247c08ff7fd0950d8a1f37bf2da29eae4a0dd65126d87a0db7cab4b400 ca705fed923ab66d6d8bfe0f65359a4b872981be5bcc1364e29aac69375af323 7fc983ea552f7c8d153fc308d621eb4f52e84aa63ecccf3a735698a11a2a4a8d Found 1696 prefixes in 54272 bytes

which, as I have highlighted, contains the twimg.com hash we have seen earlier.

Should you want to play with the server backend and run your own instance, follow the installation instructions and then go into about:config to change these preferences to point to your own instance:

browser.trackingprotection.gethashURL browser.trackingprotection.updateURL

Note that on Firefox 43 and later, these prefs have been renamed to:

browser.safebrowsing.provider.mozilla.gethashURL browser.safebrowsing.provider.mozilla.updateURL Learn more

If you want to learn more about how tracking protection works in Firefox, you can find all of the technical details on the Mozilla wiki or you can ask questions on our mailing list.

Thanks to Tanvi Vyas for reviewing a draft of this post.

Binh Nguyen: China Background, Economic Warfare, and More

Sat, 2015-11-14 00:56
- the world feels very different when you get perspectives from all over the world... if you were to simply watch the local news you'd think that the Chinese and Russians were right on our border and were ready to invade us. The other problem is that due to the language problem we only get a snippet of what they intend to say. Younger people in China aren't much different from us and censorship

Steven Hanley: [mtb] Happy Jacks and Jagungal wilderness ride on NYE 2012

Fri, 2015-11-13 14:25


David, Julie and Alex with Jagungal in the background (fullsize)

Alex and I had been thinking we should head up to near Jagungal and check out some of the trails through the wilderness there to ensure bikes could get through. This to ensure our planned route through for the next Canberra to Kosci Ride would work better than the previous one.

We decided to do a new years eve mtb ride on our single speeds in the wilderness around Mt Jagungal (the northern most mountain in Australia over 2000 metres). David and Julie came along for the fun, though I did not have my camera I was able to get my phone out easily through the day and get some good photos. They are all online in my Happy Jacks Jagungal Ride Album. Nice day out and and awesome way to finish off the year even though we arrived back in canberra tired and ready for sleep around midnight.

Steven Hanley: [mtb] ARNuts Victorian Cycling Holiday 2013

Thu, 2015-11-12 12:25


The boys on the trip overlooking Halls Gap (fullsize)

As I say i the write up, late in 2012 Alex, Lib and I started plotting a week of cycling in Victoria. Inspired to some extent by a cycling tips article, we added in mtb riding to the plan and decided to ask Bleeksie, Brooke and Aaron along for the fun.

We klicked it off with the Otway Odyssey mtb race and then did a bunch of awwesome mtb and road rides aorund the state. I was rather impressed with the grampians having never been there. Such a great week on bikes.

Photos and a few words from the trip are online in the gallery Victorian Epic Cycling Holiday February 2013. Now we just need to work out the next such trip. I suspect a south east Queensland cycling trip could be the go for some time in the future. I know there is a bunch of great riding up there.

Steven Hanley: [mtb] Whungee Wheengee Canyoning

Wed, 2015-11-11 13:25


A more open section later in the canyoning day (fullsize)

The previous ARNuts canyoning trip had been based on Mark's birthday, this time we all got up there to celebrate Alex's birthday. Another fun day out in the Blue Mountains with the ARNuts.

My photos from the day are on my Whungee Wheengee Canyoning page. I should not find it so amusing but I do that when the guides end up with a group of people who have fun in long AR events or 100KM runs and similar it seems to be a bit of a shock to them as they are used to clients that struggle with the hikes and harder bits rather than clients who are fitter than most people and simply get on with the harder bits having more fun the harder it gets.

Colin Charles: Ubuntu Online Summit: MySQL & Variants in 16.04

Tue, 2015-11-10 17:25

I personally have always enjoyed the Ubuntu Developer Summits (UDS), but nowadays they have been converted to the Ubuntu Online Summits (UOS). Attending them is not always convenient (timezone issues, might be travelling, etc.) so I watched the recorded video of a session I was interested in: MySQL & Variants in 16.04.

My key takeaways
  1. Ubuntu 16.04 Xenial Xerus is an LTS release.
  2. The term “cross-grade” is used a lot (it is not about downgrading/upgrading, but being able to use MySQL or MariaDB or Percona Server interchangeably)
  3. It would be nice to see MySQL 5.7 in this release (for Xenial as well as Debian Stretch). From Oracle there is a new packager taking over the task (Lars)
  4. MySQL 5.5 is still the default in Debian, and there needs to be upgrades tested between 5.5 to 5.7 (it looks like the ideal jump is that Ubuntu will not be seeing MySQL 5.6)
  5. Percona Server 5.7 is 60-90 days out; xtrabackup has had some new modifications and deserves an upgrade
  6. Boost is a new requirement for MySQL 5.7 & Percona Server 5.7; some old TokuDB problems in the builds are likely already fixed in MariaDB Server so this can be inherited
  7. MariaDB is waiting to iron out the bugs in 10.0, and may stick to that
My “raw” transcribed notes
  • Attendees:

    • Jon Grimm (Engineering Director for Ubuntu)
    • Robie Basak (Ubuntu)
    • Otto Kekäläinen (MariaDB Foundation)
    • Lars Tangvald, Norvald H. Ryeng (Oracle)
    • George Ormond Lorch III (Percona)
  • Robie: Waiting in Debian for a transition slot from MySQL 5.5 to MySQL 5.6. There’s some discussion with bugs, re: Akonadi, need to also resolve ABI issues with MySQL 5.6. Not really discussed MySQL 5.7 yet.

  • Norvald: 5.7, changes to installation. Client library ABI cleaned up. There may be some clients breaking because of that. No more exported symbols. See: The Client Library, Part 1: The API, the Whole API and Nothing but the API & The Client Library, Part 2: The Version Number
  • mysql_install_db is now replaced by --initialize in the server, so have to rewrite the post-install scripts. Might also have some AppArmour changes. Spoke to people @ DebConf (so best place is to put AppArmour profiles upstream (i.e. in mysql) and Debian and other distros will get it from there). AppArmour profile is in the MySQL source package now. Probably can get away with doing everything as cmake variables.
  • MySQL 5.7 has disabled the old password hashing algorithm, so if people haven’t upgraded they might have problems; so a manual intervention to fix their accounts.
  • Going from MySQL 5.7 to MySQL 5.6? It is done by dump and restore. There is no testing automated downgrades. Are there disk format changes? Norvald is not aware of any. If you use virtual columns in 5.7, you can’t downgrade easily to 5.6.
  • Robie would prefer to not release 5.6 and 5.7 concurrently. During Trusty, there was some level of user confusion. Debian – release team would prefer to see one transfer than two, so is it better to just do a single transition to 5.7?
  • Norvald says there hasn’t been testing from 5.5 -> 5.7. They only support upgrades from 5.5 -> 5.6 -> 5.7. For Ubuntu the choice can be to have 5.6 and then later do 5.7, but Jessie only just released with 5.5, so Stretch with 5.6 might not be a great idea (so users migrating from Jessie to Stretch will go from 5.5 to 5.7). Could also have 5.7 depend on a stripped 5.6 binary (like the embedded server; this is for localhost and the security team shouldn’t be too annoyed) for people to do an upgrade. Norvald says this has not been tried and there needs to be a migration path tested from 5.5 -> 5.7.
  • Conclusion: 5.7 in Stretch. Xenial is an LTS release, and 5.7 should be targeted for that.
  • If the maintainer script fails (postinstall script fails – don’t leave apt in a weird state). If it fails then upgrades, leave a debconf critical notice to say that the service is disabled and then fix it manually. Otto says that leaving /etc in a broken state is terrible, so we should avoid it.
  • Do we (Oracle) have the resources for 5.7 packaging and how soon can it be done in time for Xenial? There were patches from Lars in the git tree, but there haven’t been more recently. Lars will take over the 5.7 transition so if there is a list of work items, this will be settled (Lars will take over from Norvald).
  • There will be a separate session with Norvald/Lars/Robie outside of UOS about 5.7. Defer the Boost conversation after the session as well.
  • George: Percona is mainly looking out towards the 5.7 work and what kind of resources that will be put to that. There are new folk @ Percona to help with this. Percona inherits so much from the upstream codebase, it just works for Percona Server. There is Percona XtraDB Cluster and Percona xtrabackup, and xtrabackup has moved on quite a bit since the last upload (since last November 2014). So might be good idea to look at a refresh. There has also been a lot of work done on Percona XtraDB Cluster and there are some developments with Codership, so they are unsure if they will have their own Percona XtraDB Cluster 5.7 by the time Ubuntu is supposed to ship. When Percona is ready for something, just give Robie a shout to ensure that things happen. 60-90 days before a Percona Server 5.7 release. Just be aware of feature freeze for Xenial.
  • Norvald mentions that Percona Server 5.7 will also depend on Boost and there needs to be a decision on this. George mentions that TokuDB is now part of Percona Server, and it has some of its own requirements as well. Do we include TokuDB? It has requirements like it will only run on 64-bit platforms. Things to figure out going forward? MariaDB has been carrying TokuDB last November, but Robie remembers disabling it in Ubuntu. George says there were some licensing issues back then but they seem to be taken care of.
  • Otto says the builds for TokuDB was failing. It has a dependency on jemalloc, and that might have been the reason there were failures (says George). There may be something else where it doesn’t build on Ubuntu builders. But Otto says that there was a commit where this got fixed about last month. George will follow on, just to absorb it, since the legwork is already complete.
  • Otto: Trusty has 5.5, and Jessie and all other Ubuntu releases have 10.0, and 10.1 was released last month and I’m not quite pushing it to Debian quite yet. Fix 10.0 build fixes, upstream them, then only focus on 10.1. Blocking? (last summer) 5.6 is not in testing, so could not depend on it/changes done in 5.6 mysql-common. Here’s hoping that mysql-common going forward will be generated separately.
  • Robie will take an action to resolve the delta (probably just drop it). To sync MariaDB 10.0 to Xenial.
  • Discussion on /var/lib/mysql/*.flag thing on the list — conclusion at: mailing list — goal: within a single Ubuntu release, people can “cross-grade” between MySQL variants. The goal is to support all 3, and users want to try them, and thats when the bug reports come. Robie’s goal: move to a per-variant data directory. Otto says that once directory names change, 3rd party tools might have breakage. So a working prototype. Migration path is difficult. Maybe the best is to turn /var/lib/mysql into a symlink and store the data elsewhere. PostgreSQL does per version directories today; so studying that is going to happen.

Steven Hanley: [mtb/events] Sri Chinmoy Trail Ultra 2013 - 100 KM (first year)

Tue, 2015-11-10 16:25


At the start line (fullsize)

This was the first year Sri Chinmoy had run this Ultra, an event now in the third year (and I again ran it this year in a team and once more doing 2 legs). In 2013 they had not yet decided on the increase by 1km every year plan, however the course already showed off Canberra really well.

Alex and I decided to Alternate legs, in retrospect probably harder than doing 2 in a row each, however we had a good day out, and this year I got to do 2 in a row to compare. Of course I had my camera out there and took a bunch of photos which are online in my Sri Chinmoy 100 2013 gallery.

Steven Hanley: [mtb] Yell for Cadel, Australia's best ever XC mountain biker won the tour!

Tue, 2015-11-10 16:25
I know it has been a few weeks, however I have not exactly been on a blogging rampage, what with having my first ever month of no entries here. However I should start writing again and this is something of note for sure. How exciting it is that Cadel Evans won the tour!

I have been a fan of Cadel for a while, I guess since reading mtb magazines through out the 90s and marshaling at the mtb National Championship races in Majura Pines in Canberra when he won the title here. That he won the MTB World Cup series for two years in a row, has also won the Road World Cup series two years running, won the Road World Champs and now the Tour de France it is fairly obvious to all he is the most complete successful cyclist Australia has ever produced.

I still remember watching him lead through some of the single track at Majura in 1997 from where I was marshaling, seeming to be riding on smooth pavement through sections I rattle and bounce over, sure it was a shame when he left mountain biking, I am after all a mountain biker at heart, but there were as we all know bigger achievements in his future, there is nothing in the mtb world that could possibly excite a nation the way he has the last few years.

For the entire tour this year Cadel and his team seemed to be well organised, know what they were doing and went about everything the right way. Leopard also dealt with the race well, thus as Andy Schleck has said it definitely seems the best rider did indeed win this year. The final time trial was an incredible hour of viewing, seeing Cadel so focused and confidant at the start and then he almost won the stage and blasted away everyone else. Of course seeing him get air on a time trial bike was pretty cool too.

I really hope this helps move Australian's recognition of bikes and cycling forward, the reception for Cadel in Melbourne on Friday was awesome, with St Kilda rd lined 5 deep on each side all the way along and then Federation square packed so full along with all of us watching who did not make it down. The media coverage across the board has been positive and pretty good. Now we can all hope for a repeat performance next year. Rock on Cadel.