Planet Linux Australia

Syndicate content
Planet Linux Australia -
Updated: 1 hour 19 min ago

sthbrx - a POWER technical blog: Docker: Just Stop Using AUFS

Fri, 2015-10-30 15:06

Docker’s default storage driver on most Ubuntu installs is AUFS.

Don’t use it. Use Overlay instead. Here’s why.

First, some background. I’m testing the performance of the basic LAMP stack on POWER. (LAMP is Linux + Apache + MySQL/MariaDB + PHP, by the way.) To do more reliable and repeatable tests, I do my builds and tests in Docker containers. (See my previous post for more info.)

Each test downloads the source of Apache, MariaDB and PHP, and builds them. This should be quick: the POWER8 system I’m building on has 160 hardware threads and 128 GB of memory. But I was finding that it was only just keeping pace with a 2 core Intel VM on BlueMix.

Why? Well, my first point of call was to observe a compilation under top. The header is below.

Over 70% of CPU time is spent in the kernel?! That’s weird. Let’s dig deeper.

My next port of call for analysis of CPU-bound workloads is perf. perf top reports astounding quantities of time in spin-locks:

perf top -g gives us some more information: the time is in system calls. open() and stat() are the key culprits, and we can see a number of file system functions are in play in the call-chains of the spinlocks.

Why are open and stat slow? Well, I know that the files are on an AUFS mount. (docker info will tell you what you’re using if you’re not sure.) So, being something of a kernel hacker, I set out to find out why. This did not go well. AUFS isn’t upstream, it’s a separate patch set. Distros have been trying to deprecate it for years. Indeed, RHEL doesn’t ship it. (To it’s credit, Docker seems to be trying to move away from it.)

Wanting to avoid the minor nightmare that is an out-of-tree patchset, I looked at other storage drivers for Docker. This presentation is particularly good. My choices are pretty simple: AUFS, btrfs, device-mapper or Overlay. Overlay was an obvious choice: it doesn’t need me to set up device mapper on a cloud VM, or reformat things as btrfs.

It’s also easy to set up on Ubuntu:

  • export/save any docker containers you care about.

  • add --storage-driver=overlay option to DOCKER_OPTS in /etc/default/docker, and restart docker (service docker restart)

  • import/load the containters you exported

  • verify that things work, then clear away your old storage directory (/var/lib/docker/aufs).

Having moved my base container across, I set off another build.

The first thing I noticed is that images are much slower to create with Overlay. But once that finishes, and a compile starts, things run much better:

The compiles went from taking painfully long to astonishingly fast. Winning.

So in conclusion:

  • If you use Docker for something that involves open()ing or stat()ing files

  • If you want your machine to do real work, rather than spin in spinlocks

  • If you want to use code that’s upstream and thus much better supported

  • If you want something less disruptive than the btrfs or dm storage drivers

…then drop AUFS and switch to Overlay today.

James Purser: Do you suffer from rage watching?

Fri, 2015-10-30 13:30

There is a terrible ailment sweeping the land. Sufferers find themselves compelled to watch, listen, read or generally consume media output that causes high blood pressure, anger and a desperate, overwhelming urge to tweet about how much they truly loathe the media thing they are consuming.

Rage watching.

There seems to be spikes of Rage Watching specifically around Monday nights at 9:30pm with smaller occurances occuring on Sunday mornings (replays on Sunday afternoons). More recently there has been an uptick of Rage Watching on Wednesday nights by people who feel it absolutely necessary to tell the world exactly how bad the ABC show "Kitchen Cabinet" is for either a) Having  an evil person on as a guest or b) Not spending 22 minutes using Kitchin impliments to torture said evil person into confessing they are indeed an evil person and will do better from now on.

Why? Why do you watch these programmes if you know they're going to be terrible? You already know that you're not going to like either the show, or the person being interviewed, or in the case of the Bolt Report everything about it.

Instead be calm, turn off the tv, or switch on Netflix and binge watch your way through a series. Save your rage for when it is actually useful.

Blog Catagories: media

Sam Watkins: sswam

Thu, 2015-10-29 20:30

I wrote a simple program ramp-io, based on the redshift code, to read and write the xrandr gamma ramps for Linux / X11.  This enables me to define my own gamma ramps, and switch ramps quickly from the command line.  My preferred ramp is red-inv, dim inverse video with a low colour temperature (more red, less blue), and I set the LCD hardware brightness to maximum to reduce LED PWM flicker.  I find this is relatively easy on the eyes for work, compared to the normal glaring white backgrounds.

Linux Users of Victoria (LUV) Announce: LUV Main November 2015 Meeting: Computer Science and SELinux / Parallel Programming

Thu, 2015-10-29 12:29
Start: Nov 4 2015 18:30 End: Nov 4 2015 20:30 Start: Nov 4 2015 18:30 End: Nov 4 2015 20:30 Location: 

6th Floor, 200 Victoria St. Carlton VIC 3053


Please note that due to the Melbourne Cup this month's meeting is on Wednesday


• Russell Coker, Computer Science and SELinux

• Lev Lafayette, Parallel Programming

200 Victoria St. Carlton VIC 3053 (formerly the EPA building)

Late arrivals, please call (0490) 049 589 for access to the venue.

Before and/or after each meeting those who are interested are welcome to join other members for dinner. We are open to suggestions for a good place to eat near our venue. Maria's on Peel Street in North Melbourne is currently the most popular place to eat after meetings.

LUV would like to acknowledge Red Hat for their help in obtaining the venue and VPAC for hosting.

Linux Users of Victoria Inc. is an incorporated association, registration number A0040056C.

November 4, 2015 - 18:30

read more

Binh Nguyen: Defense Podcasts, MH17 Background, JSF Break-In, JSON Parsing, and More

Wed, 2015-10-28 20:00
- if you're interested in defense, intelligence, or geo-politics in general these soundcasts may be of interest to you. Obviously, they're US/Allied focused but they cover a wide range of affairs that face these areas. I may go through other countries at another time...[]=817

- if you've been watching the media lately you'll have realised that Russian seperatists seemed to have been implicated in the MH17 downing. If you actually go through all of the evidence (especially the hard evidence that is hard to fake. If you listen to any one side you can easily get caught up in their perspective and miss a few things) though things don't seem that clear cut and there are a lot of people who seem to be withholding (often crucial such as RADAR records, ability to access the crash site, debris/fragments from the site, etc...) evidence for some strange reason (or just missing some things which should be obvious?). Moreover, all parties involved have had a history of fabricating evidence (I wouldn't put it past Ukrainian or Russian forces planting evidence on the crash site) so I wouldn't necessarily believe whatever is finally said. Some theories have included: it could have been a 'false flag' operation to aide Russian justification for invasion of Ukraine, it could have been a 'false flag' operation to aide Ukraine justification for action against Russia, Ukraine air force operation which went extremely bad but actually does a good explanation of why the debris has such variability with regards to damage, it was a plain accident (with a lot of silliness involved all round), possible targeted assisination of Putin himself as his plane was in area at the time (about 100-200 km) and since his plane has similar markings it MH17 which could have meant it was mis-identified. Either way, if you go through the history of all parties you'll realise that all have a credibility problem...

Dutch Safety Board MH17 final report (FULL VIDEO)

Dutch Safety Board simulates MH17 being hit by BUK missile

Former CIA Analyst Ray McGovern on Who Shot Down Flight MH17 And Iran Nuclear Deal.

'MH17 crash' test simulation video: Il-86 plane cockpit hit with BUK missile

MH-17 - The Untold Story 

Flight MH17: Russia and its changing story

- the Russians state that type of missile isn't used by Russian forces, the pattern on the fuesalage doesn't completely replciate the same one as one that was later tested in one of the videos above (though some of this can be put down to the difference between a static and moving aircraft). Moreover, there while there was a supposed siting of a BUK SAM system in the area in question I'm wondering whether there haven't been more people who have come forward of evidence of before and after videos of it launching a rocket at MH17? Surely, with such a massive contrail heaps more people would have come forward with evidence indicating what was happening. Not ruling out the possibility some elements of government may have gotten involved here though to hush people up or that others were paid to tell a 'version of the truth'...

Flight MH17: searching for the truth

- either way, I doubt that we'll ever know the 'complete truth'. There will be some form of cover up because it feels like they either know what happened (and the truth is ugly) or they don't know and those who are guilty are with holding evidence. The problem is that everyone seems to be doing this to some extent so it is possible that 'a deal' may have been reached behind the scenes. My guess is that a lot of people simply stuffed up and they're partly trying to figure out the best way of apportioning blame...

- if you follow the defense/intelligence space you'll have noticed some strange happenings with regards to the JSF project. Apparently, they said that 50TB was stolen a while back but later they've said it was just ALIS and that it was just non-classified information (if you think that US intelligence/security is generally better think again. Look through enough background and you'll find that they are subject to the same limitations, problems, etc... that are faced by every other organisation. There have been some bizarre penetrations of even 'classified networks'...). Either way, I'd be very interested to know how much technology they've stolen, purchased, bribed from certain officials (based on what I've heard 'incentives' have been between 4-7 figures for information regarding stealth technologies and they've been able to procure quite a lot including information about RAM coatings, AESA RADAR, EOTS, DAS, avionics, engine design, etc...)(even with the downing of and purchase of some aircraft I'm guessing they've gained access to at least some AESA RADAR, EOTS, RAM coating, and engine technology?) and how much they've reverse engineered or is entirely native? Look at the design of some of their new stealth aircraft and some aspects seem incredibly crude... The other thing I'm curious about is if it was 50TB of genuine design material how much would Western design efforts of the JSF going to be thrown off?  Would they have to re-design or is the core system good enough? This is much like the question of security of obscurity (closed versus open source security) if you know anything about cybersecurity. Even if the stolen material was honeypot/honeynet material it has to be convincing enough to throw Chinese research off... which means it's still decent (possibly old?)...

China’s new counter-stealth radar JY-26

How China Steals U.S. Military Secrets 

Next Big Thing: China’s Aviation to Develop Long-Range Strike Bomber

Military Marvel: China Ready to Test Asia’s Largest Warship

- if you have to program regularly, you you have to read some pretty 'human unreadable' stuff at times. Some links regarding possible JSON parsers

echo '{"test":1,"test2":2}' | python -mjson.tool

Pandas are actually quite funny and peculiar animals if you read up about them...

Cute Alert!Clingy pandas don’t want to take their medicine

Clingy panda do not let zookeeper go

So Cute! Panda asks for hug to get down from tree!

Cute alert! Four baby pandas playing with zookeeper

Pandas addicted to hugging

Cute pandas playing on the slide

Some interesting quotes in the recent media:-“We’ve tried intervention and putting  down troops in Iraq,” he said. “We’ve tried invention without putting in troops in Libya. And we’ve tried no intervention at all but demanding regime change in Syria. It’s not clear to me that even if our policy did not work, subsequent policies have worked better.”

-“Whenever I’m asked this, I can say that I apologize for the fact that the intelligence we received was wrong,” Blair said. “Because even though he had used chemical weapons extensively against his own people, against others, the program in the form that we thought it was did not exist in the way that we thought. So I can apologize for that. I can also apologize, by the way, for some of the mistakes in planning and certainly our mistake in our understanding of what would happen once you removed the regime.”

-"Since 9/11, a near doubling of the Pentagon’s modernization accounts — more than $700 billion over 10 years in new spending on procurement, research and development — has resulted in relatively modest gains in actual military capability,” Defense Secretary Robert M. Gates said in an address last week.

- The United States remains the most powerful nation on earth. Yet from the immediate aftermath of the heady days of 1991 to the present, nations great and small have shown themselves unimpressed by or impervious to U.S. might. To the astonishment of many Americans, the United States, for all its power and its good intentions, has frequently failed in its efforts to lead the world, enforce its preferences or impose its will.

International relations scholars have long understood the fallacy of assuming that power routinely if not automatically provides the wherewithal to get others to do as one wishes. And yet there remains, among statesmen, politicians, policy analysts and the broader public—to say nothing of presidential candidates—an easy assumption of a correlation between a country’s overall power and its ability to persuade, entice, bribe or compel other countries to do its bidding, if not all the time, then at least when the stakes for the powerful country are sufficiently high.

- Atmar warns, "The symbiotic network of terrorists that we are confronted with is going to be a threat to every country in this region and by extension the whole world."

Obama has for years boasted of rendering al-Qaida toothless, but Atmar points out the U.S. withdrawal has reinvigorated the group founded by Osama bin Laden.

He also noted IS leader Abu Bakr al-Baghdadi's new role in choosing Taliban leaders, with the new caliphate in Iraq and Syria "commanding and controlling," "financing" and even providing the Taliban with a new strategy. "We have no doubt about that," Atmar said.

Facing an existential threat, you turn to those on whom you can depend. Right now, sadly, Putin is a better bet for Afghans than America.

- It’s all a lot to take in, and makes one wonder what G.D.P. really stands for: Generally Disorienting Predictions? Guesses Done Poorly?

“G.D.P. is accounting science built to supply a need to understand an economy’s direction,” said Steve Blitz, chief economist at ITG Investment Research. “Is there more art than science? In terms of filling in all the numbers where the answers are imputed rather than measured, the answer is yes.”

And imputed values, he added, are becoming more important as the service sector grows, while in developing nations, accurate measurement is more difficult for a variety of reasons.

Pro tip: “Whenever doing cross country G.D.P. comparisons, I have always used I.M.F. data,” Mr. Blitz said. “They scrub the data and reset so concepts are the same from country to country.” Point taken.

- In 1999, Saudi Prince Nayef bin Sultan bin Fawwaz Al Shaalan allegedly smuggled two tons of cocaine from Venezuela to France. Now believed to be living under legal shelter in Saudi Arabia, Prince Nayef was accused by France of using his diplomatic status to sneak the drugs onto a jet belonging to the Saudi royal family. He managed to escape his sentencing and was convicted in absentia in 2007. The United States also indicted him with conspiracy to distribute cocaine. 

In 2010, a leaked WikiLeaks cable described a royal underground party scene in Jeddah that was “thriving and throbbing” because Saudi officials looked the other way. The dispatch described a Halloween party, funded in part by a prince from the Al Thunayan family, where more than 150 young men and women dressed in costumes and slogged expensive alcohol, which is sold only on the black market in Saudi Arabia. “Though not witnessed directly at this event, cocaine and hashish use is common in these social circles,” the cable read.

The harsh punishments for violations of Saudi Arabia’s interpretation of sharia law tend not to apply to the some 15,000 princes and princesses who belong to the royal House of Saud. But that hasn’t stopped Riyadh from pursuing executions of foreigners and non-royal citizens accused of less egregious violations of the country’s drug laws.

- "What happens in Afghanistan really does have an impact on what happens over here," Sopko says. "Heroin use is on the rise in the U.S., and although the DEA says that most of the heroin here originates from South America, some still comes from Afghanistan. Our European allies have told me time and again their concerns about the amount of Afghan heroin reaching Europe. Heroin use is a problem in Canada, and 90 percent of Canada's heroin comes from Afghanistan." The Iraqi government has become increasingly suspicious of the US’ lack of real commitment in fighting ISIS. On the other hand, Russian strikes have thus far been so effective against ISIS that the Iraqi government has asked Russia to take on a bigger role against ISIS, than the US.

Russia has in turn signaled that it may start bombing ISIS in Iraq as well as Syria, with the permission of the Iraqi government. Unlike the US, Russia has not broken international law and has sought permission to enter Iraq and Syria from each respective state’s legitimate government.

With these actions Russia has called the US’s bluff on fighting ISIS, and is effectively forcing the US to do a better job of convincing the Iraqi government that it is truly fighting ISIS. If Russia does enter Iraqi airspace, it will more easily cross into Syrian airspace to provide supplies to the Syrian government, since the US has bullied many countries in the region to close their airspace to Russian aircrafts. Furthermore, if Iraq asks Russia to intervene it is a scenario that would reverse any of the influence the US had gained in Iraq, throughout its lengthy occupation of the country since 2003.

The US has been backed into a corner and in doing so, has exposed itself and its allies as the source of terrorism, not champions truly fighting it. Terrorism has always been a means by which the US has sought to deconstruct Russian spheres of influences. Ironically over the last decade it has also simultaneously perpetuated the myth that it is actually fighting a war against terror. However as its allied states grow increasingly tired of this game, how long can the US continue to juggle this duplicity, before the entire deck of cards crumbles?

- Financially, the war economy has largely replaced formal economic life. Incomes are increasingly conflict-dependent, whether it is through smuggling, selling weapons, kidnapping, even distributing aid. You can buy or rent a checkpoint for the day or for an hour. Hezbollah, for one, profits through control of checkpoints. Border control by armed groups is hugely lucrative. Fruitful earnings are made from forged documents such as passports and ID cards. The Syrian regime benefits from and encourages this trade, especially if it means opponents can flee abroad.

- “Iran’s nuclear problem has been solved. From Iran, there is no threat and there never had been,” Mr. Putin said. “The only reason that was used by U.S.—to start building the Missile Defense Shield—disappeared. We [Russia] might have expected that a system of MDS development to be halted.”

Mr. Putin believes the United States lied to Russia and the world on the threat of nuclear danger coming from Iran.

“Some days ago, the first tests of USA’s MDS were conducted in Europe. What does that mean? It means that when we were arguing with our American partners we were right. Russia was right from the beginning that the American Missile Defense Shield program was being developed with the goal to destroy strategic balance and to have a way to dictate her power to everyone. They were trying to deceive us, and the whole world, once again. And, to put it simply, we were lied to.”

- To the notion of America helping the “moderate opposition” in Syria, Mr. Putin responded that the division of “moderate” and “non-moderate” leads to the empowerment of Islamic terrorists.

“We shouldn’t play with words here and divide the terrorists into moderate and non-moderate,” Mr. Putin said. “The difference, according to the ‘specialists’ [a jab to the Obama advisors], seems to be that ‘moderate’ bandits behead people softly.”

- A trio of young Muslim women have been conning ISIS by setting up fake social media accounts and getting the terror group to send them money to travel to Syria to become jihadi brides, according to police.

Once the terror group wired funds to the con artists, from Chechnya, they allegedly deleted the accounts and pocketed the money.

ISIS uses social media to encourage men and women to travel to the lands controlled by ISIS to become fighters and jihadi brides. The Republic of Chechnya is a federal subject of Russia and mainly Muslim.

Now the trio have been detained by Chechen e-crime police for the scam, which has so far netted them more than £2,000, Moskovsky Komsomolets newspaper reported.

“I don’t recall any precedent like this one in Chechnya, probably because nobody digs deep enough in that direction,” officer Valery Zolotaryov told the newspaper.

Colin Charles: Sunsetting HPCloud, whom contributed to making MySQL better

Wed, 2015-10-28 10:25

Recently at Percona Live Amsterdam I gave a talk titled Databases in the Hosted Cloud (I’m told I got a 4/5 rating for this talk). It was before AWS re:Invent, so obviously some of the details in the talk have changed. For one, now there is also Amazon RDS for MariaDB. But there has also been other changes, i.e. HP’s Public Cloud (HP Helion Public Cloud) will sunset January 31 2016.

That’s a slide from my deck. I basically have to caution users as to what’s going on in the cloud world when it comes to their databases. And this one slide shows news reports about HP possibly wanting to exit the cloud world back in April 2015. See: HP Comes to Terms With the Cloud, HP: We’re not leaving the public cloud, and of course the HP blog post from Bill Hilf: HP Helion Strategy to Deliver Hybrid IT Continues Strong.

The tune has of course changed in October 2015: A new model to deliver public cloud. I find this to be quite sad considering they were all very gung ho about pushing OpenStack forward several OSCONs ago. I know many people who made this happen (many ex-MySQL’ers went on to HP to work on OpenStack). I can only feel for them. I guess their important work continues in OpenStack as a whole and all this ends up being part of the HP Helion private cloud.

I think its also worth noting the improvements that Percona Server 5.5 received thanks to HPCloud to make it easier to manage in the cloud:

This pretty much leaves only Rackspace Cloud Databases as being a large OpenStack based offering of databases in the public cloud space, doesn’t it?

HPCloud offered 3 Availability Zones (AZs) per region, and had 2 regions — US-East (Virginia) and US-West. It’s worth remembering that US-West was the only place you could use the Relational DB MySQL service. You also got Percona Server 5.5. You enjoyed 50% off pricing while it was in public beta. 

All this is basically over. Here’s wishing the team well, a big thanks to them for helping make MySQL better and in case you’re looking for more articles to read: H-P Winds Down Cloud-Computing Project

Silvia Pfeiffer: My journey to Coviu

Tue, 2015-10-27 21:07

My new startup just released our MVP – this is the story of what got me here.

I love creating new applications that let people do their work better or in a manner that wasn’t possible before.

My first such passion was as a student intern when I built a system for a building and loan association’s monthly customer magazine. The group I worked with was managing their advertiser contacts through a set of paper cards and I wrote a dBase based system (yes, that long ago) that would manage their customer relationships. They loved it – until it got replaced by an SAP system that cost 100 times what I cost them, had really poor UX, and only gave them half the functionality. It was a corporate system with ongoing support, which made all the difference to them.

The story repeated itself with a CRM for my Uncle’s construction company, and with a resume and quotation management system for Accenture right after Uni, both of which I left behind when I decided to go into research.

Even as a PhD student, I never lost sight of challenges that people were facing and wanted to develop technology to overcome problems. The aim of my PhD thesis was to prepare for the oncoming onslaught of audio and video on the Internet (yes, this was 1994!) by developing algorithms to automatically extract and locate information in such files, which would enable users to structure, index and search such content.

Many of the use cases that we explored are now part of products or continue to be challenges: finding music that matches your preferences, identifying music or video pieces e.g. to count ads on the radio or to mark copyright infringement, or the automated creation of video summaries such as trailers.

This continued when I joined the CSIRO in Australia – I was working on segmenting speech into words or talk spurts since that would simplify captioning & subtitling, and on MPEG-7 which was a (slightly over-engineered) standard to structure metadata about audio and video.

In 2001 I had the idea of replicating the Web for videos: i.e. creating hyperlinked and searchable video-only experiences. We called it “Annodex” for annotated and indexed video and it needed full-screen hyperlinked video in browsers – man were we ahead of our time! It was my first step into standards, got several IETF RFCs to my name, and started my involvement with open codecs through Xiph.

Around the time that YouTube was founded in 2006, I founded Vquence – originally a video search company for the Web, but pivoted to a video metadata mining company. Vquence still exists and continues to sell its data to channel partners, but it lacks the user impact that has always driven my work.

As the video element started being developed for HTML5, I had to get involved. I contributed many use cases to the W3C, became a co-editor of the HTML5 spec and focused on video captioning with WebVTT while contracting to Mozilla and later to Google. We made huge progress and today the technology exists to publish video on the Web with captions, making the Web more inclusive for everybody. I contributed code to YouTube and Google Chrome, but was keen to make a bigger impact again.

The opportunity came when a couple of former CSIRO colleagues who now worked for NICTA approached me to get me interested in addressing new use cases for video conferencing in the context of WebRTC. We worked on a kiosk-style solution to service delivery for large service organisations, particularly targeting government. The emerging WebRTC standard posed many technical challenges that we addressed by building , by contributing to the standards, and registering bugs on the browsers.

Fast-forward through the development of a few further custom solutions for customers in health and education and we are starting to see patterns of need emerge. The core learning that we’ve come away with is that to get things done, you have to go beyond “talking heads” in a video call. It’s not just about seeing the other person, but much more about having a shared view of the things that need to be worked on and a shared way of interacting with them. Also, we learnt that the things that are being worked on are quite varied and may include multiple input cameras, digital documents, Web pages, applications, device data, controls, forms.

So we set out to build a solution that would enable productive remote collaboration to take place. It would need to provide an excellent user experience, it would need to be simple to work with, provide for the standard use cases out of the box, yet be architected to be extensible for specialised data sharing needs that we knew some of our customers had. It would need to be usable directly on, but also able to integrate with specialised applications that some of our customers were already using, such as the applications that they spend most of their time in (CRMs, practice management systems, learning management systems, team chat systems). It would need to require our customers to sign up, yet their clients to join a call without sign-up.

Collaboration is a big problem. People are continuing to get more comfortable with technology and are less and less inclined to travel distances just to get a service done. In a country as large as Australia, where 12% of the population lives in rural and remote areas, people may not even be able to travel distances, particularly to receive or provide recurring or specialised services, or to achieve work/life balance. To make the world a global village, we need to be able to work together better remotely.

The need for collaboration is being recognised by specialised Web applications already, such as the LiveShare feature of Invision for Designers, Codassium for pair programming, or the recently announced Dropbox Paper. Few go all the way to video – WebRTC is still regarded as a complicated feature to support.

With Coviu, we’d like to offer a collaboration feature to every Web app. We now have a Web app that provides a modern and beautifully designed collaboration interface. To enable other Web apps to integrate it, we are now developing an API. Integration may entail customisation of the data sharing part of Coviu – something Coviu has been designed for. How to replicate the data and keep it consistent when people collaborate remotely – that is where Coviu makes a difference.

We have started our journey and have just launched free signup to the Coviu base product, which allows individuals to own their own “room” (i.e. a fixed URL) in which to collaborate with others. A huge shout out goes to everyone in the Coviu team – a pretty amazing group of people – who have turned the app from an idea to reality. You are all awesome!

With Coviu you can share and annotate:

  • images (show your mum photos of your last holidays, or get feedback on an architecture diagram from a customer),
  • pdf files (give a presentation remotely, or walk a customer through a contract),
  • whiteboards (brainstorm with a colleague), and
  • share an application window (watch a YouTube video together, or work through your task list with your colleagues).

All of these are regarded as “shared documents” in Coviu and thus have zooming and annotations features and are listed in a document tray for ease of navigation.

This is just the beginning of how we want to make working together online more productive. Give it a go and let us know what you think.

Colin Charles: 3 Big Announcements from MariaDB (my take for Oct 2015)

Tue, 2015-10-27 03:25

Today I received about five emails with the subject: 3 Big Announcements from MariaDB. Maybe you did as well (else, read it online). October has brought on some very interest announcements, and I think my priority for the big announcements vary a little:

  1. MariaDB Server is now available on Amazon RDS – you wouldn’t believe how many people ask for this, as many now deploy using Amazon Web Services (AWS), so now that it is available, I consider this to be extremely amazing. You get 10.0.17 today, and within 3-5 months of a GA, you get the next release (the docs are a work of art — read them!).
  2. MariaDB Server 10.1 is now a stable GA — this is a milestone. Our last stable GA came out in March 2014. There are plenty of new features and we had a developer meeting to plan what comes in 10.2 as well. Remember to read: What is MariaDB 10.1?
  3. New XAMPP with MariaDB — The new XAMPP does not ship with MySQL any longer but MariaDB Server 10.0.17. This is going to help distribution tremendously as many people use XAMPP as a development environment (it is after all the most popular PHP development environment out there). Remember to get your downloads for Windows/Linux/OSX. 

I think the above are my highlights of 3 big announcements from the MariaDB world. What are you waiting for, download it now! And remember to report bugs/feature requests to our Jira instance.

Ben Martin: ESP8266 and a few pins

Mon, 2015-10-26 23:41
The new Arduino 1.6.x IDE makes it fairly simple to use the ESP8266 modules. I have been meaning to play around with a some open window detectors for a while now. I notice two dedicated GPIO pins on the ESP8266, which is one more than I really need. So I threw in an led which turns on when the window is open. Nothing like local, direct feedback that the device has detected the state of affairs. The reed switch is attached on an interrupt so as soon as the magnet gets too far away the light shines.

I will probably fold and make the interrupt set a flag so that the main loop can perform an http GET to tell the server as soon as it knows when a state has changed.

Probably the main annoying thing I've still got is that during boot it seems the state of both the gpio pins matters. So if the reed switch is closed when you first spply power then the esp goes into some stall state.

It will be interesting to see how easy OTA firmware updates are for the device.

Sridhar Dhanapalan: Twitter posts: 2015-10-19 to 2015-10-25

Mon, 2015-10-26 01:27

Hamish Taylor: The woeful state of communications in Australia’s capital city

Sun, 2015-10-25 21:27

For those who may not know, I recently moved from Melbourne, Victoria to Canberra, Australian Capital Territory (ACT) and am now living in a house in the inner north-west. Of course, being a geek, I wanted to get the internet connected as soon as possible! After such a smooth transition I’d expected some problems and this is where they all cropped up.

In Melbourne I had an Internode ADSL connection and before I moved I called them up to relocate this service. This, of course, relied on getting an active Telstra line at the new house. I knew it would take a bit of time to relocate the service, so in the interim I bought a Telstra wi-fi internet device. This is actually a ZTE MF30 and supports up to 5 connections via wi-fi, so I can get both my iPhone and laptop on at the same time. Quite simply, this device is brilliant at what it does and I couldn’t be happier with it.

So, at the moment I’m online via the Telstra device, which is just as well really, as I soon encounter communication issue number 1: Optus.

It appears that Optus have a woeful network in Canberra. I have an iPhone 3GS, which I know can only use 850MHz and 2100MHz 3G networks. Optus uses 900MHz and 2100MHz for their 3G, so the iPhone will only work in Optus 2100MHz coverage. In Melbourne I never had a problem getting on the internet at good speeds.

When I looked at the Optus overage maps for ACT and click on “3G Single band” (the 2100MHz network coverage), it shows the inner north-west being well covered. It really isn’t. Both from home and at work in Belconnen, I can barely get two bars of GSM phone signal. The connectivity is so bad that I can barely make phone calls and send SMSs. Occasionally, I get the “Searching…” message which tells me that it has completely lost GSM connectivity. This never happened in Melbourne, where I had 4-5 bars of signal pretty much all the time.

The 3G connection drops in and out so often that I have to be standing in exactly the right location to be able to access the internet on my iPhone. Even this afternoon in Kingston in the inner south, I wasn’t able to get onto the internet and post to Twitter. I had to use the Telstra device, which hasn’t missed a beat in any location for network connectivity, to establish a connection. This really isn’t good enough for the middle of Canberra. I am seriously considering calling Optus, lodging a complaint and trying to get out of my 2 year contract (which has another 10 months to run), so I can switch over to Telstra. I never thought I’d say this, but I actually want to use a Telstra service!!!

Communications issue number 2: TransACT. From what I can find out TransACT have a cable TV network which also has telephone and internet capabilities. When this network was established about a decade ago, it was revolutionary and competitive. Today the network has been expanded to support ADSL connections, but there is no ability to get a naked service as all connections require an active phone service. Additionally, as a quick look at some of the internet connectivity plans show, after factoring in the required phone service, it is a costly service for below average download allowances.

When I moved into the house, the process of relocating the Internode ADSL service from Melbourne to Canberra triggered a visit from a Telstra technician. However, he wasn’t able to find a physical Telstra line into the house. Being an older suburb of Canberra, this house will have a Telstra cable. Or rather will have had as apparently it is not unknown for TransACT installers to cut the Telstra cables out as “You won’t need THAT anymore!”

So now I have to pay for a new cable to be installed from the house to the “Telstra network boundary” (presumably the street or nearest light pole where it can be connected to Telstra’s infrastructure). Then we have to pay again for a new Telstra connection at a cost of $299. Considering that if the Telstra cable had been left in place, the connection cost would be $55, this is turning into quite an expensive proposition just to get a naked DSL service.

All in all I am not impressed with the state of communications in Australia’s capital city, Canberra. All I can say is please, please, please bring on the National Broadband Network (NBN)!



Colin Charles: Mark Callaghan at the Korean MySQL Power User Group

Sun, 2015-10-25 14:25

The Korean MySQL Power User Group gets a special guest speaker next weekend (Oct 31 2015 – 4pm – 4:33’s offices in Gangnam — nearest train stop is Samseong station, Line 2 — post requires Cafe Naver login) — Mark Callaghan (Small Datum, @markcallaghan, and formerly High Availability MySQL). I’ve been to many of their meetups, and I think this is a great opportunity for many DBAs to learn more about how Mark helps make MySQL and MongoDB better for users at Facebook. I’m sure he’ll also talk about RocksDB.

After that, as usual, there will be a DBA Dinner. This time the tab gets picked up by OSS Korea. See you next Saturday — Halloween in Seoul will have added spice!

Colin Charles: MySQL NL Community Meetup with MariaDB speakers summary

Sat, 2015-10-24 22:25

Last week we had the MySQL Meetup with MariaDB Developers in Amsterdam, which went on easily for about 3.5 hours. Thanks to all for listening (these were lightning talks, not with a strict 5 minute clock with Q&A thrown in), and Daniël van Eeden for organising this at the eBay offices (whom kindly provided pizza, beer and soft drinks as well). We had many talks, and I’ve managed to put up most of the slides into a Google Drive folder, so feel free to access the bucket.

  1. How is the MariaDB sausage made? by Rasmus Johansson covers how MariaDB Server gets made from an engineering standpoint
  2. An inside look at the MariaDB restaurant by Daniel Bartholomew covers how MariaDB Server gets released
  3. ANALYZE for statements by Sergei Petrunia
  4. Data at Rest Encryption in MariaDB 10.1 by Sergei Golubchik (HTML presentation so will prompt a download for you to view in the browser)
  5. Cool MariaDB Plugins by Colin Charles
  6. Passwordless login with unix auth_socket by Otto Kekäläinen
  7. InnoDB in MariaDB 10.1 by Jan Lindström
  8. PCRE Regular Expressions in MariaDB by Alexander Barkov
  9. MaxScale by Massimiliano Pinto
  10. Overview of failover handling in the MariaDB Java Connector by Diego Dupin
  11. Profile Guided Optimization by Axel Schwenke
  12. MariaDB Server in Docker by Kolbe Kegel
  13. Systemd by Dan Black (there were no slides, and the work was being pushed into 10.1, so the crowd got to see svoj come up with his laptop, and Dan made the commit by hitting the Enter key. IIRC, the commit was 20c2ae39db3dd0ec4c337a9b0bd2bf4481b61e49)
  14. Engine Independent Table Statistics including Histograms by Sergei Petrunia

Georg Richter had prepared a presentation but decided not to give it, since we already had quite a lot of talks and discussion throughout the sessions. If you’re interested in MariaDB Connectors, the presentation is worth a read.

Thanks again to Daniël van Eeden and Jean-François Gagné whom really helped get this stuff going.

P/S: for some pictures, I live tweeted them:

Standing room only for @mysql NL meetup group with #mariadb developers talking today @mariadb

— Colin Charles (@bytebot) October 12, 2015

The @MySQL NL meetup with #mariadb folk – yes it's a lot of learning today @mariadb

— Colin Charles (@bytebot) October 12, 2015

When the questions get serious, we see @mariadb @montywi talk too — @mysql NL meetup #mariadb

— Colin Charles (@bytebot) October 12, 2015

Hamish Taylor: Stupidity with passwords

Fri, 2015-10-23 14:28

We all know and understand how important passwords are. We all know that we should be using strong passwords.

What’s a strong password? Something that uses:

  • lower case characters
  • punctuation, such as !@#$%^&*()<>?”:{}+_
  • and should be 8 characters or longer

So, to put it mildly, it really annoys me when I come across services that don’t allow me to use strong passwords. If I possibly could, I’d boycott these services, but sometimes that’s just not possible.

For example, my internet banking is limited to a password of between 6-8 characters. WTF?! This is hardly a secure password policy!

Another financial service I use is limited to 15 characters and doesn’t allow most of the punctuation set. Why? Is it too difficult to extend your database validation rules to cover all of the character set?

Ironically, I didn’t have a problem with Posterous, Facebook or Twitter (and others) in using properly secure passwords. So, these free services give me a decent level of security, but Australian financial services companies can’t. It’s stupidity in the extreme.

Hamish Taylor: Three Ubuntu 11.10 annoyances

Fri, 2015-10-23 14:28

A while back I posted up a few of the issues I was having with Ubuntu 10.04 “Lucid Lynx”.

I’m now using the latest version (for the next few weeks), Ubuntu 11.10 “Oneric Ocelot”. And while it works well on my new laptop, it suffers from three pretty annoying issues.

  1. IPv6 and Network Manager. I am experiencing regular wireless drop outs when I enable IPv6 on my router. When I disable IPv6 on Network Manager it is perfectly stable again.
  2. For most USB keys, write speeds are really slow. And I mean excruciatingly slow. USB HDDs seems to be OK. The issue seems to be in the way that Ubuntu deals with caching.
  3. Sandy Bridge power draining. This is a well known and documented issue, with fixes that have been issued (to be incorporated into the 3.3.x kernel). They are not being integrated into the current version of Ubuntu (which uses the 3.0.x kernel), but are being backported into the next version, 12.04 (which will use the 3.2.x kernel).

These things are quite frustrating, and while I am pretty confident that the power issues will be resolved, I really hope that the other problems are addressed for the next version which is due 26 April 2012. From those bug reports and blog posts, it looks like they will be, which is heartening.

Hamish Taylor: Test

Fri, 2015-10-23 14:28

Test post

Hamish Taylor: A call to “standardised user account requirements” arms

Fri, 2015-10-23 14:28

We need to have a standard for management of user accounts.

Given the number of high profile companies that have been cracked into lately, I have been going through the process of closing accounts for services I no longer use.

Many of these accounts were established when I was more trusting and included real data. However now, unless I am legally required to, I no longer use my real name or real data.

But I have been bitterly disappointed by the inability of some companies to shut down old accounts. For example, one service told me that “At this time, we do not directly delete user accounts…”. I also couldn’t change my username. Another service emailed my credentials in plain text.

To protect the privacy and security of all users, an enforceable standard needs to be established covering management of user accounts. It needs to be applied across the board to all systems connected to the internet. I know how ridiculous this sounds, and that many sites wouldn’t use it, but high profile services should be able to support something like this.

Included in the standard should be:

  • the ability to completely delete accounts (unless there’s some kind of legislative requirement to keep, and then they should only retain the data that is absolutely necessary)
  • the ability to change all details including usernames
  • a requirement to encrypt and salt the password (that covers the credentials in plain text issue noted above)
  • determine the minimum practicable data set that you need to maintain an account and only ask for that. If there’s no need to retain particular account details, don’t collect them. For example, I’ve never been contacted by phone by any of these companies so why was I forced to enter a phone number?

This is a short list from my frustrations today. Please comment to help me flesh this out with other things that should be done on a properly supported user account management system.

And please let me know of your experiences with companies that were unable to properly protect your privacy and security.

Hamish Taylor: Back to WordPress!

Fri, 2015-10-23 14:28

I’ve given up on Blogger and returned to WordPress. I’ll update the look and feel from the defaults and try to update it a bit more often!

Hamish Taylor: My new laptop!

Fri, 2015-10-23 14:28

In May 2010, I posted about what I thought were some pretty underwhelming specifications for laptops.

I have bitten the bullet and upgraded to a laptop with 1366×768 display resolution anyway.

But on a 13.3 inch screen. So it actually works pretty well.

It is a system worth about $2500 that I got for around $700. And no, it didn’t fall off the back of a truck! It fell off the back of the Dell Outlet Store.


  • Dell Latitude E6320
  • Core i5-2520M
  • 4GB RAM (although as an ‘Enterprise’ system, it came with Windows 7 32-bit, so only 3.2GB is visible to Windows. Fixed that by dual-booting Ubuntu 64-bit)
  • 250GB HDD
  • Wi-fi
  • Bluetooth (which I personally think is next to useless)
  • Backlit keyboard (which I think is the BEST thing ever!)
  • 6 cell battery

It’s also mil-spec hardened (or something) which means that it’s almost child-proof!

It does 1080p video and with 4 cores (2 physical and 2 virtual ‘hyper-threading’) video editing works well. Really well.

I want to post up a full review at some stage, but it may not be soon.

Hamish Taylor: In an ideal world … how to change my address

Fri, 2015-10-23 14:28

Recently I moved house.

I  hate moving. Not just for the having to pack everything into boxes at one end then then unpack everything at the destination (which for this move I didn’t have to do!), but mostly because I have to go through the pain that is changing my address.

It turns out that I interact with a lot of organisations, from finance institutions (banks, credit card companies, car insurance, house insurance, health insurance, etc), to official organisations (driver licencing, Medicare, electoral, organ donor register, etc), to community (Red Cross blood donor, 3RRRFM, etc) and mundane organisations (Costco, etc). And that’s just a fraction of them.

I was thinking that, rather than having to fill in what feels like a million forms and waste time that could be spent being a productive public servant or dad for my kid, why isn’t there a central contact details database that I update once? I’m sure that smarter minds than mine have considered this, but I think an opportunity exists for some organisation (government or private) to do this. In the day and age of ‘over-sharing’, are people still averse to putting their address, phone number and email details into a central database? Login security could be addressed using two-factor authentication, such as used by Google Authenticator, or sending a one-time code via SMS or email.

Many services, such as Twitter and Facebook, are set up to authorise other apps to access them. An example of this is when I used my Facebook account to sign up for Freecycle which operates as a Yahoo Group.  I ‘authorised’ Facebook to talk to Yahoo. I’ve also authorised Twicca on my Android smartphone to talk to my Twitter account.

In the same way, in this theoretical single contact details database, I could let the various companies and organisations that I interact with, access my updated contact details. Maybe they could poll this database once a week to look for updated details. I understand they’d have many different backend CRM systems so there may be some manipulation required, but nothing that’s too hard to fix with a bit of scripting.

I could also remove their access when I cease using their services. If I’m not longer banking with Bank A, then I revoke their access so they can’t find out how to contact me.

Does this sound sensible or silly? If sensible why hasn’t Google or someone done this already?