Planet Linux Australia

Syndicate content
Planet Linux Australia - http://planet.linux.org.au
Updated: 1 hour 22 min ago

Richard Jones: PyCon Australia 2016 Call for Proposals!

Tue, 2016-04-19 10:26

Register and submit a proposal, or read on for more information.

PyCon Australia 2016 is pleased to announce that its Call for Proposals is now open! The conference this year will be held from Friday 12th August to Tuesday 16th August. Miniconfs and special events are held on Friday 12th. The main conference is held on Saturday 13th and Sunday 14th. Monday 15th and Tuesday 16th are developer sprints.

Important Dates

  1. Proposal submission deadline: Sunday, 8th May
  2. Proposal acceptance: Sunday, 29th May

PyCon Australia attracts professional developers from all walks of life, including industry, government, and science, as well as enthusiast and student developers. We’re looking for proposals for presentations and tutorials on any aspect of Python programming, at all skill levels from novice to advanced.

Presentation subjects may range from reports on open source, academic or commercial projects; or even tutorials and case studies. If a presentation is interesting and useful to the Python community, it will be considered for inclusion in the program. We're especially interested in short presentations that will teach conference-goers something new and useful. Can you show attendees how to use a module? Explore a Python language feature? Package an application?

Special Events

Four special events will be held on Friday 12th August, as a prelude to the main conference. These special events are:

If your talk is targeted to a particular field, or requires a lot of prior knowledge, the special events might be a better fit than the main part of the conference.

We are running a combined call for proposals for all the special events and the main conference. In the submission form you can nominate which event(s) your proposal is targeting.

First Time Speakers

We welcome first-time speakers; we are a community conference and we are eager to hear about your experience. If you have friends or colleagues who have something valuable to contribute, twist their arms to tell us about it! Please also forward this Call for Proposals to anyone that you feel may be interested.

Speaker Benefits

Speakers are eligible for discounted conference registration (early bird prices) that will be waived on request. If you, or your organisation, can afford to register at full price, such payments will go directly towards our financial assistance pool to help people who could not otherwise afford to attend the conference.

You will not be automatically registered as an attendee for the conference; you will need to register yourself and failure to do so may result in loss of your talk slot(s).

Financial Assistance

PyCon Australia offers a generous financial assistance programme, so that some attendees and speakers have some -- or in rare cases, all -- of their expenses such as flight, hotel and admission provided to them from the conference budget.

PyCon Australia strongly encourages people to apply for financial assistance -- even if we can’t cover all of your expenses, we will give you free or discounted admission based on need. The application process is simple, and straightforward. It’s also very liberal -- the only caveat is that accepted speakers are given priority so that we don’t lose a good talk because of financial need. We also don’t ban anyone from applying.

Ben Martin: Making PCB with a hobby CNC machine

Mon, 2016-04-18 18:25
One of the main goals I had in mind when getting a CNC "engraving" machine was to make PCB at home. It's sort of full circle to the '70s I guess. Only instead of using nasty chemicals I just have the engraver scratch off an isolation path between traces. Or so the plan goes.





My "hello world" board is the above controller for a 3d printer. This is a follow up to the similar board I made to help use the CNC itself. For a 3d printer I added buttons to set Z=0.1 height and a higher Z height to aid in homing. The breakout headers on the bottom right are for the ESP8266 daughter board. The middle chip is an MCP32017 gpio extender. I've had good experiences using TWI on the ESP8266 and the MCP overcomes the pin limitations quite nicely. It also gives all the buttons a nice central place to go :)



The 3v3 regulator makes the whole show a plug in the AA pack and go type board. The on/off switch is the physical connection to an external battery.



One step closer to the design in the morning, physically create in the afternoon, and use in the evening goal.



OpenSTEM: OpenSTEM 3D Printing and Robotics @ Kilcoy Show

Mon, 2016-04-18 11:31

Last Friday and Saturday we had a great time at the Kilcoy Show (that’s a few hours North of Brisbane), showing visitors 3D printing in action and answering lots of questions on that topic – we actually printed some HO-scale train gear and miniature sheep for our model railway neighbours at the show!

We also let kids have a good play with the Mirobot drawing turtle robots and enjoying our cute robotic caterpillar.

The OpenSTEM booth, next to Andrew Triggs of Mt Kilcoy State School

We saw a fair amount of interest from parents and visiting teachers for our work with schools, from the workshops to our resources and complete classroom programs, of which we had some sample materials that people could browse and ask us about.

On Friday we had the opportunity to meet briefly with Deb Frecklington, QLD state MP for Nanango (Shadow Minister for Agriculture, Fisheries and Forestry), and talked about the meaning and importance of STEM. Naturally STEM is critical for our agriculture as well!

Trevor Wessling (Kilcoy Show), Deb Frecklington (Nanango MP), Arjen (OpenSTEM)

Our other neighbours at the show were Mt Kilcoy State School, where teacher Andrew Triggs showed off QUT LEGO robotics. Kids enjoyed watching the Rubik cube solver – I competed with it a few times, and was done well before it was

Colin Charles: MariaDB Berlin Meetup Notes & Slides

Thu, 2016-04-14 17:25

We had the first MariaDB Berlin Meetup on Tuesday 12.04.2016 at the Wikimedia Berlin offices at 7pm. More or less there were over 54 people that attended the event, a mix of MariaDB Corporation employees and community members. We competed with the entertainment at the AWS Summit Berlin which was apparently about 400m away! Food and drink were enjoyed by all, and most importantly there were many, many lightning talks (minimum 5 minutes, maximum 10 minutes – most were about 6-7 minutes long).

The bonus of all of this? Lots and lots of slides for you to see. Grab them from the Google Drive folder MariaDB Berlin meetup April 2016.

  1. Monty talked about improving the speed of connections to MariaDB Server, some work he’s just pushed fairly recently to the 10.2 tree.
  2. Dipti spoke about MariaDB ColumnStore and it is now clear we’ll see some source/binary drop by the end of May 2016.
  3. Sergei Petrunia and Vicentiu Ciorbaru spoke about the upcoming window functions that MariaDB Server 10.2.0 will see (yes, the alpha should be out real soon now).
  4. Jan spoke about InnoDB in 10.2.
  5. Lixun Peng spoke about a fairly interesting feature, the idea to flashback via mysqlbinlog and how you can have a “Time Machine”. I can’t wait for flashback/time machine to appear in 10.2. The demo for this is extremely good.
  6. Kolbe spoke about data at rest encryption using the MariaDB Amazon AWS KMS plugin.
  7. Sanja and Georg went up together to speak about 10.2 protocol enhancements as well as what you’ll see in Connector/C 3.0.
  8. Wlad gave us a good rundown on authenticating with GSSAPI, something you will notice is also available in MariaDB Server 10.1’s later releases.
  9. Johan Wikman gave us an introduction to MariaDB MaxScale, which started off the talks on MaxScale.
  10. Markus talked about the readwritesplit plugin.
  11. Massimiliano went into the Binlog server.
  12. Martin didn’t use slides but gave us an amazing talk titled “Rival concepts of SQL Proxy”; it was very well given and I’ve encouraged him to write a blog post about it.
  13. Community member Ben Kochie, an SRE at SoundCloud gave us a quick talk on Monitoring MySQL with Prometheus and how much they depend on the PERFORMANCE_SCHEMA.
  14. Diego Dupin spoke a little about the MariaDB Java Connector, and the idea was to do a demo but the projector via HDMI seemed to be a bit wonky (this was also true of using my Mac; the VGA output however worked fine). So it was just a quick talk without any deck.

We ended with a quick Q&A session with Monty dominating it. Lots of interesting questions around why the name Maria, licensing thoughts, ensuring all the software we have are in distributions, etc. Some ended up going for pizza while others ended up in a hotel bar at the Crowne Plaza Potsdamer Platz — and the chatter went on till at least 11pm.

Thanks again to Georg Richter who found us the venue and also did a lot of the legwork with Wikimedia Foundation.

Colin Charles: Major post-GA features in the 5.7 release!

Thu, 2016-04-14 16:25

Interesting developments in the MySQL world – it can now be used as a document store and you can query the database using JavaScript instead of SQL (via the MySQL Shell). There is also a new X Plugin (see: mysql-5.7.12/rapid/) (which now makes use of protocol buffers (see: mysql-5.7.12/extra/protobuf/)). I will agree, this is more than just a maintenance release.

Do get started playing with MySQL Shell. If you’re using the yum repository, remember to ensure you have enabled the mysql-tools-preview in /etc/yum.repos.d/mysql-community.repo. And don’t forget to load the X Plugin in the server! I can’t wait for the rest of the blog posts in the series, and today just took a cursory look at all of this — kudos Team MySQL @ Oracle.

However, I’m concerned that the GA is getting what you would think of as more than just a maintenance release. We saw 5.7.11 get at rest data encryption for InnoDB, and now 5.7.12 getting even more changes. This is going to for example, ship in the next Ubuntu LTS, Xenial Xerus. Today it has 5.7.11, but presumably after release it will be upgrade to 5.7.12. I am not a huge fan of surprises in LTS releases (predictability over 5 years is a nice thing; this probably explains why I still have a 5.0.95 server running), but I guess this small band-aid is what we need to ensure this doesn’t happen going forward?

As for the other question I’ve seen via email from several folk so far: will MariaDB Server support this? I don’t see why not in the future, so why not file a Jira?

OpenSTEM: Launch of OpenSTEM Digital Technologies Program

Thu, 2016-04-14 15:31

As promised, we delivered the OpenSTEM Digital Technologies Program for Primary Schools (F-6) to schools and individual teachers who already signed up: initial units for each year level, resource PDFs and activities, free software, a board game, optional incursions and workshops and other useful resources.

“Our goal is to make sure our students are at the cutting edge of innovation through the development of skills to become the technology architects of the digital age,” Queensland Premier Annastacia Palaszczuk said, “This will include an assessment of coding and computer science, as well as early stage robotics, something I firmly believe should be a part of our education system.”

Advance Queensland’ package announcement (July 2015)

Appreciating the very full schedule that teachers have, we have gone beyond regular integration with the initial materials for Digital Technologies (Australian Curriculum v8.1).  Instead, the base fits directly within existing curricula, particularly Maths and English.  So, doing the basics doesn’t cost any extra time!

That said, we also have some catching up to do. It’s no good tossing older students (or their teachers!) at more complicated problems when they don’t yet have the base level understanding or skills covered in the earlier years.  So we have a catch up plan integral to our initial units.

Today’s students have been immersed in the stream of new technologies since they were born. They have much to learn, but they regard the technology itself as an entirely normal part of life and society.

To be able to guide the students, all educators now also need to go beyond using specific technologies to understanding how things work on a broader scale, and how it all fits together.  So uniquely, the journey is very much a joint one and in some parts the teachers are learning along with (slightly ahead of) the students.

The more I see our teachers and students work with the programs, the more convinced I am that we have a great partnership and are doing the right thing by the kids.

— Cheryl Rowe, Principal

OpenSTEM’s related Robotics Program was recently featured on Channel TEN @ Schools coverage in Brisbane.

With schools already signed up and implementing this program in 2016, you can start any time and in a form that suits you (school wide, or individual teachers or year levels). Contact us for more details, and any questions you might have.

Feel free to ask us for a reference (teacher or principal of a school we’ve worked with).

Glen Turner: There are only two ethernet settings

Thu, 2016-04-14 12:37

I can't beleive I have to write this in 2016, more that twenty years after the bug in the DEC "Tulip" ethernet controller chip which created this mess.

There are only two ethernet speed and autonegotiation settings you should configure on a switch port or host:

1.

Auto negotiation = on

2.

Auto negotiation = off

Speed = 10Mbps

Duplex = half

These are the only two settings which work when the partner interface is set to autonegotiation = on.

If you are considering other settings then buy new hardware. It will work out cheaper.

That is all.

But...

Oh, so you know what you are doing. You know that explicitly setting a speed or duplex implicitly disables autonegotiation and therefore you need to explicitly set the partner interface's speed and duplex as well.

But if you know all that then you also know the world is not a perfect place. Equipment breaks. Operating systems get reinstalled. And you've left a landmine there, waiting for an opportunity...

A goal of modern network and systems administration is to push down the cost of overhead. That means being ruthless with exceptions which store away trouble for the future.

Binh Nguyen: Hybrid Warfare, More PSYOPS, and More

Wed, 2016-04-13 23:09
- the tactics of anti-West groups and states make no sense until you dealve a bit deeper. Basically, they're all saying the same thing. If you don't co-operate with the West you're in trouble but if you do co-operate with the West you still lose because of your loss of autonomy. The reason why South America/Ecuador is supportive of Assange is basically because he's opened up about the fact that

OpenSTEM: OpenSTEM Robotics at Seville Rd on Ten News

Tue, 2016-04-12 11:31

Seville Road State School and OpenSTEM got coverage on Channel TEN News yesterday afternoon with the Robotics Program, in their “TEN at Schools” segment. Good exposure for a great school.

Ten News at Seville Road school library Year 5/6 teacher Trent Perry talking with Josh Students + Mirobots Signing off

BlueHackers: Explainer: what’s the link between insomnia and mental illness?

Tue, 2016-04-12 11:21
The relationship between insomnia and mental illness is bidirectional: about 50% of adults with insomnia have a mental health problem; up to 90% of adults with depression have sleep problems.

Colin Charles: Trip Report: Bulgarian Web Summit

Tue, 2016-04-12 01:25

I have never been to Sofia, Bulgaria till this past February 2016, and boy did I enjoy myself. I visited the Bulgaria Web Summit and spoke there amongst many others. A few notes:

  • Almost 800 people (so more than last year); hence the event was sold out
  • Missed the RocksDB talk due to the massive Q&A session that went on afterwards.
  • Very interesting messaging
LinvoDB
  • LinvoDB (embeddable MongoDB alternative) — LinvoDB / www.strem.io
  • Library written entirely in JavaScript without any dependencies. Converts any KV store to a MongoDB-like API, with Mongoose-like models, and live queries
  • Use case: < 1 million objects (indexes are in memory using a binary search tree; so don’t use it for more). HTML5/Electron/NW.js. Best used with AngularJS/React and maybe Meteor. Can also use NativeScript or React Native. You can use it with node.js but its not recommended for server use cases.
  • Works with SQLite or LevelDB (why not RocksDB yet?). Can also use with IndexedDB/LocalStorage
  • Implemented almost entirely the MongoDB query language. Gives you automatic indexes.
  • FTS in memory (linvodb-fts) – uses trie/metaphone modules for node.js. Can also do p2p replication, persistent indexes, compound indexes
My talk

I enjoyed speaking about MariaDB Server as always, and its clear that many people had a lot of questions about it. Slides. Video. It was tweeted that I had to answer questions for about as long as my talk, afterwards, and it was true :)

I got to meet Robert Nyman at the social event (small world, since he works at the office where Jonas of ex-MySQL fame does). Also met someone very interested in contributing to InfiniDB. It was nice having a beer with my current colleague Salle too. And speaking to the track moderator, Alexander Todorov was also a highlight – since we had many common topics, and he does an amazing amount of work around automation and QA. His blog is worth a read.

Michael Still: Exploring the Jagungal

Mon, 2016-04-11 17:29
Peter Thomas kindly arranged for a variety of ACT Scout leaders to take a tour of the Jagungal portion of Kosciuszko National Park under the guidance of Robert Green. Robert is very experienced with this area, and has recently written a book. Five leaders from the Macarthur Scout Group decided to go along on this tour and take a look at our hiking options in the area.



The first challenge is getting to the area. The campsite we used for the first day is only accessible to four wheel drive vehicles -- the slope down to the camp site from Nimmo Plain is quite rocky and has some loose sections. That said, the Landcruiser I was in had no trouble making the trip, and the group managed to get two car style four wheel drives into the area without problems as well. The route to Nimmo Plain from the south of Canberra is as follows:



Interactive map for this route.



We explored two areas which are both a short drive from Nimmo Plain. We in fact didn't explore anything at Nimmo Plain itself, but as the intermediate point where the road forks it makes sense to show that bit of route first. From Nimmo Plain, it you turn left you end up where we camped for the first day, which is a lovely NWPS camp site with fire pits, a pit toilet, and trout in the river.



The route to that camp site is like this:



Interactive map for this route.



From this campsite we did a 14km loop walk, which took in a series of huts and ruins along relatively flat and easy terrain. There are certainly good walking options here for Scouts, especially those which don't particularly like hills. The route for the first day was like this:



Interactive map for this route.



Its a fantastic area, very scenic without being difficult terrain...



                                           



As you can see from the pictures, life around the camp fire that evening was pretty hard. One note on the weather though -- even at the start of April we're already starting to see very cool overnight weather in this area, with a definite frost on the tents and cars in the morning. I wouldn't want to be hiking in this area much later in the season than this without being prepared for serious cold weather.



   



The next day we drove back to Nimmo Plain and turned right. You then proceed down a dirt road that is marked as private property, but has a public right of way through to the national park. At the border of the park you can leave the car again and go for another walk. The route to this second entrance to the park is like this:



Interactive map for this route.



                     



This drive on the second morning involved a couple of river crossings, with some representative pictures below. Why does the red Landcruiser get to do the crossing three times? Well that's what happens when you forget to shut the gate...



                                                   



Following that we did a short 5km return walk to Cesjack's Hut, which again wasn't scenic at all...



Interactive map for this route.



                                     



I took some pictures on the drive home too of course...



             



Tags for this post: blog pictures 20160409-jagungal photo kosciuszko scouts bushwalk

Related posts: Scout activity: orienteering at Mount Stranger



Comment

OpenSTEM: New shirts for OpenSTEM people

Mon, 2016-04-11 11:32

Horays, our new polo shirts have arrived!  We’re very happy with how they came out with the embroidered owl logo.

We go out & about quite a bit to schools and other events, so it’s useful to be easily recognisable in those environments. The shirts standardise that effort and the colour scheme matches our branding very well.

On the rack there it looks a bit like the outsfit for a sports team

Glen Turner: Embedding files into the executable

Sat, 2016-04-09 14:20

Say you've got a file you want to put into an executable. Some help text, a copyright notice. Putting these into the source code is painful:

static const char *copyright_notice[] = { "This program is free software; you can redistribute it and/or modify", "it under the terms of the GNU General Public License as published by", "the Free Software Foundation; either version 2 of the License, or (at", "your option) any later version.", NULL /* Marks end of text. */ }; #include <stdio.h> const char **line_p; for (line_p = copyright_notice; *line_p != NULL; line_p++) { puts(*line_p); }

If the file is binary, such as an image, then the pain rises exponentially. If you must take this approach then you'll want to know about VIM's xxd hexdump tool:

$ xxd -i copyright.txt > copyright.i

which gives a file which can be included into a C program:

unsigned char copyright_txt[] = { 0x54, 0x68, 0x69, 0x73, 0x20, 0x70, 0x72, 0x6f, 0x67, 0x72, 0x61, 0x6d, 0x20, 0x69, 0x73, 0x20, 0x66, 0x72, 0x65, 0x65, 0x20, 0x73, 0x6f, 0x66, … 0x30, 0x31, 0x2c, 0x20, 0x55, 0x53, 0x41, 0x2e, 0x0a }; unsigned int copyright_txt_len = 681;

That program looks like so:

#include "copyright.i" unsigned char *p; unsigned int len; for (p = copyright_txt, len = 0; len < copyright_txt_len; p++, len++) { putchar(*p); }

If you are going to use this in anger then modify the generated .i file to declare a static const unsigned char …[]. A sed command can do that easily enough; that way the Makefile can re-create the .i file upon any change to the input binary file.

It is much easier to insert a binary file using the linker, and the rest of this blog post explores how that is done. Again the example file will be copyright.txt, but the technique applies to any file, not just text.

Fortunately the GNU linker supports a binary object format, so using the typical linkage tools a binary file can be transformed into an object file simply with:

$ ld --relocatable --format=binary --output=copyright.o copyright.txt $ cc -c helloworld.c $ cc -o helloworld helloworld.o copyright.o

The GNU linker's --relocatable indicates that this object file is to be linked with other object files, and therefore addresses in this object file will need to be relocated at the final linkage.

The final cc in the example doesn't compile anything: it runs ld to link the object files of C programs on this particular architecture and operating system.

The linker defines some symbols in the object file marking the start, end and size of the copied copyright.txt:

$ nm copyright.o 000003bb D _binary_copyright_txt_end 000003bb A _binary_copyright_txt_size 00000000 D _binary_copyright_txt_start

Ignore the address of 00000000, this is relocatable object file and the final linkage will assign a final address and clean up references to it.

A C program can access these symbols with:

extern const unsigned char _binary_copyright_txt_start[]; extern const unsigned char _binary_copyright_txt_end[]; extern const size_t *_binary_copyright_txt_size;

Don't rush ahead and puts() this variable. The copyright.txt file has no final ASCII NUL character which C uses to mark the end of strings. Perhaps use the old-fashioned UNIX write():

#include <stdio.h> #include <unistd.h> fflush(stdout); /* Synchronise C's stdio and UNIX's I/O. */ write(fileno(stdout)), _binary_copyright_txt_start, (size_t)&_binary_copyright_txt_size);

Alternatively, add a final NUL to the copyright.txt file:

$ echo -e -n "\x00" >> copyright.txt

and program:

#include <stdio.h> extern const unsigned char _binary_copyright_txt_start[]; fputs(_binary_copyright_txt_start, stdout);

There's one small wrinkle:

$ objdump -s copyright.o copyright.o: file format elf32-littlearm Contents of section .data: 0000 54686973 2070726f 6772616d 20697320 This program is 0010 66726565 20736f66 74776172 653b2079 free software; y 0020 6f752063 616e2072 65646973 74726962 ou can redistrib 0030 75746520 69742061 6e642f6f 72206d6f ute it and/or mo

The .data section is copied into memory for all running instances of the executable. We really want the contents of the copyright.txt file to be in the .rodata section so that there is only ever one copy in memory no matter how many copies are running.

objcopy could have copied an input ‘binary’ copyright.txt file to a particular section in an output object file, and that particular section could have been .rodata. But objcopy's options require us to state the architecture of the output object file. We really don't want a different command for compiling on x86, AMD64, ARM and so on.

So here's a hack: let ld set the architecture details when it generates its default output and then use objcopy to rename the section from .data to .rodata. Remember that .data contains only the three _binary_… symbols and so they are the only symbols which will move from .data to .rodata:

$ ld --relocatable --format=binary --output=copyright.tmp.o copyright.txt $ objcopy --rename-section .data=.rodata,alloc,load,readonly,data,contents copyright.tmp.o copyright.o $ objdump -s copyright.o copyright.o: file format elf32-littlearm Contents of section .rodata: 0000 54686973 2070726f 6772616d 20697320 This program is 0010 66726565 20736f66 74776172 653b2079 free software; y 0020 6f752063 616e2072 65646973 74726962 ou can redistrib 0030 75746520 69742061 6e642f6f 72206d6f ute it and/or mo

Link this copyright.o with the remainder of the program as before:

$ cc -c helloworld.c $ cc -o helloworld helloworld.o copyright.o

Lev Lafayette: Password Praise in the Future Tense

Fri, 2016-04-08 22:31

Apropos the previous post, I am coming to the conclusion that University's are very strange places when it comes to password policies. Mind you, it shouldn't really come to much of a surprise - the choice of technologies adopted are often so mind-bogglingly strange one is tempted to conclude that the decisions are more political than technical. Of course, that would never happen in the commercial world. All this aside, consider the password policy of a certain Victorian university.

read more

Colin Charles: FOSDEM 2016 notes

Fri, 2016-04-08 20:25

While being on the committee for the FOSDEM MySQL & friends devroom, I didn’t speak at that devroom (instead I spoke at the distributions devroom). But when I had time to pop in, I did take some notes on sessions that were interesting to me, so here are the notes. I really did enjoy Yoshinori Matsunobu’s session (out of the devroom) on RocksDB and MyRocks and I highly recommend you to watch the video as the notes can’t be very complete without the great explanation available in the slide deck. Anyway there are videos from the MySQL and friends devroom.

MySQL & Friends Devroom MySQL Group Replication or how good theory gets into better practice – Tiago Jorge
  • Multi-master update everywhere with built-in automatic distributed recovery, conflict detection and group membership
  • Group replication added 3 PERFORMANCE_SCHEMA tables
  • If a server leaves the group, the others will be automatically informed (either via a crash or if you execute STOP GROUP REPLICATION)
  • Cloud friendly, and it is self-healing. Integrated with server core via a well-defined API. GTIDs, row-based replication, PERFORMANCE_SCHEMA. Works with MySQL Router as well.
  • Multi-master update everywhere. Conflicts will be detected and dealt with, via the first committer wins rule. Any 2 transactions on different servers can write to the same tuple.
  • labs.mysql.com / mysqlhighavailability.com
  • Q: When a node leaves a group, will it still accept writes? A: If you leave voluntarily, it can still accept writes as a regular MySQL server (this needs to be checked)
  • Online DDL is not supported
  • Checkout the video
ANALYZE for statements – Sergei Petrunia
  • a lot like EXPLAIN ANALYZE (in PostgreSQL) or PLAN_STATISTICS (in Oracle)
  • Looks like explain output with execution statistics
  • slides and video
Preparse Query Rewrite Plugins – Sveta Smirnova / Martin Hansson
  • martin.hansson@oracle.com
  • Query rewwriting with a proxy might be too complex, so they thought of doing it inside the server. There is a pre-parse (string-to-string) and a post-parse (parse tree) API. Pre-parse: low overhead, but no structure. Post-parse: retains structure, but requires re-parsing (no destructive editing), need to traverse parse tree and will only work on select statements
  • Query rewrite API builds on top of teh Audit API, and then you’ve got the pre-parse/post-parse APIs on the top that call out to the plugins
  • video
Fedora by the Numbers – Remy DeCausemaker MyRocks: RocksDB Storage Engine for MySQL (LSM Databases at Facebook) – Yoshinori Matsunobu
  • SSD/Flash is getting affordable but MLC Flash is still expensive. HDD has large capacity but limited IOPS (reducing rw IOPS is very important and reducing write is harder). SSD/Flash has great read iops but limited space and write endurance (reducing space here is higher priority)
  • Punch hole compression in 5.7, it is aligned to the sector size of your device. Flash device is basically 4KB. Not 512 bytes. So you’re basically wasting a lot of space and the compression is inefficient
  • LSM tends to have a read penalty compared to B-Tree, like InnoDB. So a good way to reduce the read penalty is to use a Bloom Filter (check key may exist or not without reading data, and skipping read i/o if it definitely does not exist)
  • Another penalty is for delete. It puts them into tombstones. So there is the workaround called SingleDelete.
  • LSMs are ideal for write heavy applications
  • Similar features as InnoDB, transactions: atomicity, MVCC/non-locking consistent read, read committed repeatable read (PostgreSQL-style), Crash safe slave and master. It also has online backup (logical backup by mysqldump and binary backup by myrocks_hotbackup).
  • Much smaller space and write amplification compared to InnoDB
  • Reverse order index (Reverse Column Family). SingleDelete. Prefix bloom filter. Mem-comparable keys when using case sensitive collations. Optimizer statistics for diving into pages.
  • RocksDB is great for scanning forward but ORDER BY DESC queries are slow, hence they use reverse column families to make descending scan a lot faster
  • watch the video

Colin Charles: (tweet) Summary of Percona Live 2015

Fri, 2016-04-08 20:25

The problem with Twitter is that we talk about something and before you know it, people forget. (e.g. does WebScaleSQL have an async client library?) How many blog posts are there about Percona Live Santa Clara 2015? This time (2016), I’m going to endeavour to write more than to just tweet – I want to remember this stuff, and search archives (and also note the changes that happen in this ecosystem). And maybe you do too as well. So look forward to more blogs from Percona Live Data Performance Conference 2016. In the meantime, here’s tweets in chronological order from my Twitter search.

  • crowd filling up the keynote room for #perconalive
  • beginning shortly, we’ll see @peterzaitsev at #perconalive doing his keynote
  • #perconalive has over 1,200 attendees – oracle has 20 folk, with 22 folk from facebook
  • #perconalive is going to be in Amsterdam sept 21-22 2015 (not in London this year). And in 2015, April 18-21 2016!
  • We have @PeterZaitsev on stage now at #perconalive
  • 5 of the 5 top websites are powered by MySQL – an Oracle ad – alexa rankings? http://www.alexa.com/topsites #perconalive
  • now we have Harrison Fisk on ployglot persistence at facebook #perconalive
  • make it work / make it fast / make it efficient – the facebook hacker way #perconalive
  • a lot of FB innovation goes into having large data sizes with short query time response #perconalive
  • “small data” to facebook? 10’s of petabytes with <5ms response times. and yes, this all sits in mysql #perconalive
  • messages eventually lands in hbase for long term storage for disk #perconalive they like it for LSM
  • Harrison introduces @RocksDB to be fast for memory/flash/disk, and its also LSM based. Goto choice for 100’s of services @ FB #perconalive
  • Facebook Newsfeed is pulled from RocksDB. 9 billion QPS at peak! #perconalive
  • Presto works all in memory on a streaming basis, whereas Hive uses map/reduce. Queries are much faster in Presto #perconalive
  • Scuba isn’t opensource – real time analysis tool to debug/understand whats going on @ FB. https://research.facebook.com/publications/456106467831449/scuba-diving-into-data-at-facebook/ … #perconalive
  • InnoDB as a read-optimized store and RocksDB as a write-optimized store — so RocksDB as storage engine for MySQL #perconalive
  • Presto + MySQL shards is something else FB is focused on – in production @ FB #perconalive
  • loving the woz keynote @ #perconalive – wondering if like apple keynotes, we’ll see a “one more thing” after this ;)
  • “i’m only a genius at one thing: that’s making people think i’m a genius” — steve wozniak #perconalive
  • Happiness = Smiles – Frowns (H=S-F) & Happiness = Food, Fun, Friends (H=F³) Woz’s philosophy on being happy + having fun daily #perconalive
  • .@Percona has acquired @Tokutek in a move that provides some consolidation in the MySQL database market and takes..
  • MySQL Percona snaps up Tokutek to move onto MongoDB and NoSQL turf http://zd.net/1ct6PEI by @wolpe
  • One more thing – congrats @percona @peterzaitsev #perconalive Percona has acquired Tokutek with storage engines for MySQL & MongoDB – @PeterZaitsev #perconalive
  • Percona is now a player in the MongoDB space with TokuMX! #perconalive
  • The tokumx mongodb logo is a mongoose… #perconalive Percona will continue to support TokuDB/TokuMX to customers + new investments in it
  • @Percona “the company driving MySQL today” and “the brains behind MySQL”. New marketing angle? http://www.datanami.com/2015/04/14/mysql-leader-percona-takes-aim-at-mongodb/ …
  • We have Steaphan Greene from @facebook talk about @WebScaleSQL at #perconalive
  • what is @webscalesql? its a collaboration between Alibaba, Facebook, Google, LinkedIn, and Twitter to hack on mysql #perconalive
  • close collaboration with @mariadb @mysql @percona teams on @webscalesql. today? upstream 5.6.24 today #perconalive
  • whats new in @WebScaleSQL ? asynchronous mysql client, with support from within HHVM, from FB & LinkedIn #perconalive
  • smaller @webscalesql change (w/big difference) – lower innodb buffer pool memory footprint from FB & Google #perconalive
  • reduce double-write mode while still preserving safety. query throttling, server side statement timeouts, threadpooling #perconalive
  • logical readahead to make full table scans as much as 10x fast. @WebScaleSQL #perconalive
  • whats coming to @WebScaleSQL – online innodb defragmentation, DocStore (JSON style document database using mysql) #perconalive
  • MySQL & RocksDB coming to @WebScaleSQL thanks to facebook & @MariaDB #perconalive
  • So, @webscalesql will skip 5.7 – they will backport interesting features into the 5.6 branch! #perconalive
  • likely what will be next to @webscalesql ? will be mysql-5.8, but can’t push major changes upstream. so might not be an option #perconalive
  • Why only minor changes from @WebScaleSQL to @MySQL upstream? #perconalive
  • Only thing not solved with @webscalesql & upstream @mysql – the Contributor license agreement #perconalive
  • All @WebScaleSQL features under Apache CCLA if oracle can accept it. Same with @MariaDB @percona #perconalive
  • Steaphan Greene says tell Oracle you want @webscalesql features in @mysql. Pressure in public to use the Apache CLA! #perconalive
  • We now have Patrik Sallner CEO from @MariaDB doing the #perconalive keynote ==> 1+1 > 2 (the power of collaboration)
  • “contributors make mariadb” – patrik sallner #perconalive
  • Patrik Sallner tells the story about the CONNECT storage engine and how the retired Olivier Bertrand writes it #perconalive
  • Google contributes table/tablespace encryption to @MariaDB 10.1 #perconalive
  • Patrik talks about the threadpool – how #MariaDB made it, #Percona improved it, and all benefit from opensource development #perconalive
  • and now we have Tomas Ulin from @mysql @oracle for his #perconalive keynote
  • 20 years of MySQL. 10 years of Oracle stewardship of InnoDB. 5 years of Oracle stewardship of @MySQL #perconalive
  • Tomas Ulin on the @mysql 5.7 release candidate. It’s gonna be a great release. Congrats Team #MySQL #perconalive
  • MySQL 5.7 has new optimizer hint frameworks. New cost based optimiser. Generated (virtual) columns. EXPLAIN for running thread #perconalive
  • MySQL 5.7 comes with the query rewrite plugin (pre/post parse). Good for ORMs. “Eliminates many legacy use cases for proxies” #perconalive
  • MySQL 5.7 – native JSON datatypes, built-in JSON functions, JSON comparator, indexing of documents using generated columns #perconalive
  • InnoDB has native full-text search including full CJK support. Does anyone know how FTS compares to MyISAM in speed? #perconalive
  • MySQL 5.7 group replication is unlikely to make it into 5.7 GA. Designed as a plugin #perconalive
  • Robert Hodges believes more enterprises will use MySQL thanks to the encryption features (great news for @mariadb) #perconalive
  • Domas on FB Messenger powered by MySQL. Goals: response time, reliability, and consistency for mobile messaging #perconalive
  • FB Messenger: Iris (in-memory pub-sub service – like a queue with cache semantics). And MySQL as persistence layer #perconalive
  • FB focuses on tiered storage: minutes (in memory), days (flash) and longterm (on disks). #perconalive
  • Gotta keep I/O devices for 4-5 years, so don’t waste endurance capacity of device (so you don’t write as fast as a benchmark) #perconalive
  • Why MySQL+InnoDB? B-Tree: cheap overwrites, I/O has high perf on flash, its also quick and proven @ FB #perconalive
  • What did FB face as issues to address with MySQL? Write throughput. Asynchronous replication. and Failover time. #perconalive
  • HA at Facebook: <30s failover, <1s switchover, > 99.999% query success rate
  • Learning a lot about LSM databases at Facebook from Yoshinori Matsunobu – check out @rocksdb + MyRocks https://github.com/MySQLOnRocksDB/mysql-5.6 …
  • The #mysqlawards 2015 winners #PerconaLive
  • Percona has a Customer Advisory Board now – Rob Young #perconalive
  • craigslist: mysql for active, mongodb for archives. online alter took long. that’s why @mariadb has https://mariadb.com/kb/en/mariadb/progress-reporting/ … #perconalive
  • can’t quite believe @percona is using db-engines rankings in a keynote… le sigh #perconalive
  • “Innovation distinguishes between a leader and a follower” – Steve Jobs #perconalive
  • Percona TokuDB: “only alternative to MySQL + InnoDB” #perconalive
  • “Now that we have the rights to TokuDB, we can add all the cool features ontop of Percona XtraDB Cluster (PXC)” – Rob Young #perconalive
  • New Percona Cloud Tools. Try it out. Helps remote DBA/support too. Wonder what the folk at VividCortex are thinking about now #perconalive
  • So @MariaDB isn’t production ready FOSS? I guess 3/6 top sites on Alexa rank must disagree #perconalive
  • Enjoying Encrypting MySQL data at Google by @jeremycole & Jonas — you can try this in @mariadb 10.1.4 https://mariadb.com/kb/en/mariadb/mariadb-1014-release-notes/ … #perconalive
  • google encryption: mariadb uses the api to have a plugin to store the keys locally; but you really need a key management server #perconalive
  • Google encryption: temporary tables during query execution for the Aria storage engine in #MariaDB #perconalive
  • find out more about google mysql encryption — https://code.google.com/p/google-mysql/ or just use it at 10.1.4! https://downloads.mariadb.org/mariadb/10.1.4/ #perconalive
  • Encrypting MySQL data at Google – Percona Live 2015 #perconalive http://wp.me/p5WPkh-5F
  • The @WebScaleSQL goals are still just to provide access to the code, as opposed to supporting it or making releases #perconalive
  • There is a reason DocStore & Oracle/MySQL JSON 5.7 – they were designed together. But @WebScaleSQL goes forward with DocStore #perconalive
  • So @WebScaleSQL will skip 5.7, and backport things like live resize of the InnoDB buffer pool #perconalive
  • How to view @WebScaleSQL? Default GitHub branch is the active one. Ignore -clean branches, just reference for rebase #perconalive
  • All info you need should be in the commit messages @WebScaleSQL #perconalive
  • Phabricator is what @WebScaleSQL uses as a code review system. All diffs are public, anyone can follow reviews #perconalive
  • automated testing with jenkins/phabricator for @WebScaleSQL – run mtr on ever commit, proposed diffs, & every night #perconalive
  • There is feature documentation, and its a work in progress for @WebScaleSQL. Tells you where its included, etc. #perconalive
  • Checked out the new ANALYZE statement feature in #MariaDB to analyze JOINs? Sergei Petrunia tells all #perconalive https://mariadb.com/kb/en/mariadb/analyze-statement/ …

Rusty Russell: Bitcoin Generic Address Format Proposal

Fri, 2016-04-08 12:29

I’ve been implementing segregated witness support for c-lightning; it’s interesting that there’s no address format for the new form of addresses.  There’s a segregated-witness-inside-p2sh which uses the existing p2sh format, but if you want raw segregated witness (which is simply a “0” followed by a 20-byte or 32-byte hash), the only proposal is BIP142 which has been deferred.

If we’re going to have a new address format, I’d like to make the case for shifting away from bitcoin’s base58 (eg. 1At1BvBMSEYstWetqTFn5Au4m4GFg7xJaNVN2):

  1. base58 is not trivial to parse.  I used the bignum library to do it, though you can open-code it as bitcoin-core does.
  2. base58 addresses are variable-length.  That makes webforms and software mildly harder, but also eliminates a simple sanity check.
  3. base58 addresses are hard to read over the phone.  Greg Maxwell points out that the upper and lower case mix is particularly annoying.
  4. The 4-byte SHA check does not guarantee to catch the most common form of errors; transposed or single incorrect letters, though it’s pretty good (1 in 4 billion chance of random errors passing).
  5. At around 34 letters, it’s fairly compact (36 for the BIP141 P2WPKH).

This is my proposal for a generic replacement (thanks to CodeShark for generalizing my previous proposal) which covers all possible future address types (as well as being usable for current ones):

  1. Prefix for type, followed by colon.  Currently “btc:” or “testnet:“.
  2. The full scriptPubkey using base 32 encoding as per http://philzimmermann.com/docs/human-oriented-base-32-encoding.txt.
  3. At least 30 bits for crc64-ecma, up to a multiple of 5 to reach a letter boundary.  This covers the prefix (as ascii), plus the scriptPubKey.
  4. The final letter is the Damm algorithm check digit of the entire previous string, using this 32-way quasigroup. This protects against single-letter errors as well as single transpositions.

These addresses look like btc:ybndrfg8ejkmcpqxot1uwisza345h769ybndrrfg (41 digits for a P2WPKH) or btc:yybndrfg8ejkmcpqxot1uwisza345h769ybndrfg8ejkmcpqxot1uwisza34 (60 digits for a P2WSH) (note: neither of these has the correct CRC or check letter, I just made them up).  A classic P2PKH would be 45 digits, like btc:ybndrfg8ejkmcpqxot1uwisza345h769wiszybndrrfg, and a P2SH would be 42 digits.

While manually copying addresses is something which should be avoided, it does happen, and the cost of making them robust against common typographic errors is small.  The CRC is a good idea even for machine-based systems: it will let through less than 1 in a billion mistakes.  Distinguishing which blockchain is a nice catchall for mistakes, too.

We can, of course, bikeshed this forever, but I wanted to anchor the discussion with something I consider fairly sane.