Planet Linux Australia
Partook in Geeks On A Train today, from Auckland down to Wellington, quite happy with how it all went. Just about all of the photos in my LCA2015 album are taken on the train.
It was nice disconnecting from the conference and technology for a bit. And I have a feeling that the trip will be quite an important one to remember in the future.
Filed under: diary
Interactive map for this route.
Tags for this post: blog pictures 20150120-geocaching photo canberra tuggeranong bushwalk geocaching
Related posts: Lunchtime geocaching; Big Monks; Geocaching in the evening, the second; Geocaching in the evening; Point Hut Cross to Pine Island; A walk around Mount Stranger
The morning keynote really did feel like a kick in the guts to all the work that we’ve been doing, and is a horrible tail end to a conference that started with the wonderful community leadership summit. I later quipped that keynotes should be at the end of the day in case the only rational response was drinking.
Fortunately there was a light hearted Paul McKenny talk later in the day that lifted my spirits.
And on an even better note, the main organiser for Geelong is not going to put up with such shit from Linus.
Filed under: diary
As I mentioned on Twitter last week, I’m very happy SUSE was able to support linux.conf.au 2015 with a keynote giveaway on Wednesday morning and sponsorship of the post-conference Beer O’Clock at Catalyst:
— Tim Serong, Esquire (@tserong) January 13, 2015
For those who were in attendance, I thought a little explanation of the keynote gift (a Samsung Galaxy Tab 4 8″) might be in order, especially given the winner came up to me during the post-conference drinks and asked “what’s up with the tablet?”
To put this in perspective, I’m in engineering at SUSE (I’ve spent a lot of time working on high availability, distributed storage and cloud software), and while it’s fair to say I represent the company in some sense simply by existing, I do not (and cannot) actually speak on behalf of my employer. Nevertheless, it fell to me to purchase a gift for us to provide to one lucky delegate sensible enough to arrive on time for Wednesday’s keynote.
I like to think we have a distinct engineering culture at SUSE. In particular, we run a hackweek once or twice a year where everyone has a full week to work on something entirely of their own choosing, provided it’s related to Free and Open Source Software. In that spirit (and given that we don’t make hardware ourselves) I thought it would be nice to be able to donate an Android tablet which the winner would either be able to hack on directly, or would be able to use in the course of hacking something else. So I’m not aware of any particular relationship between my employer and that tablet, but as it says on the back of the hackweek t-shirt I was wearing at the time:
Some things have to be done just because they are possible.
Not because they make sense.
Interactive map for this route.
Tags for this post: blog pictures 20150119-geocaching photo canberra tuggeranong bushwalk geocaching
Related posts: Another lunch time walk; Big Monks; Geocaching in the evening, the second; Geocaching in the evening; Point Hut Cross to Pine Island; A walk around Mount Stranger
When you boot Fedora with a corruption which is not automatically repaired when systemd runs fsck -a then you are asked on the console if to enter single user mode, or if to continue. If you choose to enter single user mode then you'll find that you can't run fsck /dev/md0 as the root filesystem is mounted.
Dracut has a debugging mode with named breakpoints: it will boot up to the break-point, and then dracut will drop the console into a shell.
This is useful for solving a corrupted root filesystem, we can boot up to just before the disk is mounted, breakpoint into the Dracut shell, and then run fsck on the yet-to-be-mounted root filesystem. To do this temporarily add the Dracut breakpoint parameterdracut.break=pre-mount
to the Linux kernel.
In Fedora you do can temporarily modify the Linux kernel parameters by pressing e at the Grub bootloader prompt, arrow-ing down to the "linux" command, adding the parameter to the end of that line, and pressing F10 to run the Grub command list you see on the screen.
Dracut will load the logical volumes, assemble any RAID, and then present a shell on the console. Say fsck /dev/md0 (or whereever /etc/fstab says your / filesytem lives) and then reboot. This is a world easier than booting from a CD or USB and working out which partitions are on what logical volumes, and which logical volumes are in which RAID devices.
Breakpoints are a very fine feature of Dracut and, as this blog posting shows, very useful for solving problems which appear during the early stages of booting the machine.
The data is taken entirely from the linked pdf with one exception: N3664 is a clarification that permits optimization, not a requirement for compliance. Compilers that do not perform this optimization are no less compliant with C++14. I’ve recomputed the percentages for all compiler versions to take this into account.
In addition to the references from the previous post, the approval date of C++14 was taken from http://en.wikipedia.org/wiki/C++14
We were camping underneath a wattle tree, and this wattle tree seemed very popular with the rainbow lorikeets, so we were up at 5:30am, literally with the birds. Zoe was still very excited about camping.
We had a shower first, and then I cooked some bacon and eggs for breakfast before heading down to the beach. Eva and Layla were coming to visit us for the day, so I thought we could just meet them on the beach before coming back to the camp site for lunch.
Zoe had a great time playing around in the waves, and then we did some sand play, making sand castles. I showed her how she could dig down to the sea water in the sand above the wave, and we found heaps of small bivalves. Zoe thought they were pretty cool.
Eva and Layla were running a bit late, so we finished up at the beach and met them at the caravan park, where we cooked some hotdogs for lunch, before having a swim in the pool and then heading back to the beach. It was a really nice afternoon.
After they left, we took it easy for a while before going out for fish and chips for dinner and then calling it a night. Zoe was fast asleep by 7pm, despite the light outside the tent.
- More Surveillance Won’t Protect Free Speech http://t.co/8rgN9R7xh4 11:20:10, 2015-01-14
- Your Computer Understands Your Personality Better Than Your Friends Do http://t.co/9wgnIJgc4s 09:42:06, 2015-01-14
- Internet speeds: Australia ranks 44th, study cites direction of NBN as part of problem http://t.co/SUINPJTKPU #auspol 17:27:15, 2015-01-12
SaltStack currently doesn’t set the hostname correctly on Debian/Ubuntu. For example, this won’t work:system: network.system: - enabled: True - hostname: server1.example.com
Here’s a little shell script I wrote, to get around this problem:% cat set_hostname.sh #!/bin/bash hn=$1 hostname $hn echo $hn > /etc/hostname sed -i "1s/.*/127.0.0.1 localhost $hn/" /etc/hosts
Then apply it using cmd.script, for example:foo-hostname: cmd.script: - source: salt://soe/set_hostname.sh - args: foo.bar.com - unless: grep -q "foo.bar.com" /etc/hosts
Today was the big day. Pack up and drive to Bribie Island for two nights camping. This was the first time I've attempted camping since moving back to Australia (and being on my own). I like camping, and Zoe absolutely loves it, but I've found the idea of trying to do it all on my own a bit daunting, and it's taken me this long to get around to tackling it.
We managed to get the car packed up and be on the road by about 9:30am. Zoe was really helpful, and was able to help carry some of the lighter, less bulky stuff down to the car for me, that that was an unexpected bonus. Everything fit reasonably well with half of the back seats folded down.
We made good time getting there, and arrived at the caravan park by about 10:45am, but check in wasn't until 1:30pm, so we pottered around, and ended up back on the calm side of the island and had some lunch in the park there. After lunch, we caught up with Zoe's Great Aunty Pam for an ice cream before returning to the caravan park to check in.
We'd just started pegging down the tent when it became apparent that the very large caravan that was trying to maneuver into the site next to ours wasn't going to fit, so I offered to do a swap with them before I got any further invested in our site, so we had to pull up the tent and start over on the site next door.
This was the first time I'd put up this tent, so there was all the attendant trial and error of putting it up. I've learned all the lessons now, so I'll be better next time. Zoe was again super helpful, and we were able to put the tent up together successfully. It was ridiculously hot, and very sweaty work.
After we got the tent up, we went for a swim in the pool to cool off, before I started on dinner. The first night's dinner was just some spaghetti bolognaise. I'd pre-made the bolognaise at home and frozen it, so I just had to reheat it on the stove. The caravan park had a pretty decent camp kitchen, but I was trying to see how much I could be self-sufficient, so I did all the cooking on the gas stove I'd brought with me.
I got Zoe to bed a little bit later than normal, but she slept pretty well, despite the heat. It took me hours to get to sleep, despite being exhausted, because I found the tent uncomfortably hot. I was really happy with how the set up day had gone though, and Zoe had an absolute ball. It was totally worth all the effort.
We didn't have a lot planned for the day, and given that we were embarking on a camping trip the following day, I decided to keep it that way. As it was, we still managed to have a pretty busy day.
I started the day off with a run, and managed to do 10 kilometres for the first time in ages. It was a dreadful time, but I was going for progress over perfection.
I didn't realise I had a chiropractic adjustment, so I had to go straight to the chiropractor after my run and sweat all over everything. I felt so bad, but I haven't had an adjustment since before Christmas, so it was great.
Then Sarah dropped Zoe off, and I finally got to have a shower and some breakfast. After that, we headed over to the doctor for the obligatory weekly wart freezing appointment. I think it's been the production it has been because the doctor hasn't been able to give it a really decent hit with the liquid nitrogen, but it's definitely shrinking. She was super brave and even let the doctor give it a bit of a scrape with a scalpel to take off some of the top layers of dead skin. I'm grateful that we have the relationship that we do, because she was a bit scared, but she trusted me anyway, and it all worked out fine.
On the way home, we picked up some mail from the post office. I have to resubmit one unit of my real estate licence course, because I made a mistake, but I passed the other one. I think I'm waiting for one more unit to come back.
After that, we just hung out at home until after lunch, and then went to Woolworths to do some grocery shopping. We ran into Lachlan there, and Zoe and Lachlan had a great time hanging out while we did the grocery shopping.
Instead of getting ready for camping, I decided to have a crack at baking one of the things I want to put in Zoe's school lunchbox, some Hidden Veggie Lunchbox Scrolls. They turned out pretty good, like something you'd get from Baker's Delight. The challenge now is to make space for them in the freezer.
Interactive map for this route.
Tags for this post: blog pictures 20150118-mount_taylor photo canberra tuggeranong bushwalk trig_point
Related posts: Big Monks; A walk around Mount Stranger; Two trigs and a first attempt at finding Westlake; Urambi Trig; Walk up Tuggeranong Hill; A quick walk to Tuggeranong Trig
In computing, a DMZ (demilitarized zone) is a method for separating untrusted traffic from a trusted network. One of the most common implementations of this would be for supporting a publicly accessible server (such as web) on a local internet connection. The server sits in the DMZ and can be accessed from the Internet, but it cannot access the trusted network.
OpenWRT probably needs no introduction, the brilliant open source and community driven Linux based embedded router stack. I run it on my Netgear WNDR3800.
I have an ODRIOD-U3 (little ARM box) running Fedora, which runs a web server. This is what I want to make publicly available in my DMZ.
So, how to create a DMZ in OpenWRT? Some commercial routers have a single button “make a DMZ” and everything is handled behind the scenes for you. Not so with OpenWRT; it’s powerful, transparent, and only does what you tell it to, so we have to create it manually.
My router has a bunch of physical interfaces:
- eth0 (switch)
- eth1 (ethernet)
- wlan0 (wireless card)
- wlan1 (5GHz wireless card)
The eth1 device maps to the physical WAN port on the back of the router. It’s important to note that the physical interfaces may differ from router to router, depending on the chipsets.The Switch
The switch (eth0) includes a number of ports, including the four physical ones on the back of the router, a fifth one that’s not used, as well as one that connects to the CPU.
The switch supports VLANs (virtual LANs), and by default OpenWRT puts all of those ports into VLAN 1. This means that physical connections in those four ports at the back are on the same virtual switch and are able to communicate with each other. You can imagine that if I changed the VLAN of one of those ports to VLAN 10, that the device plugged into that port would no-longer be able to communicate with other devices on the switch. This is the basis for our DMZ.
That VLAN 1 actually creates a new interface on the router:
- eth0.1 (VLAN 1)
The configuration of the switch (including the mapping of ports to VLANs) is available under the switch menu, Network -> Switch.
Note: The port numbers on the switch in OpenWRT do not necessarily map in the right direction to the back of the router. In my case, port 0 on the switch is port 4 on the back of the router.Creating a new VLAN
The first thing we want to do is create VLAN 10 and then assign one of the ports to that VLAN, removing it from VLAN 1.
- Browse to Network -> Switch
- Click Add to make a new VLAN entry
- Set this new entry’s VLAN ID to 10
- In the VLAN 1 row, change Port 0 to off
- In the VLAN 10 row, change Port 0 to untagged
- In the VLAN 10 row, change CPU port to tagged
Setting VLAN to untagged tells the switch to add the appropriate VLAN tag to each ethernet frame as the traffic exits that port. The setting tagged means that the switch should expect that traffic leaving the port has already been tagged, perhaps by the operating system running on the device which is attached to the port.
Port 0 (port 4 on the back of the router) is now in VLAN 10, while the remaining three ports are in VLAN 1 and so it is now isolated from the others. The CPU is also in VLAN 10, else we would not be able to pass any traffic to port 0.
That new VLAN 10 creates a new interface on the router:
- eth0.10 (VLAN 10)
In OpenWRT you create virtual network interfaces which map to physical devices on the router. These are available under the Network -> Interfaces menu.
For example, my router has:
- LAN (for my internal local area network)
- WAN (for the external Internet connection)
One or more physical devices are attached to these zones, for example in my case:
- LAN (bridges VLAN 1 eth0.1, wlan1 and wlan0 together)
- WAN (eth1)
The LAN bridge creates a new interface on the router:
- br-lan (bridged LAN)
Once we have created our new VLAN, we want to create a new a interface for the DMZ. In the same way that the VLAN 1 device, eth0.1, is attached to the LAN interface, we will attach VLAN 10 device, eth0.10, to our new DMZ interface.
- Browse to Network -> Interfaces
- Click Add New Interface to make a new DMZ zone
- Set the name of the new interface to DMZ
- Leave the protocol of the new interface to static
- Ensure bridge over multiple interfaces remains unchecked
- For the interface, select only VLAN Interface: “eth0.10″
- Click Submit
You should be presented with a new configuration screen for this interface.
- Set IPv4 address to something in a new range different to LAN, e.g. if your LAN is 192.168.1.1 then set DMZ to 192.168.0.1
- Leave the rest of the settings blank, you do not need to set routes, or IPv6 if you don’t want to
- Click on the Advanced Settings tab
- Ensure Bring up on boot is ticked
- If you don’t want IPv6, untick Use builtin IPv6-management
- Click on the Physical Settings tab, should already be set to eth0.10
- Click on the Firewall Settings tab
- Under Create / Assign firewall-zone select unspecified -or- create and type dmz
- Click Save and Apply
- If you want to run DHCP on your DMZ, then under DHCP Server click Setup DHCP Server button, leave default settings
We now have a new interface or zone called for the DMZ that’s set to use out DMZ VLAN. It has a new firewall policy assigned to it, dmz, which we now need to configure.Firewall
Now we need to configure the firewall to do a few things:
- Allow the DMZ to talk to the WAN zone, so that devices can access the Internet
- Allow the LAN zone to talk to the DMZ, but not the other way around
- Add some traffic rules opening ports 53 and 67, so that devices from the DMZ can access DNS and DHCP services on the router’s DMZ IP address
- Finally, forward the HTTP port (80) from external internet WAN interface onto a device in the DMZ
Let’s do zone settings first.
- Browse to Network -> Firewall
- Under the Zones section on General Settings page, edit the dmz zone
- Leave the name set to dmz
- Set input to reject, so that we drop all incoming packets by default
- Leave output as accept, although you could set this to reject by default but you’ll require specific outgoing rules as required (like for Yum updates)
- Leave Masquerading and MSS clamping disabled
- Under Covered networks ensure that only dmz is selected
- Under the section Inter-Zone Forwarding, ensure Allow forward to destination zones is set only to WAN
- ensure Allow forward from source zones is set only to LAN
- Click Advanced Settings tab
- If you don’t want IPv6, you can set Restrict to address family to IPv4 only
- Tick Enable logging on this zone, so that we can see what’s happening
Now let’s do port forwards.
- Click on the Port Forwards tab
- Under New port forward section, give a name, such as dmz-http
- Set Protocol to TCP
- Set External zone to WAN
- Set External port to 80
- Set Internal zone to DMZ
- Set Internal IP address to your DMZ server, e.g. 192.168.0.100
- Set Internal port to 80
- Click Add when you’re happy
- Repeat for HTTPS port 443 if you want to run a secure server
Finally, let’s finish with traffic rules.
- Click on the Traffic Rules tab
- Under Open ports on router, set a name like dhcp-dns
- Under Protocol, select UDP
- Under Port, set 53
- Click Add
- Find your new rule in the list and click edit
- Set Destination address to your router’s DMZ IP address
- Repeat for DHCP port 67 UDP if you want to use router’s DHCP server, but don’t set the destination address as DHCP is broadcast
If you want to be able to ping the router from the DMZ clients, do this.
- Set a name like ping-dmz
- Set protocol to Other
- Click Add
- In the new configuration page, set Protocol to ICMP
- Set Match ICMP type to echo reply
- Set Source zone to dmz
- Leave Destination zone to Device (input)
- Set Destination address to your router’s DMZ IP address
- Click Save
Remember we told the router to log the DMZ? Well now we can monitor the firewall rules by browsing to Status -> Kernel Log. Here you should be able to see any rejects that are happening, which is useful to work out why something isn’t happening as you expect on the DMZ.
For example, disable the dmz-ping rule and then try to ping the router from your DMZ server. Refresh the Kernel Log and you should see entries appear.Testing
Plug in a device, see if it gets an IP address. Try to ping 184.108.40.206 (Google DNS server), then try to ping google.com.
Set up a web server on your DMZ box, or use netcat to listen on port 80. Get your external IP address from the router, or Google “my ip”. Now get a friend to browse to your IP and see if you see your web server.
Update: This chart has been updated and I’ve added charts for C++11 Concurrency, C++14, and C++17 here.
One of the things I wanted to see was not just how support had advanced between versions of each compiler, but how compilers had changed relative to one another over time. I extracted the numbers for C++11 from Christophe’s document, found the release dates for each compiler, and created a chart that puts it all together.
It’s interesting to see how far behind Clang starts in comparison to the others, and that it ends up in a close dance with GCC on the way to full C++11 support. It also highlights how disappointing VC++ has been in terms of language feature advancement — particularly when VS2010 was ahead of Clang and ICC for C++11 features.
Creating the chart also served as an opportunity to play around with data visualization using Bokeh. As such, you can click on the chart above and you’ll see a version that you can zoom, pan, and resize (which is only a small part of what Bokeh offers). I intend to write about my experiences with Bokeh at a later date.
Release dates for each compiler were taken from the following pages:
- Visual Studio: http://en.wikipedia.org/wiki/Microsoft_Visual_Studio
- GCC: https://gcc.gnu.org/releases.html
- Clang: http://llvm.org/releases/
- ICC: http://en.wikipedia.org/wiki/Intel_C++_Compiler
The date used to mark the approval of the C++11 standard is taken from http://en.wikipedia.org/wiki/C++11
For the lazy among you the easiest Reaktor Synthesisers that can be built can be made as so. Right click in the workspace, Instrument -> Synthesizers -> Option and then hook up to correct/relevant Voice Combiner. My designs obviously start from scratch though, as I'd like to be able to design some both for educational purposes, for resale, and if that's not possible simply to give away.
You can download my updated experiments from here:https://sites.google.com/site/dtbnguyen/Multiple-Oscillator-Sawtooth-Triangle-Sine-Filter-Interface-Delay-4.enshttps://sites.google.com/site/dtbnguyen/Multiple-Oscillator-Sawtooth-Triangle-Sine-Parabol-Impulse-Pulse-Filter-Interface-Delay-5.enshttps://sites.google.com/site/dtbnguyen/Multiple-Oscillator-Sawtooth-Triangle-Sine-Parabol-Impulse-Pulse-Filter-Interface-Delay-Pan-6.enshttps://sites.google.com/site/dtbnguyen/Multiple-Oscillator-Polyphonic-Selector-Filter-Interface-Delay-Pan-7.ens
I've been looking to build some Android software applications for a while now (curious to know whether this is a viable long term option). It's interesting how many people actually Open Source their software on the various web stores.
I recently wanted to download al the applications/archives from a particular website, http://www.vst4free.com/ so I looked at various website download programs (HTTrack, Teleport Pro, wget, curl, etc...). In spite of the filters/wildcards that were available they were too slow to be realistic.
Use wildcards to exclude or include URLs or links. You can put several scan strings on the same line. Use spaces as separators. Example: +*.zip -www.*.com -www.*.edu/cgi-bin/*.cgi
+*.png +*.gif +*.jpg +*.css +*.js -ad.doubleclick.net/*
+*.zip +*.exe +*.msi +*.tar.gz +*.tar +*.rar
+*.css +*.js -ad.doubleclick.net/*
What did I do? I built something because I noticed patterns in the way files were encoded.
Range for Instrument VSTs
was the same as
which converted to
which could then be parsed for automated download.
Range for Effects VSTs
was the same as
which converted to
which could then be parsed for automated download.
Range for Midi VSTs
You can download my script from here:https://sites.google.com/site/dtbnguyen/download_date_sections.sh.zip
As I've stated previously I've been thinking of re-spinning some versions of Linux for fun and possibly profit. The irony is that it's actually much easier to go down than it is go up. Namely, the smaller distributions such as DamnSmall don't really lend themselves to customisation going up because there are too many dependencies that need to be remedied prior to being able to come up with something workable. This has led me to work on scripts to achieve the exact opposite on smaller (but large such as Knoppix) DVD/CD based live distributions. They work based on class of program based on yum or apt package information. It'll be interesting to see what we can do.
smallest damn thing that you can possibly get?
Several of the ways in which I was thinking about making revenue was:
- distributing/re-sale on chosen media such as USB, CD, DVD, etc... http://damnsmalllinux.org/usb.html
- creating custom versions for who ever wants them. After all, if I'm currently building the code to allow for this why not? (You need to send a portion of payment now and rest on delivery.) Working perferably only on smaller distributions at this point unless the project is really interesting.
- figuring out what the public wants and then attempting to build that for them
- figuring out what the best possible distribution is and attempting to build that for the public
- support via of these distributions
NoOps with Ansible and Puppet – Monty Taylor
- didn’t know it was a contentious term
- “devs can code and let a service deploy, manage and scale their code”
- I want to change the system by landing commits. don’t want to “do ops”
- if I have to use my root access it is a bug
- Cloud Native
- Ephemeral Compute
- Data services
- Design your applications to be resilient via scale out
- Cloud scale out, forget HA for one system, forget long-lived system, shared-nothing for everything. Cloud provides the hard scale-out/HA/9s stuff
- Great for new applications
- OpenStack Infra
- Tooling, automation, and CI for the openstack project
- 2000 devs
- every commit is fully tested.
- each test runs on a single use cloud slave
- 1.7 million test jobs in the last 6 months. 18 TB of log data
- all runs in HP and rackspace public clouds
- Create Servers manually at 1st
- Step 1 – Puppet
- extra hipster because it is in ruby
- If you like ruby it is awesome. If don’t is it less-awesome
- collaboration from non-root users
- code review
- problem that it blows up when you try and install the same thing in two different places
- 3 ways to run. masterless puppet apply. master + puppet agent daemon . master + puppet agent non-daemons
- Secret stuff that you don’t want into you puppet git repo
- Step 2 – Ansible for orchestration
- Control the puppet agent so it runs it nicely and in schedule and on correct hosts first
- Open source system management tool
- Sequence of steps not description of state like puppet
- ad-hoc operation. run random commands
- easy to slowly grow over time till it takes over puppet
- yaml syntax of config files
- Step 3 – Ansible for cloud management
- Ansible config currently mixed in with puppet under – http://git.openstack.org/cgit/openstack-infra/system-config/
- Steve Walsh wins Rusty Wrench award
- Preview of Linux.conf.au 2016 in Geelong
- Much flatter than Auckland
- Deakin University – Waterfront Campus
- Waurn Ponds student accomadation 15 minutes with shuttles
- Feb 8th – 12th 2016
- CFP 1st of June 2015
- Theme “life is better with linux”
- 4 keynotes confirmed or in final stages of discussion, 2 female, 2 male
- NFS keytags
- Announcement for Linux.conf.au 2017 will be in Hobart
- Add more detailed network information to the metadata server: review 85673 (approved).
- Add separated policy rule for each v2.1 api: review 127863 (requested a spec exception).
- Add user limits to the limits API (as well as project limits): review 127094.
- Allow all printable characters in resource names: review 126696 (approved).
- Consolidate all console access APIs into one: review 141065 (approved).
- Expose the lock status of an instance as a queryable item: review 127139 (abandoned); review 85928 (approved).
- Extend api to allow specifying vnic_type: review 138808 (requested a spec exception).
- Implement instance tagging: review 127281 (fast tracked, approved).
- Implement the v2.1 API: review 126452 (fast tracked, approved).
- Improve the return codes for the instance lock APIs: review 135506.
- Microversion support: review 127127 (approved).
- Move policy validation to just the API layer: review 127160 (approved).
- Nova Server Count API Extension: review 134279 (fast tracked).
- Provide a policy statement on the goals of our API policies: review 128560 (abandoned).
- Sorting enhancements: review 131868 (fast tracked, approved, implemented).
- Support JSON-Home for API extension discovery: review 130715 (requested a spec exception).
- Support X509 keypairs: review 105034 (approved).
- Expand support for volume filtering in the EC2 API: review 104450.
- Implement tags for volumes and snapshots with the EC2 API: review 126553 (fast tracked, approved).
- Actively hunt for orphan instances and remove them: review 137996 (abandoned); review 138627.
- Add totalSecurityGroupRulesUsed to the quota limits: review 145689.
- Check that a service isn't running before deleting it: review 131633.
- Enable the nova metadata cache to be a shared resource to improve the hit rate: review 126705 (abandoned).
- Implement a daemon version of rootwrap: review 105404 (requested a spec exception).
- Log request id mappings: review 132819 (fast tracked).
- Monitor the health of hypervisor hosts: review 137768.
- Remove the assumption that there is a single endpoint for services that nova talks to: review 132623.
- Allow direct access to LVM volumes if supported by Cinder: review 127318.
- Cache data from volumes on local disk: review 138292 (abandoned); review 138619.
- Enhance iSCSI volume multipath support: review 134299 (requested a spec exception).
- Failover to alternative iSCSI portals on login failure: review 137468 (requested a spec exception).
- Give additional info in BDM when source type is "blank": review 140133.
- Implement support for a DRBD driver for Cinder block device access: review 134153 (requested a spec exception).
- Poll volume status: review 142828 (abandoned).
- Refactor ISCSIDriver to support other iSCSI transports besides TCP: review 130721 (approved).
- StorPool volume attachment support: review 115716 (approved, requested a spec exception).
- Support Cinder Volume Multi-attach: review 139580 (approved).
- Support iSCSI live migration for different iSCSI target: review 132323 (approved).
- Cells Scheduling: review 141486.
- Create an instance mapping database: review 135644 (approved).
- Flexible cell selection: review 140031.
- Implement instance mapping: review 135424 (approved).
- Populate the instance mapping database: review 136490 (requested a spec exception).
- Initial specification: review 114044 (abandoned).
- Develop and implement a profiler for SQL requests: review 142078 (abandoned).
- Enforce instance uuid uniqueness in the SQL database: review 128097 (fast tracked, approved, implemented).
- Nova db purge utility: review 132656.
- Online schema change options: review 102545 (approved).
- Support DB2 as a SQL database: review 141097 (fast tracked, approved).
- Validate database migrations and model': review 134984 (approved).
- Migrate the Docker Driver into Nova: review 128753.
- Implement support for FreeBSD networking in nova-network: review 127827.
- Allow volumes to be stored on SMB shares instead of just iSCSI: review 102190 (approved, implemented).
- Instance hot resize: review 141219.
- Add config drive support: review 98930 (approved).
- Pass through flavor capabilities to ironic: review 136104 (approved).
- Add ephemeral disk support to the VMware driver: review 126527 (fast tracked, approved).
- Add support for the HTML5 console: review 127283 (requested a spec exception).
- Allow Nova to access a VMWare image store over NFS: review 126866.
- Enable administrators and tenants to take advantage of backend storage policies: review 126547 (fast tracked, approved).
- Enable the mapping of raw cinder devices to instances: review 128697.
- Implement vSAN support: review 128600 (fast tracked, approved).
- Support multiple disks inside a single OVA file: review 128691.
- Support the OVA image format: review 127054 (fast tracked, approved).
- Add Quobyte USP support: review 138372 (abandoned); review 138373 (approved).
- Add VIF_VHOSTUSER vif type: review 138736 (approved).
- Add a Quobyte Volume Driver: review 138375 (abandoned).
- Add finetunable configuration settings for virtio-scsi: review 103797 (abandoned).
- Add large page support: review 129608 (approved).
- Add support for SMBFS as a image storage backend: review 103203 (approved, implemented).
- Allow scheduling of instances such that PCI passthrough devices are co-located on the same NUMA node as other instance resources: review 128344 (fast tracked, approved).
- Allow specification of the device boot order for instances: review 133254.
- Allow the administrator to explicitly set the version of the qemu emulator to use: review 138731 (abandoned).
- Consider PCI offload capabilities when scheduling instances: review 135331.
- Convert to using built in libvirt disk copy mechanisms for cold migrations on non-shared storage: review 126979 (fast tracked).
- Derive hardware policy from libosinfo: review 133945 (approved).
- Implement COW volumes via VMThunder to allow fast boot of large numbers of instances: review 128810 (abandoned); review 128813 (abandoned); review 128830 (abandoned); review 128845 (abandoned); review 129093 (abandoned); review 129108 (abandoned); review 129110 (abandoned); review 129113 (abandoned); review 129116; review 137617.
- Implement configurable policy over where virtual CPUs should be placed on physical CPUs: review 129606 (approved).
- Implement support for Parallels Cloud Server: review 111335 (approved); review 128990 (abandoned).
- Implement support for zkvm as a libvirt hypervisor: review 130447 (approved).
- Improve total network throughput by supporting virtio-net multiqueue: review 128825 (requested a spec exception).
- Improvements to the cinder integration for snapshots: review 134517.
- Quiesce instance disks during snapshot: review 128112; review 131587 (abandoned); review 131597.
- Real time instances: review 139688.
- Stop dm-crypt device when an encrypted instance is suspended or stopped: review 140847 (approved).
- Support SR-IOV interface attach and detach: review 139910 (requested a spec exception).
- Support StorPool as a storage backend: review 137830.
- Support for live block device IO tuning: review 136704.
- Support libvirt storage pools: review 126978 (fast tracked, approved).
- Support live migration with macvtap SR-IOV: review 136077.
- Support quiesce filesystems during snapshot: review 126966 (fast tracked, approved).
- Support using qemu's built in iSCSI initiator: review 133048 (approved).
- Volume driver for Huawei SDSHypervisor: review 130919.
- Allow portions of an instance's uuid to be configurable: review 130451.
- Allow the resize of ephemeral disks during resize: review 145736.
- Attempt to schedule cinder volumes "close" to instances: review 130851; review 131050 (abandoned); review 131051 (abandoned); review 131151 (abandoned).
- Dynamic server groups: review 130005 (abandoned).
- Improve the performance of unshelve for those using shared storage for instance disks: review 135387 (requested a spec exception).
- A lock-free quota implementation: review 135296 (approved).
- Automate the documentation of the virtual machine state transition graph: review 94835.
- Fake Libvirt driver for simulating HW testing: review 139927 (abandoned).
- Flatten Aggregate Metadata in the DB: review 134573 (abandoned).
- Flatten Instance Metadata in the DB: review 134945 (abandoned).
- Implement a new code coverage API extension: review 130855.
- Move flavor data out of the system_metadata table in the SQL database: review 126620 (approved).
- Move to polling for cinder operations: review 135367.
- PCI test cases for third party CI: review 141270.
- Transition Nova to using the Glance v2 API: review 84887 (abandoned).
- Transition to using glanceclient instead of our own home grown wrapper: review 133485 (approved).
- Enable lazy translations of strings: review 126717 (fast tracked, approved).
- Add a new linuxbridge VIF type, macvtap: review 117465 (abandoned).
- Add a plugin mechanism for VIF drivers: review 136827 (abandoned).
- Add support for InfiniBand SR-IOV VIF Driver: review 131729 (requested a spec exception).
- Neutron DNS Using Nova Hostname: review 90150 (abandoned).
- New VIF type to allow routing VM data instead of bridging it: review 130732 (approved, requested a spec exception).
- Nova Plugin for OpenContrail: review 126446 (approved).
- Refactor of the Neutron network adapter to be more maintainable: review 131413.
- Use the Nova hostname in Neutron DNS: review 137669.
- Wrap the Python NeutronClient: review 141108.
- Dynamically alter the interval nova polls components at based on load and expected time for an operation to complete: review 122705.
- A nested quota driver API: review 129420.
- Add a filter to take into account hypervisor type and version when scheduling: review 137714.
- Add an IOPS weigher: review 127123 (approved, implemented); review 132614.
- Add instance count on the hypervisor as a weight: review 127871 (abandoned).
- Add soft affinity support for server group: review 140017 (approved).
- Allow extra spec to match all values in a list by adding the ALL-IN operator: review 138698 (fast tracked, approved).
- Allow limiting the flavors that can be scheduled on certain host aggregates: review 122530 (abandoned).
- Allow the remove of servers from server groups: review 136487.
- Cache aggregate metadata: review 141846.
- Convert get_available_resources to use an object instead of dict: review 133728 (abandoned).
- Convert the resource tracker to objects: review 128964 (fast tracked, approved).
- Create an object model to represent a request to boot an instance: review 127610 (approved).
- Decouple services and compute nodes in the SQL database: review 126895 (approved).
- Distribute PCI Requests Across Multiple Devices: review 142094.
- Enable adding new scheduler hints to already booted instances: review 134746.
- Fix the race conditions when migration with server-group: review 135527 (abandoned).
- Implement resource objects in the resource tracker: review 127609 (approved, requested a spec exception).
- Improve the ComputeCapabilities filter: review 133534 (requested a spec exception).
- Isolate Scheduler DB for Filters: review 138444 (requested a spec exception).
- Isolate the scheduler's use of the Nova SQL database: review 89893 (approved).
- Let schedulers reuse filter and weigher objects: review 134506 (abandoned).
- Move select_destinations() to using a request object: review 127612 (approved).
- Persist scheduler hints: review 88983.
- Refactor allocate_for_instance: review 141129.
- Stop direct lookup for host aggregates in the Nova database: review 132065 (abandoned).
- Stop direct lookup for instance groups in the Nova database: review 131553 (abandoned).
- Support scheduling based on more image properties: review 138937.
- Trusted computing support: review 133106.
- Dynamic Management of Server Groups: review 139272.
- Make key manager interface interoperable with Barbican: review 140144 (fast tracked, approved).
- Provide a reference implementation for console proxies that uses TLS: review 126958 (fast tracked, approved).
- Strongly validate the tenant and user for quota consuming requests with keystone: review 92507 (approved).
- Pacemaker service group driver: review 139991.
- Transition service groups to using the new oslo Tooz library: review 138607.
When Everything Falls Apart: Stories of Version Control System Scaling – Ben Kero
- Sysadmin at Mozilla looking after VCS
- Primarily covering mercurial
- Primarily mercurial
- 3445 repos (1223 unique)
- 32 million commits
- 2TB+ transfer per day
- 1000+ clones per day
- Biggest customer = ourselves
- tested platforms > 12
- Also use git (a lot) and a bit of: subversion, CVS, Bazaar, RCS
- 2 * ssh servers, 10 machines mirror http traffic behind load balancer
- 1st story – know what you are hosting
- Big git repo 1.7G somebody asked to move off github
- Turned out to be mozilla git mirror, so important to move
- plenty of spare resources
- But high load straight away
- turned out to be mercurial->git converter, huge load
- Ran garbage collection – took several hours
- tweaked some other settings
- 2nd story
- 2003 . “Try” CI system
- Simple CI system (before the term existed or they were common)
- flicks off to build server, sends status back to dev
- mercurial had history being immutable up until v2.1 and mozilla was stuck on old version
- ended up with 29,000 brashes in repo
- Around 10,000 heads some operations just start to fail
- Wait times for pushes over 45 minutes. Manual fixes for this
- process was “hg serve” only just freezein gup, not any debug info
- had to attached debugging. trying to update the cache.
- cache got nuked by cached push, long process to rebuild it.
- mercurial bug 4255 in process of being looked at, no fix yet
- The new system
- More web-scalable to replace old the system
- Closer to the pull-request model
- leverage mercurial bundles
- stores bundles in scalable object store
- hopefully minimal retooling from other groups (lots of weird systems supported)
- Planet release engineering @ mozilla
SL[AUO]B: Kernel memory allocator design and philosophy – Christopher Lameter
- NOTE: I don’t do kernel stuff so much of this is over my head.
- Role of the allocator
- page allocator only works in full page size (4k) and is fairly slow
- slab allocator for smaller allocation
- SLAB is one of the “slab allocators”
- kmeme_cache , numa aware, etc
- SLOB: K&R 1991-1999 . compact
- SLAB: Solaris 199-2008 . cache friendly, benchmark friendly
- SLUB: 2008-today , simple and instruction costs count, better debugging, defrag, execution time friendly
- 2013 – work to split out common code for allocators
- manages list of free objects with the space of free objects
- have to traverse list to find object of sufficient size
- rapid fragmentation of memory
- queues per cpu and per node to track cache hotness
- queues for each remote node
- complete data structures
- cold object expiration every 2 seconds on each CPU
- large systems with LOTS of CPUs have huge amount of memory trapped, spending lots of time cleaning cache
- A lot less queuing
- Pages associated with per-cpu. increased locality
- page based policies and interleave
- de-fragmentation on multiple levels
- current default in the kernel
- slabinfo tool for SLUB. tune, modify, query, control objects and settings
- can be asked to go into debug mode even when debugging not enabled with rest of the kernel
- SLUB faster (SLAB good for benchmarks)
- SLOB slow
- SLOB less memory overhead for small/simple systems (only, doesn’t handle lots of reallocations that fragment)
- More common framework
- Various other speedups and features