Planet Linux Australia

Syndicate content
Planet Linux Australia - http://planet.linux.org.au
Updated: 59 min 23 sec ago

David Rowe: SM1000 Part 3 – Rx Working

Thu, 2014-08-21 13:29

After an hour of messing about it turns out a bad solder joint meant U6 wasn’t connected to the ADC1 pin on the STM32F4 (schematic). This was probably the source of “noise” in some of my earlier unit tests. I found it useful to write a program to connect the ADC1 input to the DAC2 output (loudspeaker) and “listen” to the noise. Software signal tracer. Note to self: I must add that sort of analog loopback as a SM1000 menu option. I “cooked” the bad joint for 10 seconds with the soldering iron and some fresh flux and the rx side burst into life.

Here’s a video walk through of the FreeDV Rx demo:

I am really excited by the “analog” feel to the SM1000. Power up and “off air” speech is coming out of the speaker a few 100ms later! Benefits of no operating system (so no boot delay) and the low latency, fast sync, FreeDV design that veterans like Mel Whitten K0PFX have designed after years of pioneering HF DV.

The SM1000 latency is significantly lower that the PC version of FreeDV. It’s easy to get “hard” real time performance without an operating system, so it’s safe to use nice small audio buffers. Although to be fair optimising latency in x86 FreeDV is not something I have explored to date.

The top level of the receive code is pretty simple:



/* ADC1 is the demod in signal from the radio rx, DAC2 is the SM1000 speaker */

 

nin = freedv_nin(f);  

nout = nin;

f->total_bit_errors = 0;

 

if (adc1_read(&adc16k[FDMDV_OS_TAPS_16K], 2*nin) == 0) {

  GPIOE->ODR = (1 << 3);

  fdmdv_16_to_8_short(adc8k, &adc16k[FDMDV_OS_TAPS_16K], nin);

  nout = freedv_rx(f, &dac8k[FDMDV_OS_TAPS_8K], adc8k);

  //for(i=0; i<FREEDV_NSAMPLES; i++)

  //   dac8k[FDMDV_OS_TAPS_8K+i] = adc8k[i];

  fdmdv_8_to_16_short(dac16k, &dac8k[FDMDV_OS_TAPS_8K], nout);              

  dac2_write(dac16k, 2*nout);

  //led_ptt(0); led_rt(f->fdmdv_stats.sync); led_err(f->total_bit_errors);

  GPIOE->ODR &= ~(1 << 3);

}



We read “nin” modem samples from the ADC, change the same rate from 16 to 8 kHz, then call freedv_rx(). We then re-sample the “nout” output decoded speech samples to 16 kHz, and send them to the DAC, where they are played out of the loudspeaker.

The commented out “for” loop is the analog loopback code I used to “listen” to the ADC1 noise. There is also some commented out code for blinking LEDs (e.g. if we have sync, bit errors) that I haven’t tested yet (indeed the LEDs haven’t been loaded onto the PCB). I like to hit the highest risk tasks on the check list first.

The “GPIOE->ODR” is the GPIO Port E output data register, that’s the code to take the TP8 line high and low for measuring the real time CPU load on the oscilloscope.

Running the ADC and DAC at 16 kHz means I can get away without analog anti-aliasing or reconstruction filters. I figure the SSB radio’s filtering can take care of that.

OK. Time to load up the switches and LEDs and get the SM1000 switching between Tx and Rx via the PTT button.

I used this line to compress the 250MB monster 1080p video from my phone to a 8MB file that was fast to upload on YouTube:



david@bear:~/Desktop$ ffmpeg -i VID_20140821_113318.mp4 -ab 56k -ar 22050 -b 300k -r 15 -s 480x360 VID_20140821_113318.flv

David Rowe: SM1000 Part 2 – Embedded FreeDV Tx Working

Thu, 2014-08-21 08:29

Just now I fired up the full, embedded FreeDV “tx side”. So speech is sampled from the SM1000 microphone, processed by the Codec 2 encoder, then sent to the FDMDV modulator, then out of the DAC as modem tones. It worked, and used only about 25% of the STM32F4 CPU! A laptop running the PC version of FreeDV is the receiver.

Here is the decoded speech from a test “transmission” which to my ear sounds about the same as FreeDV running on a PC. I am relieved that there aren’t too many funny noises apart from artefacts of the Codec itself (which are funny enough).

The scatter plot is really good – better than I expected. Nice tight points and a SNR of 25 dB. This shows that the DAC and line interface hardware is working well:

For the past few weeks I have been gradually building up the software for the SM1000. Codec 2 and the FDMDV modem needed a little tweaking to reduce the memory and CPU load required. It’s really handy that I am the author of both!

The hardware seems to be OK although there is some noise in the analog side (e.g. microphone amplifier, switching power supply) that I am still looking into. Thanks Rick Barnich KA8BMA for an excellent job on the hardware design.

I have also been working on various drivers (ADC, DAC, switches and LEDs), and getting my head around developing on a “bare metal” platform (no operating system). For example if I run out of memory it just hangs, and when I Ctrl-C in gdb the stack is corrupted and it’s in an infinite loop. Anyway, it’s all starting to make sense now, and I’m nearing the finish line.

The STM32F4 is a curious combination of a “router” class CPU that doesn’t have an operating system. By “router” class I mean a CPU found inside a DSL router, like a WRT54G, that runs embedded Linux. The STM32F4 is much faster (168MHz) and more capable than the smaller chips we usually call a “uC” (e.g. a PIC or AVR). Much to my surprise I’m not missing embedded Linux. In some ways an operating system complicates life, for example random context switches, i-cache thrashing, needing lots of RAM and Flash, large and complex build systems and on the hardware side an external address and data bus which means high speed digital signals and PCB area.

I am now working on the Rx side. I need to work out a way to extract demod information so I can determine that the analog line in, ADC, and demod are working correctly. At this stage nothing is coming out of U6, the line interface op-amp schematic here). Oh well, I will take a look at that tomorrow.

Andrew Pollock: [life] Day 203: Kindergarten, a run and cleaning

Wed, 2014-08-20 21:25

I started the day off with a run. It was just a very crappy 5 km run, but it was nice to be feeling well enough to go for a run, and have the weather cooperate as well. I look forward to getting back to 10 km in my near future.

I had my chiropractic adjustment and then got stuck into cleaning the house.

Due to some sort of scheduling SNAFU, I didn't have a massage today. I'm still not quite sure what happened there, but I biked over and everything. The upside was it gave me some more time to clean.

It also worked out well, because I'd booked a doctor's appointment pretty close after my massage, so it was going to be tight to get from one place to the other.

With my rediscovered enthusiasm for exercise, and cooperative weather, I decided to bike to Kindergarten for pick up. Zoe was very excited. I'd also forgotten that Zoe had a swim class this afternoon, so we only had about 30 minutes at home before we had to head out again (again by bike) to go to swim class. I used the time to finish cleaning, and Zoe helped mop her bathroom.

Zoe wanted to hang around and watch Megan do her swim class, so we didn't get away straight away, which made for a slightly late dinner.

Zoe was pretty tired by bath time. Hopefully she'll have a good sleep tonight.

Michael Still: Juno nova mid-cycle meetup summary: nova-network to Neutron migration

Wed, 2014-08-20 14:27
This will be my second last post about the Juno Nova mid-cycle meetup, which covers the state of play for work on the nova-network to Neutron upgrade.



First off, some background information. Neutron (formerly Quantum) was developed over a long period of time to replace nova-network, and added to the OpenStack Folsom release. The development of new features for nova-network was frozen in the Nova code base, so that users would transition to Neutron. Unfortunately the transition period took longer than expected. We ended up having to unfreeze development of nova-network, in order to fix reliability problems that were affecting our CI gating and the reliability of deployments for existing nova-network users. Also, at least two OpenStack companies were carrying significant feature patches for nova-network, which we wanted to merge into the main code base.



You can see the announcement at http://lists.openstack.org/pipermail/openstack-dev/2014-January/025824.html. The main enhancements post-freeze were a conversion to use our new objects infrastructure (and therefore conductor), as well as features that were being developed by Nebula. I can't find any contributions from the other OpenStack company in the code base at this time, so I assume they haven't been proposed.



The nova-network to Neutron migration path has come to the attention of the OpenStack Technical Committee, who have asked for a more formal plan to address Neutron feature gaps and deprecate nova-network. That plan is tracked at https://wiki.openstack.org/wiki/Governance/TechnicalCommittee/Neutron_Gap_Coverage. As you can see, there are still some things to be merged which are targeted for juno-3. At the time of writing this includes grenade testing; Neutron being the devstack default; a replacement for nova-network multi-host; a migration plan; and some documentation. They are all making good progress, but until these action items are completed, Nova can't start the process of deprecating nova-network.



The discussion at the Nova mid-cycle meetup was around the migration planning item in the plan. There is a Nova specification that outlines one possible plan for live upgrading instances (i.e, no instance downtime) at https://review.openstack.org/#/c/101921/, but this will probably now be replaced with a simpler migration path involving cold migrations. This is prompted by not being able to find a user that absolutely has to have live upgrade. There was some confusion, because of a belief that the TC was requiring a live upgrade plan. But as Russell Bryant says in the meetup etherpad:



"Note that the TC has made no such statement on migration expectations other than a migration path must exist, both projects must agree on the plan, and that plan must be submitted to the TC as a part of the project's graduation review (or project gap review in this case). I wouldn't expect the TC to make much of a fuss about the plan if both Nova and Neutron teams are in agreement."



The current plan is to go forward with a cold upgrade path, unless a user comes forward with an absolute hard requirement for a live upgrade, and a plan to fund developers to work on it.



At this point, it looks like we are on track to get all of the functionality we need from Neutron in the Juno release. If that happens, we will start the nova-network deprecation timer in Kilo, with my expectation being that nova-network would be removed in the "M" release. There is also an option to change the default networking implementation to Neutron before the deprecation of nova-network is complete, which will mean that new deployments are defaulting to the long term supported option.



In the next (and probably final) post in this series, I'll talk about the API formerly known as Nova API v3.



Tags for this post: openstack juno nova mid-cycle summary nova-network neutron migration

Related posts: Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Juno nova mid-cycle meetup summary: DB2 support; Juno nova mid-cycle meetup summary: social issues; Juno nova mid-cycle meetup summary: slots



Comment

Tridge on UAVs: First flight of ArduPilot on Linux

Wed, 2014-08-20 14:24

I'm delighted to announce that the effort to port ArduPilot to Linux reached an important milestone yesterday with the first flight of a fixed wing aircraft on a Linux based autopilot running ArduPilot.

As I mentioned in a previous blog post, we've been working on porting ArduPilot to Linux for a while now. There are lots of reasons for wanting to do this port, not least of which is that it is an interesting challenge!

For the test flight yesterday I used a PXF cape which is an add-on to a BeagleBoneBlack board which was designed by Philip Rowse from 3DRobotics. The PXF cape was designed as a development test platform for Linux based autopilots, and includes a rich array of sensors. It has 3 IMUs (a MPU6000, a MPU9250 and a LSM9DSO), plus two barometers (MS5611 on SPI and I2C), 3 I2C connectors for things like magnetometers and airspeed sensors plus a pile of UARTs, analog ports etc.

All of this sits on top of a BeagleBoneBlack, which is a widely used embedded Linux board with 512M ram, a 1GHz ARM CPU and 2 GByte of EMMC for storage. We're running the Debian Linux distribution on the BeagleBoneBlack, with a 3.8.13-PREEMPT kernel. The BBB also has a two nice co-processors called PRUs (programmable realtime units) which are ideal for timing critical tasks. In the flight yesterday we used one PRU for capturing PPM-SUM input from a R/C receiver, and the other PRU for PWM output to the aircrafts servos.

Summer of code project

The effort to port ArduPilot to Linux got a big boost a few months ago when we teamed up with Victor, Sid and Anuj from the BeaglePilot project as part of a Google Summer of Code project. Victor was sponsored by GSoC, while Sid and Anuj were sponsored as summer students in 3DRobotics. Together they have put a huge amount of effort in over the last few months, which culminated in the flight yesterday. The timing was perfect, as yesterday was also the day that student evaluations were due for the GSoc!

PXF on a SkyWalker

For the flight yesterday I used a 3DR SkyWalker, with the BBB+PXF replacing the usual Pixhawk. Because the port of ArduPilot to Linux used the AP_HAL hardware abstraction layer all of the hardware specific code is abstracted below the flight code, which meant I was able to fly the SkyWalker with exactly the same parameters loaded as I have previously used with the Pixhawk on the same plane.

For this flight we didn't use all of the sensors on the PXF however. Some issues with the build of the initial test boards meant that only the MPU9250 was fully functional, but that was quite sufficient. Future revisions of the PXF will fix up the other two IMUs, allowing us to gain the advantages of multiple IMUs (specifically it gains considerable robustness to accelerometer aliasing).

I also had a digital airspeed sensor (on I2C) and an external GPS/Compass combo to give the full set of sensors needed for good fixed wing flight.

Debugging at the field

As with any experimental hardware you have to expect some issues, and the PXF indeed showed up a problem when I arrived at the flying field. At home I don't get GPS lock due to my metal roof so I hadn't done much testing of the GPS and when I was doing pre-flight ground testing yesterday I found that I frequently lost the GPS. With a bit of work using valgrind and gdb I found the bug, and the GPS started to work correctly. It was an interesting bug in the UART layer in AP_HAL_Linux which also may affect the AP_HAL_PX4 code used on a Pixhawk (although with much more subtle effect), so it was an important fix, and really shows the development benefit of testing on multiple platforms.

After that issue was fixed the SkyWalker was launched, and as expected it flew perfectly, exactly as it would fly with any other ArduPilot based autopilot. There was quite a strong wind (about 15 knots, gusting to 20) which was a challenge for such a small foam plane, but it handled it nicely.

Lots more photos of the first flight are available here. Thanks to Darrell Burkey for braving a cold Canberra morning to come out and take some photos!

Next Steps

Now that we have ArduPilot on PXF flying nicely the next step is a test flight with a multi-copter (I'll probably use an Iris). I'm also looking forward to hearing about first flight reports from other groups working on porting ArduPilot to other Linux based boards, such as the NavIO.

This projects follows in the footsteps of quite a few existing autopilots that run on Linux, both open source and proprietary, including such well known projects as Paparrazi, the AR-Drone and many research autopilots at universities around the world. Having the abiity to run ArduPilot on Linux opens up some interesting possibilities for the ArduPilot project, including things like ROS integration, tightly integrated SLAM and lots of computationally intensive vision algorithms. I'm really looking forward to ArduPilot on Linux being widely available for everyone to try.

All of the code needed to fly ArduPilot on Linux is in the standard ArduPilot git repository.

Thanks

Many thanks to 3DRobotics for sponsoring the development of the PXF cape, and to Victor, Sid and Anuj for their efforts over the last few months! Special thanks to Philip Rowse for designing the board, and for putting up with lots of questions as we worked on the port, and to Craig Elder and Jeff Wurzbach for providing engineering support from the 3DR US offices.

Andrew Pollock: [life] Day 202: Kindergarten, a lot of administrative running around, tennis

Wed, 2014-08-20 12:25

Yesterday was a pretty busy day. I hardly stopped, and on top of a poor night's sleep, I was pretty exhausted by the end of the day.

I started the day with a yoga class, because a few extraordinary things had popped up on my schedule, meaning this was the only time I could get to a class this week. It was a beginner's class, but it was nice to have a slower pace for a change, and an excellent way to start the day off.

I drove to Sarah's place to pick up Zoe and take her to Kindergarten, and made a bad choice for the route, and the traffic was particularly bad, and we got to Kindergarten a bit later than normal.

After I dropped Zoe off, I headed straight to the post office to get some passport photos for the application for my certificate of registration. I also noticed that they now had some post office boxes available (I was a bit miffed because I'd been actively discouraged from putting my name down for one earlier in the year because of the purported length of the wait list). I discovered that one does not simply open a PO box in the name of a business, one needs letters of authority and print outs of ABNs and whatnot, so after I got my passport photos and made a few other impulse purchases (USB speakers for $1.99?!) I headed back home to gather the other documentation I needed.

By the time I'd done that and a few other bits and pieces at home, it was time to pop back to get my yoga teacher to certify my photos. Then I headed into the city to lodge the application in person. I should get the piece of paper in 6 weeks or so.

Then I swung past the post office to complete my PO box application (successfully this time) and grab some lunch, and update my mailing address with the bank. By the time I'd done all that, I had enough time to swing past home to grab Zoe's tennis racquet and a snack for her and head to Kindergarten to pick her up.

Today's tennis class went much better. Giving her a snack before the class started was definitely the way to go. She'd also eaten a good lunch, which would have helped. I just need to remember to get her to go to the toilet, then she should be all good for an interruption-free class.

I dropped Zoe directly back to Sarah after tennis class today, and then swung by OfficeWorks to pick up some stationery on the way home.

Arjen Lentz: Two Spaces After a Period: Why You Should Never, Ever Do It | Slate.com

Wed, 2014-08-20 11:25

The cause of the double space may be manual typewriters with their monospace font. But we all use proportional fonts these days.

Related Posts:
  • No related posts

Rusty Russell: POLLOUT doesn’t mean write(2) won’t block: Part II

Wed, 2014-08-20 00:27

My previous discovery that poll() indicating an fd was writable didn’t mean write() wouldn’t block lead to some interesting discussion on Google+.

It became clear that there is much confusion over read and write; eg. Linus thought read() was like write() whereas I thought (prior to my last post) that write() was like read(). Both wrong…

Both Linux and v6 UNIX always returned from read() once data was available (v6 didn’t have sockets, but they had pipes). POSIX even suggests this:

The value returned may be less than nbyte if the number of bytes left in the file is less than nbyte, if the read() request was interrupted by a signal, or if the file is a pipe or FIFO or special file and has fewer than nbyte bytes immediately available for reading.

But write() is different. Presumably so simple UNIX filters didn’t have to check the return and loop (they’d just die with EPIPE anyway), write() tries hard to write all the data before returning. And that leads to a simple rule.  Quoting Linus:

Sure, you can try to play games by knowing socket buffer sizes and look at pending buffers with SIOCOUTQ etc, and say “ok, I can probably do a write of size X without blocking” even on a blocking file descriptor, but it’s hacky, fragile and wrong.

I’m travelling, so I built an Ubuntu-compatible kernel with a printk() into select() and poll() to see who else was making this mistake on my laptop:

cups-browsed: (1262): fd 5 poll() for write without nonblock cups-browsed: (1262): fd 6 poll() for write without nonblock Xorg: (1377): fd 1 select() for write without nonblock Xorg: (1377): fd 3 select() for write without nonblock Xorg: (1377): fd 11 select() for write without nonblock

This first one is actually OK; fd 5 is an eventfd (which should never block). But the rest seem to be sockets, and thus probably bugs.

What’s worse, are the Linux select() man page:

A file descriptor is considered ready if it is possible to perform the corresponding I/O operation (e.g., read(2)) without blocking. ... those in writefds will be watched to see if a write will not block...

And poll():

POLLOUT Writing now will not block.

Man page patches have been submitted…

Andrew McDonnell: Unleashed GovHack – an Adelaide Adventure in Open Data

Tue, 2014-08-19 22:27
Last month I attended Unleashed Govhack, our local contribution to the Australian GovHack hackathon. Unleashed Essentially GovHack is a chance for makers, hackers, designers, artists, and researchers to team up with government ‘data custodians’ and build proof of concept applications (web or mobile), software tools, video productions or presentations (data journalism) in a way that […]

Lev Lafayette: A Source Installation of gzip

Tue, 2014-08-19 21:28

GNU zip is a compression utility free from patented algorithms. Software patents are stupid, and patented compression algorithms are especially stupid.

read more

Linux Users of Victoria (LUV) Announce: LUV Main September 2014 Meeting: AGM + lightning talks

Tue, 2014-08-19 18:28
Start: Sep 2 2014 19:00 End: Sep 2 2014 21:00 Start: Sep 2 2014 19:00 End: Sep 2 2014 21:00 Location: 

The Buzzard Lecture Theatre. Evan Burge Building, Trinity College, Melbourne University Main Campus, Parkville.

Link:  http://luv.asn.au/meetings/map

AGM + lightning talks

Notice of LUV Annual General Meeting, 2nd September 2014, 19:00.

Linux Users of Victoria, Inc., registration number

A0040056C, will be holding its Annual General Meeting at

7pm on Tuesday, 2nd September 2014, in the Buzzard

Lecture Theatre, Trinity College.

The AGM will be held in conjunction with our usual

September Main Meeting. As is customary, after the AGM

business we will have a series of lightning talks by

members on a recent Linux experience or project.

The Buzzard Lecture Theatre, Evan Burge Building, Trinity College Main Campus Parkville Melways Map: 2B C5

Notes: Trinity College's Main Campus is located off Royal Parade. The Evan Burge Building is located near the Tennis Courts. See our Map of Trinity College. Additional maps of Trinity and the surrounding area (including its relation to the city) can be found at http://www.trinity.unimelb.edu.au/about/location/map

Parking can be found along or near Royal Parade, Grattan Street, Swanston Street and College Crescent. Parking within Trinity College is unfortunately only available to staff.

For those coming via Public Transport, the number 19 tram (North Coburg - City) passes by the main entrance of Trinity College (Get off at Morrah St, Stop 12). This tram departs from the Elizabeth Street tram terminus (Flinders Street end) and goes past Melbourne Central Timetables can be found on-line at:

http://www.metlinkmelbourne.com.au/route/view/725

Before and/or after each meeting those who are interested are welcome to join other members for dinner. We are open to suggestions for a good place to eat near our venue. Maria's on Peel Street in North Melbourne is currently the most popular place to eat after meetings.

LUV would like to acknowledge Red Hat for their help in obtaining the Buzzard Lecture Theatre venue and VPAC for hosting, and BENK Open Systems for their financial support of the Beginners Workshops

Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

September 2, 2014 - 19:00

read more

Michael Still: Juno nova mid-cycle meetup summary: slots

Tue, 2014-08-19 18:27
If I had to guess what would be a controversial topic from the mid-cycle meetup, it would have to be this slots proposal. I was actually in a Technical Committee meeting when this proposal was first made, but I'm told there were plenty of people in the room keen to give this idea a try. Since the mid-cycle Joe Gordon has written up a more formal proposal, which can be found at https://review.openstack.org/#/c/112733.



If you look at the last few Nova releases, core reviewers have been drowning under code reviews, so we need to control the review workload. What is currently happening is that everyone throws up their thing into Gerrit, and then each core tries to identify the important things and review them. There is a list of prioritized blueprints in Launchpad, but it is not used much as a way of determining what to review. The result of this is that there are hundreds of reviews outstanding for Nova (500 when I wrote this post). Many of these will get a review, but it is hard for authors to get two cores to pay attention to a review long enough for it to be approved and merged.



If we could rate limit the number of proposed reviews in Gerrit, then cores would be able to focus their attention on the smaller number of outstanding reviews, and land more code. Because each review would merge faster, we believe this rate limiting would help us land more code rather than less, as our workload would be better managed. You could argue that this will mean we just say 'no' more often, but that's not the intent, it's more about bringing focus to what we're reviewing, so that we can get patches through the process completely. There's nothing more frustrating to a code author than having one +2 on their code and then hitting some merge freeze deadline.



The proposal is therefore to designate a number of blueprints that can be under review at any one time. The initial proposal was for ten, and the term 'slot' was coined to describe the available review capacity. If your blueprint was not allocated a slot, then it would either not be proposed in Gerrit yet, or if it was it would have a procedural -2 on it (much like code reviews associated with unapproved specifications do now).



The number of slots is arbitrary at this point. Ten is our best guess of how much we can dilute core's focus without losing efficiency. We would tweak the number as we gained experience if we went ahead with this proposal. Remember, too, that a slot isn't always a single code review. If the VMWare refactor was in a slot for example, we might find that there were also ten code reviews associated with that single slot.



How do you determine what occupies a review slot? The proposal is to groom the list of approved specifications more carefully. We would collaboratively produce a ranked list of blueprints in the order of their importance to Nova and OpenStack overall. As slots become available, the next highest ranked blueprint with code ready for review would be moved into one of the review slots. A blueprint would be considered 'ready for review' once the specification is merged, and the code is complete and ready for intensive code review.



What happens if code is in a slot and something goes wrong? Imagine if a proposer goes on vacation and stops responding to review comments. If that happened we would bump the code out of the slot, but would put it back on the backlog in the location dictated by its priority. In other words there is no penalty for being bumped, you just need to wait for a slot to reappear when you're available again.



We also talked about whether we were requiring specifications for changes which are too simple. If something is relatively uncontroversial and simple (a better tag for internationalization for example), but not a bug, it falls through the cracks of our process at the moment and ends up needing to have a specification written. There was talk of finding another way to track this work. I'm not sure I agree with this part, because a trivial specification is a relatively cheap thing to do. However, it's something I'm happy to talk about.



We also know that Nova needs to spend more time paying down its accrued technical debt, which you can see in the huge amount of bugs we have outstanding at the moment. There is no shortage of people willing to write code for Nova, but there is a shortage of people fixing bugs and working on strategic things instead of new features. If we could reserve slots for technical debt, then it would help us to get people to work on those aspects, because they wouldn't spend time on a less interesting problem and then discover they can't even get their code reviewed. We even talked about having an alternating focus for Nova releases; we could have a release focused on paying down technical debt and stability, and then the next release focused on new features. The Linux kernel does something quite similar to this and it seems to work well for them.



Using slots would allow us to land more valuable code faster. Of course, it also means that some patches will get dropped on the floor, but if the system is working properly, those features will be ones that aren't important to OpenStack. Considering that right now we're not landing many features at all, this would be an improvement.



This proposal is obviously complicated, and everyone will have an opinion. We haven't really thought through all the mechanics fully, yet, and it's certainly not a done deal at this point. The ranking process seems to be the most contentious point. We could encourage the community to help us rank things by priority, but it's not clear how that process would work. Regardless, I feel like we need to be more systematic about what code we're trying to land. It's embarrassing how little has landed in Juno for Nova, and we need to be working on that. I would like to continue discussing this as a community to make sure that we end up with something that works well and that everyone is happy with.



This series is nearly done, but in the next post I'll cover the current status of the nova-network to neutron upgrade path.



Tags for this post: openstack juno nova mid-cycle summary review slots blueprint priority project management

Related posts: Juno nova mid-cycle meetup summary: social issues; Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Juno nova mid-cycle meetup summary: DB2 support



Comment

Andrew Pollock: [life] Day 201: Kindergarten, some startup stuff, car wash and a trip to the vet ophthalmologist

Mon, 2014-08-18 22:25

Zoe woke up at some point in the night and ended up in bed with me. I don't even remember when it was.

We got going reasonably quickly this morning, and Zoe wanted porridge for breakfast, so I made a batch of that in the Thermomix.

She was a little bit clingy at Kindergarten for drop off. She didn't feel like practising writing her name in her sign-in book. Fortunately Miss Sarah was back from having done her prac elsewhere, and Zoe had been talking about missing her on the way to Kindergarten, so that was a good distraction, and she was happy to hang out with her while I left.

I used the morning to do some knot practice for the rock climbing course I've signed up for next month. It was actually really satisfying doing some knots that previously I'd found to be mysterious.

I had a lunch meeting over at the Royal Brisbane Women's Hospital to bounce a startup idea off a couple of people, so I headed over there for a very useful lunch discussion and then briefly stopped off at home before picking up Zoe from Kindergarten.

Zoe had just woken up from a nap before I arrived, and was a bit out of sorts. She perked up a bit when we went to the car wash and had a babyccino while the car got cleaned. Sarah was available early, so I dropped Zoe around to her straight after that.

I'd booked Smudge in for a consult with a vet ophthalmologist to get her eyes looked at, so I got back home again, and crated her and drove to Underwood to see the vet. He said that she had the most impressive case of eyelid agenesis he'd ever seen. She also had persistent pupillary membranes in each eye. He said that the eyelid agenesis is a pretty common birth defect in what he called "dumpster cats", which for all we know, is exactly what Smudge (or more importantly, her mother) were. He also said that other eye defects, like the membranes she had, were common in cases where there was eyelid agenesis.

The surgical fix was going to come in at something in the order of $2,000 an eye, be a pretty long surgery, and involve some crazy transplanting of lip tissue. Cost aside, it didn't sound like a lot of fun for Smudge, and given her age and that she's been surviving the way she is, I can't see myself wanting to spend the money to put her through it. The significantly cheaper option is to just religiously use some lubricating eye gel.

After that, I got home with enough time to eat some dinner and then head out to crash my Thermomix Group Leader's monthly team meeting to see what it was like. I got a good vibe from that, so I'm happy to continue with my consultant application.

Craige McWhirter: Introduction to Managing OpenStack Via the CLI

Mon, 2014-08-18 14:28
Assumptions: Introduction:

There's great deal of ugliness in OpenStack but what I enjoy the most is the relative elegance of driving an OpenStack deployment from the comfort of my own workstation.

Once you've configured your workstation as an OpenStack Management Client, these are some of the commands you can run from your workstation against an OpenStack deployment.

There are client commands for each of the projects, which makes it rather simple to relate the commands you want to run against the service you need to work with. ie:

$ PROJECT --version

$ cinder --version 1.0.8 $ glance --version 0.12.0 $ heat --version 0.2.9 $ keystone --version 0.9.0 $ neutron --version 2.3.5 $ nova --version 2.17.0 Getting by With a Little Help From Your Friends

The first slice of CLI joy when using these OpenStack clients is the CLI help that is available for each of the clients. When each client is called with --help, a comprehensive list of options and sub commands is dumped to STDOUT, which is useful but but not expected.

The question you usually find yourself asking is, "How do I use those sub commands?", which answered by utilising the following syntax:

$ PROJECT help subcommand

This will dump all the arguments for the specified subcommand to STDOUT. I've used the below example for it's brevity:

$ keystone help user-create usage: keystone user-create --name <user-name> [--tenant <tenant>] [--pass [<pass>]] [--email <email>] [--enabled <true|false>] Create new user Arguments: --name <user-name> New user name (must be unique). --tenant <tenant>, --tenant-id <tenant> New user default tenant. --pass [<pass>] New user password; required for some auth backends. --email <email> New user email address. --enabled <true|false> Initial user enabled status. Default is true. Getting Behind the Wheel

Before you can use these commands, you will need to set some appropriate environment variables. When you completed configuring your workstation as an OpenStack Management Client, you will have completed short file that set the username, passowrd, tenant name and authentication URL for you OpenStack clients. Now is the time to source that file:

$ source <username-tenant>.sh

I have one of these for each OpenStack deployment, user account and tenant that I wish to work with. Sourcing the relevent one before I commence a body of work.

Turning the Keystone

Keystone provides the authentication service, assuming you have appropriate privileges, you will need to:

Create a Tenant (referred to as a Project via the Web UI).

$ keystone tenant-create --name DemoTenant --description "Don't forget to \ delete this tenant" +-------------+------------------------------------+ | Property | Value | +-------------+------------------------------------+ | description | Don't forget to delete this tenant | | enabled | True | | id | painguPhahchoh2oh7Oeth2jeh4ahMie | | name | DemoTenant | +-------------+------------------------------------+ $ keystone tenant-list +----------------------------------+----------------+---------+ | id | name | enabled | +----------------------------------+----------------+---------+ | painguPhahchoh2oh7Oeth2jeh4ahMie | DemoTenant | True | +----------------------------------+----------------+---------+

Create / add a user to that tenant.

$ keystone user-create --name DemoUser --tenant DemoTenant --pass \ Tahh9teih3To --email demo.tenant@example.tld +----------+----------------------------------+ | Property | Value | +----------+----------------------------------+ | email | demo.tenant@example.tld | | enabled | True | | id | ji5wuVaTouD0ohshoChoohien3Thaibu | | name | DemoUser | | tenantId | painguPhahchoh2oh7Oeth2jeh4ahMie | | username | DemoUser | +----------+----------------------------------+ $ keystone user-role-list --user DemoUser --tenant DemoTenant +----------------------------------+----------+----------------------------------+----------------------------------+ | id | name | user_id | tenant_id | +----------------------------------+----------+----------------------------------+----------------------------------+ | eiChu2Lochui7aiHu5OF2leiPhai6nai | _member_ | ji5wuVaTouD0ohshoChoohien3Thaibu | painguPhahchoh2oh7Oeth2jeh4ahMie | +----------------------------------+----------+----------------------------------+----------------------------------+

Provide that user with an appropriate role (defaults to member)

$ keystone user-role-add --user DemoUser --role admin --tenant DemoTenant $ keystone user-role-list --user DemoUser --tenant DemoTenant +----------------------------------+----------+----------------------------------+----------------------------------+ | id | name | user_id | tenant_id | +----------------------------------+----------+----------------------------------+----------------------------------+ | eiChu2Lochui7aiHu5OF2leiPhai6nai | _member_ | ji5wuVaTouD0ohshoChoohien3Thaibu | painguPhahchoh2oh7Oeth2jeh4ahMie | | ieDieph0iteidahjuxaifi6BaeTh2Joh | admin | ji5wuVaTouD0ohshoChoohien3Thaibu | painguPhahchoh2oh7Oeth2jeh4ahMie | +----------------------------------+----------+----------------------------------+----------------------------------+ Taking a Glance at Images

Glance provides the service for discovering, registering and retrieving virtual machine images. It is via Glance that you will be uploading VM images to OpenStack. Here's how you can upload a pre-existing image to Glance:

Note: If your back end is Ceph then the images must be in RAW format.

$ glance image-create --name DemoImage --file /tmp/debian-7-amd64-vm.qcow2 \ --progress --disk-format qcow2 --container-format bare \ --checksum 05a0b9904ba491346a39e18789414724 [=============================>] 100% +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | checksum | 05a0b9904ba491346a39e18789414724 | | container_format | bare | | created_at | 2014-08-13T06:00:01 | | deleted | False | | deleted_at | None | | disk_format | qcow2 | | id | mie1iegauchaeGohghayooghie3Zaichd1e5 | | is_public | False | | min_disk | 0 | | min_ram | 0 | | name | DemoImage | | owner | gei6chiC3hei8oochoquieDai9voo0ve | | protected | False | | size | 2040856576 | | status | active | | updated_at | 2014-08-13T06:00:28 | | virtual_size | None | +------------------+--------------------------------------+ $ glance image-list +--------------------------------------+---------------------------+-------------+------------------+------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+---------------------------+-------------+------------------+------------+--------+ | mie1iegauchaeGohghayooghie3Zaichd1e5 | DemoImage | qcow2 | bare | 2040856576 | active | +--------------------------------------+---------------------------+-------------+------------------+------------+--------+ Starting Something New With Nova

Build yourself an environment file with the new credentials:

export OS_USERNAME=DemoTenant export OS_PASSWORD=Tahh9teih3To export OS_TENANT_NAME=DemoTenant export OS_AUTH_URL=http://horizon.my.domain.tld:35357/v2.0

Then source it.

By now you just want a VM, so lets knock one up for the user and tenant you just created:

$ nova boot --flavor m1.small --image DemoImage DemoVM +--------------------------------------+--------------------------------------------------+ | Property | Value | +--------------------------------------+--------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | instance-0000009f | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | W3kd5WzYD2tE | | config_drive | | | created | 2014-08-13T06:41:19Z | | flavor | m1.small (2) | | hostId | | | id | 248a247d-83ff-4a52-b9b4-4b3961050e94 | | image | DemoImage (d51001c2-bfe3-4e8a-86d8-e2e35898c0f3) | | key_name | - | | metadata | {} | | name | DemoVM | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | BUILD | | tenant_id | c6c88f8dbff34b60b4c8e7fad1bda869 | | updated | 2014-08-13T06:41:19Z | | user_id | cb040e80138c4374b46f4d31da38be68 | +--------------------------------------+--------------------------------------------------+

Now you can use nova show DemoVM to provide you with the IP address of the host or acces the console via the Horizon dashboard.

Michael Still: Juno nova mid-cycle meetup summary: scheduler

Mon, 2014-08-18 13:27
This post is in a series covering the discussions at the Juno Nova mid-cycle meetup. This post will cover the current state of play of our scheduler refactoring efforts. The scheduler refactor has been running for a fair while now, dating back to at least the Hong Kong summit (so about 1.5 release cycles ago).



The original intent of the scheduler sub-team's effort was to pull the scheduling code out of Nova so that it could be rapidly iterated on its own, with the eventual goal being to support a single scheduler across the various OpenStack services. For example, the scheduler that makes placement decisions about your instances could also be making decisions about the placement of your storage resources and could therefore ensure that they are co-located as much as possible.



During this process we realized that a big bang replacement is actually much harder than we thought, and the plan has morphed into being a multi-phase effort. The first step is to make the interface for the scheduler more clearly defined inside the Nova code base. For example, in previous releases, it was the scheduler that launched instances: the API would ask the scheduler to find available hypervisor nodes, and then the scheduler would instruct those nodes to boot the instances. We need to refactor this so that the scheduler picks a set of nodes, but then the API is the one which actually does the instance launch. That way, when the scheduler does move out it's not trusted to perform actions that change hypervisor state, and the Nova code does that for it. This refactoring work is under way, along with work to isolate the SQL database accesses inside the scheduler.



I would like to set expectations that this work is what will land in Juno. It has little visible impact for users, but positions us to better solve these problems in Kilo.



We discussed the need to ensure that any new scheduler is at least as fast and accurate as the current one. Jay Pipes has volunteered to work with the scheduler sub-team to build a testing framework to validate this work. Jay also has some concerns about the resource tracker work that is being done at the moment that he is going to discuss with the scheduler sub-team. Since the mid-cycle meetup there has been a thread on the openstack-dev mailing list about similar resource tracker concerns (here), which might be of interest to people interested in scheduler work.



We also need to test our assumption at some point that other OpenStack services such as Neutron and Cinder would be even willing to share a scheduler service if a central one was implemented. We believe that Neutron is interested, but we shouldn't be surprising our fellow OpenStack projects by just appearing with a complete solution. There is a plan to propose a cross-project session at the Paris summit to cover this work.



In the next post in this series we'll discuss possibly the most controversial part of the mid-cycle meetup. The proposal for "slots" for landing blueprints during Kilo.



Tags for this post: openstack juno nova mid-cycle summary scheduler

Related posts: Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Juno nova mid-cycle meetup summary: DB2 support; Juno nova mid-cycle meetup summary: social issues; Juno nova mid-cycle meetup summary: slots



Comment

Michael Still: Juno nova mid-cycle meetup summary: bug management

Mon, 2014-08-18 13:27
Welcome to the next exciting installment of the Nova Juno mid-cycle meetup summary. In the previous chapter, our hero battled a partially complete cells implementation, by using his +2 smile of good intentions. In this next exciting chapter, watch him battle our seemingly never ending pile of bugs! Sorry, now that I'm on to my sixth post in this series I feel like it's time to get more adventurous in the introductions.



For at least the last cycle, and probably longer, Nova has been struggling with the number of bugs filed in Launchpad. I don't think the problem is that Nova has terrible code, it is instead that we have a lot of users filing bugs, and the team working on triaging and closing bugs is small. The complexity of the deployment options with Nova make this problem worse, and that complexity increases as we allow new drivers for things like different storage engines to land in the code base.



The increasing number of permutations possible with Nova configurations is a problem for our CI systems as well, as we don't cover all of these options and this sometimes leads us to discover that they don't work as expected in the field. CI is a tangent from the main intent of this post though, so I will reserve further discussion of our CI system until a later post.



Tracy Jones and Joe Gordon have been doing good work in this cycle trying to get a grip on the state of the bugs filed against Nova. For example, a very large number of bugs (hundreds) were for problems we'd fixed, but where the bug bot had failed to close the bug when the fix merged. Many other bugs were waiting for feedback from users, but had been waiting for longer than six months. In both those cases the response was to close the bug, with the understanding that the user can always reopen it if they come back to talk to us again. Doing "quick hit" things like this has reduced our open bug count to about one thousand bugs. You can see a dashboard that Tracy has produced that shows the state of our bugs at http://54.201.139.117/nova-bugs.html. I believe that Joe has been moving towards moving this onto OpenStack hosted infrastructure, but this hasn't happened yet.



At the mid-cycle meetup, the goal of the conversation was to try and find other ways to get our bug queue further under control. Some of the suggestions were largely mechanical, like tightening up our definitions of the confirmed (we agree this is a bug) and triaged (and we know how to fix it) bug states. Others were things like auto-abandoning bugs which are marked incomplete for more than 60 days without a reply from the person who filed the bug, or unassigning bugs when the review that proposed a fix is abandoned in Gerrit.



Unfortunately, we have more ideas for how to automate dealing with bugs than we have people writing automation. If there's someone out there who wants to have a big impact on Nova, but isn't sure where to get started, helping us out with this automation would be a super helpful way to get started. Let Tracy or I know if you're interested.



We also talked about having more targeted bug days. This was prompted by our last bug day being largely unsuccessful. Instead we're proposing that the next bug day have a really well defined theme, such as moving things from the "undecided" to the "confirmed" state, or similar. I believe the current plan is to run a bug day like this after J-3 when we're winding down from feature development and starting to focus on stabilization.



Finally, I would encourage people fixing bugs in Nova to do a quick search for duplicate bugs when they are closing a bug. I wouldn't be at all surprised to discover that there are many bugs where you can close duplicates at the same time with minimal effort.



In the next post I'll cover our discussions of the state of the current scheduler work in Nova.



Tags for this post: openstack juno nova mi-cycle summary bugs

Related posts: Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Michael's surprisingly unreliable predictions for the Havana Nova release; Juno nova mid-cycle meetup summary: DB2 support



Comment

Andrew Pollock: [tech] Solar follow up

Mon, 2014-08-18 11:25

Now that I've had my solar generation system for a little while, I thought I'd write a follow up post on how it's all going.

Energex came out a week ago last Saturday and swapped my electricity meter over for a new digital one that measures grid consumption and excess energy exported. Prior to that point, it was quite fun to watch the old analog meter going backwards. I took a few readings after the system was installed, through to when the analog meter was disconnected, and the meter had a value 26 kWh lower than when I started.

I've really liked how the excess energy generated during the day has effectively masked any relatively small overnight power consumption.

Now that I have the new digital meter things are less exciting. It has a meter measuring how much power I'm buying from the grid, and how much excess power I'm exporting back to the grid. So far, I've bought 32 kWh and exported 53 kWh excess energy. Ideally I want to minimise the excess because what I get paid for it is about a third of what I have to pay to buy it from the grid. The trick is to try and shift around my consumption as much as possible to the daylight hours so that I'm using it rather than exporting it.

On a good day, it seems I'm generating about 10 kWh of energy.

I'm still impatiently waiting for PowerOne to release their WiFi data logger card. Then I'm hoping I can set up something automated to submit my daily production to PVOutput for added geekery.

Sridhar Dhanapalan: Twitter posts: 2014-08-11 to 2014-08-17

Mon, 2014-08-18 09:27

Andrew Pollock: [life] Day 198: Dentist, play date and some Science Friday

Sun, 2014-08-17 21:25

First up on Friday morning was Zoe's dentist appointment. When Sarah dropped her off, we jumped in the car straight away and headed out. It's a long way to go for a 10 minute appointment, but it's worth it for the "Yay! I can't wait!" reaction I got when I told her she was going a few days prior.

Having a positive view of dental care is something that's very important to me, as teeth are too permanent to muck around with. The dentist was very happy with her teeth, and sealed her back molars. Apparently it's all the rage now, as these are the ones that hang around until she's 12.

Despite this being her third appointment with this dentist, Zoe was feeling a bit shy this time, so she spent the whole time reclining on me in the chair. She otherwise handled the appointment like a trooper.

After we got home, I had a bit of a clean up before Zoe's friend from Kindergarten, Vaeda and her Mum came over for lunch. The girls had a good time playing together really nicely for a couple of hours afterwards.

I was flicking through 365 Science Experiments, looking for something physics-related for a change, when I happened on the perfect thing. The girls were already playing with a bunch of balloons that I'd blown up for them, so I just charged one up with static electricity and made their hair stand on end, and also picked up some torn up paper. Easy.

After Vaeda left, we did the weekend grocery shop early, since we were going away for the bulk of the weekend.

It was getting close to time to start preparing dinner after that. Anshu came over for dinner and we all had a nice dinner together.

Michael Still: Juno nova mid-cycle meetup summary: cells

Fri, 2014-08-15 14:29
This is the next post summarizing the Juno Nova mid-cycle meetup. This post covers the cells functionality used by some deployments to scale Nova.



For those unfamiliar with cells, it's a way of combining smaller Nova installations into a thing which feels like a single large Nova install. So for example, Rackspace deploys Nova in cells of hundreds of machines, and these cells form a Nova availability zone which might contain thousands of machines. The cells in one of these deployments form a tree: users talk to the top level of the tree, which might only contain API services. That cell then routes requests to child cells which can actually perform the operation requested.



There are a few reasons why Rackspace does this. Firstly, it keeps the MySQL databases smaller, which can improve the performance of database operations and backups. Additionally, cells can contain different types of hardware, which are then partitioned logically. For example, OnMetal (Rackspace's Ironic-based baremetal product) instances come from a cell which contains OnMetal machines and only publishes OnMetal flavors to the parent cell.



Cells was originally written by Rackspace to meet its deployment needs, but is now used by other sites as well. However, I think it would be a stretch to say that cells is commonly used, and it is certainly not the deployment default. In fact, most deployments don't run any of the cells code, so you can't really call them even a "single cell install". One of the reasons cells isn't more widely deployed is that it doesn't implement the entire Nova API, which means some features are missing. As a simple example, you can't live-migrate an instance between two child cells.



At the meetup, the first thing we discussed regarding cells was a general desire to see cells finished and become the default deployment method for Nova. Perhaps most people end up running a single cell, but in that case at least the cells code paths are well used. The first step to get there is improving the Tempest coverage for cells. There was a recent openstack-dev mailing list thread on this topic, which was discussed at the meetup. There was commitment from several Nova developers to work on this, and notably not all of them are from Rackspace.



It's important that we improve the Tempest coverage for cells, because it positions us for the next step in the process, which is bringing feature parity to cells compared with a non-cells deployment. There is some level of frustration that the work on cells hasn't really progressed in Juno, and that it is currently incomplete. At the meetup, we made a commitment to bringing a well-researched plan to the Kilo summit for implementing feature parity for a single cell deployment compared with a current default deployment. We also made a commitment to make cells the default deployment model when this work is complete. If this doesn't happen in time for Kilo, then we will be forced to seriously consider removing cells from Nova. A half-done cells deployment has so far stopped other development teams from trying to solve the problems that cells addresses, so we either need to finish cells, or get out of the way so that someone else can have a go. I am confident that the cells team will take this feedback on board and come to the summit with a good plan. Once we have a plan we can ask the whole community to rally around and help finish this effort, which I think will benefit all of us.



In the next blog post I will cover something we've been struggling with for the last few releases: how we get our bug count down to a reasonable level.



Tags for this post: openstack juno nova mid-cycle summary cells

Related posts: Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Juno nova mid-cycle meetup summary: DB2 support; Juno nova mid-cycle meetup summary: social issues



Comment