Planet Linux Australia

Syndicate content
Planet Linux Australia - http://planet.linux.org.au
Updated: 28 min 48 sec ago

Brad Hards: Microsoft releases PST specification document

Fri, 2014-07-04 15:26
KDE Project:

Looks like Microsoft has released the PST format specification.

I don't normally like to link to MSDN, but I'll do it this once:

http://msdn.microsoft.com/en-us/library/ff385210.aspx

As usual with these documents, I recommend reading the PDF version rather than the HTML. Also, Firefox seems to handle MSDN a bit better than my (KDE 4.3.5) Konqueror.

If you were a mad-keen PIM hacker, and looking for a GSoC project, might be worth a look.

[Thanks to Tom Devey for the heads-up on this]

Brad Hards: OpenChange BoF - LCA 2010

Fri, 2014-07-04 15:26
KDE Project:

I'm not at Camp KDE, but instead at LCA 2010 (in Wellington, NZ).

Andrew Tridgell, Andrew Bartlett, Jelmer Vernooij and I will be running "birds of a feather" (BoF) sessions during the last part of the conference (Friday 22 January 2010 starting at 1430 in the "Civic 3" room, which is over in the Town Hall building).

The first part of the BoF session will focus on Samba 4 and its ability to replace (or work with) Active Directory. I've been working on an installer GUI for Samba4 and OpenChange, and if you are a sysadmin who might be interested in installing this stuff yourself, then you can influence the design of the tool.

There will also be some demos.

If nothing else, you'll get to see Tridge hack, which is always fun.

So come along (remember 1430 to 1630 on Friday), hang out, observe, contribute and enjoy!

Brad Hards: Goodbye Okular

Fri, 2014-07-04 15:26
KDE Project:

The Okular team has never been all that big. Recently we lost Pino as the maintainer. His reasons are his reasons, but I can't say I blame him. I can personally no longer tolerate the level of abuse that we're seeing on bug reports. The latest example is Wishlist item 157284

I'm unsubscribed from the okular-devel mailing list. I'm not going to be in #okular. I'll still look at XPS bugs if I notice them.

It is difficult to leave. I really do care about Okular - I gave quite a lot of my time to improve it. It is a really nice application. However every time I looked at some bug comment insisting on a change (in something like the GUI that is subjective in the absence of actual usability study), I get disheartened. Even when I'm supposed to be working on something else, it gets to me. I've got better things to work on, where I'm not seeing that level of criticism of volunteer efforts.

I'd like to thank everyone involved in Okular, especially Albert and Pino for their maintainership, Piotr for the initial work, and Tobias for some inspirational work on the generator side.

Best wishes to Okular and its happy users.

Andrew Pollock: [life] Day 155: Jet lagged, some shopping and more swimming

Fri, 2014-07-04 11:25

The jet lag is totally kicking my butt at the moment. I woke up at about 1:30am and stayed awake, no matter what I tried. Zoe woke up briefly not long after me, but went back to sleep in my bed and slept until nearly 8:30am when I woke her up. She's making this whole thing look easy.

So after a few coffees to kick start myself, we went out to do a spot of shopping. Turns out I forgot to pack my swim gear and shaving cream, so we went to Target to get that, and also my favourite breakfast cereal.

After that, I went to the T-Mobile store to sort out a pre-paid SIM for my phone. It's good to have a number and some mobile data now.

We had lunch at Chick-fil-A and the kids all had a bit of a play in the play area.

I crashed out for nap again after we got home. I'm slightly concerned I might be getting a sinus infection, but it's hard to tell if it's just the massive sleep deficit I'm running.

After everyone resurfaced from their various naps, (Zoe just had some quiet time playing on her tablet), we went to the pool again for a swim. Zoe got her confidence up and was swimming back to the side from where she dived in off the diving board, without requiring the use of a float.

The girls had a great hair washing line happening at bath time, which made that nice and easy.

I'm really hoping I can sleep better tonight.

Andrew Pollock: [life] Day 154: Pottering around in Decatur, museum and swimming

Thu, 2014-07-03 11:25

Zoe woke up at about 2am, Atlanta time. That corresponded with about dinner time in Brisbane. I'd actually woken up slightly before she did. She'd crawled into bed with me not long after going to bed the night before.

I'm not sure what time she went back to sleep, but I think it was around 4am. She slept until 7:30am when I woke her up to have a video call with Sarah before she went to bed. Unfortunately I wasn't able to get back to sleep, so I felt like a bit of a zombie by morning. A couple of cups of coffee helped take the edge off it for a while.

Once we got going, we went to Fernbank Museum of Natural History. Zoe got along really well with Clara, 7 and Lucy, 4. The museum was pretty nice. it had a kid's area that kept everyone occupied for a quite a while.

We got back and had some lunch, and then I went and had a nap while James, 2, also had one. It took me about an hour to actually get up after I woke up from the nap. I'm glad I had it though, because I don't think I'd have made it through the day. Zoe apparently had a great time playing with Lucy while I napped.

After that, we went to the neighbourhood pool for a swim. The pool had a low diving board, and Zoe had lots of fun jumping in off it.

I managed to get Zoe to bed relatively on time, and she passed out straight away. Hopefully with a better dinner in her belly, she'll sleep better tonight. I'm hoping I do too.

Andrew Pollock: [life] Day 153: The long day of traveling

Thu, 2014-07-03 11:25

"Yesterday" was a very long day. I'm sure I'm going to lose track of which day is what somewhere along the line...

I had one of the best night's sleep I've had the night before I know I need to get up early for travel. Usually I spend the whole night telling myself to hurry up and get to sleep because I have to get up early, and thus fail to do so and sleep terribly. I slept well, and was showered, dressed and breakfasted before Zoe woke up naturally herself.

We were all packed up and downstairs with our luggage before Sarah arrived to take us to the airport.

We had a comfortable amount of time at Brisbane airport waiting for our flight down to Sydney. I haven't flown Qantas for a while, and they now have iPads in all of the seat pockets as their in-flight entertainment. That kept Zoe nicely amused for that flight.

In Sydney, we transited over to the International terminal without incident. Zoe's a great size for her Trunki now, and dragging her along on that allowed for us to cover distance at pretty much my walking pace, which was perfect.

We got to the International terminal and cleared Customs with enough time grab some lunch before we boarded. They were boarding a flight to LAX and DFW at adjacent gates at roughly the same time, and in conjunction with the additional round of document checks that they like to do for flights to the US it was total bedlam. We got to the gate just as they were starting the pre-boarding for people with children, so that worked out well, and we got to jump the queue.

Zoe did absolutely fabulously for the flight from Sydney to Dallas. She watched some movies and then went to sleep only a little bit later than her normal bed time and pretty much slept all the way through until I had to wake her up for breakfast, which was at about 3:30am Brisbane time. She didn't sleep again until we got to Atlanta, but wasn't the worse for wear for it. My head cold also didn't give me much grief. I didn't need to resort to any drugs on the flight.

Our arrival into Dallas went pretty well. The only thing I could have done better was figure out where the car seat had been put a bit sooner (it wasn't put with where the signed oversize baggage was) so we burned a bit of time finding it. They have a new electronic self-service kiosk thing for their border control. It was pretty cool, but it still bounced us to a human for processing. I'm not sure if it was because I had declared food, or because I was traveling with a minor US citizen, or some combination of the above.

The US border control did a better job of handling the fact that I was traveling on my own with a child than the Australian passport control did on the way out. The Australian departure passport control didn't seem to care at all that I was traveling on my own with Zoe, but the US guy wanted to know where her mother was, and what my wife did. It wasn't until I said that she was almost my ex-wife that he wanted to know if she knew I was traveling with Zoe, and I produced the letter I had brought with me as a precaution. That was the extent of the questioning I received (aside from how I planned to support myself for my stay).

That said, even though I had planned for that level of questioning, the interaction still left me feeling a bit unsettled. I can expect at least that much scrutiny for any travel I do with Zoe from now on. I guess it's a good thing, but I don't like the default assumption that I'm possibly an international child abductor. It also left me wondering if the same level of scrutiny would occur if Zoe was traveling with her mother, or if it was just a gender stereotype.

We cleared immigration to then get stuck in a longish line for Quarantine, which went fine. I had hoped that we could recheck our luggage immediately on the other side, like I believe you can do in LAX, but that was not the case. We had to schlepp all our luggage over to the Delta terminal on one of the shuttles. We possibly missed a shuttle while I was figuring all this out.

By the time we got a shuttle and got to the right terminal, we'd just missed the bag check deadline for our flight, so we had to get bumped from the 3:45pm flight to the 5pm one. It wasn't a big deal, but it delayed our pick up in Atlanta, and made for a longer day of traveling. Delta was really nice and waived all the baggage fees though, since we were coming from an international flight. That was nice.

That said, it did give us a more leisurely time in the terminal in Dallas. I had time to call Chris and Briana and let them know of our delay, and get Zoe something to eat.

The flight from Dallas to Atlanta was uneventful. Zoe entertained herself with her tablet on the flight.

When we arrived in Atlanta, Zoe remembered the inter-terminal train from her last trip with Sarah. We collected our luggage and rendezvoused with Chris on the kerb.

We had a hasty dinner and Briana gave Zoe a bath for me, and I put her to bed, and went to bed not long after her. It ended up being a mammoth day, in the order of about 32 hours door to door. I was so incredibly impressed with how well Zoe handled it all. Not one single meltdown. The only time she was really sad was briefly when she managed to overbalance on the chairs in the Sydney International terminal just before we boarded our flight, and landed on her head, all while I was putting our passports away and not watching her.

Michael Still: Configuring a spa3000 or spa3102

Thu, 2014-07-03 10:27
I've had a spa3000 for a long time and then it blew up. That meant I had to replace it with a spa3102, but without being able to refer to the configuration I had on the spa3000. It wasn't too bad, although the software on these devices is truly terrible (even with a modern firmware). Some pointers:



  • Use the WAN port on the spa3102, even if you're only plugging in one ethernet port.
  • Get firmware from Cisco. You can upload this to the spa3102 by putting it on a web server and then hitting http://SPA3102_IP/admin/upgrade?http://yourwebserver/firmware.bin.
  • Then follow the instructions here.
  • When you have problems with the dial plan, realize that Chris Yeoh has already solved this and go here.
  • Finally, fix hangup detection by stealing the config line from here.
  • Job done.




Of course, there's no way to backup the config on one of these devices, so I am sure I will be back here again.



Tags for this post: voip ata spa3000 spa3102 cisco linksys sip

Related posts: Setting up Cisco 7961 IP phones with asterisk; Slides from the NSLU2 talk at LUV; Debian bootstrap complete; Configuring Asterisk



Comment

Michael Still: Interesting things found whilst surfing

Thu, 2014-07-03 10:27
Some interesting things I found recently that I think others might be interested in:







You'll notice all of these videos are from Matthias Wandel. He has a YouTube channel and seems like my sort of geek. I am sure there will be more links from him as I work through his 7 years (!) of videos.



Tags for this post: link wood gadget

Related posts: A couple of days of turning; More wood turning; Initial play with wood turning; Another bowl



Comment

Donna Benjamin: Drupal Diversity

Thu, 2014-07-03 09:26
Thursday, July 3, 2014 - 09:16

Drupal project founder Dries just published a blog post entitled Fostering inclusivity and diversity.

I took this photo at DrupalCamp Sydney last weekend. Not only were these three men from very different parts of the world, but they also had different levels of expertise, and different interests. Sure, no women in this picture, but I was taking the photo ;-)

max, taps and vignesh at #dcsyd2014 #drupalcampsydney  pic.twitter.com/GbL4vj508f

— kattekrab (@kattekrab) June 29, 2014

Difference is a good thing. Erynn Petersen spoke at DrupalCon Austin about the real bottom line advantage diversity brings to teams. She also spoke about the importance of tolerating some level of conflict and dissent. It makes us stronger. Challenging and testing ideas usually makes those ideas better.

It's a balancing act.

Donna Benjamin: DrupalSouth Melbourne 5-7 March 2015

Wed, 2014-07-02 20:26
Wednesday, July 2, 2014 - 20:09

Hey @drupal Beware the ides of March. Over 5-7 March 2015 @DrupalSouth is coming. See http://t.co/YP1LJMSD30 & sign up for announcements

— kattekrab (@kattekrab) June 29, 2014

Craige McWhirter: Making Your Debian Workstation an OpenStack Management Client

Wed, 2014-07-02 14:27
Assumptions:
  • You've already created your OpenStack tenant.
  • You've already created your OpenStack user.
  • You've already given that user permissions to that tenant.
Pre-Requisites for Debian 7.x workstations only:

If you're running testing or unstable, you have the lastest packages so move onto the next section. However if you're running Debian 7.x (wheezy) you will need to first add these repositories:

# echo "deb http://archive.gplhost.com/debian icehouse main" >>/etc/apt/sources.list # echo "deb http://archive.gplhost.com/debian icehouse-backports main" >>/etc/apt/sources.list

and update your repository database before installing the key:

$ sudo apt-get update && sudo apt-get install gplhost-archive-keyring Install the Clients!:

Update your repository database then install the meta package openstack-clients:

$ sudo apt-get update && sudo apt-get install openstack-clients The following NEW packages will be installed: ieee-data{a} openstack-clients python-babel{a} python-babel-localedata{a} python-cinderclient{a} python-cliff{a} python-cliff-doc{a} python-cmd2{a} python-colorama{a} python-crypto{a} python-d2to1{a} python-distlib{a} python-eventlet{a} python-glanceclient{a} python-greenlet{a} python-heatclient{a} python-html5lib{a} python-httplib2{a} python-iso8601{a} python-json-patch{a} python-json-pointer{a} python-jsonpatch{a} python-jsonschema{a} python-keyring{a} python-keystoneclient{a} python-mock{a} python-netaddr{a} python-neutronclient{a} python-novaclient{a} python-oslo.config{a} python-pbr{a} python-pip{a} python-prettytable{a} python-pyparsing{a} python-secretstorage{a} python-setuptools{a} python-simplejson{a} python-stevedore{a} python-swiftclient{a} python-tz{a} python-warlock{a} python-wheel{a} python-yaml{a} 0 packages upgraded, 43 newly installed, 0 to remove and 3 not upgraded. Need to get 5,780 kB of archives. After unpacking 30.8 MB will be used. Do you want to continue? [Y/n/?] y

You'll notice that it installs a whole world of Python. Python was cool in 2005.

Configure Your Clients:

To configure your OpenStack clients you merely need to set four environment variables:

  • OS_USERNAME - this contains your OpenStack username.
  • OS_PASSWORD - this contains your OpenStack password.
  • OS_TENANT_NAME - the name of the OpenStack tenant you wish to connect to.
  • OS_AUTH_URL - the URL to your OpenStack authentication service.

The easiest way to do this is to put them in a file to be sourced when needed. Your TENANT_NAME-openrc.sh may look like this:

export OS_USERNAME=your_username export OS_PASSWORD=your_password export OS_TENANT_NAME=your_tenant_name export OS_AUTH_URL=http://keystone.your.openstack.domain:35357/v2.0

When you need to use your OpenStack clients, you can source the credentials as follows::

$ source TENANT_NAME-openrc.sh

This also allows you to perform operations as different users or on different tenants by sourcing different openrc files.

Craige McWhirter: Find the IP Address of a Virtual Machine

Wed, 2014-07-02 11:27

So, you're running libvirt, you've run up a new VM that's using DHCP and you'd like to know the IP address without logging into the console.

If you haven't already, set LIBVIRT_DEFAULT_URI. The below example assumes it's localhost:

$ export LIBVIRT_DEFAULT_URI=qemu:///system

Next you need to obtain the virtual machine's MAC address:

$ virsh dumpxml vm-name | grep "mac address"

Then you can use the MAC address to get the running VM's IP address:

$ /usr/sbin/arp -an | grep mac-address

So an example of this would look like this:

$ export LIBVIRT_DEFAULT_URI=qemu:///system $ virsh dumpxml debian-7 | grep "mac address" <mac address='52:54:00:ec:fe:cd'/> $ /usr/sbin/arp -an | grep 52:54:00:ec:fe:cd ? (192.168.122.197) at 52:54:00:ec:fe:cd [ether] on virbr0

You can also write a nice little bash script to do this for you:

#!/bin/bash # # This script uses virsh and arp to determine the IP address of a VM # Where are your tools? virsh=/usr/bin/virsh arp=/usr/sbin/arp grep=/bin/grep sed=/bin/sed # Get the VM name as an argument: VM_NAME=$1 # Ensure the default URI is set for your environment: export LIBVIRT_DEFAULT_URI=qemu:///system # Obtain the MAC address from libvirt: MAC_ADDRESS=`$virsh dumpxml $VM_NAME | $grep "mac address" | $sed "s/.*'\(.*\)'.*/\1/g"` # Use arp to find the IP address you're looking for via it's MAC address: $arp -an | $grep $MAC_ADDRESS exit 0

An example output of that script looks likes this:

$ ./vm_ip.sh debian-7 ? (192.168.122.197) at 52:54:00:ec:fe:cd [ether] on virbr0

You can get a copy of that script here.

Ben Martin: The long road to an Autonomous Vehicle.

Tue, 2014-07-01 23:20
Some people play tennis, some try to build autonomous wheeled robots. I do the later and the result has come to be known as “Terry”. In the long road toward that goal what started as a direct drive, “give the motor a PWM of X% of the total power”, lead to having encoder feedback so that the wheel could be turned an exact amount over a given time.  This has meant that the control interface no longer uses a direct power and direction slider but has buttons to perform specific motion tasks. The button with the road icon and a 1 will move the wheels forward a single rotation to advance Terry 6π inches forwards (6 inch wheels).



The red tinted buttons use the wheel encoders to provide precision movement. Enc left and Enc right turn Terry in place (one wheel forward, one wheel backwards). The Circus and Oval perform patrol like maneuvers. Circus moves forward, turns in place, returns back, and turns around again. It's like a strafing patrol. On the other hand, the Oval turns move like a regular car, with both wheels going forward but one wheel moving slower than the other to create the turning effect. The pan and tilt control the on board camera for a good Terry eye view of the world.



I added two way web socket communications to Terry this week. So now the main battery voltage is updated and the current, err, current for each motor is shown at the bottom of the page. The two white boxes there give version information so you can see if the data is stale or not. I suspect a little async javascript on some model value will be coming so that items on the page will darken as they become stale due to lag on Terry or communications loss.



I've been researching proximity detection and will be integrating a Kinect for longer distance measures soon. Having good reliable distance measurements from Terry to objects is the first step in getting him to create maps of the environment for autonomous navigation.



Russell Coker: Blog Comments

Tue, 2014-07-01 14:26

John Scalzi wrote an insightful post about the utility of blog comments with the way the Internet works nowadays [1]. He starts out focusing on hate comments that could reasonably be described as terrorism (death threats with the aim of preventing people writing about politics meet any reasonable definition of “terrorism”). Terrorists on the Internet are a significant problem but it’s one that doesn’t get much attention as it generally only affects people who aren’t straight-acting white men.

Blogging About Technology

One corner case that John doesn’t seem to consider is that of writing about technology. Issues related to programming often aren’t related to politics and are often testable so comments will be based on things that have been shown to work rather than stuff people invent or want to believe. I’ve received many useful and educational comments on my technical posts with little hostility. Even getting a snarky comment is rare when writing a strictly technical blog post.

The comments problem for technology blogging is spam. I’ve been using the WordPress plugin Block Spam by Math [2] (which is obsolete but still works) for years. Initially it stopped almost all spam, but now I’m getting at least 20 spam comments a day.

Extremely Popular Blogs

The comments section of a blog is sometimes described as a “conversation”. When a blog post gets comments from less than 10 people it is possible for them to have something that resembles a conversation with the author that is of benefit to other readers and doesn’t take excessive amounts of time for the author. When a blog is very popular and every post gets comments from 50+ people it’s not really possible. So a traditional blog comment section seems to work best when the blog is primarily read by a small well connected group of people who sometimes comment and some casual readers who never comment (but sometimes find value in the comments of others).

Discussions of blog comment systems usually include a reference to a post written by someone who disabled comments on their blog and found it to be a good thing, it always seems that the person who writes such a post has a large and varied audience who’s comments would take a lot of time to moderate. John followed the usual form in this regard by linking to a reasonably popular SF author who would presumably have a lot of fans with good net access.

I’m not going to criticise anyone for disabling comments when their blog becomes really popular, but any advice that they have to offer about such things won’t apply to the vast majority of blogs. Due to the long-tail effect the small blogs would probably comprise the majority of all comments so in terms of the way the blog environment works I don’t think it makes much difference when the small minority of very popular blogs disable comments. The vast majority of blogs that I regularly read only have a small number of comments.

One thing that should be noted is that getting a lot of readers shouldn’t be the only factor for writing a successful blog. For example some of my blog posts about SE Linux are aimed at a small audience of Linux programmers and have an even smaller number of people who are qualified to comment. When I write a post that can only receive comments other than “please explain more because I don’t understand” from a few dozen people that doesn’t make it any less important. Sometimes the few dozen people who know a topic well need to work together to educate the few thousand who can implement the ideas for the benefit of millions of users of the software.

Disabling Comments on Contentious Posts

One interesting method John uses is to disable comments early when posting about contentious issues. It’s a general practice when running a blog to disable comments on posts after a certain period of time (3 months to 1 year seem to be common time limits for comments). This means that the moderators can concentrate on recent posts and not be bothered with spam bots hitting ancient posts as the interest in writing legitimate comments on an old post is vanishingly small. John has a practice of disabling comments after a couple of days when the comments start to lose quality.

No matter how contentious the issue is I’m not likely to get the 400+ comments that John gets. But the idea of closing comments quickly still has some merit for my blog and other blogs with less traffic.

Not Accepting Comments While Asleep

John has a practice of closing comments while he’s asleep to avoid allowing a troll to get 8 hours of viewing for a nasty comment. The most immediate down-side to that is that it inconveniences people who don’t want to wait 8 hours to comment and prioritises comments from people in the same time zone, this makes me think of Cory Doctorow’s novel Eastern Standard Tribe (which is available for free download and I highly recommend reading it) [3]. It seems that a better solution to that problem would be to have a team of moderators to watch things 24*7 which is what a lot of popular blogs that allow comments do. The WordPress capabilities model doesn’t support granting a user no special privileges other than moderating comments [4], as WordPress is the most popular self-hosted blog software this limits the possibilities for people moderating comments on other people’s blogs.

No variation of this would work for me. I have lots of things that require my ongoing attention and don’t want to add my blog to the list. If I have other things to work on for a few days I want to just not bother with my blog. This means that my blog needs to be able to run on autopilot for days at a time – however I do monitor my blog closely after publishing a post that is likely to attract nasty comments. One extra problem that I have is that the Android client for WordPress has problems in synchronising comments.

Using a Forum for Comments

Popular Planet installations such as Planet Debian and Planet Linux Australia syndicate more than a few blogs that have comments disabled. A forum installation for such a Planet would be useful to allow people to comment on all posts and also support bloggers who are thinking of disabling comments. While the use of a forum for blog comments has been proven to work well for Boing Boing forums have their own issues of spam and anti-social behavior.

Debian already has a forum [5], if a section of that was devoted to discussing blog posts from Planet Debian then it shouldn’t make much of an increase to the work of the forum administrators while providing a benefit to the community. Also if the Debian forum had such a section it would probably attract use from more Debian Developers, I would use that forum if it was a place to comment on blogs that don’t have a comment section and I might also comment on other forum discussions.

It would be good if there was a forum for discussing Linux in Australia. I’m not volunteering to run it but I would help out if someone else wants to be the main sysadmin and I can offer free hosting.

Related posts:

  1. reviewing blog comments and links It seems that the swik.net site is mirroring all my...
  2. a Heartbeat developer comments on my blog post Alan Robertson (a major contributor to the Heartbeat project) commented...
  3. Blog Friends? There are some people who’s blogs I read and often...

David Rowe: Reducing FDMDV Modem Memory

Tue, 2014-07-01 13:28

For the SM1000 (SmartMic) project I need to run the FDMDV HF modem on the STM32F4 micro-controller. However The STM32F4 only has 192k of internal RAM, and the modem in it’s original form uses over 400k. I wrote a unit test to break down the memory usage:



david@bear:~/tmp/codec2-dev/build_dir$ ./unittest/fdmdv_mem

struct FDMDV..........: 409192

prev_tx_symbols.......: 168

tx_filter_memory......: 1008

phase_tx..............: 168

freq..................: 168

pilot_lut.............: 5120

pilot_baseband1.......: 1840

pilot_baseband2.......: 1840

pilot_lpf1............: 5120

pilot_lpf2............: 5120

S1....................: 2048

S2....................: 2048

phase_rx..............: 168

rx_filter_memory......: 161280

rx_filter_mem_timing..: 3360

rx_baseband_mem_timing: 215040

phase_difference......: 168

prev_rx_symbols.......: 168

fft_buf...............: 4096

kiss_fft_cfg..........: 4



Looks like the memory usage is dominated by just two arrays. This modem has a rather high “over sampling rate”, M. The symbol (or baud) rate of each carrier is just 50 Hz. However the sample rate at the output is 8000 Hz, which gives M = 8000/50=160. We tend to do our processing at the symbol rate. This means that for every symbol we process, we need 160 samples. If we are running at 1600 bit/s there are 17 carriers (one is used for sync and carries no data). For some operations (like filtering), we need a record of the last 5 symbols. Each sample is 8 bytes as it’s a complex number (cos and sin sample) with two floats. So it’s easy to chew up say (160)(17)(5)(8)=108,800 bytes in a single array.

However the are many ways to implement a modem. By re-arranging the processing steps we can sometimes save memory and/or MIPs, or trade them off to get the combination we want for our target platform.

New Re-sampler

The first step was to change the re-sampling algorithm. The demodulator (figure below) has a timing estimation routine that works out the best place to sample a received symbol. This is important for modem performance, if you don’t sample at the right place you get noisy symbols and more errors. Often the position we wish to sample is half way between two existing samples. So we need a way to work out the value of a sample between two existing samples.

The original algorithm re-ran the 5 symbol filter at the optimal timing point. As we have oversampled by a large factor, we can choose any of the M=160 output samples per symbol. However this meant keeping a large memory buffer (rx_baseband_mem_timing) above. Instead, I implemented a linear re-sampler. This just fits a straight line between two existing filtered samples, then uses the year 8 equation of a straight line y=mx + c to find the new sample y at the optimum timing instant x.

The scatter diagram is a good way to evaluate the new, linear re-sampler. The points should be nice and tight. Here are the scatter diagrams for the original filter based re-sampler, followed by the linear re-sampler.

Any modem changes like this can have a big impact on Bit Error Rate (BER) performance, so we need to test very carefully. I used the Octave fdmdv_ut.m to measure the BER at the nominal operating point SNR of 4dB:



octave:3> fdmdv_ut

Bits/symbol.: 2

Num carriers: 14

Bit Rate....: 1400 bits/s

Eb/No (meas): 7.30 (8.20) dB

bits........: 2464

errors......: 16

BER.........: 0.0065

PAPR........: 13.02 dB

SNR...(meas): 3.99 (5.51) dB



The BER is about the same as the previous version so all good. Sixteen errors isn’t very many, so I also tested at a lower Eb/No (SNR) to get more than 100 errors.

Storing One Signal Instead of 17

The next array to tackle was rx_filter_memory. The demod takes the FDM modem signal centred on 1500 Hz, then downconverts it to 17 parallel baseband signals, one for each carrier. To make matters worse we need to store 5 symbols worth of each baseband signal. This uses a lot of storage. So I changed the processing steps to:

  1. Keep 5 symbols worth of demod input samples (the FDM signal centred on 1500 Hz). This is just one signal, rather than 17.
  2. For each carrier, downconvert all 5 symbols to baseband, filter, then throw away the baseband signal, freeing the memory.

This is wasteful of CPU, as we downconvert 5 symbols worth (5*160 samples), rather than just the 160 new samples. However it saves a lot of memory. The final memory breakdown uses 10% of the original:



david@bear:~/tmp/codec2-dev$ ./build_dir/unittest/fdmdv_mem

struct FDMDV..........: 41916

prev_tx_symbols.......: 168

tx_filter_memory......: 1008

phase_tx..............: 168

freq..................: 168

pilot_lut.............: 5120

pilot_baseband1.......: 1840

pilot_baseband2.......: 1840

pilot_lpf1............: 5120

pilot_lpf2............: 5120

S1....................: 2048

S2....................: 2048

phase_rx..............: 168

rx_fdm_mem............: 8960

rx_filter_mem_timing..: 3360

phase_difference......: 168

prev_rx_symbols.......: 168

fft_buf...............: 4096

kiss_fft_cfg..........: 4



Automated Testing

I use a set of automated tests to check the C and Octave versions are identical. First I run the C version, which processes 25 frames of modem data and logs all of the internal states as Octave-format vectors which are saved to a text file.



david@bear:~/tmp/codec2-dev/unittest$ pushd ../build_dir/ && make && popd && ../build_dir/unittest/tfdmdv



The command line is a bit tricky as the new Cmake build system builds out of the source tree, and my program needs to run from the unittest directory in the source tree. Next we run the Octave version, which should be identical. Some automated tests make sure they are. Actually as they are two different float implementations I make sure the total error between a C and Octave vector is within 1 part in 1000:



octave:4> tfdmdv

tx_bits..................: OK

tx_symbols...............: OK

tx_baseband..............: OK

tx_fdm...................: OK

pilot_lut................: OK

pilot_coeff..............: OK

pilot lpf1...............: OK

pilot lpf2...............: OK

S1.......................: OK

S2.......................: OK

foff_coarse..............: OK

foff_fine................: OK

foff.....................: OK

rx filt..................: OK

env......................: OK

rx_timing................: OK

rx_symbols...............: OK

rx bits..................: OK

sync bit.................: OK

sync.....................: OK

nin......................: OK

sig_est..................: OK

noise_est................: OK

 

passes: 23 fails: 0



The Octave version also plots the C and Octave states (vectors) against each other, so you can work out what went wrong. This one is the output of the rx filter, which was the site of my recent rewiring:

You may be interested in software development “process”, or delegating software work to teams of people with varying skills levels, or making a profit from software, or a business that requires software development to not destroy said business. I have used similar, bit exact, schemes for fixed point DSP development, which I whimsically describe in this allegorical tale.

These automated tests give me a lot of confidence. So many things can go wrong with a complex system. So as painful as it is, it’s worthwhile to have some quality “gates” wherever you can. Now when I move to the STM32F4 micro-controller, I can be reasonably sure there aren’t any algorithm or C porting bugs. Reasonably sure.

Lets Do the Time Warp Again

Check out this little snippet of code:



        /*

          now downconvert using current freq offset to get Nfilter+nin

          baseband samples.

    

                     Nfilter             nin

          |--------------------------|---------|

                                      |

                                  phase_rx(c)

    

          This means winding phase(c) back from this point to ensure

          phase continuity.

        */

 

        windback_phase           = -freq_pol[c]*NFILTER;

        windback_phase_rect.real = cos(windback_phase);

        windback_phase_rect.imag = sin(windback_phase);

        phase_rx[c]              = cmult(phase_rx[c],windback_phase_rect);



There are a bunch of local oscillators that are defined by their current phase phase_rx[c], and frequency freq[c]. To calculate the next oscillator output sample we increment the current phase by the frequency. That’s how you make an oscillator in software. It’s normally all done in rectangular coordinates (real and imaginary parts, cos and sin, or in-phase and quadrature depending on where you went to school) as it’s easy on the CPU.

Now, we need to down-convert the signal from 5 symbols (960 samples) in the past. So we need to “wind” the oscillator phase backwards to where it was 960 samples ago. Think of the phase like a hand on a clock. We normally increment it a few minutes for every sample but now we want to wind it back several “days”.

Now that I can send oscillators backwards in time I’ll get to work on that warp drive. Or breaking the Shannon limit. Now that would be useful for these HF channels.

Is There a Better Way?

This work took me about an hour of creative thinking (the fun bit) and several days of implementation pain, off by one errors, fighting to understand filter memories (again), and tracking down differences between the Octave and C versions.

I can’t help wondering if there is an easier way, like snapping something together in GNU radio, or some better way of expressing digital filters in software. A more domain-specific language, or some set of functions that take care of filter memories being shifted so my head doesn’t hurt any more. If the code is easier to understand is would be more hack-able too, and maintainable, and be a better tool for teaching.

I have tried various “toolkit” ideas over the years. They were all going to make DSP software development painless and fast. These sorts of tools are great fun to implement but the idea of a near-zero effort implementation utopia tends to break down somewhere in the messy details. So here I am still grinding away with Octave and C, like I have been since 1990.

Or maybe it’s just hard, and it isn’t going to get any easier? Like hacking the Linux kernel? I mean, I am trying to replace SSB …..

Daniel Patton: jizzlobber - faith no more

Tue, 2014-07-01 05:25
Joy!

I just got my tv tuner card working through Linux, would be just about as long as I've had my computer that I've been trying to do that.



On a similar subject, I've been out and bought some necessary supplies for L.C.A, not least of which is a nifty new bag for my new laptop, cheap too, only $50 AUD.



I'll post some photos tomorrow, can't be bothered chasing up the camera right now.



Oh yeah, seems photobucket has changed their service, so you can't post screenshots anymore. Looks like I'll have to get my own web server up soon.



So, Until tomorrow,

Chow.

Daniel Patton: Stuff

Tue, 2014-07-01 05:25
I finally have enough time to update my journal! Incredible.



On the work subject; I finally know how much I can 'blog about. So, the company that's employing me is Global Multimedia, as a computer technician. I can't really name the schools I'm working at until I've gained permission from each one individually.



On the journal subject; Crimson_wishes (Trinergy's new alt) has just added me to her friends list on livejournal, so a few new people might be reading my rants through her friends page. Hi all!



On the programming subject; A little more work on GNOME-MUD has been done as a result of a little prodding by me, I started out with a patch I thought solved a crash on 64 bit systems, but it turns out it was the wrong thing to do, that still got the same result in this case. So, checkout HEAD branch if you have troubles running GNOME-MUD on 64 bit systems. If you like that kind of thing, that is.



On the l.c.a subject; It's nearly all sorted out now, the only thing now is to install Ubuntu on my shiny new laptop, transfer my key to it, pack, and get to the airport! Joy.

Daniel Patton: tenacious d - tribute

Tue, 2014-07-01 05:25
Seems I've done well with the company I'm been working with, the probation period is up and they want me to stay on.



Sweet.



Unfortunately, I still don't know how much I can blog about.

Daniel Patton: Car crash

Tue, 2014-07-01 05:25
Lisa and I were in a car accident on Thursday. Luckily, no-one was injured, just our cars.



She was driving her car, with me in the passenger seat, down near Altona Library. Some stooge decided they'd go right through a give way sign, and ploughed straight into us, pushing us all into a car waiting at a give way sign on the other side. The accident instigator was being a bit funny about the story, so the police were called in to take statements from everybody. Looks like a bit of court time & probable demerit points for the instigator. Lisa's car had to be towed, along with the instigators. Probably a couple of weeks before it's fixed. Very annoying, as she depends on her car for a lot.



Sigh.

Daniel Patton: Sin City

Tue, 2014-07-01 05:25
Just watched Sin City last night, I really enjoyed it.



Brittany Murphy is a really sucky actor though. Obviously hired for the tits.