Planet Linux Australia

Syndicate content
Planet Linux Australia - http://planet.linux.org.au
Updated: 1 hour 28 min ago

OpenSTEM: This Week in HASS – term 1, week 8

Mon, 2017-03-20 10:04

As we move into the final weeks of term, and the Easter holiday draws closer, our youngest students are looking at different kinds of celebrations in Australia. Students in years 1 to 3 are looking at their global family and students in years 3 to 6 are chasing Aunt Madge around the world, being introduced to Eratosthenes and examining Shadows and Light.

Foundation to Year 3

Our standalone Foundation/Prep students (Unit F.1) are studying celebrations in Australia and thinking about which is their favourite. It may well be Easter with its bunnies and chocolate eggs, which lies just around the corner now! They also get a chance to consider whether we should add any extra celebrations into our calendar in Australia. Those Foundation/Prep students in an integrated class with Year 1 students (Unit F.5), as well as Year 1 (Unit 1.1), 2 (Unit 2.1) and 3 (Unit 3.1) students are investigating where they, and other family members, were born and finding these places on the world map. Students are also examining features of the world map – including the different continents, North and South Poles, the equator and the oceans. Students also get a chance to undertake the Aunt Madge’s Suitcase Activity, in which they follow Aunt Madge around the world, learning about different countries and landmarks, as they go. Aunt Madge’s Suitcase is extremely popular with students of all ages – as it can easily be adapted to cover material at different depths. The activity encourages students to interact with the world map, whilst learning to recognise major natural and cultural landmarks in Australia and around the world.

Years 3 to 6 Aunt Madge

Students in Year 3 (Unit 3.5), who are integrated with Year 4, as well as the Year 4 (Unit 4.1), 5 (Unit 5.1) and 6 (Unit 6.1) students, have moved on to a new set of activities this week. The older students approach the Aunt Madge’s Suitcase Activity in more depth, deriving what items Aunt Madge has packed in her suitcase to match the different climates which she is visiting, as well as delving into each landmark visited in more detail. These landmarks are both natural and cultural and, although several are in Australia, examples are given from around the world, allowing teachers to choose their particular focus each time the activity is undertaken. As well as following Aunt Madge, students are introduced to Eratosthenes. Known as the ‘Father of Geography’, Eratosthenes also calculated the circumference of the Earth. There is an option for teachers to overlap with parts of the Maths curriculum here. Eratosthenes also studied the planets and used shadows and sunlight for his calculations, which provides the link for the Science activities – Shadows and Light, Sundials and Planets of the Solar System.

Next week is the last week of our first term units. By now students have completed the bulk of their work for the term, and teachers are able to assess most of the HASS areas already.

 

sthbrx - a POWER technical blog: Erasure Coding for Programmers, Part 1

Mon, 2017-03-20 09:43

Erasure coding is an increasingly popular storage technology - allowing the same level of fault tolerance as replication with a significantly reduced storage footprint.

Increasingly, erasure coding is available 'out of the box' on storage solutions such as Ceph and OpenStack Swift. Normally, you'd just pull in a library like ISA-L or jerasure, and set some config options, and you'd be done.

This post is not about that. This post is about how I went from knowing nothing about erasure coding to writing POWER optimised routines to make it go fast. (These are in the process of being polished for upstream at the moment.) If you want to understand how erasure coding works under the hood - and in particular if you're interested in writing optimised routines to make it run quickly in your platform - this is for you.

What are erasure codes anyway?

I think the easiest way to begin thinking about erasure codes is "RAID 6 on steroids". RAID 6 allows you to have up to 255 data disks and 2 parity disks (called P and Q), thus allowing you to tolerate the failure of up to 2 arbitrary disks without data loss.

Erasure codes allow you to have k data disks and m 'parity' or coding disks. You then have a total of m + k disks, and you can tolerate the failure of up to m without losing data.

The downside of erasure coding is that computing what to put on those parity disks is CPU intensive. Lets look at what we put on them.

RAID 6

RAID 6 is the easiest way to get started on understanding erasure codes for a number of reasons. H Peter Anvin's paper on RAID 6 in the Linux kernel is an excellent start, but does dive in a bit quickly to the underlying mathematics. So before reading that, read on!

Rings and Fields

As programmers we're pretty comfortable with modular arithmetic - the idea that if you have:

unsigned char a = 255; a++;

the new value of a will be 0, not 256.

This is an example of an algebraic structure called a ring.

Rings obey certain laws. For our purposes, we'll consider the following incomplete and somewhat simplified list:

  • There is an addition operation.
  • There is an additive identity (normally called 0), such that 'a + 0 = a'.
  • Every element has an additive inverse, that is, for every element 'a', there is an element -a such that 'a + (-a) = 0'
  • There is a multiplication operation.
  • There is a multiplicative identity (normally called 1), such that 'a * 1 = a'.

These operations aren't necessarily addition or multiplication as we might expect from the integers or real numbers. For example, in our modular arithmetic example, we have 'wrap around'. (There are also certain rules the addition and multiplication rules must satisfy - we are glossing over them here.)

One thing a ring doesn't have a 'multiplicative inverse'. The multiplicative inverse of some non-zero element of the ring (call it a), is the value b such that a * b = 1. (Often instead of b we write 'a^-1', but that looks bad in plain text, so we shall stick to b for now.)

We do have some inverses in 'mod 256': the inverse of 3 is 171 as 3 * 171 = 513, and 513 = 1 mod 256, but there is no b such that 2 * b = 1 mod 256.

If every non-zero element of our ring had a multiplicative inverse, we would have what is called a field.

Now, let's look at a the integers modulo 2, that is, 0 and 1.

We have this for addition:

+ 0 1 0 0 1 1 1 0

Eagle-eyed readers will notice that this is the same as XOR.

For multiplication:

* 0 1 0 0 0 1 0 1

As we said, a field is a ring where every non-zero element has a multiplicative inverse. As we can see, the integers modulo 2 shown above is a field: it's a ring, and 1 is its own multiplicative inverse.

So this is all well and good, but you can't really do very much in a field with 2 elements. This is sad, so we make bigger fields. For this application, we consider the Galois Field with 256 elements - GF(2^8). This field has some surprising and useful properties.

Remember how we said that integers modulo 256 weren't a field because they didn't have multiplicative inverses? I also just said that GF(2^8) also has 256 elements, but is a field - i.e., it does have inverses! How does that work?

Consider an element in GF(2^8). There are 2 ways to look at an element in GF(2^8). The first is to consider it as an 8-bit number. So, for example, let's take 100. We can express that as as an 8 bit binary number: 0b01100100.

We can write that more explicitly as a sum of powers of 2:

0 * 2^7 + 1 * 2^6 + 1 * 2^5 + 0 * 2^4 + 0 * 2^3 + 1 * 2^2 + 0 * 2 + 0 * 1 = 2^6 + 2^5 + 2^2

Now the other way we can look at elements in GF(2^8) is to replace the '2's with 'x's, and consider them as polynomials. Each of our bits then represents the coefficient of a term of a polynomial, that is:

0 x^7 + 1 x^6 + 1 x^5 + 0 x^4 + 0 x^3 + 1 x^2 + 0 x + 0 * 1

or more simply

x^6 + x^5 + x^2

Now, and this is important: each of the coefficients are elements of the integers modulo 2: x + x = 2x = 0 as 2 mod 2 = 0. There is no concept of 'carrying' in this addition.

Let's try: what's 100 + 79 in GF(2^8)?

100 = 0b01100100 => x^6 + x^5 + x^2 79 = 0b01001111 => x^6 + x^3 + x^2 + x + 1 100 + 79 => 0 + x^5 + x^3 + 0 + x + 1 = 0b00101011 = 43

So, 100 + 79 = 43 in GF(2^8)

You may notice we could have done that much more efficiently: we can add numbers in GF(2^8) by just XORing their binary representations together. Subtraction, amusingly, is the same as addition: 0 + x = x = 0 - x, as -1 is congruent to 1 modulo 2.

So at this point you might be wanting to explore a few additions yourself. Fortuantely there's a lovely tool that will allow you to do that:

sudo apt install gf-complete-tools gf_add $A $B 8

This will give you A + B in GF(2^8).

> gf_add 100 79 8 43

Excellent!

So, hold on to your hats, as this is where things get really weird. In modular arithmetic example, we considered the elements of our ring to be numbers, and we performed our addition and multiplication modulo 256. In GF(2^8), we consider our elements as polynomials and we perform our addition and multiplication modulo a polynomial. There is one conventional polynomial used in applications:

0x11d => 0b1 0001 1101 => x^8 + x^4 + x^3 + x^2 + 1

It is possible to use other polynomials if they satisfy particular requirements, but for our applications we don't need to worry as we will always use 0x11d. I am not going to attempt to explain anything about this polynomial - take it as an article of faith.

So when we multiply two numbers, we multiply their polynomial representations. Then, to find out what that is modulo 0x11d, we do polynomial long division by 0x11d, and take the remainder.

Some examples will help.

Let's multiply 100 by 3.

100 = 0b01100100 => x^6 + x^5 + x^2 3 = 0b00000011 => x + 1 (x^6 + x^5 + x^2)(x + 1) = x^7 + x^6 + x^3 + x^6 + x^5 + x^2 = x^7 + x^5 + x^3 + x^2

Notice that some of the terms have disappeared: x^6 + x^6 = 0.

The degree (the largest power of a term) is 7. 7 is less than the degree of 0x11d, which is 8, so we don't need to do anything: the remainder modulo 0x11d is simply x^7 + x^5 + x^3 + x^2.

In binary form, that is 0b10101100 = 172, so 100 * 3 = 172 in GF(2^8).

Fortunately gf-complete-tools also allows us to check multiplications:

> gf_mult 100 3 8 172

Excellent!

Now let's see what happens if we multiply by a larger number. Let's multiply 100 by 5.

100 = 0b01100100 => x^6 + x^5 + x^2 5 = 0b00000101 => x^2 + 1 (x^6 + x^5 + x^2)(x^2 + 1) = x^8 + x^7 + x^4 + x^6 + x^5 + x^2 = x^8 + x^7 + x^6 + x^5 + x^4 + x^2

Here we have an x^8 term, so we have a degree of 8. This means will get a different remainder when we divide by our polynomial. We do this with polynomial long division, which you will hopefully remember if you did some solid algebra in high school.

1 --------------------------------------------- x^8 + x^4 + x^3 + x^2 + 1 | x^8 + x^7 + x^6 + x^5 + x^4 + x^2 - x^8 + x^4 + x^3 + x^2 + 1 ------------------------------------------- = x^7 + x^6 + x^5 + x^3 + 1

So we have that our original polynomial (x^8 + x^4 + x^3 + x^2 + 1) is congruent to (x^7 + x^6 + x^5 + x^3 + 1) modulo the polynomial 0x11d. Looking at the binary representation of that new polynomial, we have 0b11101001 = 233.

Sure enough:

> gf_mult 100 5 8 233

Just to solidify the polynomial long division a bit, let's try a slightly larger example, 100 * 9:

100 = 0b01100100 => x^6 + x^5 + x^2 9 = 0b00001001 => x^3 + 1 (x^6 + x^5 + x^2)(x^3 + 1) = x^9 + x^8 + x^5 + x^6 + x^5 + x^2 = x^9 + x^8 + x^6 + x^2

Doing long division to reduce our result:

x ----------------------------------- x^8 + x^4 + x^3 + x^2 + 1 | x^9 + x^8 + x^6 + x^2 - x^9 + x^5 + x^4 + x^3 + x ------------------------------------------------- = x^8 + x^6 + x^5 + x^4 + x^3 + x^2 + x

We still have a polynomial of degree 8, so we can do another step:

x + 1 ----------------------------------- x^8 + x^4 + x^3 + x^2 + 1 | x^9 + x^8 + x^6 + x^2 - x^9 + x^5 + x^4 + x^3 + x ------------------------------------------------- = x^8 + x^6 + x^5 + x^4 + x^3 + x^2 + x - x^8 + x^4 + x^3 + x^2 + 1 ----------------------------------------------- = x^6 + x^5 + x + 1

We now have a polynomial of degree less than 8 that is congruent to our original polynomial modulo 0x11d, and the binary form is 0x01100011 = 99.

> gf_mult 100 9 8 99

This process can be done more efficiently, of course - but understanding what is going on will make you much more comfortable with what is going on!

I will not try to convince you that all multiplicative inverses exist in this magic shadow land of GF(2^8), but it's important for the rest of the algorithms to work that they do exist. Trust me on this.

Back to RAID 6

Equipped with this knowledge, you are ready to take on RAID6 in the kernel (PDF) sections 1 - 2.

Pause when you get to section 3 - this snippet is a bit magic and benefits from some explanation:

Multiplication by {02} for a single byte can be implemeted using the C code:

uint8_t c, cc; cc = (c << 1) ^ ((c & 0x80) ? 0x1d : 0);

How does this work? Well:

Say you have a binary number 0bNMMM MMMM. Mutiplication by 2 gives you 0bNMMMMMMM0, which is 9 bits. Now, there are two cases to consider.

If your leading bit (N) is 0, your product doesn't have an x^8 term, so we don't need to reduce it modulo the irreducible polynomial.

If your leading bit is 1 however, your product is x^8 + something, which does need to be reduced. Fortunately, because we took an 8 bit number and multiplied it by 2, the largest term is x^8, so we only need to reduce it once. So we xor our number with our polynomial to subtract it.

We implement this by letting the top bit overflow out and then xoring the lower 8 bits with the low 8 bits of the polynomial (0x1d)

So, back to the original statement:

(c << 1) ^ ((c & 0x80) ? 0x1d : 0) | | | | > multiply by 2 | | | | | > is the high bit set - will the product have an x^8 term? | | > if so, reduce by the polynomial | > otherwise, leave alone

Hopefully that makes sense.

Key points

It's critical you understand the section on Altivec (the vperm stuff), so let's cover it in a bit more detail.

Say you want to do A * V, where A is a constant and V is an 8-bit variable. We can express V as V_a + V_b, where V_a is the top 4 bits of V, and V_b is the bottom 4 bits. A * V = A * V_a + A * V_b

We can then make lookup tables for multiplication by A.

If we did this in the most obvious way, we would need a 256 entry lookup table. But by splitting things into the top and bottom halves, we can reduce that to two 16 entry tables. For example, say A = 02.

V_a A * V_a 00 00 01 02 02 04 ... ... 0f 1e V_b A * V_b 00 00 10 20 20 40 ... ... f0 fd

We then use vperm to look up entries in these tables and vxor to combine our results.

So - and this is a key point - for each A value we wish to multiply by, we need to generate a new lookup table.

So if we wanted A = 03:

V_a A * V_a 00 00 01 03 02 06 ... ... 0f 11 V_b A * V_b 00 00 10 30 20 60 ... ... f0 0d

One final thing is that Power8 adds a vpermxor instruction, so we can reduce the entire 4 instruction sequence in the paper:

vsrb v1, v0, v14 vperm v2, v12, v12, v0 vperm v1, v13, v13, v1 vxor v1, v2, v1

to 1 vpermxor:

vpermxor v1, v12, v13, v0

Isn't POWER grand?

OK, but how does this relate to erasure codes?

I'm glad you asked.

Galois Field arithmetic, and its application in RAID 6 is the basis for erasure coding. (It's also the basis for CRCs - two for the price of one!)

But, that's all to come in part 2, which will definitely be published before 7 April!

Many thanks to Sarah Axtens who reviewed the mathematical content of this post and suggested significant improvements. All errors and gross oversimplifications remain my own. Thanks also to the OzLabs crew for their feedback and comments.

David Rowe: Codec 2 700C and Short LDPC Codes

Sat, 2017-03-18 14:03

In the last blog post I evaluated FreeDV 700C over the air. This week I’ve been simulating the use of short LDPC FEC codes with Codec 2 700C over AWGN and HF channels.

In my HF Digital Voice work to date I have shied away from FEC:

  1. We didn’t have the bandwidth for the extra bits required for FEC.
  2. Modern, high performance codes tend to have large block sizes (1000’s of bits) which leads to large latency (several seconds) when applied to low bit rate speech.
  3. The error rates we are interested in (e.g. 10% raw, 1% after FEC decoder) are unusual – many codes don’t work well.

However with Codec 2 pushed down to 700 bit/s we now have enough bandwidth for a rate 1/2 code inside a standard 2kHz SSB channel. Over coffee a few weeks ago, Bill VK5DSP offered to develop some short LDPC codes for me specifically for this application. He sent me an Octave simulation of rate 1/2 and 2/3 codes of length 112 and 56 bits. Codec 2 700C has 28 bit frames so this corresponds to 4 or 2 Codec 2 700C frames, which would introduce a latencies of between 80 to 160ms – quite acceptable for Push To Talk (PTT) radio.

I re-factored Bill’s simulation code to produce ldpc_short.m. This measures BER and PER for Bill’s short LDPC codes, and also plots curves for theoretical, HF multipath channels, a Golay (24,12) code, and the current diversity scheme used in FreeDV 700C.

To check my results I compared the Golay BER and ideal HF multipath (Rayleigh Fading) channel curves to other peoples work. Always a good idea to spot check a few values and make sure they are sensible. I took a simple approach to get results in a reasonable amount of coding time (about 1 day of work in this case). This simulation runs at the symbol rate, and assumes ideal synchronisation. My other modem work (i.e experience) lets me move back and forth between this sort of simulation and real world modems, for example accounting for synchronisation losses.

Error Distribution and Packet Error Rate

I had an idea that Packet Error Rate (PER) might be important. Without FEC, bit errors are scattered randomly about. At our target 1% BER, many frames will have 1 or 2 bit errors. As discussed in the last post Codec 2 700C is sensitive to bit errors as “every bit counts”. For example one bit error in the Vector Quantiser (VQ) index (a big look up table) can throw the speech spectrum right off.

However a LDPC decoder will tend to correct all errors in a codeword, or “die trying” (i.e. fail badly). So an average output BER of say 1% will consist of a bunch of perfect frames, plus a completely trashed one every now and again. Digital voice works better with this style of error pattern than a few random errors in each codec packet. So for a given BER, a system that delivers a lower PER is better for our application. I’ve guesstimated a 10% PER target for intelligible low bit rate speech. Lets see how that works out…..

Results

Here are the BER and PER curves for an AWGN channel:

Here are the same curves for HF (multipath fading) channel:

I’ve included a Golay (24,12) block code (hard decision) and uncoded PSK for comparison to the AWGN curves, and the diversity system on the HF curves. The HF channel is modelled as two paths with 1Hz Doppler spread and a 1ms delay.

The best LDPC code reaches the 1% BER/10% PER point at 2dB Eb/No (AWGN) and 6dB (HF multipath). Comparing BER, the coding gain is 2.5 and 3dB (AWGN and HF). Comparing PER, the coding gain is 3 and 5dB (AWGN and HF).

Here is a plot of the error pattern over time using the LDPC code on a HF channel at Eb/No of 6dB:

Note the errors are confined to short bursts – isolated packets where the decoder fails. Even though the average BER is 1%, most of the speech is error free. This is a very nice error distribution for digital speech.

Speech Samples

Here are some speech samples, comparing the current diversity scheme used for FreeDV 700C to LDPC, for AWGN and LDPC channels. These were simulated by extracting the error pattern from the simulation then inserting these errors in a Codec 2 700C bit stream (see command lines section below).

AWGN Eb/No 2dB Diversity LDPC HF Eb/No 6dB Diversity LDPC

Next Steps

These results are very encouraging and suggest a gain of 2 to 5dB over FreeDV 700C, and better error distribution (lower PER). Next step is to develop FreeDV 700D – a real world implementation using the 112 data-bit rate 1/2 LDPC code. This will require 4 frames of buffering, and some sort of synchronisation to determine the 112 bit frame boundaries. Fortunately much of the C code for these LDPC codes already exists, as it was developed for the Wenet High Altitude Balloon work.

If most frames at the decoder input are now error free, we can consider more efficient (but less robust) techniques for Codec 2, such as prediction (delta coding). This will decrease the codec bit rate for a given speech quality. We could then choose to reduce our bit rate (making the system more robust for a given channel SNR), or raise speech quality while maintaining the same bit rate.

Command Lines

Generating the decoded speech, first run the Octave ldpc_short simulation to generate “error pattern file”, then subject the Codec 2 700C bit stream to these error patterns.

octave:67> ldpc_short $ ./c2enc 700C ../../raw/ve9qrp_10s.raw - | ./insert_errors - - ../../octave/awgn_2dB_ldpc.err 28 | ./c2dec 700C - - | aplay -f S16_LE -

The simulation generate .eps files as direct generation of PNG leads to font size issues. Converting EPS to PNG without transparent background:

mogrify -resize 700x600 -density 300 -flatten -format png *.eps

However I still feel the images are a bit fuzzy, especially the text. Any ideas? Here’s the eps file if some one would like to try to get a nicer PNG conversion for me! The EPS file looks great at any scaling when I render it using the Ubuntu document viewer.

Update: A friend of mine (Erich) has suggested using GIMP for the conversion. This does seem to work well and has options for text and line anti-aliasing. It would be nice to be able to generate nice PNGs directly from Octave – my best approach so far is to capture screen shots.

Links

LowSNR site Bill VK5DSP writes about his experiments in low SNR communications.

Wenet High Altitude Balloon SSDV System developed with Mark VK5QI and BIll VK5DSP that uses LDPC codes.

LPDC using Octave and the CML library

FreeDV 700C

Codec 2 700C

OpenSTEM: St Patrick’s Day 2017 – and a free resource on Irish in Australia

Sat, 2017-03-18 10:05

Happy St Patrick’s day!

And “we have a resource on that” – that is, on the Irish in Australia and the major contributions they made since the very beginning of the colonies. You can get that lovely 5 page resource PDF for free if you check out using coupon code TRYARESOURCE. It’s an option we’ve recently put in place so anyone can grab one resource of their choice to see if they like our materials and assess their quality.

Back to St Patrick, we were briefly in Ireland last year and near Dublin we drove past a ruin at the top of a hill that piqued our interest, so we stopped and had a look. It turned out to be Slane Abbey, the site where it is believed in 433 AD, the first Christian missionary to Ireland, later known as St Patrick, lit a large (Easter) celebration fire (on the Hill of Slane). With this action he (unwittingly?) contravened orders by King Laoghaire at nearby Tara. The landscape photo past the Celtic cross shows the view towards Tara. Ireland is a beautiful country, with a rich history.

Photos by Arjen Lentz & Dr Claire Reeler

OpenSTEM: Am I a Neanderthal?

Wed, 2017-03-15 10:05
Early reconstruction of Neanderthal

The whole question of how Neanderthals are related to us (modern humans) has been controversial ever since the first Neanderthal bones were found in Germany in the 19th century. Belonging to an elderly, arthritic individual (a good example of how well Neanderthals cared for each other in social groups), the bones were reconstructed to show a stooping individual, with a more ape-like gait, leading to Neanderthals being described as the “Missing Link” between apes and humans, and given the epithet “ape-man”.

Who were the Neanderthals? Modern reconstruction – Smithsonian Museum of Natural History

Neanderthals lived in the lands surrounding the Mediterranean Sea, and as far east as the Altai Mountains in Central Asia, between about 250,000 and about 30,000 years ago. They were a form of ancient human with certain physical characteristics – many of which probably helped them cope with the cold of Ice Ages. Neanderthals evolved out of an earlier ancestorHomo erectus, possibly through another species – Homo heidelbergensis. They had a larger brain than modern humans, but it was shaped slightly differently, with less development in the prefrontal cortex, which allows critical thinking and problem-solving, and larger development at the back of the skull, and in areas associated with memory in our brains. It is possible that Neanderthals had excellent memory, but poor analytical skills. They were probably not good at innovation – a skill which became vital as the Ice Age ended and the global climate warmed, sea levels rose and plant and animal habitats changed.

Neanderthals were stockier than modern humans, with shorter arms and legs, and probably stronger and all-round tougher. They had a larger rib cage, and probably bigger lungs, a bigger nose, larger eyes and little to no chin. Most of these adaptations would have helped them in Ice Age Europe and Asia – a more compact body stayed warmer more easily and was tough enough to cope with a harsh environment. Large lungs helped oxygenate the blood and there is evidence that they had more blood supply to the face – so probably had warm, ruddy cheeks. The large nose warmed up the air they breathed, before it reached their lungs, reducing the likelihood of contracting pneumonia. Neanderthals are known to have had the same range of hair colours as modern humans and fair skin, red hair and freckles may have been more common.

They made stone tools, especially those of the type called Mousterian, constructed simple dwellings and boats, made and used fire, including for cooking their food, and looked after each other in social groups. Evidence of skeletons with extensive injuries occurring well before death, shows that these individuals must have been cared for, not only whilst recovering from their injuries, but also afterwards, when they would probably not have been able to obtain food themselves. Whether or not Neanderthals intentionally buried their dead is an area of hot controversy. It was once thought that they buried their dead with flowers in the grave, but the pollen was found to have been introduced accidentally. However, claims of intentional burial are still debated from other sites.

What Happened to the Neanderthals? Abrigo do Lagar Velho

Anatomically modern humans emerged from Africa about 100,000 years ago. Recent studies of human genetics suggests that modern humans had many episodes of mixing with various lineages of human ancestors around the planet. Modern humans moved into Asia and Europe during the Ice Age, expanding further as the Ice Age ended. Modern humans overlapped with Neanderthals for about 60,000 years, before the Neanderthals disappeared. It is thought that a combination of factors led to the decline of Neanderthals. Firstly, the arrival of modern humans, followed by the end of the Ice Age, brought about a series of challenges which Neanderthals might have been unable to adapt to, as quickly as necessary. Modern humans have more problem solving and innovation capability, which might have meant that they were able to out-compete Neanderthals in a changing environment. The longest held theory is that out ancestors wiped out the Neanderthals in the first genocide in (pre)history. A find of Neanderthals in a group, across a range of ages, some from the same family group, who all died at the same time, is one of the sites, which might support this theory, although we don’t actually know who (or what) killed the group. Cut marks on their bones show that they were killed by something using stone tools. Finally, there is more and more evidence of what are called “transitional specimens”. These are individuals who have physical characteristics of both groups, and must represent inter-breeding. An example is the 4 year old child from the site of Abrigo do Lagar Velho in Portugal, which seems to have a combination of modern and Neanderthal features. The discovery of Neanderthals genes in many modern people living today is also proof that we must have interbred with Neanderthals in the past. It is thought that the genes were mixed several times, in several parts of the world.

Am I a Neanderthal?

So how do we know if we have Neanderthals genes? Neanderthal genes have some physical characteristics, but also other attributes that we can’t see. In terms of physical characteristics, Neanderthal aspects to the skull include brow ridges (ridges of bone above the eyes, under the eyebrows); a bump on the back of the head – called an occipital chignon, or bun, because it looks like a ‘bun’ hairstyle, built into the bone; a long skull (like Captain Jean-Lu Picard from Star Trek – actor Patrick Stewart); a small, or non-existent chin; a large nose; a large jaw with lots of space for wisdom teeth; wide fingers and thumbs; thick, straight hair; large eyes; red hair, fair skin and freckles! The last may seem a little surprising, but it appears that the genes for these characteristics came from Neanderthals – who had a wide range of hair colours, fair skin and, occasionally, freckles. Increased blood flow to the face also would have given Neanderthals lovely rosy cheeks!

Less obvious characteristics include resistance to certain diseases – parts of our immune systems, especially with reference to European and Asian diseases; less positively, an increased risk of other diseases, such as type 2 diabetes. Certain genes linked to depression are present, but ‘switched off’ in Neanderthals. The way that these genes link to depression, and their role in the lifestyles of early people (where they may have had benefits that are no longer relevant) are future topics for research and may help us understand more about ourselves.

Neanderthals genes are present in modern populations from Europe, Asia, Northern Africa, Australia and Oceania. So, depending on which parts of the world our ancestry is from, we may have up to 4% of our genetics from long-dead Neanderthal ancestors!

Craige McWhirter: Japanese House in Python for Minecraft

Wed, 2017-03-15 09:04

I have kids that I'm teaching to hack. They started of on Scratch (which is excellent) and are ready to move up to Python. They also happen to be mad Minecraft fans, so now they're making their way through Adventures in Minecraft.

As I used Scratch when they were, I'm also hacking in Python & Minecraft as they are. It helps if I hit the bumps and hurdles before they do, as well as have a sound handle on the problems they're solving.

Now I've branched out from the tutorial and I'm just having fun with it and leaving behind code the kids can use, hack whatever. This code is in my minecraft-tools repo (for want of a better name). It's just a collection of random tools I've written for Minecraft aren't quite up to being their own thing. I expect this will mostly be a collection of python programs to construct things inside Minecraft via CanaryMod and CanaryRaspberryJuicePlugin.

The first bit of code to be shaken out of the tree is japanese_house.py which produces a Minecraft interpretation of a classic Japanese house. Presently it only produces the single configuration that is little more than an empty shell.

I intend to add an interior fit out plus a whole bunch of optional configurations you can set at run time but for now it is what it is, as I'm going to move onto writing geodesic domes and transport | teleport rings (as per the Expanse, which will lead to eventually coding a TARDIS, that will you know, be actually bigger on the inside ;-)

David Rowe: Testing FreeDV 700C

Tue, 2017-03-14 18:06

Since releasing FreeDV 700C I’ve been “instrumenting” the FreeDV GUI program – adding some code to perform various tests of the 700C waveform, especially over the air.

With the kind help of Gerhard OE3GBB, Mark VK5QI, and Peter VK5APR, I have collected some samples and performed some tests. The goals of this work were:

  1. Compare 700C Over the Air (OTA) to simulation on an AWGN channel.
  2. Compare 700C OTA to SSB on an AWGN channel.

Instrumentation

Here is a screen shot of the latest FreeDV GUI Options screen:

I’ve added some features to the top three rows:

Test Frames Send a payload of known test bits rather than vocoder bits Channel Noise Simulate a channel using AWGN noise SNR SNR of AWGN noise Attn Carrier Attenuate just one carrier Carrier The 700C carrier (1-14) to attenuate Simulated Interference Tone Enable an interfering sine wave of specified frequency and amplitude Clipping Enable clipping of 700C tx waveform, to increase RMS power Diversity Combine for plots Scatter and Test Frame plots use combined (7 carrier) information.

To explore these options it is useful to run in full duplex mode (Tools-PTT Half Duplex unchecked) and using a loopback sound device:

$ sudo modprobe snd-aloop

More information on loopback in the FreeDV GUI README.

Clipping the 700C tx waveform reduces the Peak to Average Power ratio (PAPR) which may be result in a higher average power over the channel. However clipping distorts the waveform and add some “shoulders (i.e. noise) to the spectrum adjacent to the 700C waveform:

Several users have noticed this distortion. At this stage I’m unsure if clipping is useful or not.

The Diversity Combine option is useful to explore each of the 14 carriers separately before they are combined into 7 carriers.

Many of these options were designed to explore tx filtering. I have long wondered if any of the FreeDV carriers were receiving less power than others, for example due to ripple or a low pass response from the crystal filter. A low power carrier would have a high bit error rate, adversely affecting overall performance. Plotting the scatter diagram or bit error rate on a carrier by carrier basis can measure the effect of tx filtering – if it exists.

Some of the features above – like attenuating a single carrier – were designed to “test the test”. Much of the work I do on FreeDV (and indeed other projects) involves carefully developing software and writing “code to test the code”. For example to build the experiments described in this blog post I worked several hours day for several weeks. Not glamorous, but where the real labour lies in R&D. Careful, meticulous testing and experimentation. One percent inspiration … then code, test, test.

Comparing Analog SSB to Digital Voice

One of my goals is to develop a HF DV system that is competitive with analog SSB. So we need a way to compare analog and DV at the same SNR. So I came with the idea of a wave files of analog SSB and DV which have the same average (RMS) power. If these are fed into a SSB transmitter, then they will be received at the same SNR. I added 10 seconds of a 1000Hz sine wave at the start for good measure – this could be used to measure the actual SNR.

I developed two files:

  1. sine_analog_700c
  2. sine_analog_testframes700c

The first has the same voice signal in analog and 700C, the second uses test frames instead of encoded voice.

Interfering Carriers

One feature described above simulates an interfering carrier (like a birdie), something I have seen on the air. Here is a plot of a carrier in the middle of one of the 700C carriers, but about 10dB higher:

The upper RH plot is a rolling plot of bit errors for each carrier. You can see one carrier is really messed up – lots of bit errors. The average bit error rate is about 1%, which is where FreeDV 700C starts to become difficult to understand. These bit errors would not be randomly distributed, but would affect one part of the codec all the time. For example the pitch might be consistently wrong, or part of the speech spectrum. I found that as long as the interfering carrier is below the FreeDV carrier, the effect on bit error rate is negligible.

Take away: The tx station must tune away from any interfering carriers that poke above the FreeDV signal carriers. Placing the interfering tones between FreeDV carriers is another possibility, e.g. a 50Hz shift of the tx signal.

Results – Transmit Filtering

Simulation results suggest 700C should produce reasonable results near 0db SNR. So that’s the SNR I’m shooting for in Over The Air (OTA) testing.

Mark VK5QI sent me several minutes of test frames so I could determine if there were any carriers with dramatically different bit error rates, which would indicate the presence of some tx filtering. Here is the histogram of BERs for each carrier for Mark’s signal, which was at about 3dB SNR:

There is one bar for each I and Q QPSK bit of the 14 carriers – 28 bars total (note Diversity combination was off). After running for a few minutes, we can see a range of 5E-2 and 8E-2 (5 to 8%). In terms of AWGN modem performance, this is only about 1dB difference in SNR or Eb/No, as per the BER versus Eb/No graphs in this post on the COHPSK modem used for 700C. One carrier being pinned at say 20% BER, or a slope of increasing BER with carrier frequency – would have meant tx filtering trouble.

Peter VK5APR, sent me a high SNR signal (he lives just 4 km away). Initially I could see a X shaped scatter diagram, a possible sign of tx filtering. However this ended up being some amplitude pumping due to Fast AGC on my radio. After I disabled fast AGC, I could see a scatter diagram with 4 clear dots, and no X-shape. Check.

I performed an additional test using my IC7200 as a transmitter, and a HackRF as a receiver. The HackRF has no crystal filter and a very flat response, so any distortion would be due to the IC7200 transmit filtering. Once again – 4 clean dots on the scatter diagram and no X-shape.

So I am happy to conclude that transmit filtering does not seem to be a problem, at least of the radios tested. All performance issues are therefore likely to be caused by me and my algorithms!

Results – Low SNR testing

Peter, VK5APR, configured his station to send the analog/700C equi-power test wave files described above at very low power, such that the received SNR at my station was about 0dB. As we are so close it was reasonable to assume the channel was AWGN, indeed we could see no sign of NVIS fading and the band (40M) was devoid of DX at the 12 noon test time.

Here is the rx signal I received, and the same file run through the 700C decoder. Neither the SSB or the decoded 700C audio are pretty. However it’s fair to say we could just get a message through on both modes and that 700C is holding it’s own next to SSB. The results are close to my simulations which was the purpose of this test.

You can decode the off air signal yourself if you download the first file and replay it through the FreeDV GUI program using “Tools – Start/Stop Play File from Radio”.

Discussion

While setting up these tests, Peter and I conversed comfortably for some time over FreeDV 700C at a high SNR. This proved to me that for our audience (experienced users of HF radio) – FreeDV 700C can be used for conversational contacts. Given the 700C codec is really just a first pass – that’s a fine result.

However it’s a near thing – the 700C codec adds a lot of distortion just compressing the speech. It’s pretty bad even if the SNR is high. The comments on the Codec 2 700C blog post indicate many lay-people can’t understand speech compressed by 700C. Add any bit errors (due to low SNR or fading) and it quickly becomes hard to understand – even for experienced users. This makes 700C very sensitive to bit errors as the SNR drops. But hey – every one of those 28 bits/frame counts at 700 bit/s so it’s not surprising.

In contrast, SSB scales a bit better with SNR. However even at high SNRs, that annoying hiss is always there – which is very fatiguing. Peter and I really noticed that hiss after a few minutes back on SSB. Yuck.

SSB gets a lot of it’s low SNR “punch” from making effective use of peak power. Here is a plot of the received SSB:

It’s all noise except for the speech peaks, where the “peak SNR” is much higher than 0dB. Our brains are adept at picking out words from those peaks, integrating the received phonetic symbols (mainly vowel energy) in our squishy biological receive filters. It’s a pity we didn’t evolve to detect coherent PSK. A curse on your evolution!

In contrast – 700C allocates just as much power to the silence between words as the most important parts of speech. This suggests we could do a better job at tailoring the codec and modem to peak power, e.g. allocating more power to parts of the speech that really matter. I had a pass at Time Variable Quantisation a few years ago. A variable rate codec might be called for, tightly integrated to the modem to pack more bits/power into perceptually important parts of speech.

The results above assumed equal average power for SSB and FreeDV 700C. It’s unclear if this happens in the real world. For example we may need to “back off” FreeDV drive further than SSB; SSB may use a compressor; and the PAs we are using are generally designed for PEP rather than average power operation.

Next Steps

I’m fairly happy with the baseline COHPSK modem, it seems to be hanging on OK as long as there aren’t any co-channel birdies. The 700C codec works better than expected, has plenty of room for improvement – but it’s sensitive to bit errors. So I’m inclined to try some FEC next. Aim for error free 700C at 0dB, which I think will be superior to SSB. I’ll swap out the diversity for FEC. This will increase the raw BER, but allow me to run a serious rate 0.5 code. I’ll start just with an AWGN channel, then tackle fading channels.

Links

FreeDV 700C
Codec 2 700C

Matthew Oliver: Setting up a basic keystone for Swift + Keystone dev work

Tue, 2017-03-14 16:07

As a Swift developer, most of the development works in a Swift All In One (SAIO) environment. This environment simulates a mulinode swift cluster on one box. All the SAIO documentation points to using tempauth for authentication. Why?

Because most the time authentication isn’t the things we are working on. Swift has many moving parts, and so tempauth, which only exists for testing swift and is configured in the proxy.conf file works great.

However, there are times you need to debug or test keystone + swift integration. In this case, we tend to build up a devstack for keystone component. But if all we need is keystone, then can we just throw one up on a SAIO?… yes. So this is how I do it.

Firstly, I’m going to be assuming you have SAIO already setup. If not go do that first. not that it really matters, as we only configure the SAIO keystone component at the end. But I will be making keystone listen on localhost, so if you are doing this on anther machine, you’ll have to change that.

Further, this will set up a keystone server in the form you’d expect from a real deploy (setting up the admin and public interfaces).

 

Step 1 – Get the source, install and start keystone

Clone the sourcecode:
cd $HOME
git clone https://github.com/openstack/keystone.git

Setup a virtualenv (optional):
mkdir -p ~/venv/keystone
virtualenv ~/venv/keystone
source ~/venv/keystone/bin/activate

Install keystone:
cd $HOME/keystone
pip install -r requirements.txt
pip install -e .
cp etc/keystone.conf.sample etc/keystone.conf

Note: We are running the services from the source so config exists in source etc.

 

The fernet keys seems to assume a full /etc path, so we’ll create it. Maybe I should update this to put all config there but for now meh:
sudo mkdir -p /etc/keystone/fernet-keys/
sudo chown $USER -R /etc/keystone/

Setup the database and fernet:
keystone-manage db_sync
keystone-manage fernet_setup

Finally we can start keystone. Keystone is a wsgi application and so needs a server to pass it requests. The current keystone developer documentation seems to recommend uwsgi, so lets do that.

 

First we need uwsgi and the python plugin, one a debian/ubuntu system you:
sudo apt-get install uwsgi uwsgi-plugin-python

Then we can start keystone, by starting the admin and public wsgi servers:
uwsgi --http 127.0.0.1:35357 --wsgi-file $(which keystone-wsgi-admin) &
uwsgi --http 127.0.0.1:5000 --wsgi-file $(which keystone-wsgi-public) &

Note: Here I am just backgrounding them, you could run then in tmux or screen, or setup uwsgi to run them all the time. But that’s out of scope for this.

 

Now a netstat should show that keystone is listening on port 35357 and 5000:
$ netstat -ntlp | egrep '35357|5000'
tcp 0 0 127.0.0.1:5000 0.0.0.0:* LISTEN 26916/uwsgi
tcp 0 0 127.0.0.1:35357 0.0.0.0:* LISTEN 26841/uwsgi

Step 2 – Setting up keystone for swift

Now that we have keystone started, its time to configure it. Firstly you need the openstack client to configure it so:
pip install python-openstackclient

Next we’ll use all keystone defaults, so we only need to pick an admin password. For the sake of this how-to I’ll pick the developer documentation example of `s3cr3t`. Be sure to change this. So we can do a basic keystone bootstrap with:
keystone-manage bootstrap --bootstrap-password s3cr3t

Now we just need to set up some openstack env variables so we can use the openstack client to finish the setup. To make it easy to access I’ll dump them into a file you can source. But feel free to dump these in your bashrc or whatever:
cat > ~/keystone.env <<EOF
export OS_USERNAME=admin
export OS_PASSWORD=s3cr3t
export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_ID=default
export OS_PROJECT_DOMAIN_ID=default
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_URL=http://localhost:5000/v3
EOF

source ~/keystone.env

 

Great, now  we can finish configuring keystone. Let’s first setup a service project (tennent) for our Swift cluster:
openstack project create service

Create a user for the cluster to auth as when checking user tokens and add the user to the service project, again we need to pick a password for this user so `Sekr3tPass` will do.. don’t forget to change it:
openstack user create swift --password Sekr3tPass --project service
openstack role add admin --project service --user swift

Now we will create the object-store (swift) service and add the endpoints for the service catelog:
openstack service create object-store --name swift --description "Swift Service"
openstack endpoint create swift public "http://localhost:8080/v1/AUTH_\$(tenant_id)s"
openstack endpoint create swift internal "http://localhost:8080/v1/AUTH_\$(tenant_id)s"

Note: We need to define the reseller_prefix we want to use in Swift. If you change it in Swift, make sure you update it here.

 

Now we can add roles that will match to roles in Swift, namely an operator (someone who will get a Swift account) and reseller_admins:
openstack role create SwiftOperator
openstack role create ResellerAdmin

Step 3 – Setup some keystone users to auth as.

TODO: create all the tempauth users here

 

Here, it would make sense to create the tempauth users devs are used to using, but I’ll just go create a user so you know how to do it. First create a project (tennent) for this example demo:
openstack project create --domain default --description "Demo Project" demo

Create a user:
openstack user create --domain default --password-prompt matt

We’ll also go create a basic user role:
openstack role create user

Now connect the 3 pieces together by adding user matt to the demo project with the user role:
openstack role add --project demo --user matt user

If you wanted user matt to be a swift operator (have an account) you’d:
openstack role add --project demo --user matt SwiftOperator

or even a reseller_admin:
openstack role add --project demo --user matt ResellerAdmin

If your in a virtual env, you can leave it now, because next we’re going to go back to your already setup swift to do the Swift -> Keystone part:
deactivate

Step 4 – Configure Swift

To get swift to talk to keystone we need to add 2 middlewares to the proxy pipeline. And in the case of a SAIO, remove the tempauth middleware. But before we do that we need to install the keystonemiddleware to get one of the 2 middlware’s, keystone’s authtoken:
sudo pip install keystonemiddleware

Now you want to replace your tempauth middleware in the proxy path pipeline with authtoken keystoneauth so it looks something like:
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache bulk tempurl ratelimit crossdomain container_sync authtoken keystoneauth staticweb copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server

Then in the same ‘proxy-server.conf’ file you need to add the paste filter sections for both of these new middlewares:
[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
auth_host = localhost
auth_port = 35357
auth_protocol = http
auth_uri = http://localhost:5000/
admin_tenant_name = service
admin_user = swift
admin_password = Sekr3tPass
delay_auth_decision = True
# cache = swift.cache
# include_service_catalog = False
[filter:keystoneauth]
use = egg:swift#keystoneauth
# reseller_prefix = AUTH
operator_roles = admin, SwiftOperator
reseller_admin_role = ResellerAdmin

Note: You need to make sure if you change the reseller_prefix here, you change it in keystone. And notice this is where you map operator_roles and reseller_admin_role in swift to that in keystone. Here anyone in with the keystone role admin or SwiftOperator are swift operators and those with the ResellerAdmin role are reseller_admins.

 

And that’s it. Now you should be able to restart your swift proxy and it’ll go off and talk to keystone.

 

You can use your Python swiftclient now to go talk, and whats better swiftclient understands the OS_* variables, so you can just source your keystone.env and talk to your cluster (to be admin) or export some new envs for the user you’ve created. If you want to use curl you can. But _much_ easier to use swiftclient.

 

Tip: You can use: swift auth to get the auth_token if you want to then use curl.

 

If you want to authenticate via curl then for v3, use: https://docs.openstack.org/developer/keystone/devref/api_curl_examples.html

 

Or for v2, I use:
url="http://localhost:5000/v2.0/tokens"
auth='{"auth": {"tenantName": "demo", "passwordCredentials": {"username": "matt", "password": ""}}}'

 

curl -s -d "$auth" -H 'Content-type: application/json' $url |python -m json.tool

 

or

curl -s -d "$auth" -H 'Content-type: application/json' $url |python -c "import sys, json; print json.load(sys.stdin)['access']['token']['id']"

To just print out the token. Although a simple swift auth would do all this for you.

James Morris: LSM mailing list archive: this time for sure!

Tue, 2017-03-14 10:01

Following various unresolved issues with existing mail archives for the Linux Security Modules mailing list, I’ve set up a new archive here.

It’s a mailman mirror of the vger list.

Ian Wienand: Thunderbird 54 external editor

Tue, 2017-03-14 05:30

For many years I've used Thunderbird with Alexandre Feblot's external editor plugin to allow me to edit mail with emacs. Unfortunately it seems long unmaintained and stopped working on a recent upgrade to Thunderbird 54 when some deprecated interfaces were removed. Brian M. Carlson seemed to have another version which also seemed to fail with latest Thunderbird.

I have used my meagre Mozilla plugin skills to make an update at https://github.com/ianw/extedit/releases. Here you can download an xpi that passes the rigorous test-suite of ... works for me.

Sridhar Dhanapalan: How to Create a Venn Diagram with Independent Intersections in PowerPoint

Mon, 2017-03-13 22:02

A Venn diagram can be a great way to explain a business concept. This is generally not difficult to create in modern presentation software. I often use Google Slides for its collaboration abilities.

Where it becomes difficult is when you want to add a unique colour/pattern to an intersection, where the circles overlap. Generally you will either get one circle overlapping another, or if you set some transparency then the intersection will become a blend of the colours of the circles.

I could not work out how to do this in Google Slides, so on this occasion I cheated and did it in Microsoft PowerPoint instead. I then imported the resulting slide into Slides.

This worked for me in PowerPoint for Mac 2016. The process is probably the same on Windows.

Firstly, create a SmartArt Venn Diagram

Insert > SmartArt > Relationship > Basic Venn

Separate the Venn circles

SmartArt Design > Convert > Convert to Shapes

Ungroup shapes

Shape Format > Group Objects > Ungroup

Split out the intersections

Shape Format > Merge Shapes > Fragment

From there, you can select the intersection as an independent shape. You can treat each piece separately. Try giving them different colours or even moving them apart.

This can be a simple but impactful way to get your point across.

OpenSTEM: This Week in HASS – term 1, week 7

Mon, 2017-03-13 10:05

This week our youngest students are looking in depth at different types of celebrations; slightly older students are examining how people got around in the ‘Olden Days’; and our older primary students have some extra time to finish their activities from last week.

Foundation to Year 3 First car made in Qld, 1902

In the stand-alone Foundation (Prep) unit (F.1), students are discussing celebrations – which ones do we recognise in Australia, how these compare with celebrations overseas, and what were these celebrations like in days gone by. Our integrated Foundation (Prep) unit (F.5) and students in Years 1 (1.1), 2 (2.1) and 3 (3.1), are examining Transport in the Past – how did their grandparents get around? How did people get around 100 years ago? How did kids get to school? How did people do the shopping? Students even get to dream about how we might get around in the future…

Years 3 to 6 Making mud bricks

At OpenSTEM we recognise that good activities, which engage students and allow for real learning, take time. Nobody likes to get really excited about something and then be rushed through it and quickly moved on to something else. This part of the unit has lots of hands-on activities for Year 3 (3.5) students in an integrated class with Year 4, as well as Year 4 (4.1), 5 (5.1) and 6 (6.1) students. In recognition of that, two weeks are allowed for the students to really get into making Ice Ages and mud bricks, and working out how to survive the challenges of living in a Neolithic village – including how to trade, count and write. Having enough time allows for consolidation of learning, as well as allowing teachers to potentially split the class into different groups engaged in different activities, and then rotate the groups through the activities over a 2 week period.

Binh Nguyen: Prophets/Genesis/Terraforming Mars, Seek Menu, and More

Sun, 2017-03-12 19:02
Obvious continuation from my previous other posts with regards to prophets/pre-cogs: http://dtbnguyen.blogspot.com/2017/03/prophetspre-cogsstargate-program-8.html http://dtbnguyen.blogspot.com/2017/02/life-in-india-prophetspre-cogsstargate_82.html http://dtbnguyen.blogspot.com/2017/02/life-in-iran-examining-prophetspre-cogs.html http://dtbnguyen.blogspot.com/2017/02/

Craige McWhirter: Solarized xmobar, trayer and dmenu for xmonad

Fri, 2017-03-10 17:36

Like many other people, Ethan Schoonover's precision colours for machines and people, known as Solarized is my preferred colour pallet, especially as I spend most of my time in terminals.

I've "Solarized" most things in my work environment except my window manager, xmonad. Well to be more specific, the xmobar, trayer and dmenu applications that I use within xmonad.

The commits:

Modifications were made to xmonad.hs, .xmobarrc and trayer in .xsession that switches out the existing and default colours for those I've selected from the Solarized palette. I now have a fully Solarized window manager that fits in much better with the rest of my workspace.

Relevant code snippets:

xmonad.hs

, logHook = dynamicLogWithPP $ xmobarPP { ppOutput = hPutStrLn xmproc , ppCurrent = xmobarColor "#859900" "" . wrap "[" "]" , ppVisible = xmobarColor "#2aa198" "" . wrap "(" ")" , ppLayout = xmobarColor "#2aa198" "" , ppTitle = xmobarColor "#859900" "" . shorten 50 } , modMask = mod4Mask -- Rebind Mod to the Windows key --, borderWidth = 1 } `additionalKeys` -- Custom dmenu launcher [ ((mod4Mask, xK_p ), spawn "exe=`dmenu_path | dmenu -fn \"Open Sans-10\" -p \"λ:\" -nb \"#073642\" -nf \"#93a1a1\" -sb \"#002b36\" -sf \"#859900\"` && eval \"exec $exe\"")

.xmobarrc

-- Appearance font = "xft:OpenSans:size=10:antialias=true" , bgColor = "#073642" , fgColor = "#93a1a1" ... --Plugins , commands = -- CPU Activity Monitor ... , "--low" , "#2aa198" , "--normal" , "#859900" , "--high" , "#dc322f" ... -- cpu core temperature monitor ... , "--low" , "#2aa198" , "--normal" , "#859900" , "--high" , "#dc322f" ... -- Memory Usage Monitor ... , "--low" , "#2aa198" , "--normal" , "#859900" , "--high" , "#dc322f" ... -- Battery Monitor ... , "--low" , "#dc322f" , "--normal" , "#859900" , "--high" , "#2aa198" , "--" -- battery specific options -- discharging status , "-o" , "<left>% (<timeleft>)" -- AC "on" status , "-O" , "<fc=#2aa198>Charging</fc>" -- charged status , "-i" , "<fc=#859900>Charged</fc>" ... -- Weather Monitor ... , "--low" , "#2aa198" , "--normal" , "#859900" , "--high" , "#dc322f" ... -- Network Activity Monitor ... , "--low" , "#2aa198" , "--normal" , "#859900" , "--high" , "#dc322f" ... -- Time and Date Display , Run Date "<fc=#268bd2>%a %b %_d %H:%M</fc>" "date" 10 ...

.xsession

exec /usr/bin/trayer --edge top --align right --SetDockType true --SetPartialStrut true --expand true --width 10 --transparent true --alpha 0 --tint 0x073642 --height 22 &

That at least implements the colours I selected from Solarized and should be a good starting point for anyone else.

James Morris: Hardening the LSM API

Thu, 2017-03-09 22:01

The Linux Security Modules (LSM) API provides security hooks for all security-relevant access control operations within the kernel. It’s a pluggable API, allowing different security models to be configured during compilation, and selected at boot time. LSM has provided enough flexibility to implement several major access control schemes, including SELinux, AppArmor, and Smack.

A downside of this architecture, however, is that the security hooks throughout the kernel (there are hundreds of them) increase the kernel’s attack surface. An attacker with a pointer overwrite vulnerability may be able to overwrite an LSM security hook and redirect execution to other code. This could be as simple as bypassing an access control decision via existing kernel code, or redirecting flow to an arbitrary payload such as a rootkit.

Minimizing the inherent security risk of security features, is, I believe, an essential goal.

Recently, as part of the Kernel Self Protection Project, support for marking kernel pages as read-only after init (ro_after_init) was merged, based on grsecurity/pax code. (You can read more about this in Kees Cook’s blog here). In cases where kernel pages are not modified after the kernel is initialized, hardware RO page protections are set on those pages at the end of the kernel initialization process. This is currently supported on several architectures (including x86 and ARM), with more architectures in progress.

It turns out that the LSM hook operations make an ideal candidate for ro_after_init marking, as these hooks are populated during kernel initialization and then do not change (except in one case, explained below). I’ve implemented support for ro_after_init hardening for LSM hooks in the security-next tree, aiming to merge it to Linus for v4.11.

Note that there is one existing case where hooks need to be updated, for runtime SELinux disabling via the ‘disable’ selinuxfs node. Normally, to disable SELinux, you would use selinux=0 at the kernel command line. The runtime disable feature was requested by Fedora folk to handle platforms where the kernel command line is problematic. I’m not sure if this is still the case anywhere. I strongly suggest migrating away from runtime disablement, as configuring support for it in the kernel (via CONFIG_SECURITY_SELINUX_DISABLE) will cause the ro_after_init protection for LSM to be disabled. Use selinux=0 instead, if you need to disable SELinux.

It should be noted, of course, that an attacker with enough control over the kernel could directly change hardware page protections. We are not trying to mitigate that threat here — rather, the goal is to harden the security hooks against being used to gain that level of control.

Rusty Russell: Quick Stats on zstandard (zstd) Performance

Thu, 2017-03-09 12:02

Was looking at using zstd for backup, and wanted to see the effect of different compression levels. I backed up my (built) bitcoin source, which is a decent representation of my home directory, but only weighs in 2.3GB. zstd -1 compressed it 71.3%, zstd -22 compressed it 78.6%, and here’s a graph showing runtime (on my laptop) and the resulting size:

zstandard compression (bitcoin source code, object files and binaries) times and sizes

For this corpus, sweet spots are 3 (the default), 6 (2.5x slower, 7% smaller), 14 (10x slower, 13% smaller) and 20 (46x slower, 22% smaller). Spreadsheet with results here.

Binh Nguyen: Prophets/Pre-Cogs/Stargate Program 8, Github Download Script, and More

Wed, 2017-03-08 20:41
Obvious continuation from my previous other posts with regards to prophets/pre-cogs: http://dtbnguyen.blogspot.com/2017/02/life-in-india-prophetspre-cogsstargate_82.html http://dtbnguyen.blogspot.com/2017/02/life-in-iran-examining-prophetspre-cogs.html http://dtbnguyen.blogspot.com/2017/02/life-in-venezuela-examining-prophetspre.html http://dtbnguyen.blogspot.com/2017/01/

Matthew Oliver: pudb debugging tips

Wed, 2017-03-08 12:06

As an OpenStack Swift dev I obviously write a lot of Python. Further Swift is cluster and so it has a bunch of moving pieces. So debugging is very important. Most the time I use pudb and then jump into the PyCharms debugger if get really stuck.

Pudb is curses based version of pdb, and I find it pretty awesome and you can use it while ssh’d somewhere. So I thought I’d write a tips that I use. Mainly so I don’t forget

OpenSTEM: Oceanography and the Continents

Wed, 2017-03-08 10:04

Marie Tharp (30 July, 1920 – 23 August, 2006) was an oceanographer and cartographer who mapped the oceans of the world. She worked with Bruce Heezen, who collected data on a ship, mapping the ocean floor.

Tharp and Heezen

Tharp turned the data into detailed maps. At that time women were not allowed to work on research ships, as it was thought that they would bring bad luck! However, Tharp was a skilled cartographer, and as she made her maps of the floor of the oceans of the world, with their ridges and valleys, she realised that there were deep valleys which showed the boundaries of continental plates. She noticed that these valleys were also places with lots of earthquakes and she became convinced of the basics of plate tectonics and continental drift.

Between 1959 and 1963, Tharp was not mentioned in any of the scientific papers published by Heezen, and he dismissed her theories disparagingly as “girl talk”. As this video  from National Geographic shows, she stuck to her guns and was vindicated by the evidence, eventually managing to persuade Heezen, and the scientific community at large, of the validity of the theories. In 1977, Heezen and Tharp published a map of the entire ocean floor. Tharp obtained degrees in English, Music, Geology and Mathematics during the course of her life. In 2001, a few weeks before her 81st birthday, Marie Tharp was awarded the Lamont-Doherty Heritage Award at Columbia University, in the USA, as a pioneer of oceanography. She died of cancer in 2006.

The National Geographic video provides an excellent testimony to this woman pioneer in oceanography.

Lev Lafayette: Multicore World 2017: A Review

Tue, 2017-03-07 16:05

Multicore World is a small conference held annually in New Zealand hosted by Open Parallel. What it lacks in numbers however it makes up in quality of the presenters. The 2017 conference included a typically impressive array of speakers dealing with some of the most difficult issues facing computational science, and included several important announcements in the fields of supercomputing, the Internet of Things, and manufacting issues.

read more