Leonora Tindall on Nora Codes

Manually Restoring Scroll Position on Reload with JavaScript

Sat, 03 Jan 2026 10:33:29 -0600

I’m working on another site, written directly in HTML and stitched together with the C preprocessor (it’s really quite ergonomic!) and using the excellent Python package reloadserver to automatically reload the pages in my browser whenever I make changes. Unfortunately, Firefox, at least, doesn’t remember and restore scroll position on reload, which may be the behavior one wants when actually browsing, but is really annoying during the writing and design process.

Fortunately, the JS browser APIs provide very simple interfaces to the page’s scroll position in window.scrollY and window.scroll(x, y). scrollY is a subpixel-precise measurement of where the user has scrolled, and scroll() takes integer arguments for horizontal and vertical coordinates to which to scroll. All we need to do is save the (rounded) value from the first whenever the user scrolls and feed it into the second when the user reloads the page.

We don’t want to use a cookie or localStorage, because those methods of data storage persist between browser sessions and across browser tabs. Instead, we want to use sessionStorage, a very simple KV store whose values are wiped away when the tab or window is closed.

The document API also provides a way to run code on many different events, including scrolling the page, via addEventListener. The document’s 'scroll' event fires whenever the user scrolls, so that’s what we’ll attach to.

Combining these APIs in a short JavaScript snippet, as follows, at the bottom of my web page resulted in exactly the behavior I was looking for.

<script language="javascript">
    document.addEventListener('scroll', (_evt) => {
        const scrollPositionY = Math.round(window.scrollY);
        window.sessionStorage.setItem('scroll_position_y', scrollPositionY);
    });
    const y = sessionStorage.getItem('scroll_position_y') || 0;
    window.scroll(0, y);
</script>

'scroll_position_y', here, is an arbitrarily chosen key for the session storage KV store. It could be whatever you like. If you want to store horizontal position too, add another key, like 'scroll_position_x', storing that value.

Now, the script added by reloadserver reloads the page, which runs my script, which reads the previous scroll position out of session storage and teleports me right back where I was; the delay is imperceptible.

Cheers, and if you have improvements to suggest or find issues with this approach, let me know!

Infrastructure Photography Manifesto (and Decker-in-HTML)

Sat, 08 Nov 2025 19:04:22 -0600

Hey look, I made a zine in Decker! You can read it right here, or download the deck file.

And now for the nerd shit. Decker is really cool. It carries on the legacy of HyperCard, one of the first real hypertext media systems; it’s got that dithered, lo-fi, retro flair that I adore; and it runs in a lot of places - anywhere with SDL, and anywhere with a full-fledged browser.

I made a zine with it a while back, but I didn’t publish it, or make anything else with it, because I didn’t realize that it was possible to publish Decker decks without the Decker UI. (You can do this by going to File -> Properties -> Protect…, which doesn’t add DRM but rather just emits a Decker deck that has the UI turned off. It can still be opened in Decker and edited.)

There was still the problem, though, that you can only really export a full-page HTML document with a single Decker canvas in the middle of it, and only at 2x scale. This is normally really good, but I’m a weirdo and wanted Decker embedded on my website. Several folks were happy to tell me that this could be done with an iframe, but I had a hunch that I could produce a better experience with a straight-in embed of the Decker canvas. And, well:

This page, in my Hugo setup, includes two shortcodes like this:

<script language="decker">
{{< include "static/code/decks/infrastructure exp04.deck" >}}
</script>

{{< decker >}}

That first shortcode, include, is very simple:

{{ .Get 0 | readFile | safeHTML }}

The second, decker, looks like this:

<a id="target" style="display:none"></a>
<img id="loader" style="display:none">
<input id="source" type="file" style="display:none">
<canvas id="render" style="display:none"></canvas>
<canvas id="lrender" style="display:none"></canvas>
<canvas id="rrender" style="display:none"></canvas>
<canvas width="0" height="0" id="ltools"></canvas>
<div class="decker-container">
    <canvas class="decker-display" id="display"></canvas>
</div>
<canvas width="0" height="0" id="rtools"></canvas>

<script src="/code/decker-itself/lil.js"> </script>
<script src="/code/decker-itself/decker.js"> </script>

This is almost verbatim from the Decker-web HTML file (as of v1.60+8360e66), with a few modifications:

  <canvas width="0" height="0" id="ltools"></canvas>
+ <div class="decker-container">
- <canvas class="decker-display" id="display"></canvas>
+   <canvas class="decker-display" id="display"></canvas>
+ </div>
  <canvas width="0" height="0" id="rtools"></canvas>

+ <script src="/code/decker-itself/lil.js"> </script>
+ <script src="/code/decker-itself/decker.js"> </script>

Those two scripts are also copied verbatim from the repo, at commit 8360e66.

I used a small CSS snippet to change the size and position of the Decker display canvas:

div.decker-container {
  display: flex;
  align-items: center;
}

canvas.decker-display {
  width: 512;
  height: 324;
  margin-left: auto;
  margin-right: auto;
  cursor: default;
}

Some issues remain. Namely:

You can only have one Decker embed per page.
Decker resets the page’s title.

That’s expected, but there are some straight-up bugs I ran into during this process, too:

Hugo freaks out if you put the include step in the Decker shortcode. I have no idea why, but it inserts \n instead of actual newlines, breaking the embedded Decker deck.
Decker can’t handle there being whitespace before the start of the deck’s data.
You can’t right-click on most of the page; Decker intercepts it everywhere, not just inside the Decker canvas.
If the intro is too long, the Decker canvas is broken (you can’t interact with it) and too big.

If folks have advice on how to fix it, or if we could get an embed-into-page export format for Decker, that’d be great!

Huge thanks to Millie Millennium for inspiring me to get back into Decker, and for giving me helpful info on this particular silly quest.

nimi sin

Tue, 01 Apr 2025 00:00:00 +0000

This post is part of April Cools Club: an April 1st effort to publish genuine essays on unexpected topics. Please enjoy these true stories, and rest assured that the tech content will be back ~~soon~~ eventually!

sina o pali e nimi sin. sina o pana e nimi mute tawa mi.
nimi o kama sama pi soweli lili. mi wile e ni: nimi li kama mute a.

That’s toki pona, a constructed language with somewhere between 118 and 181 words, depending on who you ask. It was meant to be quick and easy to learn, but more importantly, to guide the mind toward a simpler, better, more pona way of being. The fewer words, the broader their semantic space, the more contextual and general their meaning, the better. That’s the toki pona philosophy - but that’s not what I’m asking you for.

I’m asking you to make more words. New words. Strange words. Specific words. I want our words to multiply like rats.

In software, as in math and gaming and many kinds of engineering, we have far too few words, and we load them with meaning far beyond the breaking point. In what other field are “kind” and “type” so likely to be entirely different things?

We talk about tables (on the web, for displaying tabular data and nothing else; in a database, a two-dimensional array of a particular shape; in Lua, an associative array) and maps (applying a function to each element of an array? an associative array itself? who knows) and classes and types and vectors and packets and references and a dozen other things with three or five meanings each. In my professional work, I find myself differentiating between the “gateway server” and the “gateway server project” and the “gateway server solution”. I often slip up, and it often leads to problems.

But it’s not inevitable. There are other fields, other things full of complexity and craft and nuance, with their own words - fields that aren’t fifty or sixty years old but hundreds, or thousands. Those fields give us words like bight (a curved bit of rope) and chaff (the dry outside of the seeds of grain) and adze (a cutting tool with the cutting edge perpendicular to the handle) and cordwain (shiny leather from the rump of a horse.) These words accumulated over centuries, as people from different corners of the world with different etymological lineages met and had to communicate about the things they were building and doing and selling and buying.

Those words have become overloaded, too, as software and geography and aerospace engineering have spread out their tentacles, grabbing for signifiers they can attach to the new, specific meanings they’re creating - and I tried that too, for a while. I grabbed at concepts like true name to describe the set of information required to identify a particular cached web response, or acronyms like MISO (Metadata Inside Small Objects) to describe a technique for reducing indirection in a particular datastore. Cultural vampirism will get you far; that’s how we ended up with concepts like the terminal (from terminal emulator, meaning a program that emulates a computer terminal, itself from from the railway term terminal which comes from the concept of a terminal station - the end of the line.) For me, though, it’s not far enough.

So I’m making nimi sin - new words. When I have a new concept in my code, or my tabletop preparation, or just in my life, I spend some time on Wiktionary figuring out what I can call it. I describe the concept in a few words, trace their etymologies back a few hops, and rummage around for something that sounds good. My tabletop setting of little bounded realities is divided by plamurs, from planet and murus. Stochastic monitoring at a previous job was done by selecting beorends, from the Old English beorht (bright) and ærende (message, from when we get “errand”.)

In making nimi sin (or nimisin, as some toki pona speakers prone to the creation of nimi sin have taken to calling them), we lose some things. A well-selected overload - something semantically adjacent or metaphorically appropriate - can be very intuitive to some people. Most nimi sin can’t trigger nearly as many associations or bring forth as many new connections, and frankly, they sound a little silly to a lot of people.

On the other hand, we gain a lot. nimi sin are much more searchable (grep beorend will get you far on that codebase) on account of their specificity. Having a word that means only one or two things enhances the redundancy that underpins successful spoken communication, too, in that using the nimi sin reinforces the context of the conversation, rather than relying on it to convey meaning. Simply hearing “plamur” tips my players off that I’m talking about the cosmology of my fictional world.

So try it out! Come up with a few words and start using them. Sound a little silly. It’s been worth it for me.

Adding Tags to My Projects Page

Thu, 03 Oct 2024 11:23:02 -0400

I’ve wanted to change up my projects page for a while, but wasn’t quite sure what to do. I didn’t want to use any JavaScript, but I did want to organize projects in multiple different ways, rather than just a single set of categories like I’ve had for a long time.

Ultimately what I settled on is a tag system. The tags look like this:

Project Name author library rust is a project!

These tags are just text with some styling applied, and they’re inserted with a Hugo shortcode. I originally thought to use individual shortcodes for each kind of tag, like {{< t-rust >}} or {{< t-author >}}, but after making a few shortcode files that looked like this:

<span class="tag t-author">author</span>

I decided to automate the whole thing. Hugo has a very powerful template syntax, and iterating over the arguments to a shortcode is easy, so I ended up with a shortcode like {{< t-all author library rust >}} implemented using a loop:

{{- range (seq 0 (sub (len .Params) 1) ) }}
    {{- $arg := $.Get . }}
    <span class="tag t-{{ printf `%s` $arg}}">
        <span class="tag-text">{{ printf `%s` $arg}}</span>
    </span>
{{- end }}

This means there’s a span around the whole element, on which I can put the background styling, and an internal span to style the text itself. This is easy enough to do in CSS:

/* Tags for project items */
.tag {
  display: inline-block;
  color: var(--bg-color);
  background-color: var(--fg-color);
  opacity: 60%;
  padding-left: 0.25em;
  padding-right: 0.25em;
  border-radius: 8px;
}

.tag-text {
  opacity: 100%;
  font-weight: bold;
  font-size: 75%;
}

--bg-color and --fg-color are defined elsewhere; they’re the background and text of the blog, respectively, so flipping them like this makes tags very much stand out, by default. You might notice that, except for one-off tags like this , these tags are actually colored. That’s accomplished with a little more CSS:

.t-rust,.t-python,.t-javascript,.t-bash,.t-cpp {
  background-color: #cc3300;
}

.t-author,.t-contributor {
  background-color: #11aa00;
}
/* ... and so forth */

This is the most manual part of the setup, but it’s no big deal; I did consider giving tags a “class” so they could automatically pick a color, but it was much too complicated to be worth it. When I learn a new language, I’ll add to my CSS.

I’m really happy with how this came out, but I’d love to hear feedback from you all on whether you find this useful and/or readable!

A Matter of Time

Sun, 02 Jun 2024 00:00:00 +0000

“The Temper of Revenge” is one of my favorite filksongs, though I was introduced to it by the Star Trek II-related parody, “The Temperature of Revenge”. A few weeks ago, on a bus from Madison to Chicago, I decided to see if I could fit some lyrics about my favorite video game, Outer Wilds, to the tune. It took a lot of revising to do so without massively spoiling the game - and, if you ever intend to play it, do be careful of spoilers! - but I think I’ve succeeded.

A Matter of Time

(to the tune of The Temper of Revenge, by Julia Ecklar)

We were born on a fragile, ancient world
Raised up with tree and reed
There’s no time for what we could become
Yet I’ve all the time I need

I wake from sleep, it seems, again
With star-fire strong in my heart
I step out from the light, walk again through the night
And towards a black sky, I depart

My memory’s filled with names and sights
Fresh in my mind, a web is unfurled
This time I search not for trinkets or lore
But how I can shield my world

CHORUS:
So find me a ship, no matter how small
Find me a hatchling to answer the call
Hearth’s wood, Nomai metal, with courage will fly
In a friendship to rival the death in the sky

Bring me my scout, bring my scope
Far from this world, there still may be hope
I’ll search far and wide; towards the stars I will climb
For it’s only a matter of time -
It’s only a matter of time

My village is all I’ve ever known
Beneath skies so vivid and clear
We started our lives at the end of the world
But our story could never end here

My ship holds my air, my spacesuit, and me
As out from the timbers I speed
I’ll scour every world for the secrets they hold
For knowledge is all that I need

I’ll search every ship, each ruin and moon
I’ll read every text and ponder each rune
I’ll fly for an age, if that’s what it’ll take
Until I find the the answers I’m due!
CHORUS

As a reminder, these lyrics, like everything on this site, are licensed CC-BY-SA. That means you can copy them, remix them, and sell things based on them, so long as you retain an attribution notice and the CC-BY-SA license. Note that the tune is not so licensed.

Methods Should Be Object Safe

Sat, 04 May 2024 18:06:36 -0500

I think we should use “method”, in Rust, to refer specifically to associated functions which would be object safe if put in a trait.

In this terminology scheme:

associated function refers to any function defined in an impl block, which is how most people use it now¹^,²
trait function refers to any function defined in a trait
method specifically refers to associated functions with receivers like those specified in the object safety section of the Reference³
and we should come up with another name for associated functions that take self but are not object-safe, such as taking self by value.

That’s the whole idea; if you’re unfamiliar with what any of that means, read on.

Object Oriented Programming in Rust

I was chatting with a friend about object oriented programming and wanted to use Rust to illustrate why I find it odd that programmers often act like “OOP” is one big concept, when, really, it’s more like a kind of Vietnamese spring roll of concepts that can be inserted or taken out depending on preference.

For instance, in Rust, it’s entirely possible - indeed, quite normal - to write so-called free functions, like so:

// counter_free.rs
pub struct Counter {
    pub count: u32,
}

pub fn increment(c: Counter) -> Counter {
    Counter { count: c.count + 1 }
}

mod test {
    use super::{Counter, increment};

    #[test]
    fn count_up() {
        let c = Counter { x: 0 };
        let c = increment(c);
        assert_eq!(c.count, 1);
    }
}

It’s hard to call this “object oriented” in any meaningful way, although I’m sure some readers will enthusiastically correct me (which I welcome!) Counter does not encapsulate its data, nor is its behavior in any way bound to that data.

We can also write associated functions, which are a lot like free functions but are named differently and written inside impl blocks. Since the compiler knows which type the function is “about”, it also gives us a shortcut; instead of naming the type when it’s used in that function, we can just write Self.

// counter_associated.rs
pub struct Counter {
    pub count: u32,
}

impl Counter {
    pub fn increment(c: Self) -> Self {
        Self { count: c.count + 1 }
    }
}

mod test {
    use super::Counter;

    #[test]
    fn count_up() {
        let c = Counter { x: 0 };
        let c = Counter::increment(f);
        assert_eq!(c.count, 1);
    }
}

Here, Counter still doesn’t encapsulate its data, but its behavior - increment - is bound, at least in name, to Counter as a type. I could define a Cycle::increment in the same module without any confusion occurring. Is this OOP? I say no; we’ve essentially just changed the name of the function, and made it a bit more ergonomic to import.

Rust gives us another bit of special syntax, though, which changes things. This is the self argument, or receiver, and it looks like so:

// counter_receiver.rs
pub struct Counter {
    count: u32,
}

impl Counter {
    pub fn new() -> Self {
        Self { count: 0 }
    }

    pub fn count(&self) -> u32 {
        self.count
    }

    pub fn increment(self) -> Self {
        Self { count: self.count + 1 }
    }
}

mod test {
    use super::Counter;

    #[test]
    fn count_up() {
        let c = Counter::new();
        let c = c.increment();
        assert_eq!(c.count(), 1);
    }
}

This looks a lot more like object oriented programming, at least to me. The data is encapsulated; we could have some complex memory-saving method of recording the count value inside of Count, for all the user knows, and just convert it to a u32 when Counter::count is called.

Generics and Static Dispatch

But - is it object oriented in Rust? In Rust, we use object to mean something very specific: a type with a name including dyn Trait. Counter could implement a trait, certainly; let’s call it Increment.

// counter_trait.rs
pub struct Counter {
    count: u32,
}

impl Counter {
    pub fn new() -> Self {
        Self { count: 0 }
    }

    pub fn count(&self) -> u32 {
        self.count
    }
}

pub trait Increment {
    fn increment(self) -> Self;
}

impl Increment for Counter {
    fn increment(self) -> Self {
        Self { count: self.count + 1 }
    }
}

mod test {
    use super::{Counter, Increment};

    #[test]
    fn count_up() {
        let c = Counter::new();
        let c = c.increment();
        assert_eq!(c.count(), 1);
    }
}

Again, we’ve changed very little here. We’ve indirectly renamed increment again; it can be referred to as Increment::count, or as <Count as Increment>::increment.

We’ve also given ourselves, as programmers, some flexibility. Since Increment could apply to many different types, it’s possible to make our code a bit more future proof, using generics. For instance, we could write a function like this:

// counter_trait.rs (append)
fn double_count<Inc: Increment>(i: Inc) -> Inc {
    i.increment().increment()
}

#[test]
fn counter_double_count() {
    let c = Counter::new();
    let c = double_count(c);
    assert_eq!(c.count(), 2);
}

Not very novel, but it’s interesting in that it would work just as well with any other type that is Increment. For instance:

// counter_trait.rs (append)
pub struct Turnstyle {
    entries: u32,
    exits: u32,
}

impl Turnstyle {
    fn new() -> Self {
        Self {
            entries: 0,
            exits: 0,
        }
    }

    fn occupants(&self) -> u32 {
        assert!(self.entries >= self.exits);
        self.entries - self.exits
    }
}

impl Increment for Turnstyle {
    fn increment(self) -> Self {
        Self {
            entries: self.entries + 1,
            exits: self.exits
        }
    }
}

#[test]
fn turnstyle_double_count() {
    let t = Turnstyle::new();
    let t = double_count(t);
    assert_eq!(t.occupants(), 2);
}

This is getting a lot closer to what languages like Java tend to use their object oriented paradigm for. double_count doesn’t care much what the datatype it’s called on is, so long as it implements Increment. The compiler will generate one version of double_count that works for Counters and one that works for Turnstyles in a process known as monomorphization.

Unlike Java, though, we can’t have a collection (say, a Vec) of Increment; we have to either explicitly name a particular type, like Vec<Counter>, or use dynamic dispatch.

Dynamic Dispatch and Object Safety

This is where the existing definition of Increment falls short. If we try to name such a collection - say, for instance, by putting our various Increment types on the heap with Box - we get a very interesting compiler error.

// counter_trait.rs (append)
fn increment_all(items: Vec<Box<dyn Increment>>) {
    items.into_iter().map(|i| i.increment()).collect()
}
// ERROR: `Increment` cannot be made into an object
// note: for a trait to be "object safe" it needs to allow building a vtable to
// allow the call to be resolvable dynamically

In Rust, grouping values by trait rather than type is done through dynamic dispatch. Types like Box, Arc, and so forth allow pointers to the data to be stored along with pointers to the behavior related to that data, known as a vtable. The compiler doesn’t necessarily know the types of all implementers of Increment at the time increment_all is being generated, so monomorphization isn’t an option.

An object-safe version of Increment would take &mut self and modify the Counter, Turnstyle, or whatever other value in place, much like a Java method with similar functionality.

For that reason, all the functions in a trait that’s being used to define a trait object (like &dyn Increment, Box<dyn Increment>, etc.) must take pointer receivers, like &self, Box<Self>, and so forth.

This is why I say we should only call associated functions that would be object safe “methods”. They are the associated functions that could be used in an “object-oriented” way, both in that they work with what we call “objects” in Rust, and in that they can be used in the dynamic way that “method” often connotes in object-oriented languages.

The Rust Reference 6.15 Associated Items - Associated functions and methods ↩︎
Rust by Example 9.1 Associated functions & Methods ↩︎
The Rust Reference 6.11 Traits - Object Safety ↩︎

Adding Full Post Content to my RSS Feed

Tue, 27 Feb 2024 09:26:52 -0600

A reader reached out to me the other day to let me know that my RSS feed for this blog didn’t include post content, just descriptions, and that this hindered their ability to read my posts offline. This obviously isn’t ideal, so I set about adding the full post content.

Customizing the RSS Feed

Hugo comes with a builtin RSS feed template, but it’s easy enough to override. Simply copy the contents of this file to the themes/[theme name]/layouts/_default directory, and then edit to your heart’s content.

By default, the post-related part of this feed looks something like this:

<item>
    <title>{{ .Title }}</title>
    <link>{{ .Permalink }}</link>
    <pubDate>{{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}</pubDate>
    {{- with $authorEmail }}<author>{{ . }}{{ with $authorName }} ({{ . }}){{ end }}</author>{{ end }}
    <guid>{{ .Permalink }}</guid>
    <description>{{ .Summary | transform.XMLEscape | safeHTML }}</description>
</item>

In Hugo, post content is available in .Content. It’s tempting to simply change the <description> tag:

- <description>{{ .Summary | transform.XMLEscape | safeHTML }}</description>
+ <description>{{ .Content | transform.XMLEscape | safeHTML }}</description>

but there is a better way: <content:encoded>.

The content: Namespace

The default Atom XML namespace doesn’t provide a way to have both a post description and the full post content, but there is a namespace that does: the Content Namespace Extension. Adding this name space is easy; simply change the opening rss tag to include it:

- <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
+ <rss version="2.0"
+    xmlns:atom="http://www.w3.org/2005/Atom"
+    xmlns:content="http://purl.org/rss/1.0/modules/content/">

and then, within the <item>, add a tag that contains the URL-encoded content:

 <item>
     <title>{{ .Title }}</title>
     <link>{{ .Permalink }}</link>
     <pubDate>{{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}</pubDate>
     {{- with $authorEmail }}<author>{{ . }}{{ with $authorName }} ({{ . }}){{ end }}</author>{{ end }}
     <guid>{{ .Permalink }}</guid>
     <description>{{ .Summary | transform.XMLEscape | safeHTML }}</description>
+    <content:encoded>{{ .Content | transform.XMLEscape | safeHTML }}</content:encoded>
 </item>

This way, RSS readers can display the post description in their feeds/lists, but can still get access to the full post content and make it available to readers offline.

Pinning nixpkgs with Morph and Colmena

Sun, 18 Feb 2024 17:55:24 -0600

I’ve used NixOS and Morph to manage the servers in my house for about a year now, and while I have many gripes with both the Nix language and the documentation around the nixpkgs and NixOS ecosystem, it’s been a great help.

This post is for users of Morph and its near-drop-in replacement, Colmena. If you don’t already use one of these tools, take a look at them first!

Why Pin nixpkgs?

“Pinning”, in this context, refers to making the version of nixpkgs against which my configuration is built an explicit argument, written down and version controlled, rather than an implicit feature of whatever machine the build is run on and its Nix channels.

For a long time, I used Nix channels to manage the versions of packages deployed to my servers. This approach has a lot of drawbacks; for example, it’s difficult to pin a specific version of a package, and it can mean that a simple change requires a full rebuild if the channel has updated. Pinning nixpkgs also means that fixing a broken configuration is just a git revert away, and I can share my config with friends with the confidence that they’ll get the same result as I did.

The breaking poing for me, though, was installing home-manager on my laptop, which is also best managed through a Nix channel - and not the same one.

My servers run NixOS, meaning I need to use the NixOS-versioned home-manager channel for them. On my laptop, though, I gain a lot of useful tools by using the more quick-moving unstable channel. Doing both at the same time is a pain, and that gave me the impetus needed to switch to a pinned nixpkgs.

Pinning with Morph

The easiest way to do this is to simply add a call to builtins.fetchTarball to my Morph network (or Colmena hive, see below). My old network.nix looked like this:

{
  network = {
    description = "Home network";
  };

  "felonyspork" = { ... }: {
    imports = [
      ../../common/default.nix
      ../../hosts/felonyspork/configuration.nix
    ];
    deployment.targetUser = "root";
    deployment.targetHost = "felonyspork";
  };

  # ... other hosts
}

Each host imports common/default.nix, which contained some configuration for nixpkgs:

{ config, pkgs, ... }:
{
  nixpkgs.config = {
    config.allowUnfree = true;
  };

  # ... other configuration
}

fetchTarball requires two arguments: a URL, and the expected hash of the file at that URL, so Nix can tell you if and when it changes. To get the current state of the NixOS nixpkgs channel, first ask Git to get the commit ID for you. In this example, I am using nixpkgs 23.11, but you should use whatever the version is that you’re deploying to your servers.

$ git ls-remote https://github.com/nixos/nixpkgs nixos-23.11
84d981bae8b5e783b3b548de505b22880559515f        refs/heads/nixos-23.11

A simple cut will get just the hash, which can then be inserted into the GitHub archive URL:

$ echo https://github.com/nixos/nixpkgs/archive/$(git ls-remote https://github.com/nixos/nixpkgs nixos-23.11 | cut -f1).tar.gz
https://github.com/nixos/nixpkgs/archive/84d981bae8b5e783b3b548de505b22880559515f.tar.gz

Finally, Nix can prefetch the file, adding it to the Nix store for later use, and emit the hash at the same time.

$ nix-prefetch-url --unpack https://github.com/nixos/nixpkgs/archive/$(git ls-remote https://github.com/nixos/nixpkgs nixos-23.11 | cut -f1).tar.gz
path is '/nix/store/wb41vb32nf7748dr42psfgqw4cjc1pyf-84d981bae8b5e783b3b548de505b22880559515f.tar.gz'
0d6j5d31kzfla0x8f64ranp681dhd0hwxihbf3jjpb18cnddxag8

Equipped with these two pieces of information, you can add a fetchTarball invocation in a binding at the top of your network.nix:

let
  nixos_23_11 = builtins.fetchTarball {
    name = "nixos-23.11-2024-02-18";
    url = "https://github.com/nixos/nixpkgs/archive/84d981bae8b5e783b3b548de505b22880559515f.tar.gz"; 
    sha256 = "0d6j5d31kzfla0x8f64ranp681dhd0hwxihbf3jjpb18cnddxag8";
  };
in {
  network = {
  # ...

This creates a binding (that is, a name for a value) to a path in the Nix store, which can be loaded with import. Loading the Nix files in that folder results in a function which, when called with whatever options you need to set, produces the nixpkgs attrset. That’s then passed to your configuration files as the pkgs argument by Morph.

Note that I’ve used the URL and hash computed here, and used the current date as part of the name. Don’t do that unless you’re reading this on release day, because NixOS will probably have updated since then!

Morph allows you to specify the nixpkgs data in the network metadata section. This is the most natural place to add any configuration options such as allowUnfree. My network.nix ended up looking like the following:

let
  nixos_23_11 = builtins.fetchTarball {
    name = "nixos-23.11-2024-02-18";
    url = "https://github.com/nixos/nixpkgs/archive/84d981bae8b5e783b3b548de505b22880559515f.tar.gz"; 
    sha256 = "0d6j5d31kzfla0x8f64ranp681dhd0hwxihbf3jjpb18cnddxag8";
  };
in {
  network = {
    description = "Home network";
    pkgs = (import nixos_23_11) {
      config.allowUnfree = true;
    };
  };

  "felonyspork" = { ... }: {
    imports = [
      ../../common/default.nix
      ../../hosts/felonyspork/configuration.nix
    ];
    deployment.targetUser = "root";
    deployment.targetHost = "felonyspork";
  }; 

  # ... other hosts
}

Since the nixpkgs options are configured here, they can be removed from other files; in my case, common/default.nix.

Pinning with Git Directly

Nix also provides a way to work with Git repos directly, via builtins.fetchGit. The main advantage over fetchTarball is to be able to specify a branch or other ref, rather than a specific commit SHA. In this case, specifying a commit is the whole idea, so fetchGit is not appropriate.

Pinning with Colmena

Colmena uses a slightly different hierarchy for its network metadata. A hive.nix with the same NixOS 23.11 pin as the above network.nix would look like this:

let
  nixos_23_11 = builtins.fetchTarball {
    name = "nixos-23.11-2024-02-18";
    url = "https://github.com/nixos/nixpkgs/archive/84d981bae8b5e783b3b548de505b22880559515f.tar.gz"; 
    sha256 = "0d6j5d31kzfla0x8f64ranp681dhd0hwxihbf3jjpb18cnddxag8";
  };
in {
  meta = {
    description = "Home network";
    nixpkgs = (import nixos_23_11) {
      config.allowUnfree = true;
    };
  };

  "felonyspork" = { ... }: {
    imports = [
      ../../common/default.nix
      ../../hosts/felonyspork/configuration.nix
    ];
    deployment.targetUser = "root";
    deployment.targetHost = "felonyspork";
  }; 

  # ... other hosts
}

Note that network is renamed to meta, and pkgs is renamed to nixpkgs.

Using a Pinned nixpkgs

One drawback of this approach is having to manually update the pinned version. If this really gets on your nerves, you could use a shell script to do so, as long as your NixOS config is version controlled.

As of writing this post, the policy is for the NixOS binary cache server to hold on to builds forever. This helps with pinned NixOS versions, because it means that even if your pinned version is very out of date, you won’t have to manually rebuild things (except non-free packages, or packages you modify.) This policy, however, is going to change; see this forum thread for more details.

Fly On, Columbia! (Fly Columbia II)

Mon, 28 Aug 2023 00:00:00 +0000

“Fly, Columbia” is a deeply hopeful song. Written in 1981 by Diana Gallagher, Leslie Fish’s performance on Minus Ten and Counting was one of the first filk recordings that really captured me. The chorus names OV-102 “the promise of better days to come”, and blesses her to “Sail an orbit free, track the moon and chase the sun.”

Yet, that was not her fate, as we know all too well.

When I heard The Longest Johns cover “Roll Northumbria”, I immediately thought of this song, and this ship; so I set out to filk it.

Fly On, Columbia! (Fly Columbia II)

(to the tune of Roll Northumbria, by The Dreadnaughts)

’twas in seventy-five ’neath the Palmdale sun
That work on the queen of the skies was begun
We built her from tungsten and carbon long-spun
Fly, Columbia, fly

For though lunar missions had come to an end
We still had our place in the stars to defend
On pillars of flame to the sky she’d ascend

CHORUS:
Fly, Columbia, fly, me boys
Fly, Columbia, fly
And it’s one for dark sky above
Two for the crewfolk we love
And it’s three for the millions that watch from below
Fly on, Columbia
Fly, Columbia, fly

Challenger, Enterprise, Buran and all
She was the first; with her they’d stand or fall
From the Vandenberg desert through the East’s ocean squalls
Fly, Columbia, fly

In three years and two thousand her engines did burn
And seven doomed spacers did see the Earth turn
As we watched them drift down with naive unconcern
CHORUS

So come all eager spacers who yearn for the sky
And beware of the small thing that passes your eye
Lest the powers above your return will deny
Fly, Columbia, fly

Fifteen minutes from land, a dread silence o’ercame
For the space program’s heart ne’er again was the same
And seven good spacers were lost to the flame
CHORUS x3

The Horror

Thu, 08 Jun 2023 12:55:11 -0500

Twelve

It starts when you’re about twelve.

You haven’t thought about it much until then; people call you things, but you’re just a kid, and that doesn’t matter yet. You ask for the toys you like, and you don’t think about whether they’re “boy” or “girl” toys. You have the friends you have, and maybe a few people make jokes about you having a “girlfriend”, but that’s all.

You heard your parents tell a story once about how your mother thought she was having a girl, and she got you all girl’s baby clothes, and she couldn’t stand to see them on you. People laugh. You don’t get it.

You go to the doctor every year for a checkup. They look up your nose, tell you to breathe in and breath out, have you stand up and sit down, put a cold stethoscope on your chest. It’s boring, but the doctor is funny, and sometimes you get a candy.

This time, something’s different. The doctor pulls your parents aside, and they talk about something you can’t quite hear. They all look very serious, like the time you spilled something on your dad’s laptop and he got really quiet and asked you to step out of his study. But they’re not mad at you. You get a lollipop and a sticker, and the doctor leaves.

On the drive home, they tell you things are going to change. You’re going to start seeing differences in your body. You ask if you’re going to get taller and they say yes, and that makes them smile. You ask what else, and they say, “We’ll talk about it later.”

You don’t talk about it later.

Thirteen

When you’re thirteen, you start seeing the first hairs on your arms. It’s freaky. Some of the other kids are happy about it; they talk about how it means they’re becoming men. One of them flexes and shows how big his bicep muscles are. You think it’s all kind of stupid, but you show off your one hair anyway and everyone marvels at it. Your parents tell you that it’s normal.

You’re in health class. The teacher talks about how boys and girls are different - how boys have more body hair, and their voices are deeper, and they’re taller and stronger and smell worse. You think about how you have to wear deodorant now, and it makes you feel strange.

You start having nightmares. You see people in masks, people with knives. You want to move, to scream, to run, but you can’t. Your body feels wrong.

At night, when you wake in a cold sweat, it’s almost like you can feel the hairs pushing out of your skin. You stop riding you bicycle for fun because hate the feeling of the wind across your skin. It just reminds you of the hairs. You buy a razor, but your parents say it’ll just grow back darker. That makes you cry, and your mother gives you a strange look. Confusion - maybe disgust.

Fifteen

You’re fifteen. Your voice starts to crack and deepen. It’s embarrassing for your friends, too, but you don’t get the sense that they feel the same way you do. They don’t get the pit in their stomach when it happens. It doesn’t make them feel like their body is betraying them. You start talking less. Your report card calls you “withdrawn”.

Eventually, you mention it to a friend. You say you don’t like how hairy you are, or how you smell, or how your voice sounds. He calls you a faggot, as a joke, and tells you that’s just what being a man is like. You don’t laugh.

On the way back from the store one day, your mom asks if you have a girlfriend. You don’t. A boyfriend? No. She tells you that you can asked her anything, if you want to talk about sex. You definitely don’t want to talk about sex.

Sixteen

You start skipping class, hanging out with a bunch of girls in the cafeteria. You get sent to the guidance counselor and you tell her you feel gross, and dirty, and ashamed. She tells you to it’s normal, that it’s all just part of growing up. It doesn’t help. You think that maybe you are a faggot.

The nightmares get worse. Nightmares about doctors and knives. Nightmares about the hospital. Nightmares about other kids holding you down and hurting you. The kids turn into your parents. You decide to start taking diphenhydramine to sleep without dreaming.

You quit school and go online. The courses are only middling, and you spend most of your time playing video games, but at least you don’t have to be around the other kids every day. You can live in your own little world and feel, if not okay, at least less like a monster.

Eighteen

You get into a middling college, with a good scholarship. People wonder why, with your grades, you didn’t go to a bigger, better, fancier school. Your family could afford it. You tell them there was a mixup with your transcript. It’s true, but you could have fixed it. In reality, you just didn’t want to be around those people - people you respected. You knew they’d see right through you, know you for a monster.

It finally gets so bad that you confront your parents. “What’s happening to me? I don’t want to be a boy. I don’t like this.”

You mother tells you that she can’t understand why you’d want to be a woman. Don’t you know how bad women have it? To her, womanhood is pain, and fear, and oppression. You tell her you think it’s beautiful to be a woman, that it’s meaningful. You can’t get what you mean across to her, or she doesn’t want to listen. Either way, it feels like a slap in the face.

The worst part is, you know it doesn’t matter. You’ve been doing research. You’ve read that the effects of testosterone are permanent. They can’t be reversed. There’s surgery for your voice, but that might make you unable to speak. There’s laser hair removal, but it’s expensive, and painful, and anyway you don’t want to “mitigate” the body hair, you want it gone.

It’s over. You’ve lost a war against your body that nobody even told you could be won. This is the horror.

This is the Horror

This is not a story about a forced transition, though it might read like one. Were those parents giving you testosterone injections, or gel, or pills against your will? Almost certainly not; this simply doesn’t happen, except for instances of intersex children being forcibly transitioned into a binary gender presentation their parents or doctors are more comfortable with.¹

No, this is a story about my life. This is very similar to what happened to me, with a few slight inventions and omissions so as not to immediately give away my assigned gender at birth. From the ages of 13 to 20, I was incredibly uncomfortable with what was happening to my body because of my endogenous testosterone, and nobody told me there was an alternative. I almost killed myself twice because of it.

I cannot describe to you the level of horror that comes from seeing your body twist into shapes and forms you do not want, do not feel are even your own. Had I had access to puberty blockers or hormone replacement therapy at 13, I would not have a voice I am uncomfortable using to this day. I would not have spent thousands on laser hair removal to feel safe walking down the street. I would be fitter, happier, and more accomplished, because I wouldn’t have been wallowing in dysphoria and depression for a third of my life.

Fight the Horror

We fight the horror with acceptance, with access to gender-affirming care, with telling people that it’s okay to be trans, it’s okay to be gay, and it’s okay to need to figure it out.

We fight the horror with education, and legislation, and by tearing down the transphobic edifice of a society that expects everyone to have a binary, immutable gender.

When people talk about reducing access to transition for children, this is what they are inflicting. A small fraction of patients may go through something similar because of opting into gender affirming care that isn’t right for them, but those people can end the horror at any time - they can simply stop taking hormones and be better off than I was.

100% of trans people who seek access to gender affirming care as children and are denied go through the horror. 100% of trans children who never know that gender affirming care exists go through the horror. And for what?

When you vote for bills that reduce access to puberty blockers, you are condemning another generation of trans kids to live in the horror for their entire adolescence. When you vote to ban books about transgender people from school libraries, you are spreading the horror. When you go on the Internet and talk about how you’re not transphobic, you’re just worried about detransitioners and the “transgender trend”, you’re spreading the horror. When you advocate for “correcting” the genitals or hormones of intersex children without their consent, you’re spreading the horror.

It’s your responsibility to fight the horror. The horror is my enemy, and it should be yours.

I had the great privelage to meet and protest with the wonderful intersex activist Pidgeon Pagonis in 2018, and highly recommend their work. If you’re interested in the topic of intersex children’s rights, I suggest starting with this article and this interview regarding their activism. ↩︎

The Fediverse is Already Dead

Thu, 23 Feb 2023 21:21:52 -0600

The Fediverse is not dying, nor is it crumbling. As an ideoform in the collective consciousness of the ’net, the Fediverse is already dead.

What was the Fediverse?

Stripped of technical terminology, the Fediverse (“federated universe”) refers to a bunch of web services which let people share text, images, video and audio with each other without a centralized authority.

Isn’t That Mastodon?

You might be used to hearing about “Mastodon” rather than the Fediverse. Mastodon-the-corporation has done a lot of advertising and promotion, and there’s a good deal of documentation out there on how to start a Mastodon server - but Mastodon isn’t all there is to the Fediverse.

The Fediverse originally consisted of computers running a piece of software called StatusNet, which eventually became GNUSocial. ¹ Each server allowed people to post and read messages, which were distributed through a social graph much like that of Twitter or Tumblr. Unlike Twitter or Tumblr, but like e-mail or IRC, those posts were distributed through a “federation protocol” that let users on different servers converse. This network existed as early as 2010, and went through several protocol migrations over time, eventually landing on something called OStatus.

More recently, a German software engineer named Eugen Rochko released software called Mastodon in 2016 ², which implemented OStatus and allowed its users to talk to each other, and to users of GNUSocial. This software had a better user experience for many people than GNUSocial, and wasn’t tainted by association with free software extremism, and as a result began to catch on.

Others have documented the sordid tale of Mastodon’s development better than I could, but suffice it to say that many queer people, people of color, and women have been cast aside by Rochko despite significant and often culturally defining contributions to the software and ecosystem.³^,⁴The organization built around it, Mastodon gGmbH, remains a sigificant cultural force in the distributed social media landscape due to its large budget and market share, using its tens of thousands per month to pay Rochko’s salary, hire contractors, and produce marketing material.

So much for Mastodon.

Diversity of Design

Let’s rewind to 2017. I grew up at the tail end of the Jabber era; my happiest memories of computer-aided socialization involve being signed in to Facebook, Google, and two other XMPP providers from one piece of software at the same time, while chatting about Supernatural. I hated Twitter, missed the culture of content warnings and near-normalization of transness and queer culture I’d experienced on Tumblr, and had a severe mental bias toward open protocols, so I jumped on Mastodon hard - but the dream of the Fediverse was never Mastodon-the-software taking over the social web.

The dream was being able to access the same social graph from any piece of software. Ideally, I would be able to use a Tumblr-alike interface to send posts that could be read by people using a Twitter-like one, and look at images posted by people using an Instagram-alike. The dream was a common underlying protocol used across a single unified network that would allow anyone to talk to anyone else using the software they were most comfortable using.

Fast forward to 2022 and that was exactly what we had - or, you know, we had the Dollar Tree version. Most of the software out there that wasn’t Mastodon was either alpha quality or developed by the worst guy you know. ⁵ None of it had any innovative moderation features worth talking about, and the Tumblr-alike interface never sprang up, despite my halfhearted attempts to get out-of-band tags into Mastodon-the-software. But we did have the basic gist: a big snuggly social fabric where anyone could talk to anyone else.

Fediverse as Ideoform

That’s where the Fediverse is still stuck in the public consciousness. Whether you call it “the Fediverse” or “Mastodon”, it’s that thing that you see when you log in to mastodon.social, or vulpine.club, or plural.cafe, or witches.live, weirder.earth, or rage.love, or tech.lgbt, or hachyderm.io, or pawoo.net.

The Fediverse is the hopes and dreams of naive kids like me, and of Eugen Rochko’s unified Twitter clone, and of Gab’s “freeze peach” brigade, and of a thousand server operators and six million active users. The Fediverse is a place, with a culture, and you can learn about it by observing it. I don’t just mean the phrase “the Fediverse” - there’s nothing wrong with it, except than that the thing it signifies isn’t meaningful anymore.

The Fediverse is a very particular way of thinking about federated social media that encompasses technical, social, cultural, and mythical elements - and that is exactly why I think it’s dead.

What Killed the Fediverse?

It turns out that the unified social fabric sucks. There are a lot of people out there that I don’t want to talk to, and lots of them don’t want to talk to me (though some really do.) There are people who post things that are illegal in my jurisdiction. There are people who harass people of color incessantly for the crime of daring to talk about their day to day lives, and people who just cannot wait to tell anyone who isn’t using their favorite flavor of open source desktop environment that they should die in a fire. ⁶

Communities as the Unit of Moderation

See, because you need an always-on computer in order to really reliably use decentralized social media, people tend to run Mastodon or other such software on computers in datacenters (“in the cloud”), and then log in from their personal devices, like logging in to check your e-mail. These servers are responsible for collecting messages for people and sending them back and forth - that’s “federation”.

So, naturally, the people who set these servers up had the ability to define some basic rules and community norms, and as Mastodon-the-software grew moderation features, it became much easier for those server operators to moderate their users’ experiences (or delegate that responsibility to a few other people.)

Mastodon, and soon other software as well, got the ability to exchange reports. If a user on one server had a problem with a user on another, not only could they complain to their server’s administrators, but to the people that provided the offending user access as well. Norms began to form around expectations of behavior, both from users and moderators on other servers, and some servers began to defederate - that is, stop talking to - others for various reasons.

I’ll be the first to admit that some of these reasons are, frankly, pretty silly - personal spats between admins, intentional escalations of intra-community arguments, and so forth - but others aren’t. After all, if you run an LGBTQ+ focused server, why would you let users on the FRC server talk to yours after the first instance of harassment? It’s not going to go well.

I would like to acknowledge the work of Black fedizen Marcia X in starting and stewarding #FediBlock, a hashtag that many admins subscribe to and to which many people post information about potentially problematic instances. It’s not a blocklist, but rather a loose standard and set of norms that enables people doing critical community work to coordinate and share information.

So, what was a unified social fabric very quickly became torn, with some servers totally isolating themselves and others simply cutting off contact with a few they found distasteful. Problems started to be obscured, circumscribed, and segregated.

What kind of problems, you ask?

Right Wing Extremism

There are a number of right-wing extremists, up to and including white nationalists and self-proclaimed neo-Nazis, who run Mastodon or Mastodon-compatible servers and communicate over them. I’ve never seen most of these servers, because the servers I use block them all and they tend to block us too (a rift I have in the past called the Great Fedi Divide), but they’re definitely out there.

“The Fediverse”, as an ideoform, has become tainted by extremism because extremists do use the software and protocols that it comprises.

Racism

Racism has been a problem in the GNUSocial-descended distributed social graph for a long time, and has been talked about extensively. From the demise of PlayVicious, ⁷ a Black-run server, to Black users being “asked” (harassed) about hiding their personal life experiences behind “politics” CWs, it’s undeniable that it’s a huge issue in even the non-right-wing communities.

A Quick Aside on CWs

Top-notch demoscener Blackle Mori implemented so-called “content warnings” for Mastodon-the-software in 2017. CWs allow the author of a post to hide it behind a <details> element so the post’s text and media don’t automatically show up; a single click is required to read the post. It’s also possible to set one’s Mastodon account to always, automatically expand content warnings. It looks kind of like this:
it’s been
ONE WEEK SINCE YOU LOOKED AT ME

People use CWs for media spoilers and for sexual, triggering, or upsetting content, but also for just about everything else including jokes and pranks, inconsistently and idiosyncratically. On other compatible software, it’s sometimes just called “subject”, as in the subject line of an e-mail.

I want to be extremely clear that it is my opinion is that the behavior of harassment and the cultural expectation that marginalized people hide their experiences behind “politics” CWs is the problem, not the use of CWs in the first place.

“The Fediverse”, as an ideoform, has become tainted by racism because people who use the software and protocols comprising it do racist things, and that software and the social structures surrounding it are bad at defending its users of color, and especially its Black users, against harassment.

Instance Drama

When users sign up for a fediverse server (also known as an “instance”, as in, “an instance of the Mastodon software” or “an instance of that dungeon”), they sign up for the operators’ moderation decisions. Sometimes, that means that two users will sign up on two different servers, intending to talk to each other, and at a later time, for reasons inscruitable to them, find themselves unable to do so. This has happened a few times, and it’s probably extremely frustrating every time. (It’s never happened to me, thankfully.)

“The Fediverse”, as an ideoform, has become tainted by instance drama because users of the software and protocols that comprise it sometimes lose access to parts of their social graph unexpectedly and without much explaination.

Misattribution of Malice

The problem is, these issues - and there are many others, believe me - are misattributed in two directions. First, outsiders look at “the Fediverse”, the holistic social fabric that connects all of its users, and see stories about horrible racism and people losing access to their friends and think that “the Fediverse” is racist and fragile. Second, insiders look at “the Fediverse” and see their curated timelines, their intact social connections, their diverse groups of contacts, and their meaningful relationships and think that “the Fediverse” can’t possibly be racist and fragile.

In other words, people on the outside attribute malice too broadly to the entire network, while people on the inside attribute malice too narrowly to a few individuals or, at most, a few servers.

I’ve certainly been guilty of the latter. While I’m too well aware of the racist harassment people of color face via distributed social media to discount it, I’m sure I don’t grasp its full extent, and my positive interactions via the exceptionally well-maintained social filters of Weirder Earth definitely skew my perception of “the Fediverse” towards the positive. Similarly, my Mastodon accounts have been more durable than any other social media I’ve had, and my social graph more durable still, so I have to put in some effort to empathise with people who have a worse experience.

As a result, conversation about “the Fediverse” is impossible. We can’t meaningfully argue over whether or not “the Fediverse” (or “Mastodon”-the-network) is racist; of course it is, of course it isn’t only. We can’t meaningfully argue over whether or not “the Fediverse” is fragile; of course it is, of course it isn’t only.

I think any distributed social media “insider” who can stand some discussion of child sexual abuse (yeah, sorry) should read this absolutely awful piece of journalism entitled “A Social Media Platform For Pedophiles”, which sums things up perfectly:

It doesn’t matter that you can’t see them, as a Mastodon user you are sharing a social media platform with them.

We Killed the Fediverse

We killed the Fediverse with bad language. Thinking back to my original definition at the top of this article, I hope it’s clear that “the Fediverse” is essentially a useless term:

the Fediverse refers to a bunch of web services which let people share text, images, video and audio with each other without a centralized authority.

What’s a “web service”? Is it the software that defines how people can post and how posts move around? Is it the servers running that software that people log into? Is it the communities that surround those servers, the people on them and the policies those people set? Is it the set of connections between those servers?

It seems clear to me that simply using the same social media software as someone doesn’t make you part of their social network. As an example, some Mastodon instances have a “allowlist”, meaning they don’t talk to any other servers by default. There’s very little sense in which those servers are part of my social network, because I can’t see or interact with their posts.

But what about if my server talks to another server that talks to your server? We can have mutual friends; my posts can make it to your server, and vice versa. Are we part of the same social network? Sure.

And what about three links removed? What if our servers could talk to each other - they don’t block each other, and neither one is in allowlist mode - but nobody on mine has ever reached out to anyone on yours, and vice versa? Technically, they have no knowledge of each other, but they could at any moment. Are we both in “the Fediverse”?

It’s a meaningless term, because the social graph is so complicated, it’s almost impossible to say whether a given server is “part of the Fediverse”.

Gargron Killed the Fediverse

Rochko (whose handle is Gargron) didn’t help. Branding the first server running Mastodon-the-software “Mastodon Dot Social” was a terrible idea, and branding Mastodon gGmbH as such and calling himself “CEO of Mastodon” in Forbes was even worse. That’s where we get misconceptions like the article above saying that everyone using Mastodon is using a social network for people who like to look at explicit images of children, even though the vast majority of Mastodon servers outside of jurisdictions where it’s legal don’t connect to the servers in question.

The Press Killed the Fediverse

That journalists don’t even try to sort through this mess is understandable, but extremely frustrating, especially when Vice and Forbes coverage is likely the first exposure many people will have to the concept. Even when the press does mention the concept of the Fediverse, they rarely try to define it, because it’s a useless, meaningless word.

We Have to Keep Going

So, what do I think we should do? You have no particular reason to listen to me, but here’s my advice.

“The Fediverse” needs to end, and I don’t think anything should replace it. Speak instead about communities, and prioritize the strength of those communities. Speak about the way those communities interact, and don’t; the way they form strands and islands and gulfs. I’ve taken to calling this the Social Archipelago.

This change is already becoming formalized. Some servers, especially those that are often blocked by others, have started pacts that aim to establish an island of continued communication by applying legalism and formal reviews to problems of interpersonal communication. ⁸ Some server software (such as Akkoma and Calkey) has explicit technical support for the idea of a “suggested timeline” or “bubble” of federated instances - a sort of halfway allowlisting that allows server operators to further curate their users’ experiences. ⁹

For those of us already using distributed social media:

Speak about being Octodon or Universedon or Hachyderm or Weirder Earth users, rather than Fediverse users. Take inspiration from communities like Merveilles Town that define a unique and cohesive local culture.
Have a relationship, even a superficial one, with your server’s operators. Understand their policies, their limits, and their needs. Donate to keep the lights on if you can.
Don’t pretend that every problem people discuss about “the Fediverse” only applies to other people and other servers.
Take reports of harassment and racism seriously, even if you can’t see it from your own vantage point. Combat racism within your own social circles and take the time to educate yourself on antiracism.

For current and prospective server operators:

Keep things small. Moderation is key, and community building gets harder the more people you have to include.
Accept the Archipelago. Don’t try to connect to every server there is; some of them definitely contain content you and your users don’t want to see, and many of them probably don’t want your users interacting with theirs. Have a set of policies about who you federate with, and stick to them.
Have a plan for burnout, technical problems, and interpersonal conflict. Have policies and funds to fall back on.
Build governance structures into your operations. Co-ops with tight-knit technical and moderation committies are probably ideal, but even something as simple as a succession document can save everyone a lot of trouble.
Ask your users for help! They’re part of your community, and their goals likely align strongly with yours.

And finally, for people on the outside:

Please don’t judge every island in the Social Archipelago by the bad things that have happened one other servers running the same genre of software.
Don’t act like harmless aspects of the culture of certain parts of the Social Archipelago - liberal use of CWs, for instance - are axiomatically bad because they come from “the Fediverse”.
If you don’t see a space in the Archipelago that looks good to you, consider making one. There’s some great starting advice at RunYourOwn.Social.
Give it a chance, or don’t. Sign up for a cool looking community with moderators you trust, or don’t. It’s not corporate social media; it’s not going to enclose your social graph in a proprietary shell you have to have an account on to access. That’s the whole point.

Ultimately, I’m still an idealist. I don’t think proprietary, centralized solutions - even cozy bug themed ones - are the answer to social media. I think that medium-sized co-op communities are as good as it gets for now, until we get easy to use, resiliant, secure peer to peer social media. I believe that progress can be made on racism and on stability in the Social Archipelago.

I believe in communities, and I believe in us. Let’s do it!

A Quick Guide to the Free Network ↩︎
Mastodon 0.1.0 ↩︎
I Left Mastodon Yesterday ↩︎
Mourning Mastodon ↩︎
This originally read “all the software”; this isn’t fair. There is some pretty good ActivityPub software out there, like Peertube. Also, being alpha quality isn’t a bad thing! GoToSocial is alpha, but I use it, and it’s nice. It just, you know, isn’t feature complete and definitely isn’t something to rely on. ↩︎
I use KDE Plasma, by the way. ↩︎
Recuerda PV, Pt 1 ↩︎
Such as the United Federation of Instances ↩︎
Thanks to @alcapurrias@seaside.cafe for pointing this out. ↩︎

Fallout: New Vegas Is Like a TTRPG With a Bad DM

Thu, 12 Jan 2023 20:09:56 -0600

I love Fallout: New Vegas. I keep going back to it, in a way that I don’t for Oblivion, or Skyrim, or Fallout 3. And yet, every time I sit down to play the game, whether reliving old memories or showing one of my favorite pieces of art to someone new, my enthusiasm quickly fades. For a long time, I had no idea why, but today I figured it out.

Minor spoilers ahead.

Nevada Route 164

See, Fallout: New Vegas, like all the Bethesda Fallout games, is set in a real place. The Mojave Outpost sits in Mountain Pass, CA, just southeast of Clark Mountain, where the Barstow Fwy becomes the Mojave Fwy. Primm, the Powder Ganger-afflicted resort town, is a conceptual fusion of the real world towns of Primm, NV and Jean, NV. I’ve driven the stretch of I-15 between Goodsprings and Mountain Pass, CA, several times over on the way to and from the DEF CON convention.

It’s a perfect place to set a video game. The huge elevation changes, the long stretch of empty desert from Mountain Pass up to Jean, and the encircling mountains make for impressive but constrained sight lines, as well as a multitude of foothills and vales to set side-quests and hidden locations in. Obsidian Entertainment, unsurprisingly, took full advantage of the landscape to set the first free-roaming section of the game.

A map of the Fallout: New Vegas playspace, made by Reddit user waiii23.

In most playthroughs, the player character will make their way from the sleepy town of Goodsprings, down the gangster- and scorpion-infested I-15, through the embattled town of Primm, and into the relative safety of the Mojave Outpost at Mountain Pass. There, the game begins to open up; several characters give the player quests, if they go looking for them; the collectable Star Bottle Caps appear for the first time; and there are even a few trading caravans and random encounters in the area in order to keep inquisitive players busy.

All of these signs, though, point toward a single path: Nevada Route 164 (in reality, Nipton Rd., which becomes Nevada Rt. 164 after passing through Nipton.) After Nipton, the player enters Nevada proper, and the game opens up completely. A normal playthrough could easily lead to Ranger Station Charlie, or Ranger Station Echo, or to fascist-occupied Cottonwood Cove. This stretch of road is the last place where Obsidian has a reasonable amount of narrative control over the player; there’s nowhere to go but east.

So what do they do with that control?

They throw two pointless, unavoidable combat encounters at you.

Killers with No Fixed Abode

Fallout: New Vegas is a lot like Dungeons and Dragons, and not just in the way most action RPGs are kinda, sorta like D&D. In fact, it’s an awful lot like the 3rd edition of D&D.

It’s got ability scores in the form of SPECIAL Attributes. These scores influence a large number of more specialized skills, which can each be individually increased. It’s got experience points which, once enough are accumulated, allow a player character to improve their skills and acquire unique abilities. It lets players choose which skills they change when they level up, unlike Skyrim’s skill training system. It’s got a universal currency, despite some trappings of state currencies, and unique, powerful items that can’t be purchased at any random vendor.

And, like Dungeons and Dragons, the most surefire way to get experience points, cash, and unique items is seeking out life-or-death fights and winning. In Dungeons and Dragons, taking this structure too seriously can lead to a problem commonly known as “murderhoboism” - player characters that wander the land looking for fights, just so they can get kills, gold, and magic items.

There’s nothing wrong with playing that way; it can be a lot of fun. But for players who value more character-focused games, which Fallout: New Vegas certainly strives to be, a good Dungeon Master will steer the players towards other methods of conflict resolution when possible. The best D&D games do include combat, of course, but in my experience they rarely involve unavoidable combat encounters that pop up out of the blue.¹

A Jackal Gang Member, on the stretch of I-15 heading from Goodsprings to the Mojave Outpost in Mountain Pass, CA.

Player characters in D&D often find themselves on that stretch of Rt. 164: on the way from one major location to another, having just healed up, filled up on minor magic items, and maybe even picked up a hireling or two. It’s absolutely reasonable to throw a few encounters at them on the way. For instance, a DM might put some highwaymen on the road between two towns. The player characters might hear rumors about them in town, if they ask around; either way, the players would be wise to have their characters look out for signs of an ambush. With high stats and good luck, they might notice fresh tracks, foliage out of place, or even something moving in the trees. If they notice the ambush, the player characters might be able to sneak past, or stage a diversion, especially if they use some of their resources, like a wizard’s spells or an invisibility potion.

Even if they can’t make it past undetected, mortal peril isn’t necessarily on the cards. The highwaymen aren’t murderers by trade, after all; they’re robbers! They’ll ambush the player characters, surround them, and ask them - what else? - “Your money or your life!” If the PCs pay up, nobody gets hurt. If they don’t want to pay (and who would!), it’s a great opportunity for roleplay. A brash, headstrong barbarian might want to fight, but be tempered by the wiser cleric or ranger. A city-bred noble might be unused to encountering such danger, and might simply try to command the bandits to stand down. A happy-go-lucky halfling bard might try to sway them with a story and a song.

If all of that is past and combat does break out, the player characters killing or seriously injuring a single highwayman will likely lead to a second parlay, and a chance to negotiate an end to hostilities. No matter what, it gives the DM a chance to make the world feel more real, and the players a chance to flex their roleplaying muscles.

That’s not a option in Fallout: New Vegas.

Zero to Climax in No Tension Flat

A proficient player, or one who has spent a lot of time and skill points on stealth, might be able to sneak past the Jackal Gang Members lying in wait on that stretch of the highway - but that means completely avoiding the content. So does simply leaving the highway, and that’s likely to lead to combat anyway.

Any engagement with the Jackals that Obsidian chose to place, in the ruined buildings they modelled and textured and designed into that part of the map, is a combat encounter. Any combat encounter is a fight to the death. That frustrates me.

It frustrates me because it’s just about the least effective way to build tension. An NPC tells you the road is dangerous; you walk down the road and maybe see some enemies; you fight them and either die or kill them all. That’s it. It’s throwing away perhaps a dozen opportunities to “fail forward”; to let the game ratchet up the tension, give the players the opportunity to fail and end up in a worse situation, and the opportunity to make the right choice and move forward.²

It would be easy to fix, too. The game already has a method of valuing items based on individual NPCs and player characters’ stats; why not just have a “robbery” version, where you’re asked at gunpoint to hand over a certain caps value of items? The gang members tell you to fork over the caps; you can try to talk them out of it, and if you fail, the trading menu pops up. If you can’t or won’t meet the minimum, it’s combat. Then, the first time one of the gangsters loses a limb or dies, it’s back to dialogue to negotiate for a lower price to end the fight, or try to convince them that you’re dangerous enough to leave you alone.

I’m not saying a fight to the death shouldn’t be on the table, but it’s much less satisfying to go from zero to sixty the way Bethesda games so often do, and in a game as narrative and character focused as New Vegas, it absolutely stands out.

The exception to this is spontaneous, unavoidable combat encounters that make sense in the narrative and exist to push the players into new ways of thinking or give them additional resources. Matt Colville talks about this in the video “Orcs Attack” ↩︎
Matt Colville talks about this in his excellent video “Many Fail States” ↩︎

Scaling Mastodon in the Face of an Exodus

Thu, 10 Nov 2022 11:15:18 -0600

TL;DR: Mastodon’s Sidekiq deferred execution jobs are the limiting factor for scaling federated traffic in a single-server or small cluster deployment. Sidekiq performance scales poorly under a single process model, and can be limited by database performance in a deployment of the default Dockerized configuration.

If you are an embattled instance admin, go here.

I recently moved to a well-established Mastodon (Hometown fork) server called weirder.earth, a wonderful community with several very dedicated moderators and longstanding links to other respected servers in the fediverse. Overall, it’s been lovely; the community is great, the Hometown fork has some nice ease-of-use features, and the moderators tend to squash spam and bigotry pretty quickly.

Then Elon Musk bought Twitter.

The Setup

Just as tens of thousands of people¹ fled Twitter to join the fediverse, Weirder Earth had a storage migration pending. It did not go well, for reasons unrelated to this post, and the instance was down for several hours.

Normally, this would not be a problem; ActivityPub², the protocol that powers Mastodon and other fediverse software, is very resilient to outages, and most implementations have plenty of built-in mechanisms like exponential backoff and so forth to get their neighbors up to date when they come back online. This particular outage, however, coincided with a huge amount of traffic, so when the server did come back online, there were many tens of thousands of messages waiting for it. Each of these messages necessitates a Sidekiq task which integrates it into the database, users’ timelines and notification lists, and so forth.

Sidekiq did not succeed in handling this load.

A screenshot from the Weirder Earth Sidekiq dashboard, showing processed and failed jobs over time. Created and edited by Packbats.

Users began to experience multi-hour delays in incoming federation, though outgoing federation was unaffected. To users of other instances, nothing was wrong; weirder.earth posts showed up within seconds, or about two minutes at the worst. Even to weirder.earth users, the web interface worked perfectly well; it’s just that there was little new content to populate it.

The Sidekiq queue grew and grew over time, eventually reaching a peak of over 200,000 queued jobs. In conversation with the admin of another instance of a roughly similar size, I learned that processing a 6-digit number of jobs is a day is only a fairly recent phenomenon. So, this was an issue.

The Cause

I became involved with this situation during the initial outage, helping with Docker problems, alongside another user, the Packbats,³ and another instance admin. As the Sidekiq queue grew and grew, myself and two of the instance’s admins looked into every possible cause, mostly sifting through documentation, since it was not an area of expertise for any of us.

It was immediately clear that at least part of the problem was Sidekiq’s inability to successfully use the 8GB of memory and 4 CPU cores available to the server; the RSS of the single Sidekiq process was around 350 MB, and the load factor was averaging around 3 – not even fully utilizing the 4 available cores.

So, we started increasing the number of workers available to Sidekiq. This was done by increasing MAX_THREADS and passing the -c (“concurrency”) argument to Sidekiq. (We later learned that DB_POOL is more appropriate; see below.) We increased first to 15, then to 25, and finally to 50, which is the mythical “stable limit” of Sidekiq according to several StackOverflow posts.

It helped, but the queue depth was still growing, so we increased our database container’s max connections to 200 (by adding -c ‘max_connections = 200’ to the invocation in docker-compose.yml) and increased to 150 Sidekiq threads. Again, this helped, but overnight the queue continued to grow.

At this point, I realized that the bottleneck was not in Sidekiq anymore; individual jobs had gone from completing in seconds to tens of seconds and even sometimes over a minute. One of the instance admins pulled up PgHero and we realized that, indeed, the database was being incredibly conservative with memory. By default, it was using only 128MB for shared buffers, which is dramatically low on a server with 4GB of memory and a workload like Mastodon, where a huge volume of traffic is just fetching the same few posts and conversations over and over again. So, we increased that to the aggressive value of 2GB, and again saw an improvement in performance. Rather than growing, the queue was now stable, at 194,000 events and a latent time of about 14 hours.

We had conquered the first derivative.

The Solution

Ultimately, the real solution as a combination of three things:

Increasing the memory available to the database for shared buffers
Increasing the number of threads available to Sidekiq (and, accordingly, the database connection pool size and maximum number of connections via DB_POOL and max_connections)
Splitting Sidekiq’s queues off into their own processes.

See, Mastodon uses 5 Sidekiq queues: default, push, pull, mailers, and scheduler. For us, push was conquered early on, and mailers and scheduler were never an issue, having very low volume. pull, for getting incoming federation events, and default, for building notifications and timelines, were the real trouble. So, we split them into their own Sidekiq processes.

We had gone up to 200 max_connections and a concurrency of 150, so I guesstimated that the following split would work well:

a single container/process for the push, mailers, and scheduler queues with 25 threads
a container/process each for pull and default with 60 threads each

This worked great for pull. We went from holding steady to dropping about 500 events every minute, and rapidly cleared the entire pull queue – but the default queue was still holding on. ⁴

This was harder to diagnose, but eventually we realized that the Sidekiq instance working on the default queue wasn’t at its full potential. Looking at the PgHero connection status dashboard, we could see that many of our 200 connections were held by inactive or low-activity Sidekiq processes with names like sidekiq 6.2.1 mastodon [0 of 25 busy] while the active ones were only holding a few connections. But that should be okay, right? 60 + 60 + 25 is much less than 200, so there should be plenty of connections to go around.

As it turned out, Puma was also configured to use lots of database connections (MAX_THREADS = 150), so it was hogging many that it didn’t need. We sorted that out, refined our Postgres config to allow even more connections, split the Sidekiq processes up even more, with fallbacks and various different ordered combinations of queues. We all went to sleep and woke up to an empty queue!

We’ve been experimenting and optimizing since, but the server has been coping with the load handily since.

Lessons Learned

The default Mastodon configuration is broken. It’s fine for a tiny instance on a tiny server, but once you start to grow, you must scale it up. Here’s what we did:

Increase Your Database’s Resources

By default, especially in the default Docker setup, Postgres is not using any appreciable percentage of your memory. I suggest using PgTune, choosing an “online transaction processing” or OLTP workload, setting about half of your server’s memory, and applying whatever it says.

In Docker, it’s easiest to apply this through adding -c "option=value" arguments to the invocation in docker-compose.yml; this can also work if you’re invoking Postgres through a SystemD unit. Or, you can use postgresql.conf. Don’t forget to restart the database.

Split Your Sidekiq Queues

Sidekiq can scale to as many as hundreds of threads on a single process, but using multiple processes is much more efficient. Before increasing concurrency (that’s the next step!), split out the queues into different processes.

All the queues in Mastodon can be split among multiple processes, except for scheduler, which must only ever be on a single process. I recommend having a single Sidekiq process each for the mailers and scheduler queues, and then processes that prioritize pull, push, and default. You can specify queues with the -q option in docker-compose.yml or your SystemD unit files.

Specifying multiple -q options gives Sidekiq a fallback order for working on jobs, so for instance -q default -q pull means that when all default jobs are done, that process will start working on pull jobs. Make sure that there is at least one process prioritizing every queue, or else processing that queue could stall entirely!

Because the queues you want to prioritize are default, push, and pull, it might be sensible to make processes with all the permutations of default, push, and pull:

default, push, pull
default, pull, push
pull, default, push
pull, push, default *
push, default, pull
push, pull, default *

However, in reality, push is generally much lower volume than the others, so it makes more sense to drop the number of queues to four, removing the fourth and final orderings (marked with *).

To do this in Docker, simply copy and paste the sidekiq service definition, change the name, and add arguments to the invocation.

For instance, we might go from this default definition:

sidekiq:
  build: .
  image: tootsuite/mastodon:v3.4.6
  restart: always
  env_file: .env.production
  command: bundle exec sidekiq
  depends_on:
    - db
    - redis
  networks:
    - external_network
    - internal_network
  volumes:
    - ./public/system:/mastodon/public/system
  healthcheck:
    test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]

To these definitions:

sidekiq-mailers:
  build: .
  image: tootsuite/mastodon:v3.4.6
  restart: always
  env_file: .env.production
  command: bundle exec sidekiq -q mailers -q scheduler
  depends_on:
    - db
    - redis
  networks:
    - external_network
    - internal_network
  volumes:
    - ./public/system:/mastodon/public/system
  healthcheck:
    test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]

sidekiq-scheduler:
  build: .
  image: tootsuite/mastodon:v3.4.6
  restart: always
  env_file: .env.production
  command: bundle exec sidekiq -q scheduler
  depends_on:
    - db
    - redis
  networks:
    - external_network
    - internal_network
  volumes:
    - ./public/system:/mastodon/public/system
  healthcheck:
    test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]

sidekiq-default-push-pull:
  build: .
  image: tootsuite/mastodon:v3.4.6
  restart: always
  env_file: .env.production
  command: bundle exec sidekiq -q default -q push -q pull
  depends_on:
    - db
    - redis
  networks:
    - external_network
    - internal_network
  volumes:
    - ./public/system:/mastodon/public/system
  healthcheck:
    test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]

sidekiq-default-pull-push:
  build: .
  image: tootsuite/mastodon:v3.4.6
  restart: always
  env_file: .env.production
  command: bundle exec sidekiq -q default -q pull -q push
  depends_on:
    - db
    - redis
  networks:
    - external_network
    - internal_network
  volumes:
    - ./public/system:/mastodon/public/system
  healthcheck:
    test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]

sidekiq-pull-default-push:
  build: .
  image: tootsuite/mastodon:v3.4.6
  restart: always
  env_file: .env.production
  command: bundle exec sidekiq -q pull -q default -q push
  depends_on:
    - db
    - redis
  networks:
    - external_network
    - internal_network
  volumes:
    - ./public/system:/mastodon/public/system
  healthcheck:
    test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]

sidekiq-push-default-pull:
  build: .
  image: tootsuite/mastodon:v3.4.6
  restart: always
  env_file: .env.production
  command: bundle exec sidekiq -q push -q default -q pull
  depends_on:
    - db
    - redis
  networks:
    - external_network
    - internal_network
  volumes:
    - ./public/system:/mastodon/public/system
  healthcheck:
    test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]

Fully Utilize Your Database

Now that it’s got more to work with, take note of the max_connections parameter you set for Postgres. Ideally, those connections would be mostly, but not completely, utilized. Database contention will slow you down.

The DB_POOL variable controls how many database connections a Ruby on Rails process will use. (MAX_THREADS controls this for Puma, the server used in web.) In Mastodon, the relevant Rails processes are in the Docker-Compose services web, streaming, and whatever Sidekiq services you’ve configured. In docker-compose.yml, you can change these variable by adding a section like the following to each relevant service:

environment:
  - DB_POOL=value

On sidekiq services, also add the -c option to the sidekiq invocation, with the same value as the DB_POOL variable.

command: bundle exec sidekiq -q default -q push -q pull -c 25
environment:
  - DB_POOL=25

In addition, the web service takes a variable called WEB_CONCURRENCY to control how many processes it runs. Similarly, streaming has STREAMING_CLUSTER_NUM to control the number of processes.

The sum of MAX_THREADS times WEB_CONCURRENCY in web, STREAMING_CLUSTER_NUM times DB_POOL in streaming, and all the sidekiq DB_POOL variables, must be less than max_connections in your Postgres config. If it’s more, you’ll experience database contention.

In the example above, assuming the rest of the configuration is default and you have 200 database connections available, I’d set the following:

web: MAX_THREADS = 10, WEB_CONCURRENCY=3 for 30 connections
streaming: STREAMING_CLUSTER_NUM = 3, DB_POOL = 15 for 45 connections
sidekiq-default-push-pull: DB_POOL = 25, -c 25 for 25 connections
sidekiq-default-pull-push: DB_POOL = 25, -c 25 for 25 connections
sidekiq-pull-default-push: DB_POOL = 25, -c 25 for 25 connections
sidekiq-push-default-pull: DB_POOL = 25, -c 25 for 25 connections
sidekiq-push-scheduler: DB_POOL = 5, -c 5 for 5 connections
sidekiq-push-mailers: DB_POOL = 5, -c 5 for 5 connections

For a sum of 185 connections. This means there will be 15 loose database connections for things like migrations and manually connecting to the database to do queries and maintenance.

Mastodon 4.0

Weirder Earth does not use Mastodon 4.0, but if you do, you will also have to consider the ingress queue.

Into the Future

These numbers may not work perfectly for you, and you may have to tweak things as workloads change.

There is more work left to do, and we’ll only know if this is successful once the next big wave hits, but it’s definitely better than it was before. We also have not figured out how to monitor the instance for future events, though the top candidate is, of course, Prometheus and Grafana.

Good luck. May your timelines run smoothly and your processes never be OOM killed.

Wilfred Chan, “Mastodon gained 70,000 users after Musk’s Twitter takeover. I joined them”, The Guardian (2022 11 02). ↩︎
ActivityPub is a W3C Recommendation. See https://activitypub.rocks/ ↩︎
@packbat@weirder.earth ↩︎
I also typo’d mailers as mailer, which lead to some stalled emails. Don’t do that. ↩︎

It's Time to Get Hyped About Const Generics in Rust

Sat, 06 Nov 2021 00:00:00 +0000

One of the most interesting emergent properties of the Rust language has been the desire and ability to push more and more information into the type system. Other programming languages allow this, but Rust’s combination of speed, safety, and extremely powerful compile-time computation through both types and macros, have lead to a large ecosystem of highly capable libraries that do much of their work before the program ever runs.

Recent developments in the type system are making it easier than ever for programmers to take advantage of that power, and things are only going to get more interesting.

Constant Generic Parameters

In Rust 1.51, the “const generics MVP” was stabilized into the language, enabling tons of API and performance improvements. These so-called “const generics”, or “constant generic parameters”, permit values of integral types to used as parameters to generic types, traits, and functions, like this example from my nslice crate:

/// A region of memory containing at least `N` `T`s.
pub struct MinSlice<T, const N: usize> {
    /// The bounded region of memory. Exactly `N` `T`s.
    pub head: [T; N],
    /// Zero or more remaining `T`s after the `N` in the bounded region.
    pub tail: [T],
}

Here, T is a normal type parameter, but N is a constant generic parameter, meaning that the minimum number of elements in the MinSlice is encoded in its type; a MinSlice<_, 16> isn’t the same as a MinSlice<_, 32>, for instance.

Const Generics Reduce Runtime Complexity

This is great, and lets us avoid some runtime checks. Whereas a normal slice means getting an Option (or a possible panic) every time we read from it, even if we know the size:

let slice: &[u8] = b"Hello, world";
let reference: Option<&u8> = slice.get(6);
// We know this value is `Some(b' ')`,
// but the compiler can't know that.
assert!(reference.is_some())

With a MinSlice, we can perform the length check one time and get a compile-time guarantee that future accesses will succeed:

let slice: &[u8] = b"Hello, world";
// Length check is performed when we construct a MinSlice,
// and it's known at compile time to be of length 12.
// If the `unwrap()` succeeds, no more checks are needed
// throughout the `MinSlice`'s lifetime.
let minslice = MinSlice::<u8, 12>::from_slice(slice).unwrap();
let value: u8 = minslice.head[6];
assert_eq!(value, b' ')

This is powerful, but the current implementation of const generics is quite limited, on purpose. There are remaining design questions around a lot of aspects of the world of type-level computation in Rust, such as error handling and even syntax for default values.

Generic Constant Expressions

One important feature that’s not yet available on stable is the ability to do computations on constant generics at compile time, beyond very simple combinations of literals. That is, you can write MinSlice::<_, 3 + 4>, but you can’t (yet) write something like:

impl <T, const N: usize> MinSlice<T, N> {
    /// Returns a longer `MinSlice` if enough elements
    /// are available in `tail`.
    fn <const M: usize> extend(&self) -> Option(&MinSlice<T, N + M>) 
}

Arithmetic and comparisons that work with literals aren’t available with constant generic parameters.

All of that changes, however, when we enable the feature generic_const_exprs. This feature still has many design questions, and likely won’t be stabilized for a long time, but when it is, we’ll be able to put a great deal of information in the type system that could previously be computed only at runtime.

Conditional Implementation

Obviously computing the lengths of arrays at compile time is convenient, but we can do even better. For example, Rust already has a way to conditionally implement traits on types; it’s just that the only condition available is whether that type implements another trait already.

impl<T> MyTrait for MyType<T> where MyType<T>: Copy {}

We can even make chains of inference using generic parameters, like this:

impl<T> Copy for MyType<T> where T: Copy {}

Now, whenever we construct a MyType<T> for a T that is Copy, that automatically implements both Copy and MyTrait. So, MyType<u32> is Copy and MyTrait, but MyType<String> is neither.

Const Implementation Bounds

Let’s say we’re working on a piece of embedded software running in a real-time environment, and it’s become clear that cloning values that are too large could lead to violations of our real-time constraints. That is, taking the time to loop through and copy 32 or 64 values is fine, but if we’re spending time copying 128 values from place to place, we’ll infringe on our sensor polling or something.

(Yeah, it’s contrived - but it’s just an example.)

Ideally, we’d be able to enforce this at compile time, and indeed with just a little bit of infrastructure, we can conditionally implement traits based on the values of compile-time expressions encoded entirely in the type system.

Specifically, we need a trait that is implemented only if some condition is met. Enter Assert and IsTrue:

struct Assert<const COND: bool> {}

trait IsTrue {}

impl IsTrue for Assert<true> {}

These empty types and traits, of course, vanish during compilation, but they do give us enough infrastructure to make some compile-time assertions:

let _: Assert<false> = Assert::<{10 > 11}> {};

If all of our data is wrapped in a custom type, like this:

/// The struct we're going to conditionally implement for
#[derive(Clone, Debug)]
struct Foo<const N: usize>{ inner: [usize; N] }

We can then go ahead and make a conditional implementation based on size:

impl<const N: usize> Copy for Foo<N> where Assert::<{N < 128}>: IsTrue {}

Now, it’s ergonomic to copy around arrays of any size up to 127, like so:

let my_foo = Foo::<16> { inner: [0; 16] };
let foo_too = my_foo;  
println!("{:?}", foo_too);
do_something(my_foo);

But, as soon as they get too big, the compiler will stop us:

let my_copynt_foo = Foo::<128> { inner: [0; 128] };
let foo_moved = my_copynt_foo;
println!("{:?}", my_copynt_foo);
// ERROR: borrow of moved value
// move occurs because `my_copynt_foo` has type `Foo<128_usize>`,
// which does not implement the `Copy` trait

This is a fairly ergonomic form of almost-not-quite dependent types, and it’s all encoded in the familiar language of traits and generics that Rustaceans use daily.

Moving Forward

As I mentioned, there are many hurdles ahead in the implementation and stabilization of these features in the language, and their subsequent use in the library ecosystem. It’s also worth mentioning that there is already a library that provides many of these features on stable, called typenum. It’s a useful library if you need these features today, but it inherently has poor compile times and is somewhat limited in terms of its integration into the compiler and overall ecosystem in a way that first-class const generics are not.

Nonetheless, I think it’s time to get excited about constant generics in Rust; start figuring out how you want to use them and how they can help you solve problems, and give feedback to the Rust project about what you’d like to see as the full-fledged constant generic implementation begins to unfold.

Modular Synthesis and UNIX

Sat, 01 Aug 2020 11:06:41 -0500

I’ve always loved strange and interesting music and sounds. For the last few months, I’ve been assembling a collection of patch cables and circuit boards on my desk which, when connected just right, can create anything from melodic sequences to fast, driving beats to wacky electronic soundscapes.

Mescaline at the Theme Park by Daughter of Eris

This is a modular synthesizer in the Eurorack format. Eurorack is a de-facto standard created by German synthesizer manufacturer Doepfer designed around the concept of voltage control. Levels of voltage between 10 and -10 volts, or changes in voltage, represent and control everything: audio, pitch, speed, slope, and even color.

As I learn more about Eurorack and the modular ecosystem, it reminds me more and more of the magical feeling of working with UNIX tools: it is a world that exchanges programs for filters, oscillators, and VCAs, text for voltage, and pipes for patch cables.

Text, Audio, and Control Voltage

We’re all used to voltage representing audio. An “AUX cord”, correctly called a 3.5mm TRS (tip, ring, and sleeve) stereo cable¹, carries two channels of voltage and a common ground between audio devices through a nearly universal² standard. For audio signals, this voltage jumps between a maximum and minimum voltage. Its value over time is eventually turned into the movement of a speaker or headphone diaphragm, which in turn moves the air and can be heard.

In modular synthesis, though, these analog signals don’t just represent the movements of air that our ears hear as audio. We also use them to control other parameters: pitch, for example. Like the text streams that UNIX tools use to move around data and text, or the JSON payloads that power a huge portion of the modern web, voltage-based communication is simple and versatile.

The micro-Ornament and Crime module displaying changing pitch values, as represented by the voltage coming into its CV1 input.

These voltages are, by their nature, analog - there are infinitely many values between C4 and D4, or even between C4 and C#4. Among other things, this means that complex microtonal music is not just possible, but just as easy for the hardware to produce as music in any Western scale.

Text, of course, isn’t analog, but it does have similar characteristics. It’s not always easy to parse or generate exactly the right text, but with enough information, it’s always possible to interoperate with something that uses a text interface, or even JSON structured data.

Oscillators, Filters, `grep`, and `awk`

The traditional synthesizer, as invented by Bob Moog and co., produces sound through a so-called subtractive process. Sound, some kind of complex wave-form, is produced by an oscillator, according to control voltage representing pitch. That waveform, which has lots of harmonics, is put through an amplifier which turns the sound on when a note is being played, and off when it is not, and through a filter, which removes some harmonics and emphasizes others.

Just one module, removed from my synthesizer. This is a four-oscillator retro digital voice, which produces three channels of square waves and one “complex” waveform (noise, stepped triangle, etc.)

Each voice has four voltage inputs and one output, along with two buttons and two knobs. The module also provides a “mixed” output.

Of course, this is not so different from a standard UNIX log mangling pipeline, where cat grabs some output from a file, awk formats it in some easy-to-work-with way, grep cuts out irrelevant information, and head, tail, or a second awk invocation formats the output. This is the versatility of universal, standard interfaces like text, JSON, or voltage; each tool does what it does, and interoperates cleanly with all the others.

Envelopes, Signal Flow, and `mkfifo`

In my rack, I have a simple signal flow mapped out by the sequence of modules. My Winterbloom Sol turns digital signals into control voltage. The Kinks, Pique and Ornament and Crime modules shape that CV. Then come two oscillators, a filter, a set of amplifiers, and some effects.

But that’s not necessarily the order I actually patch the signal chain.

Most synths have an “envelope” control, sometimes controlling the level of a signal, and sometimes controlling a filter. The magic of modular is that each of those components is discrete. One module generates control voltage every time a note is played. Another module (a voltage controlled amplifier or VCA) uses that control voltage to change the volume of the note, turning a rising voltage into attack and falling voltage into decay. A third module (a voltage controlled filter) uses that control voltage to change the harmonic content of the sound. Just by moving a cable from the “FM” input to the “Q” input, that filter can go from subtracting harmonics to emphasizing them, or maybe both!

This is the flexibility of modular synthesis. Every signal exists within a constrained realm, and every module manufacturer knows what their interface should expect. Each module does some transformation - in a sense, every module represents a mathematical funtion, transforming voltage-over-time and knob positions into voltage-over-time, which can itself be transformed by further modules.

This is not something we see often in UNIX programming - but it is possible! With mkfifo, we can create more complex data flows and unique ways of moving information around, using multiple FIFOs as input and output files. In web systems, this kind of many-way connection is very common; even relatively simple systems like If This Then That allow users to create long, wide, branching data flows, and even flows that contain cycles. Ideally, each program or API does work like a mathematical function, taking command line arguments and input and transforming them into output and perhaps some side effects, like making a Mastodon post or writing to disk or a database.

Modularity, Open Source, and Market Forces

In the forward to The UNIX Time-Sharing System³, the seminal 1978 paper on UNIX, Doug McIlroy articulates the philosophy of modularity thus:

Make each program do one thing well. To do a new job, build afresh.

Expect the output of every program to become the input of another, unknown, program.

Design and build software […] to be tried early, ideally within weeks. ⁴

These are the same values that motivate the design of the best Eurorack modules. Each module is easy to experiment with, designed to accomplish one or a few tasks, explicitly built to integrate with other modules, and well-documented.

Eurorack’s modularity works well because there is no incentive to lock in users - indeed, there is a significant disincentive, since no one small company could ever hope to produce a universal product line. Building modules that work well in as many systems as possible is the best way to maximize sales.

My whole synthesizer is controlled by this Beatstep Pro, and optionally by Linux software like 100r ORCA.

Indeed, even in the cases where larger companies (like Intellijel) have tried to invent their own standards to get at least some degree of lock-in revenue, the size of the market limits the effectiveness of that strategy. Their smaller, 1U form factor has been widely adopted, and there are now nearly a dozen small manufacturers making compatible modules.

We, as software engineers, should strive to build software that is amenable to interoperability, because that is the most effective way to support creativity. Documented, open, and capable APIs are good, if your software must be centralized; federation through open protocols is even better.

Critically, your interface does not need to be optimal. Control voltage in 1v/Oct certainly isn’t; it requires difficult and imprecise exponential conversion circuits, or slow and expensive floating point computations. Consider, for instance, ActivityPub and the Mastodon ecosystem. Eugen Rochko could have easily stuck with OStatus, which does have some advantages, but the Mastodon project’s move to ActivityPub catalzyed an entire ecosystem, with dozens of other microblogging products, as well as blogs, video sharing, music sharing, and photo sharing software springing up, all able to interoperate at least somewhat.

Modularity also promotes longevity. The predecessor to ls was written in 1961⁵; the GNU find command has history as far back as 1987⁶. These programs are so deeply embedded in the brain stem of computing that building a modern server operating system without something compatible is nearly unthinkable.

Every module in this photograph is open source - software and hardware.

I should also point out that, while free and open source software is, in my opinion, a moral good, software freedom and open protocols/interfaces are tangential concerns. While many Eurorack modules are open source, some of the most popular are not; for example, Make Noise’s analog computer Maths is not an open source design, but it is the heart of many modular systems because it integrates well into the ecosystem and has a clearly documented and well-specified interface.

Amazon’s Simple Storage Service, for instance, is a proprietary service with an open protocol, and it is hugely successful - so much so that many projects have sprung up to implement its interface, and a huge volume of cloud software is designed to work with “S3 - and other stuff that works the same.”

Modularity is the most direct path to innovation. As software engineers, we should do our absolute best, both technically and politically, to build programs that do one thing well, programs that can be tried early, and programs that work well together. Anything else guarantees stagnation, lock-in, frustration, and eventual obscurity.

In Eurorack, most connections are made with mono, rather than stereo, cables, which are called TS (tip and sleeve) as they omit the additional contact for a second audio channel. Anyone who plays guitar will be familiar with TS cables, though probably in the 1/4th inch / 7mm format used by amplifiers and lots of other audio equipment. ↩︎
Thanks, Apple and Samsung, for doing your damndest to kill it. ↩︎
https://archive.org/details/bstj57-6-1899/page/n3/mode/2up ↩︎
Yes, McIlroy prefigured “agile” development in 1978. Agile development is, so to speak, not a new development, unless you think of software as proprietary, monolithic, and restricted to a priestly class. ↩︎
https://www.tldp.org/LDP/LG/issue48/fischer.html ↩︎
https://git.savannah.gnu.org/cgit/findutils.git/commit/?id=c030b5ee33bbec3c93cddc3ca9ebec14c24dbe07 ↩︎

Back to Zsh

Tue, 14 Jul 2020 09:36:23 -0500

I’ve gone back to using zsh¹ rather than fish² as my primary shell, for the same reason I use Vim over Emacs - zsh is lightweight, compatible, present, and (with Oh My Zsh!³ and Starship⁴) very featureful.

First of all, zsh feels snappy. To be totally fair to fish, which is a wonderful project, I did not objectively measure this, but subjectively, zsh starts faster than fish. I tried using zsh without any extensions for a while, but found that a lot of fish features were missing, and I was a lot less productive. In the end, I wound up installing Oh My Zsh! and Starship, but even with the added burden of all this code, tapping Meta+Enter to pull up a terminal feels snappier with zsh.

The big downside of fish is that it’s not anywhere close to POSIX compliant. Of course, that’s one of its stated goals - to build a new kind of shell - but, nonetheless, it really hampers its utility in a lot of situations. While compatibility layers do exist, I found them fiddly at best and hopelessly broken at worst. zsh, on the other hand, does a great job running all the bash and sh scripts and snippets I use to configure machines, most of which I didn’t write, and some of which have been around for nearly a decade.

Of course, it would be possible to take the time to write all of these scripts in fish, or even just port them to nice self-contained shell scripts that can be run using the appropriate shell. On the other hand, this code is usually run at most once every few months, and generally much less. As this excellent comic demonstrates, it’s probably not worth my time.

Presence is something of the elephant in the room - presence, specifically, on Mac OS. I’m beginning a position with CancerIQ, Inc. tomorrow, and their development environment is standardized on Mac OS. I’ve used Apple’s operating system professionally before, but that was back when I, and they, were still blissfully unaware that there were any alternatives to good old⁵ bash.

With the introduction of Mac OS Catalina, Apple’s default shell is now zsh. ⁶ This means that, with my GNU stow-based dotfiles,⁷ complete environment configuration is one simple git clone away. zsh is also available on pretty much every Linux system I’ve ever used, and works alright in WSL.

Of course, zsh is not as featureful as fish - at least not out of the box. With the addition of Oh My Zsh! and Starship, however, it’s both beautiful and information-dense.

For me, case- and hyphen-insensitive tab completion and a context-aware prompt are the most critical components of a productive shell environment. While Oh My Zsh! can provide both, I prefer Starship’s aesthetic and functionality. It’s also extremely fast, taking only about ten milliseconds to generate a prompt in even the most complex environment (git detached head with multiple language environments).

While I loved fish, I feel that zsh provides most of the features I used while being much less in my way - it’s a fast, compatible shell that’s available where I need it and does what I want.

Names, Legal Names, and Fractally Deferred Responsibility

Mon, 11 May 2020 00:05:15 -0500

A person’s “preferred name” is their name, their “legal name” is something else, and it’s our duty as software engineers not to muddle the two up.

That means software should avoid keeping track of users’ legal names, and software that must record legal names should have a “name” field and a “legal name” field, rather than a “name” field meant for a legal name and a “preferred name” field.

Names

I suspect few would disagree that a name is a collection of characters¹ which represents a collection of sounds,² which people use to refer to an individual in a particular context.

That statement is worded very deliberately. People use names to refer to other people; yes, organizations have names, but those are almost always much more fixed and less commonly abbreviated. Elizabeth may be Lizzie, Liz, and Ella to different friends and relatives, and of course William can be Will, Willy, Bill, or Billy, but it’s a rare company that gets such nicknames, beyond initilisms.

Names are both sound and text; few people have names that are only written, or only spoken. And the purpose of a name is to refer to an individual within a given context. You may be Danny to your buds at the bar and Mister Khan at the office; those are both “your name.”

Experience shows that names are not unique, and are not guaranteed to follow any specific format, to be of any minimum or maximum length, to be the same for one person throughout their life, or even to exist for each person you (or your software) will encounter.³

Despite this belabourment, names, to the software engineer, are very simple. A name is a collection of characters that identifies a person. It almost certainly doesn’t identify them uniquely⁴, and importantly, it need not bear any relation to their “legal name”. As software engineers, we are descendants of a proud⁵ heritage⁶ of hackers who often take great pleasure in assigning and using names, called “handles”, which are completely unrelated to their given or legal names.⁷

Legal Names

Nonetheless, many national, regional, and local legal systems make some assumptions about names. For example, in the United States, the Social Security Administration assumes that no name will be longer than 32 characters - two rows of 16. Most critically, many legal systems, and thus the bureaucracies attached to them, require that those who interact with them have a name, generally a first and last name, and often no more than a single middle name.⁸

Legal names are complex entities that live in a strange world of convoluted, arbitrary, and often contradicting rules and data formats. They are exactly the kind of things programmers should stay away from as much as possible.

Names and Software Extension

By way of example, let us examine the situations in which I come into contact with my “legal name”. My legal first (or given) name is “Leonora” (lay-oh-no-rah), but nobody really calls me that; I go by “Nora” (no-rah) online, to my family and friends, to my colleagues, and in most contexts in which a human is directly interacting with me.

My college has an ERP system which it uses to, among other things, relate students’ billing profiles to their meal plans. This is, of course, a critical function, as it allows us to eat on campus. The ERP system has a “preferred name” field, but it was added after the critical integrations were made. Therefore, I am “Leonora” to residential life, my professors, and the person who swipes my ID at the student commons.

In my case, this isn’t a huge problem, but it serves to illustrate a point: when a structured collection of data includes a “name” field, it is more likely than not that future developers - either internal to your organization, or external developers writing code that interfaces with your system - will use that field to identify users, preferentially even to other “name-like” fields (such as “preferred name” or “nickname”.)

Harmful Uses of Legal Names

Hearing my full name called out in roll call, or being addressed strangely at the door to the dining hall, isn’t a terrible burden. There are, however, larger consequences of the neglect of the “preferred name” field.

For instance, many international students choose to adopt an Americanized name to avoid the embarrassment and inconvenience of hearing every professor and staff member in the school mispronounce or outright skip over their name. Transgender people often have to interact with systems that use their legal name unnecessarily, causing distress and sometimes outing them to people to whom they would otherwise not be interested in discussing such a personal aspect of their lives.

Changing one’s legal name, except during marriage, is a long, difficult, and expensive process. I legally changed my name last year, and it cost me over $500; depending on the jurisdiction and the situation, name changes can require months of newspaper listings, thousands of dollars, and even court appearances.

Proving that one’s legal name has changed is also difficult. Even in cases where software systems are set up to accept legal name changes, there is almost always a human in the loop somewhere verifying that the documents submitted by the user are valid. This is harmful to the user, who must sometimes even appear in person to prove their name change, and to the operators of the software, who must dedicate personnel to the mind-numbing task of verifying court orders.

As I mentioned, I changed my name last year. I am still finding systems in which my name hasn’t been updated; I’ve long surpassed the commonly-cited 100 hour mark,⁹ and I’m sure I’ve cost at least that much time in labor from other institutions.

Fractal Deferment of Responsiblity

I was prompted to write this article by the experience of a friend of mine who was somewhat embarrassed by having his full, legal name called out in front of a room full of people who only knew him by another name. The was the result of a software system suffering from precisely the ailment I described: someone used the “name” field for the name displayed to kitchen workers who were preparing an order. There was no need for his legal name to be used there, but either the integration was made before the “preferred name” field was added, or the developer of the integration simply didn’t think to use it.

Upon asking about why that name had been used, he was told that “they just used what was in the system.” And, of course, that’s true! It’s not their fault. In fact, there was nothing they could do about it. There’s probably nothing their boss could do about it, either, or his boss, or the CIO of the college, or the President of the college.

“That’s just how the software works.”

For another example, my email address uses the fancy new “.codes” top level domain. Many old software systems don’t think it’s valid, which makes it somewhat hard for me to sign up for things. This happened with my health insurance company, and I had to spend three hours on the phone to get a complaint registered. They special-cased my account after a month; I still don’t think they’ve fixed the system.

We, as software engineers, build software that is inscrutable to the majority of our own, let alone people who don’t work with computers. Even open source software is difficult to change on a moment’s notice, and I would bet my bottom dollar that this software was not open source.

So, people will fractally defer their responsibility. It’s not the line workers’ fault. It’s not their boss’s fault. It’s not the fault of the person who procured the software, or the manager of the engineers, and if you ever make it this far down the chain, I’ll give you even odds that the engineer who made the decision has left for greener pastures.

We have to get things like this right, or they’ll never get fixed, because they’re nobody’s responsibility.

So, returning to names: what should we do?

Do Not Record Legal Names (Unless You Have To)

My advice is to save yourself and your organization the trouble. If you’re not paying people or interacting with ICANN, the NHS,¹⁰ or the Selective Service System, it’s unlikely that you need to record legal names.

If people are paying you, you’re almost certainly recording a billing address for each transaction; let them enter a payment name there. That’s likely to be useful for a variety of reasons, from corporate reimbursement to parental charity. In any case, it’s possible that their “legal name” isn’t the name on their credit card or bank account, if, for instance, they recently changed it;¹¹ if your software can’t account for that, it’s not just annoying, it’s incorrect.

If you create a form with a “name” field, label it somehow with the information that it isn’t asking for a “legal name”. Many websites with forms have hover-based explanatory text. This is a great place to put something like, “Whatever most people call you,” “What you’d like us to call you,” or, if you’re particularly stuffy, “Legal name not required.”

Names Aren’t Legal Names

If you do find yourself needing to record legal names, those shouldn’t be the main way your system refers to your users. Give them a “Name” and “Legal Name (if different)” field on signup forms, or let them uncheck a “This is my legal name” checkbox if you feel like that adds too much clutter. Then let them change their name any time, and only use the “legal name” field when you need to.

Names Are Mutable

Furthermore, the “name” field should be trivial for a user to change.¹² Becuase a “legal name” field can be used when official documents are involved, there are few ramifications that won’t be controlled by existing social structures. This means using something else as a unique key, like a UUID or even a simple serial index. Many operating systems, for instance, are guilty of making it very difficult to change a username.

I promise you that your users with long names, your transgender users, and your users who are getting ready to get married, or divorced, or to leave abusive living situations and want to leave their old name behind¹³, will thank you. And so will your customer success team, because they won’t have to process stacks of name change papers!

Most critically, though, be clear about how information will be used, especially names, at the time it is collected. If you’re asking for my name on this particular form so you can give me a W2, say, “This is the name we’ll put on your W2.” If you’re asking my what my name is so you can strip everything but the first cluster of letters and put it in your newsletter, say so!

As several commenters pointed out, it’s not just letters, as Mrs. O’Leary or X Æ A-12 Musk will tell you. ↩︎
It is possible that, in our modern society, the letters come before the sounds, and the sounds represent the letters. I’ll leave that one to the philosophers and linguists to hash out. ↩︎
“Falsehoods Programmers Believe About Names”, https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ ↩︎
When we allow a user to choose a name-like unique identifier, it’s often called a username, a handle, or an “at” (from the Twitter convention of using @whatever to signify that “whatever” is inline metadata referring to a user by username.) This is not a name. ↩︎
Perhaps too proud, at times. ↩︎
The prime history of which is, sadly, now maintained by Eric Raymond. ↩︎
See The New Hacker Dictionary, “handle”. ↩︎
I have two middle names. While my Social Security card lists part of my second middle name, and my California ID card lists both in their entirety, I rarely see my second middle name survive the machinations of bureaucracy. ↩︎
https://medium.com/bits-and-behavior/100-hours-of-name-change-labor-c652c22a89b9 ↩︎
In the UK, there isn’t a single concept of a “legal name”, and indeed, a lot of software that makes this assumption is guilty not only of being annoying and incorrect but also US-centric. The NHS, however, does require an “official name” which can be verified against birth date, NHS ID, and other information. ↩︎
Contrary to what many told me, I’ve been assured by the court that changed my name that this is perfectly legal, at least in California. You’re under no obligation to tell anyone in particular about your name change. ↩︎
Hacker News user jcrawfodor rightly suggested that this point be discussed more directly. It’s very important. ↩︎
Some commentary brought up the possibility of abuse of name-change mechanisms. While I think it’s possible that name changes could be abused to, e.g., create offensive-sounding names, this strikes me as rare and easy to detect - and if changing a name isn’t a big deal in your system, someone can just change it back. Furthermore, a lot of of people do actually have names that you might consider to be “offensive”. After the initial publication of this article, someone with the surname “Hooker” contacted me to tell me that he had been denied access to systems in the past because people assumed his name was an attempt to insert offensive content. ↩︎

Stop Making Students Use Eclipse

Sat, 11 Apr 2020 00:00:00 +0000

Using Java or Python in a professional IDE like IntelliJ IDEA, NetBeans, PyCharm, or Eclipse is not a good first introduction to programming for computer science students, whether they’re in the field to become web developers, systems software engineers, or academic computer science researchers.

Some IDEs, such as jGRASP,¹ provide education-specific tooling, like the “automatic generation of visualizations for improving the comprehensibility of the software”; that is, when you write a linked list and run your code in jGRASP, you see a diagram of a linked list on the screen.

jGRASP, an education-focused IDE, can be used on its own or, as here, integrated with IntelliJ IDEA.

Most IDEs, though, primarily serve two purposes for students. First, in a Java-focused curriculum, it insulates the student from the javac command line program, and the command line environment itself. Second, it catches some basic mistakes and allows the student to defer learning about the finnicky language requirements that aren’t deemed core to the curriculum, like imports and file naming requirements.

Crucially, I do believe that these are real problems. A student who has not written an if statement doesn’t need to understand the philosophy behind putting each public class in its own file, or what public or “class” even means.²

What’s the deal with these IDEs?

The whole purpose of an IDE is to provide an integrated environment - it’s even in the name. It also provides a homogeneous environment for the instructor to instruct on; rather than teaching students how to install Java and a platform-specific programmer’s editor on each platform, they can just say “install this IDE, open it, and click New Project.” This is valuable in an introductory course, as it avoids wasting class time and lowers the barrier to entry, but it’s crucial to introduce these inconvenient details eventually.

In my experience as a student, after the first or second year of CS classes, students tend to grasp the basic control structures. They can write branches and loops and functions and use basic data structures like lists and maps. The ACM’s 2013 guidance on CS curricula agrees with me here; they recommend a focus on basic structures of programming languages and problem solving.

What they can’t do, unless they’ve figured it out on their own, is operate a computer outside of the confines of the IDE they’ve been taught. In none of the curricula I’ve seen, through personal experience or reading syllabi provided by other students, is there a place for students to get past the myriad of barriers that constitute the use of a computer in the modern day.

Students who use Windows aren’t taught that, while their file system is case-insensitive, not all filesystems are. They probably aren’t taught that a “file system” is a concept until a 300-level operating systems course. Students who use Mac OS aren’t taught what the .DS_Store directory is, or why it’s irrelevant to their project submissions. Students learning Java don’t know that javac is their compiler and java their virtual machine, at least until they take a course in compilers. Nobody focuses on things like ASCII, Unicode, and UTF-8, or on how programs interoperate, or on how to share and distribute programs that students write.

Introductory CS curricula focus on abstract ideas of programming, and use IDEs to accomplish that.

Why is this a problem?

So, okay, students are being taught how to write code and solve problems with math instead of grapple with the intricate details of the eldrich monstrosities that are modern operating systems. Surely that’s a good thing?

The issue is, as Kevlin Henney is fond of saying, “Software is nothing but the details.” When students don’t understand what a file is, or haven’t ever edited text in anything but Microsoft Word and don’t realize they can edit code outside of an IDE³, they will not be able to do the crucial work of self-directed learning that is a hallmark of all computer science success. When students have only ever programmed in Java using some bespoke learning library provided by their professor, it will take them much longer than necessary to figure out other languages, other libraries, and other approaches. In a field that moves as fast as this one does, that’s a very serious problem.

It also undermines their ability to learn in a classroom setting going forward. Among my fellow students, those who merely do what is expected of them - merely attend each and every class, do the reading, and do their problem sets and homework - aren’t prepared to learn about networking, or git, or project management, because their knowledge is disconnected from the real world. Teaching someone to use git is very difficult if they’ve never been taught that a file is an logical unit composed of bytes and metadata. Teaching someone the importance of agile methodology is difficult if they’ve never distributed a piece of software, even to their professor or friends.

Most importantly, though, it limits the ability of their peers to learn. If a 300-level software engineering class which budgeted a week to teach basic version control skills has to take a two-day detour to teach the Windows users how to get rid of the CRLFs in their commits, and teach the Mac users to remove the .DS_Store files from their repositories, and get everyone set up in Eclipse EGIT, that’s wasted time. If the professor has to schedule time with students outside of class to demonstrate their code because so many students aren’t able to submit their code in a form that successfully runs on the professor’s computer, that’s wasted time for both professors and students, and it undermines important lessons about portability and good practices.

Ultimately, my core belief is this: Students need to know how to use computers before they can program them in a serious way.

Moving forward - or backwards, or sideways?

I should be clear - I’m not talking about MIT. I’m not talking about CMU or UC Berkeley (oh, I’m sorry, “Cal”). I’m not even talking about Stanford. I’m talking about other places - the SLACs that have four or five CS professors, the polytechnic universities, everywhere else. The places where “computer science” and “computer engineering” might not be different majors, and even if they are, they aren’t really different departments, different buildings, different schools.

So how do we move forward? How do we build computer science programs that prepare students to build their own futures? We have to recognize what the ACM has been telling us since 2002 - that there are two or three tracks to “computer science”: what we might call “theoretical computer science”; what we might call “computer engineering”; and what we might call “software engineering”. We move forward by giving students a taste of all three when they enter the field, and letting them specialize as they wish.

I am very sympathetic to the idea that getting students programming quickly is important. So do that.

Provide a standardized environment - as a VM, perhaps, or using something like repl.it or ideone - perhaps a similar software designed specifically for education.
Use a language that teaches the fundamentals of the paradigm you’re interested in, like Scheme or Python. (Please, please not Java.)
Provide support for students who are interested in doing their own thing. This means having teaching assistants (and paying them!) who can help students beyond just answering questions about their homework.

Even just an effective Python or Scheme curriculum at the 100 level would be a big improvement, but it’s not enough. We need to teach students about computers themselves.

After the first foray into programming, take time to teach students about the UNIX command line. (If you’re really, really committed to Windows for some reason, PowerShell is a great piece of software too!)
Show students how things they’re familiar with - graphical file managers, for instance - interact with these new command line skills, and how their programming languages interact with those.
Commit to teaching a standardized environment that all students have access to. Ubuntu LTS is great for this, because people with their own computer can run it in a VM and the school can provide computers, and adventurous Windows users can use WSL.

And of course, mathematics is the foundation of computer science. Blending that perspective with more practical elements is important so that students learning algorithms can implement them in the real world, and students building software can analyze it appropriately. Alongside the purely mathematical understanding and models of computing should come some exposure to “weird” programming languages like Haskell, Erlang, Prolog, Elm, etc.

These are hard changes for a lot of departments, especially those with faculty that hasn’t writted production software in decades, or has only used a non-UNIX OS for a long time. I don’t think that there’s any good reason to force students to figure this stuff out on their own.

Students who have a good understanding of theory, competence in a standard UNIX environment with some language of choice, and some exposure to other languages will be better professors, better engineers, and better web developers. Let’s improve this field for the future of computing!

A commenter on Lobste.rs pointed out that there are some great resources from MIT around learning tooling and build systems: The Missing Semester of Your CS Education

Cross, Hendrix, and Umphress, 2004. DOI 10.1109/FIE.2004.1408803 ↩︎
This is why I think Python 3 is a better teaching language than Java, but that’s beside the point here. ↩︎
Example given by feonixrift ↩︎

Packaging Desktop Apps With Flatpak

Tue, 10 Sep 2019 15:00:00 -0500

As mentioned in the previous post, the web platform is popular for creating apps that would make plenty of sense as desktop software. This is partly because it unifies many platforms, but mostly it’s because it solves the distribution problem. That is, having written a program, how do I get it into the hands of users?

Consider the process of installing a desktop app for a user with a fresh install of their operating system:

Download the installer for the application
Execute the installer
Run the application

On the other hand, the web platform simplifies all of that into a single step:

Open the application in the web browser

Package managers remove the requirement to find and execute unvetted installation code, but they have other flaws, such as requiring specific packaging for each operating system distribution (Windows, Ubuntu/Debian, Arch, Fedora, CentOS, Nix, etc). Someone has to do this work, whether or not the software developer themselves take it on.

Flatpak is one way to distribute applications in a more standardized way, and this post demonstrates how to roll up the application and all its dependencies into a Flatpak so that users can easily install it.

The Makefile

The Makefile in the last post was extremely simple. This one is a bit more general, and thus a bit more complicated. Firstly, it needs to be changed to support the Makefile conventions recommended by the GNU project, primarily by abstracting the system directories and installation programs. At the top of the new Makefile, there’s the following code:

# Install to /usr unless otherwise specified, such as `make PREFIX=/app`
PREFIX=/usr

# What to run to install various files
INSTALL=install
# Run to install the actual binary
INSTALL_PROGRAM=$(INSTALL)
# Run to install application data, with differing permissions
INSTALL_DATA=$(INSTALL) -m 644

# Directories into which to install the various files
bindir=$(DESTDIR)$(PREFIX)/bin
sharedir=$(DESTDIR)$(PREFIX)/share

Then, in the install and uninstall targets, rather than simply cping files into the correct places, $(INSTALL_PROGRAM) and $(INSTALL_DATA) are used, and every filepath is prefixed with $(sharedir) or $(bindir).

This way, if someone wants to install this program onto a distribution that requires installing into a weird place like /var/<program name>/whatever, they can just set the PREFIX variable - and we can use it to do the same for the Flatpak.

How Flatpak Works

Flatpak packages use a number of Linux kernel technologies (cgroups, namespaces, seccomp, and bind mounts) and some open standards from the Open Container Initiative to provide a universal and standardized way to distribute applications. It provides a runtime on which bundled libraries and applications can be executed (more info at the official docs).

The critical aspects for application developers and users are security through isolation and dependability due to built-in dependency management, as well as ease of use in terms of installation.

Making a Flatpak

Making a Flatpak requres a program called flatpak-builder, in addition to the Flatpak tools themselves. In addition, we need a few extra components:

# Add flathub and the gnome-nightly repo
flatpak remote-add --user --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
flatpak remote-add --user --if-not-exists gnome-nightly https://sdk.gnome.org/gnome-nightly.flatpakrepo

# Install the gnome-nightly Sdk and Platform runtime
flatpak install --user gnome-nightly org.gnome.Sdk org.gnome.Platform

# Install the required rust-stable extension from flathub
flatpak install --user flathub org.freedesktop.Sdk.Extension.rust-stable//18.08

With these installed, we need a Flatpak metadata file. We’ll create the development version first. So, in the data directory of the repo, the codes.nora.gDiceRoller-development.json file will contain the app’s build metadata:

{
    "app-id" : "codes.nora.gDiceRoller",
    "tags" : [ "development" ],

This section establishes the name of this application (app-id), which must be globally unique, and any tags that should be applied to the specific version built by this Flatpak specification.

    "runtime" : "org.gnome.Platform",
    "runtime-version" : "3.32",
    "sdk" : "org.gnome.Sdk",
    "sdk-extensions" : [
        "org.freedesktop.Sdk.Extension.rust-stable"
    ],

This section tells Flatpak and flatpak-builder which pieces of software and which libraries are needed to build and run the application. The runtime should pretty much always be org.gnome.Platform, while the runtime-version is dictated by your requirements. Remember that newer versions are available to fewer users, typically.

sdk, like runtime, is usually the same, but sdk-extensions depends heavily on which tools and languages you use to create the application. In this case, that’s the Rust extension we installed earlier.

    "command" : "codes.nora.gDiceRoller",
    "desktop-file-name-suffix" : " ☢️",
    "finish-args" : [
        "--socket=x11",
        "--socket=wayland",
        "--device=dri"
    ],

First off, the command specifies which file should be run to actually start the application.

desktop-file-name-suffix specifies that a “radioactive” symbol should be added to the end of the application’s desktop file. This is a common practice for Flatpak devs distributing both a development and a production version of an app, so people can install both and tell them apart in their application directory.

The finish-args section specifies that the program needs access to the various desktop subsystems, so it can actually draw windows on the screen. I don’t, for instance, specify --socket=network or other external I/O, because the application doesn’t need it.

    "build-options" : {
        "append-path" : "/usr/lib/sdk/rust-stable/bin",
        "build-args" : [],
        "env" : {
            "CARGO_HOME" : "/run/build/gRiceRoller/cargo",
            "RUSTFLAGS" : "--error-format=short --remap-path-prefix =../",
            "RUST_BACKTRACE" : "1"
        }
    },

build-options simply sets the appropriate location for the Rust build tools and turns on backtraces for debugging.

    "modules" : [
        {
            "name" : "gDiceRoller",
            "buildsystem" : "simple",
            "build-commands": [
                "make",
                "make install PREFIX=/app" 
            ],
            "sources" : [
                {
                    "type" : "dir",
                    "path" : "../"
                }
            ]
        }
    ]
}

The modules section specifies what code actually needs to be built and how to do that. In this case, that’s a “simple” build system, which essentially means “just run the commands here”. Crucially, the PREFIX variable is set to /app, which is a Flatpak requirement.

In the previous section, we set up the Makefile so that setting the PREFIX variable will install the app in a different location. flatpak-builder will take the contents of /app and package that as the application, so that’s where it gets installed.

For the development version, we specify the local directory so that changes a developer makes are reflected immediately.

The last step for this is to add building this to the Makefile:

# Build a Flatpak package
flatpak-development: target/release/gDiceRoller
    mkdir -p flatpak-development
    flatpak-builder flatpak-development data/codes.nora.gDiceRoller-development.json

And, to clean the built package:

# Remove supplemental build files
clean :
    rm -rf flatpak/ flatpak-development/

Development vs. Production

The production version, in data/codes.nora.gDiceRoller.json, is totally identical, except that:

There is no desktop-file-name-suffix key, since this version doesn’t need to differentiate itself
There is no tags key, since this is the gold-standard version and doesn’t need a tag
The sources key is different.

Here, sources specifies the Git repository, along with the version tag and the commit hash:

"sources" : [
  {
    "type" : "git",
    "url" : "https://gitlab.gnome.org/NoraCodes/gDiceRoller.git",
    "tag": "v1.1.3",
    "commit": "57ad7d8cc886c8d9c838e665441e347d71637b1d"
  }
]

Critically, this decouples this Flatpak specification from the code itself. This will be important when uploating it to Flathub to be distributed.

In order to test this easily, there’s also an appropriate Makefile target:

# Build the Flatpak bundle for release
flatpak-release: target/release/gDiceRoller
	mkdir -p flatpak
	flatpak-builder flatpak data/codes.nora.gDiceRoller.json

For either the release or the development version, it’s possible to test the build as documented here.

Flathub

AppData

The final purpose of packaging the application into a Flatpak is to distribute it, and Flathub is the place to be for distribution. In order to be permitted onto Flatpak, an application must provide:

Desktop files
Application icons
An AppData file

gDiceRoller has everything but an AppData file, so far. You can find the AppData specification from FreeDesktop. The AppData file for gDiceRoller is at gdiceroller/data/codes.nora.gDiceRoller.appdata.xml, and the Makefile installs it to $prefix/share/metainfo/.

It’s an XML document, and is kind of long, so I’ve excerpted the relevant sections.

<id>codes.nora.gDiceRoller</id>
<name>Dice Roller</name>
<project_license>GPL-3.0</project_license>
<metadata_license>CC0-1.0</metadata_license>
<developer_name>Leonora Tindall</developer_name>
<summary>Roll dice of many different shapes and sizes in all possible combinations.</summary>
<url type="homepage">https://gitlab.gnome.org/NoraCodes/gDiceRoller/</url>
<update_contact>nora@nora.codes</update_contact>

This section specifies the crucial information about the app - the unique ID, the display name, my name, a short summary, the project’s URL, and the relevent licenses. metadata_license is the license for the info within the AppData file and should almost always be CC0 or similar. update_contact should be an e-mail at which someone can be reached who can actually change things in the package, if it needs to be updated. This is very important if someone other than the main development team is maintaining the package.

<description>
    <p>
    Roll dice of many different shapes and sizes in all possible combinations.

    gDiceRoller provides several kinds of dice, from a d4 to a d100, which can be
    rolled with a simple button press. In addition, users can enter their own
    dice with any number of sides, and perform arbitrary arithemtic on their
    results.
    </p>
</description>
<categories>
    <category>Game</category>
    <category>GTK</category>
</categories>
<screenshots>
    <screenshot type="default">
    <image type="source">https://gitlab.gnome.org/NoraCodes/gDiceRoller/raw/master/data/screenshot.png</image>
    </screenshot>
</screenshots>
<releases>
    <release version="1.1.3" date="2019-06-28"/>
    <release version="1.1.2" date="2019-06-28"/>
    <release version="1.1.1" date="2019-06-28"/>
    <release version="1.1" date="2019-06-27"/>
</releases>

This section provides some more metadata about the application. description can contain HTML which most application stores, like Flathub, will render. screenshots will be shown in header, usually below the application’s name and above the description.

categories is important, because it places the application in view of users in the relevant categories in app stores.

The releases array contains release elements. Here, there’s nothing within them, but they can contain sub-descriptions which will be rendered, usually below the main description. Use them to document changes in your application.

<launchable type="desktop-id">codes.nora.gDiceRoller.desktop</launchable>
<provides>
    <binary>codes.nora.gDiceRoller</binary>
</provides>

Finally, a few housekeeping tags. These set the actual desktop file that launches the application, and the binary it provides, so that app stores can figure out conflicts.

Publishing

With the AppData file finished, we can move on to actually publishing the project to Flathub. This is a pretty simple process:

Fork the Flathub repository (https://github.com/flathub/flathub)
Clone the fork
Create a new branch with the name of the app
Add and commit the production manifest
Make a pull request

For example, here’s the PR I made for this application.

Once that process is finished, the Flathub staff will review your application. Assuming they like it, they’ll create a new repository and make it available on a page like this, and after a while, it’ll show up in desktop app stores.

Congratulations!

HackRF Tripups with GNURadio

Sat, 07 Sep 2019 16:48:29 -0500

The HackRF One is an amazing tool, but the documentation about how to use it with GNURadio is… scattered, to say the least. I got mine working today after a few hours of effort, and I wanted to share what I did so others could benefit, because it’s a little arcane.

Specifically, I was using the wrong output sample rate.

Symptoms of having the wrong output sample rate (too low) are scratchy audio and/or only transmitting in “chunks” (because the device is waiting to get a full buffer before transmitting).

The options for the Osmocom sink, used with the HackRF.

In GNURadio, communicating with the HackRF One is done via the Osmocom sink block. In this block, there are two important fields for connection to the radio, in addition to gain, frequency, et cetera.

First, sample rate must be between 2 and 20 Msps (million samples per second), so the “Sample Rate (sps)” field must be between 2000000 and 20000000. However, it’s not a great plan to do all your processing at 2 and 20 Msps, since that will massively load down your CPU.

My solution is to use three sample rates - audio/input rate (in the case of my simple FM transmitter, 16k), “internal” rate (in my case 128k), to which the default “samp_rate” variable should be set, and finally transmitter rate “tx_rate”, 2M or more.

A Rational Resampler block converting from the internal sampling rate to the transmission rate.

I store all of these in variables: tx_rate, samp_rate (where most things operate), and input_rate. For many flowgraphs, the modulator block will convert from input_rate to samp_rate and only just before the sink is the rate converted to tx_rate with a Rational Resampler block, with the incoming rate in decimation and the outgoing rate in interpolation.

Second, you can use the HackRF One without setting any device options in the “Device Arguments” section, but if you’re using more than one HackRF One, you can use the hackrf_info tool to get the serial number and then set hackrf=<serial> in that field.

Arduino Geiger Counter Dosimeter

Mon, 19 Aug 2019 17:19:44 -0700

It’s a simple and humble apparatus, but it’s quite useful.

One of my first electronics projects was the excellent β and γ radiation detector kit, from MightyOhm. It’s all through-hole construction, a beautiful yellow PCB that’s easy to assemble and fun to play with.

Recently, though, I found myself interested in more than just a simple beep-beep-beep of incoming radiation - I wanted to actually compare the radioactivity of several samples without just manually counting the ticks. The kit does actually include a serial readout of the data, but I thought it would be fun to make something a little more visual.

The whole assembly was easy to house inside a single small cardboard box.

For this project, I picked up a display, with an I2C “backpack”, a Metro Mini microcontroller, and a battery pack with a 6V nominal output. Both the Metro Mini and the Geiger counter board can easily handle that input, so it works well and won’t sag so much as to brown out as the batteries age.

Once I’d done that, it was only a matter of a little soldering and creative cardboard-cutting to get everything situated inside an old RadioShack product box with a slit for incoming radiation. First, I soldered the four display elements onto the backpack board and added the headers to the Metro Mini. Then, I used a small piece of perfboard to connect the Mini Metro to the display and the pulse output from the Geiger counter. I also removed the battery pack from the Geiger counter and used the perfboard to route power everywhere it needed to go.

After putting on the heatshrink tubing and soldering everything together, I was ready to get coding. The method I ended up implementing uses interrupts to register pulse events from the sensor board. It simply takes a rolling average of the number of pulses over the last 30 seconds, in 42 steps per second. That allows it to easily detect up to the maximum displayable 9999 counts per minute. Actually displaying the reading is made simple by the I2C backpack board. All you really have to do is use the provided library to send commands to blank the display and set the right characters.

The radiation emmitted by this uranium ore is mostly β particles, which only go a short distance, so even a little air can block a lot of the signal to the detector.

This means that the whole apparatus can be placed next to a sample for about 30 seconds, and the correct reading will show up on the screen. I also managed to snag a sample of uranium ore from United Nuclear which measures about 1500 CPM when in direct contact with the sensor. It was a fun project and will, I hope, prove quite useful in comparing mildly radioactive β and γ emmitters.

One interesting lesson was just how aggressively radiation falls off through air, water, and other common substances. The difference between two centimenters of air between the vial and the detector, and direct contact, is about 50%, and the glass of the vial appears to block about 30% of the signal. It’s pretty clear, then, why it is that the main danger of radioactive material is not mere exposure but ingestion or inhalation, which exposes sensitive organs to the radiation directly.

You can find the code for this project here. It requires the Adafruit support library for their I2C display backpacks.

Simple Elixir Functions

Sat, 20 Jul 2019 12:03:23 -0700

I’ve been playing around with Elixir, which is a pretty cool language - specifically, I’m reading Programming Elixir 1.6 which is free for anyone with a .edu email address from The Pragmatic Bookshelf.

reverse/1

I enjoy functional programming, so of course the first two functions I made were list operations. First up is reverse(list) (written as reverse/1, meaning it takes one argument), which reverses a list.

# reverse/1
#   Reverse the given list.
def reverse([]), do: []
def reverse(list) do
    reverse(tl list) ++ [hd list]
end

The first def line here says that if you pass reverse/1 an empty list, the result is an empty list; that’s the base case.

The second def defines the recursive case, in which reverse/1 takes the tail of the list (tl list), reverses it, and appends the first element (hd list).

palindrome/1

Another classic FP function is palindrome(list), evaluating to true if the given list is a palindrome (like [1, 2, 3, 2, 1]).

This function takes advantage of the _ special variable to say that any list with exactly one element is automatically a palindrome, as a second base case.

It also uses the |> pipe operator to turn the pruning operation, which turns [1, 2, 3, 2, 1] into [2, 3, 2] for recursion, from a nested function call List.delete_at(tl(list), -1) into the somewhat more readable tl(list) |> List.delete_at(-1).

I also went out of my way to use reverse/1 here, just to say that I did. It’s definitely not the most efficient way to implement this algorithm.

# palindrome/1
#   Return true if the given list is palindromic
#   or false otherwise.
def palindrome([]), do: true
def palindrome([_]), do: true
def palindrome(list) do
    (tl(list) == tl(reverse list)) and palindrome(tl(list) |> List.delete_at(-1))
end

There’s not much point to this post other than to say - Elixir is fun, and a pretty language!

What Is Rust's unsafe?

Fri, 12 Jul 2019 10:00:00 -0700

I’ve seen a lot of misconceptions around what the unsafe keyword means for the utility and validity of Rust and its marketing as a “safe systems language”. The truth is a lot more complicated than a single pithy tweet can possibly sum up, unfortunately; here it is as I see it.

Basically, the unsafe keyword does not turn off the advanced type system that keeps Rust code honest. It only allows a few select “superpowers”, like dereferencing raw pointers. It is used to implement safe abstractions over a fundamentally unsafe world so that the majority of Rust code can use those abstractions and avoid memory unsafety.

The Promise of Safety

Rust promises safety as one of its core tenets; it is, in some ways, the raison d’être of the language. It does not, however, go about providing that safety in the traditional way, using a runtime and a garbage collector; rather, Rust uses a very advanced type system to keep track of which values are safe to access when, and the compiler then statically analyzes each Rust program to ensure that certain invariants are upheld.

Safety in Python

Let’s take, as an example, the Python programming language. Pure Python code cannot corrupt its memory. List accesses have bounds checking, references returned by functions are reference counted to prevent dangling pointers, and there’s no way to perform arbitrary pointer arithmetic.

This has two consequences: first, a lot of types have to be “special”. For example, it’s not possible to implement an efficient Python list or dict in pure Python; instead, the CPython interpreter implements lists and dicts internally. Second, access to external (non-Python-managed) functions, called “foreign function interface”, requires the use of the special ctypes module and breaks the language’s safety guarantees.

In a certain sense, this means that everything written in Python is memory unsafe.

Safety in Rust

Rust also provides safety, but instead of implementing unsafe structures in C, it provides a so-called “escape hatch”: the unsafe keyword. This means that the foundational data structures in Rust like Vec, VecDeque, BTreeMap, and String are all implemented using Rust.

“But Nora,” I hear you asking, “if Rust provides an escape hatch from its guarantees, and the standard library is implemented using that escape hatch, isn’t everything written in Rust unsafe?”

In a word, dear reader, yes - in exactly the same way that everything in Python is. Let’s dig into that.

What Is Prohibited in Safe Rust?

Safety, in Rust, is very well-defined; we think about it a lot. In essence, safe Rust programs cannot:

Dereference a pointer that does not point to the type the compiler thinks it points to. This means no null pointers (because they point to nothing), no memory-out-of-bounds and/or segmentation faults, and no buffer overflows, but it also means no use-after-free or double-free (because freeing memory counts as dereferencing a pointer), and no type punning.
Cause there to be either multiple mutable references or both mutable and immutable references to the same data at the same time. This is, if you have a mutable reference to some data, only you have that reference, and if you have an immutable reference to that data, it will not change while you hold that reference. This means it is impossible to cause a data race in safe Rust, which is a guarantee most other safe languages do not provide.

Rust encodes this information in the type system, either through the use of algebraic data types like Option<T> to encode presence/absence and Result<T, E> to encode failure/success, or references and lifetimes like &T vs &mut T to encode the difference between shared (immutable) and exclusive (mutable) references and &'a T versus &'b T to denote references that are valid in different contexts/frames. (These are usually elided; that is, the compiler is generally smart enough to figure them out.)

Examples

For example, the following code does not compile because it would cause a dangling reference; specifically, my_struct does not live long enough. In other words, the function would return a reference to something that no longer exists, and therefore the compiler won’t (and really, doesn’t know how to) compile it.

fn dangling_reference(v: &u64) -> &MyStruct {
    // Create a new value of type MyStruct with the value field set to v,
    // the function's one parameter.
    let my_struct = MyStruct { value: v };
    // Return a reference to the local variable my_struct.
    return &my_struct;
    // my_struct is deallocated (popped off the stack, really).
}

This code does the same thing, but tries to get around the problem by placing the value on the heap (Box is Rust’s name for a basic smart pointer with no wacky behavior.)

fn dangling_heap_reference(v: &u64) -> &Box<MyStruct> {
    let my_struct = MyStruct { value: v };
    // Put the struct in a Box, allocating space for it on the heap and moving it there.
    let my_box = Box::new(my_struct);
    // Return a reference to the local variable my_box.
    return &my_box;
    // my_box is popped off the stack. It "owns" my_struct and thus is responsible
    // deallocating it, so the MyStruct is deallocated.
}

The correct code returns the Box<MyStruct> itself, rather than a reference to it. This encodes the transfer of ownership - responsibility for deallocation - in the type signature of the function. Just by looking at the signature, it’s clear the the caller is responsible for what happens to the Box<MyStruct>, and indeed, the compiler handles this automatically.

fn no_dangling_reference(v: &u64) -> Box<MyStruct> {
    let my_struct = MyStruct { value: v };
    let my_box = Box::new(my_struct);
    // Return local variable my_box by value.
    return my_box;
    // Nothing is deallocated. The caller is now responsible for managing the heap memory
    // allocated in this function; it will almost certainly be deallocated automatically
    // when the `Box<MyStruct>` goes out of scope in the caller, barring a double-panic.
}

Some bad things are not prohibited in safe Rust. For example, it is absolutely permissible, from the point of view of the compiler, to:

deadlock a program
leak an arbitrarily large amount of memory
fail to close file handles, database connections, or missile silo covers

A strength of the Rust ecosystem is that many projects take the approach of using the type system to enforce correctness to heart, but the compiler does not require such enforcement except for the case of memory safety.

What is Permitted in Unsafe Rust?

Unsafe Rust is Rust code annotated with the unsafe keyword. unsafe can be applied to a function or a block of code. When applied to a function, it means, “this function requires that its caller manually uphold variants otherwise upheld by the compiler”; when applied to a block of code, it means, “this block of code manually upholds variants required to prevent causing memory unsafety, and should therefore be permitted to do unsafe things”.

In other words, on a function, unsafe means “you need to check”, and on a block of code, it means “I checked”.

As mentioned in TRPL, code in a block annotated with the unsafe keyword can:

Dereference a raw pointer. This is the key “superpower” that lets us implement doubly-linked lists, hashmaps, and other fundamental data structures.
Call an unsafe function or method. More discussion on this below.
Access or modify a mutable static variable. Static variables whose scope is not controlled cannot be statically checked, so their use is inherently unsafe.
Implement an unsafe trait. Unsafe traits are used to mark whether or not specific types guarantee certain invariants; for instance, the Send and Sync traits determine whether or not a type can be sent across a thread boundry or used in multiple threads at once.

Those dangling pointer examples I gave above? Annotate the functions with unsafe and you’ll just get the compiler complaining twice as much, because it doesn’t like the use of unsafe on code that doesn’t need it.

Instead, the unsafe keyword is used to implement safe abstractions over otherwise arbitrary manipulations of pointers. For example, the Vec type is implemented using unsafe, but it’s safe to use, since it checks accesses and doesn’t allow overflow, and while it does provide operations like set_len which could cause memory unsafety, those operations are all marked unsafe.

For example, we could do the same thing as the no_dangling_reference example with the gratuitous use of unsafe:

fn manual_heap_reference(v: u64) -> *mut MyStruct {
    let my_struct = MyStruct { value: v };
    let my_box = Box::new(my_struct);
    // Convert the Box into a regular old pointer.
    let struct_pointer = Box::into_raw(my_box);
    return struct_pointer;
    // Nothing is dereferenced; this functions just returns a pointer.
    // The MyStruct remains where it is on the heap.
}

Note the lack of unsafe; creating raw pointers is totally safe. As written, this is a memory leak risk, but nothing more, and leaking memory is safe. Calling the function is safe too; its only when something wants to dereference the pointer that unsafe is needed. As an added benefit, it will deallocate the memory automatically.

fn main() {
    let my_pointer = manual_heap_reference(1337);
    let my_boxed_struct = unsafe { Box::from_raw(my_pointer) };
    // Prints "Value: 1337".
    println!("Value: {}", my_boxed_struct.value);
    // my_boxed_struct goes out of scope. It now owns the memory on the heap, so it
    // deallocates the MyStruct.
}

With optimizations, that code is exactly equivalent to just returning the Box in the first place. Box is a safe abstraction over the use of pointers, because it prevents spreading that raw pointer all over the place. For instance, the following version of main will result in a double-free.

fn main() {
    let my_pointer = manual_heap_reference(1337);
    let my_boxed_struct_1 = unsafe { Box::from_raw(my_pointer) };
    // DOUBLE FREE BUG!
    let my_boxed_struct_2 = unsafe { Box::from_raw(my_pointer) };
    // Prints "Value: 1337", twice.
    println!("Value: {}", my_boxed_struct_1.value);
    println!("Value: {}", my_boxed_struct_2.value);
    // my_boxed_struct_2 goes out of scope. It owns the memory on the heap, so it
    // deallocates the MyStruct.
    // my_boxed_struct_1 then goes out of scope. It also owns the memory on the heap,
    // so it also deallocates the MyStruct. This is a double-free bug.
}

Just What is a Safe Abstraction?

A safe abstraction is one that uses the type system to provide an API that cannot be used to violate the safety guarantees I mentioned above. Box is safer than *mut T because it cannot be used to produce the double-free just illustrated.

Another great example of this is Rust’s Rc type. It’s a reference counted pointer - a read-only reference to some data stored on the heap. Because it allows multiple simultaneous access to a single memory location, it must prevent mutation to be considered safe.

In addition, it’s marked as not thread-safe; if you want thread-safety, you have to use the Arc (Atomic Reference Counting) type, which takes a performance hit from using atomic values for its reference count to prevent the possibility of data races in multi- threaded environments.

The compiler will stop you from using Rc where you should use Arc, because the people who implemented Rc did not mark it as thread safe. If they had, that would be “unsound”: a false promise of safety.

When is Unsafe Rust Necessary?

Unsafe Rust is needed whenever performing an operation that might cause one of those two rules mentioned above to be violated. For example, in a doubly-linked list, not having two mutable references to the same data (from the node before and after a given node) defeats the whole purpose. With unsafe, the implementor of a doubly-linked list can write that code using *mut Node<T> pointers and then encapsulate it in a safe abstraction.

Another example is when working in the embedded space. Often, microcontrollers use a set of registers whose values are determined by the actual physical state of that device. The world won’t just pause for you when you take an &mut u8 of that register, so the device support crates need unsafe in order to work with them - but they generally strive to encapsulate that state in transparent safe wrappers that copy data when possible, or use other techniques to uphold the guarantees of the compiler.

It’s sometimes unavoidable to do an operation that might lead to simultaneous reading and writing, or memory unsafety, and that’s when unsafe is needed. But, so long as there is a way to ensure that the Rust safety invariants are checked before the user of safe (that is, non-unsafe-marked) code touches anything, that’s completely fine.

On Whose Shoulders Does It Fall?

So, we come back to the point I made above - yes, the Rust language’s utility is built on a foundation of unsafe code. Though it’s done in a slightly different way than the unsafe implementations of core data structures in Python, it’s just as true that the implemenation of Vec, HashMap, et cetera must use raw pointer manipulations at some level.

We say that safe Rust is safe with the fundamental assumption that unsafe code we use through dependencies, whether on the standard library or other library code, has been correctly written and encapsulated. The fundamental advance of Rust is that the code is corralled into unsafe blocks, which are supposed to be verified by their authors.

In Python, the burden of ensuring that raw memory manipulation is safe falls only on the interpreter maintainers and the users of the foreign function interface. In C, that burden is on every single programmer.

In Rust, the burden falls on the user of the unsafe keyword. It’s clear, within the code, where variants need to be manually upheld, and it is common to strive for zero unsafe code anywhere in a library or application’s codebase. The unsafety is identified, segregated, and clearly marked, so if your Rust code segfaults, you have discovered either a bug in the compiler or a bug in your few lines of unsafe code.

It’s not a perfect system, but if what you need is the trifecta of speed, safety, and concurrency, it’s the only option out there.

Speedy Desktop Apps With GTK and Rust

Fri, 05 Jul 2019 14:00:00 -0700

The web platform is the delivery mechanism of choice for a ton of software these days, either through the web browser itself or through Electron, but that doesn’t mean there isn’t a place for a good old fashioned straight-up desktop application in the picture.

Fortunately, it’s easier than ever to write a usable, pretty, and performant desktop app, using my language of choice (Rust) and the wildly successful cross-platform GUI framework GTK. In this blog post, we’ll walk through the source code of gDiceRoller. In future posts, I’ll explain how I packaged it for different systems.

A screenshot of gDiceRoller, the program this post is about.

What App?

gDiceRoller is a truly simple app. It leverages the excellent rfyl dice notation library to allow users to roll dice of arbitrary numbers of sides and perform arithmetic on the results.

Several polyhedral dice of the type commonly used for gaming.

Dice notation is the de facto standard notation used by tabletop gaming systems such as Dungeons and Dragons to express how many and which kind of dice a player should roll to determine what happens in a game. A roll is expressed as NdM, where N is the number of dice to be rolled, and M is the number of sides each die should have.

Dungeons and Dragons is known for using an icosahedron, or d20, for many of its rolls. When determining the effects of attacks in combat, it’s common to roll 2d8 or more, and in older editions of the game it was possible to build up truly complex sets of rolls, like 1d20 + 1d6 - (1d4 / 2) for a skill check or 12d6 + 2 for some damage rolls.

The most common dice are those that correspond to the platonic solids; tetrahedrons (d4), cubes (d6), octahedrons (d8), decahedrons (d10), dodecahedrons (d12), and icosahedrons (d20). People also like to roll two d10s and use the first as the first digit and the second as the second digit to make a makeshift “d100”, for percentages.

This application will allow its users to generate pseudorandom numbers based on parameters specified in dice notation, or by clicking on any of the several common dice provided, as well as permitting common operations like halving results with specific rounding behavior.

That’s it - it’s quite simple!

UI Prototyping

Glade, from the GNOME project, is easy to use and very powerful.

Given the simple interface for this program, it would be fairly easy to build the entire interface from within Rust code, but such an approach becomes unwieldy for larger projects. Fortunately, the GNOME project provides a best-in-class UI design program called Glade (available from your distribution or at glade.gnome.org).

Glade is pretty intuitive; I suggest picking up the UI definition for this project and pulling it up yourself. This is an XML-based file that defines the tree structure of the user interface, much like HTML does for web applications. (If you want to deep-dive on Glade, check out the dedicated tutorial series from the GNOME project.)

Glade’s killer feature, in my opinion, is its “preview snapshot” functionality, which opens the current UI definition in an actual native window. This lets you play with any standard UI functionality like notebooks (tabs), scroll bars, and tree views without writing any code at all.

Because it’s a tree structure, you can prototype fragments of the UI, like complex list entries or modals, in their proper place and then simply drag-and-drop them into their own tree to be used in the program once they look good.

Once the UI looks great, just save it for later. Integration into gtk-rs, the Rust GTK3 bindings, is very easy.

Dependencies

Note: from this point on, anywhere you see codes.nora.gDiceRoller or gDiceRoller, you will need to use your own project’s name. I would have called the crate codes.nora.gDiceRoller, in standard reverse domain notation used in many packaging systems, but that’s not permitted, so I simply called it gDiceRoller.

All you need to start writing this code is a working cargo installation and the GTK development libraries. On Ubuntu, that’s libgtk-3-dev.

Within the Cargo.toml, the dependencies are actually pretty simple; we need GTK, GLib, and RFYL (which handles the dice rolling).

With GTK, we have to specify a feature corresponding to the minimum toolkit version the program will support, through the use of cargo feature tags. This project deliberately uses an old version for maximum compatibility, but you’re free to choose whatever you want.

[package]
name = "gdiceroller"
version = "0.1.0"
authors = ["Leonora Tindall <nora@nora.codes>"]
edition = "2018"

[dependencies]
rfyl = "0.3.1"
glib = "0.7"

[dependencies.gtk]
version = "0.6"
features = ["v3_16"]

Factorization

Thanks to @federicomena for help refactoring this program to be easier to read.

gDiceRoller is an example of an easy-to-use factorization template for any GTK Rust application. It keeps its application logic, state management, and GUI management in seperate modules, and uses the main() function to tie them together.

Rolling Dice

The most important thing this program does is to roll dice - that is, take a string of dice notation and generate a number from it. All of that logic is provided by rfyl, and it’s only really necessary to have a single function that handles the whole process.

That’s roll_expression; it lives in src/rolls.rs and takes a string, passes it to rfyl::roll, and returns either the result or an error. Putting this logic in its own file also gives us an obvious place to do some basic tests.

State Management

Then there’s the application’s state to think about. For gDiceRoller, that’s simple. It stores the last rolled value and, if there was a problem with the user-entered dice notation, what caused that error.

Whatever your application’s global state is, it can go in its own module, state. In gDiceRoller, that’s a single file src/state.rs, and it contains the actual state as a State struct with functions to operate on it. These functions are what the rest of the application will use to change the state, and they take an &mut self. In gDiceRoller, there’s just one; the application does sometimes directly change the value inside, but the only complex behavior is encapsulated in update_from_roll_result, which is tested in the same file.

GUI Handling

Now comes the actual “front-end” portion; the GUI. In gDiceRoller, all the handles into the UI are contained in a single struct, MainWindow, in `src/main_window.rs’

pub struct MainWindow {
    window: gtk::Window,
    result: gtk::Label,
    popover: gtk::Popover,
    error_label: gtk::Label,
    user_spec_entry: gtk::Entry,    
    buttons: HashMap<String, gtk::Button>,       
}

In the new() function for this struct, the include_str!() macro is used to load the Glade template file. I just saved it in the src/ folder and did include_str!("mainwindow.glade").

Then, each element of the struct is initialized with builder.get_object("objectName").unwrap(). This takes these objects from the stringly-typed world of the XML file to their exact types in the MainWindow struct.

The exception to this rule is the buttons. There are a lot of buttons, and they all do the same thing with a minor variation, so a for loop is used to run through a slice of all the names and add them to a HashMap<String, gtk::Button>. It doesn’t provide quite the same compile-time guarantees, but at least get() on that HashMap returns a gtk::Button rather than an arbitrary GObject.

The MainWindow struct also has a separate run-time function, start(), which sets the application’s name, its wmclass, and its delete event, and shows it and its children.

Finally, MainWindow has a function update_from which takes an &State and changes all the visible fields to their appropriate values. In this case, that’s just showing the error popover if there’s an error, and updating the number displayed at the top.

This is all the data structures in use; the rest of the work is done in main.rs.

Functionality

main.rs functions as something of a plugboard, wiring together the actual computation with the State and MainWindow structs, and starting the whole thing rolling.

The program first tries to initialize GTK. Then, the GUI state and the application state are initialized and placed in Arc<> and Arc<RefCell<>>, respectively. State can now be safely modified from within callbacks, thanks to RefCell’s interior mutability (see Section 15.05 in The Rust Programming Language for more information on this pattern).

RefCell and RwLock are both good candidates for use as the interior-mutability abstraction here. RefCell is used in this case because there is no need to share between threads, and RefCell has less complexity than RwLock. Mutex is also a viable candidate, or a custom data structure using std::sync::atomics.

The use of Arc instead of Rc is unnecessary here, since the program is single-threaded. This is an error on my part; the data structure should be Rc<RefCell<T>> or Arc<Mutex<T>>/Arc<RwLock<T>>. This will be corrected soon.

After that, main just needs to add all the callbacks for the various buttons. All the standard dice can be done with a loop. Those for clearResult, halveDownResult, et cetera are done on their own.

There’s a pattern to these callbacks. Firstly, each callback is created in its own local scope, so as to make lifetime juggling easier. Then, the appropriate control is fetched, and Arc::clone() is called on &gui and &state to get a local reference to the global state. Finally, the actual code is written in a move closure, so these references are captured.

For example, the clearResult callback is written thus:

{
    let button = gui.button("clearResult");
    let gui = Arc::clone(&gui);
    let state = Arc::clone(&state);
    button.connect_clicked(move |_| {
        let mut state = state.borrow_mut();
        state.value = 0;
        gui.update_from(&state);
    });
}

Finally, main calls gui.start() and gtk::main() and the program runs!

You can run the tests with cargo tests, or actually run the program with cargo run.

Putting the “Desktop” in “Desktop App”

Just a pretty GUI and speedy functionality aren’t enough, though; a good desktop app needs integration. The most basic integration, on the Linux desktop, is provided through a .desktop file and an application icon. In addition, applications can provide an appdata.xml file describing their features and history. All of these files are held in the data directory, in the case of gDiceRoller.

For gDiceRoller, the .desktop file codes.nora.gDiceRoller.desktop looks like this:

[Desktop Entry]
Name=Dice Roller
GenericName=Dice Roller
Comment=Roll dice of many different shapes and sizes in all possible combinations.
Categories=Game;GTK
Icon=codes.nora.gDiceRoller
Exec=codes.nora.gDiceRoller
Terminal=false
Type=Application

This format is well documented, and supports a lot of nice features like translations. It should be placed in /usr/share/applications/ or somewhere similar (depending on distribution).

You’ll notice that the Icon field uses the same name as the application’s full ID. That is because the icon can also be installed on the system. gDiceRoller provides an SVG icon which can be scaled to any size, as well as raster icons at 64x64 and 128x128 resolutions. These are placed in /usr/share/icons/hicolor/<size>/applications. These icon names can then be used to set the window’s icon in the Glade file as well.

GTK and desktop environments automatically find icons by name, but it’s sometimes necessary to refresh their cache before they can actually find new icons.

All of this is getting kind of complex. Time to introduce: the build system!

The Build System

The build system, in this case using make, lets us easily install and uninstall the program from a system.

So, why did I use make? Well, it’s really simple and pretty much just does what I want, and gets out of the way. In addition, using a make-based build makes it easier for other distributions to package the project in the future. Here’s the whole Makefile I used during early development, which will act as the template for the rest of the project.

# Just tell make that clean, install, and uninstall doesn't generate files
.PHONY: clean clean-all install uninstall

# Build the application
target/release/gDiceRoller : src
    cargo build --release

# Install onto the system
install : target/release/gDiceRoller
    # Install the binary
    cp target/release/gDiceRoller /usr/bin/codes.nora.gDiceRoller
    cp data/codes.nora.gDiceRoller.desktop /usr/share/applications/codes.nora.gDiceRoller.desktop
    cp data/codes.nora.gDiceRoller.svg /usr/share/icons/hicolor/scalable/applications/codes.nora.gDiceRoller.svg

# Remove from the system
uninstall :
    rm -f /usr/bin/codes.nora.gDiceRoller
    rm -f /usr/share/applications/codes.nora.gDiceRoller.desktop
    rm -f /usr/share/icons/hicolor/scalable/applications/codes.nora.gDiceRoller.svg

# Rebuild from scratch
clean-all : clean
    cargo clean

# Does nothing, yet
clean :
    true

With this Makefile, it’s possible to simply:

make
sudo make install

and have a totally working installation. To fix a few issues with the system not picking up on the new icon, adding touch /usr/share/icon/hicolor to the install target after installing the icon does the trick.

Then, if you want to remove the program:

sudo make uninstall

NOTE: This Makefile is incomplete; please check out the one used by the actual project as a better template here and/or wait for the next post to explain all the aspects of that Makefile.

Conclusion and Recap

gDiceRoller isn’t a groundbreaking application, but I hope it demonstrates how easy it is to make useful, non-web-based software. I only showed how to install the software on a dev machine, but in the next few posts, I’ll look at Flatpak, Snap, .deb packages, and even Windows packaging. GTK is a cross-platform toolkit, after all!

System76 Thelio: A Review

Sat, 23 Feb 2019 11:31:34 -0600

The bottom line: Thelio, System76’s new “open hardware” desktop, is a small, beautiful, and powerful desktop computer that hits every high point anyone could have expected, faltering only in the inherent limitations of its small size.

Pros: pretty, small, performant, well cooled, well supported, and easy to modify

Cons: somewhat noisy, no front I/O, not as many USB ports as one might desire

The smallest Thelio, thelio-r1, was made available in January 2019.

In November 2018, System76, renowned Linux laptop company, announced the Thelio though an ARG-like process of story-based videos. The product itself was rarely featured, but a good deal of hype built up around the whole affair. When specs and a configurator were finally made available, along with the details of the open source I/O add-in board and the manufacturing process that System76 was spinning up in Colorado, that hype appeared at least partially justified. I’ve had a Thelio for a couple of weeks now, and it’s pretty much lived up to its manufacturers lofty promises.

The Exterior

The smallest Thelio, designated “thelio-r1”, stands 32 centimeters tall, 20 centimeters wide, and 28 centimeters deep, coming to only a total of 18 liters in volume, a good deal smaller than even the tiniest commercial full-GPU mITX form factor cases, most of which are at least 22 liters.

thelio-r1 is absolutely tiny. The ruler in this photo is 15cm, or 6 inches.

The front, sides, and top are all made of powder coated metal, colored a deep and even black. On the front and metal side are subtle laser-etched designs: a System76 logo and a Colorado mountainscape, respectively. Covering much of the right side and the right portion of the front is the wrap-around wood accent panel, which can be purchased in either dark maple or “high-contrast” cherry.

The maple on my unit blends well with my dorm room furniture and will match with any natural or dark-colored desk or media console. For rooms with a lighter color scheme, the cherry version might be a better option. The wood is rough and not finished with a varnish or stain, which could leave it vulnerable to damage over time.

Aside from a single LED-lit metal button, the front, top, and sides have no cutouts or ports. The case comes down in the front and back to touch the surface the computer rests upon, though four small rubber posts on the underside take the weight of the system itself. Between those areas there is a small cut-out section to permit sufficient airflow to the underside intake fan.

The rear panel provides all the I/O on the entire system.

The choice of motherboard and GPU determines the back panel’s ports and configuration. My AMD model has

HDMI and DisplayPort from the motherboard
line out, line in, and headphone out
Gigabit Ethernet
2xUSB type A 3.1 gen 1
2xUSB type A 3.1 gen 2
Video connectors from the GPU

The Intel version also includes a USB type C port, but either way, the small form factor will mean a shortage of USB ports. I recommend placing a USB 3.1 gen 2 hub either atop the machine or on your desk or console just beside it for easy access.

Having machined cutouts for the back panel I/O is a boon for reliability and durability, but it does make upgrading the motherboard very difficult. On the other hand, System76 has chosen relatively high-end motherboards which are likely to be supported for some time; AMD in particular plans to release new CPUs on Socket AM4 as far out at 2022, making Thelio a long-term investment for those willing to pull out the guts of their machines once or twice.

The Mechanics

Clustering all the ports on the back contributes to the overall clean and modern aesthetic, as well as making it very easy to remove the entire wood and aluminum shroud. It requires only the removal of four tool-free thumb screws, which are powder-coated to the same color as the case. Once these are taken out, the whole case shroud simply slides up on a set of extruded rails that fit like a glove and are simple to slot back into place.

Thelio’s metal shroud is decordated with a beautiful stylized mountainscape, echoing its Colorado origins.

The metal power button connects to the rest of the system via a set of four pins which slide in and out of their sockets with ease, comfortably long after the shroud has settled into its grooves; there’s little danger of bending them or missing the socket during shroud replacement.

Within is a durable and competently engineered mini-ITX system. My unit came with a Radeon RX580 GPU and 16GB of RAM, and everything was properly cable managed, including the thin and bend-sensitive coaxial cables which feed into the WiFi and Bluetooth antennae. Because the right side panel is wood, not metal, these antennae can be entirely internal, leaving the exterior smooth and durable. The chassis design preempts much of the damage than occurs in typical systems when shipped with metal cross-bars and a well-secured GPU.

The interior cables are well-managed, which is important in such a tiny space.

The case design centers on combination of miniaturized computing and modular storage. All systems ship with an NVMe drive mounted directly on the motherboard, and include four slots for SATA 6GB/s hard drives. These hard drives, as well as all the system fans, connect to a custom built open hardware I/O board which routes SATA traffic and performs smart fan management.

The I/O board is reliable and low-latency, not impacting the read speed or seek times on my Samsung 850 or 860 EVO SSDs in any detectable way. It also provides enough juice via the SATA connectors to run four old and inefficient 1TB hard drives simultaneously. Unfortunately, at the time of writing, System76 does not make a configuration utility available for this I/O board, although they do provide kernel modules which allow Linux users to monitor temperatures and fanspeeds.

The drive slots use rails designed to work with the 16 included vibration-isolating thumb screws, stored in specially machined screw-holes inside the chassis. Once inserted, some significant force is required to remove a drive, but not so much as to place the drive cage at risk of damage. Unsurprisingly, given the small volume of the system, the cage is too small to support 3.5 inch drives.

Thermal and Acoustic Performance

A positive-pressure system fed through a filtered fan below the chassis provides cooling air to the CPU, which exhausts through a massive channeled fin cooler, and GPU, which blows out into the chassis. This reduces the potential of the system internals to build up dust as well as ensuring that the maximum amount of air passes through the heat exchanger, though it does have the potential to starve the GPU fans of room-temperature air.

Both of the Thelio’s vents show the solar system at the time of the UNIX epoch. The filter on this bottom intake does a good job even in the very dusty environment of my dorm.

Though not as effective as liquid cooling, this system proved more than adequate for my system. I was unable to force the system’s temperature more than about 30℃ above ambient, providing plenty of headroom for voltage increases for those who wish to overclock. GPU temps were not nearly as good on my RX 580, reaching almost 50℃ above ambient, but I observed no thermal throttling. This likely depends on the CPU and GPU configured; luckily, the Thelio supports all tiers of current-gen AMD Ryzen and Intel Core CPUs and several Nvidia and AMD GPUs.

Of course, no air-cooled system can be as quiet as a good water cooling solution, but the Thelio manages to keep its acoustic load low and low-pitched thanks to the two large, slow-spinning fans and the efficiency with which it uses the air they move. Because of the positive-pressure design, it can’t spin down its fans when cool, so the system is audible all the time, but it’s never obnoxious and rarely ramps up, even under load.

Customer Experience

The worst part of my Thelio experience was the waiting. Not that there was so much of it, mind you; System76 custom-builds each order in their Colorado factory and ships via UPS, so it arrived about 3 weeks after I ordered it, including the 5-day shipping, which can be upgraded. System76’s web-based configurator provides many options ranging from low-end to ultra-powered for CPU, GPU, storage, main memory, and even warranty. The company even includes the option to get a shipping label to a recycling facility for an old computer the Thelio might be replacing.

The Thelio’s packaging did stand out in my campus mail center, with a huge stylized moon dominating the package’s top along with the word “Thelio” and the System76 logo. Inside was a simple postcard, some instructions on how to get customer support, and a great deal of what felt to me like very firm foam securing the system in place. The cardboard provides hand-holds for carrying, a welcome feature during my trek back across my frozen campus.

Software

The included Pop!_OS, despite having some of the worst naming on the Linux desktop today, provides a great Ubuntu-based experience with minimal but well-thought-out modifications to the GNOME 3 desktop. System76 included the drivers for my AMD RX 580 and a firmware update program for future use, though no updates were needed upon arrival. Unfortunately, there is no configuration utility for the custom I/O board, though while the provided open source drivers do make fan speed measurements available to the OS.

The lack of the Ubuntu Snap subsystem by default makes access to many proprietary apps more difficult than necessary. On the other hand, the product is targeted at developers, and installing and using Snap support is trivial from the command line, so this isn’t too much of a black mark.

Drives are inserted on rails using vibration-isolating gromits. If only it were this easy to get RAID working!

With so many drive slots, a graphical configuration utility for software RAID would be great; unfortunately, GNOME Disks no longer supports creating mdraid volumes, though it can manage them. Again, this is probably OK, as mdraid is simple to configure from the command line, though I did find that the motherboard used in the AMD model does not play well with partition-on-mdraid configurations; rather, each mdraid member needs to be partitioned and those partitions set up as members of the mdraid array. This is common, but disappointing.

Price

For all this, my Thelio weighed in at a hefty $1,866, plus $31 for 5-day UPS shipping. That’s not a trivial amount of money; I can easily build a respectable gaming computer for a quarter of that, and even a decent laptop workstation from Lenovo is just pushing $1,500. Larger Thelio models like the Thelio Major and Thelio Massive, which starts at $3000, and even higher-end configurations of the thelio-r1, can reach absolutely astronomical prices.

After doing some math, though, it looks like the margins aren’t particularly high, especially for the level of integration and service provided. The Thelio costs about 18% to 22% more than the prices of the individual components; $200 dollars more than an almost equivalent PC Part Picker build for the base model, or about $400 more for my model. This gap widens only slightly with the ultra-high-end models.

The Verdict

System76’s new “open hardware” desktop, is a small, beautiful, and powerful desktop computer that hits every high point anyone could have expected, faltering only in the inherent limitations of its small size. It’s pretty, it’s tiny, it’s fast, it’s well cooled, and the software support is top-tier. Despite being somewhat noisy and lacking front I/O, it’s certainly a good machine for any Linux user who can swallow the 18% - 22% upcharge for assembly and custom engineering.

Go/No Go

If you are a Linux user looking for a durable, competently engineered, and beautiful desktop PC, the System76 Thelio is for you, so long as you have a flexible budget.

If you don’t currently use Linux and want to get started, the Thelio can still do what you need, but you might want to go with stock Ubuntu instead of Pop!_OS. If you’re a Mac OS developer who is looking to switch to Linux, I suggest checking out Mike Dominic’s review for a Mac user perspective.

If you’re a software developer who likes to game, the Thelio is a no-brainer. With the off-the-shelf hardware and excellent expandability, you can easily add enough storage for even the largest games and, if needed, dual boot Windows.

If you need more than one PCI-e expansion card, more than four USB ports, or more than about 12 TB of storage, you might want to look elsewhere, though the Thelio Major or Thelio Massive might meet your needs.

If you’re comfortable building your own PCs and don’t need to fit a powerful system into a small space, you can save a buck by going that route instead, and probably end up with better thermals.

Finally, if you need workstation power on the go, the Thelio is simply the wrong product, but System76, Lenovo, and Dell all offer excellent Linux-supporting portable workstations.

Let's Build a Terrible Search

Sun, 03 Feb 2019 21:09:52 -0600

Searching: the second-most studied problem in computer science, next to sorting (or so it would seem from my CS curriculum). There are pretty good methods for doing it, too; for example, everyone’s favorite, binary search, can find an item in a list of 260,000 possibilities after looking at just 18 items (the base-2 logarithms of the size of the array).

Even the most obvious search, linear search, isn’t terrible, especially for problem sizes which can fit in memory close to the CPU, like low-level cache. But, fast programs are boring. I want a bad search! So, let’s get there.

Linear Search

Let’s look at the Python (“executable pseudocode”) for a basic linear search:

def pythonseek(array, x):
    return array.index(x)

This is great, except that if we call it with an x that doesn’t exist in the list, we get an exception. Let’s change that:

def pythonseek(array, x):
    try:
        return array.index(x)
    except ValueError as e:
        return

So, that’s easy. What if we want to see the steps, though? It’s pretty easy to build a simple linear search in Python. We iterate over each element in the array, just checking if that element is the one we’re looking for. If so, we return the index.

def linearseek(array, x):
    for (index, element) in enumerate(array):
        if element == x:
            return index

Random Search

This feels a little too… systematic, though. What if we didn’t look through in any particular order? Maybe we can just, you know, cast about at random.

import random
def randomseek(array, x):
    index = random.randint(0, len(array) - 1)
    while array[index] != x:
        index = random.randint(0, len(array) - 1)
    return index

This does have a problem. If the element isn’t in the array, it will never terminate! So, we need to keep track of whether or not we’ve searched the whole array.

It’s pretty easy to add in a hash table to do this. Every time we look at an index, we add it to the hash table, and we keep searching until the hash table is the same size as the input array.

import random
def randomseek(array, x):
    already_looked = {}

    while len(already_looked) < len(array):
        i = random.randint(0, len(array) - 1)
        if array[i] = x:
            return i
        else:
            already_looked[i] = True

Beautiful! Now, let’s figure out exactly how bad this is. Python has first-class functions, which is a boon for comparing algorithms. We can create a simple function to run a bunch of searches and time them.

Benchmarking

In essence, this function just takes a function of the form of the ones above, and runs it a thousand times, averaging the time per element and printing it.

def test_seek(f, name):
    total_elems = 0
    total_time = 0
    size = 5000
    arr = [random.randint(0, size) for index in range(0, size)]
    for in in range(1, 1000):
        x = random.randint(0, size)
        t1 = time.time
        f(arr, x)
        t2 = time.time
        nanoseconds = (t2 - t1) * 1000000000
        total_elems += n
        total_time += nanoseconds
    print(name + ": On average " + str(total_time/total_elems) + "ns per element")

I also threw in “Sort and Seek”, which runs .sort() (Timsort) on the array and then binary searches it. And the results are in:

Python .index linear search: ~130 ns/element
Custom linear search: ~130 ns/element
Sort and Seek: ~82 ns/element
Randomseek: ~131,831 ns/element

(These results are from my ThinkPad T430s with a Core i5 3320M and 16GB of memory.)

So, yes, randomseek is a very bad search. In fact, it’s almost exactly a thousand times worse than linear search! But it’s still not that bad.

Conclusion

130,000 nanoseconds is not a long time. According to Wolfram Alpha, it’s about a fifth as long as a baseball remains in contact with the bat during a hit, or half the period of the highest tone audible to most humans. We could search through an array of 7,585 items in one second. That’s a lot.

On the other hand, 130,000 nanoseconds is a long time. Light can travel 25 kilometers in that amount of time. It’s 82 times the half-life of radium; in the time our search has finished one item, the amount of radium in a sample will have diminished to basically nothing. It would take almost 12 hours to search a list regarding the whole population of the U.S., which our other searches can do in under a minute.

So, the moral of this story? As Julia Evans tells us, computers are fast. But they are not fast enough to hide our sins. Randomseek is a silly example, but we do lots of things that waste time. Things like reaching out to the network before launching a program that doesn’t need the internet, or launching an entire virtual machine and generic layout and rendering engine to power a chat application. Don’t write randomseek.

Fedidict Implementation: Setting Up the Database

Thu, 06 Sep 2018 12:31:37 -0500

This is the first of several posts on the implementation of my ActivityPub-enabled Rust web application, FediDict. Each of these posts was released to my Patreon patrons before being make publicly available. I recommend that you read the security design post first.

The core of any stateful service is its datastore, and FediDict uses PostgreSQL, a very performant, ACID-compliant, free and open source relational database. In this post, I’ll look at how to make an efficient, normalized database for FediDict’s data.

That’s a lot of jargon, so let’s break it down.

“Performant”, here, means that it is capable of many simultaneous reads, since that is likely to be the main load.
“ACID-compliant” basically means that PostgreSQL won’t lose data, unless I explicitly ask it to.
“Free and open source” is important becuase I want anyone to be able to run FediDict.
“Relational” means that relations, such as “user x owns definition y”, are encoded in the database itself, rather than being something FediDict has to handle at the application level. This is good for both performance and security.

From the Ground Up

FediDict is mosly interested in its list of definitions, and that’s where I’ll begin with the database implementation as well. A definition requires a lot of attributes. These will go in the Definitions table of the database.

Definition:

An arbitrary unique ID value
The term being defined
The part of speech being defined (refers to a part of speech; a foreign key for PartOfSpeech)
The actual text of the definition
A list of related (“see also”) terms
Date of creation
Created by (foreign key for User)
Date of approval (it may not exist; we call this nullable)
Approved by (foreign key for User; nullable)

Normalization

This is a lot of columns for one database table, and furthermore, some of these items depend on one another. For instance, it’s valid to have no approval date AND no approved by user ID, or to have both an approval date and an approved by user ID, but not one or the other.

This is not desirable; it’s better to have such constraints encoded in the database, and this is impossible with a single table solution. Furthermore, a bulk approval could result in multiple records storing data that refers to a single event, which is a violation of database normalization. To avoid this, I’ll split out approval into its own table.

I’ll also replace the two fields for date of approval and approved by user ID with a single nullable foreign key reference for an Approval. If it’s null, the definition has not been approved; if it has a value, the definition has been approved, and the system can look up which user approved it.

Definition:

An arbitrary unique ID value
The term being defined
The part of speech being defined (that is, “kick, n.” is not the same as “kick, v.”)
The actual text of the definition
A list of related (“see also”) terms
Date of creation
Created by (foreign key for User)
Approval record (nullable, foreign key for Approval)

Written in SQL, the language of the database, that looks like:

CREATE TABLE Definitions (
    id SERIAL PRIMARY KEY,
    term TEXT NOT NULL,
    part_of_speech INT NOT NULL,
    definition TEXT NOT NULL,
    see_also TEXT NOT NULL,
    created_at TIMESTAMP NOT NULL,
    created_by INT NOT NULL,
    approval INT,
    FOREIGN KEY (part_of_speech) REFERENCES PartsOfSpeech(id),
    FOREIGN KEY (created_by) REFERENCES Users(id),
    FOREIGN KEY (approval) REFERENCES Approvals(id)
);

Approval:

An arbitrary unique ID value
The user responsible for approving the definition (foreign key for User)
The date on which the definition was approved

In SQL, this looks like:

CREATE TABLE Approvals (
    id SERIAL PRIMARY KEY,
    approved_by INT NOT NULL,
    approved_at TIMESTAMP NOT NULL,
    FOREIGN KEY (approved_by) REFERENCES Users(id)
);

The final definition-related table is PartsOfSpeech. I considered using a hard-coded list of parts of speech, but I realized that this isn’t scalable across languages, so I will definitely need to let users define their own. Fortunately, it’s very simple.

Part of Speech:

An arbitrary unique ID value
The name of the part of speech, like “noun”
The plural of the name, like “nouns”
The symbol for the part of speech, like “n.”

In SQL:

CREATE TABLE PartsOfSpeech (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    plural TEXT NOT NULL,
    symbol TEXT NOT NULL
);

Identity Crisis

The other important entity in the system is the user. Users have already been referred to several times in the schema, but given the discussion in the previous post, how I refer to them is pretty important.

User:

An arbitrary unique ID value
The user’s name
The user’s email
The hash salt for the user’s password
The hashed password used for login
The federation partner associated with this user (foreign key for FederationPartner, nullable)
The date upon which the account was created in FediDict’s database
The RBAC record for the user (foreign key for Roles)

This definition has brought in a new structure, as well: a Federation Partner. That’s what I’m calling the other ActivityPub sites that FediDict will cooperate with. For now, I’ll keep them simple.

FederationPartner:

An arbitrary unique ID value
A domain name for the partner
The date upon which this partner became known to FediDict.

Users with a non-null partner ID will be written as username@partner.domain, while users with a null partner ID are just username.

Here’s how these two look in SQL:

CREATE TABLE Users (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    email TEXT NOT NULL,
    salt TEXT NOT NULL,
    passhash TEXT NOT NULL,
    partner INT,
    created_at TIMESTAMP NOT NULL,
    roles INT NOT NULL,
    FOREIGN KEY (partner) REFERENCES FederationPartners(id),
    FOREIGN KEY (roles) REFERENCES Roles(id)
);

CREATE TABLE FederationPartners (
    id SERIAL PRIMARY KEY,
    domain TEXT NOT NULL,
    created_at TIMESTAMP NOT NULL
);

Finally, I need to define the RBAC table. This is based on the previous post:

CREATE TABLE Roles (
    id SERIAL PRIMARY KEY,
    definition_reader BOOLEAN,
    definition_submitter BOOLEAN,
    definition_evaluator BOOLEAN,
    definition_remover BOOLEAN,
    queue_reader BOOLEAN,
    queue_approver BOOLEAN,
    queue_rejecter BOOLEAN,
    account_creator BOOLEAN,
    account_remover BOOLEAN,
    account_role_assigner BOOLEAN,
);

And that’s it!

Next Steps

Now that the database is defined, I need to actually create it. I’m going to use the Diesel database mapper for Rust. In essence, Diesel allows me to write Rust code that is automatically converted into SQL, with all the type safety benefits of Rust code.

Installing Diesel

Most Rust crates are just a Cargo.toml edit away, but Diesel is a bit more complex, as it also has a CLI tool which I’d like to use. So, I first have to:

$ cargo install diesel_cli

Note that this requires both the mysql and postgresql client libraries be installed on my system, despite the fact that I’m only using PostgreSQL.

Now I do need to add it to my Cargo.toml, along with some other libraries. I’m using serde for serialization and deserialization, along with its code generator from serde_derive and JSON functionality from serde_json. I’m also using dotenv to configure the database with a .env file and chrono to handle dates and times.

[package]
name = "fedidict"
version = "0.1.0"
authors = ["Leonora Tindall <nora@nora.codes>"]

[dependencies]
serde = "1"
serde_derive = "1"
serde_json = "1"

diesel = { version ="1", features = ["postgres"] }
dotenv = "0.13"
chrono = "0.4"

Installing PostgreSQL

I’m going to use Docker to handle the database.

docker create -p 127.0.0.1:5432:5432 \
    --name fedidict-db \
    -e POSTGRES_PASSWORD="password" \
    -t postgres:alpine

docker start fedidict-db

And, in a file called .env in the project folder, I’ll make an environment variable:

DATABASE_URL="postgresql://postgres:password@localhost/fedidict"

and then run source .env to get this variable in my shell.

Now I can use Diesel to set up the database. This creates both the database and a directory to hold SQL files for database setup.

diesel setup

Heading North

Diesel’s CLI tool helps manage database state, and in fact I’ll use it to create the database to begin with.

I’ll add a new migration using diesel_cli and put some SQL code in it telling Diesel how to set up (and tear down) the database. This is the only SQL I expect to write, since Diesel will handle interaction with the database in most cases.

$ diesel migration generate initial_schema
Creating migrations/2018-09-10-170446_initial_schema/up.sql
Creating migrations/2018-09-10-170446_initial_schema/down.sql

In Diesel, migrations go both “up” and “down”; up migrations add new functionality, while down migrations remove that functionality.

The up migration is the combination of the SQL I’ve written so far in the post, but the order is important; tables must be created in dependency order. For instance, Approvals must be created before Definitions because Definitions has a foreign key that points to Approvals.

The order I decided on is PartsOfSpeech, FederationPartners, Roles, Users, Approvals, Definitions.

The down migration is much simpler. I simply drop all the tables I created in tue up migration.

DROP TABLE Roles, Definitions, Approvals, PartsOfSpeech, Users, FederationPartners;

It’s good practice to test migrations by running them and then rerunning them.

$ diesel migration run                                   
Running migration 2018-09-10-170446_initial_schema
$ diesel migration redo 
Rolling back migration 2018-09-10-170446_initial_schema
Running migration 2018-09-10-170446_initial_schema

This verifies that the down migration at least doesn’t leave anything obvious hanging around.

After running this migration, two new files will show up: diesel.toml, Diesel’s config file, and src/schema.rs, the Rust mapping of the database schema.

That’s it - the database is set up. In the next post, I’ll discuss how I map these data structures into Rust, how I test them, and what behavior I need to set up for them.

FediDict UX Design, Part 2

Sun, 02 Sep 2018 00:00:00 +0000

This is the second of several posts on the design of my current ActivityPub-enabled Rust web application, FediDict. Each of these posts was be released to my Patreon patrons before being made publicly available. I recommend that you read the first UX design post first.

Any software that accepts input over the network will, eventually, be subject to attack, of some kind. Any federated software service must accept input over the network - that’s the whole point. In the case of FediDict, these attacks could come from either the user-facing side or the federation-facing, or server-to-server, side.

Either way, some of the technology choices I’ve made will help FediDict withstand attack, and in this post, I’ll explore how a solid access control model can help secure a system while still enabling rich control by administrators and a good user experience.

Inherent Sturdiness

FediDict is built in Rust, a language that promises memory safety for all programs written without the unsafe keyword. This eliminates (or, at least, mitigates to the point of insignificance) the majority of common memory-based attacks, like buffer overruns, use-after-free bugs, double-free bugs, and other such issues.

I’m also using a proven database system, though I’m not sure if PostgreSQL or MariaDB will be my final choice yet. Either one has a track record of high security, and FediDict will interface with them using Diesel, an object-relational mapping library which uses static analysis to ensure that certain types of database errors and bugs (like SQL injection) are not possible.

Access Control

Nonetheless, bugs exist in any piece of software, and application-level bugs are often far more impactful than are those in underlying code, precisely because they are so hard to predict and expose precisely what the attacker wants - the data held in the application.

One of the best ways to mitigate application-level bugs is to have a consistent (and consistently enforced) access control model. Rust’s type system helps me here.

A Pyramid Model

A dictionary produced by a small team of professionals is an easy thing to secure, but a crowdsourced (or, a semi-crowdsourced) one is not. In FediDict, I’ll have a few basic levels of access:

The Guest. A guest has absolutely no write access, at all, but they have read access to most things. I’ll define exactly what later.
The Contributor. A contributor has all the privelages a Guest has, as well as write access to the moderation queue (but, you’ll note, not to the site directly), and to the vote count of existing definitions, and to the report queue.
The Moderator. A moderator has all the privelages a Contributor has, as well as being able to read the moderation queue and write from the moderation queue to the public database of definitions.

This is a tidy and convenient pyramid of access. Each level builds on the last, each group is smaller than the last, and all these privelages can be enforced with simple conditions, even at the database level.

Access becomes slightly more complex when I bring federated users into the mix. Because ActivityPub supports “liking” posts remotely, and I wish to support this as well, some people can write to the vote count of definitions without being Contributors.

Roles, not Tiers

This would appear to pose a problem, but in fact, it’s easy to solve with a concept known as Role Based Access Control, or RBAC. RBAC is a technique used by many complex systems, including Amazon Web Services and Google Cloud, but is applicable to simpler systems as well.

With RBAC, I’ll define a set of roles and assign those roles to users based on various factors. Then, when a user attempts to perform an action, FediDict will check for these roles. For instance, the roles could be:

DefinitionReader: can access the public database of definitions.
DefinitionSubmitter: can submit new definitions to the moderation queue.
DefinitionEvaluator: can submit likes and reports for definitions.
DefinitionRemover: can remove definitions from the public database.
QueueReader: can read definitions submitted to the moderation queue.
QueueApprover: can move definitions from the moderation queue to the public database.
QueueRejecter: can remove definitions from the moderation queue.
AccountCreator: can create new local FediDict accounts.
AccountRemover: can remove local FediDict accounts.
AccountRoleAssigner: can assign and remove roles on accounts.

I could use these roles in Rust through an enum. I’m not sure just how it will look, but for example:

impl Definition {
    // ... other code for definitions
    /// approve() sets a definition as approved, by the authority of the
    /// given user.
    fn approve(self, authority: &User) -> Result<Definition, Error> {
        if authority.has_role(Roles::QueueApprover) {
            let mut new = self;
            new.approved_by = Some(authority.id);
            return Ok(new);
        } else {
            return Err(Error::PermissionDenied(
                        Roles::QueueApprover,
                        "approve a definition from the moderation queue");
        }
    }
}

This could then be used in a pipeline. For instance:

// Assume I have user, the current user, and def, a definition, from elsewhere
// in the code.
def.approve(&user)?.commit()?;

In the case of an error, the information given is enough to render a message like:

Error: Permission denied. In order to approve a definition from the moderation queue, you need the QueueApprover role, which you do not have. Contact an administrator for assistance.

More Flexibility Without (Much) Complexity

I can still model tiers with these roles. A Guest has only the DefinitionReader role, while a Contributor has DefinitionReader, DefinitionSubmitter, and DefinitionEvaluator. A Moderator has all of those roles, in addition to QueueReader, QueueApprover, and QueueRejecter.

I can also construct other types of user from these roles. For instance, a site admin needs AccountCreator, AccountRemover, and AccountRoleAssigner, but could delegate account creation to others by giving them only AccountCreator.

Someone the organization trusts to remove spam but not to fully evaluate the validity of new definitions could have QueueReader and QueueRejecter but not QueueApprover.

A user interacting over ActivityPub can like and report as well as viewing the database, so they would get DefinitionReader and DefinitionEvaluator, but not any other roles.

With roles, I get the security provided by a solid tiered model with a flexibility that a tiered model can never provide.

Implementing Roles

This might seem like a difficult technique to implement, but actually, it’s very natural, especially using the combination of Rust’s expressive type system and a powerful RBDMS like PostgreSQL.

Each user will have a database record in the Users table with an associated ID. That record will specify their user name, e-mail address, (hashed) password, and any other information, including the ID of a row in the Roles table.

The Roles table will associate user IDs with a set of roles. Each role can be on, off, or none. This maps naturally to a nullable Boolean value in SQL, or to an Option<bool> in Rust.

Say, for instance, ther are three users, an admin, a moderator, and a contributor. The users table might hold three records, like so:

ID	Username	email	password	roles
12	ltindall	a@b.c	d7239dhs	02358
87	chughes	c@x.y	a2342nb0	83950
34	djanes	q@m.z	passw0rd	22741

And, the Roles table would have some associated records:

ID	definition-reader	definition-submitter	queue-reader	queue-approver	account-creator
02358	null	yes	yes	yes	yes
83950	null	yes	yes	yes	no
22741	null	null	null	null	null

Here, some roles have explicit values and some do not. For instance, entry 83950 (associated with UID 87, user chuges) has no explicit value (null) for definition-reader and explicit values for all the other fields, while entry 22741 (associated with UID 34, user djanes) has default values everywhere.

This is valuable because, if a problem with the default settings is discovered, it would be quite difficult to set the roles for every user created before that point. With this system, values are only non-null if they have been explicitly set; null values resolve to defaults set elsewhere.

This approach also provides the flexibility I was looking for. If I decide to trust djanes with creating accounts but not, say, approving new definitions, I need only change the account-creator column of the Roles table to yes (or true, or however the database stores Boolean values) and that account can now create new accounts.

There is a major issue with this model, however. It does not have a good way to handle access control for users with no identity, or with an identity I don’t know about until they first interact with us over a federation protocol.

Anonymous Identity

Anonymous users - those interacting through an unauthenticated web interface and without identity provided by a federation partner - are actually pretty easy to handle. I already laid out the need for default permissions for logged-in users, so I can just add another set of defaults for not-logged-in users (probably just DefinitionReader).

New Introductions

Like other federated systems (such as e-mail), an ActivityPub identity is determined by both a username (like ltindall) and a domain (for instance, nora.codes). Assuming I hosted a FediDict instance on computerjargon.online with the user accounts shown above, their full federated usernames would be ltindall@computerjargon.online, chughes@computerjargon.online, and djanes@computerjargon.online. These are identities ComputerJargon.Online already knows about, but the system could get a “like” from a user on another ActivityPub-supporting instance at any time.

Let’s say Eugen, creator of Mastodon, follows the definitions created by users of ComputerJargon.Online and sees a definition he really likes. He could click the “like” button in the Mastodon UI, which would eventually notify FediDict that a like had been sent by gargron@mastodon.social. Assuming ComputerJargon.Online had never interacted with Eugen before, he would not have an account in the ComputerJargon.Online database. So, how does FediDict determine his access rights?

Defaults come to the rescue again. With a set of defined defaults for accounts coming in from federation partners, FediDict simply has to create a new record for Eugen’s account in the accounts table (noting its remote origin) with a null access control row, giving it default values for remote accounts.

A Holistic View

These considerations alone are not enough to make a system secure, but with good underlying technologies and a solid access control model, which I’ll refine in future posts and as I begin to write more code, it should be easy to keep egregious bugs out and fix and issues that do make it into the codebase.

Security comes in layers, and this post has laid out a few of them: one at the very bottom, the language and underlying technology used to build the system, and one near the top, the access control model. I can fill in the rest as time goes on.

FediDict UX Design, Part 1

Thu, 23 Aug 2018 18:04:14 +0000

This is the first of several posts on the design of my current ActivityPub-enabled Rust web application, FediDict. Each of these posts was be released to my Patreon patrons before being made publicly available.

FediDict, a portmanteau of Federated and Dictionary, is my current open source project. Most of the time, I’ve taken the approach of “dive in and write some code; the design will shake itself out”. This isn’t a great idea for something I want to be widely adopted, so I took a different approach for this project.

I do have some code written - mostly dealing with Diesel and the database, as well as simple HTTP server boilerplate - but I have yet to work on the meat of the application. In this series, I’ll walk through some of the design work.

What does it do?

FediDict lets people define jargon that they know and look up jargon that they don’t, and share those definitions between disciplines and across the web.

‘Jargon’ is an important word here; I want FediDict to help people who are reading a text from a specific discipline find out the meaning of words they don’t know, and get some context on those words in other disciplines as well.

While the word “proband” probably only has meaning in the medical community, something like “stall” means one thing in aviation and another in automotive repair. Some words, like “normal”, are so overloaded as to have multiple meanings in multiple fields. Wikipedia alone lists six in psychology, one in chemistry, two in physics, and upwards of 25 in various fields of mathematics, including linear algebra, geometry, and statistics!

Such words can be confusing if misinterpreted, so I’d like each instance of FediDict to connect to others. That way, they can send definitions around, with attribution, so that each FediDict instance has access to the whole corpus of information while making it clear where each definition comes from.

Seeking Knowledge

The Guest

A guest coming to the FediDict site just wants to know a specific piece of information. Let’s imagine that I host “computerjargon.online” and that Danielle A. Hacker wants to know what a “PIC” is. She knows that computerjargon.online is the place to be to learn about computer jargon, so she enters that URL.

My top priority here is to get out of her way and let her get the info she needs. She needs to have a big search bar, front and center, where she can start typing the word she wants to define, so we need a distraction-free home page with search bar.

While she’s typing, she might wonder if the ComputerJargon FediDict instance even has the definition for “PIC”. To reassure her, and to make typing long terms (or terms a user doesn’t fully know how to spell) easier, it would be great to have some kind of typeahead for terms the FediDict instance already knows.

Once the term is entered, FediDict needs to search the database for all the terms matching the search, and display them.

It’s also important to have some way to order these terms. In order to maintain relevance, the terms from the instance the user actually searched on should come first. In addition, there should be some mechanism for determining whether or not a given definition is accurate and relevant. For now, that’s the “score”; manipulating it comes later.

The Member

Imagine that another person, Jake P. Grammer, wants to look up the same information. He has a good idea that a “PIC” is a programmable something, but he isn’t sure what. He goes through basically the same workflow as Danielle the guest, but with two pre- conditions: he’s made an account, been approved by the moderation team, and signed in.

He navigates to computerjargon.online, enters “PIC” in the search bar, and gets a list of results, sorted as mentioned above. Unlike Danielle, however, he has the option to voice his opinion on the relevance of the results. Let’s say the results are as follows.

PIC, n., by duck3345

A PIC, or photonic integrated circuit, is an electronic component which uses light rather than electricity for its internal operation.

PIC, n., by user123

A PIC, standing for programmable interrupt controller, is a component of a computer which does x, y, z, and q, and blah blah.

PIC, n., by farmer1372

PIC, particulate inorganic carbon, is some chemistry thing that Nora doesn’t know about, but it’s certainly not relevant to ComputerJargon.online’s database.

The first definition is totally irrelevant to Jake, but he doesn’t know if it’s wrong, and it doesn’t break any rules. He does, however, know that the entry by user123 on programmable interface controllers is relevant to his needs. He should be able to signal this by liking the definition. This action is associated with his account and increments the score of the post.

The third definition might also catch his eye. It makes no sense to have a chemistry term on the ComputerJargon database, and Jake wants to help out the site by making the moderator team aware of this. He should be able to do that by reporting the definition, another action associated with his account. It will reduce the score of the definition, as well as sending a message of his choosing to the moderators of the site.

Jake has successfully gotten the information he needs, and has contributed some contextual information back to the site. As more and more users use the site and look up “PIC”, the most domain-relevant information will be liked most often and will drift to the top of the list, while lesser-used information will drift downwards and totally irrelevant information (and downright rule-breaking) can be deleted by moderators after being reported.

Perhaps another user of the site, Eric D. Velopoer, just attended a seminar on frobnitzem and wants to make sure that everyone knowns the definition of “frobnicate”.

He follows the basic flow of the other users to determine whether or not the definition already exists in the database. In this case, it doesn’t exist in the ComputerJargon database, but it might exist on other instances. Whether or not it does, the result of the search is clear.

Once he has found that the definition doesn’t exist, Eric has a clear path to create a new definition. (He also has the option to add a new definition for which terms already exist.)

Note that I deliberately do not want people to be able to add definitions without searching for them first, because that is likely to lead to duplication.

Once Eric has decided to add a new definition, he’s presented with a form to add the definition. It has a prefilled, uneditable entry with the term being defined, a dropdown for part of speech (verb, noun, etc), and a large text box for the definition.

It might also be useful to provide a way to enter related terms or “see also"s for a definition. For instance, Eric might want to mention that “frobnicate” is related to “frobnitz”, “twiddle”, and “tweak”.

Eric’s definition might end up looking like this (from the Original Hacker’s Dictionary):

frobnicate, v.

To manipulate or adjust, to tweak. Derived from “frobnitz”. Usually abbreviated to frob. Thus one has the saying “to frob a frob”. “Frob”, “twiddle”, and “tweak” sometimes connote points along a continuum. Frob connotes aimless manipulation; twiddle connotes gross manipulation, often a coarse search for a proper setting; tweak connotes fine-tuning. If someone is turning a knob on an oscilloscope, then if he’s carefully adjusting it he is probably tweaking it; if he is just turning it but looking at the screen he is probably twiddling it; but if he’s just doing it because turning a knob is fun, he’s frobbing it.

See also: frobnitz, tweak, twiddle.

That definition, once submitted, goes onto a moderation queue, so that the site moderation team can decide whether or not to allow it to go public.

Next Steps

These are the basic workflows for FediDict users - looking things up, giving feedback on existing definitions, and submitting new definitions. In the next post, we’ll look at the security aspect of design, including anti-abuse tooling.

PDF Embedding Attacks

Sat, 04 Aug 2018 12:17:11 -0500

PDF, or Portable Document Format, is an incredibly complex file format, governed by many standards and semi-standards. Like HTML and CSS, it was primarily designed for document layout and presentation. Also like HTML and CSS, it has been augmented with a JavaScript engine and document API that allows programmers to turn PDF documents into applications - or vehicles for malware.

Embedding Files in PDF Documents

It’s very easy to embed any kind of file in a PDF document. Every document includes the EmbeddedFiles name tree, along with support for collections of files, known as portfolios.

Most PDF libraries provide support for this; we’ll examine PyPDF2, which supports everything we need and is pure Python.

PyPDF2’s PdfFileWriter provides a method called addAttachment which takes a name and some bytes and embeds them as a file in the PDF (docs).

This is how malware is usually concealed in a PDF document - as an embedded file.

Opening Files from PDF Documents

Now that we have a payload embedded in a PDF document, we need to actually open it. The basic method for this is to also embed a script in the PDF document. In our case, we want to add a document level script. This script will execute as soon as the PDF is opened.

Fortunately, PyPDF2 also supports this! We can simply add a JavaScript object with the method addJS, and that JavaScript will be registered to run on the PDF opening.

Our JavaScript payload is pretty simple: we just add a single call to exportDataObject, a function provided by the PDF reader. This function takes an object with 2 parameters:

cName, the name of the embedded object, and
nLaunch, an instruction as to what the PDF reader should do with the exported object

nLaunch is just an integer, and it has three valid values:

Prompt the user for a path and save the file there
Prompt the user for a path, save the file, and ask the operating system to open it
Pick a temporary location, save the file there, and ask the operating system to open it

That last option sounds great for malware. Assuming we embedded a file called myExploit.exe, we would add the following JavaScript:

this.exportDataObject({
    cName: "myExploit.exe",
    nLaunch: 2,
});

and it would run as soon as the PDF was opened, right? Well, not quite. Unfortunately, there’s a bit more to it; Adobe Reader (and most other readers) will prevent the launch of common executable files. For example, .exe, .js, .vba, and .bat files cannot be opened.

Evading the Blacklist

There are many ways to evade the blacklist, such as Microsoft Word documents with malicious macros embedded in them (read more), but recently, researchers discovered that another kind of file could be used: .SettingContent-ms. As explained by the SpecterOps team, these are just XML documents pointing to specific places in the Windows 10 settings GUI. However, they contain a field called DeepLink which can contain any arbitrary executable which will be run when the SettingsContent-ms file is executed.

Deploying a Payload

Let’s bring this all together. How would we use this in a real attack?

There are a few steps we need to perform in order to get this attack working.

1: Create a Payload

There are many great ways of creating effective payloads. In this case, I’ll assume you have a payload already; my payload of choice is a Meterpreter reverse shell encoded with some type of cloaker.

2: Encode the Payload in Base64

We can use certutil.exe -encode InputFile EncodedFile on Windows or base64 input > output on Linux to encode and decode files with Base64. This will let us more readily insert it where it is needed.

3: Embedded Files

We need to run 3 commands, so we’ll use the above method to embed three files. They will all be valid SettingContent-ms XML, differing only in the DeepLink node.

PutFile.SettingContent-ms which will echo the Base64 encoded payload to disk at a known path (echo b46c827y... > %APPDATA%\evil.b64).
Decode.SettingContent-ms which will decode the Base64 encoded payload into an EXE ( certutil -decode %APPDATA%\evil.b64 %APPDATA%\evil.exe)
Execute.SettingContent-ms which will actually run that file. (%APPDATA%\evil.exe)

4: Scripts

Now, we need a single script which will run all of these:

var files = ["PutFile", "Decode", "Execute"];
for (var i = 0; i < files.len; i++) {
	this.exportDataObject({
		cName: files[i] + ".SettingContent-ms",
		nLaunch: 2,
	});
}

5: Pwnage!

Now we have our Metasploit payload running on the target! Congratulations, time to move into post-exploitation.

Alternative Methods

Another method would be to use Reflective PE Injection, converting a PowerShell process into the process for our executable. To learn more about this, check out this post.

Conclusion

PDF files are extremely complex, and the applications that read them tend to be old and full of cruft, designed without security in mind. A black-listing method will never be effective in eliminating all dangerous files, as we can see here. This technique requires no exploits; we just ask the OS to run a file for us and it does so!

Building Readable Tests with Fluent Testing APIs

Mon, 23 Jul 2018 09:56:24 -0500

One of the biggest challenges in software testing is defining the input for code under test in a way that is expressive and powerful enough to test complex situations but doesn’t distract from the intent of the test or clutter the test code to a degree that makes it difficult to read.

Many dynamic languages have testing APIs which take advantage of their looser and later type checking to provide easy mocking and stubbing, but strict, statically typed languages can make it difficult to build up suitable instances of the types needed in the test. This is especially true with nested data.

Motivation

At CancerIQ, we have a large body of tests to ensure we correctly implement our risk models. These tests are thorough and give us a high confidence that new features or refactors aren’t breaking old functionality.

We do have one problem, however: fragility and verbosity. Our tests are very complete but, until the changes described here, they didn’t express what they were testing well, and they were very difficult to change.

In this post, you’ll see how we got from fragile and hard-to-understand tests to short, expressive, and functionally equivalent tests with fluent-style testing APIs.

The Literal Approach

Originally, these tests were written using literals. For instance, consider a Person that looks something like this:

struct Person {
    pub id: String,
    pub age: Option<u8>,
    pub mother_id: Option<String>,
    pub father_id: Option<String>,
    pub children: Option<Vec<String>>,
    pub medical_history: Option<MedicalHistory>,
    pub reproductive_history: Option<ReproductiveHistory>,
    // ... and many more fields
}

Of course, the MedicalHistory and ReproductiveHistory structs are themselves nontrivial, and because we are dealing with data from inconsistent sources (patient reporting), almost every field is Optional. Our tests ended up looking something like this:

#[test]
fn patient_with_field_that_matters_value_is_flagged() {
    let patient = Patient {
        id: "1".into(),
        age: None,
        mother_id: None,
        father_id: None,
        children: None,
        medical_history: Some(MedicalHistory{
            diagnoses: vec![Diagnosis {
                type: DiagnosisType::SomeIllnessOrOther,
                field_a: None,
                field_b: None,
                field_that_matters: Some(31337),
                field_d: None,
            }]
        }),
        reproductive_history: None,
        // Followed by many more Nones
    };
    assert!(function_under_test(patient),
           "Function did not return True for patient who should be at risk!");
}

Is it clear what we’re doing here? Not really. It turns out, we only care about one field here: field_that_matters, on the Patient’s MedicalHistory. In addition, this test is 21 lines long - far too long for a test which is only evaluating a single function call.

More importantly, though, this test is totally inflexible. Whenever we had to add a field to the Diagnosis struct, our tests stopped compiling, even though the field was Optional and not important here. With hundreds of such fragile tests in the codebase, making even basic changes became very slow.

First Steps

The most obvious way to solve both fragility and verbosity was to encapsulate this initialization into a function, which we did. These functions took some basic info and then tests with more complex requirements would add the properties they needed. For instance:

fn patient_with_field_that_matters_is_flagged() {
    let mut patient = make_basic_patient("1", None, None, None, None);
    let diagnosis = make_diagnosis(
        Diagnosis_Type::SomeIllnessOrOther,
        None,
        None,
        Some(31337),
        None);
    patient.medical_history = MedicalHistory{
        diagnoses: Some(MedicalHistory{
            diagnoses: vec![diagnosis]
        })
    }
    assert!(function_under_test(patient),
           "Function did not return True for patient who should be at risk!");
}

This approach does make the test more concise, but it actually reduces readability due to the large number of unnamed function arguments. Without looking up the signature of make_diagnosis, a reader has no way to know that the value given is going into field_that_matters.

This can also be done with Rust’s struct update syntax, either by implementing Default or with an explicit ::base() associated function:

fn patient_with_field_that_matters_is_flagged() {
    let patient = Patient {
        id: "1".into(),
        medical_history: Some(MedicalHistory{
            diagnoses: Some(vec![ Diagnosis {
                field_that_matters: Some(31337),
                .. Diagnosis::base(DiagnosisType::SomeIllnessOrOther)
            }])
        }),
        .. Default::default()
    }
    assert!(function_under_test(patient),
           "Function did not return True for patient who should be at risk!");
}

Now, only the ::base() method or the implementation of Default has to be modified when a struct field is added or modified. This is a good first step, but it doesn’t fully solve the problem of clarity.

Taking Inspiration

I don’t have a lot of occasion to work with dynamic languages, but I was pairing with another engineer on some Ruby code and noticed how nice the testing API was with RSpec. This lead me down a rabbit hole which eventually brought me to fluent programming.

In short, the idea of a fluent API is that it can be read almost like a sentence in a natural language. We built an API with a similar idea, which is what is currently used in the risk model tests. They look something like this:

fn patient_with_field_that_matters_is_flagged() {
    let patient = Patient::base("1").with_medical_history(
        MedicalHistory::base().and_diagnosis(
            Diagnosis::base(DiagnosisType::SomeIllnessOrOther)
                .with_field_that_matters(31337)
        )
    );
    assert(function_under_test(patient),
           "Function did not return True for patient who should be at risk!");
}

The advantages of this approach become even clearer when working with tests that require many associated structs. For example, consider this code:

let patient = Person::base("1", Gender::Male)
    .with_father("2")
    .with_mother("3")
    .with_children(&["4", "5"])
    .with_age(53);
let father = Person::base("2", Gender::Male).deceased_at(84);
let mother = Person::base("3", Gender::Female).with_medical_history(
    MedicalHistory::base()
        .and_diagnosis(Diagnosis::base(DiagnosisType::CommonCold).at_age(32))
        .and_diagnosis(Diagnosis::base(DiagnosisType::AvianFlu)
            .with_treatment(Treatment::base(Treatment::SomeFluTreatment))));
let child1 = Person::base("4", Gender::Female);
let child2 = Person::base("5", Gender::Female)
    .with_mutation(GeneticMutation::BlueHairAllele_pg5s77);

This is a lot of information! We’ve effectively defined a DAG (a family tree) in fewer lines than our original test, and it’s trivial to read through this.

“Who’s the father?” “Oh, he’s a male who died at 84.”

“Who’s the mother?” “She’s a woman who got the cold at 32 and was treated for avian flu.”

“Who’s the patient?” “He’s a 53 year old man with two children.”

The critical idea here is that the API doesn’t require specifying that you don’t have some information; rather, you use methods to specify which pieces of information you do have.

Implementation

Implementation of these methods is efficient and straightforward in Rust, using the aforementioned update syntax. Consider the following struct:

struct ShippingItem {
    pub volume: f64,
    pub condition: Option<Condition>,
    pub sender: Option<String>,
    pub recipient: Option<String>,
    pub certified: Option<bool>,
    pub contents: Option<Vec<InventoryObject>>,
    // et cetera
}

This struct has 1 required field and several optional fields. Because of the required field, just implementing Default isn’t an option, so the ::base() method is the way to go:

impl ShippingItem {
    pub fn base(volume: f64) -> ShippingItem {
        ShippingItem {
            volume,
            condition: None,
            sender: None,
            recipient: None,
            certified: None,
            contents: None,
        }
    }
}

Then, for each basic Optional field, we’ll add a method. This includes condition, sender, and recipient.

impl ShippingItem {
    pub fn with_condition(self, cond: Condition) -> Self {
        ShippingItem {
            condition: cond,
            ..self
        }
    }

    pub fn with_sender(self, sender: &str) -> Self {
        ShippingItem {
            sender: sender.into(),
            ..self
        }
    }

    pub fn with_recipient(self, recipient: &str) -> Self {
        ShippingItem {
            recipient: recipient.into(),
            ..self
        }
    }
}

Each of these takes ownership of self and returns a new ShippingItem, allowing the creation of long sentence-like descriptions.

For boolean values, two methods are preferrable.

impl ShippingItem {
    pub fn certified(self) -> Self {
        ShippingItem {
            certified: Some(true),
            ..self
        }
    }
    
    pub fn not_certified(self) -> Self {
        ShippingItem {
            certified: Some(false),
            ..self
        }
    }
}

Optional lists provide something more of a challenge. We settled on a two-method API, providing one method for replacing the contents with a slice and one for adding a single item.

impl ShippingItem {
    pub fn with_contents(self, contents: &[InventoryObject]) -> Self {
        ShippingItem {
            contents: Some(contents.into()),
            ..self
        }
    }

    pub fn and_contents(self, object: InventoryObject) -> Self {
        let contents = self.contents.unwrap_or(Some(Vec::new()));
        contents.push(object);
        ShippingItem {
            contents,
            ..self
        }
    }
}

Now we can write our tests with this API! For example, let’s say we wanted to test a shipping validation function. We can specify only the required information, rather than all the possible fields.

#[test]
fn same_sender_and_receiver_not_shippable() {
    let item = ShippingItem::base(1.0)
        .with_sender("Somebody T. Something")
        .with_receiver("Somebody T. Something"); 
    assert!(!item.validate());
}

Conclusion and Use Cases

This technique isn’t for every project. The main drawback is that it requires a fair amount of boilerplate for each struct, and that the ::base() method needs an argument for every required item in the struct (as opposed to a builder-style pattern).

However, for any system in which you need to test a lot of computations on a struct with a lot of optional fields, you should consider this fluent-style testing pattern. It’s made our tests more readable, easier to write, and easier to modify, and it can probably do the same for you.

Note: This post was written while I was working at CancerIQ as a software engineering intern, and was also posted on their engineering blog. You can find it here.

Chaining Fallible Operations with Combinators

Tue, 26 Jun 2018 12:54:53 -0500

Rust’s Iterator trait is one of its most useful features. It allows lazy processing of item-by-item streams of anything from the bytes of a file to threads to complex and exotic data structures.

Most of the useful functionality, though, is provided by combinators, functions that allow us to combine iterators and process them in useful ways. These include map, fold, filter, and many other useful functions (including those from the excellent itertools crate).

Concision and Clarity

Consider that we have the following functions which we’d like to apply to some list of numbers:

fn operation_one(a: i32) -> i32 { a + 1 }
fn operation_two(a: i32, b: i32) -> i32 { a + b }
fn check(a: i32) -> bool { a > 3 }

That is, we want to apply operation_one to every item, apply operation_two to every pair of those results, and finally return only the results which satisfy the condition in check.

Procedurally, that looks a bit like this:

fn procedural(values: &[i32]) -> Vec<i32> {
    let mut temp = Vec::with_capacity(values.len());
    for value in values {
        temp.push(operation_one(*value));
    }
    
    let mut temp2 = Vec::with_capacity(values.len());
    for index in 1..temp.len() {
        temp2.push(operation_two(temp[index - 1], temp[index]));
    }
    
    let mut temp3 = Vec::with_capacity(values.len());
    for value in temp2 {
        if check(value) {
            temp3.push(value)
        }
    }
    
    return temp3;
}

A function with that signature must allocate at least once (to have a Vec to return), but this function has three allocations, three mutable bindings, and is 19 lines long! Surely we can do better with functional programming:

extern crate itertools; // 0.7.8
use itertools::Itertools;
fn functional(values: &[i32]) -> Vec<i32> {
    values.iter()
        .map(|x| operation_one(*x)).tuple_windows()
        .map(|values: (i32, i32)| operation_two(values.0, values.1))
        .filter(|x| check(*x)).collect()
}

Even including the imports, this is 11 lines shorter and, in my opinion, far more readable. It also has only a single allocation (in the collect() call) and no mutable bindings.

So, all is rosy and fun with functional programming, right? Not quite. Enter fallibility.

Fallible Maps

I work with microservices a lot, and one of the big problems with microservices is the introduction of system boundaries where all type information is lost.

For instance, many architectures involve passing around JSON, and therefore every ingestion routine has to consider the possibility that it was called with nonsensical input; JSON that parses is not necessarily JSON that makes sense.

Often, the desired behavior is to simply ignore the problematic input. This is easy with flat_map:

// Given that parse() return a Result&lt;Item, whatever&gt;
let parsed: Vec&lt;Item&gt; = inputs.iter()
    .flat_map(|input| parse(input).into_iter())
    .collect();

This results in throwing away the errors, which may not be what we want. We could filter with a match expression, or do other iterator magic, if we simply wanted to log the errors, but what if we want to abort on error?

Fortunately, Result<_, _> implements a trait called FromIterator, which allows the collect() iterator method to collect into a result, inverting the containment relationship:

let items_or_errors: Vec&lt;Result&lt;Item, Error&gt;&gt;;
let single_error_or_items: Result&lt;Vec&lt;Item&gt;, Error&gt; =
    items_or_errors.into_iter().collect();

This is really cool! In cooperation with the ? operator, this can be used to easily, efficiently and concisely abort a function as soon as an error value is encountered - as long as you only have to do it once.

Chaining Fallible Operations

In a real-world scenario, there are often a number of fallible operations that have to be performed. For example, the microservice that inspired this post had to:

Ingest untrusted input into a list of unnormalized input values, possibly rejecting the input as unparsable
Normalize the input values, possibly rejecting them as unnormalizable
Perform a computation on the values, possibly rejecting them as uncomputable
Aggregate the results into a single resulting value

Because of the nature of the service, the correct behavior was to abort as soon as a nonsensical, unnormalizable, or unprocessable value was encountered.

struct DataUnnormalized { data: i32 }
struct Data { data: i32 }

fn ingest(input: String) -&gt; Result&lt;Vec&lt;DataUnnormalized&gt;, String&gt; {
    // In reality, this can also fail
    Ok( vec![DataUnnormalized { data: 1 }] )
}

impl DataUnnormalized {
    fn normalize(self) -&gt; Result&lt;Data, String&gt; {
        // In reality, this can fail
        Ok(Data {data: self.data})
    }
}

impl Data {
    fn perform_operation(&self) -&gt; Result&lt;i32, String&gt; {
        // In reality, this can fail too
        Ok(self.data)
    }
}

fn aggregate(agg: i32, curr: i32) -> i32 {
    agg + curr
}

Unfortunately, composing these functions wasn’t as simple as I expected. I eventually came up with the following solution:

fn compute(input: String) -> Result&lt;i32, String&gt; {
    Ok(ingest(input)? // Ingest and abort if error. Simple enough
        .into_iter()
        .map(|i| i.normalize()) // Maps into Vec&lt;Result&lt;Data, String&gt;&gt;
        .collect::&lt;Result&lt;Vec&lt;Data&gt;, String&gt;&gt;()? // Flatten and abort if error
        .into_iter()
        .map(|i| i.perform_operation()) // Maps into Vec&lt;Result&lt;i32, String&gt;&gt;
        .collect::&lt;Result&lt;Vec&lt;i32&gt;, String&gt;&gt;()? // Flatten and abort if error
        .into_iter()
        .fold(0, |agg, curr| aggregate(agg, curr?))) // Aggregate
} ``` 

This is... not good. It has two allocations which are effectively unnecessary, and are only there to convert between types. It has lots of syntactic noise, and is difficult to read.

After asking around, I found out that I was thinking about this wrong. Rather than mapping from success to success/failure, I should have started out by embracing the fallibility.

Making each `map` operation take a `Result` type makes this much prettier, and with the newly stablized [`try_fold`](https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.try_fold), it can be done in only a few lines. This is because the `?` operator is allowed to operate correctly in these contexts.

```rust
fn compute(input: String) -> Result<i32, String> {
    ingest(input)? // Ingest and abort if error. Simple enough
        .into_iter()
        .map(|i| i.normalize()) // Maps into Result<Data, String>
        .map(|i| i?.perform_operation()) // Maps into Result<i32, String>
        .try_fold(0, |agg, curr| Ok(aggregate(agg, curr?))) // Maximize
}

Lessons

Thanks for reading, and a huge thanks to CUViper for helping me come up with this solution to my problem. The takeaways from this process for me have been threefold:

Functional combinators in Rust are very powerful. They fit well in any problem that involves processing many homogeneous data points.
Design around fallibility. Pretending that fallible operations can’t fail is a recipe for convoluted, type-weird code.
When a problem can’t be solved at one level, step back and look a the level above it. I spent hours trying to work out the types for the .collect::<>() call when I should have reexamined the whole pipeline.

If you enjoyed this tutorial, you might like to read more about Rust types, or perhaps learn how session types can make your code better.

Improved User Interface 0.3.0!

Wed, 13 Jun 2018 08:46:55 -0500

The Improved User Interface crate has had its 0.3.0 release, adding new input fields (Checkbox and Combobox), new layout options (LayoutGrid), as well as finally working 100% on Windows, and with many bug fixes. This comes with the 0.1.3 release of the underlying ui-sys crate to support these features.

It’s been a big undertaking to get to this point, and I’m excited to grow from here, now that libui itself is moving forward again as well.

I want to give a huge shoutout to several GitHub users who helped with the project, specifically:

pgvee, who helped fix Windows CI and worked on LayoutGrid
huangjj27, who did an amazing job fixing Windows builds
ZakCodes, who fixed an API bug
masche842, who implemented several new features

From the project management side, we have a vastly improved README, a CONTRIBUTING guideline file, and have resolved to keep a changelog.

I’d like to extend a huge thank you to all of y’all for keeping this ball rolling. I’m excited to see where we go from here.

#DeleteFacebook and FOSTA/SESTA

Wed, 28 Mar 2018 11:16:18 -0500

We, each and every one of us, need to make the decision to move to free, open source, and decentralized online services. It will be painful. It will be difficult. It may mean giving up some comforts, like sending money instantly to friends without fees. It is also the only way to prevent some seriously bad things from happening.

Inciting Events

In recent weeks, two major things happened:

Facebook’s business model - gathering as much information about you as possible, then selling it - was used in a totally predictable way that, nonetheless, nobody seemed prepared for. Cambridge Analytica paid 270,000 users and ended up with data from 50 million, none of whom consented to that data being used. They then targeted vulnerable individuals in a (successful) attempt to play on their fears and get Donald Trump elected president.
The United States government passed two laws, FOSTA and SESTA, which are completely ass-backwards attempts to combat sex trafficking that actually severly harm victims and sex workers. In response, many web services are cracking down on any sexually explicit content, including articles written about the porn industry and its issues.

These are not good things, but, and this may be hard to hear: they are entirely our fault, as Web users.

How Could This Happen?

A lot of people are running around with their hair on fire, wondering, “how could this happen?”. The answer is simple: we trusted large, centralized, profit-driven services with everything we care about - our photos, our messages, our opinions, our livelihoods, and even our relationships.

Had Facebook not been the primary way for hundreds of millions of people to communicate and express their opinions, Cambridge Analytica could not have used the fears and anxieties expressed in those communications to manipulate voters.

Had we not become reliant on Skype and Google Drive to carry out business and pleasure, FOSTA and SESTA would not be hitting sex workers and survivors of rape so hard.

Had we interrogated, for even one instant, what it was that funded and powered the “free” online services we entrusted with our social and professional lives, we would have glimpsed the writhing horror we were building, and turned away in disgust.

Is There Any Hope?

Fortunately, there is a light at the end of the tunnel. Those of us who did understand the true nature of “free” services like Facebook have been building a solution.

This solution is decentralized services. This means that anyone - any organization or individual with Internet access - can participate, not just as a user but as a service provider.

One of the key pieces of software here is called Mastodon. It is a microblogging platform, like Tumblr or Twitter, but rather than every user handing over their data to Twitter-the-company, there are thousands of “Mastodon sites”. There is cyber.space and witches.town, mastodon.social and pawoo.net. The crucial feature is this: if you have an account on mastodon.social (or any other Mastodon site), and I have one on cybre.space (or any other Mastodon site), I can talk to you and you can talk to me.

If you want to see a sample Mastodon profile, check out mine: noracodes@weirder.earth.

Let’s say I join mastodon.social with all my friends, but the people who run the site don’t agree with me - perhaps they don’t adequately protect me from harassment, and I want to move to kitty.town which is more aggressively moderated. With a traditional, centralized model, I would have to convince my friends to come with me, or be resigned to losing contact. With decentralized social media, you stay connected even if you move service providers.

That means that the problems mentioned above would be very, very short-lived. One Mastodon site (or “instance” as they’re called) starts doing shady things with user data. All the users can simply move to another instance with practically zero effort.. The network of instances can also cut off instances that are sources of spam or harassment, and that decision is up to the moderators and admins for each instance. If you don’t like the decisions the moderation team of your instance makes, you can move to another, more agreeable one with almost no effort.

Remember, there have been massive issues with moderation on Twitter. Those issues are largely solved by smaller, more tailored moderation teams, which Mastodon enables.

While Mastodon is modelled after Twitter, the same underlying systems are being applied to Facebook-like and Tumblr-like sites. Even more interestingly, these will be able to talk to each other. Imagine being able to see Twitter posts in Facebook, or Facebook posts on Tumblr. This further reduces lock-in. Don’t like the Mastodon format? Move to Aardwolf or Pleroma or … wherever. You will still be connected to your friends.

What You Need To Do

There are a few things you can do to help.

Make a Mastodon account. joinmastodon.org will help you get started. Then, aggressively convince your friends to switch from Facebook, Twitter, et cetera. Be “that person”. It’s painful at first but the benefits are worth it.
Use other decentralized software. E-mail, for instance, is better than Facebook Messenger, even if you use GMail or Yahoo! Mail or another large e-mail provider, because there is the option to move to another.
Share this post. Share it on Facebook and Twitter and Instagram and everywhere. Or write your own and share that. Make it resonate.

Together, we can build an internet that’s not just safer, but more enjoyable. Thank you. Your effort and struggle are valuable.

U2F Zero in Firefox

Mon, 26 Mar 2018 11:48:21 -0500

I recently got a U2F Zero, a tiny (about 2 inches long) and cheap (about 8 dollars) device implementing the FIDO Universal Second Factor protocol. It’s open source, too, which is awesome.

A close-up shot of the U2F Zero.

I primarily use Firefox. Unfortunately, Firefox doesn’t support U2F out of the box; you have to enable security.webauth.u2f in about:config.

Once enabled, I tried to register it with GitHub. Unfortunately, it kept on failing, despite the little green light on the U2F Zero going blue.

Eventually, I discovered that one security measure - Linux isolating Firefox from my USB devices - was getting in the way of this new one. I added the contents of this file to /etc/udev/rules.d/10-u2f.rules and ran sudo udevadm control --reload to reload the rules, and it worked!

The process is as follows:

Log into a website that supports U2F
Enable second factor with some 2FA app, like Google Authenticator
Find the interface to register a U2F device
Plug in U2F Zero
When light is green, press the only button on the device

The light turns blue and, if all goes well, the device will be registered. Now, instead of pulling out your phone and entering a code, just whack in the U2F Zero and hit the button when you log in.

Works like a charm, and the bare matte-black PCB means you can keep your hacker aesthetic going, or 3D print a case to your liking.

A Methodology for Fontconfig Editing

Wed, 07 Mar 2018 11:16:38 -0600

One of the hardest parts of building beautiful Linux systems is fonts. Font precedence on Linux is generally handled with fontconfig. In essence, fontconfig is used to permit many fonts to be installed and uninstalled over time without breaking applications which specify a font or font family, while letting users configure which fonts are used when a missing font, font family, or missing glyph is requested.

This is really useful piece of technology; having a defined configuration system for which fonts are used in which scenarios is a boon for configurability, but fontconfig has no real GUI editors or usable interactive configuration tools. Users are expected to manually edit XML configuration files.

As with most Unix styling topics, Eevee has a great piece on fontconfig’s complexities. She digs into how to disable and re-configure fonts, how to set fallbacks, and how to verify that the correct resolution order is set. Fontconfig relies on a set of config files, generally in /etc/fonts/conf.d, which are loaded in alphabetical order. These are usually prefixed with a number, so it’s easy to determine the order.

Unfortunately, it can be very complex to determine where a specific font or option is configured. In my recent case, I wanted to switch from DejaVu as my default to Bitstream Vera, and I spent the better part of an hour flipping around different files changing mentions of Deja Vu X to Bitstream Vera X.

Eventually, I settled on the following methodology:

Identify problematic resolution result (either by observing it or using fc-match -s).
In /etc/fonts/conf.d, use grep or rg to search for the incorrectly resolved font (e.g. rg DejaVu .*).
Open highest-numbered file with a match. For me, this was 69-language-selector-zh-tw.conf.
Determine whether or not this config file is causing the problematic match. In the case of 69-language-selector-zh-tw.conf, it was only selecting DejaVu Sans Mono for language zh-tw, which is actually correct as Bitstream Vera Mono doesn’t include zh-tw glyphs.
If that file might be causing the problematic match, modify it.
Check if the problematic resolution still occurs (using fc-match). If so, repeat.

I’ve been very successful with this methodology so far. In my specific case, I had to modify 56-emojione.conf, which was setting the default serif, sans serif, and monospace fonts to resolve to DejaVu followed by Emoji One.

Additional Exercises in Reverse Engineering

Sat, 03 Feb 2018 11:00:00 -0600

This is a sequel to my (rather long) introduction to reverse engineering. It is something of a “whirlwind tour” of some useful methods that weren’t shown in that tutorial, and provides a number of exercises to hone your skills.

The CrackMe Programs

You can find the CrackMes discussed here on GitHub. Clone that repository and, without looking at the source code, build each CrackMe with make crackme01, make crackme02, etc.

Tools and Software

These CrackMes only work on Unix systems, and I wrote this tutorial using Linux. You need the essentials of a development environment installed - a C compiler (gcc), object inspection utilities (objdump, objcopy, xxd), et cetera. You also need Radare2, an advanced open-source reverse engineering toolkit. On Debian-derived systems, the following command should get you set up:

sudo apt install build-essential gcc xxd binutils

You can install Radare2 from here.

On other systems, install the equivalent packages from your package manager.

Solutions

crackme05.c

This crackme is very similar to those presented in the previous tutorial, but is more modularized. It has functions (remember, aaa to analyze and then afl to list functions) for success and failure conditions, which print out the appropriate string and then exit.

The main function has a number of calls to sym.fail, the failure function, each stemming from a different condition. Only by passing all of these conditions does execution reach 0x880 where RDI (the first function argument) is loaded with the input string and then sym.success is called.

Some of these checks are self-evident; for instance, at 0x7d7, the string’s length must be exactly 16:

call sym.imp.strnlen
cmp eax, 0x10
jne 0x850

Other checks, however, call the function check_with_mod, each time with three arguments. For instance, at 0x81a:

lea rdi, [rbx + 8]
mov edx, 5
mov esi, 4
call sym.check_with_mod

Here, RBX is argv[1], so this is really check_with_mod(argv[1] + 8, 5, 4). The third argument, here a 4, appears consistent across calls, while the second argument changes. So what does check_with_mod do?

As usual, s sym.check_with_mod followed by pdf will give us the answer. It’s actually a very simple function, just 20 instructions. At its heart is a loop which adds up the values of some bytes in the input string (argument 1), dictated by the 3rd argument. In our case this will always be 4 bytes.

The function then performs integer division idiv r8d. This divides RDX by R8 (the second argument), saving the quotient in RAX and the remainder in EDX. The code then checks that RDX is zero and discards the quotient, making this a modulus operation.

So the mystery is solved. The code checks that the four bytes after each offset sum up to something divisible by the given value. Returning to the main function, we can see that there are four chunks, which need to sum up to 3, 4, 5, and 4, respectively.

That’s not enough, though. It also compares byte 2 to ‘B’ and byte 0xd (13) to ‘Q’.

So the string looks like this so far: ..B..........Q..

Now we need to figure out those blank spaces. By doing a little math we can find EEBD,,,,2222QQOO, which is correct!

crackme06.c

This crackme is pretty simple to solve using Radare2 static analysis (that is, just looking at the code), but I’d like to demonstrate another tool in the toolbox of a reverse engineer - strace.

strace prints out every system call a program makes as it runs. This makes it very useful to figure out the basic behavior of a program, as well as letting you easily isolate specific kinds of behavior like network connections and file I/O.

Here, we’ll just run strace ./crackme06.64 test:

execve("./crackme06.64", ["./crackme06.64", "test"], [/* 56 vars */]) = 0
brk(NULL)                               = 0x5645ad7a0000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=249222, ...}) = 0
mmap(NULL, 249222, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f8dded0d000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\22\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1960656, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8dded0b000
mmap(NULL, 4061792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f8dde743000
mprotect(0x7f8dde919000, 2097152, PROT_NONE) = 0
mmap(0x7f8ddeb19000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1d6000) = 0x7f8ddeb19000
mmap(0x7f8ddeb1f000, 14944, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f8ddeb1f000
close(3)                                = 0
arch_prctl(ARCH_SET_FS, 0x7f8dded0c4c0) = 0
mprotect(0x7f8ddeb19000, 16384, PROT_READ) = 0
mprotect(0x5645ab906000, 4096, PROT_READ) = 0
mprotect(0x7f8dded4a000, 4096, PROT_READ) = 0
munmap(0x7f8dded0d000, 249222)          = 0
brk(NULL)                               = 0x5645ad7a0000
brk(0x5645ad7c1000)                     = 0x5645ad7c1000
openat(AT_FDCWD, "test", O_RDONLY)      = -1 ENOENT (No such file or directory)
dup(2)                                  = 3
fcntl(3, F_GETFL)                       = 0x8402 (flags O_RDWR|O_APPEND|O_LARGEFILE)
fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
write(3, "PANIC! Aborting due to: No such "..., 50PANIC! Aborting due to: No such file or directory
) = 50
close(3)                                = 0
exit_group(-1)                          = ?
+++ exited with 255 +++

Much of this output is relatively useless - loading shared library preloads which aren’t set, and mapping bits of memory in, all done by the shell. Eventually, though, there’s a call to openat:

openat(AT_FDCWD, "test", O_RDONLY)      = -1 ENOENT (No such file or directory)
...
write(3, "PANIC! Aborting due to: No such "..., 50PANIC! Aborting due to: No such file or directory
) = 50

The program calls openat with the filename test, attempting to open it in read-only mode, gets an error, and then prints that error out.

Clearly, it expects the file to exist; by creating that file and putting some content in it, we see:

openat(AT_FDCWD, "test", O_RDONLY)  = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=13, ...}) = 0
read(3, "some content\n", 4096)         = 13
read(3, "", 4096)                       = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
write(1, "Access denied.\n", 15Access denied.
)        = 15

Here, the code successfully opens the file and gets a file descriptor back (3). It reads the file and gets back “some content\n” and then reads it again and gets nothing. Finally, it prints “Access denied”.

So we know the program is reading bytes from a file. Diving into Radare, it’s trivial to see that those bytes are then compared to the string “scrambled egg 42”. If you load that file with these bytes, the crackme will succeed.

Taking the time to trace the program statically would have yielded the same result, but it would have taken a lot longer. Dynamic analysis has a place, especially for identifying critical sections of the target executable.

crackme07.c

This crackme is a pretty simple one, but it uses a slightly odd mechanism for input. It compares its first arg to a static string, but only succeeds if the current time is in a certain range.

There is a function sym.cur_hour which makes a call to the libc function localtime. It takes the resulting localtime struct and returns an quadword from 8 bytes into that struct.

call sym.imp.localtime
mov edi, dword [rbx]
test edi, edi
jne 0x8f9                   ; jump to error handling if error
mov eax, dword [rax + 8]    ; this value gets returned

That struct looks like this, according to the documentation for localtime (3):

struct tm {
    int tm_sec;         /* seconds */
    int tm_min;         /* minutes */
    int tm_hour;        /* hours */
    int tm_mday;        /* day of the month */
    int tm_mon;         /* month */
    int tm_year;        /* year */
    int tm_wday;        /* day of the week */
    int tm_yday;        /* day in the year */
    int tm_isdst;       /* daylight saving time */
};

So 8 bytes in is tm_hour, the current hour. In the main function, that value is used thus:

call sym.cur_hour
mov ebx, eax
...
sub ebx, 5
cmp ebx, 1
jbe 0x985

In other words, the hour must be between 5 and 6 (that is, the time must be between 0500 and 0659). Either stay up late or just temporarily change your system clock to get this crackme to succeed.

crackme08.c

This is another “computed” crackme; that is, it computes the correct code on the fly.

One important feature of this program is that it allocates memory, at 0x869:

mov edi, 0xf
call sym.imp.malloc

This is a very common call for C programs - it’s how they get memory on the heap - but previous crackmes haven’t really needed to do this. This particular call allocates 0xf (15) bytes.

Immediately afterward, there’s something that isn’t normally seen: the cpuid instruction. This instruction is a way for a CPU to identify itself to software, and it places “characteristic bytes” in EBX, EDX, and ECX, in that order. While these could be any bytes, they are usually printable ASCII characters. Intel processors return the bytes “GenuineIntel”, AMD processors return “AuthenticAMD” (or “AMDisbetter!” on old engineering samples), and other vendors have their own patterns ranging from the utilitarian “Vortex86 SOC” to the tongue-in-cheek “CyrixInstead”.

Anyway, these bytes are then moved into other registers and passed to the function sym.shift_int_to_char along with the pointer returned by that previous call to malloc. Looking at that function, it’s pretty simple; it moves the first, second, third, and fourth bytes of the given doubleword into memory sequentially. This is being used my the main code to destructure the weird encoding from cpuid into an actual string.

Hopping back to main, we see some other bytes get set: ‘3’, ‘Q’, and the null byte. It’s a good bet that the password is made up of the three doublewords from the CPUID instruction plus “3Q”, and this checks out: three doublewords is 12 characters, plus two more is 14 plus null is 15 (or 0xf), the length of the allocated buffer.

Sure enough, that buffer is immediately checked using strcmp against the input and then released back to the system with free.

This is the first of many crackmes that will have different solutions on different computers. For me, it’s “GenuineIntel3Q”.

Appendix

Thank you for reading this second tutorial! I hope it’s been instructive. My next priority is to create some more real-world use cases, such as reverse engineering nontrivial C applications, looking at minified JavaScript, and doing some more in-depth dynamic analysis with GDB.

An Intro to x86_64 Reverse Engineering

Thu, 16 Nov 2017 18:29:37 +0000

This document presents an introduction to x86_64 binary reverse engineering, the process of determining the operation of a compiled computer program without access to its source code, through a series of CrackMe programs.

There are a lot of excellent tutorials out there, but they mostly focus on the 32-bit x86 platform. Modern computers are, almost without exception, 64-bit capable, so this tutorial introduces 64-bit concepts immediately.

A CrackMe is an executable file which takes (typically) a single argument, does some check on it, and returns a message informing the user if it’s correct or not. The challenge is to determine the correct argument without looking at the source code. Here, I present some CrackMe programs which I wrote, and demonstrate how to arrive at their solutions.

Prerequisites

Knowledge

This tutorial assumes a reasonable familiarity with programming, but not any particular skill with assembly, CPU architecture, or even C programming specifically. You should know what a compiler does, but you don’t have to know how to implement one. Similarly, you should know what a register is, but you don’t need to have the x86 registers or instructions memorized. I certainly don’t.

If you are a fluent programmer but don’t know assembly, I suggest you look at the x86 Crash Course. It’s a 10 minute video that should give you the background you need to understand this tutorial.

The CrackMe Programs

You can find the CrackMes discussed here on GitHub. Clone that repository and, without looking at the source code, build each CrackMe with make crackme01, make crackme02, etc.

Tools and Software

These CrackMes only work on Unix systems, and I wrote this tutorial using Linux. You need the essentials of a development environment installed - a C compiler (gcc), object inspection utilities (objdump, objcopy, xxd), et cetera. This tutorial will also show you how to use the program Radare2, an advanced open-source reverse engineering toolkit. On Debian-derived systems, the following command should get you set up:

sudo apt install build-essential gcc xxd binutils

You can install Radare2 from here.

On other systems, install the equivalent packages from your package manager.

The Solutions

Note: In later solutions, I discuss file offsets. These may be different for you, but I never use them without discussing where I got them, so if you’re confused, just search for the offset using Ctrl+F or Find In Page and you should see how I got them quite easily.

crackme01.c

crackme01.64 is a relatively simple program. When you run it, you’ll see this output:

$ ./crackme01.64
Need exactly one argument.

Provide some random argument. I used lmao:

$ ./crackme01.64 lmao
No, lmao is not correct.

This is as expected. We don’t know the password. The first thing to try, when faced with a problem such as this, is to think about what the program is doing. The simplest way to check a string is to simply compare it against another string, stored in the binary. The binary may appear opaque to us, but in fact it is not. It is a file full of data like any other; it’s just structured in a special way.

Try It Yourself: Try examining the executable file with cat, less, or your favorite text editor.

If we simply cat it, we get gibberish. Luckily, there is a standard Unix tool called strings which will try to extract all the valid pieces of text (printable characters followed by the null character) in a given file.

$ strings ./crackme01.64
/lib/ld-linux.so.2
WXZd
libc.so.6
_IO_stdin_used
__printf_chk
puts
__cxa_finalize
__libc_start_main
_ITM_deregisterTMCloneTable
__gmon_start__
_Jv_RegisterClasses
_ITM_registerTMCloneTable
GLIBC_2.3.4


...

.dynamic
.data
.bss
.comment
.debug_aranges
.debug_info
.debug_abbrev
.debug_line
.debug_str
.debug_loc

This is a lot of output. We can learn some useful things from it, but for now, we’re looking for a password.

Try It Yourself: Try to find the password in the output of strings. This is all you need to solve the challenge!

Solution

In this case, it is enough to simply scroll through the listing. Eventually, you’ll see these lines:

...
[^_]
Need exactly one argument.
password1
No, %s is not correct.
Yes, %s is correct!
;*2$"
...

You can see two strings we already know about: Need exactly one argument. and No, %s is not correct.. Note that %s is the control sequence that tells C’s printf function to print a string, presumably the one we entered at the command line.

Between these known strings is something rather suspicious looking. Let’s try it:

$ ./crackme01.64 password1
Yes, password1 is correct!

Success! You’d be surprised how much useful knowledge can come from a simple invocation of strings on a binary.

Exercise: There is a file called crackme01e.c which can be solved using this same technique. Compile and attempt to solve it, to cement your skills.

crackme02.c

This CrackMe is a little more difficult. You can try the same procedure as above, but the password you uncover won’t work!

Try It Yourself: Try to figure out why this might be, without reading ahead.

We’ll need to look at the actual behavior of the program using objdump. You may need to install it with your system’s package manager. objdump is an extremely powerful tool for examining binaries.

A binary program like this is a sequence of machine instructions, represented as number. objdump allows us to disassemble these machine instructions and represent them as slightly more readable assembly mnemonics.

In this case, if we run objdump -d crackme02.64 -Mintel | less, we will get an assembly listing. I’ve piped it through less because it’s very long.

The first line tells us what we’re looking at: crackme02.64: file format elf64-x86-64. It’s a 64-bit ELF executable file, on the Intel x86_64 (that is, AMD64) CPU architecture. Following this line are a number of sections that look like this:

Disassembly of section .init:

0000000000000590 <_init>:
 590:   48 83 ec 08             sub    rsp,0x8
 594:   48 8b 05 3d 0a 20 00    mov    rax,QWORD PTR [rip+0x200a3d]        # 200fd8 <__gmon_start__>
 59b:   48 85 c0                test   rax,rax
 59e:   74 02                   je     5a2 <_init+0x12>
 5a0:   ff d0                   call   rax
 5a2:   48 83 c4 08             add    rsp,0x8
 5a6:   c3                      ret 
 ...

Most of these are inserted by the linker immediately after compilation, and so aren’t associated with the the algorithm for checking the code. We can skip everything but the .text section. It begins like this:

Disassembly of section .text:

00000000000005e0 <_start>:
 5e0:   31 ed                   xor    ebp,ebp
 5e2:   49 89 d1                mov    r9,rdx
 5e5:   5e                      pop    rsi
 5e6:   48 89 e2                mov    rdx,rsp
 5e9:   48 83 e4 f0             and    rsp,0xfffffffffffffff0
 5ed:   50                      push   rax
 5ee:   54                      push   rsp
 ...

Again, this is a stub function inserted by the linker. We don’t care about anything until the main function, so keep on scrolling until you see it.

0000000000000710 <main>:
 710:   48 83 ec 08             sub    rsp,0x8
 714:   83 ff 02                cmp    edi,0x2
 717:   75 68                   jne    781 <main+0x71>
 719:   48 8b 56 08             mov    rdx,QWORD PTR [rsi+0x8]
 71d:   0f b6 02                movzx  eax,BYTE PTR [rdx]
 720:   84 c0                   test   al,al
 ...

On the far left column, the addresses of each location are listed (in base 16). Just to the right are the raw machine code bytes, represented as hex pairs (pairs of base 16 digits). Finally, objdump generates and displays the equivalent assembly to the far right.

Let’s start picking apart this program. First, we see sub rsp,0x8. This is moving the stack pointer down by 8, allocating space on the stack for 8 bytes worth of variables. Note that we don’t know anything about these variables yet. These could be 8 chars, or a single pointer (remember, it’s a 64-bit executable).

Moving on, there’s a pretty standard jump-if condition:

cmp    edi,0x2
jne    781

If you don’t know what these instructions do, you can look them up; in this case, we’re comparing (cmp) the edi register to the hex number 2, and then jumping if it’s not equal (jne).

So the question is, what’s in that register? This a Linux x86_64 executable, so we can look up the calling convention (Wikipedia is your friend). It turns out that edi is the lower 32 bits of the Destination Index register, which is where the first argument to a function goes. If you recall how the main function is written in C, its signature is: int main(int argc, char** argv). So this is the register holding the first argument: argc, the number of arguments to the program.

Finding String Literals

So that compare-and-jump is checking if there are exactly two arguments to the program. (Note: the first argument is the name of the program, so it’s really checking if there’s one user-supplied argument.) If not, it jumps to another part of the main program, at 781:

lea    rdi,[rip+0xbc]
call   5c0 <.plt.got>
mov    eax,0xffffffff
jmp    77c <main+0x6c>

Here, we’re loading the address (lea) of a value into rdi (if you remember, this is the first argument to a function) and then calling a function at address 5c0. Looking at the disassembly of that line, we see:

5c0:   ff 25 02 0a 20 00       jmp    QWORD PTR [rip+0x200a02]        # 200fc8 <puts@GLIBC_2.2.5>

objdump has helpfully annotated this instruction, telling us that it is jumping into the libc function puts. If you look up that function, you’ll see that it takes a single argument: a pointer to a string, which is then printed to the console. So this block prints a string. But what string?

To answer that, we need to see what’s being inserted into rdi. If we look at that instruction, it says: lea rdi,[rip+0xbc]. That means calculate the pointer to the position 0xbc ahead of the instruction pointer (which will be pointing at the next instruction) and store that address in rdi.

So whatever we’re printing is 0xbc bytes ahead of this instruction. We can do the math ourselves: 0x788 (next instruction) + 0xbc (offset) = 0x845.

We can use another standard Unix binary tool to view the raw data from a specific offset: xxd. In this case, let’s issue xxd -s 0x844 -l 0x40 crackme02.64. -s is “seek” or “skip”; it makes the hexdump start at the offset we’re interested in. -l is “length”; it makes the hexdump only 0x40 characters long, rather than the whole rest of the file. We see:

$ xxd -s 0x844 -l 0x40 crackme02.64
00000844: 4e65 6564 2065 7861 6374 6c79 206f 6e65  Need exactly one
00000854: 2061 7267 756d 656e 742e 004e 6f2c 2025   argument..No, %
00000864: 7320 6973 206e 6f74 2063 6f72 7265 6374  s is not correct
00000874: 2e0a 0070 6173 7377 6f72 6431 0059 6573  ...password1.Yes

So, now we know. This block prints a string reading “Need exactly one argument.”, as you’d expect from looking at the program’s behavior when too many or too few arguments are specified.

Basic Flow Analysis

The most important part of this block is the unconditional jump at the end, which goes to address 77c:

add    rsp,0x8
ret

This block removes the local variables from the stack and returns. So, that’s it. If there aren’t exactly 2 strings provided to the binary - its own name and one command line argument - it will quit.

We can start to write this program out in C code:

int main(int argc, char** argv){
    if (argc != 2) {
        puts("Need exactly one argument.");
        return -1;
    }

    // Magic happens here
}

To find out just what magic happens in that lower part of the program, we need to look at the program’s flow. Assuming the argc check succeeds ( the jump at 0x717 isn’t taken), program execution proceeds into this block:

mov    rdx,QWORD PTR [rsi+0x8]
movzx  eax,BYTE PTR [rdx]
test   al,al
je     761 <main+0x51>

That first instruction moves the quadword (64-bit value) at the address [rsi+0x8] into rdx. What’s in rsi, the full 64-bit Source Index register? Turns out, that’s the second argument in the Linux x86_64 calling convention. So in C, this is the value argv + 8 or, because argv is of type char**, argv[1].

The next instruction moves and extends with zeroes (movzx) a single byte from the memory address stored in rdx; in other words, *argv[1], or argv[1][0]. The Accumulator register now has all zeroes except for the last 8 bits, which contain the first byte of argv[1], the the command line argument to the program.

test al,al is equivalent to cmp al, 0. al is the lower 8 bits of the Accumulator register. Essentially, this block is equivalent to the C code:

if (argv[1][0] == 0) {
    // do something
}

So, what’s at address 0x761? It’s the following block:

lea    rsi,[rip+0x119]        # 881 <_IO_stdin_used+0x41>
mov    edi,0x1
mov    eax,0x0
call   5c8 <.plt.got+0x8>
mov    eax,0x0
add    rsp,0x8
ret

One of the most important skills for a reverse engineer is noticing patterns, and you should be seeing one right away. Here, the program leas a relative offset from the instruction pointer into rsi and then calls a function.

That function is, according to the same technique used above, printf. printf takes a format string and a variable number of arguments. All variadic functions need the Accumulator register to hold a value telling them how many arguments to look for in the FPU registers (in this case, none, as we can see from the mov eax, 0x0 instruction). The rdx register already holds the pointer argv[1], so that’s the second argument.

So what’s the format string? We use the same technique as before, but this time I didn’t take out the comment that objdump added in where it did the math for us.

So, running xxd -s 0x881 -l 0x40 crackme02.64 gives us the answer: the format string here is Yes, %s is correct!. This looks promising! Additionally, we can see that, after the function call (at 0x77c; this is a useful address to remember), the space for local variables is removed from the stack, and the function returns. The return value is always placed in the Accumulator, so here the program returns a zero; success!

So, our C code looks like this:

int main(int argc, char** argv){
    if (argc != 2) {
        puts("Need exactly one argument.");
        return -1;
    }

    if (argv[1][0] == 0) {
        printf("Yes, %s is correct.", argv[1]);
    }

    // Magic happens here
}

All we have to to is provide a string whose first byte is 0; that is, the empty string:

$ ./crackme02.64 ""
Yes,  is correct!

In a certain sense, we’re done, but it is instructive to look at the remainder of the code. Let’s move on.

If that check fails, the code goes to this block (at 0x724):

cmp    al,0x6f
jne    794 <main+0x84>

Recall that, since we’re working from the assumption of that jump to success not being taken, al has argv[1][0] in it still. Here, we’re checking if it’s not equal to 0x6f (111 in decimal; ‘o’ in ASCII). If so, we jump to address 0x794.

lea    rsi,[rip+0xc4]        # 85f <_IO_stdin_used+0x1f>
mov    edi,0x1
mov    eax,0x0
call   5c8 <.plt.got+0x8>
mov    eax,0x1
jmp    77c <main+0x6c>

This is another print-and-return block; that unconditional jump (jmp) at the end jumps to 0x77c, where the program removes its stack space for local variables and returns.

Rather than printing the success message, this block prints “No, %s is not correct.” formatted with the command line argument, and then returns a failure code (1). We now know, for sure, that the correct message starts with the letter ‘o’, because without it, there’s an automatic fail condition.

int main(int argc, char** argv){
    if (argc != 2) {
        puts("Need exactly one argument.");
        return -1;
    }

    if (argv[1][0] == 0) {
        printf("Yes, %s is correct.", argv[1]);
        return 0;
    }

    if (argv[1][0] != 'o') {
        printf("No, %s is not correct.", argv[1]);
        return 1;
    }

    // Magic happens here
}

Assuming that jump is not taken, we go on to this code, at 0x728:

mov    esi,0x1
mov    eax,0x61
mov    ecx,0x1
lea    rdi,[rip+0x139]        # 877 <_IO_stdin_used+0x37>
movzx  ecx,BYTE PTR [rdx+rcx*1]
test   cl,cl
je     761 <main+0x51>

Here, we load up some registers with constants, and then load a pointer into rdi. That pointer is to the string “password1”. But we know this isn’t the right password. So what’s going on?

The next instruction places the byte at the address rdx + rcx. What’s in rdx? Well, if we backtrack to 0x719, we see that it was loaded with the value at rsi + 0x8; a.k.a. argv[1]. So we’re indexing into that string here; ecx = argv[1][1].

Again, the most important skill for reverse engineering is recognizing patterns. Here’s a common one we saw above: testing a register against itself, followed by a je, means “jump if that register is zero”.

So, if there’s a zero at argv[1][1], we jump to 0x761. Where is that? It’s the block we just reversed above; it prints the success string and exits with the return code of 0. Our pseudocode looks like this:

int main(int argc, char** argv){
    if (argc != 2) {
        puts("Need exactly one argument.");
        return -1;
    }

    if (argv[1][0] == 0 || argv[1][1] == 0) {
        printf("Yes, %s is correct.", argv[1]);
        return 0;
    }

    if (argv[1][0] != 'o') {
        printf("No, %s is not correct.", argv[1]);
        return 1;
    }

    // Magic happens here
}

What happens if it’s not zero, though? The program progresses down to 0x746.

movsx  eax,al
sub    eax,0x1
movsx  ecx,cl
cmp    eax,ecx
jne    794 <main+0x84>

Here, we zero out all but the lowest 8 bits of eax, then subtract one from it. Then we zero out all but the last 8 bits of ecx and compare eax to ecx. If they’re not equal, we jump to 0x794. Where is that? It’s another block we’ve already reversed; it prints the failure string and exits with the return code of 1.

What is this actually doing? From above, eax contains a single byte; 0x61 (decimal 97, or ‘a’ in ASCII). It has one subtracted from it, so it’s 0x60 (decimal 96, ‘`’ in ASCII). So we know that the first two characters of the key are ‘o`’. Our pseudocode looks like this:

int main(int argc, char** argv){
    if (argc != 2) {
        puts("Need exactly one argument.");
        return -1;
    }

    if (argv[1][0] == 0 || argv[1][1] == 0) {
        printf("Yes, %s is correct.", argv[1]);
        return 0;
    }

    if (argv[1][0] != 'o' || argv[1][1] != 0x60) {
        printf("No, %s is not correct.", argv[1]);
        return 1;
    }

    // Magic happens here
}

If they are equal, we move on to 0x753:

add    esi,0x1
movsxd rcx,esi
movzx  eax,BYTE PTR [rdi+rcx*1]
test   al,al
jne    73e <main+0x2e>

First off, the program increments esi. (esi was set to 1 in the previous block.) It then moves that value into the lower 32 bits of rcx.

Then, the program loads a single byte from rdi + rcx. rdi is argv[1], and rcx is esi + 1 (which is, at this moment, 2). So here, the program is loading argv[1][2]. More correctly, though, it’s loading argv[1][rcx] (you’ll see why this matters in a moment).

The program then checks if that value is zero; if not, it jumps to 0x73e. Where is that?

movzx  ecx,BYTE PTR [rdx+rcx*1]
test   cl,cl
je     761 <main+0x51>

We’ve seen this before. It’s just the check from a few sections above. It loads a byte from argv[1][ecx] and checks if it’s zero; if so it jumps to success, and if not it moves down to the code we just reversed. This is another pattern you should recognize in the future: a loop.

Now that we’ve identified the whole loop, let’s look at all of its instructions, from 0x73e to 0x75f.

Recall that, a few blocks ago, rdi was loaded with the address of the string password1, but this wasn’t the right password. Here we can see why; bytes are being loaded from that string and having one added to them before they’re compared with the actual input.

movzx  ecx,BYTE PTR [rdx+rcx*1] ; load a byte from argv[1]
test   cl,cl                    ; check if that byte is zero
je     761 <main+0x51>          ; if so, jump to success
movsx  eax,al                   
sub    eax,0x1                  ; decrement comparison byte
movsx  ecx,cl                   
cmp    eax,ecx                  ; check if the correct byte == the input byte
jne    794 <main+0x84>          ; if it doesn't match, jump to failure
add    esi,0x1                  ; increment index into comparison string
movsxd rcx,esi                  ; place that index in CX
movzx  eax,BYTE PTR [rdi+rcx*1] ; load the next byte from the comparison string
test   al,al                    ; Check that that byte isn't zero
jne    73e <main+0x2e>          ; If it's not zero, loop

While we as humans would write this like the following C code, the compiler actually moved the second part of the loop check to the end of the loop, and loads the next byte for the comparison string there.

Note: This code was incorrect in the original version of this tutorial. Thanks to empwill for correcting it. I note this so that readers will be aware that even experienced reverse engineers make mistakes, and that such mistakes are both expected and recoverable!

Try It Yourself: The following C code should be all you need to determine the correct code. Try it!

int main(int argc, char** argv){
    if (argc != 2) {
        puts("Need exactly one argument.");
        return -1;
    }
    // This pointer is in rdi in our disassembled binary
    char* comparison = "password1";
    // This is the value used to index the argv[1]
    int i = 0;

    while (argv[1][i] != 0 && (comparison[i]) != 0) {
        if (argv[1][i] != comparison[i] - 1) {
            printf("No, %s is not correct.", argv[1]);
            return 1;
        }
        i++;
    }

    printf("Yes, %s is correct.", argv[1]);
    return 0;
}

This is really all we need. Simply subtracting one from each letter of password1 in ASCII gives us “o`rrvnqc0”. Let’s try it:

$ ./crackme02.64 o\`rrvnqc0
Yes, o`rrvnqc0 is correct!

The sharp-eyed among you might have noticed that this binary has a problem, however; it will actually accept any of these characters. o, o`, o`r, o`rr, et cetera all work! Clearly not a very good method to use for your product key. Also, as pointed out by an astute GitHub commenter, an empty password (./crackme02.64 "") will be accepted.

If you made it this far, good job! Reverse engineering is difficult, but this is the core of it, and it only gets easier from here.

Exercise: There is a file called crackme02e.c which can be solved using this same technique. Compile and attempt to solve it, to cement your skills.

crackme03.c

The next crackme is slightly more difficult. In crackme02, we were able to manually inspect each branch, building up the flow of execution mentally. This technique breaks down as programs become more complex.

The Radare Analysis Tool

Fortunately, the reverse engineering community is rife with smart people, and there are excellent tools to automate a great deal of this analysis. Some of them, like Ida Pro, cost as much as $5000, but my personal favorite is Radare2 (Random data recovery), which is entirely free and open source.

Running crackme03.64, we can see that it (basically) works about the same way as the other two. It needs exactly one argument, and when we provide it with one, it helpfully tells us that it’s wrong.

This time, rather than objdumping the file, we’re going to open it with radare2 (or r2): r2 ./crackme03.64. You’ll see a prompt. Type “?”, and you’ll get a help listing. Radare is an immensely powerful tool, but for this challenge we won’t need most of its functionality. I’ve removed many entries, paring it down to the bare necessities.

[0x000005e0]> ?
Usage: [.][times][cmd][~grep][@[@iter]addr!size][|>pipe] ; ...
Append '?' to any char command to get detailed help
Prefix with number to repeat command N times (f.ex: 3x)
| a[?]                    Analysis commands
| p[?] [len]              Print current block with format and length
| s[?] [addr]             Seek to address (also for '0x', '0x1' == 's 0x1')
| V                       Enter visual mode (V! = panels, VV = fcngraph, VVV = callgraph)

The important thing to note is that Radare is self-documenting. If you ever want to know what you can do with a command, simply type a ? after it. For instance, we want to analyze the current program:

[0x000005e0]> a?
|Usage: a[abdefFghoprxstc] [...]
| ab [hexpairs]    analyze bytes
| aa[?]            analyze all (fcns + bbs) (aa0 to avoid sub renaming)
| ac[?] [cycles]   analyze which op could be executed in [cycles]
| ad[?]            analyze data trampoline (wip)
| ad [from] [to]   analyze data pointers to (from-to)
| ae[?] [expr]     analyze opcode eval expression (see ao)
| af[?]            analyze Functions
| aF               same as above, but using anal.depth=1
| ag[?] [options]  output Graphviz code
| ah[?]            analysis hints (force opcode size, ...)
| ai [addr]        address information (show perms, stack, heap, ...)
| ao[?] [len]      analyze Opcodes (or emulate it)
| aO               Analyze N instructions in M bytes
| ar[?]            like 'dr' but for the esil vm. (registers)
| ap               find prelude for current offset
| ax[?]            manage refs/xrefs (see also afx?)
| as[?] [num]      analyze syscall using dbg.reg
| at[?] [.]        analyze execution traces
Examples:
 f ts @ `S*~text:0[3]`; f t @ section..text
 f ds @ `S*~data:0[3]`; f d @ section..data
 .ad t t+ts @ d:ds

Try It Yourself: I suggest traversing the help for a while. Google every term you don’t understand. There is a lot of cool functionality that I won’t touch on here, but which might inspire you to try something.

Automatic Analysis

It turns out that the command we want it aaaa: analyze using all functions using all normal and all experimental techniques.

This means that Radare has a list of functions available to us. We can view it with afl: analyze functions, displaying a list.

[0x000005e0]> afl
0x00000000    3 73   -> 75   fcn.rsp
0x00000049    1 219          fcn.00000049
0x00000590    3 23           sym._init
0x000005c0    1 8            sym.imp.puts
0x000005c8    1 8            sym.imp.__printf_chk
0x000005d0    1 16           sym.imp.__cxa_finalize
0x000005e0    1 43           entry0
0x00000610    4 50   -> 44   sym.deregister_tm_clones
0x00000650    4 66   -> 57   sym.register_tm_clones
0x000006a0    5 50           sym.__do_global_dtors_aux
0x000006e0    4 48   -> 42   entry1.init
0x00000710    7 58           sym.check_pw
0x0000074a    7 203          main
0x00000820    4 101          sym.__libc_csu_init
0x00000890    1 2            sym.__libc_csu_fini
0x00000894    1 9            sym._fini

We only care about main and check_pw.

Try It Yourself: Figure out why I can immediately discard the other functions for initial analysis. Search engines are your friend.

Radare can disassemble just a single function for us with pdf@main: print disassembly of a function @ (at) the symbol called main. Radare also supports tab completion in many contexts. For instance, if you type pdf@sym. and hit the tab key, you’ll get a list of all the functions in the symbol table.

Anyway, the first thing to notice is that Radare does syntax highlighting, adds a lot of comments, and even names some variables for us. It does some analysis to determine the types of variables, too; in this case, we have 9 local stack variables. Radare names them local_2h, local_3h, et cetera based on their offsets from the stack pointer.

The beginning of our program is pretty familiar. Starting at 0x74a:

push rbx
sub rsp, 0x10
cmp edi, 2
jne 0x7cc

We have the function prologue allocating 16 bytes of memory for our local variables and an if statement. Recall that the DI register holds the first argument, and since this is main, that argument is argc. So, if (argc != 2) jump somewhere.

In Radare, look to the left of the jne instruction. You’ll see an arrow coming out of that instruction and running down to 0x7cc, where we see:

lea rdi, str.Need_exactly_one_argument. ; 0x8a4 ; "Need exactly one argument." ; const char * s
call sym.imp.puts           ; int puts(const char *s)
mov eax, 0xffffffff         ; -1
jmp 0x7c6

Remember how annoying it was to search for string literals in our binary? Radare does it for us, giving us the address, a convenient alias, and the value of the string literal. It also analyzes the functions being called, which is very convenient. Here, we can see without much effort that the binary is printing the string “Need exactly one argument.”.

It then loads eax with -1 and jumps to 0x7c6. We can follow the arrow (or just scroll to the address) to see where that is, but there’s a more interesting way.

Visual Flow Analysis

Radare provides a feature known as “visual mode”. To use it, we need to move Radare’s internal cursor to the function we want to analyze with the seek command: s main. You’ll notice that the prompt changes from [0x000005e0]> to [0x0000074a]>, indicating that the current location has moved to the first instruction in the main function. Then, type VV (Visual mode 2). You should see ASCII-art boxes with different parts of the program.

Every time a jump instruction appears, the block ends and lines come out, pointing to other blocks. For instance, in the top block (the beginning of the function), the jne instruction which checks for the argument number causes a red line to come out to the left and a green one to the right.

On the right, you should see a block that looks a bit like this:

.---------------------------------------------.                                
|  0x7cc ;[ga]                                |                                
|      ; const char * s                       |                                
|      ; 0x8a4                                |                                
|      ; "Need exactly one argument."         |                                
| lea rdi, str.Need_exactly_one_argument.     |                                
| call sym.imp.puts;[gh]                      |                                
|    ; -1                                     |                                
| mov eax, 0xffffffff                         |                                
| jmp 0x7c6;[gg]                              |                                
`---------------------------------------------'

That’s the block we just analyzed. Use the arrow keys to follow the blue (unconditional) line down to see what happens after this block. You’ll see, at the bottom of the graph, a block at 0x7c6 that is unconditionally jumped to from a number of places in the program:

add rsp, 0x10                                                      
pop rbx                                                 
ret

This simply frees stack space and returns. So this program behaves like the others we’ve looked at: if there aren’t the right number of arguments, it prints a string and exits with an error code (remember, eax was loaded with -1).

Try It Yourself: Look at the rest of the program in the control flow graph. Try to find the block that prints the failure message. There are two checks that can lead there; can you figure out what they do?

Recall that test eax, eax followed by je means “jump if eax is zero”, and that the x86 instruction set is well documented; if you don’t know what an instruction does, look it up!

If we progress down the red branch from the first block, where execution flows if the jne isn’t taken (that is, if there are exactly 2 strings passed to the binary), you’ll see these instructions at 0x754:

mov dword [local_9h], 0x426d416c ; [0x426d416c:4]=-1
mov word [local_dh], 0x4164 ; [0x4164:2]=0xffff
mov byte [local_fh], 0
mov word [local_6h], 0
mov byte [local_8h], 0
mov byte [local_2h], 2
mov byte [local_3h], 3
mov byte [local_4h], 2
mov byte [local_5h], 3
mov byte [local_6h], 5
mov rbx, qword [rsi + 8]    ; [0x8:8]=0
mov eax, 0
mov rcx, 0xffffffffffffffff
mov rdi, rbx
repne scasb al, byte [rdi]
cmp rcx, 0xfffffffffffffff8
je 0x7df

Mostly all this block does is load a bunch of values into memory. Here, rather than showing the actual addresses, Radare has named each local variable based on its stack offset. Scrolling up to the initial block, we can see that local_2h through local_fh are all of type int (or, at least, that’s what Radare thinks) and they’re each one byte in size.

After loading those values into local variables, it loads something in memory at the address rsi + 8 into rbx. If we recall the SystemV x86_64 calling convention, rsi is the second argument: argv. So rsi + 8 is argv[1]. It then loads up eax with 0, rcx with 0xffffffffffffffff , and rdi with rbx, the value just loaded from argv[1].

Then it runs repne scasb. This is a weird but fast quirk of x86: it has a native instruction for string length determination. repne means repeat while not equal, and scasb means string compare and subtract, byte variant - see the reference here for more info.

So, this instruction compares bytes to the value of al (which is zero here), starting at the memory address in rdi and counting up, while subtracting from rcx (remember, C is the counter register). In essence, this instruction is measuring the length of a string. Isn’t x86 a fun time?

Anyway, once the repne scasb operation is done, rcx will hold 0xffffffffffffffff minus the length of the string, and we can see that the next instruction compares it to 0xfffffffffffffff8. Therefore, if the string is 0xffffffffffffffff - 0xfffffffffffffff8 = 7 bytes long (note that this includes the terminating character), the jump is taken; otherwise it is not.

If the jump isn’t taken, the code proceeds to a block at 0x7a8, where the failure string is printed. Therefore, we can immediately determine that any correct passcode will be precisely six bytes (plus the null terminator).

Functions

What is more interesting is the block that is executed when the jump is taken.

lea rdx, [local_2h]
lea rsi, [local_9h]
mov rdi, rbx
call sym.check_pw
test eax, eax
je 0x7a8

The binary loads the addresses of some local variables, loads argv[1] again (rbx, remember?), and then calls a function: sym.check_pw. Of course, the actual binary just has the offset of the function, but Radare was smart enough to look up that offset in the symbol table and put the name in for us. check_pw sounds pretty promising, as cunction names go, and we can verify that by continuing: after the call, the program jumps to failure if the function returned zero, and continues on to success if it did not (recall that test eax, eax followed by je jumps if eax is zero).

So what exactly does this function do? First, recall that the SystemV x86_64 calling convention says that rdi, rsi, and rdx (the three registers loaded prior to the call) are the first three arguments to the function. So in C, the call looks like this:

int result = check_pw(argv[1], &local_9h, &local_2h);
if (result == 0) {
    // fail
} else {
    // succeed
}

The question, then, is what, exactly, does check_pw do? To find that out, we need to exit visual mode (q followed by q) and seek to it (s sym.check_pw), then look at the flow diagram (VV).

It is immediately clear that this function contains a loop. Unlike the main function, which continues consistently downward no matter which jumps you take, in check_pw a block near the bottom has a jne that jumps up to the top. Looking a little more closely, we can see that there are three opportunities to return. One of them (at 0x73e) returns 0 (failure) and the other two (at 0x744 and 0x748) return 1 (success).

This kind of high-level analysis is only possible with a flow diagram, and is one of the major advantages of using a tool like Radare. When I was getting started with reverse engineering, I drew out flow diagrams by hand, simply because I was unaware that free tools existed. Don’t do that; it’s a waste of time.

The function starts off by loading r8d, a 64-bit general purpouse register, with the value 0. It then jumps to the following block (at 0x716):

movsxd rax, r8d
movzx ecx, byte [rdx + rax]
add cl, byte [rsi + rax]
cmp cl, byte [rdi + rax]
jne 0x73e;[gb]

This block sets rax to r8d (which we know is zero), then loads a single byte from its third argument, indexed by eax. Going back to our arg list, this argument is &local_2h, so it’s loading (&local_2h)[0].

It then adds a byte loaded from the second argument indexed by eax ((&local_9h)[0]), and compares it to a byte loaded from its first argument indexed by eax (argv[1][0]). Remember that this is a loop, so eax will change. In other words:

while (/* something?? */) {
    char temp  = arg3[eax] + arg2[eax];
    if (temp != arg1[eax]) {
        return 0; // failure
    }
}

If the jump isn’t taken, this code is run (at 0x725):

add r8d, 1 
movsxd rax, r8d
cmp byte [rsi + rax], 0
je 0x744;[gd]

This increments the loop counter, then checks if the second argument indexed by the loop counter is zero. If so, it jumps to code (at 0x744) that returns success. Otherwise, it continues looping. Our updated C code looks like this:

while (arg2[eax] != 0) {
    char temp  = arg3[eax] + arg2[eax];
    if (temp != arg1[eax]) {
        return 0; // failure
    }

    eax++;
}

return 1;

At this point, it’s pretty easy to see what check_pw is doing: it’s comparing two strings, but it’s modifying each byte of one of the strings.

Looking at the arguments passed in main, we can see that the program is adding (&local_2h)[eax] to (&local_9h)[eax]. I suggest going back to the main function (exit visual mode; pdf@main) to look at what each of those values will be.

Both of these are just locations on the stack. We also know that check_pw will only be run on a string with 6 characters in it, so we only need to look at 6 values. Here are the values after local_2h (you can see them being set in main): 2, 3, 2, 3, 5. That’s only 5 values. What’s going on?

If we look again, the stack variables are set starting at offset 0x754:

mov dword [local_9h], 0x426d416c ; [0x426d416c:4]=-1
mov word [local_dh], 0x4164 ; [0x4164:2]=0xffff
mov byte [local_fh], 0
mov word [local_6h], 0
mov byte [local_8h], 0
mov byte [local_2h], 2
mov byte [local_3h], 3
mov byte [local_4h], 2
mov byte [local_5h], 3
mov byte [local_6h], 5

Prior to the values being set in order from local_2h to local_6h, by moving byte-sized values into them, local_6h (that is, rsp+0x6) is loaded with a word-sized 0 (this is Intel-syntax, so a word is 16 bits; see this historical note) value. That means that both rsp+0x6 and rsp+0x7 are set to zero.

Note that Radare didn’t even realize that these values were in an array, let alone tell us what it was initialized to, despite it being entirely static. This is the part of reverse engineering that requires a human brain; the computer knows what’s there, but it can’t know what it’s being used for.

Anyway, our table of values starting at local_2h is [2, 3, 2, 3, 5, 0]. These aren’t printable ASCII characters, so presumably the hard-coded password is stored at the other input value: local_9h.

The mov instruction above is sized for a double word (dword); a 32-bit value. It’s followed by a word-sized value, then a byte-sized zero. That works out to 4+2 = 6 bytes, plus a null terminator, so it’s a good bet that these three locations together form a string. Indeed, if we write out the values with byte seperation, it makes sense: 42 6d 41 6c 41 64 00 is a well-formed null-terminated string, with values in the printable ASCII range.

All that’s left is to add the offsets to them, giving us 44 70 43 6e 44 64 00. Translating these to ASCII, we get: DpCnDd.

Sure enough, putting this into the binary gives us… the failure message. What happened?

Try It Yourself: Why didn’t this work? It’s something pretty fundamental about how x86 processors organize data.

What’s happening here is that x86 processors are little-Endian. That means that bytes are read from the right to the left, not the other way around, in multi-byte values. This is easily corrected by just flipping the order of local_9h and local_dh. 42 6d 41 6c becomes 6c 41 6d 42 and 41 64 becomes 64 41, making our whole string 6c 41 6d 42 64 41 00 and our correct string 6e 44 6f 45 69 41 00 or nDoEiA.

Congratulations on making it through this part of the tutorial. You’ve now got all the tools you need for static reverse engineering! Don’t forget to solve the exercises to cement your skills.

crackme04.c

Now that you’ve seen all the tools and techniques you need to reverse engineer these crackmes, I’m only going to highlight the most important sections of each crackme. With crackme04, you can use the same basic process as before: open it up in Radare, run analysis, and seek to the main function. The flow graph should lead you right to a loop at the heart of the code, which looks like this:

movsx eax, al
add esi, eax
add ecx, 1
movsxd rax, ecx
movzx eax, byte [rdx + rax]
test al, al
jne 0x72e;[ge]

That jump is either back to the top if al (a single byte from the input string) isn’t zero, or to a block that just compares ecx with 0x10 and exits with failure if it’s not equal, and to another check if it is. That check jumps to failure if esi isn’t 0x6e2, and success if it is.

So what are ecx and esi? It’s easy to see that ecx is just a counter; at each loop iteration, it gets incremented, and is used to index the input string. So, after the loop is done, it’s equal to the number of non-zero bytes in the string.

esi is only modified in one line in the loop: it’s the sum of the numerical values of the characters in the string. It’s later compared to 0x632. So we need a 16-character string with the sum 0x6e2 (1762).

My method is to simply divide and then add in the remainder. 1913 over 16 is 110 remainder 2, so we just use character 110 (’n’) followed by a single 112 (‘p’): nnnnnnnnnnnnnnnp.

Appendix

The Makefile

The Makefile used is fairly straightforward, but it does have a few quirks. Prime among them is the use of objcopy on the compiled executables. I use it to strip out the FILE symbol, which would otherwise be used by Radare to display source code listings alongside the disassembly, completely defeating the purpouse of the exercise.

Exercises

The files named crackme01e.c, crackme02e.c, et cetera are modified versions of their non-suffixed counterparts. They are intended as exercises, and can be solved with exactly the same techniques as are shown in the respective sections of this tutorial. You will have a much better experience if you solve them before moving on.

Media

Saturday, Jan. 6, 2018: This tutorial was featured on Hackaday, which briefly brought down my server. Hi to all of you who make it here from there. Please consider checking out my other tutorials!

RandomUA

Wed, 08 Nov 2017 00:00:00 +0000

Install for Firefox or Chrome, or view on GitHub

What is RandomUA?

RandomUA is a WebExtension that changes your User-Agent string to a sensible-looking, desktop-browser-like UA string which is different for every request.

A User-Agent string is a piece of information that your computer sends to every website you go to. It tells them what software you use to access the Internet, what kind of computer you’re using, and what kind of operating system you use. Some of this is really important to those websites, but most of it isn’t, and it can be used to help track you without your consent.

Many browser fingerprinting solutions use the User-Agent as a relatively constant string, because most users UAs only change when their browser updates. Therefore, having a different UA on each request make it a lot easier to evade these tracking schemes.

Sample UA Strings

Mozilla/5.0 (Windows NT 5.64; rv:193.0) (KHTML, like Gecko)  AppleWebKit/193.0;
Mozilla/5.0 (Windows NT 7.59; rv:869.0) Gecko/20100101  Chrome/531.0 Firefox/869.0;
Mozilla/5.0 (Linux i686 on x86_64; rv:514.0) Gecko/20100101 Mobile Chrome/514.0;
Mozilla/5.0 (Windows NT 9.81; rv:77.0) Gecko/20100101  Chrome/77.0;

These generated UAs preserve important markers (like the Mozilla/5.0 and Gecko or like Gecko compat markers, and the Mobile flag) so that your browser isn’t served unusable content, but randomizes parameters that websites really shouldn’t care about (like OS version, browser version, and processor architecture) to protect your privacy.

Permissions

RandomUA requires access to all URLs and web request data for its core function: it has to be able to change your user agent string!

Other Platforms

Simply run the included make_xpi.sh script and install the generated XPI file.

Contributing

Contributions are welcome, either as PRs or issue reports. Please use GitHub issues for both bugs and feature requests.

Locking Down Firefox

Mon, 06 Nov 2017 11:32:43 -0600

I recently switched back to Firefox from Google Chrome. Firefox has gotten a lot faster recently, and is in the process of getting event faster, partly thanks to Mozilla’s Rust language, which is being used to write important components in a provably memory-safe way. This allows them to be made much more concurrent and parallel, leading to large speed improvements.

One big draw of Firefox is that it’s not a data collection tool for Google, and I’d like to double down on that promise. So, here’s a list of simple (and not-so-simple) ways to lock down Firefox 57, to keep your browsing private.

Extensions

Ad Blockers

First off, there are a lot of great browser extensions (using the standard WebExtension platform) which can improve your experience. The most important on the modern web is an ad-blocker of some kind. I personally use AdNauseam, which attempts to not only block ads but actually pretends to click on them, throwing off tracking and costing advertisers money. There are many other options available; I used to use the software that AdNauseam is based on, which is called uBlock Origin; it’s an excellent piece of software which simply blocks ads, with a very low memory and CPU overhead.

CDN Caching

Many websites use pieces of JavaScript and CSS code imported from CDNs (Content Delivery Networks) elsewhere on the web. Unfortunately, this allows the publishers of those libraries to track anyone who visits multiple websites that use them. It’s also kind of slow. Decentraleyes saves many of these shared pieces of code to your computer, then intercepts requests for them and uses your saved copy. This both prevents the CDN operators from tracking you and speeds up your browsing experience.

Adaptive Tracker Blocking

The Electronic Frontier Foundation publishes and maintains a great piece of software called Privacy Badger. This browser extension learns which domains appear over and over again on different websites, and uses some logic to determine whether or not they might be tracking you. It then gives you the option to allow, block, or block cookies from these domains. Combined with DecentralEyes, this is unlikely to break your browsing experience, but because it adaptively learns from your browsing behavior, it’s very effective at preventing certain kinds of tracking.

Fingerprint Resistance

There are a lot of pieces of information that a website can use to track you, even without explicitly dropping a cookie or other tracking data onto your computer. This is called browser fingerprinting; you can learn more at the EFF’s Panopticlick project. Firefox has a built-in tracking resistance mode, which I’ll cover later, but it’s very aggressive and breaks some websites. For me, this wasn’t good enough, so I built an extension that changes your User-Agent string, a major part of the browser’s fingerprint, on every single request. This makes it very hard to correlate your behavior.

HTTPS Everywhere

HTTPS is the basic underlying encryption technology for the web. While properly-configured websites generally upgrade HTTP requests to HTTPS when possible, you can install the HTTPS Everywhere extension (again by the great folks at the EFF) to make sure your browser check if HTTPS is known to be available and uses to it if possible. This not only improves your tracking protection, but also makes your computer significantly more resistant to the SSLStrip attack.

Config Settings

Third Party Cookies

First-party cookies - cookies set and read by the website you’re visiting - are immensely useful, allowing you to stay logged in across page loads and keep preferences saved. However, third-party cookies - cookies that are set by another site than the one you are visiting - are basically useless, except for tracking. You can disable them thus:

Go to Preferences.
Go to Privacy and Security.
Scroll to History.
Select Firefox will: Use custom settings for history.
Change “Accept third-party cookies” to Never.

First Party Cookies

You can also configure Firefox to clear all your cookies when you close it. If you’re ok with logging in to websites again, this is a good step to take; since I use a password manager, it’s not much of a hassle. In the same place as the third-party setting above, change “Keep until” to I close Firefox.

Tracking Protection

Firefox has a list of known trackers, and it can automatically block them, but only when you’re using Private Browsing mode. To enable this:

Go to Preferences.
Go to Privacy and Security.
Scroll to Tracking Protection.
Check the box entitled Use Tracking Protection in Private Browsing.

Combined with Privacy Badger, this effectively covers both your private and non-private use of Firefox for tracker blocking.

Multi Account Containers

This is a great feature which is not really in the scope of this guide, but I suggest you check it out here. It allows you to segregate some tabs from others, allowing you to have multiple accounts open in a single browser window, and effectively preventing all CSRF attacks.

Plugins

Plugins, like Adobe Flash and some kinds of DRM video decryption, can contain secret, unauditable code. They also often contain vulnerabilities. Firefox allows you to set these plugins to click-to-run, meaning that they require explicit user permission to execute. This has very little impact on your browsing experience, but a huge impact on security. To set this option:

Navigate to “about:addons”.
Select Plugins.
Set them all to Never Activate, except for the OpenH264 Video Codec and Widevine Decryption Module, which should be Always Activate, and Shockwave Flash, which should be Ask to Activate.

Isolate to First-Party Domains

Firefox has a lot of config options that aren’t normally exposed to the user, all accessed via the url “about:config”. When you go there, you’ll be warned that you might break Firefox. This is true, but not if you follow my instructions, so go ahead and click through.

Search for the setting privacy.firstparty.isolate and go ahead and set it to true. This will segregate sites from each other much more strongly than usual, making tracking very hard.

Done!

Congratulations! Your browser is now pretty dang difficult to track. If you have suggestions for more hardening methods, please contact me. I’d like to give a shoutout to @david_ross@mastodon.social and @gemlog@mastodon.sdf.org for some suggestions I used here.

Note: This post has been edited to correct errors in the spelling and description of operation of Decentraleyes and HTTPSEverywhere.

A Gentle Introduction to Practical Types

Fri, 29 Sep 2017 00:00:00 +0000

Programmers talk a lot about types, but what is a “type”, anyway? It is, in essence, the set of all possible values for some variable. Defining such a set gives us some information about what we can do with the value of that variable, in general.

For example, when speaking about numbers, we might say, “let x be any integer” or “let y be any real number not equal to zero”. These statements tell us what we can do with these values; we know that we can add, subtract, or multiply x and y. We cannot necessarily divide by x (because x might be equal to zero), though we can divide by y.

In programming, most languages have many built in types, and not just numbers; strings (of characters; that is, text) and more exotic things like pointers are all types of values that can be found in most programs, and the types of those values essentially tell us what we can do with them.

Types also tell us how to represent data. For instance, a variable of type int is not “any integer”; it is any integer between -2^31 and 2^31 - 1, because it is represented in memory as 32 bits, the first of which is the sign (negative or positive) and the rest of which are the absolute value of the number.

But before we can really talk about integers, decimal numbers, or text, we need to understand a little more about types themselves.

The Naming of Things

The first type I’m going to make is something that mathematicians call a Σ type or sum type. What it really is is a list of options; an enumeration of possible values, or, in many programming languages, an enum. Each option has its own type, and the enumeration of these options creates a new type.

Here’s a type that may be familiar to you: a Boolean value. It can be either True, or False. Nothing else. Here’s how we would write that down in a programming language (Rust, in this case):

enum Boolean {
    True,
    False
}

This is saying, “Create a new enumeration type. It’s called the Boolean type, and every value of this type must be one of the following: True, or False.”

We can figure out a few things about this. For instance, we can check if two values of type Boolean are equal or not; we can also define the Boolean functions like and, or, and xor, if we wish.

We need not be restricted to two options, however. For instance, here’s a type that you might use when giving someone directions:

enum StreetDirection {
    Left,
    Right,
    Straight,
}

We can use these two types to apply some type theory to a conversation. Let’s say Alice is giving Bob directions. Bob might ask, “Which way do I go at Laurel Street?” He is requesting a value of type StreetDirection; it would make no sense if Alice were to say “Blue” or “False”.

We can extend this construct to create very useful types like an unsigned integer (the always-positive variant of the bounded int discussed above). It can hold values from 0 to 2^64 - 1. Were we to define it using this syntax, it would look something like this:

enum PositiveBoundedInteger {
    0,
    1,
    2,
    3,
    (skip a few)
    18446744073709551615,
    18446744073709551616
}

Of course, (almost) nobody would ever actually write out all the integers from 0 to 18446744073709551616, but we could define the type that way. We could also begin to write functions like addition, subtraction, and multiplication.

Doing More with Enumerations

Consider the power of these sum or enumeration types; we can write the type of anything that is only one of a number of options, whether those options are ordered (like the PositiveBoundedInteger type above) or unordered (like Boolean values and directions).

These types are called sum types because the number of possible values is the sum of the number of options; for now, that’s just 1 each - there are 2 possible values of type Boolean, for instance - but that won’t always be true. We can define types that are the combinations of these types.

For example, we could define a type that is the combination of itself and a StreetDirection. We could imagine a controller giving a robot directions on how to move around with this type:

enum MovementInstruction {
    Go(StreetDirection),
    NoGo,
}

The MovementInstruction type has 2 options, but one of those options has a StreetDirection in it! Therefore, we could write little sets of instructions for our hypothetical robot to solve a maze:

Go(Straight)
Go(Right)
Go(Straight)
NoGo

It’s pretty easy to figure out how many options there are: Go(Straight), Go(Right), Go(Left), and NoGo. That is, the sum of the first option’s possible values (3) and the second option’s possible values (1). Thus, the number of possibilities for values is the sum of the number of possiblities for values of the types of the enumerated options. Enumeration types are sum types.

Getting Productive

Here’s a challenge for you, as a sort of perverse reward for making it through sum/enum types. How would you use an eunm type to represent values that record how many cats and dogs someone has?

I’d start like this:

enum Pets{
    NoCatsNoDogs,
    NoCatsSomeDogs(BoundedPositiveInteger),
    SomeCatsNoDogs(BoundedPositiveInteger),
}

This is a good start. We can talk about people with no pets, or only one type of of pet. We do have a problem, though; what about people who have dogs and cats? This is where product types or structured types come in. We can write something like this:

struct Pets {
    dogs: BoundedPositiveInteger,
    cats: BoundedPositiveInteger,
}

This definition means, “Create a new structured type. Its name is Pets, and it consists of one value of type BoundedPositiveInteger, called dogs, followed by another value of type BoundedPositiveInteger, called cats.” We call it a structured type because we define it not by a list of possible values but by the structure of any possible value; in this case, two arbitrary BoundedPositiveIntegers.

We could denote the state of owning no pets as:

Pets {
    dogs: 0,
    cats: 0,
}

That is, no dogs and no cats. A person with, say, 4 dogs and 3 cats would be:

Pets {
    dogs: 4,
    cats: 3,
}

Consider the number of possible values of the Pets type. There are 2^64 possible values of BoundedPositiveInteger, and here there are two BoundedPositiveIntegers, neither dependent on the other. Simple combinatorics tells us that there are 2^64 * 2^64 possible permutations; the product of the number of values of the two consituent types.

Combining the Combinations

Product and sum types can be combined to create any type. For instance, we could create a signed integer type, based on our existing BoundedPositiveInteger:

struct BoundedInteger {
    isNegative: Boolean,
    absoluteValue: BoundedPositiveInteger,
}

This would have as many values as BoundedPositiveInteger (2^64) times as many values as Boolean (2); in other words, 2^65 values.

More Exotic Types

So far, we’ve talked about types with sets of values with cardinality between 2 (Boolean) and 2^65 (BoundedInteger). Most useful types will have sizes like this; however, there are uses for types with both infinite cardinality and only one possible value.

I L 1 N I L 2 N I L 3 N x

Infinitely Large Types

We can use the combination of sum types and product types in another way: to define lists. Let’s say we wanted a list of numbers; perhaps winning lottery numbers. For this, we need two types:

struct IntegerListNode {
    value: BoundedInteger,
    link: IntegerListLink,
}

enum IntegerListLink {
    HasNextNode(IntegerListNode),
    NoNextNode,
}

Each node holds a link which points to either another node (the next in the list) or to nothing (signifying the end of the list). We could construct a list and then look at each of its elements in sequence. Writing such a list out is a bit daunting; The list 1, 2, 3 would read, in full:

IntegerListNode {
    value: 1,
    link: HasNextNode(
        IntegerListNode {
            value: 2,
            link: HasNextNode(
                IntegerListNode {
                    value: 3,
                    link: NoNextNode,
                }
            ),
        }
    ),
}

We do run into a bit of a problem, though, when looking at the size of this type. Clearly, the IntegerListNode type has as many possible values as a BoundedInteger times as many possible values as an IntegerListLink. We know that the BoundedInteger type has 2^65 values, but how many does IntegerListLink have? Well, it’s the sum of 1 (for NoNextNode) plus… however many possible values IntegerListNode has! This list type is a type with infinitely many possible values, because the nesting of nodes and links can go on for arbitrarily long.

Sidebar: Representing Infinite Types in Real Computers

The property just described actually causes the Rust compiler (which sadly must exist in the real world) to reject this type as invalid, because it cannot generate a representation that would fit into a computer’s memory. Because each node contains all subsequent nodes, and because one of the things that makes Rust programs fast is that values are, by default, allocated all at once, the compiler would have to ask the computer to allocate the maximum number of nodes that could ever exist. Since we didn’t introduce any limit to the number of nodes, that would require an infinite amount of memory. This is where indirection via pointers comes in; see the Rust Book¹ for more info.

Any programming language that puts values on the stack by default will have this problem; for instance, C and Rust will require explicit indirection, while e.g. Python and Java do it for you, at a bit of a speed cost.

It’s important to remember that while the current Rust implementation doesn’t like this type, there’s no reason that Rust as a language can’t handle it. It’s a very useful type, and a good logical description of how linked lists work, even though we have to add some incantations to get them to work in the real world.

Zero Sized or Single Value Types

One thing that’s interesting to note is that we have a bit of a discrepency between two kinds of enumeration type options. Some of them have other types associated with them, and some don’t. Recall our type MovementInstruction:

enum MovementInstruction {
    Go(StreetDirection),
    NoGo,
}

This is actually a simplification of the “real” types here. NoGo technically “contains” a value of what we call the “unit type”, which we denote () or ○. Values of this type can only have one value; the unit value, also denoted () or ○. So, we can re-write this type:

enum MovementInstruction {
    Go(StreetDirection),
    NoGo(()),
}

We can see that this has a size of 0: there is only one possible value, so whenever a value with this type appears, we know what the value is. We also can’t do anything at all with the unit value, meaning that we can’t do anything with variables of the unit type.

However, the unit type can be useful for conveying information to the compiler, or to help the compiler figure out when the programmer has made a mistake. For instance, in Rust, we say that things which return nothing (for instance, if something { end_the_world(); }) actually return the unit value. This gives us errors like this:

error[E0308]: mismatched types
 --> test.rs:2:24
  |
2 |   fn something() -> bool {
  |  ________________________^
3 | |     println!("This doesn't return anything!");
4 | | }
  | |_^ expected bool, found ()
  |
  = note: expected type `bool`
             found type `()`

Recap & Further Reading

It’s been a long journey. We’ve gone from nothing, through finite lists of possible values (enumerations/sum types), to combinations of those lists (structured types/product types) and, finally, types with only one possible value and those with infinitely many possible values. Go forth and apply this knowledge!

To help you in actually understanding how to use the type system of Rust specifically, I suggest the excellent Rust Book¹ and Sean Griffin’s talk on the type system² from RustConf 2017. Traits and lifetimes are the next topics I’d look into in order to develop a better understanding of how Rust leverages the type system to make guarantees about your programs. Most of these ideas are applicable to other languages, like TypeScript, Java, and of course OCaml-like and Haskell-like languages.

The examples here will not compile on their own; I have elided use statements and other such language-specific constructs to make a point. Refer to the Rust Book for information on how to actually use the types.

The Rust Programming Language, 2nd Edition, by S. Klabnik and C. Nichols. 2018. O’Reilly. ↩︎ ↩︎
Type System Tips for the Real World, by Sean Griffin at RustConf 2017. ↩︎

BattleDome VR, a Review

Fri, 16 Jun 2017 00:00:00 +0000

I hadn’t really decided whether or not I liked Battle Dome until I punched a wall with my Vive wand while trying to poke my rifle out from behind cover to distract the sniper that was killing all my teammates. Then I decided that I liked it very, very much.

Battle Dome, available on Steam, is a 5v5 first person shooter that manages to combine all the best elements of second-generation teathered VR with a solid core of multiplayer shooter gameplay. It’s not a particularly innovative game in the way Quanero or Accounting are; its graphics are decent, but not mindblowing, and it has no story to speak of. Upon starting the game, you’re presented with a “lobby” area in which you can try out any of the vast array of weapons, from pistols to laser rifles to grenade launchers (each one of which has seperately tracked personal stats) and, when ready, join or create a game.

There are a number of gamemodes available, from traditional FPS deathmatch to an interesting cooperative horror mode. There is also a lot of variation between maps, even in mechanics; some allow players to alter gravity, some have jet-packs available, and many are so called “paint” maps. This is where one of the most interesting mechanics comes into play; in order to cater to both players who enjoy free movement in vr and those who get motion sick from the difference between percieved movement and inner-ear inertial measurement, the game allows both teleportation and trackpad-based free movement, but on paint maps, players can only teleport to places where their teams color has been painted. This leads to some very exciting Splatoon-style paint raids. In one of my recent matches, two players with a machine gun and a grenade launcher guarded me while I used dual paint guns to spray a path to the enemy base so we could destroy their core in the attack-defense mode.

The huge selection of weapons in the game is one of its greatest features. Three basic types of weapons exist: bullet-firing, plasma, and laser weapons, each of which has unique advantages and disadvantages. There are large and small guns, too, from pistols and one-handed automatic weapons to assault rifles, light machine guns, and even a huge-scoped sniper rifle, and all of them are designed for VR. Every one, even the huge rocket launcher, can be weilded one-handed, but many of the larger ones have an optional two-handed mode which I’ve found increases both accuracy and immersion. There are more unique weapons, too, like the multi-use grenade launcher, which has utility grenades such as smoke, and the ricochet gun, whose multicolored rounds bounce off walls, ceiling, and floor to devastating effect. In addition, the shortsword and lance can be paired with the energy shield for that authentic space-Spartan feel.

I do have a few complaints. The graphics, especially the character models, can be somewhat offputting, with low-resolution textures and weapons floating in mid-air. The teleporting movement is balanced using a cooldown timer which isn’t adjusted based on distance, so tiny adjustments freeze you in place for just as long as full-range hops. There are a lot of maps, but each has different features enabled, like paint, gravity control, and jetpacks; it would be very nice to be able to enable and disable those features at will, for maps that support them.

That’s not to say you shouldn’t play it, though. Battle Dome, like the original Unreal Tournament, is an well-built, solid game that, while not particularly innovative, combines most of the mechanics of a new medium to create a game that can be learned in a few minutes but has nearly infinite replayability. I give it an 8/10.

Quanero VR, a Review

Sat, 27 May 2017 00:00:00 +0000

Quanero VR, available from “Laserboys3000” on Steam for nothing, is the best of the many VR experiences I’ve encountered so far.

This isn’t because it’s particularly beautiful (it’s graphics are competent, much more so than those I’m able to create, but not on a AAA level) nor because it’s gameplay is particularly well designed (in fact, it has almost no “gameplay”). Quanero is amazing because it’s the first truly “player-motivated” VR experience I’ve seen.

VR provides the ability to create almost perfectly immersive virtual worlds, and Quanero demonstrates how to use that immersion to tap into intrinsic motivations of players - in this case, mostly curiosity and a little bit of completionism. Quanero presents a tiny virtual world - just a terrace and small bar - with the hints of a large, cyberpunky, Blade Runner-esque world around it.

The experience begins with a little minor explanation. You are informed that you’re a detective and that you can run time forwards and backwards, but you may only observe, not alter things. This sets the perfect stage for pure exploration, since the player’s actions can’t have any consequences. You’re then shown an array of thumbnails, blurred out, of events and interactions you haven’t yet seen. That sets up the completionism; what player could resist at least trying to fill those in?

You’re dropped onto the terrace and, running the time forward with the right wand’s trigger, you see a peaceful scene rudely interrupted by a massive explosion from the outdoor grill, which leads to a dramatic rescue and a fistfight. Personally, I was very curious as to why the explosion occurred, so I teleported myself over to the grill and ran time backward, forward, backward again - and this was where the experience really got immersive for me. I saw a flash of green just before the explosion and I was hooked.

Long story short (and less spoilerific), I spent about twenty minutes figuring out what the green flash was, then where it had come from, and then piecing together the whole story. The scene is maybe five minutes long, but it took me much longer to discover the true sequence of events, and the experience was utterly compelling throughout, without any extrinsic motivation. Looking at that fistfight in great detail didn’t unlock a new level or improve my combat stats - it was just interesting. That’s the true promise VR, in my opinion - it allows game designers to engage players in a much more immersive and intrinsic manner, and to drop many of the traditional gimmicks and Skinner-box tactics that motivate players in traditional games.

This is not to say that VR games automatically (or necessarily) tap into these motivations, however. For instance, Raw Data from Survios Inc. is a traditional first person combat game that smoothly integrates the unique traits of VR without drastically changing the formula; the player’s motivation basically comes from the infinite cycle of beat up robots, get better at beating up robots, find more and more interesting robots to beat up, repeat. That doesn’t make it a bad game; in fact, Raw Data is one of my favorite VR experiences. It just isn’t revolutionary in the way Quanero VR is.

My ultimate wishlist VR item is a longer, more intense, more complex, and more diverse version of Quanero in which the player must find clues in one part of the environment to understand events in other parts. In Quanero, this essentially only happens with the strange green flash that causes the explosion; in a larger game, there could be more such events and more levels of uncertainty. There could also be a few more people who weren’t ripped white dudes - and no, one blue alien, one skinny female pool player with no characterization, and a fat dude who dies right away don’t count.

So, since this seems to have become a review, I guess I’ll give a score. Quanero gets 8/10; not perfect, but pretty damn good, especially given the price.

Thoughts on Virtual Reality

Fri, 26 May 2017 00:00:00 +0000

I was recently fortunate enough to be the recipient of a grant for $2000 to learn about virtual reality and develop a video game using the HTC Vive. So far, I’ve mostly been exploring prior art.

There is the problem of cost; the $2000 went pretty quickly, first for a GTX 1080 to produce the graphics and then for the $800 Vive itself, along a bit of money for tripods for the position-tracking lighthouses. Then there’s the issue that making independent games for VR is difficult, mostly due to art assets needing to be fairly detailed. There’s only so much blank, untextured whiteness, so many perfectly flat surfaces, that one can stand when staring at them in virtual reality. Once these hurdles are leapt, however, the result is… pretty impressive.

I started out my VR journey in Valve’s The Lab, a collection of VR experiments, all impressively animated and modelled. I started with the phyics-based games: a Galaxians-like shoot-em-up in which the player’s hand position controls the ship, an archery simulation, a slingshot game. Then I went into some of the more passive experiments: a Scandinavian mountaintop, a magician’s shop, the human body. All in all, very impressive stuff.

From there, I sought out some more substantial games and hit upon Raw Data, the premier full-length game for the HTC Vive. It blew me away with its simplicity: you get a pistol or a sword and you beat up evil robots. Maybe most cliché video game possible, and yet the sense of actually holding the sword, actually dodging the enemy’s blows, made it fresh and new.

I’ve gone through a number of ideas for VR games to make myself. I’m limited somewhat by my relative lack of 3D art knowledge, but one can always buy assets from the Unity asset store, and (as another part of the grant) I’m taking a Udemy course in 3D art, so perhaps my skills will be sufficient. In any case, it’s an interesting medium and I hope I’ll be able to produce something thought-provoking and/or enjoyable.

SBrain, an extension of BrainF.ck

Tue, 02 May 2017 00:00:00 +0000

SBrain, or Semantic Brain, is a language based on Urban Müller’s famous language Brainf*ck with only 8 symbols (3 bit instructions). SBrain’s additions increase the number of symbols to 32 (6 bit instructions), including bit-shifting and arithmetic, and add a stack and a register.

Having these additional facilities allows SBrain to be far more expressive while retaining its attractiveness as a genetic medium for evolutionary algorithms. I’ve been interested in genetic programming for a long time. After my less-than-stellar results from a Python implementation of a Brainf*ck (which you can see here, though the code is truly awful), I decided to try a new tack. I designed SBrain over about two years and implemented an interpreter in Rust, which you can find here.

SBrain is a strict superset of its parent language. It does have some important differences other than its additional instructions, most notably that unbalanced jump ([ and ]) instructions are valid. For instance, the program .+] will print ascending numbers forever (as opposed to the strictly legal [.+] required in BF.)

Evolving Computer Programs

Once I finished an initial implementation, I decided to try and get evolution working. I ended up with a program whose output looks a bit like this:

Generation     1 Cost  3075: aS|.p!.S
Generation     5 Cost  3074: aS|.^!p!.S
Generation     6 Cost  3073: aS|.^!ap!S.S
Generation     7 Cost  2052: aS|.^!ap-.S.S
Generation     9 Cost  1028: a&|.^!ap..S.S
Generation    11 Cost     7: a&|.^!ap..S..S
Generation    16 Cost     5: a&|.^-!ap..S..S
Generation    29 Cost     3: a&m|.^-!ap..+..S
Generation    34 Cost     2: a&&m|.^-!ap..+.d.S
Generation   285 Cost     1: a&&m|+.^-!p..+.d.S
Generation  1101 Cost     0: a&&m|+.^!p.+.+.d.S
Program found after 1101 tries.
a&&m|+.^!p.+.+.d.S
Ran for 128 cycles and did not halt
[] -> a&&m|+.^!p.+.+.d.S -> [1, 2, 3, 4, 5]

This particular run was looking for the output [1,2,3,4,5] given no input. Built with --release, it took about 3 seconds (2.93, to be precise). The algorithm is fully parallelized using Rayon and keeps all four of my laptop’s CPU cores at around 80 to 100 percent most of the time. The program it found is not particularly efficient (the optimal program being +.+.+.+.+.), but it’s not extremely inefficient either.

How it works

The actual algorithm is pretty simple. It consists of a few basic steps:

Generate an inital, entirely random population of programs.
“Cost” the population (that is, execute them and see how far off they are from the desired outputs)
Remove the bottom 50% of programs and replace them with variants of the top program, crossed with the programs in the top 50%.
Repeat until at there is a program with the cost of 0

The program reports its state only when there is an improvement; as you can see above, improvements come thick and fast in the beginning and slow down a great deal later.

Generation

The generation step is fairly simple. A set of valid source code symbols is hard-coded into the program, and the initally generated population is a set of purely random strings of those characters.

Costing

Costing is the most important part of the process, but is really quite simple in this case. The program is run in the SBrain VM and provided with the correct input. Its output is then analyzed against the expected output. There are two steps of this analysis. If the lengths of the actual vs. expected outputs don’t match, the program is heavily penalized (2 to the 10th points for each missing or extra character). Then, the characters of each input are compared and their difference is added to the cost.

Mutation

Mutation actually occurs at two levels: the population level and the program level. The population is sorted by cost, then essentially cut in half. The top half, the most successful programs, are mutated, and their mutated genomes are crossed with a mutated variation of the top program.

This crossing is immensely important to the rapid generation of workable algorithms. It allows two relatively sucessful genomes to share their best features, which further allows “breakthrough” steps in evolution. I initally implemented the algorithm without crossing, and it was incredibly slow at finding even very simple programs.

At the program level, mutation is merely a random choice between three alternative: adding a source character, changing a source character, and deleting a source character.

Conclusions

This algorithm has proven to be pretty effective for trivial problems like addition and generating ranges of values. There is currently a great deal of copying present in the code, but I hope to spend some time in the next few weeks (once finals are over, that is) making a zero-copy or almost-zero-copy implementation.

If you have ideas for how to make my code or algorithm more effective or faster, please oh please email me or send in a PR! The code is on Github.

MLeM, a VM for genetic programming

Mon, 17 Apr 2017 04:48:50 +0000

I’ve recently been working on a project called the Machine Learning Machine, or MLeM. It’s a VM implemented in the Rust programming language which I hope to use as a basis for some genetic programming.

It’s a Harvard architecture machine, meaning that it has separate representations and memory for data and program segments. While this is not the way most modern computers work, it does model the more secure W XOR X functionality that exists in operating systems such as BSD and allows me to properly utilize the amazing type system of the Rust language to do compile time verification of a lot of the system.

The machine has 8 general purpose registers, a “hardware” stack (that is, it has Stack Pointer and Base Pointer registers and PUSH and POP instructions), and built in I/O instructions. This makes is slightly abstracted over a real machine, but still makes any code written for it relatively easy to turn into actual machine code for real CPUs.

All data is stored in 64-bit registers and memory cells. Instructions, however, are represented using Rust’s enums. I chose to go with a unified-addressing scheme. This glosses over the realities of memory access, but because all accesses are already tagged by virtue of the system itself, I can easily add performance penalties for main memory access and even a cache simulation when needed.

A sample instruction looks like this:

Add(a, b)

where a and b are of type Address. So

Add(RegAbs(R1), MemReg(R2))

would add the value in R1 to the value at the memory address in R2 and place the result in R1. Note also that R1 and R2 are enum variants of Register; trying to create something like

Add(RegAbs(R1), MemReg(100))

would produce a compiler error.

I’ve also developed an extremely simple assembly syntax for testing purposes; the above instruction would look like:

add r:r1 p:r2

That is, add (register) r1 to (pointer) r2.

The primary goal for this project is to provide a more stable and closer-to-real-CPU language than Muller’s Brainf*** or my own SBrain, both of which I’ve previously used for genetic programming with rather mixed results.

The ability to generate instructions I know are valid helps short-circuit the process of culling unparseable genes from the population, which, while not particularly difficult, slows the process of evolution significantly.

Look out for a future post to discuss my previous experiences with GP and the future of this project.

Getting started with Piston, a game library for Rust

Tue, 14 Mar 2017 04:48:50 +0000

Author’s note: Piston has moved on since March of 2017. This tutorial, for 0.64, is preserved for posterity; I suggest you look instead at the official getting started guide.

I’ve been interested in game programming for a while, and I periodically sample the Rust ecosystem’s offerings around game programming. Piston seems like the most promising candidate, but the tutorials are out of date, so here’s a simple one.

This tutorial will show you how to build a simple windowed application with a time-locked update rate and keyboard controls. You can extend it with image loading, music, and many other features of the Piston library.

Note: This tutorial assumes that you have a recent version of Rust installed, that you know how to build hello world in Rust, and that you know about creating projects and installing packages with Cargo, Rust’s package manager.

We’re going to create a simple windowed application that draws a square of color which bounces around. This will exercise the basic 2D abilities of Piston and set you up for future projects.

To get started, just cargo new --bin with whatever name you like. Our Cargo.toml only needs one dependency: piston_window = "0.64.0". All of our code will go in src/main.rs. Go ahead and delete the existing main function.

A Prototype

First, we have to make some simple physics. We’ll need only one simple data structure, a ColoredRectangle, which will store three fields: position/size, color, and velocity.

Piston needs to know the position/size (an array of f64 values [x_position, y_position, x_size, y_size]) and the color (f32 values [r, g, b, alpha]), but only the rectangle itself needs to know about its velocity, so the fields look like this:

struct ColoredRect {
    pub color: [f32; 4],
    pub position: [f64; 4],
    velocity: [f64; 2]
}

We can also create a new() function to create a default ColoredRect. The default will start at the top left corner, a size of 100x100 pixels, with a color of (1, 0.5, 0.25, 1.0), and have a 0.3 pixel per frame velocity in both dimensions.

impl ColoredRect {
    fn new() -> Self {
        ColoredRect {
            color: [1.0, 0.5, 0.25, 1.0],
            position: [0.0, 0.0, 100.0, 100.0],
            velocity: [0.3, 0.3]
        }
    }
}

Now we need to make the update routine, which will be called every frame. Colors are between 0.0 and 1.0, so we’ll simply make a method to cycle and wrap. Add this to the impl block:

fn update_color(color: f32)->f32 {
    if color <= 0.0 {
        1.0
    } else {
        color - 0.001
    }
}

Now comes the big update function. It has a few responsibilities; first it should change all the colors (but not the alpha). Then, it should move the block by its velocity. It also has to check whether or not the block has hit the edge of the window, in which case it should negate the velocity. Here’s just the color updating

fn update(&mut self, size: (f64, f64)) {
    self.color[0] = Self::update_color(self.color[0]);
    self.color[1] = Self::update_color(self.color[1]);
    self.color[2] = Self::update_color(self.color[2]);
    
    // Collision check X
    // Move X

    // Collision check Y
    // Move Y
}

The collision check is pretty easy, but does require a little thought. Checking the left and top sides are easy; position[0] and position[1], respectively, are X and Y of the top left corner of the block; if either of these is less than 0, the block should bounce. The bottom and right, however, are position[0] + position[2] and position[1] + position[3]; that is, the upper or left position plus the size in that dimension. These are checked against the window size, passed in as a parameter.

To bounce all we have to do is velocity[0] = -velocity[0]; for X and similar for Y. Applying velocity is as easy as position[0] += velocity[0]; position[1] += velocity[1];. Here’s the final code:

fn update(&mut self, size: (f64, f64)) {
    self.color[0] = Self::update_color(self.color[0]);
    self.color[1] = Self::update_color(self.color[1]);
    self.color[2] = Self::update_color(self.color[2]);
    // X updates
    if self.position[0] + self.position[2] >= size.0 ||
        self.position[0] < 0.0 {
        self.velocity[0] = -self.velocity[0];
    }
    self.position[0] += self.velocity[0];

    // Y updates
    if self.position[1] + self.position[3] >= size.1 || 
        self.position[1] < 0.0 {
        self.velocity[1] = -self.velocity[1];
    } 
    self.position[1] += self.velocity[1];
}

Now have to hook this up to something that will actually display it; we need a main function.

First we create a ColoredRect with the new() function. Then, we create a new PistonWindow with the builder provided by the library. The constructor takes a title and resolution (in this case 640 x 480) and the two additional functions set the window to close when the Escape key is pressed, and turn on vertical sync.

A small aside: the online Piston docs are hopelessly out of date. I suggest running cargo build (which will fail, but will download all the dependency packages) and then cargo doc. The full, up-to-date docs will be available in <project>/target/doc/piston-window/; point your favorite web browser at it and go nuts.

fn main() {
    let mut rect = ColoredRect::new();
    let mut window: PistonWindow =
        WindowSettings::new("Hello Piston!", [640, 480])
        .exit_on_esc(true)
        .vsync(true)
        .build().unwrap();
    // Do things that have to be done every frame
} // Rust's automagical resource management means we're done!

Now comes the event loop - that // Do things that have to be done every frame. The window has a function .next() which returns an Option of the event, so we simply iterate over all the events and match the two we’re interested in: Render and Update. These both come with some useful information such as delta-time, but for now we’re just throwing that out.

Updating is easy; there’s a single method call, which is given the resolution of the window. Drawing is pretty easy as well; the PistonWindow instance has a .draw_2d() function that takes a lambda. In this case we tell it to clear the whole window to a [1.0, 1.0, 1.0, 1.0] color, or an opaque white, and then create a rectangle with the color and position arrays we already have in our ColoredRectangle.

while let Some(e) = window.next() {
    match e {
        Input::Render(_) => {
            window.draw_2d(&e, |c, g| {
                clear([1.0; 4], g); // Clear to white
                rectangle(rect.color, // Color
                          rect.position, // Position/size
                          c.transform, g);
            });
        }
        Input::Update(_) => {
            rect.update((640.0, 480.0));
        }
        _ => {}
    }
}

This is now a complete program; cargo run results in a nice white window with a colored square bouncing off the walls.

Controls & Full Event Support

No game engine is complete without user interaction, of course; so let’s give the user the ability to set the velocity! This is also a great time to integrate the delta-time provided by the update function; it tells the game logic how many fractions of a second have passed since the last update. This helps keeps updates consistent and lets you avoid the pitfalls of old Windows games. This entails a few small changes. First, we have to add dt arguments to the update and update_color functions. Because colors are calculated in f32 space and position is calculated in f64 space, we make the dt argument to update an f64 and the one to update_color an f32, and downcast inside of update. Then we multiply all the time sensitive transforms by dt, and grab the UpdateArgs object from the Event::Update in the match expression. The example was running at around 120 FPS on my machine, so I stuck in a * 120 to make the speed about the same.

So the changes are like this:

impl ColoredRect {
    fn new() -> Self {
        ColoredRect {
            color: [1.0, 0.5, 0.25, 1.0],
            position: [0.0, 0.0, 100.0, 100.0],
            velocity: [0.3, 0.3]
        }
    }
    fn update(&mut self, dt: f64, size: (f64, f64)) {
        self.color[0] = Self::update_color(dt as f32, self.color[0]);
        self.color[1] = Self::update_color(dt as f32, self.color[1]);
        self.color[2] = Self::update_color(dt as f32, self.color[2]);
        // X updates
        if self.position[0] + self.position[2] >= size.0 ||
            self.position[0] < 0.0 {
            self.velocity[0] = -self.velocity[0];
        }
        self.position[0] += self.velocity[0] * dt * 120.0;

        // Y updates
        if self.position[1] + self.position[3] >= size.1 || 
            self.position[1] < 0.0 {
            self.velocity[1] = -self.velocity[1];
        } 
        self.position[1] += self.velocity[1] * dt * 120.0;
    }
    fn update_color(dt: f32, color: f32)->f32 {
        if color <= 0.0 {
            1.0
        } else {
            color - 0.001 * dt * 120.0
        }
    }
}

fn main() {
    let mut rect = ColoredRect::new();
    let mut window: PistonWindow =
        WindowSettings::new("Hello Piston!", [640, 480])
        .exit_on_esc(true)
        .vsync(true)
        .build().unwrap();
    while let Some(e) = window.next() {
        match e {
            Input::Render(_) => {
                window.draw_2d(&e, |c, g| {
                    clear([1.0; 4], g); // Clear to white
                    rectangle(rect.color, // Color
                                rect.position, // Position/size
                                c.transform, g);
                });
            }
            Input::Update(u) => {
                rect.update(u.dt, (640.0, 480.0));
            }
            _ => {}
        }
    }
}

We can now also add in resizeablilty! The Event::Render enum provides a RenderArgs struct which, among other things, has width and height fields. We can use a mutable binding to allow this to be resized every frame, and pass that into update() instead of the hardcoded 640 x 480. Unfortunately, these are given as f32 and our physics code needs f64, so we have to upcast.

let mut window_size: (f64, f64) = (0.0, 0.0);

while let Some(e) = window.next() {
    match e {
        Input::Render(r) => {
            window_size = (r.width as f64, r.height as f64);
            window.draw_2d(&e, |c, g| {
                clear([1.0; 4], g); // Clear to white
                rectangle(rect.color, // Color
                            rect.position, // Position/size
                            c.transform, g);
            });
        }
        Input::Update(u) => {
            rect.update(u.dt, window_size);
        }
        _ => {}
    }
}

For controls, one of the simplest things to add is a velocity control. We can do this by adding a change_velocity function to the ColoredRect object. We’ll then see how to call that from keyboard events. The function itself is simple:

fn change_velocity(&mut self, factor: f64) {                                                                                                                                                                                                        
    self.velocity[0] *= factor;
    self.velocity[1] *= factor;
}

We also have to complicate the match statement. All input events are encapsulated in Event::Input(Input), so we match on that and add a sub-match that looks for the Press, and a submatch under that for the keys we want (w and s, in this case). When W is pressed, we increase velocity by a factor of 1.1, and when S is pressed, we use a factor by 0.9.

while let Some(e) = window.next() {
    match e {
        Input::Render(r) => {
            window_size = (r.width as f64, r.height as f64);
            window.draw_2d(&e, |c, g| {
                clear([1.0; 4], g); // Clear to white
                rectangle(rect.color, // Color
                            rect.position, // Position/size
                            c.transform, g);
            });
        }
        Input::Update(u) => {
            rect.update(u.dt, window_size);
        }
        Input::Press(b) => {
            match b {
                Button::Keyboard(k) => {
                        match k {
                            Key::W => {
                                rect.change_velocity(1.1);
                            }
                            Key::S => {
                                rect.change_velocity(0.9);
                            }
                            _ => {} // Catch all keys
                        };
                    }
                    _ => {} // Catch non-keyboard buttons
                };
            }
        _ => {} // Catch uninteresting events
    }
}

Finishing Touches

This is a totally working example of drawing and keyboard input with Piston, specifically piston_window. However, to really polish it up, there are a few things to add. First, we can change the name to “Flying Square”. I also added a parameter to update_color to control how much the color should change, and added a match so that the F5 key resets the square to the top left. The final program looks like this:

extern crate piston_window;
use piston_window::*;

struct ColoredRect {
    pub color: [f32; 4],
    pub position: [f64; 4],
    velocity: [f64; 2]
}

impl ColoredRect {
    fn new() -> Self {
        ColoredRect {
            color: [1.0, 1.0, 1.0, 1.0],
            position: [0.0, 0.0, 100.0, 100.0],
            velocity: [0.3, 0.3]
        }
    }
    fn update(&mut self, dt: f64, size: (f64, f64)) {
        self.color[0] = Self::update_color(dt as f32, self.color[0], 0.001);
        self.color[1] = Self::update_color(dt as f32, self.color[1], 0.002);
        self.color[2] = Self::update_color(dt as f32, self.color[2], 0.003);
        // X updates
        if self.position[0] + self.position[2] >= size.0 ||
            self.position[0] < 0.0 {
            self.velocity[0] = -self.velocity[0];
        }
        self.position[0] += self.velocity[0] * dt * 120.0;

        // Y updates
        if self.position[1] + self.position[3] >= size.1 || 
            self.position[1] < 0.0 {
            self.velocity[1] = -self.velocity[1];
        } 
        self.position[1] += self.velocity[1] * dt * 120.0;
    }
    fn update_color(dt: f32, color: f32, change: f32)->f32 {
        if color <= 0.0 {
            1.0
        } else {
            color - change * dt * 120.0
        }
    }
    fn change_velocity(&mut self, factor: f64) {
        self.velocity[0] *= factor;
        self.velocity[1] *= factor;
    }
}

fn main() {
    let mut rect = ColoredRect::new();
    let mut window: PistonWindow =
        WindowSettings::new("Flying Square", [640, 480])
        .exit_on_esc(true)
        .vsync(true)
        .build().unwrap();

    let mut window_size: (f64, f64) = (0.0, 0.0);

    while let Some(e) = window.next() {
        match e {
            Input::Render(r) => {
                window_size = (r.width as f64, r.height as f64);
                window.draw_2d(&e, |c, g| {
                    clear([1.0; 4], g); // Clear to white
                    rectangle(rect.color, // Color
                                rect.position, // Position/size
                                c.transform, g);
                });
            }
            Input::Update(u) => {
                rect.update(u.dt, window_size);
            }
            Input::Press(b) => {
                        match b {
                            Button::Keyboard(k) => {
                                match k {
                                    Key::W => {
                                        rect.change_velocity(1.1);
                                    }
                                    Key::S => {
                                        rect.change_velocity(0.9);
                                    }
                                    Key::F5 => {
                                        rect = ColoredRect::new();
                                    }
                                    _ => {} // Catch all keys
                                };
                            }
                            _ => {} // Catch non-keyboard buttons
                        };
                    }
            _ => {} // Catch uninteresting events
        }
    }
}

You can also check out the associated GitHub repository; comments about the code or improvements are welcome as issues or PRs there, or comments on this post. You might want to look at my Rust state machine tutorial or my post about session types in Rust.

Happy hacking!

Update: I’ve reorganized the project and added an FPS counter to the top left hand corner. See my comments on that commit to learn a bit more.

Session Types

Thu, 16 Feb 2017 01:49:50 +0000

Session types are a technique for using a rich type system, like that of Rust or OCaml, to prevent the representation of certain kinds of illegal states. Here, I’ll illustrate them with a (somewhat contrived) example.

What is the use-case?

Let’s take the example of a system representing packaging and shipping boxes. I want to create a Package data structure, pack data into it, close it (preventing adding data), address it, and then ship it. It makes no sense to send an un-addressed Package, or to insert data into a closed one.

We could represent such a data structure like this:

struct Package<T> {
    is_closed: bool,
    is_addressed: bool,
    data: T
}

We would then have lots of runtime typechecking of whether boxes submitted for shipping were addressed and such, which costs time and is error-prone. Wouldn’t it be nice if the compiler could enforce this?

How is it implemented?

The easiest way to implement this is with three different types: an OpenPackage, a ClosedPackage, and an Addressed Package.

struct OpenPackage<T> {
    pub contents: T
}

Crucially, the contents field is pub. You can poke and prod and change that data all you want.

This has a few capabilities: pack, which takes control of whatever is supposed to go in the package and creates an OpenPackage around it, unpack, which destroys the OpenPackage and gives back its contents, and finally close, which converts the OpenPackage to a ClosedPackage.

impl <T: Sized> OpenPackage<T> {
    fn new(contents: T) -> Self {
        OpenPackage::<T> {contents: contents}
    }

    fn pack(contents: T) -> Self {
        println!("\tPut some data in a package.");
        Self::new(contents)
    }

    fn unpack(self) -> T {
        println!("\tPackage unpacked.");
        self.contents
    }

    fn close(self) -> ClosedPackage<T> {
        println!("\tPackage closed.");
        ClosedPackage::<T>::new(self.contents)
    }
}

This leads naturally to the ClosedPackage struct:

struct ClosedPackage<T> {
contents: T
}

Very similar, but without the pub attribute. This means that a ClosedPackage isn’t in danger of having its contents manipulated in any way.

ClosedPackages can be opened again, yielding an OpenPackage, or addressed, creating an AddressedPackage.

impl <T: Sized> ClosedPackage<T> {
    fn new(contents: T) -> Self {
        ClosedPackage::<T> { contents: contents }
    }
    fn open(self) -> OpenPackage<T> {
        println!("\tPackage opened.");
        OpenPackage::<T>::new(self.contents)
    }
    fn address(self, address: String) -> AddressedPackage<T> {
        println!("\tAddressed a closed package.");
        AddressedPackage::new(self.contents, address)
    }
}

Finally, the AddressedPackage struct represents one with a specified destination. I used a String for the address here, but it would be trivial to create a generic version.

struct AddressedPackage<T> {
    contents: T,
    pub address: String
}

To understand the access controls here, just think of a physical package. The address is on the outside; anyone can read it or cross it out with a sharpie. The contents, however, are sealed away.

This struct can be turned back into a ClosedPackage by receiveing it:

impl <T: Sized> AddressedPackage<T> {
    fn new(contents: T, address: String) -> Self {
        AddressedPackage::<T> { contents: contents, address: address }
    }
    fn receive(self) -> ClosedPackage<T> {
        println!("\tPackage recieved.");
        ClosedPackage::<T>::new(self.contents)
    }
}

Finally, I created an example function to “send” the package somewhere.

fn send_package<T: Sized+std::fmt::Display>(package: AddressedPackage<T>) -> Result<String, String> {
    // Save the address.
    let address = package.address.clone();

    // Destroy the package to get at the contents
    let contents = package.receive().open().unpack();
    println!("Destination recieved: {}", contents);
    // Success! Theoretically this function could fail, but not with this implementation.
    Ok(format!("Sent package to {}", address))
}

fn main() {
    // Make a box and then destroy it.
    let contents: String = "Here is some data.".into();
    let package = OpenPackage::pack(contents);
    // The package owns its contents.
    // println!("{}", contents); is invalid.
    println!("{}", package.unpack());

    // Now, make a box, close it, and address it.
    let contents: String = "Here is some MORE data.".into();
    let package = OpenPackage::pack(contents);
    let closed_package = package.close();
    // We now can't unpack the package to get to its contents. This is an error:
    // let contents = closed_package.unpack();
    // because ClosedPackage doesn't have .unpack()
    // Also, package is no longer valid, so no duplication can occur.
    // Finally, we can't send_package() this package; we have to address it.
    let addressed_package = closed_package.address("6902 East Pass, Madison, WI".into());
    // Now we can send the package.
    println!("{:?}", send_package(addressed_package));
}

This ends up printing out:

$ ./session_types
    Put some data in a package.
    Package unpacked.
Here is some data.
    Put some data in a package.
    Package closed.
    Addressed a closed package.
    Package received.
    Package opened.
    Package unpacked.
Destination received: Here is some MORE data.
Ok("Sent package to 6902 East Pass, Madison, WI")

In the real world, the Rust crate hyper makes heavy use of session types to ensure the integrity of HTTP requests and responses.

socketserver: the Python networking module you didn't know you needed

Sat, 24 Dec 2016 05:04:24 +0000

I occasionally spend time randomly surfing the Python standard library docs; there is a lot of useful functionality included in the language’s standard distribution, such as, for instance, the socketserver module, which I didn’t know about until this evening and which is one of the most useful I’ve seen in a while. As ever, the docs are straightforward in their self-description:

The `socketserver` module simplifies the task of writing network servers.

This is something of an understatement. To demonstrate this, here is a simple CaaS (capitalization as a service) server written with socketserver and one with socket:

import socket

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.bind(('', 50006))
    s.listen(1)
    conn, addr = s.accept()
    with conn:
        while True:
            data = conn.recv(1024)
            if not data: break
            conn.sendall(data.upper())

And here is the same functionality with socketserver:

import socketserver

class CaaSHandler(socketserver.StreamRequestHandler):
    def handle(self):
        data = self.rfile.readline()
        self.wfile.write(data.upper())

if __name__ == "__main__":
    server = socketserver.TCPServer(('', 50007), CaaSHandler)
    server.serve_forever()

Both of these take connections synchronously and sequentially, capitalize the data they recieve, and return it. The main difference is that the socketserver version can accept as much data as there is memory, while the socket version can accept only a limited amount (1024 bytes in this example).

This is because socketserver’s StreamRequestHandler provides the file-like objects rfile and wfile which expose all the normal luxuries of Python 3 files, like readline and read. The parent class of the handler you write will deal with setting the buffer size, looping until a newline or EOF is encountered, and dealing with client-first and server-first protocols. We could just as easily add a welcome message/prompt to the program; just make the CaaSHandler class look like this:

class CaaSHandler(socketserver.StreamRequestHandler):
    def handle(self):
        self.wfile.write(b"Enter some data to be capitalized:\n")
        data = self.rfile.readline()
        self.wfile.write(data.upper())

without any changes to the client’s behavior. Adding that functionality in the socket version is somewhat nontrivial; how, for instance, one would handle both clients that expect to send data first and clients that expect to receive it first is less than obvious.

The second useful facility that socketserver provides is the xxxServer classes. I use TCPServer here, to which I passed a tuple of (hostname, portnumber) and the name of my handler class, CaaSHandler. I could also have used UDPServer for datagrams or UnixStreamServer/UnixDatagramServer for Unix sockets.

The socketserver module also provides mixins for threading and forking servers, which makes writing asynchronous network services much less painful than using socket and threading or even asyncio.

Rewriting tinyhttpd in Rust, Part One

Fri, 21 Oct 2016 14:07:09 +0000

In 1999, J. David Blackstone, or, as he is known online, jdavidb, was taking CSE 4344 (Network Concepts) at UT Arlington. Those were the glory days of Sparc Solaris, and Blackstone wrote, for his college course, a C program called tinyhttpd. It is, essentially, a very short version of the immensely complex programs that seem run the world these days: web servers. Unlike the million-line behemoths (think Apache, nginx, et cetera), tinyhttpd is a HTTP 1.1 web server in 532 lines of well-commented C.

HTTP 1.1 is a ubuqitously supported protocol that is useful for a great many applications, and in this modern era of embedded (a.k.a “Internet of Things”) computing applications, small web servers have never been more important.

This program is also a small, manageable example of a legacy application - an old program written for an obsolete operating system that still gets the job done, but exposes any organization using it to not only the cost of maintaining ancient operating systems and hardware, but also to the risk of the security vulnerabilities present in tinyhttpd itself and the software it needs to run.

For the purposes of these posts, I’ll be looking at tinyhttpd from the perspective of a company that uses it internally, and wants to transition to a more modular, portable, and maintainable design, rather than one which either ships it as a product or buys it as a product from another company and wants to replace it; these situations are similar, but have additional challenges.

The first thing to do is to analyze the existing source. I’ve gone ahead and created a GitHub repository to host both the old and new source code, and I’ll link to specific commits in these posts. For instance, here is the commit with nothing but the unmodified source of the legacy app.

The first thing to do is to build the existing app. In order not to clutter the repository with object files, I created a .gitignore file from GitHub’s default C gitignores. Now all I have to do is run make, right?

11:32:58: nora [~/Projects/rtinyhttpd/legacy]
$ make
gcc -W -Wall -lpthread -o httpd httpd.c
/tmp/ccbqEOVd.o: In function `main':
httpd.c:(.text+0x1a85): undefined reference to `pthread_create'
collect2: error: ld returned 1 exit status
Makefile:4: recipe for target 'httpd' failed
make: *** [httpd] Error 1

What’s this, it doesn’t compile? Well, you’ll remember I mentioned it was written for an ancient version of Sparc Solaris - that’s the whole reason we’re rewriting it. Luckily, the original author anticipated this. Looking at legacy/httpd.c (where the error is), I see this comment at the top:

/* This program compiles for Sparc Solaris 2.6.
 * To compile for Linux:
 * 1) Comment out the #include <pthread.h> line.
 * 2) Comment out the line that defines the variable newthread.
 * 3) Comment out the two lines that run pthread_create().
 * 4) Uncomment the line that runs accept_request().
 * 5) Remove -lsocket from the Makefile.
 */

I made a note of this in my analysis folder and made those changes - except that they didn’t apply. The makefile didn’t have -lsocket, and there was only one occurrence of pthread_create. They did make the app build, but it didn’t work!

In order to figure out what’s happening, I looked up pthread_create on man7.org. It’s part of the POSIX threading API, and it is definitely available on Linux. Furthermore, if we look at the main() function, we can see why commenting out those lines caused a problem - it’s an infinite loop that does nothing but accept connections!

while (1)
{
    client_sock = accept(server_sock, (struct sockaddr *)&client_name, &client_name_len);
    if (client_sock == -1)
    {
        error_die("accept");
    }

    // Commented out in order to build on Linux
    /* if (pthread_create(&newthread , NULL, (void *)accept_request, (void *)&client_sock) != 0)
    {
        perror("pthread_create");
    }
    */
}

So, we need to get POSIX threads working to make this app run properly. (Note that this problem isn’t an uncommon one when looking at legacy apps; there is often not a good set of build instructions.)

In our case, luckily, this is easy: just revert the commenting and change -lpthread in the Makefile to -pthread, as mentioned on the manual page.

Doing this allows the app to build and run correctly, binding to port 9999. When I open localhost:9999 in my web browser, I get a page back. Success!

Now that we have a compiling and running version of the legacy tinyhttpd, it’s time to go through the source code. Luckily for us, tinyhttpd is entirely contained in a single file. Let’s start off with the top:

/* J. David's webserver */
/* This is a simple webserver.
 * Created November 1999 by J. David Blackstone.
 * CSE 4344 (Network concepts), Prof. Zeigler
 * University of Texas at Arlington
 */
/* This program compiles for Sparc Solaris 2.6.
 * To compile for Linux:
 * 1) Comment out the #include <pthread.h> line.
 * 2) Comment out the line that defines the variable newthread.
 * 3) Comment out the two lines that run pthread_create().
 * 4) Uncomment the line that runs accept_request().
 * 5) Remove -lsocket from the Makefile.
 */

Here is some information which will often be included in legacy programs - some short information about the author and purpose of the program, and some (in this case out of date and inaccurate) information about building and running the program. Removing the misleading lines makes this section a lot more concise and is probably a good idea.

Skipping the #includes, which aren’t very helpful in this case, we find two #define statements:

#define ISspace(x) isspace((int)(x))

#define SERVER_STRING "Server: jdbhttpd/0.1.0\r\n"

The SERVER_STRING definition is pretty straightforward; it’s an identifier of the software, which will be sent to clients. In our version, I would prefer to not include the \r\n terminator in the definition itself. As to the ISspace definition, though, I’m not immediately sure. A quick search of the source shows no definition of a function isspace taking an integer, so it’s probably coming from one of the includes.

If this program had multiple files, I’d search through them next; but, as there are none, I’m going straight to the Internet. Turns out, it does just what you’d expect - it checks if a given integer represents whitespace or not. This definition simply allows calling it directly on char values without writing out an explicit cast every time. I’ve made a note of this in my analysis documents.

After the head macros, we can see explicit definitions of all the functions used in the program.

void accept_request(void *);
void bad_request(int);
void cat(int, FILE *);
void cannot_execute(int);
void error_die(const char *);
void execute_cgi(int, const char *, const char *, const char *);
int get_line(int, char *, int);
void headers(int, const char *);
void not_found(int);
void serve_file(int, const char *);
int startup(u_short *);
void unimplemented(int);

Because they have no comments, these definitions are not particularly useful, so let’s go down to the bottom of the page and look at which functions are called in the program’s entry function, main().

int main(void) {
    int server_sock = -1;
    int client_sock = -1;
    u_short port = 9999;
    struct sockaddr_in client_name;
    socklen_t client_name_len = sizeof(client_name);
    pthread_t newthread;

    signal(SIGPIPE, SIG_IGN);

    server_sock = startup(&port);
    printf("httpd running on port %d\n", port);

    while (1)
    {
        client_sock = accept(server_sock, (struct sockaddr *)&client_name, &client_name_len);
        if (client_sock == -1)
        {
            error_die("accept");
        }

        if (pthread_create(&newthread , NULL, (void *)accept_request, (void *)&client_sock) != 0)
        {
            perror("pthread_create");
        }

    }

    close(server_sock);
    return(0);
}

Let’s break this down further. This function takes void, meaning that the program has no arguments or command line options. This probably means it’s not very customizable, something I’d like to change in the rewritten version.

After the function signature come the definitions of some local variables: server_sock and client_sock, port, client_name, client_name_len, and newthread. server_sock and client_sock are just ints, but they represent file handles, as we’ll see in a moment. port is clearly a port number. client_name is the address of the client, and client_name_len is its length.

Below that, the program uses signal() to ignore SIGPIPE, the signal that programs receive when they write to a file handle which has been closed. It seems to me that this should be handled more appropriately in the rewrite.

Immediately afterward, the server_sock variable is filled by the result of the function startup, which is given a pointer to the port number. This seems odd to me - why does it need a reference and not just the value? - so I look at that function’s definition. It is commented with:

/**********************************************************************/
/* This function starts the process of listening for web connections
 * on a specified port. If the port is 0, then dynamically allocate a
 * port and modify the original port variable to reflect the actual
 * port.
 * Parameters: pointer to variable containing the port to connect on
 * Returns: the socket */
/**********************************************************************/

That makes more sense now - it allows dynamically generating a port number. That’s useful, but the functionality isn’t exposed through the command line interface, which is annoying. In our program, I’d like to expose that, and I’d also like to move away from the C convention of modifying inputs. In the rewrite, I think I’ll return a tuple. Since this is a fairly complex idea, I’ll take this time to write some notes down.

That’s enough to understand a bit more about main. After a simple status message, the program moves on to the main loop:

while (1)
{
    client_sock = accept(server_sock, (struct sockaddr *)&client_name, &client_name_len);
    if (client_sock == -1)
    {
        error_die("accept");
    }

    if (pthread_create(&newthread , NULL, (void *)accept_request, (void *)&client_sock) != 0)
    {
        perror("pthread_create");
    }
}

This is an infinite loop which accepts a connection, as can be seen if we look up accept(), which is where client_sock gets its value. It returns a file handle representing the socket. It returns -1 if it fails for some reason, and the next few lines check for that eventuality. This is another suboptimal design imposed by C’s lack of algebraic data types - in Rust, this idea can be represented with an Option or a Result.

The next few lines try (and handle errors for) spawning a new thread that runs accept_request. Looking at the comments here is not quite as illuminating as one might hope:

/**********************************************************************/
/* A request has caused a call to accept() on the server port to
 * return. Process the request appropriately.
 * Parameters: the socket connected to the client */
/**********************************************************************/

I’m not really sure what processing the request “appropriately” entails. For now, though, it’s enough to know that this is the main function for dealing with incoming requests.

The only code after this is cleanup code we won’t need in the rewrite, so we have enough info to write a short pseudocode summary of the server:

Open and configure a server socket

Until the process is terminated:
     Wait for a client to request a connection
     Try to open a connection with that client
     Open a new thread to deal with that connection's request

That’s a lot simpler than one might have imagined from the length of this post, and it doesn’t tell us much about the actual functionality of the server, but it gives you a good idea of the process one often has to go through to understand legacy code.

Now that we have examined the basic structure of the server’s execution, I’m going to dive into the actual functionality and logic of the server, which is encapsulated primarily in the function accept_request, whose signature is void accept_request(void *arg). This is a signature that is totally unrevealing, and which in Rust would require an unsafe block; this function takes a raw pointer with no type information at all. We’ll have to do quite a bit of work to understand what the function actually does.

First of all, are there any clues about what the argument might represent? Well, we can look back at how the function is called:

pthread_create(&newthread , NULL, (void *)accept_request, (void *)&client_sock)

This is a little complicated, but essentially a new thread is being spawned which will execute accept_request(&client_sock). This is the only place this function is called, so the argument is presumably expected to be only a pointer to an integer file descriptor to a socket - but the compiler knows none of that! That’s a lot of unchecked assumptions and unsafe memory access. Rust, and more importantly the Rust standard library, has better invariant checking, which will make the re-implementation a great deal safer and thus easier to extend.

Moving on to the body of the function, we see the creation of a lot of local variables which I’ll go into as they’re used. It is important to note, though, that there is a group of buffers created with absolute lengths. These appear, at first glance, to be possible introduction points for overflow vulnerabilities - something that is mitigated by the Rust idiom of defaulting to using Vecs instead of arrays.

One of these buffers, of length 1024, is populated using the function get_line, which, according to the comments above its definition, reads a line into a buffer and null-terminates it, with length checking, and returns the number of bytes stored. That buffer is printed and dissected over the course of the next 90 lines or so.

Now that it’s clear how I dissect each line of code, I’m going to move a bit faster, translating the entire program into pseudocode function by function. What we currently have is this:

Open and configure a server socket

Until the process is terminated:
 Wait for a client to request a connection
 Try to open a connection with that client
 Open a new thread to deal with that connection's request

And we’re examining the idea of “dealing with the client”. This is all done in the accept_connection function, whose pseudocode looks a bit like this:

accept_request takes a socket connecting to the client
     read a line from the client
     log (to stdout) the received request
     
     copy what is assumed to be the method into another buffer
     if the method isn't GET or POST:
          handle an unimplemented method somehow
     if the method is POST:
          make a note that this request will require executing a CGI script

     copy what is assumed to be the URL into another buffer
     if the method is GET and the url has a ? in it:
          make a note that this request will require executing a CGI script
     construct the path to the requested resource by prepending "htdocs" to the url
          (note - I'd like to make this customizable in the rewrite)

     if the URL is /:
          add 'index.html' to the path

     if the resource being requested doesn't exist:
          handle a not found error somehow
     if the resource is a directory:
          append "/index.html" to the file path
          (note- the existence of THIS file isn't checked!)
     if the file is executable:
          make a note that this request will require executing a CGI script
     
     if this request requires executing a CGI script:
          handle executing a CGI script
     otherwise:
          handle serving a static file

This analysis is pretty revealing: essentially all this function does is determine some properties of a request and then pass it off to be handled appropriately by other functions.

This particular function should be fairly easy to translate into more efficient Rust code, especially if we look at using Rust’s more advanced type system. In particular, rather than having a large number of buffers, I’d like to use slices and ADTs. For example, I might create an enum HTTPMethod:

enum HTTPMethod {
     Get,
     Post,
     Other
}

Then I could use a match expression to appropriately dispatch the request, whether to the static server, CGI handler, or error response.

In the next post, I’ll take a look at the handler functions and how they handle the various conditions and actions a request can trigger - unimplemented method, resource not found, static file serving, and CGI execution. I’ll also discuss the Rust idioms that can be used to better model the intended behavior and internal structure of this server.

Am I in a Terminal?

Tue, 13 Sep 2016 01:55:38 +0000

Sometimes, it can be useful to know if your program is running in a terminal. Since Python 3.3, this functionality has been available in the os module:

#!/usr/bin/env python3

# Test if this Python script is running in a terminal or not.

import os

try:
    size = os.get_terminal_size()
    print("I am in a terminal of size {}x{}"
        .format(size[0], size[1]))
except OSError:
    print("I am not in a terminal.")

Here is an example of it in operation:

$ ./am_i_term.py 
I am in a terminal of size 80x24

$ ./am_i_term.py | tee
I am not in a terminal.

This is useful for many reasons. For example, scripts which have interactive “beautifications” like progress bars, no-freeze spinners, and animations should cease these antics when piped into the input of other programs or redirected to files. Additionally, programs being run from scripts can disable all performance-impacting interactivity, including interactive KeyboardInterrupt handling; if a user Ctrl+Cs a script, they want it to stop, immediately, not ask to quit.

Learning Japanese the Python Way

Mon, 29 Aug 2016 22:06:19 +0000

Now that I’m in college, I’m taking a lot of non-computer science classes, and one of them is Japanese. I’m just starting out, and I need to be able to rapidly read numbers in Japanese and think about them without translating them consciously. I could make a bunch of flash cards, or use a service like Quizlet… or I could write some Python!

For those of you who are unfamiliar, Japanese doesn’t have the ridiculous numerical system that English does. One through ten are defined, and eleven is simply (ten)(one). Twenty three, for example, is (two)(ten)(three) (にじゅうさん). This means that rather than having a long list of numbers and special cases, I can just have the numbers zero to ten “hard coded”.

After that, the program is pretty simple: if the number is less than 11, simply look it up. If it’s more than 11 but less than 20, build it with じゅう plus the second digit. If it’s larger than 20, build it with the first digit plus じゅう plus the second digit.

The interactive part is pretty simple too: it runs a loop that randomly generates numbers, checking that they haven’t been done before, translates them, and asks me to translate them back. If I succeed, it moves on; if not, it doesn’t record the number as having been completed, so I have to do it again at some point in the same run.

This simple program came out to 136 lines of very verbose and error-checked Python. It’s a good piece of code for a beginner to try and modify - for example, can you get it to incorporate the alternate form of four (し) as well as the primary form? Can you make one that teaches Kanji numbers? (I plan to do both of those things at some point.)

#!/usr/bin/env python3

"""
This is a program to help you study the Japanese numbers.
It currently goes from 0 to 99; I will extend it at a later date.

It can be executed as follows:

    ./numerica.py

which will do all the available numbers, or

    ./numberica.py 10

which will go only up to 10.
"""

numbers = ['ZERO',
           'いち',
           'に',
           'さん',
           'よん',
           'ご',
           'ろく',
           'なな',
           'はち',
           'きゅう',
           'じゅう',
           ]


class OutOfRangeException(Exception):
    pass


def small_to_japanese(n):
    "Convert a number (0-10) to Japanese."
    if n > 10 or n < 0:
        raise OutOfRangeException

    return numbers[n]


def medium_to_japanese(n):
    "Convert a number from 11 - 100 to Japanese"
    if n > 100 or n < 11:
        raise OutOfRangeException

    digits = list(map(
        int, str(n)
    ))

    out = ""

    # Omit いち in numbers > 10
    if digits[0] > 1:
        out += numbers[digits[0]] + " "
    out += numbers[10] + " "
    out += numbers[digits[1]]
    return out


def number_to_japanese(n):
    try:
        return small_to_japanese(n)
    except OutOfRangeException:
        pass

    try:
        return medium_to_japanese(n)
    except OutOfRangeException:
        pass

    print("No way to represent numbers of that magnitude!")


if __name__ == "__main__":
    from random import randint
    from sys import argv

    # Check if there is a command line option for max numbers
    if len(argv) >= 2:
        try:
            MAX_NUM = int(argv[1])
        except ValueError:
            MAX_NUM = -1

        # A little edge case handling
        if MAX_NUM > 99:
            print("Impossible - this program doesn't "
                  "work with numbers over 99.")
            exit(1)

    else:
        # If a max wasn't given, default to 99
        MAX_NUM = 99

    given = ""
    done_so_far = []
    number_done = 0

    while True:
        n = randint(0, MAX_NUM)
        # If and as long as n has already been done, get a new number.
        while n in done_so_far:
            n = randint(0, MAX_NUM)

        try:
            given = input("What is {} in Roman numbers? ".format(
                number_to_japanese(n)))
        except KeyboardInterrupt:
            print("Bye!")
            exit(1)
        except EOFError:
            print("Bye!")
            exit(1)

        if given.lower() == 'quit':
            print("Bye!")
            exit(0)
        if number_done >= MAX_NUM:
            print("You did all the numbers in that set!")
            exit(0)

        try:
            given_n = int(given)
        except ValueError:
            given_n = -1

        if given_n == n:
            print("You got it!")
            done_so_far.append(n)
            number_done += 1
        else:
            print("No, that's incorrect. This is {}.".format(n))

Why Linux on the PC Needs a Focus on Hardware Support

Tue, 02 Aug 2016 16:28:47 +0000

Linux cannot fall behind in hardware support. In fact, if we ever want to capture the desktop market, we have to race ahead.

A few days ago, I had an interesting and somewhat frustrating experience with a friend of mine. Their laptop was dying, so they asked me to give them some suggestions for a new one.

Their requirements were a computer with a display that was good for reading, enough power to be responsive and able to multitask well, and rapidly accessible storage, but not necessarily a lot of it. Of course, I immediately thought of the System76 Lemur, which happened to be on sale at the time; however, after going through a whole list of pros and cons of Ubuntu with the friend, they told me that they wanted to go with Windows or Mac OS X as they “didn’t have time to tinker”.

They’re right, of course, but this really got under my skin. The thing is, if you’re not trying to game on Linux, there are absolutely no problems using it on a desktop in almost all cases. Desktop hardware is nicely standardized and “just works”. But on a laptop? Nope. It’s a crapshoot as to whether your wireless Internet and Bluetooth will work, or if your touchpad’s multi-touch will be usable. On my Dell Inspiron 7000-series laptop, which works almost flawlessly with Ubuntu GNOME, the wireless chipset will occasionally forget that any but a single network, to which I’m not connected, exists.

Why is this? Well, it’s because laptops are often very, very custom. They have custom form factor motherboards with non-standard sets of features. Battery life is often the primary concern rather than compliance, and release cycles are very tight, so if a new hardware system is developed, drivers are produced for the target platform (almost always Windows) and released there. The Linux community has to hack together our own after the fact.

Windows runs on most laptops, and has a lot of big issues. Privacy concerns, resource overutilization, extremely poor real-time performance, and a massive lack of customization are the obvious ones, along with a downright byzantine user interface without much power to back it up (in the consumer versions, that is). Mac OS X looks simple on the surface while still exposing the massive power of a UNIX to its power users and developers (although this is becoming progressively less true). On the other hand, it costs an absolute fortune to buy into that ecosystem, and that is where Linux comes in.

In reality, Linux is a modern UNIX like Mac OS X, and it is far more flexible and powerful, but to many people, it’s just “Windows that costs less”. What we need to be is a Mac OS X that can run anywhere. Linux needs to be simple on the surface, which most DEs accomplish brilliantly, while exposing the power of the underlying OS, which isn’t hard given a terminal emulator. Where Linux falls short is the “runs anywhere” part.

An update, four years on. Things are better now; a lot better. I’ve managed to stay almost entirely on Linux as my desktop OS by relying on the excellent support for ThinkPad hardware. The mainline kernel supports a lot of hardware out of the box. On the other hand, as more custom laptops come out with highly integrated hardware, there is a class of very cheap laptops that would be nice to use as Linux machines that have hardware that isn’t even really well supported on Windows, and is totally unusable on Linux.

Porting Deucalion to Rust

Mon, 11 Jul 2016 15:33:38 +0000

A few months ago, I made a proof-of-concept for an RPG engine based on SFML and the Tiled map editor, called Deucalion. Over time, the source code became unwieldy, leaked a great deal of memory, and was nearly impossible to build. I ended up spending more time configuring build systems than actually working on the code, and I abandoned it in favor of SBrain and schoolwork.

Recently, though, the Rust game development story has gotten a lot better, and I’ve gotten a bit of free time. With the help of a friend of mine, Dan Janes, I’ve been porting the existing code to Rust and refining the design for the game-dev-facing API. It’s been interesting, since it’s my first time running a project on which I am not the sole contributor.

I’ve certainly run into some problems because of the relative immaturity of the Rust ecosystem - for example, many projects don’t use the standard Error trait, which makes using the handy macros that rely on it such as try! nearly impossible, but I’ve also found that as a whole, the community is very responsive to having these issues pointed out and solved.

Deucalion isn’t quite at the level it was before I decided to port it - I’m still struggling to get tilemaps to draw with decent performance, and a lot of design work needs to be done - but it’s doing better than I thought it would, and I’ve discovered some of the best features of Rust so far.

For example, while Rust doesn’t have exceptions (because exception handling requires a heavyweight runtime), the convention of returning Result<T, Error> from functions that might fail allows programs to act as if it did. Deucalion implements a single, shared error struct DeucalionError that encapsulates every possible error type (Currently IoError, LuaError, TiledError, NotImplementedError, and Other), allowing callers of risky functions to act according to the actual failure that occurred.

I also like the module system much more than I thought I would at first. While learning when and where to use mod vs use can be a bit of a hassle, the fact that multiple includes create an automatic compiler error is very welcome when compared with C++.

Rust is a great language, and its ecosystem is on its way to becoming as good as that of Python or Ruby. I’m excited for every step along the way.

I Repaired My Headphones

Sat, 04 Jun 2016 14:53:55 +0000

Hardware manufacturers are missing out on a huge potential source of revenue: the thrifty tech user market.

I just woke up to find that my favorite pair of headphones was making only one sound, and it wasn’t the one I was putting in as an electrical signal. It was rattling.

Feeling adventurous, I popped the around-the-ear pads off and - lo and behold! - found four Torx screws. I removed them, found that the drivers had separated from their sockets, and put them back in. I also glued them in with a bit of hot snot, since the drivers happened to be a little deeper than their sockets, giving me a nice lip around which I could run the hot glue gun.

These headphones cost about $25. They’re not expensive and they’re not fancy, but they’re repairable. I bought them for that, in a sort of shallow sense: they’re advertised as having a 3.5mm accepting jack on the left ear, which means that the cable can be replaced. I abuse my headphones constantly, and I’ve already replaced the 3.5mm cable twice.

My only question is, why didn’t they advertise to me that the insides were trivial to repair as well? Every other pair of around-ear headphones I’ve owned has been ultrasonically welded, not affixed with screws; the drivers have been glued to the frame, not inserted into a socket that’s integral to the frame; and the around-the-ear cups have been either glued on or attached with a huge lip that’s nearly impossible to remove.

In Idoru, Gibson predicted that, by around now, we would have reverted to a model of non-disposable, easily repairable electronics. I’d pay so much more for headphones that were advertised not just with shots of the outside but with schematics and examples of how easy repairs are. I’d be willing to wait longer for my shipment if I didn’t think I’d need to buy a new pair of headphones next month.

I want headphones made by the Sandbenders out of coral and turquoise and the interior surface of renewable nuts, and I’d be willing to pay a lot for them, not for aesthetic reasons but because I know I won’t have to replace them for a long, long time.

Open Source for Normal People

Mon, 11 Apr 2016 05:54:17 +0000

This is not an article for technical people. It is an article normal people who just use their computers to get things done: to look at Facebook and Twitter, check their work email, write novels, design houses… whatever.

Maybe you’ve heard about open source by using a open source program like Firefox, OpenOffice, or Inkscape. Maybe a friend suggested you look into it. Whatever the reason, you’re here, so here you go: a simple, non-technical explanation of what it is and why it’s important.

TL;DR (Too long, didn’t read)

For the quick overview: Most software is secret; open source software isn’t. As counterintuitive as it may seem, it’s not necessary to keep software secret in order to make money, and when software is not secret, competition is fiercer and innovation often progresses at a faster pace. Keeping software transparent also prevents the authors of that software from using anti-consumer and anti-competitive tactics like proprietary format lock-in.

Background

Some background first (really, just a little bit, I promise!)

Computers really just do math. That’s it; they don’t know what words are, or what a web page is. Software turns things humans are interested in, like words or web pages or building plans, into numbers that computers know how to work with. Software is made of code. First, there’s the source code, which is a kind of code that humans can (sort of) read. Programmers make source code that describes what a computer should do to numbers to get the result that you, the user, want. It’s a little something like a recipe.

That recipe gets “baked” into machine code, which very, very few humans understand. It’s only for computers. That’s what you download from the App Store or get on a CD when you buy a piece of software like Angry Birds or Microsoft Word.

Closed Source

Most software, like Microsoft Word or Angry Birds, is closed source or proprietary. This means that all you, the customer, get is the machine code. You can’t read it, but your computer can, so you can click on the little W icon and your computer will open up a window with a blank page, ready to be filled with the next great American novel. The machine code tells the computer what each keypress means, how to save and load files, and when to draw that little red squiggly to inform you that you spelled “antidiesstablishmentarianism” wrong.

That seems all fine and dandy, right? Well, it is, except for two things.

First, the people who made the software are the only ones who know how it works.

This is problematic for a lot of reasons. There’s the “bus factor”: what happens if the developer is hit by a bus, or, more realistically, the company shuts down or decides to stop supporting that product? No more updates for you, and there’s nothing anyone can do about it, because nobody has the recipe, only the pre-baked machine code. It also means that, if the people who made the software don’t want you to use it with their competitors’ software, they can stop you (or, at least, try).

This happens a lot in the real world, and it’s called lock-in. If your whole company uses Solidworks or Adobe Photoshop, for example, it’s going to be a real pain for you to switch to an alternative, even if some other company came out with a better, cheaper CAD package or photo editor. Once you’ve started using a particular piece of proprietary software, there’s no guarantee that you can switch to a better or cheaper program without a lot of headache. This is nothing, though, compared to the other problem.

Second, the people who made the software are the only ones who are really sure about what it does.

In the same way that the cook is the only one who really knows what’s in the cake you’re eating, the developers are the only ones who really know what their program does. This is really, really bad when the closed source software is something that’s essential to a lot of people’s everyday lives and businesses, like Microsoft Windows, which, starting with version 10, forces users to install updates, sends information about them back to Microsoft, and even removes programs they’ve installed without asking. Imagine if that happened to a program you used for work the day before a major deadline!

The creators of closed source software can also artificially limit what the software is able to do. This is the model of the EagleCAD printed circuit board design program, which sells licenses ranging from $0 to $50,000. The software you get with each license is exactly the same, but the more you pay, the more features get unlocked, so most users end up with a powerful CAD package they can never use to its fullest extent.

Open Source to the Rescue

These problems are why many users and even businesses are embracing open source software. Open source software is software whose source code, or recipe, is available for everyone to see. This means that anyone who can understand source code (a lot more people than can understand machine code) can make sure that the software does only what it’s supposed to.

It also means that anyone can change that code. Not like a wiki, mind you ! Changes have to be reviewed by the product’s maintainers before they are accepted into the mainstream distribution of the product, so if you decide to download an open source product, you can be sure that you’re getting the best version that’s available. Just like deciding to throw in that pinch of cinnamon, even if the original baker hated spices, you can change your own copy whenever you like, and (usually) redistribute that copy to others. If you have a problem with the way your software works, you can just change it or, if you can’t program, you can hire any developer in the world to do it for you. This may not seem like a big deal, but it has far-reaching consequences.

Imagine trying to get Microsoft to change something in Windows or Word for you, or trying to convince Adobe to add your favorite feature to Photoshop. They’d never do it because they can’t justify the cost, and even if they did agree to your change, it might take years before it was incorporated into a consumer version. On the other hand, with an open source product, you can download the software and use it just like with a closed source product, but if you need something to be changed and can’t wait for the people who made the software to do it, you can get it done right away to your exact specifications. Open source gives you that freedom. If you don’t like the way something works, you can often change it yourself, and if you can’t, any developer in the world can.

The Open Source Business

While on the surface it might seem impossible to make money on open source software, many companies do so. Red Hat Inc. sells an enterprise operating system based on the open source Linux kernel. While their product is almost entirely open source, and is freely available to anyone, many industrial facilities, financial institutions, and governments pay huge sums for technical support. MongoDB Inc. publishes an eponymous open source database solution under a similar business model.

Other companies, especially small ones with between one and ten developers, work partially on what are called open source bounties. The software is free, but customers can make a contract with the developers which essentially says, “for some amount of money, the developers will create some feature by some specific date”. This allows developers to make money, allows the customer to be sure that they will get the features they need when they are needed, and allows the entire community to benefit from these new features once they’ve been created.

Even electronic hardware companies are embracing open source. While Linux, KiCAD, and other open source software has been popular in electronics for a long time, businesses like SparkFun, Adafruit Industries, and many others are creating open source hardware. Electronics that are open source are sold by the companies that create them just like normal, but all the schematics, diagrams, printed circuit board designs, and bills of materials are distributed for free to anyone who wants them.

Open Source and Competition

All of these practices encourage competition and make anticompetitive tactics like lock-in via proprietary file formats impossible. What is more, the moment an open source product is improved (whether by the community or the product’s owners), that innovation is available to everyone. Other developers can immediately begin to improve it and build upon it, and users can immediately begin to integrate it into their workflows and business processes.

Try It!

Open source is not the answer to every problem, but it is the answer to many of them. Open source products exist for nearly everything.

Office applications (like word processing, presentations, and spreadsheets);
web browsing;
computer aided design in architecture, mechanical engineering, and electrical engineering;
digital photo processing;
digital art;
professional and amateur video editing;
accounting and financial auditing;
tax processing;
professional and amateur audio processing;
web servers;
blogging software (in fact, this site is powered by the open source Jekyll software);
almost anything you can think of

It costs you nothing but a download and a few clicks to try a piece of open source software, so go ahead. If you have a problem, most products have a method by which you can bring that problem to the developers’ attention (called filing a bug against the product), so they can fix it.

If you’re curious about open source as a concept, and how it can help you or your business, please contact me at nora@nora.codes. I’m more than happy to answer any questions you might have.

The Sinclair ZX-81/TS-1000

Sun, 10 Apr 2016 15:42:56 +0000

I recently watched the excellent film Micro Men, about Sir Clive Sinclair, Sinclair Research, Acorn Computers, and the fight for the British low-end computer home computer market in the early 1980s. This, along with a healthy (unhealthy?) dose of procrastination, lead me to spend a few hours browsing eBay, looking for old PCs. “Timex Sinclair 1000 Retro Home Computer, New In Box”. I checked the price - $29.99 plus shipping. It has been one of the most interesting purchases of my life.

The Timex Sinclair 1000 was originally the ZX81, designed by Rick Dickenson for Sir Clive Sinclair’s Sinclair Research, and built by Timex (yes, the watch company) in Scotland. It sold extremely well in Britain. In America, however, the Sinclair brand was mostly unknown, so the computer was released there as the Timex Sinclair 1000. It also recieved some modifications and upgrades, most notably a whole extra kilobyte of RAM. (I make fun of it now, but at the time it was a big deal.) Over its three-year run, the ZX81 sold 1.5 million units under that name and an unknown number in America as the TS1000. In addition, several unauthorized clones were created in the Soviet Union in 1982 and sold well there until at least 1986.

Like most computers at the time, the ZX81 used the Zilog Z80A processor. It was clocked at 3.85 MHz, and that was approaching the practical limits of the Z80, which was not well binned and not very good at disdissipating heat. Even so, industrious Z80 users have managed to increase the clock to as much as 7.3 MHz. The ZX81 came with 1 KB of RAM, or 2 KB in the TS1000. This was a lot for £99, but it wasn’t enough to have, say, a real parser for the included Sinclair BASIC - tokens like ‘PRINT’ were entered whole, as Shift + P, for example. The Shift key could be used in this way because the ZX81 actually didn’t support lower-case characters.

Because even 2 KB of RAM isn’t really sufficient for anything more than simple BASIC programs, official and unofficial RAM packs became extremely popular. The ZX81 was, however, notorious for connection problems with this hardware, which plugged directly into the mainboard with a very flimsy connector. In addition, the requirement for a tape drive to save and load programs has prompted more modern ZX81 enthusiasts to add EPROM, EEPROM, and flash chips to the system in order to store programs.

If the Z80A processor was the brain of the ZX81, the Ferranti Uncommitted Logic Array was the heart. It did everything the all-digital Z80A couldn’t: video and audio synthesis, clock stabilization, and keyboard control, among other tasks. Because most households in the 1980s didn’t have a TV with composite video input, the ZX81’s ULA fed a composite signal, PAL for the UK and Europe, and NTSC for America, into the RF modulator. This modulator (pictured below with my bypass buffer modification) made the video appear to be a broadcast TV signal, allowing home TVs to display it with minimal user effort. The TS1000 included an RF switch that allowed connecting the real TV antenna and the computer so that they could be switched between at any time.

When I received my Timex Sinclair 1000, I was amazed at the small size. It’s really not larger than a typical hardback YA novel (you know the type), and is pretty well put together for something built down to a price. I would have loved to use it unopened, but unfortunately there’s no way to do so these days, so I figured I may as well take some photos. You can find them with annotations here. The main modification I’ve done so far is to bypass the RF modulator. This let me use the computer with my projector, which is the only composite display I owned at the time I bought the computer (I have since bought a $15 car backup camera monitor, which works well with the same modification). I used a single NPN transistor with a 220 Ohm resistor to create a collector follower buffer to avoid stressing the ULA, then added a set of diodes in series to drop the voltage to a level that my projector can tolerate. You can see more in-depth instructions here. I plan to add a reset button, an on/off switch, a power LED, and possibly a stepper switch and a bus visualizer. This would make it much less authentic but also much more interesting to watch in operation, since the entire computer could be stopped and clocked at a human-viewable rate, with the bus visualizer allowing me to view the status of the various interfaces to the CPU. I might also add a fake tape drive or an EEPROM to allow saving and loading programs.

IPFS, the Interplanetary File System

Sun, 10 Apr 2016 15:37:11 +0000

IPFS, the InterPlanetary File System, is a content-addressable network. This means that rather than asking the network for a particular site or domain name (like google.com), you ask for a particular piece of content, and you’re guaranteed to recieve it. There’s no possibility of a “monkey in the middle” attack, in which someone maliciously modifies the web page you’re trying to access.

To explain it another way, on the normal Web, when you access google.com, the network translates it to an IP address, like 216.58.216.14 or 2607:f8b0:4003:c00::6a. Then, your computer connects to the server that address refers to and asks it, “Could you send me the content for google.com, please?”. This means that Google can change the content on their front page whenever they like - or, if someone malicious is inbetween you and Google, they could change the content. They might, for example, change the login form so that the passwords you enter are sent to them.

On IPFS, however, when you ask for something, you don’t request an IP address from the network, but instead ask for a hash of a file - a web page, an image, a video, or whatever. For example, /ipfs/QmbKM1C3ggNVdQtTnQuhvWruyodK6TUnoxjYwg31Q3crcn is the address of a specific version of my GNU/Linux tutorial series. If I change the content, the hash changes.

Of course, people still want to be able to change their content without breaking all the links to it. For that, we have IPNS, InterPlanetary Name System. IPNS allows you to securely point to mutable content with a hash-like address (/ipns/).

These addresses are still not human-readable, but DNS can be used to resolve human-readable names to IPNS addresses, just like it’s currently used to resolve IP addresses from human-readable names.

Benefits

One of the coolest things about IPFS is that the load of serving files is shared between users. Ever heard of the “Slashdot effect”, wherein something cool gets linked to on social media and the server hosting it collapses under the sudden load? With IPFS, that won’t happen.

That’s because when you request a particular piece of content over IPFS, it gets cached to your machine and sits there until the allotted space is used up, or you decide to remove it. Anyone who asks for that piece of content has a chance of connecting to your IPFS node before anyone elses and recieving the content from you. They in turn will cache it, and serve it to future clients. Everyone who is looking at a particular piece of content shares the load of serving it to everyone else.

Technical Concerns

There have been worries about hash collisions in the data store. IPFS uses multihash, which allows its hashing function to be upgraded, and currently, implementations use SHA-256. This StackOverflow post makes a good point about the likleyhood of such a collision:

If we have a “perfect” hash function with output size n, and we have p messages to hash (individual message length is not important), then probability of collision is about p2/2n+1 (this is an approximation which is valid for “small” p, i.e. substantially smaller than 2n/2). For instance, with SHA-256 (n=256) and one billion messages (p=109) then the probability is about 4.3*10-60. A mass-murderer space rock happens about once every 30 million years on average. This leads to a probability of such an event occurring in the next second to about 10-15. That’s 45 orders of magnitude more probable than the SHA-256 collision. Briefly stated, if you find SHA-256 collisions scary then your priorities are wrong.

So, perhaps, when IPFS is truly interplanetary, we will have to switch to a new hash function. That’s fine - the way IPFS is built, that’s entirely possible.

Keys and Fingerprints

Fri, 22 Jan 2016 00:00:00 +0000

Here you will find the key material needed to identify me on various services.

OMEMO

Here are the keys for my XMPP clients.

Desktop:

F0AB7263 B6C022EF A8C38705 65076963
FCF10F75 B53666FB 92619D86 807D9619

Laptop:

2140AA59 2159DD24 01508815 90BA116C
9DD7AF4F 63FBFAA0 92346EF7 2AC36361

Phone:

1FC18869 CB8A790E 8264034E 9927DDFF
9424C8CF F0BAA6BE F8BD4DF4 3FFA3A32

GPG

My GPG key is as follows.

Off-site Writing

Mon, 01 Jan 0001 00:00:00 +0000

I’ve written for other sites as well; this is an incomplete directory of that work.

SCP Foundation

The SCP Foundation Wiki is a collaborative microfiction project to which I am an occasional contributor.

SCP-2568 “Snap to Grid”
SCP-7443 “Spear Point, NJ” (removed)

Freifunk Community Blog

I worked with Freifunk for Google Summer of Code 2019 to build a network simulation suite for Irdest, then called qual.net. Among other things, this work resulted in several blog posts.

Leonora Tindall on Nora Codes

Manually Restoring Scroll Position on Reload with JavaScript

Infrastructure Photography Manifesto (and Decker-in-HTML)

nimi sin

Adding Tags to My Projects Page

A Matter of Time

A Matter of Time

Methods Should Be Object Safe

Object Oriented Programming in Rust

Generics and Static Dispatch

Dynamic Dispatch and Object Safety

Adding Full Post Content to my RSS Feed

Customizing the RSS Feed

The content: Namespace

Pinning nixpkgs with Morph and Colmena

Why Pin nixpkgs?

Pinning with Morph

Pinning with Git Directly

Pinning with Colmena

Using a Pinned nixpkgs

Fly On, Columbia! (Fly Columbia II)

Fly On, Columbia! (Fly Columbia II)

The Horror

Twelve

Thirteen

Fifteen

Sixteen

Eighteen

This is the Horror

Fight the Horror

The Fediverse is Already Dead

What was the Fediverse?

Isn’t That Mastodon?

Diversity of Design

Fediverse as Ideoform

What Killed the Fediverse?

Communities as the Unit of Moderation

Right Wing Extremism

Racism

A Quick Aside on CWs

Instance Drama

Misattribution of Malice

We Killed the Fediverse

Gargron Killed the Fediverse

The Press Killed the Fediverse

We Have to Keep Going

Fallout: New Vegas Is Like a TTRPG With a Bad DM

Nevada Route 164

Killers with No Fixed Abode

Zero to Climax in No Tension Flat

Scaling Mastodon in the Face of an Exodus

The Setup

The Cause

The Solution

Lessons Learned

Increase Your Database’s Resources

Split Your Sidekiq Queues

Fully Utilize Your Database

Mastodon 4.0

Into the Future

It's Time to Get Hyped About Const Generics in Rust

Constant Generic Parameters

Const Generics Reduce Runtime Complexity

Generic Constant Expressions

Conditional Implementation

Const Implementation Bounds

Moving Forward

Modular Synthesis and UNIX

Text, Audio, and Control Voltage

Oscillators, Filters, grep, and awk

Envelopes, Signal Flow, and mkfifo

Modularity, Open Source, and Market Forces

Back to Zsh

Names, Legal Names, and Fractally Deferred Responsibility

Names

Legal Names

Names and Software Extension

Harmful Uses of Legal Names

Fractal Deferment of Responsiblity

Do Not Record Legal Names (Unless You Have To)

Oscillators, Filters, `grep`, and `awk`

Envelopes, Signal Flow, and `mkfifo`