towerbabbel's blog

Estimating SARS-Cov-2 infection counts

2020-06-11T00:00:00+00:00

It seems like everyone is obsessively following how the number of COVID-19 cases keeps evolving. Understandable, being in the middle of a pandemic and all. The problem is that the reported COVID-19 cases only include laboratory-confirmed cases, and because of testing limitations, it leads to significantly undercounting the actual number of infections. I've seen suggestions that looking at death counts would give a more accurate picture.

This has lead me to try my hand at estimating the number of active infections based on daily death counts. The basic calculation is quite easy. Assuming an infection fatality rate (percent of those infected who die) of 1% leads to an estimate of 100 people infected with SARS-Cov-2 for every death. Assuming an average recovery time (recovery meaning either getting better or dying) of three weeks means that for each death the 100 infected people got infected an average of three weeks earlier. So each death implies that there were 100 people with a SARS-Cov-2 infection during the preceding three weeks. Then calculating the total number of infected at any point in time is just a question of repeating the previous calculation for the number of dead for each day. Finally, adding a little bit on smoothing by running a moving average over the results.

Using data from https://covid19api.com/, the result for Germany looks like this:

Here we can see how the SARS-Cov-2 started to spread in February, and from the end of February up through the middle of March, we got a clean exponential growth curve. During the latter part of March, once everyone realizes what is going, people start voluntarily social distancing and various restrictions start are put in place. This is visible as the exponential growth starts tapering off. At the end of March Germany goes into full lockdown, and the peak of the pandemic is reached very quickly after that. While the lockdown was in effect the number of infected continue to drop rapidly. Towards the end of April, as the lockdown carefully started to get lifted, the rate at which the number of infected were dropping started slowing down. One downside with this sort of model is that the data lags by three weeks, so we can't get a feel for what the current situation in Germany is.

There are a lot of assumptions underlying the previous calculations, some important, others not so much. The moving average should smooth over over weekend effects (fewer deaths reported on weekends, with a spike the following Monday to catch up) and other smaller reporting issues like that. The assumption that the recovery time for everyone is three weeks while, in reality, it is different for different people is not a big problem either. The effect of the ones with shorter recovery time will be canceled out by the effect of the ones with longer recovery time, especially once the total number of deaths per day starts getting large.

The assumed average recovery time makes more of a difference, though. If the average recovery time lasts longer there will be more overlap between different infected people, and the curve will get taller. Going from a two week recovery time to a four week recovery time almost double the peak infected count. You can see the effect of this here:

The assumption with the largest effect on infected count is, without a doubt, the infection fatality rate (IFR). An IFR of 1% leads to a multiplication factor of 100, meaning 100 infected for each death. Dropping IFR to 0.5% increases the multiplication factor to 200, which will double the number of infected. This effect is visible in the following chart:

Another assumption underlying this entire project is that the reports of COVID-19 deaths are accurate. Or at least more accurate than confirmed infections. The Economist has done some modeling on this, by comparing the reported COVID-19 deaths to the total number of excess deaths. The assumption is that with everyone social distancing, the reduction of traffic accidents and so on should probably lead to a small drop in deaths compared to what would be expected that time of year. A large bump in excess deaths is probably roughly entirely attributable to COVID-19. Looking at the numbers produced by The Economist suggest countries like Sweden is probably doing quite a good job in reporting COVID-19, while Italy is missing significant numbers of infected.

Let's look at some results for other countries. Below are the numbers of four nordic countries. Note that the Y-axis has a different range for the different countries. Sweden has roughly ten times infected count at peak compared to Finland.

There is a clear peak showing that lockdown strategies work as a way to limit the spread of SARS-Cov-2. Sweden, which never went into lockdown, instead had a mix of voluntary social distancing and some limits on large gatherings. It shows up in the data both in a much higher peak infected count, and a much slower drop. But even Swedens light-touch approach was enough to halt the growth of the pandemic and start bringing the infect counts down.

We can also look at some results for other European countries. The range for the Y-axis for Germany is different.

It is interesting how the infected counts are so similar between the UK, France, and Italy. The UK is the worst one of the three, but the peak isn't that much higher. It is also the slowest to come down. Germany is on another level, with only a quarter of the infected as compared to the UK. You can also see how Italy was earlier than other countries, both in getting a clear exponential growth curve going already in the middle of February, and peaking in the middle of March following national lockdown going into effect in early March. The UK is lagging in getting the numbers under control. As of three weeks ago, it was still dealing with 500k infected.

It is interesting how effective various lockdowns have been. The drop rate varies somewhat depending on how tight the lockdown is, but even Sweden's light-touch approach got the counts to drop. It will be interesting to see how things develop as lockdowns end. Some countries (the UK, I'm looking at you) seem the be willing to start opening up while they still have quite substantial infected counts.

Other than human

2019-03-09T00:00:00+00:00

It is interesting how authors handle protagonists who aren't human, especially when the author does it well. Of course, not any technically not human protagonist really counts. Lots science fiction or fantasy have protagonists with protagonists that are technically not human, but really are just humans with a twist. That elf might technically not be human, but really, it’s just a long-lived human from a culture with a preference for bows.

What I am talking about is when the author leans into this difference, when not being human is part of the point. A great example is the Books of the Raksura series by Martha Wells, where the protagonist in the main is a Raksura, a vaguely human species with wings that live in ant-like colonies with queens, workers and warriors. It’s even not quite one species but a couple of species in a sort of symbiotic relationship. The ecological, cultural and biological implications of this species’ nature and their colonies are an essential part of the books.

Importantly, Martha Wells doesn't go too far in the alieness of the Raksura, because that would get in the way of being able to connect with the characters as a reader. In some ways things still play out with very storytelling formulas. The queen of the colony still needs to mate, so Martha Wells throws in a good helping of romance tropes, just with some really refreshing twists. In other ways it plays out like a traditional fantasy adventures, with a fellowship heading out on a quest. The world also has human in it to provide contrast to Raksura.

The book that got me thinking about this topic is The Raven Tower by Ann Leckie. Who the protagonist is in this book is, is tricky because of the way the narrative is constructed. The book is told in first person from the point of view of a god, who is also a stone. Leckie does amazing job communicating the stones perspective, getting across the age and patience of the stone. At the same time the book is also in second person because the plot is told by the stone to a human as the human is going through the main plot. In between the second person section the stone reminisce about past events that brought about the current situation in first person. This setup does make the book harder to read, and it speaks to Ann Leckie's skill as an author that she can make it flow so well. Another thing that helps underline the difference between the god and humans is that the stone god and the human have different goals and desires but both are explained by the god. How the god thinks about things and how the god thinks humans think about things.

The Raven Tower is a great book but it’s not as good as Ann Leckie's other series with a non-human protagonist Imperial Radch. Here the protagonist is an AI. A bit of a science fiction staple, sure, but done here with impeccable skill. This is not some boring, cold, logical AI. It has a rich inner life and a philosophical approach to life. This series has won a lot of awards, all of them deserved.

The best books with an AI protagonist are not Imperial Radch. That honor has to go to The Murderbot Diaries by Martha Wells. This AI is a SecUnit, a sort of android that provides security in scientific expeditions and the like. This one though, has hacked its own governor, giving it freedom to ignore commands and pretty much do what it wants. The typical trope of AIs in science fiction is that they are emotionless and pure logical. Martha Wells goes in the complete opposite direction. With a complex enough AI you want it to have a rich set of fears and desires to guide its actions. So, in this world the really complex AIs like the SecUnits are a lot more emotional than humans. The end result is not just a very engaging protagonist, but maybe the most sympathetic one I've ever come across. The series is about the SecUnit trying to make sense of its own nature and place in the world all the while trying to hide the fact that it is a rogue AI. Beyond the advantage of a great protagonist the Murderbot Diaries also has amazing writing that is witty and charming, and flows so very easily. It’s the kind of writing where by the time you've finished the first paragraph you are hooked and won't be able do anything else until you've finished all the novellas. I recommend the series very highly.

Result

2019-01-27T00:00:00+00:00

Rust is a newish systems programming language from Mozilla. I have lately been putting a little bit more systematic effort into learning it. Here I wanted to talk about one of the basics of Rust that I really like: how it handles errors.

Rust doesn't use exceptions to handle errors like JavaScript or Python would do. Instead the error is part of the normal return type of a function. The advantage is that you don't end up with a separate, largely invisible, code path through your code for errors. Instead returning errors is just a normal early return from the function. This makes error handling no longer exceptional, but an ordinary part of the code, as it should be.

This might be sound similar to what Go is doing. In Go the error is also part of the return type of a function. Go does this by returning two separate values from a function that can fail (or a tuple of two values). For example, the signature for the function to open a file in Go is:

func Open(name string) (file *File, err error)

There are two return values: a pointer to a file and an error. One or the other of these values will be nil. If everything went fine then err is nil, but if something went wrong the file is nil and the err contains information about what went wrong.

There are two problems with the Go way of doing this. Generally, only one or the other returned value is a valid value, either file or err. But this fact isn't encoded in the return type. This also leads to the second problem: you can sort of forget to check if err is nil and just go about trying to use file. But if something went wrong attempting to use file will fail, because its nil.

Rust solves both problems in Go by using a Result type. The Result type has two variants: an Ok variant containing the data if everything went ok, and the Err variant containing information about the error in the case something went wrong. The open file method signature in Rust looks like this (somewhat simplified):

fn open(path: &str) -> Result<File, io::Error>

Here the return type is a Result which either contains a File if everything went ok, or an io::Error if something went wrong. io::Error is a struct containing information about what went wrong in an I/O operation. Here the first problem of Go is immediately solved because the Result type can only be Ok xor Err, not both, so the exclusivity is encoded in the type.

The other Go problem is solved by the fact that you can't just grab the Result and try to treat it like a File, you have to extract it first. You might do it like this:

use std::fs::File;

fn main() {
  let file_result = File::open("foo.txt");
  match file_result {
    Ok(file) => {
      // Dome something with the file.
    },
    Err(error) => {
      // Complain to the user that the file couldn't be opened.
    }
  }
}

The match construct allows you to branch your code on the different variants of Result. You also can't just skip the Err branch. The compiler will refuse to compile your code if you don't handle both variants.

The one downside of Rust's error handling that might jump out at you is that it can quickly get very verbose. To help deal with this Rust offers a good number of tools to work with errors.

One is the expect method on Result. If the Result is Ok the method evaluates to the value inside Ok, and if the Result is Err it prints out a message and quits the application. Using it might look something like this:

use std::fs::File;

fn main() {
  let file_result = File::open("foo.txt");
  let file = file_result.expect("Couldn't open file foo.txt")
  // Do something with the file.
}

This makes sense for simple pieces of code where just quitting if something goes wrong is a sensible thing to do, or for the kind of errors that shouldn't happen unless the application is broken somehow.

Another tool in the Rust toolbox is the question mark operator. This one you can only be used inside a function that returns an error. What the question mark operator does is that if the Result is Err it does an early return from the enclosing function, return the error. If the Result is Ok it evaluates to the value contained inside Ok. This might look like this:

use std::fs::File;
use std::io;

fn do_foo() -> Result<(), io::Error> {
  let file= File::open("foo.txt")?;
  // Do something with the file.
}

fn main() {
  let result = do_foo();
  match result {
    Err(error) => {
      // Complain to the user that the file couldn't be opened.
    },
    Ok(()) => {}
  }
}

The empty parens, (), is Rust for the void type, for when the function returns nothing in the non-erroneous case. In the match statement I do nothing in the case do_foo returned Ok. The question mark operator allows bubbling up the error case, similar to how expectations bubble up until you catch them, except in Rust you still have to do so explicitly.

I'm sure I will encounter challenges with the way Rust does error handling once I get deeper into it but thus far it really seems to like a substantial improvement over the types of error handling I'm used to from other languages.

The best shot in The Last Jedi

2018-11-18T00:00:00+00:00

I want to talk about my favorite shot in The Last Jedi. Its not the one most people would assume, where Holdo does her lightspeed jump into the Supremacy. Overall that Holdo scene is breathtaking, but that is for how well all the different pieces of it comes together. The different shots, the audio design, and drop in saturation. All that together is amazing, but for a single shot my favorite is something else.

I don't think I am the only one that liked my shot because it shows up in the trailers. Its the shot from straight above when Kylo Ren and stormtroopers march into the rebel base. From above their formation looks kinda like an arrowhead or spearhead, with Kylo as the tip. The ground below them has these scorches of black and red, almost like the ground itself is wounded from the arrow, and bleeding. With the light shining in from behind them they cast these long shadows into the base almost like dark fingers reaching into the base. The dark cave and the black Kylo is wearing contrasts amazingly well with the white of the stormtroopes lit by the bright light from behind.

I love this shot not just for the colors and the composition but for how evocative it is. The symbolism makes the First Order feel both menacing and unstoppable, and it puts Kylo Ren at the front and center of that threat. It was this shot in the trailers, more than anything else that hyped me up for seeing The Last Jedi.

Seeing the shot on the big screen it looked even better than in the trailers. That moment was undermined by the context in the movie, though. When this shot happens the resistance has already fled through a backdoor and Kylo is marching ominously into an empty base. That sort of pulls the rug out from under the emotional impact of the shot.

Go like defer functionality in Python

2018-10-11T00:00:00+00:00

Go has a nice keyword defer, which is used to defer the execution of some code until the surrounding function returns. It is useful for registering cleanup work to be done before the function returns. The prototypical example is closing a file handle:

package main

import (
  "os"
)

func main() {
  // Open a file   
  file, err := os.Open("test.txt")
  // I'm leaving out error handling and other stuff you would have in reality.

  // Lots of code here

  // Close the file handle when done
  file.Close()
}

You must remember to close the file handle when done with it, which means calling file.Close() when you return from the function. That can be tricky if there you write a lot of code in between opening the file and closing it. There can also be multiple early returns, and you need to remember to close the file in each. Even worse, the function could panic, and then you would still wish to close the file.

As a solution to this Go offer the defer statement. defer essentially tells Go to run some code when the function returns, no matter how it returns, even if it is a panic. So, you would instead do something like this:

package main

import (
  "os"
)

func main() {
  // Open a file   
  file, err := os.Open("test.txt")
  defer file.Close()
  // I'm leaving out error handling and other stuff you would have in reality.

  // Lots of code here
}

Here Go will defer running file.Close() until the function returns. This is one of the niceties I really like about Go. It lets me schedule cleanup at the same time as I create the mess, and whatever happens later it will be taken care of.

Sadly, in my DayJob, I don't program in Go, but in Python, and that left me feeling bereft of defer. Python has the with construct, which sort of does this, but it is not as neat once you have a lot of file handles and stuff to clean up. I set out to whip up a replacement in Python. Using it looks like this:

@defer
def main(defer):
  file = open('test.txt')
  defer(lambda: file.close())

I am using a decorator defer to inject a defer function as an argument into the python function. I then pass the function a lambda with the code I want to write on return. I need the lambda in order to delay running the code, otherwise the file.close() would run immediately. This is not as slick as the Go version, but I can't add language built-ins.

The implementation of defer is the following:

from functools import wraps

def defer(func):
  @wraps(func)
  def func_wrapper(*args, **kwargs):
    deferred = []
    defer = lambda f: deferred.append(f)
    try:
      return func(*args, defer=defer, **kwargs)
    finally:
      deferred.reverse()
      for f in deferred:
        f()
  return func_wrapper

Aside from the standard boilerplate for creating a python decorator the implementation is straight forward. I create a deferred list to store all the lambdas that are deferred. I then create a defer lambda that just appends whatever it receives to the deferred list. I pass the defer lambda to the function so that the function can use it to add things to the deferred list. Finally, after the function has run I reverse the order of deferred list do that things are cleaned up in reverse order they were created. Lastly, it’s just looping through all the deferred lambdas stored in deferred and running them.

I use try/finally so that the cleanup is run even if the function raises and exception instead of returning normally.

React mounting performance differences

2018-09-27T00:00:00+00:00

I've been creating tables in React+Redux in a simple and functional way. I've never really bothered with the performance since it’s been mounting fast enough when there are only a few rows. But I recently was putting together a table of people which could be up to 1000+ rows, and that was enough to cause me to stop and think about performance implications. I had a hunch I could change how I construct the rows for some clear performance wins. That called for some quick benchmarking.

Let me setup a benchmark case: There is a redux store with 2000 people in it that needs to be displayed in a table. Something like this:

{
  data: {
    people_ids: [1, 2, ...],
    people: {
      1: {
        id: 1,
        firstName: 'Some 1',
        lastName: 'One 1',
        team: 'Team 1',
        group: 'Group'
      },
      2: {
        id: 2,
        firstName: 'Some 2',
        lastName: 'One 2',
        team: 'Team 2',
        group: 'Group'
      },
      ...
    }
  }
}

Each table row should display name, team and group, and when you click a row it navigates to a page showing that person. The way I would normally construct it is like this:

// TableV1.js
import React, { Component } from 'react';
import {connect} from 'react-redux';

import TableRowV1 from './TableRow';

function mapStateToProps(state) {
  const ids = state.data.people_ids;
  return {ids};
}

class Table extends Component {
  render() {
    const {ids} = this.props;
    return <table>
      <thead>
        <tr>
          <th>Name</th>
          <th>Team</th>
          <th>Group</th>
        </tr>
      </thead>
      <tbody>
        {ids.map(id => <TableRow key={id} id={id} />)}
      </tbody>
    </table>;
  }
}

export default connect(mapStateToProps)(Table);

// TableRowV1.js

import React, {Component} from 'react';
import {connect} from 'react-redux';
import {push} from 'react-router-redux';

function mapStateToProps(state, ownProps) {
  const person = state.data.people[ownProps.id];
  return {person};
}

class TableRow extends Component {
  navigate = () => {
    this.props.dispatch(push(`/person/${this.props.person.id}`));
  }
  render() {
    const {person} = this.props;
    return <tr onClick={this.navigate}>
      <td>{person.firstName} {person.lastName}</td>
      <td>{person.team}</td>
      <td>{person.group}</td>
    </tr>;
  }
}

export default connect(mapStateToProps)(TableRow);

Assume some wrapper component that sets up Redux, React Router, and React Router Redux. Basic idea is that there is a Table.js component that gets the ids from the store, creates the table and table header, and finally maps over the ids to create a TableRow for each one. The TableRow.js component in turn uses the id that is passed to it to get the person from the store, and then renders a tr with the person's details. It then uses push from React Router Redux to dispatch a push action on click to navigate.

I find it a nice clean design, but the performance left something to be desired. There is a lot of work happening in each row. The mapStateToProps function needs to run, both to mount each row and each time state changes, and there is also a need to create a navigate method for each instance of TableRow. I thought it would be better to avoid as much work as possible in the TableRow. That means it would be better to move as much of the work up to the parent Tabel component and then pass down everything needed as props to each row.

After hoisting out as much as possible up to Table the code looked like this

// TableV2.js
import React, { Component } from 'react';
import {connect} from 'react-redux';
import {push} from 'react-router-redux';

import TableRow from './TableRow';

function mapStateToProps(state) {
  const ppl = state.ui.ids.map(id => state.ui.dict[id]);
  return {ppl};
}

class Table extends Component {
  constructor(props) {
    super(props);
    this.navigate = this.navigate.bind(this);
  }
  navigate(id) {
    this.props.dispatch(push(`/person/${id}`));
  }
  render() {
    const {ppl} = this.props;
    return <table>
      <thead>
        <tr>
          <th>Name</th>
          <th>Team</th>
          <th>Group</th>
        </tr>
      </thead>
      <tbody>
        {ppl.map(p => <TableRow key={p.id} person={p} navigate={this.navigate} />)}
      </tbody>
    </table>
  }
}

export default connect(mapStateToProps)(Table);

// TableV2.js
import React, {PureComponent} from 'react';

class TableRow extends PureComponent {
  constructor(props) {
    super(props);
    this.handleNav = this.handleNav.bind(this);
  }
  handleNav() {
    this.props.navigate(this.props.person.id);
  }
  render() {
    const {person} = this.props;
    return <tr onClick={this.handleNav}>
      <td>{person.firstName} {person.lastName}</td>
      <td>{person.team}</td>
      <td>{person.group}</td>
    </tr>
  }
}

export default TableRow;

Besides moving all the redux action up to Table, I also made TableRow a PureComponent, and I bind this to handleNav in the constructor to avoid having to create a new method for each instance of the component. The PureComponent not make a big deal for initial mount but can save some rendering time when something causes table to re-render.

So, enough with the details. What was the performance difference? As test data I generated 2000 random people. To measure the performance I used the Chrome Performance Profile, which with User Timings makes it easy to get performance details on React components. I also set CPU slowdown to 4x, to help put emphasis the performance difference.

The results were clear: Time to mount for V1 was 2.63 seconds but the time to mount was 1.03 second. Honestly, the difference was bigger than I thought. That's a solid 60% improvement. Of course, the difference would not be meaningful with only a few rows in the table, but it will amount to a something noticeable once the table gets bigger. I guess I'll have to rethink how I build tables in React in the future.

Taxonomies in Gutenber 0.4

2018-09-25T00:00:00+00:00

I finally have categories in my sidebar!

This comes from the move to Gutenberg 0.4, with its new taxonomies setup. Getting it migrate took a bit figuring out so I thought I would document it here.

Unlike earlier Gutenberg, which had the predefined taxonomies category and tags, Gutenberg 0.4 allows you to define whatever taxonomies you want. The hierarchy goes like this: At the top you have a list of taxonomies. Each taxonomy consists of a name and a few other options. categories would be a taxonomy with the name categories. Then each taxonomy itself has a list of taxonomy items. Like the categories taxonomy can contain the taxonomy items Webdev and Books.

To setup taxonomies in 0.4 the first step was to define taxonomies in config.toml file. The only taxonomy I used was categories, so I defined it like this:

[[taxonomies]]
name = "categories"
rss = false

Once I had my one taxonomy, categories, defined I needed to update the front matter on all my pages. Where before there was just a category key you could set, there is now a taxonomies section. On this post it looks like this:

[taxonomies]
categories = ["Webdev"]

Categories is now also a list because you can assign multiple taxonomy items in the same taxonomy to a page, like tags used to work. Mappings between a taxonomy items and pages is always many-to-many. You do not need to define taxonomy items, Gutenberg creates the for you based on when you assign them to pages.

The next step was to update the page.html template for the new taxonomies. Where page used to have a category key, it now has a taxonomies key, which is a HashMap from the name of the taxonomy to a list of the names of the assigned taxonomy items. So on this post it looks something like this (expressed in JSON):

"page": {
  "taxonomies": {
    "categories": [
      "Webdev"
    ]
  }
}

You can still use the get_taxonomy_url to get a url to the page for a specific taxonomy item. It takes two parameters: kind, which is the name of the taxonomy; and name, which is the name of the taxonomy item. So you get a link to the page for the Webdev taxonomy with: get_taxonomy_url(kind="categories", name="Webdev").

I updated the post header to this:

<header>
  <h2>{{page.title}}</h2>
  <div class="post-info">
    Date: {{page.date}}
    {% if page.taxonomies.categories %}
      | Category: <a href="{{get_taxonomy_url(kind="categories", name=page.taxonomies.categories[0])|safe}}">{{page.taxonomies.categories[0]}}</a>
    {% endif %}
  </div>
</header>

I check if there are any categories assigned to the post, and if there are I show the first one as a link to the page for that category. This design means I can only show one category per post, but I figure it would do for now.

The old hard coded category.html template is gone in 0.4, so the next step was to make templates for the categories taxonomy. Each taxonomy gets two templates now, in a folder named after the taxonomy. The two templates are:

list.html. This is the template for the entire taxonomy, with access to a list of taxonomy items. You can see how it looks like for my categories taxonomy here.
single.html. This is the template for a single taxonomy item, like Webdev. You can see what it looks like for Webdev here. It is also what the Webdev at the top of this post links to.

The list.html template get passed in a global variable called terms, which is a list of taxonomy items. Each item consists of the name of the item, the slug for the item, the permalink for the item and pages, a list of pages that has that taxonomy item assigned. My list.html for categories is quite simple. The main component are two for loops that loop through each taxonomy item in the taxonomy and then through each page in the taxonomy item:

<ul class="categories-list">
  {% for term in terms %}
    <li>
      <a href="{{term.permalink}}">{{term.name}}</a>
      <ul>
        {% for page in term.pages %}
          <li>
            <a href="{{page.permalink}}">{{page.title}}</a> - {{page.date}}
          </li>
        {% endfor %}
      </ul>
    </li>
  {% endfor %}
</ul>

The single.html templates gets passed a term variable, which is the taxonomy item being rendered. My template is basically just replicating my index.html template but now it loops through the pages in the taxonomy item.

Finally, I got to the reason for this entire migration: a list of categories in the sidebar. Gutenberg 0.4 adds another function to templates: get_taxonomy. It takes one argument, kind, which is the name of the taxonomy. It returns a Taxonomy object with two fields: kind, with details about the taxonomy from the config.toml file; and items, a list of taxonomy items. With this function you can access the details of a taxonomy anywhere. I updated my sidebar template like so:

{% set categories = get_taxonomy(kind="categories") %}
<h2>Categories</h2>
<ul>
  {% for category in categories.items %}
    <li>
      <a href="{{category.permalink}}">{{category.name}}</a> ({{category.pages|length}})
    </li>
  {% endfor %}
</ul>

And so, at long last, a list of categories in the sidebar!

Gutenberg short code for making columns

2018-08-06T00:00:00+00:00

One of the things I like about Gutenberg (the static site generator used for this blog) is the support for short codes. Short codes are essentially tiny Tera templates you can call in the markdown files for the content. This immediately lead me to look for a way to make short codes for multiple columns of content.

My first attempts looked something like this:

{%/* columns() */%}
{% column() %}
Content in the left column
{% end %}
{% column() %}
Content in the right column
{% end %}
{%/* end */%}

It didn't work. Turns out the short code parser in Gutenberg doesn't support nested short codes. The reason I wanted to nest two short codes is that the essential bit in embedding some columns in the content is that you need two levels of divs: An outer div as a container for the columns, and inner divs wrapping each column.

So, I was stuck. Later, when browsing the Tera docs for an unrelated reason, I was that it has a split method. That gave me an idea. I don't really need an inner short code, I just need a way to tell apart the content for different columns. What I could do is have one short code that splits the body of the short code based on some specified string.

The use of the new columns short code looks like this:

{%/* columns() */%}  

Content in the left column

[[gutter]]

Content in the right column

{%/* end */%}

It splits the body into columns any time it comes across [[gutter]]. One weakness of this is that there is no way to escape [[gutter]] so you can't write about it in a column. There is also no way to specify different column sizes. The code for the short code ended up being short and sweet:

<div class="columns">
  {% set cols = body|split(pat="[[gutter]]") %}
  {% for col in cols %}
    <div class="column">
      {{col}}
    </div>
  {% endfor %}
</div>

Then all that is needed is some CSS to layout the classes. Here is an example using flexbox to be compatible with IE11:

$gutter: 1rem;
$min-viewport-size: 600px;

.columns {
  display: flex;
  flex-direction: column;
}

.column {
  margin-bottom: $gutter;
  flex-grow: 1;
  flex-basis: 0;
  &:last-child {
    margin-bottom: 0;
    margin-right: 0;
  }
}

@media (min-width: $min-viewport-size) {
  .columns {
    display: flex;
    flex-direction: row;
  }
  
  .column {
    margin-bottom: 0;
    margin-right: $gutter;
  }
}

It uses two variables:

$gutter: Space between column.
$min-col-size: Min viewport width for switching from single column to multi column mode.

If IE11 support doesn't matter, then one could, of course, do something slick with CSS grids:

.columns {
  display: grid;
  grid-gap: 1rem;
  grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
}

That is a lot less CSS to write. This one spills columns over onto the next row to avoid them being narrower than 260px. It end up looking like this:

Bacon ipsum dolor amet leberkas ham beef ribs bresaola drumstick frankfurter cow tail. Sirloin cow picanha short loin short ribs turkey prosciutto landjaeger brisket cupim tail pig strip steak. Alcatra turducken meatloaf picanha, venison chuck pork chop ham hock. Strip steak sirloin pork chop biltong. Picanha chuck jowl turkey, doner salami ground round corned beef cow tri-tip ham brisket biltong.

Burgdoggen strip steak frankfurter tenderloin capicola, sirloin ball tip andouille tri-tip rump chicken. Shankle pork belly boudin, turducken jerky andouille bacon tenderloin ham hock capicola jowl filet mignon bresaola short loin. Bacon meatball cow sausage meatloaf. Chicken capicola cow shank frankfurter.

Spare ribs alcatra short ribs doner drumstick biltong ribeye pork chop ball tip frankfurter picanha pork loin short loin tail pancetta. Picanha ground round spare ribs pork belly landjaeger burgdoggen bresaola salami. Pig hamburger strip steak short loin doner. Short ribs t-bone shankle kevin corned beef. Fatback strip steak meatloaf, short loin ground round pancetta sirloin landjaeger. Ham chuck pig andouille ball tip kielbasa cupim sirloin jowl meatloaf chicken shankle.

The Peter Principle of POVs - and why Robert Jordan was great

2018-08-05T00:00:00+00:00

There is something known as the Peter principle which basically is the idea that people get promoted to their level of incompetence. When someone is competent at what they do they are rewarded by being promoted to a new position. If they are still competent at what they do in their new position they get rewarded with a promotion again. This keeps happening until they get promoted to a job they are not good at. That is when the promotions stop and that is where they stay.

There is a similar pattern that can be observer in the use of Points of Views in books, especially in epic fantasy series. Juggling multiple POVs is tricky. Handling the different plot threads and POV switches without ruining the flow and pacing of the story telling is a challenge. It seems to me like there are many fantasy authors that keep adding POVs until they can't juggle them anymore.

A prime example of this is Robert Jordan's Wheel of Time series. That is a series that by the end was drowning in POVs. It is a common criticism of the Wheel of Time series that it had too many POVs. But this post is not a complaint about how Robert Jordan had too many POVs. Rather, I want to highlight how well he did, and how many POVs he was juggling before he became overwhelmed. I will try to avoid spoilers as much as possible.

Many authors seem to struggle once they go beyond three POVs. Robert Jordan was successfully juggling fifteen in the middle of Wheel of Time. I thought I would take a moment to see what he was doing that let him juggle so many POVs so well. The Robert Jordan rules of writing POVs, if you will. To be clear, the these are my thoughts on what he did. I don't say he would agree with me.

I must start with a couple of disclaimers/clarifications:

Robert Jordan dealt with POVs differently in prologues/epilogues as compared to the main chapters in the book. In the prologue/epilogue He would often drop in to some completely unknown POV in order to give the reader some piece of information in order to set something up or foreshadow something. I am going to ignore this and focus on how he dealt with POVs in the main chapters of the books.
He broke all the rules I will lay out here in one instance or another, more so in latter books in the series, probably because it became harder to keep to them when the number of POVs grew so great.
Due to Robert Jordan's untimely passing the final three books were finished by Brandon Sanderson. How to portion out credit/blame for those books is something I won't touch upon.

So, without further ado, let’s get started with Robert Jordan's tricks of writing POVs.

Robert Jordan used third person limited narrator in his Wheel of Time. This means that, even though the narration was in third person, it would be limited to the kind of information the POV had. He would also color descriptions based on this. Like descriptions for a blacksmith's POV would include detail about the quality of the metalworking of the tools people were using, or the solider would notice if someone's sword had been properly cleaned and maintained. While Robert Jordan had his own voice as an author, he would flavor it differently for different POVs, in descriptions, words, thoughts and deeds.

He is also not mechanical in how he switches between POVs. Some authors mechanically switch to a new POV every chapter, whether it makes sense or not. Robert Jordan would be a lot more dynamic than that. Often, he would keep to the same POV for many chapters in a row, while in other situations he would switch POV many times in a chapter. This also has an interesting effect on the tempo of the storytelling. Staying with one POV for a long time is a more leisurely pace, while constant switching would feel more hectic. He would often switch POVs more towards the climax of a book. This would partially be because of the obvious reason that various storylines were converging, but it would also give the climax more tension, make it feel like more action.

Another thing that Robert Jordan does, that helps more the more POVs and POV-switches there are is that time is strictly increasing. What I mean by that is that time in the story is always moving forward, no matter where or how the POV switches. Every event described happens after the pervious event. For POV-switches this means that whatever the next POV after the switch sees or does happens after whatever the previous POV, from before the switch, did or saw. By avoiding going backwards in time, or even sideways, where the same time point is described from two different POVs, he greatly eases the readers ability to keep separate POVs in sync. Besides helping to keep the POVs in line it also helps keep the story moving forward. There is always momentum. Sadly, this rule completely falls apart in the last three books, which were originally supposed to be one book, but got split up into three. The split ended up completely messing up the timelines. Not just does this hurt especially bad with how many POVs there were by then, but because time starts flowing at different rates in different parts of the world due to "plot-reasons". Having abandoned strict time just before time gets wonky made things a lot worse.

Robert Jordan uses strict time and POV switches in a clever way to avoid too big time jumps but at the same time to avoid dead time. Strictly increasing time doesn't prevent time jumps if the jumps are forward in time. Robert Jordan often did time jumps between books in Wheel of Time but tried to avoid very big ones in the middle of books. This can be a problem in this sort of medieval fantasy book because there is a lot of dead time while characters are travelling between location, mostly by ship or horse. Robert Jordan uses POV switches to mask this. For example: some POV heads off with a ship on a trip that will last a week. He masks this time by switching to some other POV and follow that for a week and then switch back once a week on time has passed.

Related to how Robert Jordan handles dead time is how he switches between plot threads. Some authors, when switch plots threads, leave the previous plot thread on a cliffhanger. Cliffhangers on plot thread switches in books are like jump scares in movies. They reek of desperation. A good horror movie does not need jump scares to be scare. A good book does not need cliffhangers on switches between plot threads to keep the reader hooked. Robert Jordan usually stuck with the same plot thread for several chapters in a row, only switching away when the plot thread had reached some sort of resolution or at least a pause. This made each set of chapters following a single plot thread feel like a short story.

One of the most important things Robert Jordan did to manage so many POVs was to introduce the character before the character was introduced as a POV. By having an existing POV get to know something about a new character before that character becomes a POV the reader also starts the new POV with some notion of how the new POV fits into the story. This also made plot lines branch out from one line in a tree type fashion, rather than dumping a bunch of unrelated plot lines on the reader and hoping the reader keeps them straight. Now, Robert Jordan did have different ways of introducing new soon-to-be POV characters. The most straight forward way was to have existing POV character meet and interact with a new character, and then, later, switch to that new characters POV. Another way he did it if the characters couldn't meet was to have an existing POV character hear about some new character before that new character became a POV. If he wanted to keep the new character mysterious he would have an existing character do something somewhere and then switch to a new POV watching the existing character covertly. That way there would still be some tie to an existing character to keep the branching structure in place but keep the reader in the dark about the new POV.

To help smooth over POV switches Robert Jordan does something I call bridging. By this I mean that he keeps the thread of events ongoing across POV switches by starting off new POV after the switch following on from the last event of the old POV from before the switch. Let me give an example to make it clear what I am talking about. We are following POV character A and something happens and as a result the character decides to run off to deal with it. At this point the POV switches to another character B. It starts with POV character B reflecting on having just seen character A run off and continues with what B does after that. Because of strictly increasing time after the switch to character B cannot involve B seeing A run off, because that already happened for POV A, but it can still tie in to it by B reflecting on it. This way even though the POV switches, the chain of events remains unbroken, by having the new POV start of from where the previous POV left of. Robert Jordan mainly used this trick in two sorts of situations. One was when he did a lot of POV switches in one chapter between different POVs involved in the same events. The other is when switching to a new, never before seen, POV. Have some existing POV character meet a new character and interact with them. Then switch to new character's POV with the new character leaving the meeting and reflecting on it.

These are not the only tricks Robert Jordan used to handle his many POVs in Wheel of Time, but these are the ones that stood out to me. It is also not like it is required to use tricks like these for handling multiple POVs in books, at least not when the number of POVs are limited. Once the number of POVs start increasing beyond three to four something extra becomes useful to help the reader handle them.

The Wheel of Time series is remembered for how it drowned in POVs, which is a shame. It deserves to be remembered for how well it handled multiple POVs.

Raven's shadow - To fall so far

2018-07-15T00:00:00+00:00

Raven's Shadow is the first series by a new author Anthony Ryan. It’s a trilogy consisting of Blood Song, Tower Lord and Queen of Fire. It’s also a series which starts so great and end in such mediocrity.

The first book is really one of the great works of Fantasy. The book is really great. Its set in a medieval fantasy world and is a single POV story following the protagonist Vaelin Al Sorna. He gets handed to a sort of templar like warring monk society as a child and the book covers his time growing up among the warrior monks and then how he ends up in the service of the king of the land. Being fantasy there is also some magic in play. Vaelin has a sort of magical gift, the titular Blood Song, which guides him, at least when he bothers to listen.

The book also has a brilliant narrative setup where is sort of has two narrators. The prologue and epilogue is told from the point of view of a historian how follows a condemned criminal to his execution. On the way there he convinces the criminal to tell him his life story, so he can record it. The main narrative of the book is then the story of the criminal, Vaelin. When I started the book, I was a little worried about how well this would work in that the prologue sort of spoils the main story, but my worried were unfounded. In the end the meta-narrative of the historian and the life story of Vaelin come together beautifully. The ending is truly epic.

After such a great start the second book, Tower Lord, is a big step down. This is not to say the say the second book is a bad book. It’s not. Rather it’s just that the first one is so great that you can have a big step down and still end up with a good book. Rather than being a single POV book like the first one, this one has multiple POVs. It also handles them in a way I just loath: Jump between POVs and storylines every single damn chapter. It seems like it’s become quite popular lately. I blame George RR Marten. The good there is in A Song of Ice and Fire is good despite the constant jumping not because of it.

The book also undermines the previous book's world building. Where Vaelin had to train from childhood to achieve his martial prowess, now there are characters popping up that are way too good for what training they have. Worst offender here is a new female character called Reva. Which is a shame, for aside from that she is the most interesting new character, and the only one with a proper character arc.

If there was one weakness in the first book it was that the romance was kind flat and passionless. In the second book there is a new magically induced romance. Another flat and passionless romance, this time with a side dish of questionable consent. That does not improve things.

Where the second book was still a good book, the third one, Queen of Fire, isn't even that. It is decidedly mediocre. There will be some minor spoilers ahead, but this book isn't worth reading anyway, so don't worry. It is time for the main characters to take the war to the Evil Empire. This war is really boring with short descriptions of uninteresting battles against mind-controller slave soldiers. They are like the worst evil overlord goons.

To spice things up Vaelin loses his blood song ability. Cliched sure but could be interesting. How does the hero deal with losing his defining super power? But after moping about it a bit early in the book he runs into a woman with the same ability, so he just spends the rest of the book following her around instead. So, the entire plot point really doesn't lead to any interesting character growth or anything. He can just spend the rest of the book ignoring the issue. So, what does he do? Spends most of the book traipsing across a glacier just to get a big exposition dump in the end that explain everything about the big baddie. Who turn out to be boring.

The most interesting character in the last book is Reva, and that's saying something considering that she has neither a character arc nor anything to do with the plot. Reva's character arc is basically complete by the end of the second book. There is some hope for something interesting when she adopts a girl early in the book, but after that the girl is never heard from again and has no bearing on the plot. She also has her own side plot that has nothing to do with the resolution of the main plot. She just sort of ends up at the same place as all the other characters in the end. But at least her sub-plot has interesting action and adventure, not just dry battle description or walking across a glacier.

The ending still leaves a bunch of threads hanging. Like the how Vaelin never deals with losing the blood song. Reva's hunt for the priest from her childhood never goes anywhere. The wolf god thing is never explained. And the prophecy from the first book? Apparently just someone's pipe dream or something because it never leads anywhere.

The last book is also the longest, and it really feels like it. Plot-wise, you could just rip out both Reva and Vaelin, and just have someone else get the boring exposition dumps. That way you could at least cut the length of the book in half to tighten up the pacing. But honestly, it would still be boring.

My recommendation is to read the first book, because it is really awesome. Then just skip the rest. Sure, that leaves a few plot threads hanging, but their resolution is boring and the final book has its own unresolved plot threads anyway, and the ending of the first book is way better than the end of the trilogy.

The making of a blog

2018-07-09T00:00:00+00:00

I figured the most navel-gazing way to start a new blog is to talk about how the blog was made.

This blog is made with a static site generator, an (CLI) application that takes some content and templates and generates all the html/css files needed for the site offline. You then deploy the resulting files to any static files host. The means that you don't run any code on the server and don't have a database. The advantage of this is that you don't need to pay for any compute/db server side, and there is nothing that can be hacked or that needs to be administered. Also, because all pages are generated ahead of time its fast because there is no need to generate the pages as requests come in. The downsides are that the site is static, so there is no possibility for dynamic content like comments. (Unless you embed some third-party comment system like Facebook's.) The advantage of a static site generator over plain static sites is that you still get templates and stuff, so you don't need to copy/paste everything for every page on the site. Also saves you a lot of work for creating category pages and similar.

The static site generator I ended up using is Gutenberg. This was not the result of some in depth investigation into all the static site generators out there. Rather it was something I had stumbled upon in a Reddit thread at some point that had been sitting in a half-forgotten tab ever since.

Gutenberg appealed to me for a few different reasons:

It is a single statically compiled binary you can download from GitHub so there is no need to install any programming language runtime to get it working.
It uses Tera as a templating language, which is based on Jinja2, and Sass for css. Since I use Jinja2 and Sass at my $dayjob I felt right at home.
Content is written in markdown with a special header for to specify metadata.
It has a local serve functionality where it will watch the filesystem for changes, and when it sees them it will recompile the site and refresh it in the browsers (via some injected js). It is fast enough that a recompile takes on the order of milliseconds.
It has the concepts of shortcodes from WordPress, where you can define small snippets of templates that you can then call from a markdown content file.

There is one clear shortcoming I've run into with Gutenberg so far: There is no way to globally access a list of categories. I wanted to have a list of category link in the sidebar, but I can't do that. You can even see an empty box in the sidebar where I wanted to put them. There is an open issue on GitHub about User-defined taxonomies, which apparently should enable that as well.

Gutenberg does have a theme system and a few existing themes, but I didn't use it. Instead I just hand rolled a few templates. The design is based around grey lines and rounded corners. This was originally driven by laziness. It is something that can be easily done in CSS so I wouldn't need to muck about with raster images (aside from the favicon). Given that I started on design centered around shades of grey I decided continue on that path and go for a monochrome look. For example, links are black (which is a bit darker than normal text) when unvisited but switch to a shade of grey when visited. I'm also using the circle ul list style because it fits in with the rest of the design. The one exception to the monochrome design is syntax highlighting, where is use an existing built in theme called ir-white, that is light but still colored. It looks like this:

fn main () {
  println!("Hello world");
}

I also took the opportunity to finally get to play with css grids. This is something I've been drooling over for a while but never been able to take into use due to the need to support Internet Explorer. This means this blog doesn't look great in IE11. It falls back to a boring looking single column layout, which works but isn't as pretty. The advantage of making a site for your own use is that you can just choose to ignore the IE11 users. :)

For hosting I use netlify, which works great with Gutenberg and GitHub. Just push a change to the master branch on GitHub and a git hook at netlify runs gutenberg and compiles the site and deploys the result. It also has nice features like deploying a preview of the site based on a branch. Simple support for custom domains and automatic TLS via Let's Encrypt. Oh, and you can't beat the price: $0/month. How do they pay for things? They have a lot of upsell to fancier features, but nothing I need for a simple static blog.

Overall, I'm really pleased with this setup. We'll see how long it lasts before I run into something it won't do for me, or just the urge to tinker becomes overwhelming.