Continuously Delivering this Blog with Nix, Hugo and CircleCI

September 05, 2019

1360 words

7 min

NOTE: Some details outlined here are now out of date. I have transitioned to Zola instead of GoHugo. This was a change primarily made out of interest, I have no complaints about GoHugo. The nix and CircleCI build/deploy system remains unchanged.

Having established this blog, I figured I should briefly explain how it's built, deployed and hosted using some of my favourite tools, Nix, GoHugo and CircleCI. Most of my setup comes directly from this article on the blog kalbas.it. This post includes more information regarding continous deployment, using CircleCI, to a static site host, Netlify.

This blog is built using Hugo and I use CircleCI to continuously deploy a static version of the blog to Netlify. In order to see new content or edits applied to the production version of the blog the following things happen...

I write a post or make edits
I commit and push the changes, git add . + git commit [...] + git push
CircleCI automatically starts a build and deploy pipeline
The build deploy pipeline completes by pushing the statically built site to Netlify

The power of CircleCI, being able to automatically publish changes to the live blog, is amazing and greatly reduces the overhead of each successive addition or edit I make. I don't need to think about deployment, I just write content, save the changes to GitHub and the rest is automatic. To do Continuous Delivery with CircleCI properly I use a reproducible package management system called Nix; Nix ensures a consistent build environment that can be used by the CI system and locally.

Building Locally

Building and serving the blog locally is easy if you have the Hugo binary installed, however, I want to make sure that when I build the site within a CD pipeline the exact same binary for hugo is used. To do this, I use a nix-shell derivation that refers to a pinned version of nixpkgs and automatically downloads and links a pinned version of the site theme into the themes/ directory.

let

  # See https://nixos.wiki/wiki/FAQ/Pinning_Nixpkgs for more information on pinning
  nixpkgs = builtins.fetchTarball {
    # Descriptive name to make the store path easier to identify
    name = "nixpkgs-unstable-2019-07-31"; # Commit hash for nixos-unstable as of 2019-02-26
    url = https://github.com/NixOS/nixpkgs/archive/c0e56afddbcf6002e87a5ab0e8e17f381e3aa9bd.tar.gz;
    # Hash obtained using `nix-prefetch-url --unpack <url>`
    sha256 = "1zg28j760qgjncqrf4wyb7ijzhnz0ljyvhvv87m578c7s84i851l";
  };

in

{ pkgs ? import nixpkgs {} }:

with pkgs;

let
  hugo-theme-hello-friend-ng = runCommand "hugo-theme-hello-friend-ng" {
    pinned = builtins.fetchTarball {
      # Descriptive name to make the store path easier to identify
      name = "hugo-theme-hello-friend-ng-2019-08-01";
      # Commit hash for hugo-theme-terminal as of 2019-02-25
      url = https://github.com/rhazdon/hugo-theme-hello-friend-ng/archive/091ba7b1d2836ebf1defa1908020524174df6353.tar.gz;
      # Hash obtained using `nix-prefetch-url --unpack <url>`
      sha256 = "04hsr2wj8jxvql6cqasc56gcy5jhnf74aqcswn5jh2rmiskljdmk";
    };

    patches = [];

    preferLocalBuild = true;
  }
  ''
    cp -r $pinned $out
    chmod -R u+w $out

    for p in $patches; do
      echo "Applying patch $p"
      patch -d $out -p1 < "$p"
    done
  '';
in

mkShell {
  buildInputs = [
    hugo
    nodePackages.node2nix
  ];

  shellHook = ''
    mkdir -p themes
    ln -snf "${hugo-theme-hello-friend-ng}" themes/hugo-theme-hello-friend-ng
  '';
}

The important part in the above is defining nixpkgs using the builtins.fetchTarball function; this ensures the download matches the predetermined hash and therefore ensures that the packages defined in mkShell { buildInputs: [ hugo ] } are always consistent.

I can activate this shell locally using nix-shell which provides an environment with the pinned hugo package in $PATH.

$ which hugo
$ nix-shell
=>
[nix-shell:blog]$ which hugo
=> /nix/store/mjiyzf8sms6jgn62yl7scpp5kzh4vhd4-hugo-0.55.4/bin/hugo
[nix-shell:blog]$ exit
$ which hugo

This means that I can remove hugo from my global path or update my global version of hugo, but within the context of this nix-shell I will always have a consistent version. In practice, instead of using nix-shell whenever I want to use this consistent shell environment, I use direnv to activate the shell automatically; expect a better explanation of this soon. The key here is that I can use this same nix-shell derivation within the CI build scripts to ensure builds of the site in CI and locally are using the same tools and binaries.

Build the site using nix-shell --run "hugo"
Serve the site using nix-shell --run "hugo serve"

CircleCI

I won't dive into linking CircleCI to a GitHub repository here, it's a simple process, however, in order to have the site build using CircleCI properly I need to define a CI configuration in $(blog-root)/.circleci/config.yml. CircleCI uses this config.yml file to determine when to run builds and how to run them. First of all, I only want to run the build deploy job on the master branch, so I specify that at the start. To build the blog with the nix tools I use the lnl7/nix:2019-05-04 docker image, this image includes the nix tools. I update the tools and then install some common utilities missing by default. The build config so far looks like...

version: 2

jobs:
  build:
    branches:
      only:
        - master
    docker:
      - image: lnl7/nix:2019-05-04
    steps:
      - run:
          name: Install dependencies
          command: |
            nix-channel --update
            nix-env -u
            nix-env -iA \
              nixpkgs.gitMinimal \
              nixpkgs.nodejs-10_x \
              nixpkgs.openssh \
              nixpkgs.yarn \
              nixpkgs.gnutar \
              nixpkgs.curl \
              nixpkgs.findutils \
              nixpkgs.glibc \
              nixpkgs.gnugrep \
              nixpkgs.gnused \
              nixpkgs.gzip

To actually build the blog I add a step named 'Build', this uses the nix-shell --run "hugo" command. It is important to note here that in the above 'Install dependencies' step, the nix tools are updated along with nixpkgs but these updates won't spillover and affect the site build itself, because of the consistent nix-shell that is used. Using the nix-shell command here loads the shell.nix file and will download the pinned nixpkgs version and themes, this means that the shell environment that runs hugo to build the site is exactly the same as the local nix-shell I use to build the site.

- run:
    name: Build
    command: |
      nix-shell --pure --run "hugo"

The final component of the deploy is the 'Deploy' step. This step uses a nix shell called ./ci-shell.nix to run the npm command netlify deploy. Even though this step relies on a npm package, node2nix can be used to make the step reproducible by converting the node lockfile into a nix derivation that pins each dependency. To do this I installed the dependencies I wanted by running npm install netlify npx and then ran node2nix. node2nix creates a few files, the important one is ./default.nix (I renamed this file to ci-shell.nix for clarity), if I use nix-shell to enter into this shell environment, all the node dependencies and a consistent node version will be provided in a reproduceable way, regardless of what is installed locally. The CircleCI build config utilizes this shell in the final step, to run npx netlify deploy [...]

- run:
    name: Deploy
    command: |
      nix-shell -A shell --pure ./ci-shell.nix --run "npx netlify deploy --dir=public --message=\"$CIRCLE_SHA1\" --prod"

The final .circleci/config.yml file is...

version: 2

jobs:
  build:
    branches:
      only:
        - master
    docker:
      - image: lnl7/nix:2019-05-04
    steps:
      - run:
          name: Install dependencies
          command: |
            nix-channel --update
            nix-env -u
            nix-env -iA \
              nixpkgs.gitMinimal \
              nixpkgs.nodejs-10_x \
              nixpkgs.openssh \
              nixpkgs.yarn \
              nixpkgs.gnutar \
              nixpkgs.curl \
              nixpkgs.findutils \
              nixpkgs.glibc \
              nixpkgs.gnugrep \
              nixpkgs.gnused \
              nixpkgs.gzip
      - checkout
      - run:
          name: Build
          command: |
            nix-shell --pure --run "hugo"
      - run:
          name: Deploy
          command: |
            nix-shell -A shell --pure ./ci-shell.nix --run "npx netlify deploy --dir=public --message=\"$CIRCLE_SHA1\" --prod"

Once this config file is committed to source and CircleCI is set up to build the repository, any new commits that are pushed to master will fire off the build pipeline and the updated blog should be deployed to Netlify.

Summary

Using a build pipeline to continuously deploy my blog helps to reduce the burden of making a new post or an edit to the source. Nix helps in this process by ensuring that the commands run in the build pipeline use the exact same dependencies as the commands I run locally, this is achieved through pinning a version of nixpkgs.