Services

test automation and acceptance testing

Blog

Elixir development and test automation

Tech tips

Tech tips on different topics

Building the Blog

- 23/02/2019
elixirphoenixdevelopmentAWSS3

I wanted to update this site for a while and was looking at best options to do it. Should I use a static site generator like Gatsby.js or Jekyll? Should I use one of the many SaaS CMS platforms to host and edit the content and access it by API. Or should I use this opportunity to build the site as a Phoenix server rendered site, which is still the best option in most cases in my opinion.

Options/Alternatives

There are several options to building static sites nowadays. One of the popular new ones is Gatsby.js. The promise of JAMStack apps is that it’s all “serverless” and uses different SaaS products that are integrated. There’s often a CMS, like Contentful or a DB service like Airtable involved. The data is updated via that service and there’s no need to build the admin pages for adding content yourself. With Gatsby.js, one option is also to use local Markdown files that get built into the static site. I have played around with Gatsby.js and it is cool, but in the spirit of learning and sharpening the toolset, I wanted to see how I could have a similar experience with Phoenix where I don’t need to worry about creating content on a web form but rather stick to good local apps for writing the Markdown. Ulysses is one nice app for writing Markdown, and I could just as well just use Visual Studio Code/Emacs/Vim for it as the posts often involve some code and these editors are the natural habitat of it.

I wanted to have two different types of content for now:

  • blog posts
  • tech tips from different areas such as coding, test automation, tools, shell commands

Out of these content types, blog posts take a lot of effort to write and I don’t expect to write them too often. Tech tips are something that could happen any moment during the daily work where I’d realise that a piece of information would be useful to share. Creating content is difficult enough when there are no hurdles, so being able to do it on the spot with the same tools I’m using all day seems like a win. I could be writing some code and come up with something that could be useful for others and writing a short note about and/or copy-pasting some code right in the editor seems like the ideal solution.

Building blocks

The process I had in mind is roughly the following:

  1. Edit files locally with your favourite markdown editor
  2. Push changes to Github
  3. CircleCI builds and deploys files to S3
  4. CircleCI notifies the web server of new updates with a webhook
  5. Web server syncs the DB with the files from S3
  6. Web server invalidates the cache so that new content will be displayed

To achieve this, the following building blocks are necessary:

  • Elixir
  • Phoenix framework
  • Postgresql
  • Github
  • Amazon S3 bucket for storing posts and images
  • IAM user that has access to the bucket
  • CircleCI (or some other CI) that published content to S3
  • Imgix to serve images

It’s not mandatory to even have a database as the data could be stored in a GenServer and updated when the server starts and after a webhook event is received. I wanted to keep things flexible for now and opted in for using Postgresql.

Editing a post

Editing a post can be done in any editor. As the files are stored on github, all changes are nicely visible in git history. Once the change is committed and pushed, CircleCI starts a new job that deploys files to S3.

Syncing from S3

It’s quite a common pattern to need to pull in data from outside and make it available for website users. I have started creating a Feed umbrella app that handles some of the common services. Elixir doesn’t have a library for all SaaS type services but most of the time what the libraries do is:

  • make an HTTP request
  • authenticate (usually an API key)
  • parse the content

These things are relatively simple to implement in Elixir and are not a hurdle for adopting it.

Retrieving the data from S3

ExAws is a fully fledged AWS client library for Elixir. Each service is a separate package and for S3, ExAws.S3 is required.

The following configuration is needed on the client side:

config :ex_aws,
  region: "eu-west-2",
  access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role],
  secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :instance_role]

The code to get the posts from S3 looks like the following:

defmodule Feed.S3.Request do
  alias ExAws.S3

  @bucket "<BUCKET NAME>"
  @post_regex ~r/posts\/.*.md/
  @tip_regex ~r/tips\/.*.md/

  def get_posts do
    @post_regex
    |> list_posts()
    |> parse_list()
  end

  def get_tips do
    @tip_regex
    |> list_tips()
    |> parse_list()
  end

  defp list_items(regex) do
    @bucket
    |> S3.list_objects()
    |> ExAws.request!()
    |> Map.get(:body)
    |> Map.get(:contents)
    |> Enum.filter(fn item ->
      String.match?(item.key, regex)
    end)
  end

  defp parse_list(files) do
    files
    |> Enum.map(fn file ->
      body =
        @bucket
        |> S3.get_object(file.key)
        |> ExAws.request!()
        |> Map.get(:body)

      %{
        body: body,
        filename: file.key
      }
    end)
  end
end

get_posts and get_tips returns a list of maps containing the post body and the filename, which is used as a key in the DB to find the post later on.

Handling Markdown

The functions above returns a list of items with raw Markdown as the body. The files also have metadata that is formatted using Front Matter. The post and the metadata need to be parsed for storing into the DB.

Two libraries are used for converting the Markdown files:

Front Matter

---
title: Building the Blog
date: 2019-02-23 09:30:00
summary: I set out to modernise this site and wanted to do use this opportunity to learn something new and try to create as good and convenient platform that I could
published: true
author: Marko Honkanen
image:
tags:
  - elixir
  - phoenix
  - development
  - AWS
  - S3
---

Parsing Front Matter and Markdown

Next sections show the code for handling parsing the file. Parsing the Markdown files is inspired by this blog post.

defmodule Feed.S3.Sync.Post do
  defstruct title: "",
    summary: "",
    content: "",
    filename: "",
    author: "",
    cover_image: "",
    tags: [],
    published_at: "",
    published: false

  alias __MODULE__

  @options %Earmark.Options{code_class_prefix: "language-"}

  def compile(file) do
    file
    |> split()
    |> extract()
  end

  defp split(data) do
    [frontmatter, markdown] = String.split(data, ~r/\n-{3,}\n/, parts: 2)
    {parse_yaml(frontmatter), Earmark.as_html!(markdown, @options)}
  end

  defp parse_yaml(yaml) do
    [parsed] = :yamerl_constr.string(yaml)
    parsed
  end

  defp extract({props, content}) do
    %Post{
      title: get_prop(props, "title"),
      summary: get_prop(props, "summary"),
      author: get_prop(props, "author"),
      content: content,
      cover_image: get_prop(props, "image"),
      tags: get_prop(props, "tags"),
      published_at: get_date_prop(props, "date"),
      published: get_boolean_prop(props, "published")
    }
  end

  defp get_prop(props, key) do
    parse_value(props, key)
  end

  def get_date_prop(props, field) do
    props
    |> get_prop(field)
    |> Timex.parse!("%Y-%m-%d %H:%M:%S", :strftime)
  end

  def get_boolean_prop(props, field) do
    props
    |> get_prop(field)
    |> convert_to_boolean()
  end

  defp parse_value(props, "tags") do
    case get_value(props, "tags") do
      :undefined -> nil
      list ->
        Enum.map(list, fn item -> to_string(item) end)
    end
  end

  defp parse_value(props, key) do
    case get_value(props, key) do
      :undefined -> nil
      x ->
        to_string(x)
    end
  end

  defp get_value(props, key) do
    :proplists.get_value(String.to_charlist(key), props)
  end

  def convert_to_boolean("true"), do: true
  def convert_to_boolean("false"), do: false
end

Images

There are several SaaS services for serving images on the web. The one I’ve been using is Imgix. The service gets original images from a source, for example S3, and then serves the image allowing for transforms as specified in the URL. One example of this is having a single image that is served in multiple resolutions in the UI by specifying the width (w) and height (h) parameters.

This ties in conveniently with how the blog posts are stored (S3). The Imgix image source is set to the same S3 bucket as where the blog posts are stored. When the images are uploaded to the same Github repo from where they are pushed to the S3 bucket, the images will be directly available from the blog posts.

Triggering a sync

The last part of the puzzle is how to trigger a content refresh. In a Phoenix app, adding a new route to where CircleCI can get/post a message to trigger a new update is trivial. We’ll need some form of authentication so that someone doesn’t just trigger updates for the fun of it. When a message is received, the app calls sync functions defined in Feed.S3.Sync module.

Summary

As expected, Elixir and Phoenix are excellent options also for building a blog. While some of the tech could be an overkill for this purpose, it was a pleasure to implement. The technology is going to be reusable in other projects, something that Elixir’s flexible module system makes easy.

There are several more topics to cover but that will have to wait for future blog posts.