Adding E2E tests to an Elixir library

In WyeWorks we maintain a library that sends a notification every time an exception is raised in a Plug application. Besides adding unit and integration tests for every new feature, we used to run manual regression tests on every release to check we didn’t break previous functionality and the new ones are actually working.

However, as the number of features increased, doing manual testing started to be very time-consuming so we decided it was about time to start automating this process.

We ended up using Wallaby for the testing but any other tool like Hound would work just fine. The scope of this article is not to explain how Wallaby works but to show how it can be used to test a library.

Creating a testing application

The problem when trying to E2E test a library is that you need an actual application that does something with it. In our case we need a Phoenix application so let’s just create one.

Since this is also a test artifact it would be nice to create this app inside /test. However, we’ll have to make a couple of changes to have it working.

First, let’s move our current test files to /test/unit.

mkdir test/unit
mv test/*.exs test/unit

Now, for the tests to work, we have to tell mix we changed its default test path.

def project do
  [
    ...
    test_paths: ["test/unit"]
  ]
end

We should check our tests are still working running mix test.

Finally, let’s create our Phoenix app.

cd test
mix phx.new example_app --no-ecto --no-dashboard --no-live
cd example_app

Update: running static code tools

If we run mix format or mix credo we’ll get some nasty errors because those tools will check everything under test/**/*.{ex,exs} which includes the newly created app.

We need to exclude that folder updating their settings in the .formatter.exs and .credo.exs files.

# .formatter.exs

[
  inputs:
    Enum.flat_map(
      ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"],
      &Path.wildcard(&1, match_dot: true)
    ) -- Path.wildcard("test/example_app/**/*.*", match_dot: true)
]

# .credo.exs

%{
  configs: [
    files: %{
      ...
      excluded: [..., ~r"test/example_app/"]
    },
  ],
  ...
}

Our first E2E test

Following the official guides we know we need to install chromedriver, include the testing library as a dependency, and configure it.

npm install chromedriver

# test/example_app/mix.exs

defp deps do
  [
    ...
    {:wallaby, "~> 0.29.0", runtime: false, only: :test}
  ]
end

# test/example_app/test/test_helper.exs

Application.put_env(:wallaby, :base_url, ExampleAppWeb.Endpoint.url())
{:ok, _} = Application.ensure_all_started(:wallaby)

# test/example_app/config/test.exs

# make sure ExampleAppWeb is set up to serve endpoints in tests
config :example_app, ExampleAppWeb.Endpoint,
  ...
  server: true

The easiest way to check everything is working is to add a test that validates the default landing page.

Landing page

# test/example_app/test/landing_page_test.exs

defmodule ExampleAppWeb.LandingPageTest do
  use ExUnit.Case, async: true
  use Wallaby.Feature

  feature "Shows a welcome message", %{session: session} do
    session
    |> visit("/")
    |> find(Query.css("section.phx-hero"))
    |> assert_has(Query.css("h1", text: "Welcome to Phoenix!"))
  end
end

> mix test
.

Finished in 1.0 seconds (1.0s async, 0.00s sync)
1 feature, 0 failures

Randomized with seed 41741

Start testing our library

Now we have everything in place it’s time to include our library in the example app.

Let’s crack open the mix.exs file and include the library as a dependency using a relative path so we are always testing against the latest version of the code. Remember we are under test/example_app so the library code is a couple of folders up.

defp deps do
  [
    ...
    {:boom_notifier, path: "./../.."}
  ]
end

The next thing is to configure the example application so it uses the library we want to test. For this example, let’s configure Boom as if we were a user of this library.

For sending emails we are using the mailer that is included in Phoenix. It also provides a plug that allows us to preview the emails in an in-memory mailbox. This is particularly convenient because we can check what our emails will look like without actually sending them.

# test/example_app/lib/example_app_web/router.ex
use BoomNotifier,
    notifier: BoomNotifier.MailNotifier.Swoosh,
    options: [
      mailer: ExampleApp.Mailer,
      from: "me@example.com",
      to: "foo@example.com",
      subject: "BOOM error caught"
    ]

# this provides access to the fake email inbox
forward "/mailbox", Plug.Swoosh.MailboxPreview

Finally, we need to add a route that raises an error.

# test/example_app/lib/example_app_web/controllers/page_controller.ex

defmodule ExampleAppWeb.PageController do
  use ExampleAppWeb, :controller

  def index(conn, _params) do
    # an exception is raised and an email
    # should be sent
    raise "Boom"
    render(conn, "index.html")
  end
end

Writing an actual test

After having the setup ready, it is time to think about what flows we want to test. To keep it simple, we’re providing the smallest scenario we can test in Boom:

Visit /.
Check an error was raised.
Navigate to /mailbox.
Check the email was sent.
Check the email contains the error information.

Error page in `/`

Phoenix error

Fake inbox in `/mailbox`

Phoenix email

Send notification test

defmodule ExampleAppWeb.SendNotificationTest do
  use ExUnit.Case, async: true
  use Wallaby.Feature
  alias Wallaby.{Browser, Element, Query}

  # helpers
  def get_latest_from_inbox(page) do
    page
    |> find(Query.css(".list-group"))
    |> find(Query.css(".list-group-item"))
  end

  def click_on_email(inbox_item, session),
    do: visit(session, Element.attr(inbox_item, "href"))

  feature "sends email when an exception happens", %{session: session} do
    # raise the exception
    session
    |> visit("/")
    |> assert_text("RuntimeError at GET /")

    # got to the inbox and select the latest email
    session
    |> visit("/mailbox")
    |> get_latest_from_inbox()
    |> click_on_email(session)

    # check email metadata
    session
    |> Browser.find(Query.css(".header-content"))
    |> Browser.assert_text("BOOM error caught: Boom")
    |> Browser.assert_text("me@example.com")

    # check email content
    session
    |> Browser.find(Query.css(".body"))
    |> Browser.assert_text(
      "RuntimeError occurred while the request was processed by PageController#index"
    )
  end
end

Running the tests

At this point you can only execute the E2E tests if you run mix test inside the example_app directory. We can create a mix task so we can run this tests from the root directory of our library.

# be sure to run this in the root folder (not inside the example_app directory)
mkdir -p lib/mix/tasks
touch lib/mix/tasks/end_to_end_test.ex

# lib/mix/tasks/end_to_end_test.ex

defmodule Mix.Tasks.EndToEndTest do
  @moduledoc "Runs e2e tests"

  use Mix.Task

  @impl Mix.Task
  def run(_) do
    exit_status = Mix.shell().cmd("cd test/example_app && mix deps.get && mix test")
    exit({:shutdown, exit_status})
  end
end

And now we can add this task as an alias in our mix file.

defmodule BoomNotifier.MixProject do
  ...

  def project do
    [
      ...
      aliases: [
        ...
        e2e: ["cmd mix end_to_end_test"]
      ],
      ...
    ]
  end
  ...
end

> mix e2e
.

Finished in 1.0 seconds (1.0s async, 0.00s sync)
1 feature, 0 failures

Randomized with seed 41741

Conclusions

Not only doing manual testing was taking a lot of time but also it was a matter of time until a scenario was missed and a broken version of the library got released.

Now we can run the main scenarios in a couple of seconds and we can even configure our CI to run the end-to-end testing in every pull request.

Here is the PR with the complete code: https://github.com/wyeworks/boom/pull/78. If you have any suggestions of how this can be improved we’d love you tell us in the comments.

Blog article