Configuring Health Checks with Elixir + Phoenix

Matched Pattern logo

Herman Singh

2018-10-03

Every critical application you deploy to production should have monitoring. Application monitoring can come in different forms. Sometimes in-house solutions are developed, or more commonly, third party resources are utilitized.

In order to use PagerDuty, and many other monitoring solutions, you will need to establish an endpoint (e.g /_health) for your service that can be pinged for a status. Typically, the application will return a 200 OK, and the monitoring solution will consider the resource reachable and thereby healthy. There are a lot of automation around alerts and recovery that you can do once this is hooked up.

So for every critical application you write, you’re going to end up writing a health check endpoint. Well I’m going to show you how you can easily get a health check endpoint in your Elixir + Phoenix app by integrating ExHealth. With ExHealth you only have provide <20 LOC to get something that is specifically configured for your application.

Installation

This is pretty straightforward. ExHealth is a Hex package, so it can be added to your codebase by putting the following in your mix.exs file:

def deps do
  [
    {:ex_health, "~> 0.1.0"}
  ]
end

Now run

$ mix deps.get

Configuration

ExHealth will read from your config.exs, so you can set configuration options by adding the following:

config :ex_health,
  module: MyApplication.HealthChecks,
  interval_ms: 5000

module is the module in which you are going to define your health checks

interval is the time (milliseconds) between performing health checks

About ExHealth

So there are a couple different philosophies for how this endpoint should work. One way is to have all of the checks perform synchronously when you get a request. While I think this is a fine solution, I built ExHealth to perform the checks asynchronously. I chose to go the asynchronous route for a couple reasons:

  • If the checks are performed async, and if many requests are made to this endpoint, multiple subsequent requests will not be made. You can stop worrying about flooding the database or other upstream dependencies. The results between intervals will remain the same until the next health check is performed.
  • Since checks are async, they can live in a completely isolated process. This is the most important reason for choosing async. If part of your application goes down, ExHealth will stay up. It does not really make much sense to crash the health check if the database is not reachable. Rather, I want to be able to hit the health check endpoint and get a response back that tells me exactly what is UP and DOWN.

Here’s an actor diagram describing ExHealth:

Adding Health Checks

Now you have to define whatever module you set in the config. In the example above, we called it MyApplication.HealthChecks. Inside this module we’ll define the health checks that need to run for your application.

defmodule MyApplication.HealthChecks do
  import ExHealth

  health_check("Redis") do
    MyApplication.Redis.ping()
  end
end

If you are just monitoring an Elixir process, you now have HealthServer (a Supervised GenServer) that is running alongside your application, performing routine health checks. You can query this process by doing the following:

iex> ExHealth.status()
%ExHealth.Status{
  checks: [
    %ExHealth.Check{
      mfa: {ExHealth.SelfCheck, :hc__ExHealth_HealthServer, []},
      name: "ExHealth_HealthServer"
    }
  ],
  interval_ms: 15000,
  last_check: nil,
  result: %{check_results: [], msg: :pending}
}

This currently returns a ExHealth.Status struct. The most import data here is the result map. Inside this, you can inspect msg, which is going to return :pending | :healthy | :unhealthy. If you want more information about each individual health check, they are contained in check_results.

If you’re running a Phoenix application, then there’s only one more step to get this integerated into an endpoint. You can leverage ExHealth.Plug to handle all the controller logic for you. All you have to do is assign it to a route. Open up router.ex and add the following lines:

defmodule MyAppWeb.Router do
  ...

  scope "/" do
    pipe_through(:browser)

    # Health Check
    forward("/_health", ExHealth.Plug)

    ...
  end

  ...
end

The above example will forward requests for /_health to ExHealth.Plug. You can set it to whatever endpoint you wish. Now you should be able to go to localhost:5000/_health and get a JSON response that looks like the following:

{
  last_check: "2018-09-18T06:43:53.773719Z",
  result:{
    check_results:[
      [
        "Database",
        "ok"
      ],
      [
        "PhoenixExampleWeb_Endpoint",
        "ok"
      ]
    ],
    msg:"healthy"
  }
}

I hope this post has helped you somehow, and I hope that ExHealth makes some of this work a little easier. Thank you for reading. My name is Herman Singh. I’m a Senior Developer at Matched Pattern. If you have any questions, don’t hesitate to reach out me at herman@matchedpattern.com. Also, if your company is looking for help with your next Elixir project, a reminder that we specialize in building high performing scalable Elixir solutions.