At $work, we’ve been using Apache httpd as a webserver and gateway as a default choice.

Yet, some configuration issues led to a production outage last year, due to configuration complexity and difficulty to test.

Let’s review the issue that led to an outage, and do an overview of the available open source webservers.

1. Usecase and configuration complexity

Our webserver is used for:

  • serving some static files (typically for UI assets);

  • serving as a front-end and a reverse proxy for dedicated back-ends, or other UI assets;

The webserver is running in a container, and its configuration is the same no matter the kubernetes namespace it runs in.

This means that some reverse proxy rules must only be applied if the hostname being served matches a rule (test vs production for example).

In Apache httpd config file, this looks like this:

RewriteCond %{HTTP_HOST} !test.example.com # (1)
RewriteRule . - [S=3] # (2)
RewriteRule subpath/backend1/(.*)$ https://test-backend1.example.com/$1 [P,NC]
RewriteRule subpath/backend2/(.*)$ https://test-backend2.example.com/$1 [P,NC]
RewriteRule subpath/backend3/(.*)$ https://test-backend3.example.com/$1 [P,NC]

RewriteCond %{HTTP_HOST} !production.example.com
RewriteRule . - [S=3]
RewriteRule subpath/backend1/(.*)$ https://backend1.example.com/$1 [P,NC]
RewriteRule subpath/backend2/(.*)$ https://backend2.example.com/$1 [P,NC]
RewriteRule subpath/backend3/(.*)$ https://backend3.example.com/$1 [P,NC]

Requests are reversed-proxied to different backends depending on the subpath of the request.

Note that httpd reads this as follows:

  • if (1) is true (aka, we are not serving test system), then read (2);

  • (2) says to skip the next 3 rules.

The same happens for production.example.com.

The issue happened when someone added a RewriteRule for a new back-end, but forgot to increase the number of rules skipped.

It was missed in code reviews, and missed during testing in test systems, since [P,NC] was set for each rule.

Furthermore, nobody tested those rules locally, because starting httpd locally with its split configuration is not the easiest to do.

Surely, nobody should need to count rules manually, so there must be a better way.

This, along with how annoying it is to configure some structured logs in httpd, or have safe defaults, lead us to looking at what other webservers can do.

2. Requirements

We could have tried to have a service mesh and abstracted away the reverse proxy needs, but that’s a change bigger than we could handle.

We also thought we could "use the right tool for the job": 1 webserver for serving files, and 1 reverse proxy (like haproxy), but this would have added too much operational complexity.

So, we set out to look at other webservers than can also do some form of reverse proxy.

Our main requirements were:

  • open source;

  • widely used;

  • simple configuration / documentation;

  • good cli;

  • easy to run and test in local;

  • modular;

  • structured logs/metrics out of the box;

Performance is not on this list, because handling between 10 and 100 tps should be a piece of cake for any webserver in 2023.

3. Apache httpd, nginx, Caddy

Requirement Apache httpd

Open Source

⭐⭐⭐

Usage

⭐⭐⭐

Configuration/doc

⭐

CLI

⭐

Local Test

⭐

Modular

⭐⭐⭐

Logs/metrics

⭐

Apache httpd has been used a lot (being the second most used webserver in the world), is fully open source, and has many modules.

Yet, its configuration is tricky, even though its documentation is extensive, and its usage in local and its cli is old and not ergonomic.

Requirement Nginx

Open Source

⭐⭐

Usage

⭐⭐⭐

Configuration/doc

⭐⭐

CLI

⭐

Local Test

⭐

Modular

⭐⭐⭐

Logs/metrics

⭐

Nginx is a bit similar to Apache httpd, as they come from a similar era. It is the most used webserver in the world, with many modules, and decent documentation.

Yet, it’s switched to an "open core" model, where some of its features are locked behind an "enterprise" version. Its configuration and doc is a bit better than httpd, but the cli and local test isn’t much better (the official download page does not even provide a link to the windows version for example).

Requirement Caddy

Open Source

⭐⭐⭐

Usage

⭐⭐

Configuration/doc

⭐⭐

CLI

⭐⭐⭐

Local Test

⭐⭐⭐

Modular

⭐⭐⭐

Logs/metrics

⭐⭐⭐

Caddy is a much younger webserver, and has a much better developer experience out of the box. It’s fully open source, yet supported/developed by a company and the community, and its usage is growing rapidly. Metrics are exposed with a simple switch, and logs can be structured with a single switch too.

More and more modules are available for caddy, and its cli is clean and testing locally is as easy as caddy run.

Nothing everything is bright, though: the documentation is not always very easy to understand, and some things are only possible in 1 version of the configuration (json configuration vs Caddyfile configuration).

Finally, usually the major versions always suffer some immediate patches indicating a younger product with less testing that expected. Although, this is something that has been recognised and is now being the main focus of the Caddy community.

4. Configuration rewritten

Now, here is how the earlier configuration looks like in Caddy:

@test {
    host test.example.com
}

handle @test {
    handle_path subpath/backend1/* {
        rewrite * {uri}
        reverse_proxy https://test-backend1.example.com
    }
    handle_path subpath/backend2/* {
        rewrite * {uri}
        reverse_proxy https://test-backend2.example.com
    }
    handle_path subpath/backend2/* {
        rewrite * {uri}
        reverse_proxy https://test-backend2.example.com
    }
}

@production {
    host production.example.com
}

handle @production {
    handle_path subpath/backend1/* {
        rewrite * {uri}
        reverse_proxy https://backend1.example.com
    }
    handle_path subpath/backend2/* {
        rewrite * {uri}
        reverse_proxy https://backend2.example.com
    }
    handle_path subpath/backend2/* {
        rewrite * {uri}
        reverse_proxy https://backend2.example.com
    }
}

This is more verbose, but much clearer, and less error-prone.

It can also be shorten quite a bit by using a mapping:

@proxy {
    host test.example.com
    host production.example.com
}

handle @proxy {
    map {host} {b1} {b2} {b3} {
        test.example.com test-backend1 test-backend2 test-backend3
        production.example.com backend1 backend2 backend3
        default localhost localhost localhost
    }
    handle_path subpath/backend1/* {
        rewrite * {uri}
        vars upstream {b1}
    }

    handle_path subpath/backend2/* {
        rewrite * {uri}
        vars upstream {b2}
    }

    handle_path subpath/backend3/* {
        rewrite * {uri}
        vars upstream {b3}
    }

    reverse_proxy {vars.upstream}.example.com
}

Getting used to the new configuration style takes some time, but being much more declarative, it’s been much easier to avoid configuration issues.

(I’ve not shown serving files in both httpd and caddy, since this part was not the source of any issues.)