Why our Grafana URLs always require HTTP Basic Authentication

December 14, 2018

As part of our new metrics and monitoring setup, we have a Grafana server for our dashboards that sits behind an Apache reverse proxy. The Apache server also acts as a reverse proxy for several other things, all of which live behind the same website under different URLs.

People here would like to be able to directly access our Grafana dashboards from the outside world without having to bring up a VPN or the like. We're not comfortable with exposing Grafana or our dashboards to the unrestricted Internet, so that external access needs to be limited and authenticated. As usual, we've used our standard approach of Apache HTTP Basic Authentication, restricting the list of users to system staff.

Now, having to authenticate all of the time to see dashboards is annoying, so it would be nice to offer basic anonymous access to Grafana for people who are on our inside networks (and Grafana itself supports anonymous access). Apache can support this in combination with HTTP Basic Authentication; you just use a RequireAny block. Here's an example:

<Location ...>
   AuthType Basic
   [...]

   <RequireAny>
      Require ip 127.0.0.0/8
      Require ip 128.100.3.0/24
      [...]
      Require valid-user
   </RequireAny>
</Location>

People outside the listed networks will be forced to use Basic Auth; people on them get anonymous access.

It's also useful for system staff to have accounts in Grafana itself, because having a Grafana account means you can build your own dashboards and maybe even edit our existing ones (or share your dashboards with other staff members and edit them and so on). Grafana supports a number of ways of doing this, including local in-Grafana accounts with separate passwords, LDAP authentication, and HTTP Basic Authentication. For obvious reasons, we don't want people to have to obtain and manage separate Grafana accounts (it would be a pain in the rear for everyone). Since we're already using HTTP Basic Authentication to control some access to Grafana, reusing that for Grafana accounts makes a lot of sense; for instance, if you're accessing the server from the outside, it means that you don't have to first authenticate to Apache and then log in to Grafana if you want non-anonymous access.

But this hypothetical setup leaves us with a problem: how do you log in to Grafana when you're on our inside networks, where you won't be required to use HTTP Basic Authentication? It would be a terrible experience if you could only use your Grafana account if you weren't at work.

Before I set the server up and started experimenting, what I was hoping was that HTTP Basic Authentication was treated somewhat like cookies, in that once a browser was challenged to authenticate, it would then send the relevant Authorization header on all further accesses to the entire website. There are other areas of our web server that always require HTTP Basic Authentication, even from our internal networks, so if Basic Auth worked like cookies, you could go to one of them to force Basic Auth on, then go to a Grafana URL and the browser would automatically send an Authorization header and Apache would pass it to Grafana and Grafana would have you logged in to your account.

Unfortunately browsers do not treat HTTP Basic Authentication this way, which is not really surprising since RFC 7617 recommends a different approach in section 2.2. What RFC 7617 recommends and what I believe browsers do is that HTTP Basic Authentication is scoped to a URL path on the server. Browsers will only preemptively send the Authorization header to things in the same directory or under it; they won't send it to other, unrelated directories.

(If a browser gets a '401 Unauthorized' reply that asks for a realm that the browser knows the authorization for, it will automatically retry with that authorization. But then you're requiring HTTP Basic Authentication in general.)

The simplest, least hacky way out of this for us is to give up on the idea of anonymous access to Grafana, so that's what we've done. And that is why access to our Grafana URLs always requires HTTP Basic Authentication, however somewhat inconvenient and annoying it is. We have to always require HTTP Basic Authentication so that people can have and readily use frictionless Grafana accounts.

(As I mentioned in my entry on why we like Apache HTTP Basic Authentication, we're not willing to trust the authentication of requests from the Internet to Grafana itself. There are too many things that could go wrong even if Grafana was using, say, a LDAP backend. Fundamentally Grafana is not security software; it's written by people for whom security and authentication is secondary to dashboards and graphs.)

Sidebar: The theoretical hack around this

In theory, if browsers behave as RFC 7617 suggests, we can get around this with a hack. The most straightforward way is to have a web page at the root of the web server that we've specifically configured to require HTTP Basic Authentication; call this page /login.html. When you visit this page and get challenged, in theory your browser will decide that the scope of the authentication is the entire web server and thus send the Authorization header on all further requests to the server, including to Grafana URLs.

However I'm not sure this actually works in all common browsers (I haven't tested it) and it feels like a fragile and hard to explain thing. 'Go to this unrelated URL to log in to your Grafana account' just sounds wrong. 'You always have to use HTTP Basic Authentication' is at least a lot simpler.


Comments on this page:

Have you considered using mod_auth_tkt for single-sign-on? You can implement the backend authentication any way you like, as it only needs to issue tickets. The frontend sites can validate the tickets using the Apache module or custom code as appropriate to the application.

Written on 14 December 2018.
« Some new-to-me features in POSIX (or Single Unix Specification) Bourne shells
Link: Everything you should know about certificates and PKI but are too afraid to ask »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Dec 14 01:00:46 2018
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.