A brute force solution to nested access permissions in Apache

July 20, 2022

The simplest way to set up Grafana Loki is as a single server that handles both ingesting logs and querying them, much like Prometheus. Unlike Prometheus, Loki is a 'push' system instead of a 'pull' one, where clients send logs to Loki (via HTTP) instead of Loki reaching out to collect logs from them. This matters because of access permissions; it's one thing to allow a system to talk to Loki to send logs, but another thing to let it query everyone's logs. Loki explicitly punts on authentication and access control, leaving it up to you to put something in front of it to do this. Our solution for reverse proxying is Apache.

In a nice world, Loki's HTTP endpoints for log ingestion and log querying would be clearly separated in an URL hierarchy; you might have all push endpoints under /loki/push/ and all query related endpoints under /loki/query/, for example. In the current Loki HTTP API things are not so nicely divided. There is one HTTP endpoint for push, /loki/api/v1/push, but a bunch of other API endpoints both at the same level and above it (including not under /loki at all, for extra fun). This means that what we want to do in Apache is provide relatively open access to /loki/api/v1/push but then provide very restricted access to everything else from Loki's root URL downward, without having to inventory every URL under /loki/ and so on that we want to restrict.

There's probably a way to do this in Apache with the right set of directives, the right ordering and nesting of <Location> things, and so on. But it's at least not obvious to me how to do this, and while I was thinking about it I realized that there was a much simpler solution: you can have multiple reverse proxies to the same thing,, under separate URLs (unless what's talking to you absolutely insists on speaking to a fixed URL on your server, that must start at the root level).

So I have one reverse proxy for /loki/api/v1/push that talks to Loki (with that URL on Loki), and is relatively open. Then I have a completely separate top level URL, let's call it /loki-2/, that's a highly access restricted reverse proxy to all of Loki. Loki Promtail clients can use the push URL without any chance that I'll have made a mistake and given them access to anything else, while authorized external Grafana instances and other tools can connect to the /loki-2/ set of URLs.

(Because I'm controlling access to URLs, not filesystem directories, I'm using <Location> directives for this.)

This solution is brute force, but it works and it's simple to set up and to understand. Since they're completely separate URLs, it's entirely clear how the permissions work (and how they don't interact with each other). The one little thing is that I had to avoid a ProxyPassReverse directive, which I think doesn't work here. Since Loki is an API server, I don't think it's necessary; Loki will not normally be replying with redirects and so on.

(I'd still like to figure out how to do nested permissions here in Apache, where the parent has narrower or non-overlapping permissions than children, because I'm sure that someday I'm going to need to do it for real. But I only have so much energy for wrestling with Apache and doing Internet searches on various keyword combinations.)

Sidebar: Promtail and HTTP authentication

Promtail can use various HTTP authentication methods but in our environment all of them are awkward, and they require carefully guarding the authentication secret in Promtail across our entire fleet, because the secret would give access to all of Loki, not just the push side.

In our setup we could use HTTP authentication on the push URL to make it harder for random people to push random things into Loki. At the moment I don't think this is going to be a problem, so I'm skipping the extra complexity (and extra things that could break).

Written on 20 July 2022.
« We won't be sending systemd logs to Grafana Loki in JSON format
You can sensibly move or copy Prometheus's database with rsync »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Jul 20 22:28:41 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.