2022-08-12
My adventure with URLs in a Grafana that's behind a reverse proxy
I was oblique in yesterday's entry,
but today I'm going to talk about the concrete issue I'm seeing
because it makes a good illustration of how modern web environments
can be rather complicated. We run Grafana behind a reverse proxy as
part of a website, with all of Grafana under the /grafana/ path.
One of the things you can add to a Grafana dashboard is links,
either to other dashboards or to URLs. I want all of our dashboards
to have a link to the front page of our overall metrics site. The
obvious way to configure this is to tell Grafana that you want a
link to '/
', which as a raw link in HTML is an absolute path to
the root of the current web server in the current scheme.
When I actually do this, the link is actually rendered (in the
resulting HTML) as a link to '/grafana/', which is the root of the
Grafana portion of the website. Grafana is partially configured so
that it knows what this is, in that on the one hand it knows what
the web server's root URL for it is, but on the other hand its own
post-proxy root is '/' (in Apache terminology, we do a ProxyPass
of '/grafana/' to 'localhost:3000/'). This happens in both Firefox
and Chrome, and I've used Firefox's developer tools to verify that
the 'href
' of the link in the HTML is '/grafana/' (as opposed
to, eg, the link getting rewritten by Javascript on the fly when
you hover or click on it).
Grafana dashboards are not served as straight HTML at a given URL. Instead they are created through copious application of Javascript; the HTML you get served initially is just a vague starting point to load the Javascript. This makes it quite difficult to see what the source of the '/grafana/' link is. The information about the link could be sent from the Grafana server to the in-browser Javascript as either HTML or JSON, and it might then be rewritten by the Javascript (from either form of the information). If it's rewritten by Javascript, this could be a general rewriting of URLs that's necessary to make other URLs in the dashboard work; the Grafana server could be generating all URLs using its own root of '/' and then counting on the Javascript to fix all of them up.
(Alternately, the Grafana server could simply have decided (perhaps by accident) that all 'absolute' URLs that you provide are relative to its root, and some combination of the backend server and the frontend Javascript will rewrite them all for you.)
What I take from all of this is that a modern web application is a complicated thing and putting it behind a reverse proxy makes it more so, at least if it's sharing your web server with anything else. Of course, neither of these two things are exactly news. Now that I know a little bit more about how much 'rehydration' Grafana does to render dashboards, I'm a bit more amazed at how seamlessly it works behind our Apache reverse proxy.
PS: Configuring the link value in Grafana to be 'https:/' defeats
whatever rewriting is going on. The HTML winds up with that literal
text as the 'href
' value, and then the pragmatics of how browsers
interpret this take over.