In previous posts we talked about Nginx, that excellent lightweight web server made in Russia. Nginx is powering the Visualizeus servers since day one.
The other day I was needing some feature on Nginx. In Visualizeus there’s a file cache for some things, like the feeds and so on. Till now if the requested url has a cached version, then Nginx returned it with no more questions. If not, then Nginx transfer the request to the backend, which generates a new cached version in disk for the following requests for that resource. Pretty normal.
But Nginx wasn’t aware of the freshness of the file at all, it only understands that easy logic: file exists, then return it. So to avoid stale cache it’s obvious we need to delete from time to time that cached resources. When to delete it depends on the type of object we’re caching, of course. For example, the feeds are deleted each time a new image appears in that feed, and so on. But it was a little bit cumbersome for some objects, like the image pages, where you don’t have a clear condition that makes the page stale. It needs some cron tasks to run periodically and find files older than X, and those tasks eats a lot of resources specially when we’re talking of thousands and thousands of files and folders. So it was clear a better solution was needed here.
And that’s why I was missing a feature on Nginx, a “return this file only if it’s newer than…” flag. And lucky me, there was a patch for that developed by DeepFryed (thanks!), in the 3rd Party section of the Nginx wiki. Bad news were that I couldn’t get the patch working for the 0.7.61 version (latest stable), because some lines gave errors where trying to apply it. I don’t know what version was the patch for, but seems clear the current Nginx source code have had changed since then. Fixing that was a minor thing, it only took me a couple of minutes to realize the changes browsing the source code. But it wasn’t be that easy!
The patch, although applied and compiled perfectly, didn’t work as expected. In fact, it didn’t work at all. After some debugging with the --with-debug flag turned on, I realized the new -M flag introduced was always returning False, like it wasn’t even evaluated. It’s worth mentioning I haven’t take a look to the Nginx source code before, and while I could figured out what was the patch doing, looking beyond that, was much more work that I was willing to assume for this task.
Buut… one can’t resist to that kind of things, and curiosity finally won the battle, so after a morning diving into source code and debug logs, I finally managed to understand 1) how it works (barely); 2) what was the patch doing and adding; and 3) what was failing. It was a loong morning though, but it ended with me so happy for figuring this out ;)
By the way, I improved it a little bit, adding support for variables and not only numerical values, which was something I needed for Visualizeus. E.g.:
set $cacheTTL 1800;
if (-M $document_root/cache/file.html $cacheTTL) {
…
}
So you can download the revised version of the DeepFryed patch, adapted and running for the new stable version of Nginx and also improved a bit, here. It’s also linked in the Nginx wiki.
Compártelo
August 21st, 2009 | Development, English |