Managing varnish content with MULE

A brief post on event driven management of Varnish Cache content. This is something that Richard Guest and I have been experimenting with off and on for a while. The concepts are similar to those covered by David Harrigan in a couple of recent blog posts (part1, part2). David’s posts explain the concepts well and are worth a read. There are a few differences with our implementation that deserve a mention.

Banning Versus Purging

We want to let Varnish serve stale content when it can’t reach the origin servers. We also need to be able to use wild cards in the ban expressions. This rules out HTTP purging for us - with an HTTP purge there is no option to use wild cards. Also, with a successful purge the content is gone from the cache whether the origin server is reachable or not - there is no option for Varnish to serve stale content.

The ESB

We use MULE for our ESB and ActiveMQ for the messaging provider. Not an important difference, but useful to know that you can manage content with a variety of means.

Less Code

I like Java most of the time, but it can be frustratingly verbose to use with HTTP. I don’t want to have to write code just to add a string to the http method (e.g., BAN). The work-around is to change the Varnish config to use an HTTP method name that is available in Java.

The Implementation

We’re going to ban content with wild cards. We have no other need for the HTTP DELETE method so we use that to implement BAN in our Varnish config. We allowed the use of an additional HTTP header (X-BanExpression) to avoid having to deal with URL encoding for regular expressions.

if (req.request == "DELETE") {
	if (!client.ip ~ purge) {
		error 405 "Not allowed.";
	}
	if (req.http.X-BanExpression) {
		ban(req.http.X-BanExpression);
	}
	else {
		ban_url(req.url);
	}
	# Throw a synthetic page so the
	# request won't go to the backend.
	error 200 "Ban added";
}

This means we don’t need any custom implementations for MULE HTTP outbound endpoints:

<http:outbound-endpoint host="localhost" port="${quakevarnish.varnish.port}" method="DELETE">
    <property key="X-BanExpression" value="obj.http.X-Url ~ /quakes/*"/>
</http:outbound-endpoint>

The MULE project is available on Github.

Other Considerations

We run multiple origin servers for Varnish. This means we would like redundancy in our messaging as well. The best way to achieve redundancy depends on the messaging provider. If possible, it makes sense to run a messaging provider with each origin server and use fail over in the ESB client on each Varnish server. I’ve covered fail over options with MULE and ActiveMQ previously.

Conclusions

This is a potentially powerful solution - the option for efficient caching with new content being available as soon as possible. The flexibility in implementation is also a testament to Varnish and their focus on doing one thing and doing it really well.