Stefan Caunter

Compile a kernel with

device carp

in the config.

add the following to /etc/rc.conf, adjusting for IP address
This creates a Master CARP interface.

defaultrouter=”192.168.0.1″
hostname=”direct1″
ifconfig_em0=”inet 192.168.1.8 netmask 255.255.255.0″
cloned_interfaces=”carp0″
ifconfig_carp0=”vhid 1 advskew 10 advbase 1 pass mypass 192.168.1.7/24″

The second box needs this for its carp config – it will run as backup.

defaultrouter=”192.168.0.1″
hostname=”direct2″
ifconfig_em0=”inet 192.168.1.9 netmask 255.255.255.0″
cloned_interfaces=”carp0″
ifconfig_carp0=”vhid 1 pass mypass advbase 1 advskew 0 192.168.1.7/24″

If you want to start varnishd with
-s file,/usr/cache/topscache,30720000000
use
dd if=/dev/zero count=60000000 of=/usr/cache/topscache

This works well for a server with 32GB ram.

Twitter does this, and I’ve implemented it too, we just modify vcl_error as follows:

sub vcl_error {
set obj.http.Location = “http://oops.scaleengine.com/”;
set obj.status = 302;
return(deliver);
}

Take out the 503 guru meditation stuff and simply set a redirect to another, friendly domain.

This does not fix “cannot reach the backend” configuration errors, and does not fix poorly configured backends, but if your backends are underpowered and overwhelmed by a traffic spike, you can at least send people to a nice fail page.

This is different, and useful website speed test, especially because of the external element count and total page load time. Enjoy.

Being able to see how your load balance algorithm works in real time is a huge advantage for IIS7 – the performance monitor lets us see real time incoming http request, and real time ASP queue size. Correlating this view with a scrolling filtered view of varnishlog checking for Backend health and reuse lets us confirm that we have an optimized load balance algorithm.

We need two perfmon msc files set up and saved; one will view HTTP arrival rates across the farm (HTTP Service Request Queues / Arrival Rate). I like the histogram display for my five servers. When a box comes out of the pool, we see its “bar” dip nice and low, to give it a break from traffic until it recovers health. The other will view ASP.NET v2.0.50727 Requests Current for each IIS7 w3wp.exe in histogram view. If this “bar” starts going up, we look for correlation to varnish health checks.

The backend probe, that ultimately decides whether our server gets traffic or not, needs to check if ASP can build a page, but not be too expensive. Don’t ask for a heavy page, but get your developer to build a simple aspx page, and set a time and frequency that takes the server out if it cannot answer acceptably.

I use 200ms for my check.

.probe = {
.url = “/howyoudoin/”;
.timeout = 200 ms;
.interval = 5s;
.window = 5;
.threshold = 1;
}

This url is served from the default web site, on the IP address, and actually triggers the real ASP page (defined in web.config) that runs in our high traffic webapp. You need to tune for aggressive .timeout, but still keep the box in the pool for most traffic. No point in setting this so that varnish cannot find healthy backends.

Once you have directors set up with backend probes, as above, view perfmon on your IIS console, and set up shell windows with filtered scrolling varnishlog data on your varnish server (or the same one if you have several in front of your webapp).

varnishlog|grep Backend_health

This scrolling display shows which IIS7 servers are seen to be healthy and which are not.

varnishlog|grep BackendReuse

This scrolling display (it will go fast if you get a lot of traffic) shows which IIS7 server is preferred by your client director. You set up in a client director a preferred backend, and it will show as preferred when it is healthy. If its ASP queue gets too large, it will fail health check probe, be marked sick, and traffic will start to flow to the next preferred server defined in the client director. The incoming http traffic display to the IIS7 backends will reflect this behaviour.

We get confirmation of our settings by observing ASP queues build, the IIS7 server fail the health check, the load balance algorithm adjust to a healthy server, and the incoming http traffic sent by varnish move to another IIS7 server.