Stefan Caunter

working example vcl for a system with multiple iis7 backends
if the webapp[s] run slowly at times you will need several backends to improve client’s chances of hitting a fast server first.
we always bail out to a long wait for an answer so we don’t show 503s.
the patient and impatient backends are the same webservers, we just poll impatiently and take out slow ones
confirm restarts are working by checking in varnishlog. make sure you have hosts file entries so the hostname entry resolves for the polling check.
client should either get a cache hit if we have the object, or a fast back end on requests that we don’t send directly to the patient director, and if it’s a specific url where we need to wait for a database or some query, we are happy to wait, by using return(lookup), so the object enters the cache for next time.
I have logic for asp sessions and facebook logic too (next time)

# patient backends, no polling, long timeout
backend search1 {
.host = “192.168.0.1”;
.port = “80”;
.first_byte_timeout = 60s;
.between_bytes_timeout = 10s;
}
backend search2 {
.host = “192.168.0.2”;
.port = “80”;
.first_byte_timeout = 90s;
.between_bytes_timeout = 90s;
}
backend my1 {
.host = “web1.example.com”;
.port = “80”;
.first_byte_timeout = 100ms;
.between_bytes_timeout = 500ms;
.probe = {
.url = “/checking”;
.timeout = 100 ms;
.interval = 5s;
.window = 5;
.threshold = 3;
}
}
backend my2 {
.host = “web2.example.com”;
.port = “80”;
.first_byte_timeout = 100ms;
.between_bytes_timeout = 500ms;
.probe = {
.url = “/checking”;
.timeout = 100 ms;
.interval = 5s;
.window = 5;
.threshold = 3;
}
}
# patient director, patience is a property of the backend definition
director patient random {
{ .backend = search1; .weight = 3; }
{ .backend = search2; .weight = 3; }
}
# impatient director, impatience is a property of the backend polling
# you need several backends for best effect, we only show two for example
director impatient random {
{ .backend = my1; .weight = 3; }
{ .backend = my2; .weight = 3; }
}

sub vcl_recv {
# omitting commonly available settings
if (req.restarts == 0) {
set req.backend = impatient;
} else if (req.restarts == 1) {
set req.backend = patient;
} else if (req.restarts == 2) {
set req.backend = patient;
} else if (req.restarts == 3) {
set req.backend = patient;
} else if (req.restarts == 4) {
set req.backend = search1;
}
if (req.url ~ “/slowthing”) {
set req.backend = patient;
return(lookup);
}
if (req.url ~ “/slowthing2”) {
set req.backend = patient;
return(lookup);
}
if (req.url ~ “/control-update”) {
set req.backend = impatient;
return(pipe);
}

}
sub vcl_fetch {
# keep trying if iis does not answer, use vcl_recv restart logic
if (beresp.status != 200 && beresp.status != 403 && beresp.status != 404 && beresp.status != 301 && beresp.status != 302) {
restart;
}
# set things here on objects we send back to clients
if (req.url ~ “^/$” && req.http.host ~ “www.example.com”){
set beresp.ttl = 300s;
set beresp.http.cache-control = “max-age = 300”;
set beresp.http.X-MyCacheReason = “MyHomePage”;
set beresp.grace = 1d;
return(deliver);
}
return(deliver);
}
sub vcl_error {
# avoid the dreaded 503, increase restart number as necessary
if (obj.status == 503 && req.restarts < 5) {
restart;
}
}

Dearth of knowledge here, thought I’d share my experiences.
Getting a busy, complex ASP.NET app to show snappy, fast page response to users is challenging, especially if potential bottlenecks cause intermittent slowness. Varnish is ideal for this, but requires careful adjustment and tuning for best effect. In particular, avoiding 503 errors caused by IIS hangs is most important. Delivering a page eventually is preferable to a quick timeout and a user confronted with the varnish 503 page. With determination, we can eliminate that possibility and speed things up nicely for IIS administrators.
First, varnish must health check the IIS backends. You need a version of varnish that will do this. Second, varnish must retry if the backend does not answer in a timely manner. Same need, post 2.04 version necessary. Third, varnish must wait in certain cases if we have to wait for the backend. Small bit of magic to do this.
I guess the overriding issue here is that ASP.NET apps, or any app for that matter will have problems, and our responsibility as sysadmins is to make them run as well as possible. Externally, we want to be able to send clients to a fast, healthy server, and we want to serve cached content to the extent possible within a demanding freshness use case.
Back end webservers should be checked with requests that time out quickly, and need to be set to quickly take a slow box out of the pool. Check a simple ASP page, not a flat file, and keep the timing reasonable for your app and scale needs. If you know your app lags at times (for whatever reason), you should be able to tune this check so that a server with a lot of requests in its w3wp.exe application pool queue will get marked sick until it clears its problem. Add these backends into your “impatient” director. Serve most requests with this director.
You cannot avoid a slow server altogether this way. IIS might be heading to a slow time but not be marked sick, and it will get requests that will sit there. We have to avoid the dreaded 503 error at all costs, and with a fast timeout you will get them unless you restart in vcl_error. Instead of giving up, varnish tries again. You’ll get a healthy server, and (almost) never show 503 to the client, especially once the varnish cache heats up.
My technique to deal with “have to wait” scenarios (login, POST, search) is to create a “patient” pool. Same servers, but long timeout on the first_bytes_check. Send these type of requests to this director. Instead of a 503, we patiently wait for a response. Result: fast response on most pages, and correct response when patience is a normal part of user expectations.

Gets ground a little fine for the automatic machine, but absolutely lovely dark French roast. Super place!

Password Security Misconceptions
The anatomy of a DoS attack

When you uninstall 1.1 version, it deletes all of your server level redirects. Not fun if you have a lot of them. Here is an alternative to recreating them all in the GUI.
First, open c:\windows\system32\inetsrv\config\applicationhost.config and copy the entire <rewrite> container to another file.
The container starts with <validation />
<rewrite>
<globalRules>
And you will put it back under <validation /> after the upgrade nukes everything.
Grab everything down to </globalRules></rewrite> and save it, I used rewrite.txt
Now you are ready to uninstall 1.1. This is done in programs and features in control panel. You need to reboot after the uninstall.
When you are back up, get URW2.0 from iis.net.
Go back to your AHC file, open it, and scroll to the validation tag. No rewrite rules. No problem. Paste in your saved rules from rewrite.txt and save out.
You need to restart the web server. Check that your app pools are all alive.
If you get errors, you likely have duplicate hostname entries in the ruleset. Version 1.1 tolerated this, version 2.0 does not, and this will break your web server. Check the line in the error, and go back to AHC and remove the duplicate entries.
URL Rewriter 2.0 is an improvement over the RC version. High traffic servers still show memory leak. There is a registry entry that may help this, at the expense of CPU. Balancing this is something we haven’t tried. Solution is to recycle the app pool (32 bit) at a Private Bytes threshold that keeps it from getting unhealthy – 1.5GB for us.