19 Jul 2010 @ 11:17 AM 

Being able to see how your load balance algorithm works in real time is a huge advantage for IIS7 – the performance monitor lets us see real time incoming http request, and real time ASP queue size. Correlating this view with a scrolling filtered view of varnishlog checking for Backend health and reuse lets us confirm that we have an optimized load balance algorithm.

We need two perfmon msc files set up and saved; one will view HTTP arrival rates across the farm (HTTP Service Request Queues / Arrival Rate). I like the histogram display for my five servers. When a box comes out of the pool, we see its “bar” dip nice and low, to give it a break from traffic until it recovers health. The other will view ASP.NET v2.0.50727 Requests Current for each IIS7 w3wp.exe in histogram view. If this “bar” starts going up, we look for correlation to varnish health checks.

The backend probe, that ultimately decides whether our server gets traffic or not, needs to check if ASP can build a page, but not be too expensive. Don’t ask for a heavy page, but get your developer to build a simple aspx page, and set a time and frequency that takes the server out if it cannot answer acceptably.

I use 200ms for my check.

.probe = {
.url = “/howyoudoin/”;
.timeout = 200 ms;
.interval = 5s;
.window = 5;
.threshold = 1;
}

This url is served from the default web site, on the IP address, and actually triggers the real ASP page (defined in web.config) that runs in our high traffic webapp. You need to tune for aggressive .timeout, but still keep the box in the pool for most traffic. No point in setting this so that varnish cannot find healthy backends.

Once you have directors set up with backend probes, as above, view perfmon on your IIS console, and set up shell windows with filtered scrolling varnishlog data on your varnish server (or the same one if you have several in front of your webapp).

varnishlog|grep Backend_health

This scrolling display shows which IIS7 servers are seen to be healthy and which are not.

varnishlog|grep BackendReuse

This scrolling display (it will go fast if you get a lot of traffic) shows which IIS7 server is preferred by your client director. You set up in a client director a preferred backend, and it will show as preferred when it is healthy. If its ASP queue gets too large, it will fail health check probe, be marked sick, and traffic will start to flow to the next preferred server defined in the client director. The incoming http traffic display to the IIS7 backends will reflect this behaviour.

We get confirmation of our settings by observing ASP queues build, the IIS7 server fail the health check, the load balance algorithm adjust to a healthy server, and the incoming http traffic sent by varnish move to another IIS7 server.

Posted By: admin
Last Edit: 19 Jul 2010 @ 11:36 AM

EmailPermalinkComments (0)
Tags
Categories: IIS7, Varnish Cache
 07 Jun 2010 @ 7:55 PM 

Not much more to say. It defaults to regex. Change it to wildcard. Most simple rules can be done with wildcard. The regex engine is, shall we say, greedy? Same issue with the built in ASP.NET library. Avoid heavy use of regular expressions on IIS web apps, they really slow things down on high traffic sites.

Posted By: admin
Last Edit: 07 Jun 2010 @ 07:55 PM

EmailPermalinkComments (0)
Tags
Categories: IIS7
 06 May 2010 @ 8:15 PM 

Noticed today that the webapp struggles as it gets to within 50 meg of the virtual memory limit threshold for recycling the app pool. This happens no matter what the threshold is set to. 2 gig, 1.7, 1.6. Same thing. We speculate it is going into GC (garbage collection) mode, and CPU gets high. It hits the limit, recycles, and is happy again for an hour. Since it was recycling around once an hour, I thought I’d remove the memory limit, and just recycle once an hour. Bad Thing. With no memory limit, URL Rewrite 2.0 kills the CPU. We were pinned at 100% in 45 minutes.

Conclusion: hard won tuning settings are always worth questioning, but if they run reliably and keep you up, it’s unlikely that disabling a setting will do anything but expose other problems.

Probably worth looking at the registry settings for relations between memory and CPU for rewrite.dll but the solution is likely to be less use of the dll overall, combined with memory threshold recycling.

Posted By: admin
Last Edit: 06 May 2010 @ 09:20 PM

EmailPermalinkComments (0)
Tags
Categories: IIS7
 05 May 2010 @ 7:43 AM 

working example vcl for a system with multiple iis7 backends
if the webapp[s] run slowly at times you will need several backends to improve client’s chances of hitting a fast server first.
we always bail out to a long wait for an answer so we don’t show 503s.
the patient and impatient backends are the same webservers, we just poll impatiently and take out slow ones
confirm restarts are working by checking in varnishlog. make sure you have hosts file entries so the hostname entry resolves for the polling check.
client should either get a cache hit if we have the object, or a fast back end on requests that we don’t send directly to the patient director, and if it’s a specific url where we need to wait for a database or some query, we are happy to wait, by using return(lookup), so the object enters the cache for next time.
I have logic for asp sessions and facebook logic too (next time)

# patient backends, no polling, long timeout
backend search1 {
.host = “192.168.0.1”;
.port = “80”;
.first_byte_timeout = 60s;
.between_bytes_timeout = 10s;
}
backend search2 {
.host = “192.168.0.2”;
.port = “80”;
.first_byte_timeout = 90s;
.between_bytes_timeout = 90s;
}
backend my1 {
.host = “web1.example.com”;
.port = “80”;
.first_byte_timeout = 100ms;
.between_bytes_timeout = 500ms;
.probe = {
.url = “/checking”;
.timeout = 100 ms;
.interval = 5s;
.window = 5;
.threshold = 3;
}
}
backend my2 {
.host = “web2.example.com”;
.port = “80”;
.first_byte_timeout = 100ms;
.between_bytes_timeout = 500ms;
.probe = {
.url = “/checking”;
.timeout = 100 ms;
.interval = 5s;
.window = 5;
.threshold = 3;
}
}
# patient director, patience is a property of the backend definition
director patient random {
{ .backend = search1; .weight = 3; }
{ .backend = search2; .weight = 3; }
}
# impatient director, impatience is a property of the backend polling
# you need several backends for best effect, we only show two for example
director impatient random {
{ .backend = my1; .weight = 3; }
{ .backend = my2; .weight = 3; }
}

sub vcl_recv {
# omitting commonly available settings
if (req.restarts == 0) {
set req.backend = impatient;
} else if (req.restarts == 1) {
set req.backend = patient;
} else if (req.restarts == 2) {
set req.backend = patient;
} else if (req.restarts == 3) {
set req.backend = patient;
} else if (req.restarts == 4) {
set req.backend = search1;
}
if (req.url ~ “/slowthing”) {
set req.backend = patient;
return(lookup);
}
if (req.url ~ “/slowthing2”) {
set req.backend = patient;
return(lookup);
}
if (req.url ~ “/control-update”) {
set req.backend = impatient;
return(pipe);
}

}
sub vcl_fetch {
# keep trying if iis does not answer, use vcl_recv restart logic
if (beresp.status != 200 && beresp.status != 403 && beresp.status != 404 && beresp.status != 301 && beresp.status != 302) {
restart;
}
# set things here on objects we send back to clients
if (req.url ~ “^/$” && req.http.host ~ “www.example.com”){
set beresp.ttl = 300s;
set beresp.http.cache-control = “max-age = 300”;
set beresp.http.X-MyCacheReason = “MyHomePage”;
set beresp.grace = 1d;
return(deliver);
}
return(deliver);
}
sub vcl_error {
# avoid the dreaded 503, increase restart number as necessary
if (obj.status == 503 && req.restarts < 5) {
restart;
}
}

Posted By: admin
Last Edit: 06 May 2010 @ 08:07 PM

EmailPermalinkComments (0)
Tags
Categories: IIS7, Varnish Cache
 22 Apr 2010 @ 6:56 PM 

Dearth of knowledge here, thought I’d share my experiences.
Getting a busy, complex ASP.NET app to show snappy, fast page response to users is challenging, especially if potential bottlenecks cause intermittent slowness. Varnish is ideal for this, but requires careful adjustment and tuning for best effect. In particular, avoiding 503 errors caused by IIS hangs is most important. Delivering a page eventually is preferable to a quick timeout and a user confronted with the varnish 503 page. With determination, we can eliminate that possibility and speed things up nicely for IIS administrators.
First, varnish must health check the IIS backends. You need a version of varnish that will do this. Second, varnish must retry if the backend does not answer in a timely manner. Same need, post 2.04 version necessary. Third, varnish must wait in certain cases if we have to wait for the backend. Small bit of magic to do this.
I guess the overriding issue here is that ASP.NET apps, or any app for that matter will have problems, and our responsibility as sysadmins is to make them run as well as possible. Externally, we want to be able to send clients to a fast, healthy server, and we want to serve cached content to the extent possible within a demanding freshness use case.
Back end webservers should be checked with requests that time out quickly, and need to be set to quickly take a slow box out of the pool. Check a simple ASP page, not a flat file, and keep the timing reasonable for your app and scale needs. If you know your app lags at times (for whatever reason), you should be able to tune this check so that a server with a lot of requests in its w3wp.exe application pool queue will get marked sick until it clears its problem. Add these backends into your “impatient” director. Serve most requests with this director.
You cannot avoid a slow server altogether this way. IIS might be heading to a slow time but not be marked sick, and it will get requests that will sit there. We have to avoid the dreaded 503 error at all costs, and with a fast timeout you will get them unless you restart in vcl_error. Instead of giving up, varnish tries again. You’ll get a healthy server, and (almost) never show 503 to the client, especially once the varnish cache heats up.
My technique to deal with “have to wait” scenarios (login, POST, search) is to create a “patient” pool. Same servers, but long timeout on the first_bytes_check. Send these type of requests to this director. Instead of a 503, we patiently wait for a response. Result: fast response on most pages, and correct response when patience is a normal part of user expectations.

Posted By: admin
Last Edit: 22 Apr 2010 @ 06:56 PM

EmailPermalinkComments (0)
Tags
Categories: IIS7, Varnish Cache

 Last 50 Posts
 Back
Change Theme...
  • Users » 328
  • Posts/Pages » 47
  • Comments » 1
Change Theme...
  • VoidVoid « Default
  • LifeLife
  • EarthEarth
  • WindWind
  • WaterWater
  • FireFire
  • LightLight

Music



    No Child Pages.