There was a latent misconfiguration, specifically a cable plugged someplace it shouldn’t have been, from a few months ago. Something called the spanning tree protocol kicked in and started trying to route all of our private network traffic to a public network over a link that was much too small and slow

via WP.com Downtime Summary — Blog — WordPress.com.

That’s rough.  The spanning-tree design/config is the kind of thing that creeps up on you after years of organic growth and never really having a dedicated “network guy”.     Its just bizarre enough that you spend a lot of time and effort digging into other obscure possibilities before you stumble on it.   Its one of the ‘gotchas’ of doing stuff in-house rather than clouding it up, particularly if you mix switch vendors.

Here’s an old map of an L2 topology I inherited and caused several spanning-tree outages learning the hard way.

Background on the curl flags, “-I” tells it to only request the headers not the full page. “-H” passes it request headers. The “Accept-Encoding” header tells the webserver that its OK by the client if it compresses the response. The “Host” header is… well, the host header. If you’re not sure what that means then get off my blog.

First off test the CDN:

[jim@brandt ~]$ curl -I -H "Accept-Encoding: gzip,deflate" -H "Host: www.mycorp.com" http://www.mycorp.com/foo/bar.html
HTTP/1.1 200 OK
Server: Apache/2.2.8 (Unix)
Content-Type: text/html
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 43593
Date: Mon, 22 Feb 2010 15:28:58 GMT
Connection: keep-alive

Notice the “Content-Encoding: gzip” header in the response. That means you got a gzip’d response, compression is in effect.

Now skip the CDN and test origin directly:

[jim@brandt ~]$ curl -I -H "Accept-Encoding: gzip,deflate" -H "Host: www.mycorp.com" http://origin.mycorp.com/foo/bar.html
HTTP/1.1 200 OK
Date: Mon, 22 Feb 2010 15:12:32 GMT
Server: Apache/2.2.8 (Unix)
Content-Type: text/html

The lack of a “Content-Encoding” header means its *not* compressed.

I got a kick out of this post to the point where I made a long ass comment.

[root@sql2 ~]# df -P /
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/VolGroup00-LogVol00  81269424  66551212  10534464      87% /

Then check how much is reserved blocks for root:

[root@sql2 ~]# tune2fs -l /dev/mapper/VolGroup00-LogVol00 | egrep 'Block count|Reserved block count'
Block count:              20971520
Reserved block count:     1045937

Thats 5%, which is a pretty common default. Like a lot of old defaults its based on old hardware, we can get away with a lot less and I could use the space elsewhere (SSD is expensive). So lets tune it down to 1%:

[root@sql2 ~]# tune2fs -m 1 /dev/mapper/VolGroup00-LogVol00
tune2fs 1.39 (29-May-2006)
Setting reserved blocks percentage to 1% (209715 blocks)

Verify that:

[root@sql2 ~]# tune2fs -l /dev/mapper/VolGroup00-LogVol00 | egrep 'Block count|Reserved block count'
Block count:              20971520
Reserved block count:     209715

Looks good, lets see how much space we got:

[root@sql2 ~]# df -P /
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/VolGroup00-LogVol00  81269424  66551756  13878808      83% /

Sweet, touch over 3GB

So yea… now I gotta pay like $800 just to proceed with the demo. Wonder if I can get it working on Vista or XP.

click for the legible size

This is what happens when developers configure servers.

5:19 is lol

source: JONNO dot com – Remember when the internet was amazing?.

Getting Support
  The tired and thirsty prospector threw himself down at the edge of the
  watering hole and started to drink. But then he looked around and saw
  skulls and bones everywhere. "Uh-oh," he thought. "This watering hole
  is reserved for skeletons." --Jack Handey

Ganglia 3.1.x Installation and Configuration – ganglia.