wordpress.com outage was spanning-tree

February 23rd, 2010

There was a latent misconfiguration, specifically a cable plugged someplace it shouldn’t have been, from a few months ago. Something called the spanning tree protocol kicked in and started trying to route all of our private network traffic to a public network over a link that was much too small and slow

via WP.com Downtime Summary — Blog — WordPress.com.

That’s rough.  The spanning-tree design/config is the kind of thing that creeps up on you after years of organic growth and never really having a dedicated “network guy”.     Its just bizarre enough that you spend a lot of time and effort digging into other obscure possibilities before you stumble on it.   Its one of the ‘gotchas’ of doing stuff in-house rather than clouding it up, particularly if you mix switch vendors.

Here’s an old map of an L2 topology I inherited and caused several spanning-tree outages learning the hard way.

How to test if a webserver is using compression

February 22nd, 2010

Was in a situation where I wanted to verify that my cdn is compressing for me, but that my origin is not.

Background on the curl flags, “-I” tells it to only request the headers not the full page. “-H” passes it request headers. The “Accept-Encoding” header tells the webserver that its OK by the client if it compresses the response. The “Host” header is… well, the host header. If you’re not sure what that means then get off my blog.

First off test the CDN:

[jim@brandt ~]$ curl -I -H "Accept-Encoding: gzip,deflate" -H "Host: www.mycorp.com" http://www.mycorp.com/foo/bar.html
HTTP/1.1 200 OK
Server: Apache/2.2.8 (Unix)
Content-Type: text/html
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 43593
Date: Mon, 22 Feb 2010 15:28:58 GMT
Connection: keep-alive

Notice the “Content-Encoding: gzip” header in the response. That means you got a gzip’d response, compression is in effect.

Now skip the CDN and test origin directly:

[jim@brandt ~]$ curl -I -H "Accept-Encoding: gzip,deflate" -H "Host: www.mycorp.com" http://origin.mycorp.com/foo/bar.html
HTTP/1.1 200 OK
Date: Mon, 22 Feb 2010 15:12:32 GMT
Server: Apache/2.2.8 (Unix)
Content-Type: text/html

The lack of a “Content-Encoding” header means its *not* compressed.

To Puppet or to Chef, that is the Question | IverCore

February 6th, 2010

via To Puppet or to Chef, that is the Question | IverCore.

I got a kick out of this post to the point where I made a long ass comment.

How to reclaim root reserved space on ext3

January 26th, 2010

First off look at what you’ve got now:

[root@sql2 ~]# df -P /
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/VolGroup00-LogVol00  81269424  66551212  10534464      87% /

Then check how much is reserved blocks for root:

[root@sql2 ~]# tune2fs -l /dev/mapper/VolGroup00-LogVol00 | egrep 'Block count|Reserved block count'
Block count:              20971520
Reserved block count:     1045937

Thats 5%, which is a pretty common default. Like a lot of old defaults its based on old hardware, we can get away with a lot less and I could use the space elsewhere (SSD is expensive). So lets tune it down to 1%:

[root@sql2 ~]# tune2fs -m 1 /dev/mapper/VolGroup00-LogVol00
tune2fs 1.39 (29-May-2006)
Setting reserved blocks percentage to 1% (209715 blocks)

Verify that:

[root@sql2 ~]# tune2fs -l /dev/mapper/VolGroup00-LogVol00 | egrep 'Block count|Reserved block count'
Block count:              20971520
Reserved block count:     209715

Looks good, lets see how much space we got:

[root@sql2 ~]# df -P /
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/VolGroup00-LogVol00  81269424  66551756  13878808      83% /

Sweet, touch over 3GB

RHEV-M really won’t install on a 64bit Windows server

January 24th, 2010

They do say “Microsoft Windows Server 2003 (or R2) 32-bit edition” in the docs, but since all I have is a 64bit vm I tried it anyway.

rhevm64biterror

So yea… now I gotta pay like $800 just to proceed with the demo. Wonder if I can get it working on Vista or XP.