church of default

How to tune MySQL’s sort_buffer_size at Xaprb.

Baron’s down with the Church.

High Scalability – High Scalability – MocoSpace Architecture – 3 Billion Mobile Page Views a Month.

I love these posts.   Its super handy to see what others in what I’ve taken to calling the “middle class” of websites are doing.

Couple thoughts.  Their pageview to unique visitor ratio is waaaaay wide. I guess thats a byproduct of small micro-pages for mobile devices and the difficulty in user tracking on mobile.   I like that they clearly weren’t afraid to build where they thought they should and buy where they thought they should.  A mix of free/open-source stuff and of ‘enterprise’-vendor stuff shows a clear results-over-religion vibe.   They’ve also got the developers-pushing-live/you-break-it-you-own-it thing figured out, which is key to how they get away with 1 sysadmin with all that workload.

Came across this in my twitter stream:

Which in and of itself is just amusing, but what got me was that when I went to the AWS Status page there was no mention of it. It had already been 15 minutes since the tweet, but nothing. So I did a twitter search, and found this one:

Which got posted nearly 45 minutes before amazon updated their status page to say:

11:19 AM PDT We’re investigating reports of timeouts in California.

Now I don’t think downtime, especially infrequent and geographically limited downtime, is any sort of “see thats why you shouldn’t use teh cloud!” argument. If anything odds are people who use cloudfront have more downtime in their webapp than amazon ever will. However 45 minute response times mean 45 minutes of wasted troubleshooting time, multiplied by the number of engineers looking into it. Its that feeling of blind, uninformed, in-the-darkness while your cloud provider gets around to acknowledging things that worries me.

Here’s the press release if you’re into blahblah.

And here’s the Cortex Developer “community site” they setup.

Short version: now you can now do all the stuff the command line and web interface let you do via a REST API. Create a LUN, map it to a host, take a snapshot, check status, performance metrics, etc.

Thats all cool, and will probably lead to lots of great cross-vendor integration and automation opportunities for larger shops, but I’m not a larger shop. So why do I think its cool? Because for sysadmin’s REST API’s are *awesome*. They are so dang simple that even a complete developmentally-disabled maroon like me can figure them out well enough to wrap them in a one-liner curl script. I learned this at Sling, when I was able to add a half dozen key performance metric monitors to our zabbix setup in a single afternoon. Its hard to add *anything* to Zabbix in an afternoon.

So the two things I wanna do now are a nagios check and a ganglia plugin.

The coolest part here though is that Xiotech seems to be on the road to “getting it” that SAN’s and high-performance storage don’t need to be wizards work of proprietary enterprise tools, professional service engagements, and a culture of “lets have lunch with our buddies the sales team” lock-in.

Digg kicked off this particular back and forth when they posted “Saying Yes to NoSQL; Going Steady with Cassandra”

That led to a lot of … community analysis… the best of which I’ve seen was by Mark Callaghan here: Index only

Joe Stump, a former architect at Digg came back with a defense of their decision in NoSQL vs. RDBMS: Let the flames begin! – stu.mp.

Push comes to shove, like all engineering decisions, its a case of everything is a tradeoff so pick your priorities and go.

Mark’s analysis of how they could have optimized their mysql setup is deep mysql-kung-fu.   Its unrealistic to expect your entire dev team to be that good, so you either need a couple of devs who are, or a DBA.   Joe makes that point in his post, but frankly I find his numbers a touch overstated.    For instance his “$500K in servers and dbas” looks much more like $150K to me, if that.

Most of all I think the point Mark makes in his first paragraph is the key to this whole MySQL vs. NoSQL bruhaha.   In SQL its easy to write a bad query.  In something like Cassandra, you have to write much more of the code and logic yourself, so the responsibility for how it performs more clearly falls on the developer at the moment of authoring.   So have your devs spend more time on their design, or have a dba spend more time optimizing queries, solve for your values of available manpower/budget.

Also, I’m curious to see if a class of  NoDBA’s start to emerge.   I’ve been following Cassandra in lots of places and the operational workload of running a Cassandra cluster looks very non-trivial.  I can’t say for sure if its more or less than MySQL, but its certainly substantial and you’re going to have to factor operational-manpower into it just like you would MySQL.