church of default » SAN

more adventures in multipathd

jim — Wed, 30 Sep 2009 04:59:12 +0000

Found this handy tonight, multipathd has an interactive shell you can use by running “multipathd -k” which is a flag not documented in the man page. Actually, no flags are documented in the man page. You can get a list of the command options by typing help (or generally any command it can’t figure out what to do with). That lead me to the “fail path” option which is what I’m trying to do here, test individual path failures without having to go through the 5 step java gui dance with my fibre switches.

[root@nas ~]# multipathd -k
multipathd> show paths
hcil dev dev_t pri dm_st chk_st next_check
0:2:0:0 sda 8:0 1 [undef] [ready] [orphan]
1:0:0:1 sdb 8:16 1 [active][ready] XXXXXXXX.. 16/20
2:0:0:1 sdd 8:48 1 [active][ready] XXXXX….. 11/20

yes as you can see I clearly don’t have my blacklist setup right. Now to fail a path its “fail path $path” where $path is… again, no documentation. Turns out what they want is the “dev” column (second one)

multipathd> fail path sdd
ok
multipathd> show paths
hcil dev dev_t pri dm_st chk_st next_check
0:2:0:0 sda 8:0 1 [undef] [ready] [orphan]
1:0:0:1 sdb 8:16 1 [active][ready] XXXXX….. 10/20
2:0:0:1 sdd 8:48 1 [failed][faulty] X……… 3/20

Interesting to note, multipathd seems pretty good at its job, as the path is re-enabled within seconds when the checker notices its fine.

Sep 30 00:51:35 nas multipathd: mpath1: fail path 8:48 (operator)
Sep 30 00:51:35 nas kernel: device-mapper: multipath: Failing path 8:48.
Sep 30 00:51:35 nas multipathd: dm-2: add map (uevent)
Sep 30 00:51:35 nas multipathd: dm-2: devmap already registered
Sep 30 00:51:35 nas multipathd: 8:48: mark as failed
Sep 30 00:51:35 nas multipathd: mpath1: remaining active paths: 1
Sep 30 00:51:40 nas multipathd: sdd: readsector0 checker reports path is up
Sep 30 00:51:40 nas multipathd: 8:48: reinstated
Sep 30 00:51:40 nas multipathd: mpath1: remaining active paths: 2
Sep 30 00:51:40 nas multipathd: dm-2: add map (uevent)
Sep 30 00:51:40 nas multipathd: dm-2: devmap already registered

So the good news is failing one path and then the other worked nicely and the server didn’t freak out and remount the filesystem read-only.

that backblaze thing

jim — Thu, 10 Sep 2009 15:30:06 +0000

I’m pretty much the last guy on the internet to comment on this cuz the wordpress bookmarklet ate my post last week and I never noticed. Anyway, here’s the link:

Petabytes on a budget: How to build cheap cloud storage | Backblaze Blog.

When I first read it I thought “holy shit these guys are bordering on criminal malpractice” but remembering the age old rule “all engineering is tradeoffs” their setup makes more sense if you think about it. Basically, they must be handling data saftey at a higher network layer and not within the actual box. If you think about it that way, its really just a JBOD and there’s nothing special going on here. All the critical saftey flaws of an individual box are mitigated because you can lose the entire 45 disk box and not lose data. To be honest, I don’t even know why they bothered with raid6. The drives arent’ hot swappable so you’re taking the entire box down for a disk failure anyway. And the performance penalty is atrocious, but if you’re a backups company where most of your storage never even gets read again I suppose you don’t care.

So if you impliment your own network level data protections and don’t care at all about access speed this is a pretty cool setup. For anyone else its data suicide.

Enterprise pricing is completely arbitrary

jim — Sat, 31 Mar 2007 14:24:05 +0000

Yesterday marked the usual end of quarter blowout fun, but I had the extra fun of running a deal headfirst into 3PAR’s end of fiscal year. Finnaly eek’d the PO out at 5:30 on friday.

I’m never sure if I’m dragging out the quote negotiation proccess too far. This one started on 2/8 and just ended in a purchase on 3/30. I did the full three-way vendor shoot-out (compellent and dell), chatted with references, got interface demos, got the 2nd-tier sales engineers in for nitty gritty answers, read what must add up to eight hojillion pages of PDF whitepapers, learned an awful lot about the storage industry and in the end got the same quote, line item for line item, down from 534K to 288K.

Meanwhile, five feet away from me, my boss managed to neg Keynote to a 69% across-the-board discount on all their pricing just by saying “no we’re going with Gomez” a few times.

Three months ago I got Verizon to drop their per-meg price from $143/mb to $51 which added up to around 250K/year cheaper than before.

I honestly can’t see 50 – 75% price swings as discounts. If you can really discount that much your pricing was a work of fiction to begin with.