church of default » Works for Me

How to backup an A10 Networks AX Series Load Balancer

jim — Tue, 10 Nov 2009 20:22:47 +0000

[jim@ssh ~]$ ssh admin@ Password: Last login: Tue Nov 10 13:16:15 2009 from 10.10.24.62 [type ? for help] AX>en Password: AX#conf AX(config)#backup config tftp:///a10_backup.tar.gz System files backup succeeded AX(config)#
Notice you want to supply the .tar.gz suffix yourself.

Then on the receiving server server there’s a slight trick to unpacking it: its actually a tarball *within* a tarball thats then gzipped. It also doesn’t create a subdirectory for itself so it’ll explode all over the dir you’re in when you type the tar command.
[root@nas tftpboot]# ll total 12 -rw-rw-rw- 1 nobody nobody 11046 Nov 10 15:14 a10_backup.tar.gz [root@nas tftpboot]# file a10_backup.tar.gz a10_backup.tar.gz: gzip compressed data, from Unix, last modified: Tue Nov 10 15:14:00 2009 [root@nas tftpboot]# tar zxvf a10_backup.tar.gz backup_system.tar [root@nas tftpboot]# tar xvf backup_system.tar a10data/aflex/ a10data/aflex/mdot-redirect.arl a10data/bwlist/ a10data/cert/ a10data/cert/mycorp-wildcard.crt a10data/cert/entrust-L1B-chain.crt a10data/crl/ a10data/hm/ a10data/hm/bin/ a10data/key/ a10data/key/mycorp-private.key etc/release etc/a10_platforms a10data/etc/enable-passwd.pri a10data/etc/enable-passwd.sec a10data/etc/a10enable-passwd a10data/etc/passwd.pri a10data/etc/passwd.sec a10data/etc/a10passwd a10data/etc/startup-config.pri a10data/etc/startup-config.sec a10data/etc/startup-config.profile a10data/etc/network/ifconfig.eth0 a10data/etc/sysconfig/network a10data/etc/timezone a10data/config/ a10data/config/start a10data/web/partition_logo/ a10data/web/partition_logo/title_logo.png mnt/cf/a10data/cert/ mnt/cf/a10data/hm/ mnt/cf/a10data/hm/bin/ mnt/cf/a10data/key/ mnt/cf/a10data/etc/enable-passwd.pri mnt/cf/a10data/etc/enable-passwd.sec mnt/cf/a10data/etc/a10enable-passwd mnt/cf/a10data/etc/passwd.pri mnt/cf/a10data/etc/passwd.sec mnt/cf/a10data/etc/a10passwd mnt/cf/a10data/etc/startup-config.pri mnt/cf/a10data/etc/startup-config.sec mnt/cf/a10data/etc/startup-config.profile mnt/cf/a10data/etc/network/ifconfig.eth0 mnt/cf/a10data/etc/sysconfig/network mnt/cf/a10data/etc/timezone mnt/cf/a10data/config/

more adventures in multipathd

jim — Wed, 30 Sep 2009 04:59:12 +0000

Found this handy tonight, multipathd has an interactive shell you can use by running “multipathd -k” which is a flag not documented in the man page. Actually, no flags are documented in the man page. You can get a list of the command options by typing help (or generally any command it can’t figure out what to do with). That lead me to the “fail path” option which is what I’m trying to do here, test individual path failures without having to go through the 5 step java gui dance with my fibre switches.

[root@nas ~]# multipathd -k
multipathd> show paths
hcil dev dev_t pri dm_st chk_st next_check
0:2:0:0 sda 8:0 1 [undef] [ready] [orphan]
1:0:0:1 sdb 8:16 1 [active][ready] XXXXXXXX.. 16/20
2:0:0:1 sdd 8:48 1 [active][ready] XXXXX….. 11/20

yes as you can see I clearly don’t have my blacklist setup right. Now to fail a path its “fail path $path” where $path is… again, no documentation. Turns out what they want is the “dev” column (second one)

multipathd> fail path sdd
ok
multipathd> show paths
hcil dev dev_t pri dm_st chk_st next_check
0:2:0:0 sda 8:0 1 [undef] [ready] [orphan]
1:0:0:1 sdb 8:16 1 [active][ready] XXXXX….. 10/20
2:0:0:1 sdd 8:48 1 [failed][faulty] X……… 3/20

Interesting to note, multipathd seems pretty good at its job, as the path is re-enabled within seconds when the checker notices its fine.

Sep 30 00:51:35 nas multipathd: mpath1: fail path 8:48 (operator)
Sep 30 00:51:35 nas kernel: device-mapper: multipath: Failing path 8:48.
Sep 30 00:51:35 nas multipathd: dm-2: add map (uevent)
Sep 30 00:51:35 nas multipathd: dm-2: devmap already registered
Sep 30 00:51:35 nas multipathd: 8:48: mark as failed
Sep 30 00:51:35 nas multipathd: mpath1: remaining active paths: 1
Sep 30 00:51:40 nas multipathd: sdd: readsector0 checker reports path is up
Sep 30 00:51:40 nas multipathd: 8:48: reinstated
Sep 30 00:51:40 nas multipathd: mpath1: remaining active paths: 2
Sep 30 00:51:40 nas multipathd: dm-2: add map (uevent)
Sep 30 00:51:40 nas multipathd: dm-2: devmap already registered

So the good news is failing one path and then the other worked nicely and the server didn’t freak out and remount the filesystem read-only.

How to mount a LUN from a Xiotech Emprise 5000 using multipathd on RHEL or CentOS 5.3

jim — Wed, 05 Aug 2009 03:31:34 +0000

DO NOT DO THIS
turns out the multipath.conf example from xone is all sorts of wrong, it effectively gets ignored and you’re using the defaults (which it turns out work ok-ish). hopefully I’ll update here when I have two days to blow on the phone support game.

Well if you wanted to you could read the official xiotech docs which in turn reccomend you the redhat docs. That would be a great way to kill a day in frustration while sifting through a lot of noise. Here’s how to get a very basic example working quickly:

1.) make sure you have device-mapper-multipath installed

[jim@sql1 ~]$ rpm -qa | grep multi
device-mapper-multipath-0.4.7-23.el5_3.4

if you do, doesn’t hurt to yum update it. if not, install it.

2.) edit /etc/multipath.conf

first comment out the blacklist lines near the top

#blacklist {
#        devnode "*"
#}

then append this to the bottom

devices {
        device {
                vendor                  "XIOTECH "
                product                 "ISE1400         "
                path_grouping_policy    multibus
                getuid_callout          "/sbin/scsi_id -g -u -d /dev/%n"
                path_checker            tur
                prio_callout              "none"
                path_selector           "round-robin 0"
                failback                    immediate
                no_path_retry           12
                user_friendly_names yes
        }
}

3.) setup and prod multipath

[jim@sql1 ~]$ sudo /sbin/chkconfig multipathd on
[jim@sql1 ~]$ sudo /etc/init.d/multipathd start
Starting multipathd daemon:
[jim@sql1 ~]$ sudo /sbin/multipath -v2

notice it returns nothing, the redhat docs say it should, but it doesn’t. its just scanning the paths, if you want to see what it has discovered/conifgured run this:

[jim@sql1 ~]$ sudo /sbin/multipath -ll
mpath1 (36001f93000a63000019b000200000000) dm-2 XIOTECH,ISE1400
[size=100G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 2:0:0:1 sdb 8:16  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 3:0:0:1 sdc 8:32  [active][ready]

take note of the “mpath1″ part there

4.) filesystem stuff time

format the disk (I’m not bothering with fdisk/partitions or lvm in this example, complicate as you see fit)

[jim@sql1 ~]$ sudo /sbin/mkfs.ext3 /dev/mapper/mpath1

[jim@sql1 ~]$ sudo mkdir /mnt/xio
[jim@sql1 ~]$ sudo vi /etc/fstab

and stick this line on the bottom

/dev/mapper/mpath1      /mnt/xio                ext3         defaults,noatime        1 1

then you’re pretty much set, mount it

[jim@sql1 ~]$ sudo mount /mnt/xio
[jim@sql1 ~]$ cd /mnt/xio/
[jim@sql1 xio]$ ls
lost+found
[jim@sql1 xio]$ sudo dd if=/dev/zero of=./foo bs=8k count=1310720
1310720+0 records in
1310720+0 records out
10737418240 bytes (11 GB) copied, 24.9054 seconds, 431 MB/s
[jim@sql1 xio]$

How to reset the BIOS on a Dell M610 blade to the defaults

jim — Wed, 05 Aug 2009 02:55:41 +0000

There are (hypothetically) two ways:

in the bios: caps lock + num lock + scroll lock + alt-f

you have to be on the kvm for this to work, or at least I assume, because it didn’t work over idrac for me

jumper shuffle

shut down the blade, pull it out of the chassis, open it up, find the nvram_clr jumper (see picture), toggle it, put it all back together, boot up, watch for the “restoring factory defaults” message (or something like that, I didn’t get to see it, colo hands-and-eyes guy did), power the blade down again, pull it out, open it up, flip the jumper back, put it together again, re-rack, power up and hit F1… done.

Making sense of Dell’s M610 blade server memory configuration

jim — Fri, 17 Apr 2009 13:13:17 +0000

Over the last two weeks I’ve had a number of confrence calls with my Dell sales rep and some engineers about the new Xeon 5500 / “Nehalem” based blade servers, and specifically over the memory configurations thereof. I have learned one thing for certain, this shit is impossible to explain without pictures.

The good news is, eventually I stumbeled across this great writeup with pictures on the Dell TechCenter Wiki: Nehalem and Memory Configurations

However its still a lot of words, and requires some intepretation to adapt to the M610, so with that in mind here is my explanation in ms-paint form:

Channels, Banks, and Slots

The Good Configs

Read the linked article for why, but the short of it is:

you want all your sticks of ram to be the same size
you want to use either 6, 8, or 12 slots at a time
you want to use 4GB or 8GB sticks. Smaller would add up to less than 3GB-per-core, and given how much ass these kick at vmware that’d be a weird config.

Wait, what? Show me

6 slots

6 x 4GB = 24GB
6 x 8GB = 48GB
can run at 1333MHz or 1066MHz, uses all 3 channels per socket (best memory bandwidth option)

Eight slots

8 x 4GB = 32GB
8 x 8GB = 64GB
because two slots per channel are populated can only run up to 1066MHz

Twelve slots

12 x 4GB = 48GB
12 x 8GB = 96GB

The Neckbeard Take? (sorry storagemojo)

6×8GB for 48GB is probably the sweet spot. It gives the best memory bandwidth performance and a 6GB-per-core ratio thats a good midrange for virtualization.

NOTE: HUGE CAVEAT! – 8GB sticks aren’t available for purchase yet. So if you plan on running vmware on these, you have a two month wait ahead of you.

Links

ran into some other good posts while I was at this, here’s the links

You can have multiple default routes in linux (centos/rhel5)

jim — Thu, 16 Apr 2009 22:19:38 +0000

Just had to do a “live” default route cutover and while I was mostly sure it was gonna work its nice to have it definatively proven now. You can add the second default gateway, then remove the first, so there’s never not a default route. I’m not sure which one it kept using while they were both there, or if it per-packet loadbalances or what, but my ssh sessions didn’t even hiccup so at the very least it won’t knock you offline or interupt connections. Of course this is assuming your upstream devices aren’t doing any nat/firewall stuff to complicate things.