Archive for the ‘Works for Me’ Category

How to backup an A10 Networks AX Series Load Balancer

Tuesday, November 10th, 2009


[jim@ssh ~]$ ssh admin@<your-loadbalancer>
Password:
Last login: Tue Nov 10 13:16:15 2009 from 10.10.24.62
[type ? for help]
AX>en
Password:
AX#conf
AX(config)#backup config tftp://<yourserver>/a10_backup.tar.gz
System files backup succeeded
AX(config)#

Notice you want to supply the .tar.gz suffix yourself.

Then on the receiving server server there’s a slight trick to unpacking it: its actually a tarball *within* a tarball thats then gzipped. It also doesn’t create a subdirectory for itself so it’ll explode all over the dir you’re in when you type the tar command.

[root@nas tftpboot]# ll
total 12
-rw-rw-rw- 1 nobody nobody 11046 Nov 10 15:14 a10_backup.tar.gz
[root@nas tftpboot]# file a10_backup.tar.gz
a10_backup.tar.gz: gzip compressed data, from Unix, last modified: Tue Nov 10 15:14:00 2009
[root@nas tftpboot]# tar zxvf a10_backup.tar.gz
backup_system.tar
[root@nas tftpboot]# tar xvf backup_system.tar
a10data/aflex/
a10data/aflex/mdot-redirect.arl
a10data/bwlist/
a10data/cert/
a10data/cert/mycorp-wildcard.crt
a10data/cert/entrust-L1B-chain.crt
a10data/crl/
a10data/hm/
a10data/hm/bin/
a10data/key/
a10data/key/mycorp-private.key
etc/release
etc/a10_platforms
a10data/etc/enable-passwd.pri
a10data/etc/enable-passwd.sec
a10data/etc/a10enable-passwd
a10data/etc/passwd.pri
a10data/etc/passwd.sec
a10data/etc/a10passwd
a10data/etc/startup-config.pri
a10data/etc/startup-config.sec
a10data/etc/startup-config.profile
a10data/etc/network/ifconfig.eth0
a10data/etc/sysconfig/network
a10data/etc/timezone
a10data/config/
a10data/config/start
a10data/web/partition_logo/
a10data/web/partition_logo/title_logo.png
mnt/cf/a10data/cert/
mnt/cf/a10data/hm/
mnt/cf/a10data/hm/bin/
mnt/cf/a10data/key/
mnt/cf/a10data/etc/enable-passwd.pri
mnt/cf/a10data/etc/enable-passwd.sec
mnt/cf/a10data/etc/a10enable-passwd
mnt/cf/a10data/etc/passwd.pri
mnt/cf/a10data/etc/passwd.sec
mnt/cf/a10data/etc/a10passwd
mnt/cf/a10data/etc/startup-config.pri
mnt/cf/a10data/etc/startup-config.sec
mnt/cf/a10data/etc/startup-config.profile
mnt/cf/a10data/etc/network/ifconfig.eth0
mnt/cf/a10data/etc/sysconfig/network
mnt/cf/a10data/etc/timezone
mnt/cf/a10data/config/

more adventures in multipathd

Tuesday, September 29th, 2009

Found this handy tonight, multipathd has an interactive shell you can use by running “multipathd -k” which is a flag not documented in the man page. Actually, no flags are documented in the man page. You can get a list of the command options by typing help (or generally any command it can’t figure out what to do with). That lead me to the “fail path” option which is what I’m trying to do here, test individual path failures without having to go through the 5 step java gui dance with my fibre switches.

[root@nas ~]# multipathd -k
multipathd> show paths
hcil dev dev_t pri dm_st chk_st next_check
0:2:0:0 sda 8:0 1 [undef] [ready] [orphan]
1:0:0:1 sdb 8:16 1 [active][ready] XXXXXXXX.. 16/20
2:0:0:1 sdd 8:48 1 [active][ready] XXXXX….. 11/20

yes as you can see I clearly don’t have my blacklist setup right. Now to fail a path its “fail path $path” where $path is… again, no documentation. Turns out what they want is the “dev” column (second one)

multipathd> fail path sdd
ok
multipathd> show paths
hcil dev dev_t pri dm_st chk_st next_check
0:2:0:0 sda 8:0 1 [undef] [ready] [orphan]
1:0:0:1 sdb 8:16 1 [active][ready] XXXXX….. 10/20
2:0:0:1 sdd 8:48 1 [failed][faulty] X……… 3/20

Interesting to note, multipathd seems pretty good at its job, as the path is re-enabled within seconds when the checker notices its fine.

Sep 30 00:51:35 nas multipathd: mpath1: fail path 8:48 (operator)
Sep 30 00:51:35 nas kernel: device-mapper: multipath: Failing path 8:48.
Sep 30 00:51:35 nas multipathd: dm-2: add map (uevent)
Sep 30 00:51:35 nas multipathd: dm-2: devmap already registered
Sep 30 00:51:35 nas multipathd: 8:48: mark as failed
Sep 30 00:51:35 nas multipathd: mpath1: remaining active paths: 1
Sep 30 00:51:40 nas multipathd: sdd: readsector0 checker reports path is up
Sep 30 00:51:40 nas multipathd: 8:48: reinstated
Sep 30 00:51:40 nas multipathd: mpath1: remaining active paths: 2
Sep 30 00:51:40 nas multipathd: dm-2: add map (uevent)
Sep 30 00:51:40 nas multipathd: dm-2: devmap already registered

So the good news is failing one path and then the other worked nicely and the server didn’t freak out and remount the filesystem read-only.

How to mount a LUN from a Xiotech Emprise 5000 using multipathd on RHEL or CentOS 5.3

Tuesday, August 4th, 2009

DO NOT DO THIS
turns out the multipath.conf example from xone is all sorts of wrong, it effectively gets ignored and you’re using the defaults (which it turns out work ok-ish). hopefully I’ll update here when I have two days to blow on the phone support game.

Well if you wanted to you could read the official xiotech docs which in turn reccomend you the redhat docs.  That would be a great way to kill a day in frustration while sifting through a lot of noise.  Here’s how to get a very basic example working quickly:

1.) make sure you have device-mapper-multipath installed

[jim@sql1 ~]$ rpm -qa | grep multi
device-mapper-multipath-0.4.7-23.el5_3.4

if you do, doesn’t hurt to yum update it. if not, install it.

2.) edit /etc/multipath.conf

first comment out the blacklist lines near the top

#blacklist {
#        devnode "*"
#}

then append this to the bottom

devices {
        device {
                vendor                  "XIOTECH "
                product                 "ISE1400         "
                path_grouping_policy    multibus
                getuid_callout          "/sbin/scsi_id -g -u -d /dev/%n"
                path_checker            tur
                prio_callout              "none"
                path_selector           "round-robin 0"
                failback                    immediate
                no_path_retry           12
                user_friendly_names yes
        }
}

3.) setup and prod multipath

[jim@sql1 ~]$ sudo /sbin/chkconfig multipathd on
[jim@sql1 ~]$ sudo /etc/init.d/multipathd start
Starting multipathd daemon:
[jim@sql1 ~]$ sudo /sbin/multipath -v2

notice it returns nothing, the redhat docs say it should, but it doesn’t. its just scanning the paths, if you want to see what it has discovered/conifgured run this:

[jim@sql1 ~]$ sudo /sbin/multipath -ll
mpath1 (36001f93000a63000019b000200000000) dm-2 XIOTECH,ISE1400
[size=100G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 2:0:0:1 sdb 8:16  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 3:0:0:1 sdc 8:32  [active][ready]

take note of the “mpath1″ part there

4.) filesystem stuff time

format the disk (I’m not bothering with fdisk/partitions or lvm in this example, complicate as you see fit)

[jim@sql1 ~]$ sudo /sbin/mkfs.ext3 /dev/mapper/mpath1
<snip>
[jim@sql1 ~]$ sudo mkdir /mnt/xio
[jim@sql1 ~]$ sudo vi /etc/fstab

and stick this line on the bottom

/dev/mapper/mpath1      /mnt/xio                ext3         defaults,noatime        1 1

then you’re pretty much set, mount it

[jim@sql1 ~]$ sudo mount /mnt/xio
[jim@sql1 ~]$ cd /mnt/xio/
[jim@sql1 xio]$ ls
lost+found
[jim@sql1 xio]$ sudo dd if=/dev/zero of=./foo bs=8k count=1310720
1310720+0 records in
1310720+0 records out
10737418240 bytes (11 GB) copied, 24.9054 seconds, 431 MB/s
[jim@sql1 xio]$

How to reset the BIOS on a Dell M610 blade to the defaults

Tuesday, August 4th, 2009

There are (hypothetically) two ways:

in the bios: caps lock + num lock + scroll lock + alt-f

you have to be on the kvm for this to work, or at least I assume, because it didn’t work over idrac for me

jumper shuffle

shut down the blade, pull it out of the chassis, open it up, find the nvram_clr jumper (see picture), toggle it, put it all back together, boot up, watch for the “restoring factory defaults” message (or something like that, I didn’t get to see it, colo hands-and-eyes guy did), power the blade down again, pull it out, open it up, flip the jumper back, put it together again, re-rack, power up and hit F1… done.

onceler6


Making sense of Dell’s M610 blade server memory configuration

Friday, April 17th, 2009

Over the last two weeks I’ve had a number of confrence calls with my Dell sales rep and some engineers about the new Xeon 5500 / “Nehalem” based blade servers, and specifically over the memory configurations thereof.  I have learned one thing for certain, this shit is impossible to explain without pictures.

The good news is, eventually I stumbeled across this great writeup with pictures on the Dell TechCenter Wiki: Nehalem and Memory Configurations

However its still a lot of words, and requires some intepretation to adapt to the M610, so with that in mind here is my explanation in ms-paint form:

Channels, Banks, and Slots

dellram-m610-basic

The Good Configs

Read the linked article for why, but the short of it is:

  • you want all your sticks of ram to be the same size
  • you want to use either 6, 8, or 12 slots at a time
  • you want to use 4GB or 8GB sticks.  Smaller would add up to less than 3GB-per-core, and given how much ass these kick at vmware that’d be a weird config.

Wait, what?  Show me

6 slots

dellram-m610-six

  • 6 x 4GB = 24GB
  • 6 x 8GB = 48GB
  • can run at 1333MHz or 1066MHz, uses all 3 channels per socket (best memory bandwidth option)

Eight slots

dellram-m610-eight

  • 8 x 4GB = 32GB
  • 8 x 8GB = 64GB
  • because two slots per channel are populated can only run up to 1066MHz

 

Twelve slots

dellram-m610-twelve

  • 12 x 4GB = 48GB
  • 12 x 8GB = 96GB

The Neckbeard Take? (sorry storagemojo)

6×8GB for 48GB is probably the sweet spot. It gives the best memory bandwidth performance and a 6GB-per-core ratio thats a good midrange for virtualization.

NOTE: HUGE CAVEAT! – 8GB sticks aren’t available for purchase yet. So if you plan on running vmware on these, you have a two month wait ahead of you.

Links

ran into some other good posts while I was at this, here’s the links