Self inflicted pain
Nothing much worse in IT then server outage. What is worse, is when you do it to yourself. This is the sorry tale of August 5th, 2010 and how I inflicted a horrible day upon myself. Unrelenting stress, fear of job, the whole nine yards.
It all started innocently enough on Wednesday the 4th, when I started installing my new virtual machine backup software (Veeam). No big, right? Just carve out a little more space on my backup SAN and get my backups going. Well, it’s been awhile since I have last used OpenFiler and it took me a little while to re-figure out the interface and get my volume attached to the proper iSCSI target. But I did and everything seemed OK.
Until the 5th, when we discovered that the email archive server was not functioning. This happens to be a virtual server who’s primary data virtual hard disk resides on the same SAN as the backups. So I immediately jump to the conclusion that I screwed something up and dropped the configuration off. Not a huge deal, right? Just reconfigure it and VMware will detect it and everything will be happy. This is where it all started going bad. In retrospect, it turns out that the iSCSI target I was using went “bad” and even though it was configured OK, it was not showing up on VMware. But at the time I didn’t realize this. So I try many different things and I just can’t get the storage to attach. What’s going on? Normally with VMware if it see’s a VMware formated volume (VMFS) then it will just mount it and you’re good. Not today. But, I can see the volume so let’s try adding storage. So I do that and go through the formating process. But I know I’m not actually going to format it, because VMware doesn’t do that. Right? Nope, it formated it.
I’ve now destroyed the data volume. The data volume that I don’t have a backup of. The data volume that I was waiting until I got my virtual machine backup software before taking a backup of. Ironic, huh?
Several minutes of panic’d thoughts go by. All I can think is to reinstall the archival software, then start pulling all my monthly backups for the past year and restore them. But the email archive software, when it removes an email from Exchange, it leaves a small shortcut file. So first I need to discover if I restore an email will it overwrite the shortcut or simply duplicate?
Go to the Backup server and getting all kinds of error messages on there. Great, now I need to save the backup server before I can get to the actual problem. Why me? But as I log on, I discover that the volume where I keep my real-time backups is just gone from the server. This is great news! The volume I formated must have been THIS volume, not my archive data volume! Jump back on the SAN and still can’t get the archive data volume to get recognized. This is when I create a new iSCSI LUN and attach the storage there. BANG! Data volume back on VMware and bring the server up minutes later.
So you think this would be the end of the story, wouldn’t you? Surely I’ve learned my lesson and will stop messing with it… right?
Right?
No comments yet.
Me
LinkedIn
The Surly Admin 
Leave a Reply