The Surly Admin

Busy Weeks

No post today for a couple of reasons.  Last two weeks have been remarkably busy, what with my backup server crashing 2 weeks ago (here) and then one of our ERP’s crashing last week–yes, we have two, don’t ask!–I just haven’t had time to work on my script projects.  It’s also been pretty quiet on the Spiceworks front so nothing really interesting coming across there.

Veeam Followup

As I mentioned a couple of weeks ago, my backup server crashed.  This was due to a problem with Veeam Enterprise Manager and Veeam Backup and Recovery being installed on the same server, and upgrading B&R without upgrading Enterprise first.  The fix for the crash turned out to be restoring a week old registry hive, which at least let us get into the server, but now I had a 6.0 registry for Veeam, and a 6.1 installation on the disk.  I contacted support on Tuesday (after working on it for awhile myself) and unfortunately the level one guys were stumped, and their recommendation was to completely reformat the server and start from scratch–this really wasn’t an option for me because this is also my tier 2 storage server, my test environment storage.  Not to mention all the of the tapes that Backup Exec is keeping track of that I didn’t want to loose.  I don’t mind reinstalling things, but I hate doing it on the fly trying to recover for a disaster!

Unfortunately, it took me a couple of days (Thursday) before I finally got past these guys into the tier 2 support people.  This was a very frustrating period as I’m essentially without my primary backup solution.  Luckily I had a secondary solution I was able to use and get the “big stuff” backed up in the meantime.  I did finally get past the tier 1 people into the tier 2 and things started moving.  Ed and Cody worked incredibly hard on the problem all day Thursday and into Friday before finally calling in tier 3 support.  These guys (not sure his name!) finally got the problem resolved and Veeam is back up and functional again.

Interestingly, they are recommending reformating the server before attempting a 6.1 to 6.5 upgrade though as they don’t have a lot of confidence in the permissions on my registry which is a little nerve racking.  I really want to get to the new features in 6.5, so I’m either going to install Backup Exec System Recovery on this server and backup the OS partition (for an easy rollback) or I’ll have to buckle down and do all the prep work that will be necessary for doing a clean format.  Of course, if I do that I might as well go to Windows Server 2012 and take advantage of the new dedupe capabilities!

ERP Down

Just as we got past the Veeam issue, our secondary ERP system decided to go down.  Turns out one of the disks in the 3 disk RAID5 array decided to go belly up, and for some reason the RAID controller lost track of the other 2 disks (calling them foreign disks).  What made this one so fun is this system is in one of our Maryland sites so my administration down there had to deal with it.  Of course, this weekend he had his 4 kids and had to bring them into work to look at the server.  Once we realized the extent of the issue he then had to drop them off with their mother, who happened to be a 1.5 hour drive away.  Did I mention this was on Sunday night?  Yes, the same Sunday night when Hurricane Sandy touched down?  And just to make it more interesting it was payroll week.  When things break, they break big!

So after a bit of delay dealing with all of that he called Dell and they were able to upgrade the firmware on the server and get the RAID array back up minus the failed disk.  The plan this year was to virtualize this server anyway, so we scheduled some time Monday evening to P2V the server–and this took much longer then expected due to the poor I/O the degraded RAID5 could produce–and get that server workload on a more robust and warrantied system.

Other Stuff

Then throw into the mix the fact that we were given 2 days notice of a new sales person coming in–and we had no hardware for her–and a new IT Director pretty much nixed any scripting plans for the last two weeks.  I do have a couple of things I’d like to work on that I think would be interesting:

1. DFS cmdlets.  Based on the work I did with DFS Replication Monitor with History these cmdlets will allow you to interrogate DFS and see how it’s set up, replication members and backlog.
2. Addition to DFS Replication Monitor that it will email the detail page once a day.
3. Sharepoint 2010 Purge List problem:  one reader mentioned there was a problem with this script and I need to dig into it and find out what’s happening.  I haven’t forgotten!

That’s it for now!

November 5, 2012 - Posted by Martin9700 | General, Technical