GrinchyNet

I consider myself pretty good at what I do for a living. I don't make a lot of mistakes, I try and stay current with technology and so on. I am confident in my abilities as a SysAdmin but today was one of those days where I felt like a complete noob.

As I mentioned in a post a while back, we are moving into the virtualization world and so far it has been a pretty cool migration. I have learned a lot and have had some very frustrating days. Today was one of those days.

Currently I have four physical servers running Citrix XenServer 5.5. I have them broken into two pools. One pool holds my infrastructure server VM's like my domain controllers. The other pool is for my virtual desktop stuff. The first pool has a 12TB SAN attached to it and the other has a 16TB SAN. Through a lot of reading and research I discovered that I didn't have the SAN's configured very well. Luckily I had a spare 16TB SAN lying around. I set it up the right way and attached it to my second pool. I successfully moved all the VM"s to the new SAN. Also in the process I had moved a SQL cluster from the second pool to the first pool. This all took place a few weeks ago.

Today I got around to reconfiguring the SAN I replaced on the second pool and my plan was to move it over to the first pool. So I went ahead and deleted the LUNs and logical drives and did my reconfiguration. About 30 minutes later we get a call about one of our applications not working. As I look into it I see that they aren't connecting to the SQL database. I login to the SQL cluster and see that it is offline and it is offline because it can't find the cluster and quorum disk. Then it hit me, the SQL cluster disks were on the SAN I just reconfigured. I had completely forgot they had a direct iSCSI connection to the SAN. Oops.

So I just killed every database that was located on the cluster. Luckily there were only two since we hadn't migrated the other databases over yet. Also lucky for me was the fact we had a fairly recent backup of the database. So I spent my day getting the SQL server back up and running and we restored the database but all was not well. The application is now refusing to make connections so tomorrow shall be fun and exciting.

Let this be a lesson to you guys out there. Document what the hell you do and double check before you reconfigure a storage device.