Some of you know that the last year at my work has been building up to this weekend. We were purchased by a MUCH larger company, and we have been planning for a year to move our networks, databases, phone systems, PC's, and printers over to the new system. Well, this weekend was it. Being the only IT guy on site (thanks, Jeremy!), I knew I'd be here all weekend. But, they sent a systems tech from the Orlando office, and a cabling guy from another office to help me. And "help me" they did. They almost helped me out of a job.
Friday night, after close of business, we started moving over cables. I gave the tech very explicit instructions to unplug ONLY the yellow cables in the TOP HALF of the rack. (We would later be running cables from here to the new switch, so the short yellow patch cables had to go.) You can see those cables below:
Now, do you see those long green and yellow cables in the middle? Those are VERY important - they ARE our current network. And since all of our production databases are sitting in that room, they need to be transferred to the new system in Delaware. This is PRODUCTION data, that HAS to be correct, or someone will lose a head.
Anyway, while I was out of the room, the cabling guy walked into the room to see what was going on. He said to the tech "Don't worry, all of their old systems are being decommissioned, so just unplug everything". The tech decided to follow his directions instead of mine, and just yanked everything out.
I felt a disturbance in the Force, as if dozens of servers were crying out in terror, and then were suddenly silenced. (Actually, I just heard the alarm system beeping a warning.) I ran into the server room and freaked out. After calling an emergency meeting with both "offenders", we tried in vain to figure out where everything went. Nothing worked.
I knew I had a diagram of the network on one of the servers, but it wasn't connected to anything (due to our network collapsing when those cables were pulled). So, I sat down at it, and found the diagram. It was a PDF...and there was no PDF viewer on that server. So, I grabbed my USB stick...but that server was so old, it DIDN'T HAVE A USB PORT. (Thanks for keeping old crappy machines around, Jeremy.) It only had a floppy drive! So, I found a floppy disc (Thanks for not ever throwing anything away, Jeremy.) and copied the file over.
We went searching for another machine with a floppy, but they don't install those in PC's anymore, so it took us 15 minutes of crawling under every desk in the office before we finally found a floppy drive. I logged in, opened up the file...but it was out of date and no longer correct.
My boss called me shortly after that..."We can't move over the databases tonight if the network is down"...I know, I know. I'm running out of options at this point, though. I considered calling Jeremy (Thanks again for moving on to better things, Jeremy), but I didn't want to bug him on a Friday night.
The tech (who I will refer to henceforth as "
Mr. Random Unplugger") did have a brilliant idea, though. When he and his boss visited our site a few months ago, they took pictures of all the racks and servers in the room. So, he called his boss, and had him forward those pics to his iPhone.
Mr. Random Unplugger then bought up the pics on his iPhone, rotated, and zoomed in on them until we could see exactly which cable went where. We didn't get it exactly right, though. Still no worky.
About an hour after the initial meltdown of our network, we were all still standing there scratching our heads and testing ports. The cabling guy (who I will refer to henceforth as "
The Un-cabler"), said "let me see that again", and took the iPhone for one last look. "Ah ha!",
The Un-cabler said...I see the problem. We had one cable in the wrong spot...once it was corrected, our network came back to life, data started moving, and I passed out from relief.
OK, I didn't pass out, but it would have been understandable if I had. There has been a TON of pressure on us to get this done this weekend. "Heads will roll", I've been told, if this does not go through.
The crises was averted, so we got back to work. We had to replace those little yellow cables with the 12-foot cables we were shipped...they were being plugged into a new rack off to the right of that picture. We quickly ran into the next problem of the night - we needed 15-foot cables, not 12-footers.
The Un-cabler got on the phone and kicked off an emergency conference call. It was decided to drop ship us 15-foot cables by Next Day Air. Also, we had to go ahead and use the 12-footers because the phones and terminals needed to be installed the next morning, and they were depending on those cables to be in place. So we had to just stretch them across empty space and leave them dangling in mid air. It reminded me of
this...I wish I had taken a picture.
Anyway,
Mr. Random Unplugger and
The Un-cabler did indeed prove to be hard workers and good at their jobs as the weekend wore on, but I wanted to kill them that night. In a purely Christian way, of course.
Stay tuned for Saturday's exploits in the next post.
