Thursday, November 27, 2003

ProFUCKduction Server


It all began humbly. It was just a rutinary upgrade, you call on the all-powerfull debian, do a simple apt-get update, say yes, get on with your life. Five minutes, tops. But it all went wrong.
apt-get did it's job perfectly, updated the thing and configured, but the ever present libc got updated, and that's where it all went berserk. The pop3 server decided that it would put a message through it's standar error stream telling everybody who would dare to listen, that somebody had used errno without including errno.h, thus producing a dependency. Applications that reference errno without including (usually by doing ``unsigned int errno'') now generate a runtime warning. Well, fucking kill me if I didn't check every bug present with every piece of software on the box I was runnning.
A full hour after the devilish ap-get I had to call the real experts to devise a clever hack that would cool of things and allow the president to read his e-mail via Outlook. A clever redirection of stderr into /dev/null cleared the problem and mail could de read. It just had a little issue: Whenever somebody sent an e-mail it got send, but an error message went back to the receiver telling him that the mail could not be delivered, altough it had been. This just HAD TO BE FIXED.
THen the guy who actually handles the machine came in and tried to fix the wreck, in the process he downgraded libc, thus rendering alll of the machines services unusable.
With the machine already down, the best solution was to install the whole thing into another box, and to pass about 17GB of data from one to the other. So the very small process that was meant to take 5 minutes, actually took 17 hours, and banned me from ever using a root console on a production server. One has to admire sysadmins.



You are not your job. You are not how much money you have in the bank. You are not the car you drive. You are not the contents of your wallet. (Tyler Durden, who else?), and to comment on somebody
Stereotypes are lousy. You are not better because you have a degree, and you are not worse because you lack it. You are not wiser because you are older, or foolish because you are young.


No comments: