A watched kettle…

So I’ve spent most of my weekend doing this silly directory-flattening thing, mostly because there would be no great peril if I got distracted by something important. Given the server crashes the last two weekends I wanted to try and catch it happening this week, so I stopped by each of the server processes every 15 minutes looking for signs of a problem.

Finally around 11pm (+/-30 minutes) I step away from the computer (well, for more than 5 minutes) and I come back 30 minutes later and find the server has gone into its crazy-not-quite-dead-yet loop. Alas, when it does it registers just enough to fool all of our monitoring systems and other servers that its alive, but not enough to do anone any good, and meanwhile it’s writing “Oops, there’s a problem” to a log file so fast that in the 20 minutes between it starting and my coming back and checking all 4 80Mb log files were full of “Oops, there’s a problem”.

Fortunately, one of the servers actually had some interesting breadcrumbs for me still that might allow me to investigate this properly tomorrow. I have a nasty hunch I might have to break out Valgrind and do some rigorous memory testing.

One Comment

make it print over the weekends and leave a printer on that has plenty of paper and a UPS…plus a new ink cartridge every Friday.

Even better: Use the old dot matrix printers with the green and white paper and buckets of ink – leave that on and let it print the spool of paper onto teh floor. When you come in, you will see (printed 10 googleplex times):

All these worlds are yours, except Europa. Attempt no landing there.

Leave a Reply

Name and email address are required. Your email address will not be published.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

You may use these HTML tags and attributes:

<a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <pre> <q cite=""> <s> <strike> <strong> 

%d bloggers like this: