Upgrades, upgrades

We upgraded a bunch of stuff today; we’ve been running like headless chickens since Friday when some rookie came along and tried hack 101 on our auth server and … succeeded. Entering a username of 1′ or ‘1’=’1 got them past the auth server and into the game; it also periodically took out the auth server. Course, it also logged a bunch of stuff.

Shame, shame, shame on the coders responsible for passing data thru to an SQL query from a client. Shame on the playgate author for passing that kind of crap to the server in the first place! I would say shame on me for not spotting it sooner, but with no apparent pressing reason for it, we hadn’t upgraded the auth box in so long that we could no-longer build executables for it.

We are incredibly lucky to have gone so far with nobody trying that!

When it became clear that we weren’t going to be able to roll out a fix and that little miss smacktard was going to continue trying to get in, it was pretty late on Saturday morning, so I created a little script called “fuckthebastard.sh” which monitored and repaired the situation, while outputting logs in the format his ISP asked for.

Not ideal tho, because this would cause a restart of the authentication token sequence. Something I have long ago fixed but couldn’t build a binary for it that the auth host would run.

Another trivial issue with auth was that the old executable *had* to be run in a debugger or it died horribly. This is because MySQL provides a feature to “ping” the database and make sure the connection you’re about to make a request on is still alive. If its timedout/gone away, mysql_ping will wake it up and reconnect. Schweet!

Only some smart mysql genius also made it generate an operating system signal that you have to write specific code for or your application aborts (SIG_PIPE).

It’s widely documented – in google. Several links to MySQL’s official forums/bugtracker saying they are aware of it and will fix it in the next patch (in 1999). It’s still current in 4.11

We also took the opportunity to make use of one of the last Dell 2×3.0ghz xeon systems into the live cluster and move the primary game database, the database proxy app and the strat host onto that machine.

Previously strat was using 0.01% cpu of a 2xPIII 800Mhz system, and the dbd+database were using 45-57% of the cpu on another 2xPIII 800 system.

Combined, they are using less than 0.25% of the CPU of this new box :) It may seem a waste but they are all three transaction servers that need to have the best possible return time, so running them like this is healthy. Could be a help in reducing some of those no-AAR issues.

After this merge, all of the live cluster is now on dual xeons, which means I don’t have to faff around building a mixture of p3 and p4 versions of the live cluster libraries/binaries, and I can turn on the full -march=pentium4 -mmmx -msse -msse2 -mfpmath=sse -O4 optimization for building game hosts. CPU usage is down about 5% on each box as a result.

10 Comments

Before someone pootles along and says “OMG you don’t automatically escape queries!!!!?!?” … As I said, “we hadn’t upgraded the auth box in so long that we could no-longer build executables for it”. Auth has mostly worked for so long that none of my general code improvements had made it out there, then in the last year or so it had become impossible.

The old C code is largely very meticulous about untainting or marking as tainted data coming from outside of itself (me, I don’t trust data from outside of my current routine).

This was the one place it didn’t (and believe me, I’ve sinced checked). And it was such an obvious place and such an obvious gotcha. Outright newbie stuff.

Old code never dies it just finds an inconvenient time to crap all over you.
People love to swear by MySQL but I think its a piece of junk myself; it’s just not quite a real relational database and you always have to code around its limitations. Since I’m a java guy I use H2 for all my databases (when I have a choice).

I’d use Postgres for choice for a personal project, but I wouldn’t change WWII from MySQL. MySQL fits the requirements bill for it perfectly. If we did something crazy and pulled in 10x or 20x the customers, I’d have killer break out the Oracle licenses and go with that in a heartbeat.

Our database API actually spports all 3, MySQL has just served us really well.

quote “If we did something crazy and pulled in 10x or 20x the customers” /quote

Like put in the Americans?… /off topic :P

couldn’t resist.

postgres is my database of choice on a personal project level and a Proof of Concept. So close to Oracle in design, very easy to port to. I think they both spawn off of the Berkly DB tree.

The postgre 8.1 db might give you the speed for transactional processing your wanting. Of course with all of these DB’s its the fine tuning. Now if you don’t want to shell out the Oracle bucks you could look at EnterpriseDB. It’s a custom postgre DB. I’ve read some good remarks about it.

And to think, in my original email to you, I classified this as not your problem. Oops.

I have a brillent idea. If the hacker is in the US, sue him/his parents for billions in lost revenue. Can’t you sue anyone in the US for anything?

You can sue for anything.

You can’t win just because you sued. The courts could throw that out and possibly even apply “defendants costs” to the litigant.

Right, but he was also kind enough to do it again with a different IP address after we had his ISP watch him :) Sadly we were a little slow in getting the info to his local police, so unlikely they will raid his premises, but we’re off to a good start.

Heh sounds kind of exciting, cloak and dagger, trying to catch him red fingered….

Leave a Reply

Name and email address are required. Your email address will not be published.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

You may use these HTML tags and attributes:

<a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <pre> <q cite=""> <s> <strike> <strong> 

%d bloggers like this: