Monday, December 22, 2008

NMA Site of the Week

Yay, an award!
Ironically in the week that we redesign it for spring summer '09 is NMA's (New Media Age) site of the week:

Wednesday, December 17, 2008

eth1 Slammin' Top Tip for starting busy webservers

Much like many others of you we run servers which accept http connections on one eth and make db connections on another. In this case eth0 faces out and eth1 faces in to the db zone.

Sounds reasonable? In fact I suggest that you should put them the other way round.
We get alerts when web requests fail because of a lack of db connection, its meant to let us know when the db connections are running out.

But today colleague x executed a graceful restart of the networking on a busy web server, you can tell whats coming can't you...

Yes... eth0 starts first, processes try to connect through eth1 before its up, hundreds of errors LoL!

So in future, eth0 shall face in-the-way, so that the route to the db is available before we open the front door to the hordes of christmas shoppers.

So remember, at xmas the rule is:- eth0 lan eth1 internet

work that terminal award, command of the week

'Tis the season to be jolly, or jolly busy as we are in the waccy world of online retail.
In fact thats 99.99% of the reason i've been so quite recently.

But naturally, at this time of year, our thoughts turn to such questions as "how many MaxClients should I have?" and "are keepalives better or worse for overall performance?" but the question exercising our minds today has been "what *are* all those ip_conntrack entries?".

Rewind a little... a) I turned keepalives off, and reduced MaxClients, things seemed to be more controlled without all those idle processes ready to receive requests with no warning.
b) We'd always had a low but persistent number of "can't connect", or "no html" errors reported by external beacons, and when we applied test loads this went through the roof.

So today, dmesg says.. ip_conntrack: table full, dropping packet.

Which brings me to the point of this post, we increased the size of the table in line with the unverified facts, lies, and dodgy misinformation prevelant on the web ;-)

That helped, but now we want to know whats in the ip_conntrack?

Enter the victor of the command of the week award..

grep ^tcp /proc/net/ip_conntrack | awk '{print $4}' | sort | uniq -c

Whos output turns a whole mess of gobbledy gook into something like this:


Knowedge is power, have an award Patrick Schaaf. :-)

I know nothing, I'm not a fortune teller, and you'd be insane to think that I am. This disclaimer was cribbed from an email footer I once received. It is so ridiculous I had to have it for myself.

Statements in this blog that are not purely historical are forward-looking statements including, without limitation, statements regarding my expectations, objectives, anticipations, plans, hopes, beliefs, intentions or strategies regarding the future. Factors that could cause actual results to differ materially from the forward looking statements include risks and uncertainties such as any unforeseen event or any unforeseen system failures, and other risks. It is important to note that actual outcomes could differ materially from those in such forward-looking statements.

Danny Angus Copyright © 2006-2013 (OMG that's seven years of this nonsense)