NYCPHP Meetup

[Nicholas Tang]

I may actually have to eat my words - or at least nibble on them a bit.  ;)
 
We were benchmarking our new beta setup - Apache 2, PHP 5, and APC 3 - and seeing absolutely terrible results.  Really, really bad.  Atrocious.  As in, I had momentary fears we'd have to pick between a rewrite in PHP 4 or double the size of our webserver farm to manage it.
 
Anyways, while I can't reveal the exact results (and since they're using our in-house apps they wouldn't help much anyways) but under a load of 12,000 simultaneous simulated users (approximately 600 page views/ second), we were seeing page load times averaging over 20 seconds per page during our test on Tuesday morning - obviously completely unusable on a production website.
 
Friday morning we re-ran the test and got it down into the sub-4 seconds per-page range.
 
What changed?
 
Amazingly, while I did tweak some settings here and there, and we did nudge a few things in the code, the biggest change?  We downgraded our servers from Red Hat Enterprise 4 Update 3 to Update 2.  That was it.  Approximately 90% of the difference in the two tests came from that one shift - a shift that should have been completely inconsequential.
 
Now, before you ask - we build our own Apache/PHP/APC binaries, so none of those would have been changed.  We still haven't isolated it to any specific issue, but *something* changed drastically on the OS side, something that absolutely killed performance.  Until we can isolate it, and fix/remote/work-around it, we're officially standardizing on RHEL 4U2 - and I advise anyone out there who is running RHEL and considering moving to Update 3 to test it thoroughly before embarking on that journey.  It should be harmless, but in our case, it was anything but.
 
(On the flip side, U3 fixes several critical bugs in multipath-tools, so we've had to upgrade some of our DBs to it.  We haven't noticed any major performance degredation w/ Oracle, which is good.)
 
Nicholas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 4980 bytes
Desc: not available
URL: <http://lists.nyphp.org/pipermail/talk/attachments/20060430/1046b0f8/attachment.bin>