The internet arrived, and broke stuff…

Well the last few days have been interesting for me, here is a brief overview of what went on.

the old rubish design
The Old Design
Bodged together

A lot of my blog posts are cobbled together in around 10-20min with a bit of effort put in but nothing really too taxing, in one of my latest posts I decide to write a brief list on “30 cool ideas for your Raspberry PI project”. As you can see in the screenshot it was a fairly boring bullet point list of 30 things with links to external sites where you can read up in more detail on a project of choice.

Well after a few days the traffic started coming in from Google and I was quite content with my new blog post, while trawling through Google Analytics I noticed a bounce rate of around 90% for that page, well that sucks! First page on Google for “Raspberry Pi Ideas” but no one wants to actually read the content.

The website had also been submitted to a few social sites with similar disappointing interest, however to be honest I completely understand why! only one other up-vote on the screenshot below:

Old Hacker News

Revamp time!

So the plan was to revamp the page and see how that affected the bounce rate, basically I got some pictures for each of the 30 things and put them in a clean and simple HTML table, it did look considerably better for only a few minuets more work, how about another screen shot for comparison? Ok there you are at the right.

the updated design
The new design, with pictures!

As I finished updating the layout I decided to resubmit the page to one social news site (hackernews), then monitored the traffic using Google Analytics real time to get a better idea of what people are doing.

Two active visitors, Ten, Thirty, cool people are actually looking at the content now and not bouncing straight away to another site!…. Eighty visitors now, Two Hundred active visitors…. What’s going on here??

Well it turns out people did like the new design/layout even though the underlying content was exactly the same, and I was now on the front page of hacker news, peaking around number 3 and averaging around number 6 most of the time.

New hacker news

real-time-madness

Everything Broke when the internet arrived

Over the next few hours I was battling trying to keep the server online, I might go into this a bit further in another post once I get some more metrics. However the basics are that CloudFlare (CDN function) didn’t really help because it was PHP based CPU load that was killing me, not throughput. W3cache using a Memcache backend really saved the day, NewRelic is awesome for monitoring my configuration change effects in real time, and finally CDN77 has 100GB of free bandwidth which is also nice when you need some help ?

Even More traffic now…

Just as the site was back on it’s feet people started flooding in from Twitter, Reddit (Raspberry_Pi) and also Google +1, people actually use Google +1 well that was news to me! Actually it’s just edged out reddit for the second spot in Social traffic sources.

Another thing I have noticed, while the hackerne.ws traffic is fairly dependant on been on the front page it appears Reddit and Google are much more stable (so far), I expect this is because they have less churn of stories in their respective sections.

reddit post

The site has now been fairly stable since the recent changes, and there is still a steady flow of traffic from some of the sources.

48 hour visits

48 hour social breakdown

(Stats for last 36 hours only)

Conclusion
  • Put more effort into design, people like nice pictures and layouts
  • CloudFlare + Nginx isn’t a solution on a slow server, get caching on there (memory not disk)
  • CDN’s don’t really help with CPU usage too much
  • As always, keep your solution simple (Memcache > Varnish)
  • Will do a more technical post soon!
  • Front page of Reddit for a few min
  • Front page of hackernews for 24 hours+

If you got this far, please don’t forget to comment or even vote on HN or Reddit, Thank you 🙂

Technology enthusiastic with many ongoing online projects one of which is this personal blog PingBin. While also working full time within a data center designing and maintaining the network infrastructure.

3 comments On The internet arrived, and broke stuff…

  • Just a quick comment – it was really PHP using up the CPU, not MySQL? Can’t make out from the post whether you already had W3Cache installed.

    It’s just been my experience that the database is nearly always the canary in the coal mine – that winds up killing everything else under heavy load…

    • MySQL by default can probably cache most of my content in the RAM or at least the more current popular content, especially considering that reads are going to be fairly consistent and there aren’t that many update/writes.

      Next my CPU is quite underpowered, it’s only an Intel Atom dedicated server from ovh.co.uk, my plan was always to make use of the 2GB of RAM to cache my content where possible to overcome the CPU limitations. I should have spent more time actually implementing this plan on my blog, like I have done on my other custom coded projects.

      Even when using W3Cache to cache the objects/database queries the CPU load was still too high because PHP still had to assemble the page for end user redering, it was only when I used page cache (in memcache) that stuff got more stable.

      I also used a CDN (CDN77) to offload some of the peak traffic, however my Nginx install was already serving the images/css/js files without any real issues, the traffic certainly moved away from my server but CPU wasn’t really helped, at best it released some Nginx workers for the PHP traffic.

      • Just out of curiosity, why did you skip PHP caching? I use nginx + php-fpm with APC on my VPS and I’m able to easily pull a concurrency of 500 users with AB testing. The VPS in question is a 1 core Xen VPS with 512M of memory. I haven’t done much testing with Memcache, and I know it’s fast, but a CDN + nginx + php-fpm with some kind of PHP caching (APC is my preferred method, but XCache works too) should work >almost< as well.

Leave a reply:

Your email address will not be published.

Site Footer