PingBin

So during my misspent christmas break I decided to play around and make a new website, however I wanted to make it a success with some meaningful traffic. So I  decided to play around and see how high I could get on hacker news, and how much traffic I could actually generate from there to my site (also advertising revenue would be interesting).

From experience of reading HN (Hacker News) one of the best things to do is present the community with a recent project that you have completed by hacking stuff together, they generally offer some great feedback and you get quite a few hits (I assume).

Obviously I also wanted to make something that was useful, so I had a look around at the domains I already had, route.im and traceroute.im sprung to mind as I purchased them a few months before, with the intention of creating a simple web interface to trace route an IP address or Hostname.

Build Time

The Design

I started with a fairly simple CSS template from another unfinished project and used some very basic PHP scripting to perform a simple trace route and display the results. I was fairly happy with the product so I uploaded to my web server (also used for this blog) and did some final testing, everything looked good to go!

The DNS for the domains got updated, however route.im was actually already pointing at the right server so I could launch it even faster than normal. Traceroute.im was now updated however that would take some time to prorogate, but nothing that should stop me…

Time to go public

Route.im was now live, so I passed the url around to a few people on twitter, MSN and facebook to have a look at, beta test completed and finally time to let the public have at it, or at least I hope it is.

Picking the title wasn’t too hard “My 2 hour morning project, route.im” was short and to the point, surly that’s going to drag a few people in to have a look?

That’s it we are live, time to keep an eye on my HN points and comments, when you first post on HN you are placed in the ‘newest’ section, only a few people go through here an up vote content giving you points, the aim is to get enough points to get into the top 30 stories, as your then on the home page! I am not sure of their formula however you basically need a constant stream of points to get on the home page and then stay there.

Welcome to the home page

Thankfully enough people found my story interesting enough and with only 10 votes in a short space of time I got to number 3 on the home page of HN (I think this was the peak position although I got too busy to keep checking), I did a quick look at my server load and it didn’t seem to be high, however looking at the nginx logs there were a lot of people on the site!

Then I quickly decided I needed some better logging on the site, so I made a MySQL table and started to log all of the searches been made and timestamps, wow there was a lot! Also I created a page so vistiors could see this.

Server died…

One of the best things about HN is the comments you get back from people going on your site, so always have a window open and keep refreshing it, instantly there were 3 posts from people saying it wasn’t working…

Turns out the server died, before I only had 4 PHP processes as the PHP code processed quick and didn’t hog the resources. However while one process was waiting for a trace route to complete it couldn’t serve any other users, and there was a lot more than 4 request waiting for a process. So I quickly bumped up the PHP processes to 20, also I did the same to Nginx and restarted the services. Disaster averted and users were ‘happy’ again. Here’s a look at some of the traffic figures, and this was after the first ‘peak’.

Live Traffic View

Firefighting, is fun!

Here was a few bugs found with the script, I tried to fix these ASAP

• So many users caused the site to crash – Not enough PHP processes, added more and restarted PHP server.
• Couldn’t trace IPv6 via IP but could via Hostname – Regular expression filtered out the colon ‘:’, simple fix.
• Couldn’t trace a domain with ‘-‘ included – Regular expression filtered out the ‘-‘, simple fix

Over time I did go down the list on HN home page, however I am still there while posting this, and there is quite a few comments!

That's me at 9th

Feature Requests

With the bug reports and comments came a load of feature requestes:

• Show IP geographic location
• Show IP route on a Map
• Show ASN for the route
• Change URL format so you can enter target there, e.g. route.im/8.8.8.8
• Trace route from multiple locations to a single destination
• Auto fill the user field with their IP address

Revenue

I wasn’t going to post about revenue, however it did see a bit dishonest if I intentionally missed it out, so yes have made some money from been featured on HN, below is a screenshot of my adsense revenue. I expect this will probably die off to less than that in a month, however hopefully with some new features we can keep people coming back to the site and sustain some of it.

I am probably going to do some kind of contest on the website over the next few weeks with the generated revenue, just to say thanks

Some tips

• Make sure your app is secure, people on HN like looking for SQL injection from my logs
• Decent logs is always a good thing to have
• Pay attention to the feedback, bad feedback is the best type!
• Don’t be afraid of putting something out there so get some feedback
• Make sure your server is ready for the traffic, if its not, be ready
• Hack together bug fixes quick if you can
• Reply to all comments, you will probably get more people commenting then
• If you don’t get to the front page, change something before posting again.
• Have a look at other HN stories to get ideas.
• Please go in the new section of HN and up vote good stories, it really helps people!

A finial little graph of my Varnish Cache graph from the initial burst:

Varnish Cache is working!

Sorry for the quick post, lack of proof reading and probably vast number of typo’s. I just wanted to get my thoughts out there, I am off now to do some more coding to improve the site!

Thanks, please leave you comments below!

I recently installed Varnish Cache infront of all of my websites on this server, good news was the websites got a lot faster and the cache was great and offloading the effects of traffic spikes, my testing with apache bench showed some great results, but more on that another day.

So basically my setup is as follows:

There was a problem with this, previously users were hitting the Nginx web server and this was then reverse proxying php connections though to php-fpm. However now a cache was installed this was the users first processing stop, any connections from varnish cache to the Nginx web server were done over the server internals transparent to the user. Basically Nginx was seeing the source address as 127.0.0.1 (Varnish) no matter what the client IP address was.

This IP was playing havoc with some of my applications that monitor DDOS attacks and just general website performance/user statistics, however there was some good news.

Varnish can create the same logs as your web server would normally do, it just needs a bit of configuration, enter the following in your /etc/rc.local file, also run it from command line:

varnishncsa -a -w /var/log/varnish/access.log -D -P /var/run/varnishncsa.pid

This will create a log file in /var/log/varnish/access.log using the same setup as normal.

Not this is an access log, not an error log, more on that another day. You might also want to check out this blog if you want help of setting up stats, however I will be covering my version of that soon.

As the title probably suggests this post is going to be somewhat single sided, however I will try outline the benefits of VPS servers and where they are “appropriate”, who knows I might move back to them in a few months. 

Transparency

My day job used to be a server administrator for both Windows and Linux platforms; my principle focus for about 6 months was actually building cloud infrastructures for the company services such as mail and DNS, plus public users as a VPS product.

When it came to aggregating my websites from shared hosting to a dedicated environment it made sense to use a VPS container, however I soon realized when you only have partial view of the setup you need a lot of trust in your provider.

Simple fact is your provider can claim many things, however you never really know how true they are until things go wrong, an example or two follow below.

• What RAID is in use, if any? Are there any failed disks, anyone checking?
• Are there any backups? Tested? How long to restore one if needed?
• What was the real cause of that 2 hour downtime, I was told DDOS on another VPS….
• How secure is my host server?

My basic setup was about 3 VPS servers, 2 were normally live and the other was for testing stuff, should one fail I could fairly easy move stuff around as required. However recently I noticed issues with uptime and general performance of the nodes, it seemed like I was opening tickets at each provider weekly and moving my sites around.

Each time this happened I just started to lose trust in what these providers were actually supplying, having some data loss was the final straw in this. Although I had backups and it was only a test VPS, it didn’t inspire confidence.

Looking at Costs

Hardware is cheap and it’s getting cheaper by the day, my VPS’s were not top of the line ones and in total probably came to about £30/pm, however my server from OVH run’s in at a smooth £15/pm (inc VAT), I do admit it’s specification isn’t top of the line with a dual core ATOM, 2GB of RAM and 1TB HDD, however I do have piece of mind that them resources are going to be available 100% of the time which to me is invaluable.

Also there were many other great features like 100GB of free backup space, automated reinstalls which make it like a VPS server, and high spec servers should I need to upgrade in the future.

Let’s not forget this is saving £15/pm, which isn’t a small figure, truth be told I will probably get another server from somewhere as a backup, I have been in the industry too long to trust hardware always works, which will remove this saving.

Scaling for Traffic

My last VPS’s were fairly small with about 128 to 256MB RAM each, this was to keep costs low and I didn’t really need anything more extraordinary at the time when you looked at the load/usage. The websites were mainly PHP/MySQL and the backend was Nginx with PHP-FPM with is great with restricted server resources.

One of the paybacks I saw for VPS’s was that I could scale as required, however in practice that didn’t really work out. First you needed to know when traffic spikes were going to approach so you can preempt and upgrade, which generally just doesn’t happen as most of my traffic is from social sources like reddit or digg, which made predicting surges unreasonable.

Next there was ways that I could auto scale with some form of scripting, however this takes a lot of determination to setup and therefore wasn’t really suited to a ‘small’ setup like mine. This scaling setup also makes cost budgeting a pain to manage, as you don’t have a clue what next month is going to produce you in terms of costs and ads revenue.

My solution was to get a dedicated server and heavily underutilize the resources, server load is about  0.1 most of the time and RAM usage is generally under 15%. One of the best parts is though I have 1GB of Varnish HTTP memory cache ready for thrashing, so in the event of a social traffic spike I can cache all the content in memory and keep server load under control, this costs absolutely nothing as the RAM is always there at a fixed cost, also it’s instant!

Security

Recently I started working on a new project that will hold customer information and may be even process payments, previously my services have normally been free with no user registration so keeping client information safe has never been a concern.

OpenVZ that my hosts were using isn’t secure, a bold comment I know, but anyone with access to the host server, albeit a hacker or staff member will have full root access to my virtual machine without authentication or even leaving logs. To be completely honest I couldn’t sleep at nigh knowing this with customer information on there, especially when I have no control over the security on that VPS host, could you?

–

Please leave your thoughts on this below, I would be interested to know your server setup? 

When you need to copy file from one server to another SCP is a great tool, you can use it to either push or pull content, and this transfer is done via the SSH protocol which is typically open on all linux servers.

The reason why I mentioned push or pull is because if you have a server that doesn’t have an ssh server installed, you can still pull the content so long as the other server has an ssh server installed, even if it’s light weight dropbear.

Sytanx is fairly simple, the following will login to a remote ssh box and copy files to the local destination

scp user@hostname:/source-files /destination-files

Where as this command will copy some local files to the remote destination.

scp  /source-files user@hostname:/destination-files

However what if you want to copy more than one directory? simple you just need to add the recursive flag “-r”, a working example below:

scp -r root@my.web.server.com:/var/www/whatportis/* /var/www/whatportis/

Just enter the password when prompted!

So may times I see people installing MySQL on a linux box using a command like the below, and then trying to secure the server, usually this ends up fairly insecure and open for hackers to attack.

yum install mysql mysql-server

Good news, there is a really simple command you can run to get things secured, first make sure that your mysql server is up and running:

# /etc/init.d/mysqld start
Initializing MySQL database: Installing MySQL system tables...
OK
Filling help tables...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h ks391934.kimsufi.com password 'new-password'

Alternatively you can run:
/usr/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:
cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl
cd /usr/mysql-test ; perl mysql-test-run.pl

Please report any problems with the /usr/bin/mysqlbug script!

[ OK ]
Starting mysqld: [ OK ]

Then run this command, and answer the questions as shown below, please also make sure you chose a secure root password, and remember each database should have their own user!

# mysql_secure_installation 

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MySQL to secure it, we'll need the current
password for the root user.  If you've just installed MySQL, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MySQL
root user without the proper authorisation.

Set root password? [Y/n] Y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!

By default, a MySQL installation has an anonymous user, allowing anyone
to log into MySQL without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y
 ... Success!

By default, MySQL comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MySQL
installation should now be secure.

Thanks for using MySQL!

[root@ks391934 ~]#