Always Get Better

Archive for December, 2011

2011 In Review

Saturday, December 31st, 2011
Hot Desk
Creative Commons License photo credit: mikecogh

This year started off with a foray into Rails, an experience I won’t be rushing to complete. Most of my time was spent building a small application during the Christmas break in 2010, but in 2011 I moved that site into production and wrote a little bit about separating production and development values (a feat I repeated for the Play! framework, which I actually like, later in the year). I think the only thing I really like from Rails, and this is a bit of a stretch, is the database migrations.

Having moved entirely over to a LAMP platform professionally, and getting good at the security nuances plus everything else, I reminisced a little about some of the creature comforts I missed in C#, like operators for default null variables. But when I discovered Time Machine on my Mac, there was no going back – until the company switched directions and I got thrown back into .NET development.

That’s right – back to .NET, and deep into the Windows Azure cloud. I dealt with things like figuring out which is better – table storage or SQL Azure, and figuring out the nuances of their multiple SLAs, and how to ensure we actually have Azure Compute instances on-line when it hits the fan. At this point I have a pretty good handle on Azure’s strengths and weaknesses and my overall impression of the platform is very positive. If I continue building sites on the Microsoft stack, I would definitely continue to use Azure – it seems more expensive than other options at first glance, but it has some serious computing power behind it and takes the majority of administration headaches off of my plate. It really has enabled me to, for the most part, just focus on development whereas I was spending an increasing amount of my work day on system administration issues when supporting the LAMP platform.

Scalability and high availability have been on my mind a lot, and I’ve been looking into some more ‘off-beat’ database solutions like Drizzle for my transactional needs, as well as speeding up existing deployments by moving as much as possible into RAM. There’s always a battle between changing the way we work to take advantage of the new paradigm or changing our existing configuration to get some more life out of it.

The whole cloud computing buzz feels tired but has enabled a whole new class of online business. If you have a lot of commodity hardware you can achieve, very cheaply, feats that were only possible with an expensive dedicated network just a few years ago. Sure, it adds a lot of new choke points you will need good people to help sort through, which is giving rise to a whole new sub-category of programmer specialization to make hiring in 2012 even more challenging.

There is a downside to all the cloud computing, though, as we learned during the high profile Amazon failures – backups are important. This includes geographically-redundant systems that most organizations don’t have the experience to deal effectively with just yet. Even so, the biggest lesson I learned was never let your server run into swap space or your performance will nose-dive. The growth of this site even prompted me to move more of the site into memory which prevented me from needing to spend a lot of money upgrading my infrastructure.

Social media continues to grow, with companies realizing they can’t control its effect on their business in traditional ways and less-than-useless cons ruining it for everyone by selling CEOs on cheap gimmicks.

Since my third child was born in February, I’ve definitely taken some time to sotp and reflect on what I want to work on, why I want to keep working, and what the next steps are career-wise and life-wise. I want to provide the best that I can for my family and 2012 is going to see a radical course change as I start to shift gears and begin building something that will really last, even outlast me. When will my website start paying my bills? I don’t expect it will.

I learned a lot by running dozens (over a hundred?) job interviews in the past two years. Ignoring the old adage that you shouldn’t judge a book by it’s cover, I learned that you can tell with pretty good accuracy whether or not someone will be a good match for your company within the first five minutes of an interview. I’m less interested in hiring people with domain knowledge than I am in surrounding myself with the most intelligent developers I can find – one is a skill that can be taught, the other is an aptitude candidates need to bring to the table. Really, when it comes down to it, what I really want is for people I hire to stand up for themselves (since they are adults) and make me look good by being awesome at what they do.

I also learned a lot by being responsible for some very large projects; things like the importance of continuous integration.

What’s next in 2012? Look for mobile device use to continue growth – every developer who plans to stay employed needs to know something about mobile development, because it’s going to be ubiquitous with regular desktop programming very soon. Now that version 0.6 has been released with Windows support is Node.js ready for prime-time? I had the opportunity to play with it a lot over the past few month – look for a book early in the new year co-authored by yours truly.

Using FastCGI with Nginx for Performance on a VM

Tuesday, December 20th, 2011

This weekend I decided to play around with the configuration on my Rackspace Cloud Server. Since our various websites have been doing well lately, the relatively low-powered machine I am running on is starting to fill up its available RAM. So far so good but as everyone quickly learns – running out of memory and hitting the swap space is a performance killer. Since I want my sites to continue to do well, I decided to take action before they hit the RAM limit and start swapping to disk.

This is the old architecture I had deployed. Apache – the Internet’s workhorse – to perform all of the PHP processing, with Nginx as a reverse proxy, passing dynamic (PHP) requests to Apache but serving static files directly to give Apache a break and cut down its footprint.

I optimized MySQL with a large buffer, so it serves the vast majority of queries directly from memory.

The two places that hit the filesystem are Nginx and PHP – Nginx for static files (as mentioned, to take the load of Apache which would spin up a new instance for each file it serves) and PHP for session data (this is PHP’s default setting).

 

This is the new setup, and is very similar to the old with two key differences: Apache is gone, and Memcached is now in the mix.

As site traffic increased I noticed that Apache was using up bigger and bigger chunks of the system RAM compared to all of the other processes. I could pare this down by putting further restrictions on the number of child processes, decrease the number of connections before recycling, and limiting the maximum memory for each process, but that seemed like a lot of work when I already had one foot into a more scalable solution.

Taking advantage of the new(ish) since PHP 5.3.3 FastCGI Process Manager (FPM), I updated Nginx to send PHP traffic directly to PHP without using Apache as a middleman. The default settings were too generous for my fairly weak server, and the memory usage shot up. But by tweaking it down to 3 processes recycling after 500 requests, I’m now using half the physical memory as I was with Apache.

Previously I wrote about using Memcached as a PHP Session Handler and that’s exactly what I did here. Now the Filesystem is only hit for static files and the first run of PHP scripts – everything else is served from memory.

It may seem a little counter-intuitive that I cut memory consumption by moving more services into memory, but the trade-off improvements realized by hitting the disk less means that responses are sent out 30% faster, meaning I can fit more traffic onto the same machine and expect the same responsiveness on the web site.

One thing I could do to improve this even more would be to put Varnish in front of Nginx and serve all static content – including rendered PHP – from memory, which would give some seriously (<100ms) fast performance on read-only WordPress pages when users are not logged in. I may do that if traffic continues to rise, but for the moment the combination of Nginx's static file speed with offloading most of my site’s static files to a content delivery network (CDN) is giving me the performance I want to see.