Always Get Better

Never stop looking for ways to improve

October 3rd, 2011

Today Adobe announced its acquisition of Typekit, a web font hosting service that allows designers to use any typeface with their sites rather than relying on standard “safe” font families.

This is an interesting development. Adobe’s Flash player already supports proprietary fonts, which suggests that the company is looking at alternate technologies for its future development. Obviously Flash will remain relevant for more time to come, but as competitors increasingly jump onto the HTML5 bandwagon, Adobe is wise to increase its arsenal of standards-compliant technology.

August 6th, 2011
Mochila Firefox
Creative Commons License photo credit: jmerelo

So I got tired of using Firefox 3.6 in my Ubuntu machine and decided to upgrade to the newest version (5.0). It’s understandable that the package maintainers responsible for Ubuntu don’t put bleeding-edge cutting-edge releases in the distribution due to the possibility of introducing unstable elements into the user experience. But Firefox 4 has been out for over a year, and the migration to 5 is well underway.

Fortunately, it couldn’t be much easier to get the newest official release using our good friend aptitude.

In a terminal window, add the Mozilla team’s stable Firefox repository by issuing the following command:


sudo add-apt-repository ppa:mozillateam/firefox-stable

Next, perform an update to get the package listing, and upgrade to install the newest browser:


sudo apt-get update
sudo apt-get upgrade

That’s it – you’re done! Your shortcuts are even updated, and any bookmarks or open tabs you might have had on the go are carried forward.

I was pleasantly surprised at how easy this process was.

April 23rd, 2011
Fixed
Creative Commons License photo credit: Don Fulano

Amazon is in the news today for the failure their Elastic Block Storage (EBS) service suffered, resulting in loss of service and/or extreme latency for hundreds of sites including some of their largest customers like Foursquare and reddit. AWS has been widely regarded as the most stable and overall leader of the cloud providers, so it was a great shock to many observers that they were able to suffer such a large failure.

I think the failure is not surprising, but rather the fact that it hasn’t happened before now is surprising. It underscores my message that cloud computing is not magical but is in fact an abstraction over very real hardware. There are bound to be flaws and issues just as with “real” hosting options, the difference is the end customer has less control over the hardware, hosting and networking environment.

Not every business can afford the overhead of a large dedicated solution, so what to do?

Spread the Load
The key is redundancy. Start by spreading your content across the internet rather than relying on single server to cough up all of your visitors’ needs. Things like content delivery networks (CDNs) will reduce the incoming load on the server and help it stay online.

How can we tell if a website is offloading the right amount of content? Perform regular speed testing and identify problem areas using tools like YSlow.

Redundancy! Eliminate Single Points of Failure
Whenever you have a single system servicing part of your application, you expose the entire application to failure.

For example, suppose you have four Apache servers and a load balancer sending equal traffic to each. If one of the Apache servers fails, the other three are able to compensate for the loss with no downtime for your visitors. But what happens if the load balancer fails? Even though all four web servers are in fine working order, your site is knocked offline.

Some systems are difficult to cluster: replication schemes in the various SQL servers are a huge drain on performance – newer solutions like MySQL Cluster or DrizzleDB aim to solve this problem, but at extra expense in terms of configuration and application design.

The key to successful redundancy is in scripting your software in such a way that failures can be recovered from fast and automatically. Having a hot spare in the group isn’t useful if you need to reach an administrator at 4am to activate – by that point you’ve already lost your overseas customers for the day.

Twilio has an excellent summary of the engineering process that goes into creating a scalable cloud-ready infrastructure.

Avoid the Cloud? Never
Despite some public failures, cloud computing has not suffered any kind of blow. Large organizations will always want their own private non-cloud hosting, small sites will always be looking for an inexpensive VWS. The middle-tier which is serviced by the cloud will continue to see cost savings that greatly outweigh any physical hosting options available at that level.

Because of the low server cost, cloud computing allows smart customers the freedom to build necessary redundancy without breaking the bank. Even though this pays off big time when catastrophic failures happen, there are longer term benefits of improved overall response times to the end users even when the hosting is working well.

April 16th, 2011
footprint
Creative Commons License photo credit: orinoko42

I like redundancy, to a fault. Part of it goes to my need for comprehensive backup – as long as you have a backup prepared, you are less likely to lose anything. So it stands to reason that if you have two identical copies of your web site running, you are more tolerant to all kinds of failures – from your web server going down to an unexpected surge in traffic.

Protection Against Hardware Failure
If your server experiences hardware issues, nothing you can do (short of changing the faulty components) will keep your site online. If you have two web servers and one of them fails, the other can still pick up the slack.

This is an important consideration when you’re using a cloud provider. In many cases you only occupy a small portion of the hardware you are hosting on, which means if someone else messes up their VM instance it can potentially affect your application. Spreading across to two VMs increases the odds that you end up on different physical hardware, perhaps even a different server rack altogether. If your provider performs maintenance on one of their servers you will not necessarily get knocked offline.

Protection Against Traffic
Even if you’re on a strong fast server, you can only support a certain number of concurrent users. A load balancing situation can reduce the average load per machine in your cluster, which improves the overal response rate, and eliminates the ‘queuing’ problem you would experience with a large numbers of users all trying to hit a particular machine.

April 12th, 2011

One of my favourite aspects of the cloud is the ease with which we can create new VMs to test our wacky architecture theories. It’s so easy (and cheap!) to spin up a small server cluster for some serious load testing, and then destroy it again when done.

If nothing else, it provides a safety net and teaches you how to squeeze every ounce of performance out of big and small server instances. Let’s examine ways in which we can make our dynamic Apache settings much faster.

Turn Off Modules You’re Not Using
This should be fairly obvious, but Apache ships with a number of modules which can affect performance but which most of us never need. Check your /etc/apache/mods-enabled folder to see what can be removed.

Never Trust Defaults
The default Apache settings are optimized for a website serving static files only. Booorrring! Never be afraid to question what you see in the configuration files; the more you understand about the inner workings of the system, the better you will be able to improve its performance.

RAM is good, Swap is Bad
Running out of physical memory (RAM) and hitting the hard drive’s swap space is bad, especially in the Virtual Machine world. When this happens your performance will nose dive; your machine may even crash. The simplest solution is to increase the amount of RAM available to your server, but if that is too costly or impossible, read on.

Kill the KeepAlive
Whenever a request is made to the web server, it keeps the network connection open for a small amount of time (often 15 seconds). During that time, if the visitor’s web browser needs to get another file, it goes through the same connection thereby avoiding wasting time re-connecting to your server. The problem is the open connection will use up space in your connection pool so if your site is under heavy load new visitors will get queued up and may experience slowdowns trying to access your content.

If Apache is your front-end web server, set the KeepAliveTimeout to 2 seconds. This will keep the number of requests fluid even under heavy load.

If your server is behind a firewall like nginx or HAProxy where KeepAlives are not honoured, turn this setting off entirely.

Don’t Serve Static Files
Apache is a memory hog. Since each hit to the server is relatively heavy in terms of threads and memory, we are in better shape when we serve non-changing static content like images, stylesheets and javascript using a single-threaded server like nginx or lighttpd or even a memory server like varnish (bonus points for using a CDN to serve static files, avoiding the hit to your server at all).

Turn off HostnameLookups
This should already be done by default in your Apache configuration; if it isn’t, do it now. When HostnameLookups is on, Apache checks every incoming request’s IP address for its host name. This can dramatically increase your latency, and isn’t healthy for DNS servers either.

Disable AllowOverride
It is tempting to set AllowOverride to All in order to give your .htaccess files free reign to do as they please. The downside of this directive is that every time anything is requested Apache will need to check that folder and every one of its parents all the way down to the site root in order to check for .htaccess commands. Apache recommends setting AlloverOverride to none globally, enabling access for .htaccess files that can’t be set in the site configuration.

April 11th, 2011

Play framework must be the best-kept secret in the Java world. If you haven’t had a chance to see their totally awesome demonstration video where they build a full app before your eyes in a matter of minutes, go – go now. Then come back.

Why do I like this framework so much? Put simply, it is an elegant solution for nearly every problem I’ve ever run into in developing websites, both single-server and multi-server applications. Don’t take my word for it, see for yourself.

The Goodness of Java
To my mind, Java has the edge over more common scripting languages (like PHP) because it is compiled (fast) and statically typed (reliable). In the past, using compiled languages on the web was only possible if you were using ASP.NET or willing to put up with the hassles of existing Java frameworks and servers.

Play’s first innovation comes from wrapping the Java runtime inside a Python web server; using a Play application is as easy as running a command line script and connecting with your web browser. Play’s second innovation is its just-in-time compilation and display of error messages; if you make a mistake you will know in the amount of time it takes to hit refresh on your web browser.

Since it IS java, programmers can use libraries they have built for other applications or sourced from other vendors and plug directly into their code. This is one of the advantages Microsoft has had going for it and it is good to see it implemented so nicely in the open source world.

The Ease of Rails
Love it or hate it, Ruby on Rails has had an affect on the entire web world and its reach is definitely felt in the Play framework. Everything from the routing to JPA integration has that minimal-configuration design that is so prevalent in the Ruby world. Play has the edge though, due to Java annotations and the extra control you get as a developer.

Baked-in Unit Testing
Admittedly this is the first thing that drew me to the Play framework. Unit testing has to be one of the most important aspects of good programming; in fact, if your code is not covered by unit tests, I argue it is incomplete. Play has terrific support for unit testing, functional testing and selenium-based web testing. In version 1.1, Play added a headless web testing mode, paving the way to run framework applications in the context of an automated build environment – smart move!

Although awkward at first, using YAML files for database fixtures makes a lot of sense. Managing database access in unit tests has always been a challenge but thanks to the in-memory database server and fixture files Play offers us database integration testing – giving us the fresh-start benefits of mock frameworks along with the soundness of mind that comes from knowing you are testing the real database.

Share Nothing Architecture
Call it laziness, call it human error. At some point in the development cycle, the session always seems to end up carrying user data around. Even with data-sharing applications like memcached, that style of development does not scale well. With Play, sessions are stored in user cookies and consist of an encrypted key. The idea is the application takes care of loading any additional information it needs from this seed information, so the web cluster can be expanded to hundreds of nodes or reduced to a single server with no performance penalties on the other servers. Each Play instance operates as if it is the only one in existence, making it far easier to support complex site architectures.

April 5th, 2011

Back in 2009 I was tired of hearing the phrases “cloud computing” and “in the cloud”. These days I’m so numb to their meaninglessness that it doesn’t even phase me anymore. Somewhere along the way marketers took over the internet and ‘social media’ became a job position.

So what do I have against cloud computing? Would I rather build servers, deal with co-location, and suffer massive downtimes in order to change hardware specs? Of course not.

Let’s not lose sight of the big picture: virtualized servers are still servers. From a remote perspective the management is all the same and from a hardware perspective you still need to be responsible for your data in the event of a catastrophic failure.

While I am a huge proponent of “cloud” providers like Rackspace (heck I host all of my web sites on Cloud Server instances), let’s call a spade a spade: there is nothing magical about servers in the cloud, they are just virtualized instances running on a massively powerful hardware architecture.

Why go with virtualization over a dedicated box? Virtual servers are cheap – I don’t need to incur the startup costs that I would from a dedicated server. For a small business this is a huge deal; for larger business with intense data needs the dedicated solution will always provide the most security but for anything from tiny, small to very large applications the virtualized way is the ticket. Add more servers, remove them, reconfigure: you don’t get that kind of flexibility from traditional server hosting.

Long live cloud computing; but the name has to go. Did the term come from network diagrams where the Internet was represented as a cloud? I don’t think it’s a particularly clever analogy to consider your business assets living as disembodied entities “somewhere” in the networking cloud.

We’re fighting a losing battle if we believe we’re going to get the marketers to back off the internet now. But on the tech side let’s keep calling it what it is and try not to let the marketing buzz cloud our opinion of the technologies we use.