Always Get Better

Archive for the ‘Windows Azure’ Category

Speed up Deployments in Azure Web Apps

Saturday, March 4th, 2017

Is it possible to speed up deployments in Azure Web Apps? Without resorting to temporary fixes?

My biggest pain point with Azure is the length of time it takes to push changes. My e-commerce project has a simple architecture with a smallish number of dependencies but takes over 35 minutes to complete a build. Although there’s no downtime during the process, you don’t want to sit around waiting. Fortunately it’s easy to speed up deployments in Azure.

Problem: Starting from a Bad Place

Here’s what I’m dealing with: 35 minutes to push each commit.

35 minutes - yikes! Time to speed up deploymentsKeep in mind – this is a simple website. .NET Core MVC6 with some NuGet dependencies. On a local system it takes about 3 minutes to go from a fresh clone to a running app.

When you factor in configuration differences and thing you need to check off when setting up a live environment, that 35 minutes makes this project pretty unwieldily.

The problem stems from the way Azure deploys apps to the service fabric. Everything in your app is shared with mapped network folders which is why things like session state and file uploads are available to every instance. That’s great if you have a complicated stateful app setup, but we’re building totally RESTful services (right?) so we don’t need that feature.

Building from a network drive is a problem for web apps because our npm and bower repositories generate tons of tiny files that absolutely suck to synchronize across a network drive.

Solution: Speed Up Deployments by Skipping the Network

Instead of waiting for the same files to synchronize across the network, use the Git repository on the local machine. This skips the network latency and does all the deployment work on the local machine.

Using local cache for Git deployments

In your Application Settings, add a key for SCM_REPOSITORY_PATH with the value D:\local\repository.

Now my deployments are 10 minutes – a big improvement.

Speed up deployments on Azure by using local drive for buildsStore the Right Things

10 minutes is a big improvement but I know we can do better. Looking through my project I find 100MB of content images that should really be part of a CDN. Now my repo is only 8MB and the deployment time is slashed again.

Respectable build times5 minutes, now that is a respectable build time.


Windows Azure Thoughts – First Six Months

Tuesday, August 2nd, 2011

Having played with Windows Azure for about six months or so, I think I have a good handle on its pros and cons for the tasks I’ve been trying to do. I definitely have a lot of positive conclusions about the platform as a whole, and a few pain points I can see Microsoft working hard to eliminate.

Starting with the good:

Seamless Deployments – No Client Downtime
One of the trickiest things to set up while deploying a web cluster is the ability to deploy new builds without causing downtime, and reverting to the last build version in the event of a discovery of some critical flaw.

Azure really takes the pain out of this part of the process. Just deploy to staging, test the staging area, and hit ‘Swap VIP’ to reverse the staging & production instances. This is a great way to increase and decrease capacity as long as the end points remain the same (same number of ports, etc). (Remember, you need to have 2 or more instances in order to invoke the SLA.)

Scalable Computing as it is Meant to Be
Most of the developer confusion I have encountered during these first months of development has turned out to be related to scalable programming in general and not Azure programming in particular. I haven’t been training people to use Azure so much as I have been promoting the importance of properly separating aspects of code to make use of data sources in a non-limiting way. Caching, database access, blob storage, table storage are not new concepts, but using them from the ground-up in every project has been new to most people.

We can’t develop with one web server in mind and then scale out, we now have to think about scaling right from the beginning, which is resulting in a much cleaner architecture overall. Developers become more connected to the practical results of their decisions when they have to choose whether to use NoSQL, SQL or both for data access based on factors ranging from speed to overall cost.

Tight Code Cohesion
Working with Azure for me has meant a return to .NET and C# programming. Combined with the ASP.NET MVC 3 framework and latest SQL database versions, the experience has been overwhelmingly fun. It has been great to have code that just works nicely in an environment that has been clearly optimized for exactly what I’m trying to do. Not having to deal with server setup and maintenance has freed me to dive more deeply into the capabilities of the system without needing to worry about the configuration. In short, I’m loving this.

And the bad:

Long Deploy Times
At one point our build times were pushing 45 minutes. This is a huge problem when it is 11 o’clock at night on launch day and you have a team of developers waiting around in order to verify their fixes.

During development the situation can be made easier with incremental pushes; since each Azure instance is a VM, it isn’t hard to upload the changed files through RDP and see changes immediately. For devving this is fine, but the changes are not persistent therefore the full deploy process of uploading the cspack file, provisioning instances and starting up is needed when moving production code into a live environment.

The process can definitely be sped up by reducing the overall project size; at one point we had several hundred megabytes of supporting files which didn’t need to be part of our project. Removing these basically eliminated our upload times, but the web role instantiation still clocks in at 15-20 minutes; a long time when you want to go home to your family.

Slow System Responses
When I say slow system response, I am referring to the management API which hits the AppFabric, not the response times of our applications which have been incredibly strong.

Any time I manage our Azure account, I find myself waiting around for a lot of information. During deployments especially, the status messages are unhelpful ‘Creating deployment’, ‘Initializing instance’, ‘Instance busy’, etc. This is really frustrating given the long deployment times; having more transparency into the process would be a real confidence builder.

Unclear Management Console
This is a bit nit-picky; the Silverlight-based management console is pretty cleanly laid out. Some of the configuration options are not clear, though. For example, to grant API access to your applications, you need to upload your certificate to the ‘Management Certificates’ section, which is separate and unrelated to the Certificates folder subordinate to each hosted service. This makes sense for veterans of the system as both certificate groups serve different roles, but new users are easily confused by the distinction and can spend hours trying to figure out how to deploy their projects from Visual Studio.

Similarly, after uploading your certificate, you can get the subscription ID by clicking on the subscription and copying the ID from the info panel at the right-hand side of your screen. If you right-click on the subscription ID in the menu pane and choose the ‘Copy’ option, it copies the creation date & name of the subscription. Hardly a major problem, but potentially a stumbling block for developers new to the platform.

Ensure SLA with Multiple Web Role Instances on Windows Azure

Monday, April 25th, 2011

As I dig deeper into the Windows Azure platform, I am becoming more and more impressed by the potential it offers. Microsoft has put a lot of money and resources into developing their infrastructure and have done an incredible job at creating an interesting and powerful architecture. If anything, their major fault is in advertising – with so many different technologies with ever-changing names, it’s hard for a newcomer to wrap their mind around the services. It is pricier than some options so it’s hard to really experiment in much depth, although they do offer a free option for developers to dig in and try the services.

Ensuring 99.95% SLA
One potential ‘gotcha’ is the SLA: it differs between their difference services but hovers in the 99.9% range. Most account properties include automatic scaling and failover, but Web Roles and Worker Roles do not – in order to qualify for the 99.95% SLA you need to deploy your solution to two or more instances. By doing that, you ensure that your application is load balanced across multiple fault domains therefore will remain available even during a failure.

Of course, this doubles the cost – each compute instance is billed separately.

Fault Domain vs Upgrade Domain
Azure’s logical server space is segmented into fault domains and upgrade domains. Translated into physical terms, a fault domain refers to a single rack containing multiple computers, while an upgrade domain is one or more computers that receive software upgrades at the same time. Multiple upgrade domains may exist within any fault domain.

For our application to remain online, we need to consider where it is hosted. Although the fault and upgrade domains are largely abstracted away from developers, the SAL guarantees that 2 or more instances are split across two or more upgrade and fault domains. At a minimum, this means if you have two Web Role instances they will exist on different computers in different racks. Beyond that, Azure decides where instances are initialized and hosted from.

High Availability Concerns
Now let’s consider our application which has been split across two instances. We are guaranteed to have 99.95% network availability to our application, but there are more factors to consider.

When one of the instances goes offline – due to a server fault or OS upgrade – Azure automatically starts it up on a working server. As long as the remaining instance is able to handle all of your application’s needs, this process is transparent and results in no performance degradation.

What happens if your application is spread across two instances, each under 70% load? The response times might be good, but if one of the instances goes offline even temporarily your remaining healthy deployment will trying to handle 140% of the overall traffic. If you want your business to stay on line, you need to plan for these periods of downtime.

This is where the abstraction hurts – we can’t control the number of fault domains in our cluster but we can control the number of upgrade domains (essentially forcing each instance to load up on a new machine). Worst case scenario is half of your cluster dies when the Azure fault domain goes down – something that should never happen, but anything can happen online.

Note, however, that this kind of planning is what’s needed in general cases when your traffic might suddenly jump due to customer interest. We’re not just mitigating against host failure, but also against reputation damage from inadequate planning.

Keep Sessions in Mind
One last thing to mention: when we split our application into multiple instances, we need to consider how we handle session data. Since Azure uses a round robin situation for their load balancing meaning the end user could potentially hit a different instance each time they access a resource on your site. If you are using traditional session handlers, session data is not shared between the instances so user data is not guaranteed to be persisted on each request.

This is actually huge benefit to cloud computing, and something I’ve praised other frameworks for providing seamless support for. Our web application should be designed to avoid sharing data where possible. If session data is needed, we can use AppFabric’s session cache providers to provide fast network-based session management.

Azure Table Storage vs Azure SQL

Sunday, April 24th, 2011

There is a lot of debate among newcomers to Azure whether to use Azure SQL or file-based Table Storage for data persistence. Azure itself does not make the differences very clear, so let’s take a closer look at each option and try to understand the implications of each.

Azure SQL
Azure SQL is very similar to SQL Server Express. It is meant as a replacement for SQL Server Standard and Enterprise, but the feature set is not there yet. Reporting services, in particular, are in community technology preview (CTP) phase and at the time of writing are not ready for prime time.

Programmatically, Azure SQL is very similar to SQL in existing web applications; any ODBC classes you already wrote will work directly on Azure with no changes.

Size and cost are the major limitations with Azure SQL in its current incarnation. The largest supported data size is 50GB which runs for $499/month. Any databases larger than 50GB would need to be split across multiple instances – this requires knowledge of sharing at the application level and is not easy surgery to perform.

Table Storage
Table storage uses a key-value pair to retrieve data stored on Azure’s disk system, similar to the way memcache and MySQL work together to provide the requested data at fast speeds. Each storage container supports 100TB of data at incredibly cheap ($0.15/GB) rates.

Working with table storage involves accessing them directly from your application differently than you may be accustomed to with SQL. Going all-in ties you to the Azure platform – which is probably not a problem if you’re already developing for Azure as you will likely be trying to squeeze every ounce of performance out of all areas of the platform anyway.

Table storage does not support foreign key references, joins, or any of the other SQL-stuff we usually use. It is up to the programmer to compensate by making wide de-normalized tables and build their lists in memory. If you’re already building clustered applications, this is not a new design pattern as developers typically want to cache data in this manner.

Besides the larger space limits, table storage affords us automatic failover. Microsoft’s SLA guarantees we will always be able to access the data, and this is accomplished by replicating everything across at least three nodes. Compared to self-managing replication and failover with the SQL service, this is a huge advantage as it keeps the complexity out of our hands.

Difference in Evolution
If Azure SQL seems somewhat stunted compared to Table Storage, it’s not an accident: it is a newcomer who was not planned during the original construction of the Azure platform. Microsoft carefully considered the high-availabilty trends used for application development and found that the NoSQL way would most easily scale to their platform. Developer outrage prompted the company to develop Azure SQL, and its service offerings are improving rapidly.

Depending on your storage needs, your course of action may be to store as much data for cheap in Table Storage, and use SQL to index everything. Searching the SQL database will be incredibly fast, and can be done in parallel with any loads against persistent tables – everybody wins.