Deploying services without downtime
n today’s SAAS world it is a common requirement to deploy applications or/and services without downtime. There’re various techniques and patterns that can be used to satisfy it, e.g. blue green deployment, as explained by Martin Fowler – The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible. At any time one of them, let’s say blue for the example, is live. As you prepare a new release of your software you do your final stage of testing in the green environment. Once the software is working in the green environment, you switch the router so that all incoming requests go to the green environment – the blue one is now idle.
For the majority of our projects we use the open source continuous delivery server go.cd and AWS. In this blog post I wanted to share, a possible implementation approach in the context of a Continuous Delivery pipeline, implemented with GO.CD and AWS EC2 instances.
The above approach ensures that always there will be a tested version of the application/service under the AWS ELB that should handle user requests. This is a vital task for critical services where downtime is not acceptable.
Implementation
Assuming we have 2 small server farms, first: Staging-1 and Staging-2 servers and second: Production-1 and Production-2 servers and existing Continuous Delivery pipeline: Creating three stages in GO.CD that are going to do the deployment on the servers under a particular AWS ELB sequentially is required. Stages in GO.CD are executed in sequence.
First stage contains only one job with the following tasks:
- Detach – it detaches the first machine – Staging-1 (respectively Production-1 when we decide to apply this technique on Production) from the AWS ELB
- Deploy – it deploys the new version of the app/service on the first machine Staging-1
Second stage contains only one job with a single task:
- Attach– returns back the first machine – Staging-1 to the AWS ELB
Third stage contains only one job with the following tasks:
- Detach – the second machine Staging-2 (respectively Production-2 when we decide to apply this technique on Production) from AWS ELB
- Deploy – the new version of the API on the second machine Staging-2
- Attach – back the second machine Staging-2 to the AWS ELB
A sleep task of 60 seconds follows each GO.CD task for attaching/detaching a machine to/from the AWS ELB, should give the AWS ELB enough time to reconfigure internally so it can properly handle the user requests at that moment.
* Information about how to detach a machine from an AWS LB can be found in AWS Tools for Windows PowerShell Documentation
** Information about how to attach a machine to an AWS LB can be found in AWS Tools for Windows PowerShell Documentation
Testing
The above implementation can be tested by setting up a JMeter Test Plan to fire requests against the AWS ELB to verify if the stages completed successfully
Marian is experienced DevOps/.NET (WinForms) developer. For the past couple of years he has built (developed), automated, evolved and maintained the product continuous delivery workflow for 10 products in international technology company specialized in user interface component software development. He has deep knowledge in Team Foundation Server administration and deployment, MSBuild and PowerShell scripting, WiX and Windows Installer Technology and Software Architecture. As a part of the R&D Solutions team Marian is focused on SOA (Service Oriented Architecture), EI (Enterprise Integration), SSO (Single Sign On) and Automation software solutions. Marian holds a Bachelor degree in Computer Systems & Technologies from Sofia Technical University.