Application deployment and testing strategies  |  Cloud Architecture Center  |  Google Cloud (2024)

Last reviewed 2023-07-20 UTC

This document provides an overview of commonly used applicationdeployment and testing patterns. It looks at how the patterns work, the benefitsthey offer, and things to consider when you implement them.

Suppose you want to upgrade a running application to a new version. To ensurea seamless rollout, you would typically consider the following:

• How to minimize application downtime, if any.
• How to manage and resolve incidents with minimal impact on users.
• How to address failed deployments in a reliable, effective way.
• How to minimize people and process errors to achieve predictable,repeatable deployments.

The deployment pattern you choose largely depends on your business goals. Forexample, you might need to roll out changes without any downtime, or roll outchanges to an environment or a subset of users before you make a featuregenerally available. Each methodology discussed in this document accounts forparticular goals that you need to meet before a deployment is deemed successful.

This document is intended for system administrators and DevOps engineers whowork on defining and implementing release and deployment strategies for variousapplications, systems, and frameworks.

Deployment strategies

When you deploy a service, it's not always exposed immediately to users.Sometimes, it's only after the service is released that users see changesin the application. However, when a service is released in-place, deploymentand release occur simultaneously. In this case, when you deploy the new version,it starts accepting production traffic. Alternatively, there are deploymentstrategies for provisioning multiple service versions in parallel. Thesedeployment patterns let you control and manage which version receives anincoming request. Read Kubernetes and the challenges of continuous software delivery for more information on deployments, releases, and related concepts.

The deployment patterns discussed in this section offer you flexibility inautomating the release of new software. What approach is best for you dependsupon your goals.

Recreate deployment pattern

With a recreate deployment, you fully scale down the existing applicationversion before you scale up the new application version.

The following diagram shows how a recreate deployment works for an application.

Version 1 represents the current application version, and Version 2 representsthe new application version. When you update the current application version,you first scale down the existing replicas of Version 1 to zero, and then youconcurrently deploy replicas with the new version.

Key benefits

The advantage of the recreate approach is its simplicity. You don't have tomanage more than one application version in parallel, and therefore you avoidbackward compatibility challenges for your data and applications.

Considerations

The recreate method involves downtime during the update process. Downtime isnot an issue for applications that can handle maintenance windows or outages.However, if you have mission-critical applications with high service levelagreements (SLAs) and availability requirements, you might choose a differentdeployment strategy.

Rolling update deployment pattern

In a rolling update deployment, you update a subset of running applicationinstances instead of simultaneously updating every application instance, asthe following diagram shows.

In this deployment approach, the number of instances that you updatesimultaneously is called the window size. In the preceding diagram, therolling update has a window size of 1. One application instance is updated at atime. If you have a large cluster, you might increase the window size.

With rolling updates, you have flexibility in how you update yourapplication:

• You can scale up the application instances with the new version beforeyou scale down the old version (a process known as a surge upgrade).
• You can specify the maximum number of application instances that remainunavailable while you scale up new instances in parallel.

Key benefits

• No downtime. Based on the window size, you incrementally updatedeployment targets, for example, one by one or two by two. You directtraffic to the updated deployment targets only after the new version of theapplication is ready to accept traffic.
• Reduced deployment risk. When you roll out an update incrementally,any instability in the new version affects only a portion of the users.

Considerations

• Slow rollback. If the new rollout is unstable, you can terminatethe new replicas and redeploy the old version. However, like a rollout, arollback is a gradual, incremental process.
• Backward compatibility. Because new code and old code live side byside, users might be routed to either one of the versions arbitrarily.Therefore, ensure that the new deployment is backward compatible; that is,the new application version can read and handle data that the old versionstores. This data can include data stored on disk, in a database, or as partof a user's browser session.
• Sticky sessions. If the application requiressession persistence,we recommend that the load balancer supportsstickinessand connection draining.Also, we recommend that you invoke session-sharing when possible (throughsession replication or session management using a datastore) so that thesessions can be decoupled from underlying resources.

Blue/green deployment pattern

In a blue/green deployment (also known as a red/black deployment), you performtwo identical deployments of your application, as the following diagram shows.

In the diagram, blue represents the current application version and greenrepresents the new application version. Only one version is live at a time.Traffic is routed to the blue deployment while the green deployment is createdand tested. After you're finished testing, you route traffic to the new version.

After the deployment succeeds, you can either keep the blue deployment for apossible rollback or decommission it. Alternatively, you can deploy a newerversion of the application on these instances. In that case, the current (blue)environment serves as the staging area for the next release.

Key benefits

• Zero downtime. Blue/green deployment allows cutover to happenquickly with no downtime.
• Instant rollback. You can roll back at any time during thedeployment process by adjusting the load balancer to direct traffic back tothe blue environment. The impact of downtime is limited to the time ittakes to switch traffic to the blue environment after you detect an issue.
• Environment separation. Blue/green deployment ensures that spinningup a parallel green environment doesn't affect resources that support theblue environment. This separation reduces your deployment risk.

Considerations

• Cost and operational overhead. Adopting the blue/green deploymentpattern can increase operational overhead and cost because you mustmaintain duplicate environments with identical infrastructure.
• Backward compatibility. Blue and green deployments can share datapoints and datastores. We recommend that you verify that both versions ofthe application can use the schema of the datastore and the format of therecords. This backward compatibility is necessary if you want to switchseamlessly between the two versions if you need to roll back.
• Cutover. If you plan to decommission the current version, werecommend that you allow for appropriate connection draining on existingtransactions and sessions. This step allows requests processed by thecurrent deployment to be completed or terminated gracefully.

Testing strategies

The testing patterns discussed in this section are typically used to validate aservice's reliability and stability over a reasonable period under a realisticlevel of concurrency and load.

Canary test pattern

In canary testing, you partially roll out a change and then evaluate itsperformance against a baseline deployment, as the following diagram shows.

In this test pattern, you deploy a new version of your application alongsidethe production version. You then split and route a percentage of trafficfrom the production version to the canary version and evaluate the canary'sperformance.

You select the key metrics for the evaluation when you configure the canary. Werecommend that you compare the canary against an equivalent baseline and not thelive production environment.

To reduce factors that might affect your analysis (such as caching, long-livedconnections, and hash objects), we recommend that you take the following stepsfor the baseline version of your application:

• Ensure that the baseline and production versions of your applicationare identical.
• Deploy the baseline version at the same time that you deploy the canary.
• Ensure that the baseline deployment (such as the number of applicationinstances and autoscaling policies) matches the canary deployment.
• Use the baseline version to serve the same traffic as the canary.

In canary tests, partial rollout can follow various partitioning strategies.For example, if the application has geographically distributed users, you canroll out the new version to a region or a specific location first. For moreinformation, seeAutomating canary analysis on GKE with Spinnaker.

Key benefits

• Ability to test live production traffic. Instead of testing anapplication by using simulated traffic in a staging environment, you canrun canary tests on live production traffic. With canary rollouts, you needto decide in what increments you release the new application and when youtrigger the next step in a release. The canary needs enough traffic so thatmonitoring can clearly detect any problems.
• Fast rollback. You can roll back quickly by redirecting the usertraffic to the older version of the application.
• Zero downtime. Canary releases let you route the live productiontraffic to different versions of the application without any downtime.

Considerations

• Slow rollout. Each incremental release requires monitoring for areasonable period and, as a result, might delay the overall release. Canarytests can often take several hours.
• Observability. A prerequisite to implementing canary tests is theability to effectively observe and monitor your infrastructure andapplication stack. Implementing robust monitoring can require a substantialeffort.
• Backward compatibility and sticky sessions. As with rolling updates,canary testing can pose risks with backward incompatibility and sessionpersistence because multiple application versions run in the environmentwhile the canary is deployed.

A/B test pattern

With A/B testing, you test a hypothesis by usingvariant implementations.A/B testing is used to make business decisions (not only predictions) based onthe results derived from data.

When you perform an A/B test, you route a subset of users to new functionalitybased on routing rules, as the following diagram shows.

Routing rules often include factors such as browser version, user agent,geolocation, and operating system. After you measure and compare the versions,you update the production environment with the version that yielded betterresults.

Key benefits

A/B testing is best used to measure the effectiveness of functionality in anapplication. Use cases for the deployment patterns discussed earlier focus onreleasing new software safely and rolling back predictably. In A/B testing, youcontrol your target audience for the new features and monitor anystatistically significant differences in user behavior.

Considerations

• Complex setup. A/B tests need arepresentative sample that can be used to provide evidence that one version is better than theother. You need to pre-calculate the sample size (for example, by using anA/B test sample size calculator)and run the tests for a reasonable period to reach statistical significanceof at least 95%.
• Validity of results. Several factors can skew the test results,including false positives,biased sampling,or external factors (such as seasonality or marketing promotions).
• Observability. When you run multiple A/B tests on overlappingtraffic, the processes of monitoring and troubleshooting can be difficult.For example, if you test product page A versus product page B, or checkoutpage C versus checkout page D, distributed tracing becomes important todetermine metrics such as the traffic split between versions.

Shadow test pattern

Sequential experiment techniques like canary testing can potentially exposecustomers to an inferior application version during the early stages of thetest. You can manage this risk by using offline techniques like simulation.However, offline techniques do not validate the application's improvements becausethere is no user interaction with the new versions.

With shadow testing, you deploy and run a new version alongside the currentversion, but in such a way that the new version is hidden from the users, as thefollowing diagram shows.

An incoming request is mirrored and replayed in a test environment. This processcan happen either in real time or asynchronously after a copy of the previouslycaptured production traffic is replayed against the newly deployed service.

You need to ensure that the shadow tests do not trigger side effects that canalter the existing production environment or the user state.

Key benefits

• Zero production impact. Because traffic is duplicated, any bugs inservices that are processing shadow data have no impact on production.
• Testing new backend features by using the production load. When usedwith tools such asDiffy,traffic shadowing lets you measure the behavior of your service againstlive production traffic. This ability lets you test for errors, exceptions,performance, and result parity between application versions.
• Reduced deployment risk. Traffic shadowing is typically combinedwith other approaches like canary testing. After testing a new feature byusing traffic shadowing, you then test the user experience by graduallyreleasing the feature to an increasing number of users over time. No fullrollout occurs until the application meets stability and performancerequirements.

Considerations

• Side effects. With traffic shadowing, you need to be cautious inhow you handle services that mutate state or interact with third-partyservices. For example, if you want to shadow test the payment service for ashopping cart platform, the customers could pay twice for their order. Toavoid shadow tests that might result in unwanted mutations or otherrisk-prone interactions, we recommend that you use either stubs orvirtualization tools such asHoverfly instead of third-party systems or datastores.
• Cost and operational overhead. Shadow testing is fairly complex toset up. Also, like blue/green deployments, shadow deployments carry costand operational implications because the setup requires running andmanaging two environments in parallel.

Choosing the right strategy

You can deploy and release your application in several ways. Each approach hasadvantages and disadvantages. The best choice comes down to the needs andconstraints of your business. Consider the following:

• What are your most critical considerations? For example, is downtimeacceptable? Do costs constrain you? Does your team have the right skills toundertake complex rollout and rollback setups?
• Do you have tight testing controls in place, or do you want to test thenew releases against production traffic to ensure the stability of therelease and limit any negative impact?
• Do you want to test features among a pool of users to cross-verify certainbusiness hypotheses? Can you control whether targeted users accept theupdate? For example, updates on mobile devices require explicit user actionand might require extra permissions.
• Are microservices in your environment fully autonomous? Or, do you have ahybrid of microservice-style applications working alongside traditional,difficult-to-change applications? For more information, seedeployment patterns on hybrid and multi-cloud environments.
• Does the new release involve any schema changes? If yes, are the schemachanges too complex to decouple from the code changes?

The following table summarizes the salient characteristics of the deploymentand testing patterns discussed earlier in this document. When you weigh theadvantages and disadvantages of various deployment and testing approaches,consider your business needs and technological resources, and then select theoption that benefits you the most.

Best practices

In order to keep deployment and testing risks to a minimum, application teamscan follow several best practices:

• Backward compatibility. When you run multiple application versionsat the same time, ensure that the database is compatible with all activeversions. For example, a new release requires a schemachange to the database (such as a new column). In such a scenario, youneed to change the database schema so that it's backward compatiblewith the older version. After you complete a full rollout, you can removesupport for the old schema, leaving support only for the newest version.One way to achieve backward compatibility is to decouple schema changesfrom the code changes. For more information, seeparallel change anddatabase refactoring patterns.
• Continuous integration/continuous deployment (CI/CD). CI ensuresthat code checked into the feature branch merges with its main branch onlyafter it successfully passes dependency checks, unit and integration tests,and the build process. Therefore, every change to an application is testedbefore it can be deployed. With CD, the CI-built code artifact is packagedand ready to be deployed in one or more environments. For more information,seebuilding a CI/CD pipeline with Google Cloud.
• Automation. If you continuously deliver application updates to theusers, we recommend that you build an automated process that reliablybuilds, tests, and deploys the software. We also recommend that your codechanges automatically flow through a CI/CD pipeline that includes artifactcreation, unit testing, functional testing, and production rollout. By usingautomation tools such asCloud Build,Cloud Deploy,Spinnaker,andJenkins,you can automate the deployment processes to be more efficient, reliable,and predictable.
• IaC and GitOps. If you need to manage complicated deployment and testingstrategies, consider using Infrastructure as Code (IaC) and GitOpstools. Using IaC withTerraform andConfig Connector can help you use declarative language to define yourinfrastructure and strategies. Using GitOps withConfig Sync,andArgo CD.can help you manage your code with git.
• Rollback strategy. Sometimes, things go wrong. We recommend creating a rollback strategy to follow when the unexpected occurs. Having a reliable rollback strategy can helpadministrators and DevOps engineers manage risk. You can create a rollback strategy byusing a platform that supports rollbacks as built-in feature, such asApp Engine,andCloud Run.To support your rollback needs, you can also use release automation tools likeCloud Deploy,Spinnaker,andArgo RollOuts.
• Post-deployment monitoring. To monitor critical metrics and to alertthe responsible team when a deployment or test fails, build a monitoringsystem usingGoogle Cloud Observability.You can also enable automated rollbacks for deployments thatfail health checks. UsingError Reporting,Cloud Trace andCloud Profiler can help you find the cause of simple and complex post-deployment issues.

What's next

• Read about the concepts of this document inKubernetes and the challenges of continuous software delivery.
• Learn more about otherDevOps solutions on Google Cloud.
• Learn more aboutcontinuous integration and delivery on GKE.
• Learn more aboutCRE life lessons for reliable releases and rollbacks.
• Explore reference architectures, diagrams, and best practices about Google Cloud.Take a look at ourCloud Architecture Center.