Automation for the people: Deployment-automation patterns, Part 2

More patterns for one-click deployments


Deployment is yet another aspect of software creation that lends
itself well to automation. Automated deployments reap the benefits of a
reliable, repeatable process: improved accuracy, speed, and control. Part 1
of this two-part article describes eight deployment-automation
patterns. In this installment, I expand the discussion to cover seven
more equally beneficial approaches to deployment:

Binary Integrity
, which ensures the same artifact is promoted throughout target environments

Disposable Container
, which puts a target environment into a known state to reduce deployment errors

Remote Deployment
, which ensures that deployments can interface with multiple machines from a centralized machine or cluster

Database Upgrades
, which provides a centrally managed and scripted process to applying incremental changes to the database

Deployment Test
, which uses pre- and post-deployment checks to verify the application is working as expected based on recent deployment

Environment Rollback
, which rolls back application and database changes if a deployment fails

Protected Files
, which controls access to certain files used by a build system

Figure 1 illustrates the relationships among the
deployment patterns covered in this article (the unshaded patterns were
covered in Part 1
):

Figure 1. Deployment-automation patterns



These seven additional deployment-automation patterns build upon the first eight to help you create one-click deployments.

Compile once, deploy to many environments

Name
: Binary Integrity

Pattern
: For each tagged deployment, the same archive (WAR or EAR) is used in each target environment.

Antipatterns
: Separate compilation for each target environment on the same tag.

After numerous debates with colleagues on this topic, I've firmly come down on the side of compile once, deploy to many target environments
rather than compile and package in every target environment
.
For instance, the deployment artifact produced from a Java Web
deployment is the Web archive (WAR) or enterprise archive (EAR) file.
This archive should be checked into the version-control repository and
tagged one time — such as in the DEV environment.

Figure 2 illustrates the compile once, deploy to many
philosophy as the same brewery.war generated on the build machine is deployed to each of the target environments:

Figure 2. The same Web archive deployed to different target environments



Ant provides a

checksum

task — using the Message-Digest algorithm 5 (MD5) hashing algorithm —
to ensure the file that was compiled and packaged on the build machine
is the same one being deployed to each of the target environments.

Some
will argue that although the artifact may be the same, the deployment
configuration is different for each target environment. That is, when
you use a Single-Command, Scripted Deployment, many automated processes
can alter the application's output, regardless of whether it's the same
archive. This is true; however, you could spend needless hours trying
to troubleshoot a problem because the software was compiled and
packaged using a different JDK version on the STAGE environment than
was run in the QA environment. And opportunities for failure arise when
the JARs from a centralized dependency-management repository (such as
Ivy or Maven) that were used in DEV are different from those in the
staging environment. These risks convince me that in order to ensure
binary integrity, I must compile and package once so that I can deploy
to many environments.

Back to top

Make deployments cheap with disposable containers

Name
: Disposable Container

Pattern
: Automate the installation and configuration of Web and database containers by decoupling installation and configuration.

Antipatterns
: Manually install and configure containers into each target environment.

In an earlier Automation
for the people

installment, "Continuous Integration anti-patterns, Part 2
,"
you learned why cleaning up a "polluted" environment helps prevent
falsely positive or negative builds. The Disposable Container reduces
many of the problems that can occur when you rely on persistent
containers. The Disposable Container pattern is based on two
principles: completely remove all container components
and separate the container installation from its configuration
.
This seems like a radical concept to some, particularly systems
engineers, because it no longer assumes that containers should be
managed, and obfuscated, by a separate team, never to be touched by
developers or others. However, considering the common and costly
problems that occur during deployments, it can be an area where all
team members can realize the most benefit.

One-click deployments

Many times I've met with teams that have told me "Yep, we've got automated
deployments." When I ask a few simple questions — such as "Are you able to
type a single command (such as

ant

) to generate a working
software application?" — the response usually goes something like: "Yes, once
you install and configure the Web container..." or "Yes, once you set up the
database." My definition for a truly automated deployment is that you should be able
to start with a clean machine, install the Java platform and Ant (there are ways to
eliminate this step as well), and then type a single command
to get a working software application. If you can't, it's not
"one-click," and costly human bottlenecks will occur in the deployment
process.

The Disposable Container pattern, shown in Figure 3, is grounded in a philosophy that everything
should be in the system
— (using the Repository pattern covered in Part 1
) — not in someone's head:

Figure 3. Removing and installing container(s) during deployment



The
Ant script in Listing 1 downloads the Tomcat ZIP from the Internet,
removing any container remnants from previous deployments, then
extracts, installs, and starts Tomcat:

Listing 1. Scripted Deployment, in Ant, that removes, reinstalls, starts, and configures a container

<!-- Check to see if Tomcat is running prior to this --> ... <exec executable="sh" osfamily="unix" dir="${tomcat.home}/bin" spawn="true"> <env key="NOPAUSE" value="true" /> <arg line="shutdown.sh" /> </exec> <delete dir="${tomcat.home}" /> <get src="${tomcat.binary.uri}/${tomcat.binary.file}" dest="${download.dir}/${tomcat.binary.file}" usetimestamp="true"/> <unzip dest="${target.dir}" src="${download.dir}/${tomcat.binary.file}" /> <exec osfamily="unix" executable="chmod" spawn="true"> <arg value="+x" /> <arg file="${tomcat.home}/bin/startup.sh" /> <arg file="${tomcat.home}/bin/shutdown.sh" /> </exec> <xmltask source="${appserver.server-xml.file}" dest="${appserver.server-xml.file}"> <attr path="/Server/Service[@name='${s.name}']/Connector[${port='${c.port}']" attr="proxyPort" value="${appserver.external.port}"/> <attr path="/Server/Service[${name='${s.name}']/Connector[${port='${c.port}']" attr="proxyName" value="${appserver.external.host}"/> </xmltask> <!-- Perform other container configuration --> ... <echo message="Starting tomcat instance at ${tomcat.home} with startup.sh" /> <exec executable="sh" osfamily="unix" dir="${tomcat.home}/bin" spawn="true"> <env key="NOPAUSE" value="true" /> <arg line="startup.sh" /> </exec>

By putting an environment into a
known state and deploying containers in a controlled manner, you reduce
many common deployment errors that are the cause of most deployment
pain.

Back to top

Running commands in multiple external environments

Name
: Remote Deployment

Pattern
: Use a centralized machine or cluster to deploy software to multiple target environments.

Antipatterns
: Manually applying deployments locally in each target environment.

Once database and Web containers have been installed, getting a deployment to run on a
developer's workstation
is usually rather trivial. However, the difference between development
and production is vast. If an organization has multiple projects and
different target environments (for instance, testing or staging
environments), there's often a need to manage deployments centrally
from a single environment: a machine or a cluster. Quite often, teams
use a build server to manage deployments between each of these target
environments. In Part 1
,
I covered the Headless Execution pattern, in which you use public and
private keys so that you don't need to log in manually to each machine.
Remote Deployment, illustrated in Figure 4, relies on the Headless
Execution, Single Command, and Scripted Deployment patterns to make it
easy to deploy to remote machines:

Figure 4. Build-management server to multiple environments



To
deploy software from a centralized build server remotely, you need to
use mechanisms for securely copying and running commands remotely. The
two mechanisms I'll illustrate use Secure Copy (SCP) and Secure Shell
(SSH). From a Scripted Deployment, as shown in Listing 2, a Web archive
that was generated on a centralized build machine is remotely copied to
a target environment:

Listing 2. Securely copying a war file from one machine to another

<target name="copy-tomcat-dist"> <scp file="${basedir}/target/brewery.war" trust="true" keyfile="${basedir}/config/id_dsa" username="bobama" passphrase="" todir="pduvall:G0theD!stance@myhostname:/usr/local/jakarta-tomcat-5.5.20/webapps" /> </target>

After the WAR file is securely copied to the remote target environment, I can use something like the

SSHExec

task in Java Secure Channel to run any SSH commands — remotely from the central build machine. An alternative approach is to

ssh

to the remote environment and run any commands locally. This lessens
the back-and-forth remote traffic and can reduce deployment times.

Back to top

Putting database and data into known state

Name
: Database Upgrade

Pattern
: Use scripts and database to apply incremental changes in each target environment.

Antipatterns
: Manually applying database and data changes in each target environment.

In Figure 5, you see an example of using automated scripts to update the database as part of a Scripted Deployment:

Figure 5. Automatically applying incremental database updates



In an earlier Automation for the people
installment, "Hands-free database migration
,"
I covered the need to apply incremental database changes in an
automated fashion. Like any other part of a Scripted Deployment, the
database upgrade scripts are checked into the repository.

LiquiBase (see Resources
)
is a tool for applying incremental changes to a database so that the
same change is applied into each target environment as part of the
Scripted Deployment. In Listing 3, an SQL script is called as part of
the LiquiBase changelog. This changelog (defined in XML) is then called
by the Scripted Deployment (which is implemented in a build-scripting
tool — such as Ant).

Listing 3. Running a custom SQL file from a LiquiBase change set

<changeSet id="1" author="jbiden"> <sqlFile path="insert-distributor-data.sql"/> </changeSet>

There's quite a bit more to
learning and applying automated database upgrades, but the idea is to
perform the upgrades as part of the Scripted Deployment so that all
database changes are in the system
, not a written procedure or in someone's head.

Back to top

Smoke-testing deployments

Name
: Deployment Test

Pattern
: Script self-testing capabilities into Scripted Deployments.

Antipatterns
: Deployments are verified by running through manual functional tests that don't focus on deployment-specific aspects.

Figure 6 illustrates an example of running deployment tests before and after a deployment:

Figure 6. Running functional deployment tests against application



In Listing 4, I'm using Ant to perform predeployment tests to verify I'm using the
correct tool versions. In the Scripted Deployment, the script can check for existing
ports in use (which would cause the Web container deployment to fail), verify a
connection to the database, and check if containers have been started, along with a host of other internal deployment tests.

Listing 4. Running predeployment checks to ensure deployment efficacy

<condition property="ant.version.success"> <antversion atleast="${ant.check.version}" /> </condition> <antunit:assertPropertyEquals name="ant.version.success" value="true" /> <echo message="Ant version is correct." /> <echo message="Validating Java version..."/> <condition property="java.major.version.correct"> <equals arg1="${ant.java.version}" arg2="${java.check.version.major}" /> </condition> <antunit:assertTrue message="Your Java SDK version must be 1.5+. / You must install correct version."> <isset property="java.major.version.correct"/> </antunit:assertTrue>

A more extensive deployment test can ensure that the application's functionality
is correct. By writing deployment-specific
automated functional tests using a tool such as Selenium for Web
applications or Abbot for client applications, you can verify the
deployment changes have been properly applied. You can think of these
tests as smoke tests
: you need to test only the functionality
that is affected by the deployment. For example, Table 1 shows ways you
can use Selenium and other tools for a Web application:

Table 1. Deployment tests

Deployment test	Description
Database	Write an automated functional test that inserts data into a database. Verify the data was entered in the database.
Simple Mail Transfer Protocol (SMTP)	Write an automated functional test to send an e-mail message from the application.
Web service	Use a tool like SoapAPI to submit a Web service and verify the output.
Web container(s)	Verify all container services are operating correctly.
Lightweight Directory Access Protocol (LDAP)	Using the application, authenticate via LDAP.
Logging	Write a test that writes a log using the application's logging mechanism.

Automated
tests aren't just for testing user functionality. By creating a suite
that focuses on deployment tests, you can verify the efficacy of the
deployment, reducing downstream errors and development costs.

Back to top

Rolling back all deployment changes

Name
: Environment Rollback

Pattern
: Provide an automated Single Command rollback of changes after an unsuccessful deployment.

Antipatterns
: Manually rolling back application and database changes.

Figure 7 illustrates rolling back database changes — using
Database Upgrade — along with the automation processes for rolling back a Web
deployment:

Figure 7. Rolling back deployment changes



Whether
or not you're automating deployments, it's nice to have a way to roll
back changes when a deployment goes wrong. In some cases, erroneous
changes can result in a system outage costing an organization millions
of dollars. To perform an Environment Rollback, you need to get the
target environment back into the state it was in before the deployment.
To do this, you essentially need a rollback script for every change. A
Web deployment often requires more changes to roll back. One example of
an Environment Rollback is to copy the archive (for example, a WAR
file) prior to the deployment and provide rollback database scripts for
each change. You also need to reapply the configuration changes applied
to the Web container.

Listing 6 demonstrates an example of
providing a rollback statement for each roll-forward statement using
LiquiBase. I'm adding a new table called

brewery

while providing a corresponding

dropTable

rollback statement.

Listing 6. Provide rollback process when applying incremental data upgrades

<changeSet id="rollback-database-changes" author="bobama"> <createTable tableName="brewery"> <column name="id" type="int"/> </createTable> <rollback> <dropTable tableName="brewery"/> </rollback> </changeSet>

This simple example is meant to be illustrative, not to trivialize rollback. Reverting
to a previous deployment is often a complex and time-consuming process to perform (and
to automate). The time you invest in writing rollback scripts should be proportionate to the cost of the deployment failure.

Back to top

Protect information from prying eyes

Name
: Protected Files

Pattern
: Using the repository, files are shared by authorized team members only.

Antipattern
: Files are managed on team members' machines or stored on shared drives accessible by authorized team members.

Figure 8 shows a protected version-control repository used to host files that only authorized people or systems should access:

Figure 8. Using a protected version-control repository for sensitive files



In
some cases, not all team members should have access to
environment-specific data. However, keeping this information separate
from the deployment scripts may prevent the scripts from executing.
When discussing the Headless Execution pattern, I described using SSH
keys with the Java Secure Channel tool to copy files and run remote
commands securely without a human needing to enter commands. The
properties you've used Externalized Configuration for are likely to
contain data that not all team members should see. A technique I've
used to ensure Headless Execution while protecting the data in the
.properties files from prying eyes is to check these files into a
protected repository.

In Listing 7, I'm configuring an Apache-hosted Subversion repository to
deny access to all for a certain directory and then explicitly adding
certain users:

Listing 7. Protecting a Subversion repository using Apache with Subversion

<DirectoryMatch "^/.*/(/.svn)/"> Order deny,allow Deny from all Allow bobama,jbiden,hclinton </DirectoryMatch>

By protecting access to the
Subversion repository, a Scripted Deployment can access the properties
as one of the allowed users without being prompted for a password,
providing Headless Execution in a manner as defined by the SSH keys.

Back to top

The one-click deployment

I've
cataloged several more deployment-automation patterns than the 15 I've
documented in this two-part article, but these 15 probably address 80
percent of the deployment situations I've encountered. Each of these
patterns is intended to help get you to a literal
one-click/single-command deployment for each and every target
environment. I wish you many painless deployments!

That's all I wrote

This is my last article in the Automation
for the people

series. It's been a tremendous adventure sharing my experiences with
you for more than two years. My goal with this series has always been
to show how and why to automate myriad software development processes
so that you're spending more time on interesting problems rather than
futzing with repetitive, error-prone activities. In the series, I've
demonstrated how to automate code reviews in order to refactor
appropriately, upgrade an application database incrementally, apply
Continuous Integration practices and tools, run automated tests with
every change, generate GUI installers, create one-click deployments,
automate the generation of developer documentation, perform dependency
management, utilize version-control repositories, and use various build
scripts and tools effectively. I hope you've enjoyed reading the series
as much as I've enjoyed writing it.

Resources

Learn

"Deployment-automation patterns, Part 1
" (Paul Duvall, developerWorks, January 2009): Patterns for one-click deployments.

Software Configuration Management Patterns

(Stephen Bercuzk and Brad Appleton, Addison-Wesley Professional, 2003):
The authors describe the Repository pattern and many other software
configuration management patterns.

Patterns of Deployment
: HP Labs' SmartFrog wiki includes a catalog of many deployment patterns.

Continuous Integration: Improving Software Quality and Reducing Risk

(Paul Duvall, Steve Matyas, and Andrew Glover, Addison-Wesley Signature Series, 2007): Chapter 8, Continuous Deployment
, demonstrates examples of incorporating deployment into an automated process.

Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Apps

(Mike Clark, The Pragmatic Programmers, 2004): Enjoy pragmatic,
automatic, unattended software production that's reliable and accurate
every time.

Release It!: Design and Deploy Production-Ready Software

(Michael T. Nygard, The Pragmatic Programmers, 2007): If you're a
developer and don't want to be on call at 3 a.m. for the rest of your
working life, this book will help.

"Speed deployment with automation
" (Paul Duvall, IBM developerWorks, January 2008): Leverage automation to move software through different environments quickly.

"Continuous Integration anti-patterns, Part 2
" (Paul Duvall, IBM developerWorks, March 2008): Make your life with CI easier by learning what not to do.

"Hands-free database migration
" (Paul Duvall, IBM developerWorks, August 2008): Use LiquiBase to manage database changes.

Browse the technology bookstore
for books on these and other technical topics.

developerWorks Java technology zone
: Hundreds of articles about every aspect of Java programming.

Get products and technologies

Ant
: Download Ant and start building software in a predictable and repeatable manner.

JSch
: Download Java Secure Channel for secure communication.

Selenium
: Download Selenium to perform deployment-specific functional testing.

LiquiBase
: Download LiquiBase to begin performing automated database migrations.

Abbot
: Download Abbot to begin performing deployment-centric functional testing.