Automation for the people: Continuous Integration anti-patterns, Part 2

Make your life with CI easier by learning what not to do


In the first part
of this two-part article, I described the following six CI anti-patterns:

Infrequent check-ins, which lead to delayed integrations

Broken builds, which prevent teams from moving on to other tasks

Minimal feedback, which prevents action from occurring

Receiving spam feedback, which causes people to ignore messages

Possessing a slow machine, which delays feedback

Relying on a bloated build, which also delays feedback

These
anti-patterns delay or prevent the benefits you can experience with CI.
In this second part, I'll cover five more equally deceiving practices:

Waiting until the end of the day to commit changes, leading to Bottleneck
Commits
, which typically cause broken builds and frustrated developers

A build consisting of minimal automated processes, which results in builds that never
fail, leading to Continuous Ignorance
and delaying integration problems

Hindering build fixes through a preference for Scheduled Builds
, rather than frequently building software with every code change

Believing that code Works on My Machine
, only to discover problems later in other environments

Failing to remove old build artifacts, which leads to a Polluted Environment
causing false positive and false negative errors

Once again, to receive the manifold benefits of CI, it pays to understand these anti-patterns — and avoid them.

Stop choking on bottleneck commits

Name
: Bottleneck Commits

Anti-pattern
:
Developers commit code changes prior to leaving for the day, causing
integration build errors and preventing team members from going home at
a decent time.

What's an integration build?

An integration
build
is a type of build that checks out source files from a version control repository and is run on a separate
machine, rather than just a developer workstation.

Solution
: Check-in code frequently throughout the day.

Bottleneck commits
are a variation of the Infrequent Check-in
anti-pattern. It assumes you are checking-in code at least once a day. However, the
problem is that everyone is checking in at the same time
. In a CM Crossroads
article, Slava Imeshev describes how the vast majority of build failures occur between 5
p.m and 8 p.m. (and to a lesser extent, the lunch hour). In the "Five-O'Clock Check-In",
Imeshev associates this occurrence to a tendency for developers to check-in their code changes toward the end of the day (see Resources
).

It's five o'clock somewhere

The Five-O'Clock Check-in
is a quick way to lose friends if your team follows the
rule that no one goes home until changes are committed and a build is run. Just the wait
to submit code can be tedious. Imagine what happens, then, when the build breaks. I'm
thinking that there will be a lot of calls home to explain why you'll be late ... again.

Figure 1 illustrates a typical timeline of check-ins for a software development team.
Note that repository commits are bunched around meal time and prior to leaving for the day.

Figure 1. Bottleneck commits



The
solution to this headache is, of course, to commit changes frequently!
With this approach, you'll have smaller, but more frequent, integration
builds, and when a build error does occur, it'll likely be small and
much simpler to fix.

Five o'clock should have a good connotation, so do yourself and your team a favor by
checking-in code frequently and keep five o'clock a happy time for the masses.

Back to top

Ignorance is not bliss

Name
: Continuous Ignorance

Anti-pattern
:
When a build was successful, everyone assumes the resulting software is
working fine. In reality, the limited build consists of compilation and
a few unit tests.

Solution
: A full integration build is run with every change to the version control repository.

Sometimes teams can be lulled into a false sense of security with one successful build
after the other. In fact, an almost sure sign that your team may be applying the Continuous Ignorance
anti-pattern is when build failures seldom occur. If
your integration builds never fail, you probably aren't verifying or validating enough
of your code! The less your build is doing, the less informed
you become about whether the software is actually working as expected.

Test Ignorance

A variation on the Continuous Ignorance anti-pattern occurs when you
choose to comment-out a broken test in an effort to force a build to
succeed, rather than fixing the source of the issue. Although
practicing this technique results in a successful integration build, it
is just another delaying tactic used to defer what will likely become a
bigger issue later.

Comprehensive builds offer enlightenment

The
stack on the left in Figure 2 illustrates an integration build that
doesn't do much more than compile source code, package classes into a
binary, and deploy software into an operating environment. Of course,
this is better than never performing an integration build at all.
However, take a look at the stack on the right. Comparing the two
stacks should provide insight into problems that can be uncovered
through running more processes as illustrated on the right, such as
tests and database changes.

Figure 2. Comprehensive builds are your friend



Streamline your builds

By
including additional processes like database integration, developer
tests (unit, component, functional, and so on), automated code
inspections (such as coding standard adherence, cyclomatic complexity,
and code duplication checks), and installation distributions, you are
better able to determine if your software is actually working earlier
in the development cycle. Keep in mind, however, that the more you add
to an integration build, the slower feedback will be. Therefore, you'll
probably want to consider creating build pipelines to run
slower-running processes after the initial commit build — doing so will
create opportunities for quicker feedback and provide for a more
flexible mechanism for software validation.

Back to top

Reschedule your scheduled builds

Name
: Scheduled Builds

Anti-pattern
: Builds are run daily, weekly, or on some other schedule, but not with every change.

Solution
: A build is run with every
change applied to a source code repository.

Continuous Integration is about integrating software assets often — and by often, I mean
any time code changes
. This is, as far as I know, the absolute quickest way to spot
issues early. Spotting issues early is a good thing. For one, it saves you money. For
another, it actually results in the ability to release better code more often.

The problem with a scheduled build is that it's run regardless of whether you committed
a change to a repository. This means that builds could be run that provide no value,
because either nothing changed since the last build, or there were so many changes that
any resulting problems become difficult to untangle.

Performing CI effectively requires a proactive emphasis — when a build fails,
it's imperative that problems are fixed immediately. The nature of a scheduled build discourages
proactive action and tends to lead to an "I'll fix it when I can get to it" approach, which is the antithesis of CI.

Changing a scheduled build to be more frequent is
as easy as properly configuring a CI server. For instance, Listing 1
demonstrates a CruiseControl script that polls a version control
repository every two minutes. If a change is discovered, CruiseControl
runs a build.

Listing 1. Build with every change

... <schedule interval="120"> <ant anthome="${cc.ant.dir}" buildfile="build-${project.name}.xml"/> </schedule> ...

Don't get me wrong — builds that run on a scheduled basis can be useful in
certain scenarios. For instance, running load and performance tests could be done
nightly because of the length of time the build takes to run. As a general rule,
however, if all builds are run on a less frequent basis than with every code change, you
are limiting your chances of discovering issues early.

Back to top

But, it works on my machine!

Name
: Works on My Machine

Anti-pattern
: You run a private build that works on your machine, only to discover later that the changes don't work in other environments.

Solution
: The team uses an integration build machine that runs a build with every change committed to a version control repository.

Imagine you make a code change and run your build (through an IDE or Ant), and after
all is working as expected you commit your changes to a version control repository. A
few days later, someone deploys the code into another environment and you're
informed that your changes aren't working. You then launch the software application on
your workstation, and, lo and behold, everything works perfectly leading you to exclaim "but, it works on my machine!"

Has
this ever happened to you? If it hasn't, get ready, it probably will.
There can be many reasons for this wicked occurrence, but the typical
culprits are that you forgot to add a new file into a version control
repository or some specific configuration on your machine isn't
configured on another environment.

The world beyond your desktop

Name
: IDE-only Build

Anti-pattern
:
You run a private build using an IDE (that works locally) on your
workstation only to discover things don't work in another environment.

Solution
: Create a build script and commit it into a version control repository.
Run this same build script with every change.

There's
nothing wrong with using an IDE to write code or to create build
scripts. IDEs make your job more efficient. However, if build processes
are so tightly coupled to the IDE that you're unable to run an
integration build without having the IDE installed in the deployment
environment, you've got a problem. This is because it's unusual that
operations teams will install an IDE in staging and production
environments. If they were to do this, it could require a manual
configuration (which could lead to inconsistencies across environments
and build errors) of the IDE in order to run an integration build.
What's more, having the build depend upon the IDE
could delay finding configuration problems until late into the
development process as dependencies are removed from the IDE in the
staging and production environments.

To solve the IDE-only build problem, all you must do is create a build script (you
can still easily use it through your IDE). This same build script can then be the master
mechanism for integration builds — regardless of environment.

Myopia and the developer

Name
: Myopic Environment

Anti-pattern
: Assuming that because a build works in one environment, it'll work in any environment.

Solution
: Define build behavior in build targets; moreover, externalize
environment-dependent data into .properties files.

The
Myopic Environment anti-pattern is reflective of a developer's
mentality of falsely assuming that if their code works in one
environment, their job is done. Therefore, to combat this mind-set, CI
systems should assume nothing (within reasonable constraints). Reducing
assumptions is as easy as removing platform-specific constraints and
making them replaceable, such as through property files.

If a build needs to run in both Windows® and Linux® environments, it stands
to reason that it'll fail in Linux if there are references to the

C

drive. If a build references a machine-dependent environment
variable (for example,

GLOBUS_LOCATION

),
it'll fail if these variables have not been set up in another
environment. In these cases, all you must do is replace those
references to variables that can be replaced at build time.

The Ant XML snippet in Listing 2 demonstrates how to include a .properties file that
contains environmental values:

Listing 2. Referring to environment properties in a build script

<property file="${basedir}/TEST.properties" />

The properties in
Listing 3 show data that is particular to a deployment environment.
Note how this file reduces assumptions in the build itself by
externalizing information like database connection values, Web
container locations, host names, and authentication information, for
instance.

Listing 3. Defining data in a properties file

database.host.name=integratebutton.com database.username=myusername database.password=mypassword database.port=3306 jboss.home=/usr/local/jboss/server/default jboss.temp.dir=/tmp jboss.server.hostname=integratebutton.com jboss.server.port=8080 jboss.server.jndi.port=1099

All build behavior should be contained in targets (such as Ant targets). Any data that
is referenced more than once in the same build script should be defined in a property.
Any data that varies from one machine to the next must be externalized into a
.properties file. If you follow this simple advice, you will achieve maximum flexibility
with minimal headaches.

Back to top

Clean up your environment

Name
: Polluted Environment

Anti-pattern
:
An incremental build is run to save time; however, an older artifact
(from a previous build) produces a false positive (or false negative)
build.

Solution
: Clean previously built artifacts prior to running a build. Baseline server and configuration information.

Cleaning up before yourself

There
are few more frustrating things than running a build that fails, only
to discover that it's because artifacts from a previous build are still
resident. Such artifacts may include recently deployed WAR files,
incorrect JAR versions, or database updates. On the flip-side, and even
more troubling, is when you're duped into a false sense of security
because previously built artifacts are in the build environment causing
a successful build.

I can't stress enough that it's important to put your environment into a known state
prior to any relevant build activities. In fact, I like to run what I call a
scorched earth
policy when building software. Listing 4 is a simple example of
removing previous log directories, distributions, and reporting files using Ant's

delete

task. This drastically reduces the chances of false positives or false negatives.

Listing 4. Removing directories via Ant

<target name="clean"> <delete dir="${logs.dir}" /> <delete dir="${dist.dir}" /> <delete dir="${reports.dir}" /> <delete file="cobertura.ser" /> </target>

The code above is
simple — scorching an environment may also include removing old class
files and previously deployed EAR/WAR files, putting the database into
a known state (for instance, dropping and recreating database tables),
reinitializing a classpath, and avoiding the use of environment
variables.

Baseline deployment environments

"Scorching"
can lead you to consider more advanced techniques such as setting the
state of the operating environment where you're running integration
builds. To baseline these environments, each component and
configuration item is removed and re-applied using automation. As
compared to simply removing old files, this can be a more difficult
undertaking depending on the environment complexity. However, it can be
incrementally applied. A component could be something like a database,
Web or file server, or some proprietary software. A configuration item
may be an environment variable or a change in the database
configuration, such as its memory allocation. Consistency is key. Each
environment you run your builds in should have a similar configuration.
By baselining deployment environments, you are better equipped to find
the cause of a particular problem. Figure 3 illustrates some of the
components that may need to be scorched and re-applied in a similar
manner in each deployment environment.

Figure 3. Scorching a deployment environment



In my experience, one of the most significant benefits with scorching
your environment is to provide a way to troubleshoot problems more
expeditiously. Baselining comparative environments gives you the
ability to compare apples to apples, so to speak, and ultimately
enables more rapid fixes should problems arise between environments.

Back to top

Succeed with Continuous Integration

I
hope you've seen that while Continuous Integration is a great practice
to use in your development projects, you can enjoy more of its benefits
by avoiding certain anti-patterns. There are good reasons why
development teams use some of these practices, but they can lead to the
use of anti-patterns. For instance, sometimes there are good reasons to
run a scheduled build. Further, the effective practice of committing
code often can actually lead to bottlenecks, but that doesn't make
frequent commits a bad practice. Remember, anti-patterns aren't bad
practices, per se, but they can be bad approaches in certain situations.

Resources

Learn

"Remove the smell from your build scripts
" (Paul Duvall, developerWorks, October 2006): Refactor your build to make it easier to maintain.

Continuous Integration: Improving Software Quality and Reducing Risk

(Paul Duvall et. al, Addison-Wesley Signature Series, 2007): Read the
reference on everything related to Continuous Integration.

"Continuous Integration
" (Martin Fowler, martinfowler.com): Fowler's seminal article on Continuous Integration.

"Is Pipelined Continuous Integration a Good Idea?
" (infoq.com, September 2007): Leading CI advocates weigh in on the build pipeline.

AntiPatterns: Refactoring Software, Architectures, and Projects in Crisis

(Brown et. al, Wiley, 1998): An excellent book describing anti-pattenrs related to software development.

"
Continuous Integration anti-patterns, Part 1

" (Paul Duvall, developerWorks, December 2007): Read about the first six anti-patterns.

Automation for the people

(Paul Duvall, developerWorks): Read the complete series.

developerWorks
: Hundreds of articles about every aspect of Java programming.

Get products and technologies

Ant
: Run your Java™ build with Ant.