Twelve-Factor App Rollback

Back To The Previous Version! For those not familiar, "Twelve Factor Apps" were defined by Heroku in a beautifully succinct online document they wrote to define the 12 attributes of modern, scalable cloud applications. Such an application is ephemeral, works well with siblings and is decoupled in such a way that logging and configuration can be easily managed externally. If you are designing applications for the cloud, or writing any software for that matter, I thoroughly recommend reading The Twelve-Factor App document - it will take you about 15 minutes.

Where The Twelve-Factor App falls short, and in fairness is probably out of the scope of the document, is with upgrading applications. You might say that The Twelve-Factor App is a guide for Developers and that this is an Ops concern - and you would be 50% right. Upgrading is such a critical issue that it is not to be left to the domain of just Developers or just Operations engineers. This is a mission for DevOps!

What The Twelve-Factor App does well is treating backing services as attached resources. This makes the application runtime package-able and easily manageable. This is the cornerstone of PaaS and what makes working with Docker at scale feasible.

What are the key concerns of upgrading?

1) General feature breakage.

2) Unhappy end-users due to UI and feature changes.

3) Database upgrades or data migration.

4) Application down-time.

Points 1 and 2 are runtime concerns, usually.

Point 3 is related to attached resources, or Services in the world of Cloud Foundry.

Point 4 is a general issue with deploying applications to Cloud Foundry-based solutions. This is due to the way applications are taken offline first and then restaged. There are workarounds to this, such as deploying multiple versions of the same application under different application names and performing URL switching, but so far all solutions are external and onerous on the developer who is deploying the application.

Strongly focusing on The Twelve-Factor App enables us to solve upgrade issues with 1 and 2 quite easily. Since our application runtime is ephemeral, we can upgrade it by simply replacing it with a new version of the runtime. If there is a problem, we replace it with the previous version of the runtime. Although quite a simple concept, we are only now starting to see the adoption of this idea. This is one of the killer features of Docker. Drop-in, drop-out, drop-in something else.

For this to work, we have to assume that both versions will work with our attached resources. Database schemas must align with what the runtime expects. Dropping in a new version of the application runtime might be easy, but sometimes it also requires upgrading the database. Rolling back to a previous version of the application runtime might require the rolling back of that database.

I see data storage as the Achilles heel of PaaS. If applications did not have to store anything in databases then our life as application platform builders would be much simpler. But we do. I do not think the role of PaaS is to solve data storage. That task is for the external services that the application platform utilizes. But PaaS does need to solve how the application consumes those data services. We need to do this through best practices, while always supporting the legacy scenarios.

What are the best practices for working with data storage, in the context of application runtime roll-forward and rollback? We need to begin that discussion.

A common pattern, especially with Ruby-on-Rails developers, is versioned database migration scripts, which are run on deployment to bring the database schema either up or down the match with the deployed application code. This is a good solution and can be included in the test-suite of the application.

In an ideal world this would simply be ensuring that your application runtime is always backwards-compatible with your data-store and your data-store is always backwards-compatible with your application runtime. Either could independently move forwards, or backwards, and this would provide a Utopian level of decoupling. Unfortunately, as any half-decent mathematician will tell you, as your version numbers move towards infinity, both your application runtime code and your database schema will be severely crippled in this futile effort to support forwards-backwards compatibility.

While we have not yet reached Utopia for PaaS, we are way past just giving people faster horses. ActiveState customers are killing it with our Cloud Foundry and Docker based PaaS solution, Stackato. With Stackato 3.4, which was released today, we are taking the next step in application upgrade and rollback.

Stackato now retains previous versions of the application runtime. If you push new application code and the result is not what you expected, with a single click (or command or API call) you can rollback. Even if you only changed an environment variable, and you want the previous version, you can rollback. You can jump back to several versions prior. Any change to your application runtime can be rolled back and the Stackato administrator can configure how far you can roll back. You can even roll forward again if you changed your mind about the rollback.

What I like about this Stackato feature, is that it is a zero-downtime event. It phases over without skipping a beat.

This is both useful and powerful. Due to its power, I will give you the same advice a wise old man once told once told Spider-Man - actually it was his uncle - Uncle Ben to be specific: "With great power comes great responsibility."

We are not expecting to solve all data storage migration and rollback scenarios during an application's life-cycle. We do not restrict you from running anything in your runtime containers - though we might work with your Ops team to lock it down if they request it. We support both Twelve-Factor Apps and legacy applications. We support any Cloud Foundry service available in Stackato, in the wild, or that you can imagine and build yourself. Therefore, we cannot stop you from writing bad code, creating bad applications, bad database schemas, making bad choices of languages or bad choices when rolling your application runtime forwards and backwards. The end result is the same as if you had pushed an older version from the command-line or your IDE.

My general advice for a pain free life of upgrading would be...

1) If you can make your application easily backwards compatible with older versions of your database schema, do it.

2) If you can make your database schema easily backwards compatible with older versions of your application, do it.

3) Version your schema, version your application and document compatibilities.

4) Generally do not rollback when you have run incompatible database migrations.

5) If your application has no state stored an external data store, then this is a non-issue. Unfortunately, such an application is rare.

As with most things PaaS, the more you focus on building more modular applications, the more you can benefit from features like this. A micro-service architecture is much easier to reason about, test and iterate on. We fully recommend moving your architecture in this direction.

I would like to hear your feedback on this new feature of Stackato, but also the broader topic of working with data storage when upgrading and downgrading your application runtime. What are your best practices?

Title image courtesy of JD Hancock on Flickr under Creative Commons License.

This blog post was originally published by Phil Whelan on July 30, 2014 at ActiveState blog.