Criteria for Evaluating Deployment Systems
My goal with this blog post is to provide criteria for evaluating deployment systems – and through this evaluation to either ensure that either the first build of a deployment system or an upgrade of a deployment system meets a company’s needs. The evaluation criteria are broken down into two sections:
- User-Facing: attributes of a deployment system that that an application’s user might experience (downtime)
- Engineering facing: attributes of a deployment system that Engineering organization will experience (efficiency, ease of deployments).
The deployment system attributes are listed from “Less Mature” (left-most column) to “Most Mature” (right-hand column).
User-Facing (External) Attributes:
The diagram below compares deployment systems from a user’s perspective. Availability is the key measurement. The user only experiences a deployment when an outage or error occurs.
- Availability: can the application function normally during a deployment?
- Recoverability: if a deployment fails, can the application continue to function normally?
Engineering-Facing (Internal) Attributes:
The diagram below compares deployment systems from an Engineering employee’s perspective. Availability, efficiency, security, scale and transparency are the key measurements.
- Auditing: can the question “who deployed what, where and when” be answered?
- Dependency Support: does the deployment system also manage resources that the application depends on? Example: does the deployment system manage database schema? Does the deployment system update content delivery network files?
- Environment Support: how much effort is required to add a new environment? Example: can a “QA2” or “Load Testing” environment be added easily?
- Re-use of Tooling: can the deployment system be easily re-used to support other applications?
- Reproducibility: can a given deployment be reproduced? Example: can you give me last week’s release on last week’s Linux server?
- Rollback: in the event of a failure can the application be rolled back? Can any previous version of the application be deployed?
- Scale: can the deployment system deploy to any number of nodes?
- Target Users and Ease of Use: is the deployment system used by both Engineering and Operations or by Operations only?
- Transparency: does the deployment system integrate with existing tools? Example: are deployment events integrated with an organization’s dashboards?