Automation considerations for distributed systems - series

Tldr#

Automation Checks - A Multi Part Series#

A look though the lens of software automation when it comes to templated applications and pipelines that validate system, environment and application functionality. Dogfooding at its finest (Bon appetite)

Chapters#

1. Automated checks using sample apps
2. Deployment templates
3. Infrastructure and environment checks
4. Using FaaS to validate operability
5. Security testing in the pipeline
6. Chaos engineering tools and techniques
7. Canary/traffic shifting options
8. Performance containers, reports and dashboards
9. API contract automation

The Goal#

To build a holistic approach to deploying the testing of applications that can be managed by a centralised Engineering Effectiveness/Efficiency team but done in a way that these can be contributed to by the business as a whole leading to an abstract ownership.

Which Leads to#

Increased velocity and providing a better ROI when developers have less distractions and context switching

#1 Sample Apps Build/Deploy Sample Apps TL’DR: Building sample applications that include automated check patterns that your teams can use/reference for deployments and implementations can aid in reducing the barrier to entry and this in turn promotes accountability and collaboration

What are we trying to accomplish

The expectation here is that you can build and deploy sample applications on demand to your various environments (Dev, UAT, Prod)

This has number of benefits: Standardised deployment methods Environment checks Reusable patterns Centralised development Emphasis on cross team collaboration Reducing the gap between code complete and deployment

There have been a number of articles on Engineering Efficiency/Effectiveness around developer tooling and productivity teams. These are some of my thoughts and experiences on these viewing and experiencing these through an automation lens

This has a couple of benefits here by building sample applications based primary domain language You build in the patterns that you expect will be used by the teams consuming your services You can include and manage the templates Testing is build into the containers so is treated as a first class citizen Expose endpoints to continually poll the above apps for service availability, FaaS is perfect for these one off tasks The use of dgoss to confirm application deployment behaviour inside docker containers Have

#2 App deployment templates Standardised App deployment templates helm chart repo that's centrally maintained (team) and located (Shared repo)

Run an open source contribution model Teams deploy an appspec.yaml to configure their own infra requirements Include Helm unit tests that can be run locally and as part of the build pipeline

#3 Infra checks Infra / Environment deployment checks Use something like TerraTest thats language agnostic for testing / confirming the following Helm / puppet / packer deployments Have them spin up containers, run the infra, execute the tests and destroy the container and report the results if there is an issue (slack post test hook) Create a suite of Integration tests that validate infra deployment execution

#4 Adding FaaS (Serverless) to your testing arsenal

#5 Automated security scanning CIS Sec scanner: Implement continuous container scanning such as: Kube-bench Kube bench for EKS SAST & DAST and IAST

#5 Turmoil in the system Chaos Engineering Production failure simulation in a controlled test environment

Use of a tool like Gremlin Run in all test and tooling clusters Eventually run in Prod as maturity increases http://gauntlt.org/ Chaos monkey/Gorilla Simian

#6 traffic shifting deployments Canary Releases to prod using traffic shifting To manage and monitor traffic routing to newly deployed app containers

To allow for automated rollbacks based off traffic/app conditions Such as flagger Required to be run in conjunction with a service mesh such as istio

#7 Perf container deployment and orchestration

#9 Including API contract checks