Skip to main content

Ansible Automation

Ansible codifies how Cirrus CDN is provisioned, upgraded, and validated across environments. This chapter summarizes inventory layout, playbook orchestration, reusable roles, and the templating approach that keeps infrastructure definitions aligned with the control plane architecture.

Automation Objectives​

  • Deliver reproducible deployments for all runtime tiers (master, nodes, DNS, monitoring, acme-dns).
  • Capture environment-specific topology without duplicating logic.
  • Encapsulate Docker Compose lifecycle management behind hardened roles.
  • Provide smoke tests and operational tooling as first-class automation steps.

Inventory Model​

Two inventories reside under ansible/:

  • inventory/hosts.ini represents the production-like topology (master, registry, nodes, ns, monitor, acmedns) with shared variables in inventory/group_vars/all.yml describing Redis credentials, VRRP VIPs, DNS parameters, and health check defaults.
  • inventory.library/hosts.ini captures a "library" staging layout. Its group_vars/all.yml overrides features such as disabling CNAME (empty base domain) and providing proxy settings for Docker installs.

Both inventories rely on environment-variable lookups for sensitive inputs (for example, CIRRUS_REDIS_PASSWORD, CIRRUS_VRRP_PASS) to avoid embedding secrets in source control.

Playbook Structure​

Playbooks under ansible/playbooks/ compose automation steps into pipelines:

  • main.yml orchestrates a full production rollout: host initialization, registry deployment, image build, keepalived configuration, stack deployment, and smoke verification (including VRRP VIP checks).
  • library.yml mirrors the flow for the library environment while injecting HTTP proxy settings and skipping VRRP validation by default.
  • build-push.yml, deploy.yml, docker.yml, and keepalived.yml are reusable building blocks invoked by higher-level playbooks.
  • Smoke suites (smoke.yml, smoke_ns.yml, smoke_vrrp.yml) verify API, DNS, and VIP functionality by seeding temporary domains through /api/v1/domains and asserting outcomes via HTTP and dig probes.

Playbooks prefer strategy: free during deployments to maximize parallelism across host groups while preserving idempotency via templated Compose definitions.

Roles & Reuse​

compose_deploy​

roles/compose_deploy wraps Docker Compose v2 lifecycle management:

  • Renders a stack-specific docker-compose.yml (and optional ancillary files) into a host-scoped directory (/opt/cirrus-*).
  • Invokes community.docker.docker_compose_v2 with remove_orphans=true to reconcile services safely.
  • Supports additional templates (Grafana, Prometheus configs) via the extra_templates variable.

This role standardizes how master, node, ns, monitor, registry, and acmedns stacks are started regardless of environment.

keepalived_vrrp​

roles/keepalived_vrrp encapsulates multi-VIP failover setup:

  • Installs keepalived plus curl on supported distros (Debian, RHEL/openEuler).
  • Applies ARP hardening sysctls to avoid VRRP-induced ARP flux.
  • Templates keepalived.conf using deterministic priority rotation so VIP ownership distributes evenly across node hosts while honoring cluster_vips order.
  • Installs a curl-based health script that queries http://127.0.0.1/ (or configured check_url) to demote unhealthy nodes by check_script_weight.

Handlers ensure configuration changes trigger keepalived restarts, keeping VRRP state consistent.

Docker Compose Templates​

Compose blueprints live under ansible/templates/ and reflect the control plane architecture:

  • Master (docker-compose.master.yml.j2) runs Redis (with password enforcement), the FastAPI API, Celery worker/beat, and exposes the hidden master DNS port. Environment variables match Chapter 3, Chapter 4, and Chapter 5 requirements (CNAME_*, ACME_*, REDIS_*).
  • Nodes (docker-compose.node.yml.j2) host OpenResty, node-exporter, a Redis replica seeded from the master, and logrotate for access logs. Arguments such as NGX_METRICS_ALLOW, NGX_RESOLVER, and NGX_WORKER_CONNECTIONS are passed at build time.
  • NS (docker-compose.ns.yml.j2) packages NSD with templated SOA/NS data and NOTIFY ACL pointing to the hidden master.
  • Monitoring (docker-compose.monitor.yml.j2) builds Prometheus and Grafana images with host networking and anonymous Grafana access (overridable password via grafana_admin_password).
  • ACME DNS (docker-compose.acmedns.yml.j2) builds the acme-dns authoritative server from acmedns/, seeding records defined in inventory.

Templates share parameters from inventory vars, ensuring consistent DNS base domains, TTLs, and ACME endpoints across stacks.

Build & Release Flow​

just deploy sources environment variables (via dotenv) and invokes Ansible with ansible/playbooks/main.yml unless overridden. The build sequence embodied in build-push.yml renders temporary Compose manifests under .compose-build/ and executes docker compose build/push for each service group, pushing to the internal registry (registry_address). Deployment playbooks then pull these images during role execution, guaranteeing hosts run the newly built artifacts.

For library environments, the same pipeline runs against inventory.library, leveraging its proxy configuration and simplified DNS settings. This approach keeps staging/production parity without code duplication.

Operational Workflows​

  • Bootstrap – Run ansible-galaxy install -r requirements.yml to fetch role dependencies, export inventory-specific secrets, then just deploy (or targeted ansible-playbook -i inventory deploy.yml).
  • Rolling Updates – Re-run build-push.yml followed by deploy.yml; Compose pulls new images and restarts services with minimal disruption thanks to per-host docker-compose.yml state directories.
  • Remediation – playbooks/cleanup.yml performs whole-host Docker cleanup when recovering from drift. Smoke playbooks provide post-change validation before closing incidents.

Extensibility Guidelines​

  • Introduce new stacks by authoring additional Compose templates and invoking compose_deploy from a dedicated play or extending existing ones.
  • Inject secrets via environment variables or Ansible Vault—do not hardcode them in templates.
  • Maintain symmetry between inventory group vars and Docker environment settings to avoid configuration drift.
  • Update Chapter 11 appendices when new environment variables or Redis keys emerge from automation changes.

Ansible serves as the connective tissue between the documented architecture and running infrastructure, enabling deterministic deployments, health validation, and rapid iteration across Cirrus CDN environments.