Skip to main content

Data Plane

OpenResty acts as Cirrus CDN’s data plane, executing per-domain routing, caching, and TLS termination logic defined in LUA scripts. This chapter covers configuration templates, runtime behavior, purge processing, and telemetry emitted by the edge tier.

Nginx Configuration Template​

openresty/conf/nginx.conf.j2 is rendered at image build time with environment-controlled parameters (openresty/Dockerfile). Key directives include:

  • proxy_cache_path – Defines the shared cache (node_cache) stored on /data/cache.
  • map directives – Determine cache bypass behavior based on cookies ($has_cookie) and cache-control headers ($cc_nocache).
  • init_worker_by_lua_file – Executes init_worker.lua in each worker to start the purge subscriber and initialize metrics.
  • log_by_lua_file – Invokes log_metrics.lua after each request to update Prometheus counters.
  • listen blocks – Bind to HTTP (default 80) and HTTPS (default 443) ports, enable HTTP/2 and QUIC, and expose a separate listener on port 9145 for metrics and health checks.
  • ssl_certificate_by_lua_file – Loads per-domain certificates dynamically via ssl_loader.lua.

The template supports injection of resolver addresses, cache sizes, worker connections, and metrics access control at build time.

Access Router (Lua)​

openresty/conf/access_router.lua orchestrates request handling:

  1. Retrieves domain configuration from Redis using resty.redis. If Redis is unavailable, falls back to a 5-second shared dictionary cache (ngx.shared.dict_conf).
  2. Validates configuration; absence yields 404.
  3. Determines whether caching should be active (cache.enable flag).
  4. Parses origin definitions into a normalized list (scheme, host, port, sni, weight).
  5. Applies custom upstream headers from conf.upstream_headers, excluding Host unless explicitly set.
  6. Selects an origin using weighted round robin stored in a shared dictionary key (wrr:{domain}).
  7. Sets $origin_host, $origin_sni, and $backend Nginx variables to drive proxy_pass.
  8. Logs the selected backend via ngx.log(ngx.INFO, ...).

This logic delegates DNS resolution to Nginx itself (proxy_pass with hostname), simplifying Lua responsibilities.

Cache & Purge Mechanics​

  • Cache keys incorporate scheme, host, URI, and Range header to support byte-range caching.
  • proxy_cache_background_update enables background refreshes while serving stale content.
  • PURGE support uses nginx_cache_multipurge (compiled during the OpenResty image build):
    • API publishes commands to Redis channel cdn:purge.
    • redis_subscriber.lua runs in worker 0, consumes messages, and issues PURGE requests to /cache/purge with the target domain in the Host header.
    • cache_multipurge.lua leverages nginx_cache_keyfinder to calculate cache keys, supports wildcard purges, and integrates with configured cache zones.

TLS Handling​

ssl_loader.lua uses resty.lrucache to store up to 2000 SNI entries for 10 seconds. On handshake:

  • Fetches the server name (ngx.ssl.server_name()).
  • Attempts LRU lookup; on miss, pulls fullchain and key from cdn:cert:{domain} hash in Redis.
  • Parses PEM material via ngx.ssl.parse_pem_cert/parse_pem_priv_key.
  • Sets the cert and key for the current handshake.
  • Records Prometheus counters (via metric_ssl_handshake_errors) for parse or set failures.

Default dummy certificates (generated at image build time) ensure TLS handshakes succeed even before per-domain certs are available.

Metrics & Logging​

init_worker.lua:

  • Starts the Redis purge subscriber.
  • Initializes Prometheus metrics (requests, latency, cache status, upstream errors/timeouts, SSL errors, upstream RTT).

log_metrics.lua:

  • Increments counters per host/status.
  • Observes request latency and upstream response time histograms.
  • Counts cache statuses (HIT, MISS, STALE, etc.) and upstream errors.

Metrics are exposed via /metrics on port 9145, gated to loopback by default (configurable via NGX_METRICS_ALLOW). The same port serves /healthz, which returns 200 OK for external health probes (Celery).

Logs:

  • Access and error logs are emitted to stdout/stderr and shipped via the Loki Docker logging driver; inspect them in Grafana or through LogQL queries.
  • Error log entries include Redis connectivity, configuration decoding, and backend selection messages helpful for debugging.

Security Controls​

  • Metrics endpoint is restricted to localhost by default; deployment engineers can whitelist additional CIDRs via build arguments.
  • Purge endpoint requires local access (127.0.0.1); external purge requests must flow through the API.
  • Redis connections default to 127.0.0.1:6379 for the host-networked OpenResty container. Production deployments should enforce TLS or restrict network boundaries accordingly.
  • Origin header injection prohibits overriding the Host header unless specified in upstream_headers["Host"].

Performance Considerations​

  • Weighted round robin is implemented in Lua; for large node counts, consider migrating to consistent hashing or using nginx upstream modules.
  • Cache zone size (NGX_CACHE_MAX_SIZE) is configurable; default is 1 GiB.
  • proxy_cache_lock prevents cache stampedes by serializing cache fill operations.
  • Client support includes HTTP/2 and QUIC, enhancing protocol coverage without additional configuration.

Request Paths​

Cache Behavior Snapshot (Example)​