Infrastructure
Production Config Should Not Be a Memory Test
July 2, 2026
During a production review, I found that Sivella's backend could start with the wrong cache address even after the operator had set the right value.
The variable was DJANGO_CACHE_URL. The operator put it in .env.prod, but the production Compose file passed application settings through a hand-maintained x-django-env anchor. The cache variable was absent from that anchor, so Django used its development fallback:
redis://localhost:6380/1
Inside the backend container, localhost meant the backend container, not the Redis service. The shared cache used by request throttling was therefore unreachable.
The backend exposed the failure. worker, beat, and flower inherited the same anchor, although their configuration needs were not identical.
The root cause was the configuration path
The repository already had the setting in backend/clara/settings.py. clara is the internal Django package name left from Sivella's earlier product name; the path is current even though the public name changed.
The application knew how to read DJANGO_CACHE_URL, and the operator knew where to set it. Compose filtered the value between them. Keeping the Django setting, example files, documentation, production anchor, and preview anchor synchronized depended on someone remembering every location.
I compared the environment names read by settings.py with the keys in the Compose anchor. Roughly forty application settings were absent from the allowlist, including ADMIN_REQUIRE_OTP, RAZORPAY_*, SENTRY_*, and THROTTLE_RATE_*.
That did not mean forty production failures. Some settings were optional, had defaults, or applied only to particular deployments. It meant the application's configuration surface had grown beyond the deployment allowlist.
Building with AI
I build this system with AI. I write detailed instructions, review the design myself, and make the architectural calls. The model writes much of the implementation because it is faster at repetitive wiring.
That saves time and creates a predictable risk. A model can carry a locally valid value into another configuration file without reconsidering the network boundary. It can also extend a pattern, such as an allowlist, without asking whether the pattern still suits the system.
One correction matters here: redis://localhost:6380/1 was not a second Redis cluster. Local Compose mapped host port 6380 to 6379 on one Redis container. Production also uses one Redis service: database 0 for the broker, 1 for results, and 2 for the cache. The error was carrying a host address into a container, where localhost changed meaning. Docker's Compose networking documentation distinguishes the mapped host port from the service name and container port used between containers.
One Redis instance is appropriate for Sivella at its current scale. Logical databases provide namespaces, not separate capacity or failure domains; they share memory, CPU, persistence, eviction behaviour, and outages. Redis recommends them for separating keys within one application, while Redis Cluster supports only database 0. I would split these workloads when their scaling, persistence, access-control, or reliability needs genuinely diverge.
Configuration needs an owner
The review produced a boundary that has held up better than the original allowlist:
The operator owns product and provider values. The deployment owns network topology. The application owns validation.
Operators or a secrets system should supply provider credentials, origins, feature flags, mail settings, and throttle rates. Compose should pin container hostnames and internal ports. In this stack, Redis database 0 is the Celery broker, 1 stores results, and 2 is the Django cache. The application still has to reject unsafe production combinations.
The Compose change
Before the fix, the backend received only the explicit anchor:
x-django-env: &django-env
DEBUG: "0"
CELERY_BROKER_URL: redis://redis:6379/0
CELERY_RESULT_BACKEND: redis://redis:6379/1
services:
backend:
environment:
<<: *django-env
The change loaded operator-owned values from .env.prod and kept topology values in the higher-precedence environment: block:
x-django-env: &django-env
DEBUG: "0"
CELERY_BROKER_URL: redis://redis:6379/0
CELERY_RESULT_BACKEND: redis://redis:6379/1
DJANGO_CACHE_URL: redis://redis:6379/2
services:
backend:
env_file:
- .env.prod
environment:
<<: *django-env
worker and beat received the same service-level env_file: because their task code loads Django settings and may use the same providers as the web process.
Flower starts from the backend image with celery -A clara flower, so the fix gave it the same environment for consistency. It does not need payment, OAuth, or SMTP credentials merely to monitor Celery. For this deployment, removing drift was worth that temporary increase in exposure. Flower is the clearest candidate for a narrower environment containing only broker and monitoring configuration.
.env interpolation and service env_file: are different
The command-line option docker compose --env-file .env.prod ... supplies values while Compose interpolates ${VAR} references in YAML. It does not put every value into a container.
A service-level entry does that:
services:
backend:
env_file:
- .env.prod
If the same variable appears under environment:, that explicit value wins. This lets .env.prod carry operator settings while Compose replaces a laptop-oriented cache URL with redis://redis:6379/2.
How I checked the fix
I used a temporary, gitignored .env.prod containing synthetic values and a deliberately wrong DJANGO_CACHE_URL. Rendered Compose configuration may contain the complete service environment, so real credentials should not appear in this check or its logs.
The first check inspected the rendered plan:
docker compose \
--env-file .env.prod \
-f docker-compose.prod.yml \
config --format json \
| jq '.services.backend.environment |
{DJANGO_CACHE_URL, ADMIN_REQUIRE_OTP, THROTTLE_RATE_REGISTER}'
The result showed the pinned cache URL plus the synthetic admin and throttle values. This proved both precedence and forwarding. The preview Compose file passed the same check.
Rendered configuration is only a plan. After deployment, I checked the running backend without printing secrets:
docker compose \
--env-file .env.prod \
-f docker-compose.prod.yml \
exec -T backend sh -lc \
'printf "%s\n" "$DJANGO_CACHE_URL"'
The backend received the shared cache and throttle configuration. Worker and beat received the same operator settings without another allowlist edit.
The application needs a second guard
The Compose patch made the intended cache URL arrive, but the original development fallback still exists in Django settings. A future deployment that does not use this Compose file could start production with localhost again.
The incident fix did not add this validation. A follow-up should reject loopback cache hosts when DEBUG is false:
from urllib.parse import urlparse
from django.core.exceptions import ImproperlyConfigured
cache_host = urlparse(DJANGO_CACHE_URL).hostname
if not DEBUG and cache_host in {"localhost", "127.0.0.1", "::1"}:
raise ImproperlyConfigured(
"DJANGO_CACHE_URL must not use a loopback host in production"
)
That guard covers Compose, another container platform, and a direct process launch.
Secret distribution remains separate work
Service-level env_file: solved drift by broadening what each application and Celery-related container received. It also broadened the blast radius of SECRET_KEY, SMTP credentials, payment secrets, OAuth secrets, and provider keys. The file remains gitignored and verification prints selected non-secret values, but neither measure provides service-level isolation.
The long-term choices are narrower per-service env files, mounted secrets, or a secret manager with workload-specific access. The current broad injection is a documented trade-off, not a universal Compose pattern.
I now compare the rendered configuration with the environment inside the running container before treating this path as healthy. Application-level validation and tighter secret scopes are still unfinished work.
AI can write the configuration quickly. I still have to decide how many services the product needs, inspect what the generated wiring assumes, and prove the running topology. The ownership boundary remains the same: operators supply environment-specific values, the deployment pins its network topology, and the application should refuse unsafe production configuration.