Most production incidents aren’t dramatic crashes — they’re quiet failures: requests that hang forever, exceptions that lose their cause, “helpful” defaults that silently change behavior, and logs that don’t preserve the crime scene.
Below are 10 Python one-liners I trust in production because they bound time, bound blast radius, and leave evidence. Each one is paired with the deceptively “fine in a tutorial” version I don’t trust.
In production, the worst bugs don’t crash. They hang, swallow evidence, or quietly choose unsafe defaults. These are 10 Python lines I trust because they bound time, bound blast radius, and leave a trail.
- Every I/O call has a timeout (network and subprocess).
- Exceptions are handled with context or re-raised with the original attached.
- Config and input validation fail at startup (no silent fallbacks for secrets).
- Logging preserves the “crime scene” without leaking secrets.
- Paths and temp files use safe primitives (
pathlib,tempfile).
The rule: production lines must fail loud, fast, and explain why
What “quiet failure” looks like
Quiet failures are the ones that don’t page you immediately: a request that hangs forever, an exception that gets swallowed and returns “None”, a retry loop that DDOSes your dependency, or a default config value that accidentally points production at a dev system.
The 2am debugging test
If you’re woken up at 2am, can you answer: What did we try? How long did we wait? What failed? What input triggered it? If the code doesn’t leave that evidence by default, I don’t trust it.
- Bound time: timeouts for anything that can block.
- Bound blast radius: limits and backoff to avoid cascading failures.
- Leave evidence: structured logs, stack traces, stable error shapes.
1–3: Networking lines I trust (and the timeouts I refuse to skip)
1) HTTP requests with explicit timeouts
Trust: bounded connect+read time. I treat connect and read separately so “can’t reach host” and “server slow” don’t look identical.
r = requests.get(url, timeout=(3.05, 10))
Don’t trust: the same call without a timeout. “No timeout” is functionally “infinite hang”, and your worker pool will quietly drain until everything is stuck.
r = requests.get(url) # can hang forever
2) Surface HTTP errors immediately
Trust: fail fast on non-2xx. It forces the caller to decide what “error handling” means instead of quietly parsing an error page as JSON.
r.raise_for_status()
Don’t trust: assuming success and doing a best-effort parse. This is how you end up with
KeyError in random places (or worse: silently wrong data).
data = r.json() # may be an HTML error page, not JSON
3) Retries with backoff (only when safe)
Trust: backoff that slows down under pressure. Retries are not “reliability”; retries are load multipliers unless they’re deliberate.
sleep_s = min(2 ** attempt, 30)
Don’t trust: tight loops that hammer a dependency. If the upstream is degraded, this makes it worse and hides the root cause behind “random flakiness”.
while True: do_request() # no backoff, no cap, no stop condition
Production note: don’t retry non-idempotent operations unless you have an idempotency key or another correctness guarantee.
4–6: Exceptions and logging that don’t erase the crime scene
4) Catch specific exceptions, not everything
Trust: narrow exception handling. It’s honest about what can fail and avoids accidentally
catching programmer errors (like AttributeError) and continuing in a corrupt state.
except (TimeoutError, OSError) as e:
Don’t trust: bare except. It turns “bug” into “mystery behavior” and makes outages longer.
except: # catches too much, including KeyboardInterrupt in some cases
pass
5) Log exceptions with stack traces (and context)
Trust: logger.exception(...) inside an except block.
You get a stack trace by default, which is usually the difference between “5 minutes” and “5 hours”.
logger.exception("upstream call failed", extra={"request_id": request_id})
Don’t trust: print(e) (or worse, nothing). Prints get dropped, reordered, or lost in container logging pipelines —
and they rarely include the stack trace you’ll need.
print(e) # no stack trace, often no context
6) Re-raise with the original error attached
Trust: explicit chaining. It preserves the original exception while providing a stable, higher-level message.
raise RuntimeError(f"payment provider failed: {provider}") from e
Don’t trust: raising a new exception without chaining (or returning None).
That deletes the cause and forces you to reproduce the issue under pressure.
raise RuntimeError("payment provider failed") # original cause lost
7–8: Input and config validation that prevents mystery states
7) Validate shapes and ranges, not just “it exists”
Trust: defensive parsing that fails early with a clear error. When inputs are untrusted (HTTP, queues, files), ambiguity becomes data corruption.
user_id = int(payload["user_id"]) # raises loudly if missing/invalid
Don’t trust: “best effort” conversions that turn bad input into a default. It hides upstream problems and can create surprising security behavior (“guest mode” by accident).
user_id = int(payload.get("user_id", 0)) # silent fallback changes meaning
8) Environment variables: fail closed for secrets
Trust: os.environ[...] for required config. If a secret is missing, the service should fail at startup,
not limp into production half-configured.
DATABASE_URL = os.environ["DATABASE_URL"]
Don’t trust: defaulting secrets or critical endpoints. The “helpful” default becomes a breach, a data leak, or a split-brain incident.
DATABASE_URL = os.getenv("DATABASE_URL", "sqlite:///dev.db")
9: Filesystem and temp paths — predictable or it didn’t happen
Paths via pathlib, temp via tempfile
Trust: composition that works across platforms and avoids accidental path traversal from string concatenation.
path = Path(base_dir) / filename
Don’t trust: string-joined paths. It’s easy to get double slashes, missing separators, or accidentally allow
../../ behavior to slip into “it works on my machine”.
path = base_dir + "/" + filename
Trust: unique, race-resistant scratch files managed by the OS.
tmp = tempfile.NamedTemporaryFile(delete=True)
Don’t trust: shared temp filenames. Under concurrency, you’ll get collisions, partial writes, or (worst case) security issues.
open("/tmp/output.txt", "w") # collision-prone in production
10: Running commands safely (subprocess without foot-guns)
subprocess.run with checks, timeout, and no shell
Trust: explicit argv list, fail on non-zero exit, and a timeout. This turns “mysterious stuck worker” into a bounded, debuggable error.
subprocess.run(["git", "rev-parse", "HEAD"], check=True, timeout=5)
Don’t trust: shell=True with string interpolation. It’s fragile (quoting breaks) and can become an injection bug
the moment any part of the command becomes user-influenced.
subprocess.run(f"git show {ref}", shell=True) # injection risk, harder to reason about
A quick production checklist (what I scan for in code review)
If I’m reviewing a PR and I don’t have time to deeply reason about every edge case, I scan for these “quiet failure” triggers:
- Timeouts everywhere: HTTP, DB, queues, subprocess. No exceptions.
- Retries are deliberate: capped attempts + backoff; only for idempotent actions.
- No bare
except: catch the specific failure modes you can handle; otherwise fail loud. - Evidence is preserved:
logger.exceptionin handlers; exceptions chained withraise ... from e. - Config fails closed: required env vars accessed via
os.environ[...]; no silent defaults for secrets. - Secrets don’t leak: log messages avoid tokens/credentials; follow redaction practices.
- Paths are safe:
pathlibjoins; temp files viatempfile. - Subprocess is bounded: argv list,
check=True,timeout=, and defaultshell=False.
Related PLEX reading
References & further reading
- Python docs: Exception context (exception chaining with
raise ... from ...) - Python docs: logging (including
Logger.exception) - Python docs: subprocess (timeouts,
check=True, andshell) - Python docs: tempfile (safe temporary files)
- Python docs: pathlib (portable path handling)
- Requests docs: Timeouts