System
Python Fundamentals & Automation — scripts that read, validate, and report
Write small, reliable automation scripts in Python: isolate dependencies in a virtual environment, read and validate JSON/CSV input with pathlib and the stdlib, log instead of print, parse CLI arguments with argparse, and return meaningful exit codes so other tools can trust the result.
Quick reference — reach for these first
python3 -m venv .venv && source .venv/bin/activateCreate and activate an isolated interpreter + site-packages. Deactivate with `deactivate`. On Windows: `.venv\Scripts\activate`.pip install -r requirements.txtInstall pinned dependencies into the active venv. Capture the current set with `pip freeze > requirements.txt`.import json; data = json.load(open("data.json", encoding="utf-8"))Parse JSON from a file object into Python objects; `json.loads(s)` parses a string, `json.dumps(obj, indent=2)` serializes back.p = argparse.ArgumentParser(); p.add_argument("path"); args = p.parse_args()Declare CLI arguments; `parse_args()` reads sys.argv, auto-builds `--help`, and exits 2 on bad usage.import logging; logging.basicConfig(level=logging.INFO); logging.info("started")Structured, level-filtered output to stderr — use instead of print(). Switch to `level=logging.DEBUG` for verbosity.The read → validate → report pipeline
Text equivalent: A horizontal pipeline wrapped by a virtual environment. Inside the venv, a JSON or CSV input file flows into a read stage (pathlib and json), then a validate stage (try/except and isinstance checks), then a summarize stage over the records, and finally a report stage that prints output and returns exit code 0 on success or a non-zero code on bad input.
Common tasks
Task · Command · What to check
| Task | Command | What to check |
|---|---|---|
| Create a virtual environment | python3 -m venv .venv | ls .venv/bin/python # file exists |
| Activate the environment | source .venv/bin/activate | which python # points inside .venv |
| Install dependencies from a lockfile | pip install -r requirements.txt | pip list | grep requests |
| Pin current dependencies | pip freeze > requirements.txt | head -1 requirements.txt |
| Read a text file via pathlib | from pathlib import Path; text = Path("input.txt").read_text(encoding="utf-8") | print(len(text)) # byte/char count |
| Parse a JSON file | import json; data = json.load(open("data.json", encoding="utf-8")) | print(type(data)) # list or dict |
| Read CSV rows as dicts | import csv; rows = list(csv.DictReader(open("data.csv", newline="", encoding="utf-8"))) | print(rows[0].keys()) |
| Guard risky work with exceptions | try: risky()\nexcept ValueError as e: logging.error(e)\nelse: ok()\nfinally: cleanup() | run with bad input; see ERROR line, no traceback |
| Configure logging once at startup | import logging; logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s") | logging.info("hi") # prints 'INFO hi' to stderr |
| Parse command-line arguments | import argparse; ap = argparse.ArgumentParser(); ap.add_argument("path"); args = ap.parse_args() | python script.py --help # usage prints, exits 0 |
| Call an HTTP JSON API (stdlib, no deps) | from urllib.request import urlopen; import json; d = json.load(urlopen("https://httpbin.org/json")) | print(d) # parsed response |
| Return a useful exit code | import sys; sys.exit(0 if ok else 1) | python script.py; echo $? # 0 on success |
| Run with an isolated interpreter (ignore local junk) | python3 -I script.py data.json | echo $? # exit status of the run |
A script that reads JSON, validates it, summarizes records, and exits with a useful code
validate-and-report
#!/usr/bin/env python3
"""Summarize a JSON array of records; non-zero exit on bad input."""
import argparse, json, logging, sys
from pathlib import Path
logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
def main() -> int:
ap = argparse.ArgumentParser(description="Summarize records")
ap.add_argument("path", type=Path)
args = ap.parse_args()
try:
data = json.loads(args.path.read_text(encoding="utf-8"))
except FileNotFoundError:
logging.error("no such file: %s", args.path)
return 2
except json.JSONDecodeError as e:
logging.error("invalid JSON: %s", e)
return 2
if not isinstance(data, list):
logging.error("expected a JSON array, got %s", type(data).__name__)
return 2
valid = [r for r in data if isinstance(r, dict) and "amount" in r]
if not valid:
logging.error("no valid records found")
return 1
total = sum(float(r["amount"]) for r in valid)
logging.info("records=%d total=%.2f", len(valid), total)
print(f"{len(valid)} records, total {total:.2f}")
return 0
if __name__ == "__main__":
sys.exit(main())echo '[{"amount": 50}, {"amount": 100}]' > good.jsonpython3 summarize.py good.json; echo "exit=$?"
# INFO records=2 total=150.00
# 2 records, total 150.00
# exit=0echo '{bad' > bad.json; python3 summarize.py bad.json; echo "exit=$?"
# ERROR invalid JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
# exit=2python3 summarize.py bad.json && echo 'would run next step'
# ERROR invalid JSON: ...
# (nothing after: the && short-circuits because exit != 0)Troubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| ModuleNotFoundError: No module named 'X' | which python && pip show X | Whether the active interpreter is your venv and whether X is installed in it — the error almost always means the wrong environment is active or the package was never installed here. | Activate the right venv (`source .venv/bin/activate`), then `pip install X` or `pip install -r requirements.txt`. Confirm `which python` points inside `.venv`. |
| zsh/bash: command not found: python | command -v python3 && python3 --version | Many systems ship only the `python3` name, not `python`. If `python3` works, the plain `python` alias is simply absent — not a broken install. | Use `python3` (and `pip3`), or activate a venv which provides a `python` symlink on PATH. Avoid system-wide aliasing that masks the real binary. |
| json.decoder.JSONDecodeError: Expecting value: line 1 column 1 | head -c 200 data.json | Whether the file actually contains JSON — an empty file, an HTML error page, a BOM/trailing comma, or single quotes all trip the parser. The message names the exact line and column. | Fix the source so it is valid JSON (double quotes, no trailing commas). Wrap the parse in `except json.JSONDecodeError` and exit non-zero with a clear message rather than crashing. |
| TabError / IndentationError: inconsistent use of tabs and spaces | python3 -tt script.py | Confirms the block mixes tab and space indentation — Python 3 refuses to guess. `-tt` turns ambiguous indentation into a hard error pointing at the line. | Convert the file to spaces (4 per level): configure the editor to expand tabs, or run `expand -t4`. Never mix the two within one file. |
| A function's default list/dict keeps data from previous calls | Inspect the signature for `def f(x, items=[])` or `={}` | The mutable default is evaluated once at definition time and shared across every call — a classic gotcha, not a runtime glitch. Accumulating values across calls confirms it. | Default to `None` and create inside: `def f(x, items=None):\n if items is None: items = []`. Reserve mutable defaults only for a deliberate cache. |
| Script 'succeeds' in a pipeline even though input was bad | python3 script.py bad.json; echo $? | Whether the script returns a non-zero exit code on failure. If `echo $?` prints 0 after an error, callers (make, CI, shell `&&`) will wrongly treat it as success. | Route all exits through `sys.exit(code)` / `return code` from main: 0 for success, 1 for 'ran but no usable result', 2 for bad usage/input. Test each path with `echo $?`. |
References
- venv — Creation of virtual environmentsdocs.python.org
- json — JSON encoder and decoderdocs.python.org
- csv — CSV File Reading and Writingdocs.python.org
- argparse — Parser for command-line optionsdocs.python.org
- logging — Logging facility for Pythondocs.python.org
Python 3.8+ on a Unix-like shell (macOS/Linux); activation and the `python3` name differ on Windows (`.venv\Scripts\activate`, often `python`). Examples use only the standard library except where `requests` is mentioned as an alternative to `urllib`.