#Python
147 posts
Python Automation #4: Web Scraping Part 2 — Dynamic Pages with Playwright
Handle JavaScript-rendered pages that return empty HTML to requests by driving a real browser with Playwright. From install and a first script to wait_for_selector, login automation, infinite scroll, and saving to CSV — all in one pass.
Routing, Pydantic models, and dependency injection
Splitting routes with APIRouter, defining input/output schemas with Pydantic v2, and factoring shared logic out with Depends.
Testing and deploy — pytest, Docker, Railway/Fly
FastAPI integration tests with pytest + httpx, isolation via dependency overrides, multi-stage Docker builds, and cloud deployment on Railway/Fly.
Type checker setup and CI integration
mypy/pyright/ruff configuration and how to avoid conflicts, blocking issues locally with pre-commit, then blocking them at the PR stage with GitHub Actions.
typing in earnest — Generic, Protocol, TypedDict, Literal
The next step from basics type hints — Generic for parameterizing types, Protocol for precise duck typing, TypedDict for dict shapes, and Literal for narrow unions.
Variables, basic types, and type hints
Python is a dynamic language, but modern Python writes types from the start. int/str/bool/None and built-in generics, the int | None shortcut, and mypy/pyright.
Python Automation #3: Web Scraping Part 1 — Static Pages with httpx and BeautifulSoup
Let code read the price, notice, and stock pages you check by hand every day. We fetch HTML with httpx, pick out just the data we need with BeautifulSoup CSS selectors, and save it to CSV — static page scraping from start to finish.
Python Automation #2: Excel Automation — Ending Repetitive Reports with openpyxl
Read Excel files with openpyxl, merge per-branch reports from a folder into a single sheet, and finish with header styling and number formats. We also cover the formula and data_only pitfalls, merged cells, and large-file performance.
Python Automation #1: Ending Repetitive Work — First Scripts and File Organizing
Learn the basic shape of an automation script by building a downloads-folder organizer in Python. We sort and move files with pathlib, add a dry-run safety net, and wire up argparse arguments — one full cycle from start to finish.
Python Testing #7: Running Tests in CI — People Forget, Machines Don't
Build a CI workflow that runs pytest automatically with GitHub Actions. Covers uv caching, a Python version matrix, coverage comments on PRs, pre-commit, and separating slow tests to wrap up the series.
Python Testing #6: Test Design — Good Tests and How to Read Coverage
Test design fundamentals from the AAA pattern and behavior-driven tests to test double terminology, dealing with flaky tests, and reading pytest-cov coverage numbers the right way.
Python Testing #5: Testing the Outside World — Files, HTTP, Databases, and Web Frameworks
A test that mocks everything guarantees nothing about real behavior. A strategy for testing the outside world: write real files with tmp_path, block only the network boundary with respx, and reset databases with transaction rollbacks.