Advanced typing — Variance, ParamSpec, Self, overload

In Chapter 9 typing in earnest we covered Generic, Protocol, TypedDict, and Literal. This chapter is the next step — the more advanced parts of the typing system. Tools like variance, ParamSpec / Concatenate, Self, TypeGuard / TypeIs, and @overload.

This chapter is mostly for library authors and doesn’t need to appear in regular code. But you should know it for reading library code or building precise types. The Annotated from this chapter reappears in Chapter 24 Pydantic v2 in depth of Part 4 as a core tool of FastAPI.

Variance — covariance / contravariance #

Starting from the most confusing part. Is list[Cat] a subtype of list[Animal]?

Intuitively it’s easy to think “since Cat is an Animal, list[Cat] is list[Animal]” — but that’s wrong. Let’s see why.

def add_dog(animals: list[Animal]) -> None:
    animals.append(Dog())

cats: list[Cat] = [Cat(), Cat()]
add_dog(cats)    # ✗ if allowed, a Dog ends up in cats

list[Cat] is not list[Animal]. Because both reading and writing are possible (mutable), it’s hard to guarantee safety either way.

Three kinds of variance #

Intuition #

• Only reading → covariant. Iterable[Cat] can be used as Iterable[Animal] (when read, a Cat is an Animal)
• Both reading and writing → invariant. list[Cat], dict[K, V]
• Argument position → contravariant. Putting a “function that takes Cat” into the role of “function that takes Animal” is dangerous (even a Dog could come in, but the Animal-function role must accept it)

Looking at function types #

def feed_animal(a: Animal): ...
def feed_cat(c: Cat): ...

# Can a 'function that only takes Cat' be put in the position of 'function that takes Animal'?
fn: Callable[[Animal], None] = feed_cat   # ✗ — breaks if a Dog comes in

# The reverse?
fn2: Callable[[Cat], None] = feed_animal  # ✅ — Cat is a part of Animal, so OK

A function’s argument position is contravariant. The return position is covariant.

How do you spell variance? #

The new PEP 695 syntax def fn[T]: from Chapter 9 infers automatically. The old style (TypeVar) had you write it.

from typing import TypeVar

T_co = TypeVar("T_co", covariant=True)
T_contra = TypeVar("T_contra", contravariant=True)

class Producer(Generic[T_co]):
    def get(self) -> T_co: ...

class Consumer(Generic[T_contra]):
    def put(self, item: T_contra) -> None: ...

New code usually doesn’t need to spell this — the tools handle it well. For library authors, the scope of everyday use is being aware of the variance of standard ABCs like Iterable, Callable when using them.

ParamSpec and Concatenate #

Briefly seen in Chapter 12 decorator patterns as a tool for preserving decorator signatures. In more depth.

What ParamSpec is #

def log[**P, R](fn: Callable[P, R]) -> Callable[P, R]:
    def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
        ...
    return wrapper

P is not a type parameter but the parameter signature itself. It represents a bundle of (positional argument types + keyword argument types).

Concatenate — add arguments #

When the decorator wants to add an argument.

from typing import Concatenate, Callable

def with_logger[**P, R](
    fn: Callable[Concatenate[Logger, P], R]
) -> Callable[P, R]:
    logger = make_logger()
    def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
        return fn(logger, *args, **kwargs)
    return wrapper

@with_logger
def do_work(logger: Logger, x: int, y: int) -> int:
    logger.info(f"{x} + {y}")
    return x + y

do_work(2, 3)   # logger is auto-injected

Meaning of Concatenate[Logger, P]: “first arg is Logger, the rest is P as-is.” After decoration, the signature drops the logger argument.

A pattern where the decorator removes an argument so the caller doesn’t have to spell it — a form of dependency injection. The FastAPI Depends in Chapter 23 routing, Pydantic models, dependency injection is the same spot.

Self — own class in method returns #

A tool briefly seen in Chapter 9.

from typing import Self

class Builder:
    def add(self, item: str) -> Self:
        ...
        return self

class SubBuilder(Builder):
    def special(self) -> Self:
        ...
        return self

b = SubBuilder().add("x").special()
# the type of b is inferred precisely as SubBuilder

Using -> "Builder" (forward reference) makes the result type Builder even when called on SubBuilder, blocking the .special() call. Self solves that.

Also in classmethods #

class Item:
    @classmethod
    def from_dict(cls, data: dict) -> Self:
        return cls(**data)

Called on a subclass, the result type is precisely inferred as the subclass.

@overload — same function, different signatures #

The tool that attaches precise types to a function whose return type changes based on the call argument.

from typing import overload

@overload
def parse(value: str) -> str: ...
@overload
def parse(value: int) -> int: ...
@overload
def parse(value: list[str]) -> list[str]: ...

def parse(value):
    # the actual implementation — under the overload signatures
    return value

a = parse("hello")        # str
b = parse(42)              # int
c = parse(["a", "b"])      # list[str]

Definitions decorated with @overload are type-check only. Their body is usually .... Under them comes the single real body.

When to use it? #

• When the return type changes by the argument type
• When branching on a Literal argument

@overload
def get(key: Literal["count"]) -> int: ...
@overload
def get(key: Literal["name"]) -> str: ...
@overload
def get(key: str) -> object: ...

def get(key): ...

When the call site does get("count"), it’s inferred precisely as receiving int.

Standard functions like dict.get are defined this way #

@overload
def get(self, key: K) -> V | None: ...
@overload
def get(self, key: K, default: V) -> V: ...
@overload
def get(self, key: K, default: T) -> V | T: ...

A pattern where d.get("k") → V | None, d.get("k", 0) → int (the type of 0) — precise inference per call becomes possible.

TypeGuard and TypeIs — type-narrowing functions #

A tool for building user-defined type-narrowing functions beyond isinstance.

TypeGuard (3.10+) #

from typing import TypeGuard

def is_str_list(items: list[object]) -> TypeGuard[list[str]]:
    return all(isinstance(x, str) for x in items)

def process(data: list[object]) -> None:
    if is_str_list(data):
        # here data is narrowed to list[str]
        print(", ".join(data))

A function returning TypeGuard[T] becomes a promise that the argument is T when True.

TypeIs (3.13+) — an improved TypeGuard #

from typing import TypeIs

def is_str(x: object) -> TypeIs[str]:
    return isinstance(x, str)

def handle(x: int | str) -> None:
    if is_str(x):
        print(len(x))     # str
    else:
        print(x + 1)      # int  ← the difference is here!

The difference: TypeIs narrows on False as well. TypeGuard only on True. TypeIs is used in more intuitive and safer cases.

TypeGuard remains for compatibility, and new code is recommended to use TypeIs (3.13+).

Annotated — metadata on types #

from typing import Annotated

UserId = Annotated[int, "UserID — users.id in the DB"]
Email = Annotated[str, "email format"]

def find(id: UserId, email: Email): ...

The extra metadata in Annotated[T, ...] is ignored by type checkers. However, libraries can read it at runtime and use it in behavior. FastAPI’s Depends and Query are the most famous examples of this pattern.

from typing import Annotated
from fastapi import Depends, Query

def get_db(): ...

def search(
    q: Annotated[str, Query(min_length=3)],
    db: Annotated[Database, Depends(get_db)],
): ...

The type itself is read as str, Database as-is, while extra behavior specs are bundled along. Covered in earnest in Chapter 24 Pydantic v2 in depth and Chapter 23 routing, Pydantic models, dependency injection.

LiteralString — SQL injection defense (3.11+) #

from typing import LiteralString

def execute(query: LiteralString) -> list[Row]:
    ...

execute("SELECT * FROM users WHERE id = 1")    # OK
execute(f"SELECT * FROM users WHERE id = {user_input}")  # ✗ — the result of an f-string isn't LiteralString

LiteralString is a type that accepts only compile-time-known strings. Strings built from user input (f"...", + user_id, etc.) are rejected. A tool that statically guarantees SQL injection defense.

NewType — same shape, different type #

from typing import NewType

UserId = NewType("UserId", int)
ProductId = NewType("ProductId", int)

def get_user(id: UserId): ...

uid = UserId(123)
pid = ProductId(456)

get_user(uid)    # OK
get_user(pid)    # ✗ — treated as a different type
get_user(123)    # ✗ — bare int can't go in

NewType is just int at runtime. Only the type checker treats it as a different type. Strong in cases where two ints carry different meaning (UserId vs ProductId, USD vs KRW).

Difference from type aliases #

The type aliases in Chapter 2 variables, basic types, and type hints (type UserId = int) are just aliases — you can put UserId where int goes and the reverse. NewType is one-way: int → UserId needs explicit casting; it doesn’t happen automatically.

type aliases are just different names; NewType is separated into a different type.

Generic class — deeper again #

The class Stack[T]: from Chapter 9. Going deeper:

Multiple parameters #

class Cache[K, V]:
    def __init__(self):
        self._data: dict[K, V] = {}

    def get(self, key: K) -> V | None: ...
    def put(self, key: K, value: V) -> None: ...

cache: Cache[str, int] = Cache()

Constraints and bounds #

class SortedList[T: (int, str)]:    # only int or str
    ...

class Container[T: Comparable]:     # only subtypes of Comparable
    ...

Variable length — TypeVarTuple (3.11+) #

def stack[*Ts](*args: *Ts) -> tuple[*Ts]:
    return args

t = stack(1, "hello", 3.14)
# type of t: tuple[int, str, float]

A variable number of type parameters — used in multi-dimensional array libraries like numpy to express dimensions as types.

cast, assert_type, assert_never #

cast — force a type #

from typing import cast

raw: object = get_data()
data = cast(dict[str, int], raw)
# at runtime it's just raw; only the type checker sees dict[str, int]

A last resort when you’re sure but the checker can’t infer it. Misuse can cause runtime accidents, so it’s good to add a reason comment alongside.

assert_type — verify inference #

from typing import assert_type

x = some_function()
assert_type(x, int)    # error if the checker doesn't infer x as int

An assertion to the type checker that “this point must be exactly this type.” Frequently used in library tests.

assert_never — guarantee all cases handled #

from typing import assert_never

def handle(event: Literal["click", "key"]):
    if event == "click": ...
    elif event == "key": ...
    else:
        assert_never(event)    # if a new case is added, this breaks

Has the checker verify that every case of a union was handled. Useful for discriminated union branching.

Exercises #

1. Build a decorator using Concatenate. Have @with_db auto-inject db: Database as an argument so the caller doesn’t write db. Verify the signature is inferred precisely in pyright.
2. Write a def parse(value) function with @overload. Define three overloads str → str, int → int, list[str] → list[str] and a single body. Verify that result = parse(42) infers result as int.
3. Write is_str_list(items: list[object]) -> TypeIs[list[str]] using TypeIs. Verify that inside if is_str_list(data): data is narrowed to list[str], and in the else: branch it remains list[object] (the difference from TypeGuard).

In one line: Variance is invariant (list) / covariant (Iterable, read) / contravariant (Callable argument). ParamSpec + Concatenate preserves decorator signatures + injects arguments. Self for own-class returns in methods; @overload to branch return types by argument. TypeGuard / TypeIs (3.13+) for user-defined narrowing. Annotated is the core of FastAPI / Pydantic. LiteralString for SQL injection defense, NewType for same-shape different-type. cast / assert_type / assert_never for explicit assertions.

Next chapter #

Next, Chapter 21 performance — cProfile, py-spy, memory profiling is the last of Part 3 — covering tools for finding and fixing slow code: cProfile, py-spy, line_profiler, and memory profiling.