Squashing Bugs Earlier and More Often with Python’s Type Hints
Shifting left is the idea that you can reduce the cost of bugs by discovering and fixing (shifting) them earlier in the development process. Fixing a bug right after it’s written is far cheaper than after a pull request has been reviewed, merged, and deployed to production.
One popular way to shift left is by writing automated tests. Microsoft MVP Kevin Bost has written an excellent article on how tests help you shift left.
Let’s revisit one of the ways to shift left from Kevin’s article:
“On the left side of the spectrum, as soon as the developer starts typing, they are greeted with a wide host of information. IntelliSense tries to autocomplete keywords and identifiers. Similar to a scrupulous know-it-all code-reviewer who must be right, linters and compilers point out flaws and typos in the words that the developer has written. At this point, making changes to the code is easy, quick, and very fresh in the developer’s mind.”
Unit tests offer an excellent way to shift the cost of bugs further left, but how can we shift the cost left even further?
One way is to utilize a new feature in Python 3 – type hints.
I Thought Python Wasn’t Typed?
Python is often thought of as untyped (and it is by default), but it doesn’t have to be. Python 3.5 introduced Pythonistas everywhere to type hints with type annotations, which allow developers to enforce variable types before runtime.
As stated in PEP-484, “This PEP [Python Enhancement Proposal] aims to provide a standard syntax for type annotations, opening up Python code to easier static analysis and refactoring, potential runtime type checking, and (perhaps, in some contexts) code generation utilizing type information.”
Whereas statically typed languages like C# and Java enforce type hints at compile time, Python doesn’t enforce your type hints at compile time*, nor runtime. This is because, as explained in PEP-484, type hints will always be optional, “It should also be emphasized that Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.”
*Compile time: note that Python is indeed compiled, but not to code that your machine can run directly. Instead, it is compiled into bytecode, which is interpreted by the Python Virtual Machine.
Shifting Left with Python
Now that we’ve reviewed type hints in Python, how can we use them to shift left? Let’s look at an example.
Print All the Usernames
# example.py
def print_username(user):
print(user.name)
Code language: Python (python)
In the example above, we see that the function print_username
expects some object with a name
attribute. Let’s define a User
object with a name:
# example.py
...
class User:
def __init__(self, name):
self.name = name
Code language: Python (python)
Let’s pass an instance of the User
class to print_username
:
>>> from example import *
>>> user = User("Inigo Montoya")
>>> print_username(user)
Inigo Montoya
Code language: Python (python)
If we run pyright
, a static type-checker used by VS Code’s first-party Python language server, Pylance
, we see that there are no complaints:
# example.py
def print_username(user):
return user.name
class User:
def __init__(self, name):
self.name = name
user = User("Inigo Montoya")
print_username(user)
Code language: Python (python)
# python -m pip install pyright
$ pyright example.py
...
Searching for source files
Found 1 source file
pyright 1.1.300
0 errors, 0 warnings, 0 informations
Completed in 0.583sec
Code language: Bash (bash)
Everything is peachy so far, both at runtime and for our type checker, but currently, our type checker isn’t any help if we pass an object that doesn’t have a name
attribute:
# example.py
...
print_username(object()) # raises AttributeError
Code language: Python (python)
$ pyright example.py
...
0 errors, 0 warnings, 0 informations
Code language: Bash (bash)
At runtime, print_username
raises an AttributeError
when it attempts to access the name
attribute that does not exist on the built-in object
instance. A simple suite of unit tests could catch this error before deploying to production:
# test_example.py
"""
See https://docs.pytest.org/en/7.1.x/how-to/capture-stdout-stderr.html#accessing-captured-output-from-a-test-function
for documentation on `capsys` pytest fixture.
See https://learn.microsoft.com/dotnet/core/testing/unit-testing-best-practices#naming-your-tests
for notes on test naming conventions.
See https://learn.microsoft.com/visualstudio/test/unit-test-basics?view=vs-2022#write-your-tests
for notes on AAA (Arrange, Act, Assert).
"""
import pytest
def test__print_username__object_with_name_attr__prints_name(capsys):
# Arrange
user = User("Princess Buttercup")
# Act
print_username(user)
# Assert
captured = capsys.readouterr()
actual = captured.out.strip()
expected = "Princess Buttercup"
assert actual == expected
def test__print_username__object_without_name_attr__raises_error():
# Arrange
user = object()
# Act / Assert
with pytest.raises(AttributeError):
print_username(user)
Code language: Python (python)
Using Type Hints
Can we do better than writing a unit test to assert this very simple function? In this case, unit tests feel like a very heavy-handed solution. Enter type hints:
# example_2.py
class User:
def __init__(self, name):
self.name = name
def print_username(user: User):
print(user.name)
Code language: Python (python)
Now if we pass an object that does not have a name
attribute, pyright
gives us an error:
# example_2.py
...
print_username(User("Inigo Montoya"))
print_username(object()) # raises AttributeError
Code language: Python (python)
$ pyright example_2.py
...
/home/joeriddles/source/blogs/type_hints/example_2.py:12:16 - error: Argument of type "object" cannot be assigned to parameter "user" of type "User" in function "print_username"
"object" is incompatible with "User" (reportGeneralTypeIssues)
1 error, 0 warnings, 0 informations
Code language: Bash (bash)
Typically, your IDE would run a type checker for you and automatically report errors, giving you visual feedback (red squiggles) when objects are being used in a way that does not align with the type hints.
For a more detailed walk through the Python standard library’s typing
module, check out this presentation from the Spokane Python User Group.
If It Walks Like a Duck…
Now I hear the folks in the back crying, “But what about the ducks!?” Pythonic code does not typically care about the type of object you pass to a function; it simply cares that it walks and talks like a duck, e.g. it has the attributes and methods you expect. Fortunately, the typing
module has our solution — Protocol
.
Protocol
was introduced in PEP-544 to achieve structural subtyping, AKA duck typing. Structural subtyping allows us to write classes that adhere to an interface without explicitly inheriting from a base class. As long as the class walks and talks like the protocol, it can be used wherever the protocol is expected.
We can use the Protocol
class to define an interface-like class with a name
attribute. Then, we update our user: User
parameter to use our new protocol: has_name: HasName
.
# example_3.py
from typing import Protocol
class HasName(Protocol):
name: str
def print_username(has_name: HasName):
print(has_name.name)
Code language: Python (python)
This protocol helps us climb to the top of the Pythonic peak. If we define some other class with a name
attribute, we can now pass it to print_username
, and Python is happy at runtime and pyright
is happy when type checking.
At this point, we may consider renaming print_username
to print_name
as it’s no longer tied to the User
model.
Let’s try to call print_username
with a different class that also has a name
attribute. We’ll use a new Recipe
class with both a name
and a list of ingredients
.
# example_3.py
...
class Recipe:
def __init__(self, name: str, ingredients: list[str]):
self.name: str = name
self.ingredients: list[str] = ingredients
Code language: Python (python)
# example_3.py
...
print_username(User("Inigo Montoya"))
recipe = Recipe(
"Zuppa Toscana",
[
"Italian sausage",
"Onion",
"Chicken broth"
]
)
print_username(recipe)
Code language: Python (python)
$ pyright example_3.py
...
0 errors, 0 warnings, 0 informations
Code language: Bash (bash)
Go Forth and Type
Will type hints catch all bugs?
No, but they will certainly help you catch many bugs earlier and shift the cost left, reducing the cost of the bugs. Are type hints an excuse not to write unit tests? Certainly not! Type hints and unit tests are separate tools that help developers catch bugs earlier, but neither replaces the other.
Now, go forth and add type hints to your Python code!
Questions about type hints with Python? Drop ’em in the comments below.
Want More?
The Spokane Python User Group hosts meetups every month! Check out our upcoming events on Meetup.
Does Your Organization Need a Custom Solution?
Let’s chat about how we can help you achieve excellence on your next project!