Squashing Bugs Earlier and More Often with Python’s Type Hints
Shifting left is the idea that you can reduce the cost of bugs by discovering and fixing (shifting) them earlier in the development process. Fixing a bug right after it’s written is far cheaper than after a pull request has been reviewed, merged, and deployed to production.
One popular way to shift left is by writing automated tests. Microsoft MVP Kevin Bost has written an excellent article on how tests help you shift left.
Let’s revisit one of the ways to shift left from Kevin’s article:
“On the left side of the spectrum, as soon as the developer starts typing, they are greeted with a wide host of information. IntelliSense tries to autocomplete keywords and identifiers. Similar to a scrupulous know-it-all code-reviewer who must be right, linters and compilers point out flaws and typos in the words that the developer has written. At this point, making changes to the code is easy, quick, and very fresh in the developer’s mind.”
Unit tests offer an excellent way to shift the cost of bugs further left, but how can we shift the cost left even further?
One way is to utilize a new feature in Python 3 – type hints.
I Thought Python Wasn’t Typed?
Python is often thought of as untyped (and it is by default), but it doesn’t have to be. Python 3.5 introduced Pythonistas everywhere to type hints with type annotations, which allow developers to enforce variable types before runtime.
As stated in PEP-484, “This PEP [Python Enhancement Proposal] aims to provide a standard syntax for type annotations, opening up Python code to easier static analysis and refactoring, potential runtime type checking, and (perhaps, in some contexts) code generation utilizing type information.”
Whereas statically typed languages like C# and Java enforce type hints at compile time, Python doesn’t enforce your type hints at compile time*, nor runtime. This is because, as explained in PEP-484, type hints will always be optional, “It should also be emphasized that Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.”
*Compile time: note that Python is indeed compiled, but not to code that your machine can run directly. Instead, it is compiled into bytecode, which is interpreted by the Python Virtual Machine.
Shifting Left with Python
Now that we’ve reviewed type hints in Python, how can we use them to shift left? Let’s look at an example.
Print All the Usernames
# example.py def print_username(user): print(user.name)Code language: Python (python)
In the example above, we see that the function
print_username expects some object with a
name attribute. Let’s define a
User object with a name:
# example.py ... class User: def __init__(self, name): self.name = nameCode language: Python (python)
Let’s pass an instance of the
User class to
from example import * user = User("Inigo Montoya") print_username(user) Inigo MontoyaCode language: Python (python)
If we run
pyright, a static type-checker used by VS Code’s first-party Python language server,
Pylance, we see that there are no complaints:
# example.py def print_username(user): return user.name class User: def __init__(self, name): self.name = name user = User("Inigo Montoya") print_username(user)Code language: Python (python)
# python -m pip install pyright $ pyright example.py ... Searching for source files Found 1 source file pyright 1.1.300 0 errors, 0 warnings, 0 informations Completed in 0.583secCode language: Bash (bash)
Everything is peachy so far, both at runtime and for our type checker, but currently, our type checker isn’t any help if we pass an object that doesn’t have a
# example.py ... print_username(object()) # raises AttributeErrorCode language: Python (python)
$ pyright example.py ... 0 errors, 0 warnings, 0 informationsCode language: Bash (bash)
print_username raises an
AttributeError when it attempts to access the
name attribute that does not exist on the built-in
object instance. A simple suite of unit tests could catch this error before deploying to production:
# test_example.py """ See https://docs.pytest.org/en/7.1.x/how-to/capture-stdout-stderr.html#accessing-captured-output-from-a-test-function for documentation on `capsys` pytest fixture. See https://learn.microsoft.com/dotnet/core/testing/unit-testing-best-practices#naming-your-tests for notes on test naming conventions. See https://learn.microsoft.com/visualstudio/test/unit-test-basics?view=vs-2022#write-your-tests for notes on AAA (Arrange, Act, Assert). """ import pytest def test__print_username__object_with_name_attr__prints_name(capsys): # Arrange user = User("Princess Buttercup") # Act print_username(user) # Assert captured = capsys.readouterr() actual = captured.out.strip() expected = "Princess Buttercup" assert actual == expected def test__print_username__object_without_name_attr__raises_error(): # Arrange user = object() # Act / Assert with pytest.raises(AttributeError): print_username(user)Code language: Python (python)
Using Type Hints
Can we do better than writing a unit test to assert this very simple function? In this case, unit tests feel like a very heavy-handed solution. Enter type hints:
# example_2.py class User: def __init__(self, name): self.name = name def print_username(user: User): print(user.name)Code language: Python (python)
Now if we pass an object that does not have a
pyright gives us an error:
# example_2.py ... print_username(User("Inigo Montoya")) print_username(object()) # raises AttributeErrorCode language: Python (python)
$ pyright example_2.py ... /home/joeriddles/source/blogs/type_hints/example_2.py:12:16 - error: Argument of type "object" cannot be assigned to parameter "user" of type "User" in function "print_username" "object" is incompatible with "User" (reportGeneralTypeIssues) 1 error, 0 warnings, 0 informationsCode language: Bash (bash)
Typically, your IDE would run a type checker for you and automatically report errors, giving you visual feedback (red squiggles) when objects are being used in a way that does not align with the type hints.
For a more detailed walk through the Python standard library’s
typing module, check out this presentation from the Spokane Python User Group.
If It Walks Like a Duck…
Now I hear the folks in the back crying, “But what about the ducks!?” Pythonic code does not typically care about the type of object you pass to a function; it simply cares that it walks and talks like a duck, e.g. it has the attributes and methods you expect. Fortunately, the
typing module has our solution —
Protocol was introduced in PEP-544 to achieve structural subtyping, AKA duck typing. Structural subtyping allows us to write classes that adhere to an interface without explicitly inheriting from a base class. As long as the class walks and talks like the protocol, it can be used wherever the protocol is expected.
We can use the
Protocol class to define an interface-like class with a
name attribute. Then, we update our
user: User parameter to use our new protocol:
# example_3.py from typing import Protocol class HasName(Protocol): name: str def print_username(has_name: HasName): print(has_name.name)Code language: Python (python)
This protocol helps us climb to the top of the Pythonic peak. If we define some other class with a
name attribute, we can now pass it to
print_username, and Python is happy at runtime and
pyright is happy when type checking.
At this point, we may consider renaming
print_name as it’s no longer tied to the
Let’s try to call
print_username with a different class that also has a
name attribute. We’ll use a new
Recipe class with both a
name and a list of
# example_3.py ... class Recipe: def __init__(self, name: str, ingredients: list[str]): self.name: str = name self.ingredients: list[str] = ingredientsCode language: Python (python)
# example_3.py ... print_username(User("Inigo Montoya")) recipe = Recipe( "Zuppa Toscana", [ "Italian sausage", "Onion", "Chicken broth" ] ) print_username(recipe)Code language: Python (python)
$ pyright example_3.py ... 0 errors, 0 warnings, 0 informationsCode language: Bash (bash)
Go Forth and
Will type hints catch all bugs?
No, but they will certainly help you catch many bugs earlier and shift the cost left, reducing the cost of the bugs. Are type hints an excuse not to write unit tests? Certainly not! Type hints and unit tests are separate tools that help developers catch bugs earlier, but neither replaces the other.
Now, go forth and add type hints to your Python code!
Questions about type hints with Python? Drop ’em in the comments below.
The Spokane Python User Group hosts meetups every month! Check out our upcoming events on Meetup.
Does Your Organization Need a Custom Solution?
Let’s chat about how we can help you achieve excellence on your next project!