Are you an intermediate Python developer looking to level up? Luckily, Python provides us with a unique set of tools to make our code more elegant and readable. I’ll share practical pythonic solutions for supercharging your code with tools like Decorators, Context Managers, and NamedTuples.
A Secure and Reliable Document Management System is Essential.docx
Elegant Solutions For Everyday Python Problems - PyCon Canada 2017
1. Elegant Solutions For Everyday
Python Problems
Nina Zakharenko
@nnja
bit.ly/elegant-python-ca
ℹ There are links in these slides. Follow along ^
2. This talk is for you if:
— You're an intermediate python programmer
— You're coming to python from another language
— You want to learn about fancy features like: magic
methods, iterators, decorators, and context managers
slides: bit.ly/elegant-python-ca
@nnja
7. You're used to implementing __str__ and __repr__ --but
there's a whole other world of powerful magic methods!
By implementing a few straightforward methods,
you can make your objects behave like built-ins such as:
— numbers
— lists
— dictionaries
— and more...
@nnja
17. Making classes iterable
— In order to be iterable, a class needs to implement
__iter__()
— __iter__() must return an iterator
— In order to be an iterator a class needs to implement
__next__() which must raise StopIteration when there
are no more items to return
or next() in python2
^ can be confusing at first, but remember these guidelines for making classesGreat explanation of iterable vs. iterator vs. generator
18. class IterableServer:
services = [
{'active': False, 'protocol': 'ftp', 'port': 21},
{'active': True, 'protocol': 'ssh', 'port': 22},
{'active': True, 'protocol': 'http', 'port': 21},
]
def __init__(self):
self.current_pos = 0
def __iter__(self): # can return self, because __next__ implemented
return self
def __next__(self):
while self.current_pos < len(self.services):
service = self.services[self.current_pos]
self.current_pos += 1
if service['active']:
return service['protocol'], service['port']
raise StopIteration
next = __next__ # optional python2 compatibility
@nnja
19. >>> for protocol, port in IterableServer():
print('service %s is running on port %d' % (protocol, port))
service ssh is running on port 22
service http is running on port 21
... not bad
@nnja
20. tip: use a generator
when your iterator doesn't need to
maintain a lot of state
@nnja
21. class Server:
services = [
{'active': False, 'protocol': 'ftp', 'port': 21},
{'active': True, 'protocol': 'ssh', 'port': 22},
{'active': True, 'protocol': 'http', 'port': 21},
]
def __iter__(self):
for service in self.services:
if service['active']:
yield service['protocol'], service['port']
@nnja
22. class Server:
services = [
{'active': False, 'protocol': 'ftp', 'port': 21},
{'active': True, 'protocol': 'ssh', 'port': 22},
{'active': True, 'protocol': 'http', 'port': 21},
]
def __iter__(self):
for service in self.services:
if service['active']:
yield service['protocol'], service['port']
@nnja
23. Why does this work?
use single parenthesis ( ) to create a generator
comprehension
^ technically, a generator expression but I like this term better, and so does Ned Batchelder
>>> my_gen = (num for num in range(1))
>>> my_gen
<generator object <genexpr> at 0x107581bf8>
@nnja
24. An iterator must implement __next__()
>>> next(my_gen) # remember __len__() mapped to built-in len()
0
and raise StopIteration when
there are no more elements
>>> next(my_gen)
... StopIteration Traceback (most recent call last)
For more tools for working with iterators, check out itertools
26. alias methods
class Word:
def __init__(self, word):
self.word = word
def __repr__(self):
return self.word
def __add__(self, other_word):
return Word('%s %s' % (self.word, other_word))
# Add an alias from method __add__ to the method concat
concat = __add__
@nnja
27. When we add an alias from __add__ to concat because
methods are just objects
>>> # remember, concat = __add__
>>> first_name = Word('Max')
>>> last_name = Word('Smith')
>>> first_name + last_name
Max Smith
>>> first_name.concat(last_name)
Max Smith
>>> Word.__add__ == Word.concat
True
@nnja
28. Dog class
>>> class Dog:
sound = 'Bark'
def speak(self):
print(self.sound + '!', self.sound + '!')
>>> my_dog = Dog()
>>> my_dog.speak()
Bark! Bark!
read the docs
31. Example: command line tool with dynamic commands
class Operations:
def say_hi(self, name):
print('Hello,', name)
def say_bye(self, name):
print ('Goodbye,', name)
def default(self, arg):
print ('This operation is not supported.')
if __name__ == '__main__':
operations = Operations()
# let's assume error handling
command, argument = input('> ').split()
getattr(operations, command, operations.default)(argument)
read the docs
32. Output
$ python getattr.py
> say_hi Nina
Hello, Nina
> blah blah
This operation is not supported.
✨
additional reading - inverse of getattr() is setattr()
33. functool.partial(func, *args, **kwargs)
— Return a new partial object which behaves like func
called with args & kwargs
— if more args are passed in, they are appended to args
— if more keyword arguments are passed in, they extend
and override kwargs
read the docs on partials
34. functool.partial(func, *args, **kwargs)
>>> from functools import partial
>>> basetwo = partial(int, base=2)
>>> basetwo
functools.partial(<class 'int'>, base=2)
>>> basetwo('10010')
18
read the docs on partials
35. library I
!
: github.com/jpaugh/agithub
agithub is a (badly named) REST API client with
transparent syntax which facilitates rapid prototyping
— on any REST API!
— Implemented in 400 lines.
— Add support for any REST API in ~30 lines of code.
— agithub knows everything it needs to about protocol
(REST, HTTP, TCP), but assumes nothing about your
upstream API.
@nnja
36. define endpoint url & other connection properties
class GitHub(API):
def __init__(self, token=None, *args, **kwargs):
props = ConnectionProperties(
api_url = kwargs.pop('api_url', 'api.github.com'))
self.setClient(Client(*args, **kwargs))
self.setConnectionProperties(props)
then, start using the API!
>>> gh = GitHub('token')
>>> status, data = gh.user.repos.get(visibility='public', sort='created')
>>> # ^ Maps to GET /user/repos
>>> data
... ['tweeter', 'snipey', '...']
github.com/jpaugh/agithub
41. given a non-existant path:
>>> status, data = this.path.doesnt.exist.get()
>>> status
... 404
& because __getitem__ is aliased to __getattr__:
>>> owner, repo = 'nnja', 'tweeter'
>>> status, data = gh.repos[owner][repo].pulls.get()
>>> # ^ Maps to GET /repos/nnja/tweeter/pulls
>>> data
.... # {....}
github.com/jpaugh/agithub
43. When should I use one?
Need to perform an action before and/or after an
operation.
Common scenarios:
— Closing a resource after you're done with it (file,
network connection)
— Perform cleanup before/after a function call
@nnja
44. Example Problem: Feature Flags
Turn features of your application on and off easily.
Uses of feature flags:
— A/B Testing
— Rolling Releases
— Show Beta version to users opted-in to Beta Testing
Program
More on Feature Flags
45. Example - FeatureFlags Class
class FeatureFlags:
""" Example class which stores Feature Flags and their state. """
SHOW_BETA = 'Show Beta version of Home Page'
flags = {
SHOW_BETA: True
}
@classmethod
def is_on(cls, name):
return cls.flags[name]
@classmethod
def toggle(cls, name, on):
cls.flags[name] = on
feature_flags = FeatureFlags()
@nnja
46. How do we temporarily turn features on and off when
testing flags?
Want:
with feature_flag(FeatureFlags.SHOW_BETA):
assert '/beta' == get_homepage_url()
@nnja
47. Using Magic Methods __enter__ and __exit__
class feature_flag:
""" Implementing a Context Manager using Magic Methods """
def __init__(self, name, on=True):
self.name = name
self.on = on
self.old_value = feature_flags.is_on(name)
def __enter__(self):
feature_flags.toggle(self.name, self.on)
def __exit__(self, *args):
feature_flags.toggle(self.name, self.old_value)
See: contextlib.contextmanager
48. The be!er way: using the contextmanager decorator
from contextlib import contextmanager
@contextmanager
def feature_flag(name, on=True):
old_value = feature_flags.is_on(name)
feature_flags.toggle(name, on)
yield
feature_flags.toggle(name, old_value)
See: contextlib.contextmanager
49. The be!er way: using the contextmanager decorator
from contextlib import contextmanager
@contextmanager
def feature_flag(name, on=True):
""" The easier way to create Context Managers """
old_value = feature_flags.is_on(name)
feature_flags.toggle(name, on) # behavior of __enter__()
yield
feature_flags.toggle(name, old_value) # behavior of __exit__()
See: contextlib.contextmanager
50. Note: yield?
from contextlib import contextmanager
@contextmanager
def feature_flag(name, on=True):
""" The easier way to create Context Managers """
old_value = feature_flags.is_on(name)
feature_flags.toggle(name, on) # behavior of __enter__()
yield
feature_flags.toggle(name, old_value) # behavior of __exit__()
See: contextlib.contextmanager
51. either implementation
def get_homepage_url():
""" Method that returns the path of the home page we want to display. """
if feature_flags.is_on(FeatureFlags.SHOW_BETA):
return '/beta'
else:
return '/homepage'
def test_homepage_url_with_context_manager():
with feature_flag(FeatureFlags.SHOW_BETA):
# saw the beta homepage...
assert get_homepage_url() == '/beta'
with feature_flag(FeatureFlags.SHOW_BETA, on=False):
# saw the standard homepage...
assert get_homepage_url() == '/homepage'
@nnja
52. either implementation
def get_homepage_url():
""" Method that returns the path of the home page we want to display. """
if feature_flags.is_on(FeatureFlags.SHOW_BETA):
return '/beta'
else:
return '/homepage'
def test_homepage_url_with_context_manager():
with feature_flag(FeatureFlags.SHOW_BETA):
assert get_homepage_url() == '/beta'
print('seeing the beta homepage...')
with feature_flag(FeatureFlags.SHOW_BETA, on=False):
assert get_homepage_url() == '/homepage'
print('seeing the standard homepage...')
@nnja
54. Recap!
Decorators:
— Wrap a function in another function.
— Do something:
— before the call
— after the call
— with provided arguments
— modify the return value or arguments
@nnja
55. def say_after(hello_function):
def say_nice_to_meet_you(name):
hello_function(name)
print('It was nice to meet you!')
return say_nice_to_meet_you
def hello(name):
print('Hello', name)
>>> hello('Nina')
Hello Nina
>>> say_after(hello)('Nina')
Hello Nina It was nice to meet you!
— say_after(hello) returns the function
say_nice_to_meet_you
— then we call say_nice_to_meet_you('Nina')
@nnja
56. def say_after(hello_function):
def say_nice_to_meet_you(name):
hello_function(name)
print('It was nice to meet you!')
return say_nice_to_meet_you
@say_after
def hello(name):
print('Hello', name)
>>> hello('Nina')
Hello Nina It was nice to meet you!
— calling the decorated function hello(name)
— is the same as calling an undecorated hello with
say_after(hello)('Nina')
@nnja
57. closure example
def multiply_by(num):
def do_multiplication(x):
return x * num
return do_multiplication
multiply_by_five = multiply_by(5)
>>> multiply_by_five(4)
20
@nnja
58. decorators that take arguments
def greeting(argument):
def greeting_decorator(greet_function):
def greet(name):
greet_function(name)
print('It was %s to meet you!' % argument)
return greet
return greeting_decorator
@greeting('bad')
def aloha(name):
print ('Aloha', name)
@nnja
59. decorators that take arguments
def say_this_after(argument):
def say_after(hello_function):
def say_after_meeting(name):
hello_function(name)
print('It was %s to meet you' % argument)
return say_after_meeting
return say_after
@say_this_after('bad')
def hello(name):
print('Hello', name)
Is the same as calling this on an undecorated function:
say_after_bad = say_this_after('bad')(hello)
say_after_bad('Nina')
@nnja
60. losing context with a decorator
def say_bye(func):
def wrapper(name):
func()
print('Bye', name)
return wrapper
@say_bye
def my_name():
""" Say my name"""
print('Nina')
>>> my_name.__name__
'wrapper'
>>>> my_name.__doc__
# ... empty
@nnja
61. solution: use wraps, or wrapt library!
from contextlib import wraps
def say_adios(func):
@wraps(func) # pass in which function to wrap
def wrapper():
func()
print('Adios!')
return wrapper
@say_adios
def say_max():
""" Says the name Max"""
print('Max')
>>> say_max.__name__
'say_max'
>>> say_max.__doc__
' Says the name Max'
@nnja
64. As of python 3.2 ContextDecorators are in the standard
library. They're the best of both worlds!
— By using ContextDecorator you can easily write classes
that can be used both as decorators with @ and
context managers with the with statement.
— ContextDecorator is used by contextmanager(), so you
get this functionality
✨
automatically .
— Alternatively, you can write a class that extends from ContextDecorator or uses
ContextDecorator as a mixin, and implements __enter__, __exit__ and __call__
— If you use python2, a backport package is available here: contextlib2
@nnja
66. use it as a context manager
def get_homepage_url():
beta_flag_on = feature_flags.is_on(FeatureFlags.SHOW_BETA)
return '/beta' if beta_flag_on else '/homepage'
with feature_flag(FeatureFlags.SHOW_BETA):
assert get_homepage_url() == '/beta'
or use as a decorator
@feature_flag(FeatureFlags.SHOW_BETA, on=False)
def get_profile_page():
beta_flag_on = feature_flags.is_on(FeatureFlags.SHOW_BETA)
return 'beta.html' if beta_flag_on else 'profile.html'
assert get_profile_page() == 'profile.html'
@nnja
67. library I
!
: freezegun lets your python tests ❇ travel
through time! ❇
from freezegun import freeze_time
# use it as a Context Manager
def test():
with freeze_time("2012-01-14"):
assert datetime.datetime.now() == datetime.datetime(2012, 1, 14)
assert datetime.datetime.now() != datetime.datetime(2012, 1, 14)
# or a decorator
@freeze_time("2012-01-14")
def test():
assert datetime.datetime.now() == datetime.datetime(2012, 1, 14)
read the source sometime, it's mind-bending!
@nnja
68. NamedTuple
Useful when you need lightweight representations of
data.
Create tuple subclasses with named fields.
@nnja
69. Example
from collections import namedtuple
CacheInfo = namedtuple(
"CacheInfo", ["hits", "misses", "max_size", "curr_size"])
@nnja
70. Giving NamedTuples default values
RoutingRule = namedtuple(
'RoutingRule',
['prefix', 'queue_name', 'wait_time']
)
(1) By specifying defaults
RoutingRule.__new__.__defaults__ = (None, None, 20)
(2) or with _replace to customize a prototype instance
default_rule = RoutingRule(None, None, 20)
user_rule = default_rule._replace(prefix='user', queue_name='user-queue')
@nnja
71. NamedTuples can be subclassed and extended
class Person(namedtuple('Person', ['first_name', 'last_name'])):
""" Stores first and last name of a Person"""
__slots__ = ()
def __str__(self):
return '%s %s' % (self.first_name, self.last_name)
>>> me = Person('nina', 'zakharenko')
>>> str(me)
'nina zakharenko'
>>> me
Person(first_name='nina', last_name='zakharenko')
@nnja
72. Tip
Use __slots__ = () in your NamedTuples!
— It prevents the creation of instance dictionaries.
— It lowers memory consumption.
— Allows for faster access
@nnja
73. "Perfection is achieved, not when
there is nothing more to add, but
when there is nothing left to take
away."
— Antoine de Saint-Exupery
@nnja
74. New Tools
— Magic Methods
— make your objects behave like builtins (numbers,
list, dict, etc)
— Method ❇Magic❇
— alias methods
— * getattr
— functool.partial
@nnja
75. — ContextManagers
— Close resources
— Decorators
— do something before/after call, modify return value
or validate arguments
— ContextDecorators
— ContextManagers + Decorators combined!
@nnja
76. — Iterators & Generators
— Loop over your objects
— yield
— NamedTuple
— Lightweight classes
@nnja