Table of Contents
Common Python notes
Magic methods
Methods with “dunder” or “__” like “__init__” are magic methods. Magic methods are related to OOP classes in Python and one of the biggest advantages of using Python’s magic methods is that they provide a simple way to make objects behave like built-in types. That means you can avoid ugly, counter-intuitive, and nonstandard ways of performing basic operators. like some languages will have you do:
if instance.equals(other_instance): # do something
You could certainly do this in Python, too, but this adds confusion and is unnecessarily verbose. Different libraries might use different names for the same operations, making the client do way more work than necessary. With the power of magic methods, however, we can define one method (__eq__, in this case), and say what we mean instead:
if instance == other_instance: #do something
Making classes iterable
Iterators
Iterable classes are a great way to implement the lazy loading pattern. Here all the data is not returned in one go. Instead one needs to keep calling __next__ until it raises exception when there are no more items to return.
To make a class object iterable, you need to implement “__iter__” and “__next__” (next in Python 2) method.
class Yes(collections.Iterator): def __init__(self, stop): self.x = 0 self.stop = stop def __iter__(self): return self def __next__(self): if self.x < self.stop: self.x += 1 return 'yes' else: # Iterators must raise when done, else considered broken raise StopIteration next = __next__ # Python 2 compatibility
Generators
Generators are a subtype of Iterators. In the sense that all generators are iterators but not all iterators are generators. As you can see above, iterators need you to maintain state(self.x
). In most scenarios, this can be simplified with syntactic sugar so that this state management is done for you out of the box. This is exactly what generators do. If state-management is more complex, then one needs to write an iterator and generator is not possible.
A generator function is a function with yield in it. A function with yield in it is still a function, that, when called, returns an instance of a generator object. Simply put, yield gives you a generator. You’d use yield
where you would normally use a return
in a function. Again:
yield is only legal inside of a function definition, and the inclusion of yield in a function definition makes it return a generator.
Same thing as above using generators:
def yes(stop): for _ in range(stop): yield 'yes'
Or perhaps simpler, a Generator Expression (works similarly to list comprehensions):
* Generator Expressions are implemented using single parentheses ()
yes_expr = ('yes' for _ in range(stop))
They can all be used in the same way:
>>> stop = 4 >>> for i, y1, y2, y3 in zip(range(stop), Yes(stop), yes(stop), ('yes' for _ in range(stop))): ... print('{0}: {1} == {2} == {3}'.format(i, y1, y2, y3)) ... 0: yes == yes == yes 1: yes == yes == yes 2: yes == yes == yes 3: yes == yes == yes
Two EXCELLENT resources for yield/generators:
https://jeffknupp.com/blog/2013/04/07/improve-your-python-yield-and-generators-explained/
https://jeffknupp.com/blog/2018/06/04/a-common-misunderstanding-about-python-generators/
getattr
getattr(object, 'x')
is completely equivalent to object.x
.
There’s only in two reasons to use it:
- you can’t write
object.x
, because you don’t know in advance which attribute you want (it comes from a string). very useful for meta-programming. - you want to provide a default value.
object.y
will raise anAttributeError
if there’s no y. Butgetattr(object, 'y', 5)
will return 5.
functools.partial
functools is a library dealing with higher order functions. partial is part of functools.
What functools.partial
does is:
- Makes a new version of a function with one or more arguments already filled in.
- New version of a function documents itself.
Example:
def power(base, exponent): return base ** exponent from functools import partial square = partial(power, exponent=2) cube = partial(power, exponent=3) def test_partials(): assert square(2) == 4 assert cube(2) == 8
A great explanation to partial functions is here: https://www.pydanny.com/python-partials-are-fun.html
A great usage of partial functions is here: https://github.com/tkaemming/django-subdomains/blob/master/subdomains/utils.py#L65-L72
Context Managers
Context managers help us to manage resources efficiently so that we can specify what and howto setup or teardown when working with certain objects. For example the with open()
for file read.
with open('sample.txt', 'w') as f: f.write('Lorem Ipsum') # this is equivalent to: f = open('sample.txt', 'w') f.write('Lorem Ipsum') f.close()
In more general terms, we need a context manager when we need to perform an action before and /or after an operation
Common scenarios:
- Closing a resource after you’re done with it (file, network connection)
- Perform cleanuup before/after a function call
Example – Feature Flags
We want to turn features of an application on and off easily
Uses of feature flags:
– A/B testing: we want to see whether a feature turned on generates more customer conversion vs when turned off
– Rolling eleases: we want to turn a feature on only to say 5% of our users
– Show beta version to users opted-in to Beta Testing Program
Let’s see this in a FeatureFlag class:
class FeatureFlags: ''' example class which stores Feature Flags and their state. we only have on feature flag which is called SHOW_BETA and this tells us to show beta version or not. Defalt, flag is True. I have two methods, one to show whether the flag is on or off and another to toggle the flag value. ''' SHOW_BETA = "Show beta version of the home page" flags = { SHOW_BETA: True } @classmethod def is_on(cls, name): return cls.flags[name] @classmethod def toggle(cls, name, on): cls.flags[name] = on # instantiate the class feature_flag = FeatureFlags()
How do we temporarily turn features on and off when testing flags?
Want:
with feature_flag(FeatureFlags.SHOW_BETA): assert '/beta' == get_homepage_url()
We can do this by making FeatureFlags into a context manager by implementing __enter__ and __exit__ magic methods
class FeatureFlags: ''' example class which stores Feature Flags and their state. we only have on feature flag which is called SHOOW_BETA and this tells is to show beta version or not. Defalt, flag is True. I have two methods, one to show whether the flag is on or off and another to toggle tha flag value. ''' SHOW_BETA = "Show beta version of the home page" flags = { SHOW_BETA: True } def __init__(self, name, on=True): self.name = name self.on = on self.old_value = FeatureFlags.is_on(name) def __enter__(self): FeatureFlags.toggle(self.name, self.on) def __exit__(self): FeatureFlags.toggle(self.name, self.old_value) @classmethod def is_on(cls, name): return cls.flags[name] @classmethod def toggle(cls, name, on): cls.flags[name] = on # instantiate the class feature_flag = FeatureFlags() with feature_flag(FeatureFlags.SHOW_BETA): assert '/beta' == get_homepage_url()
There is an even better way to implement context managers via the @contextmanager
decorator. That is going to allow us to create a factory function for with
statement without having to explicitly insert the __enter__ and __exit__ methods in the FeatureFlags class.
class FeatureFlags: ''' example class which stores Feature Flags and their state. we only have on feature flag which is called SHOOW_BETA and this tells is to show beta version or not. Defalt, flag is True. I have two methods, one to show whether the flag is on or off and another to toggle tha flag value. ''' SHOW_BETA = "Show beta version of the home page" flags = { SHOW_BETA: True } @classmethod def is_on(cls, name): return cls.flags[name] @classmethod def toggle(cls, name, on): cls.flags[name] = on from contextlib import contextmanager @contextmanager def feature_flag(name, on=True): old_value = FeatureFlags.is_on(name) FeatureFlags.toggle(name, on) yield FeatureFlags.toggle(name, old_value) with feature_flag(FeatureFlags.SHOW_BETA): assert '/beta' == get_homepage_url()
A very useful link: https://jeffknupp.com/blog/2016/03/07/python-with-context-managers/
Decorators
Decorators are higher order functions that allow us to modify an underlying function. By higher order function, we mean that decorator is a callable that returns a callable.
We often use decorators to:
- Wrap a function in another function
- Do something:
- before the call
- after the call
- with provided arguments
- modify the return value or arguments
A very simple decorator
def say_after(func): def wrapper(*args, **kwargs): func(*args, **kwargs) print('Nice to meet you') return wrapper @say_after def say_hello(name): print(f'hello {name}') say_hello('sam')
Will create this output
hello sam Nice to meet you
Another simple example that shows some more inner workings:
def decorator_function(func): def wrapper(*args, **kwargs): print(f'args passed to wrapper:{args}') print(f'Executed before function "{func.__name__}"') func(*args, **kwargs) print(f'Executed after function "{func.__name__}"') return wrapper @decorator_function def say_something(arg1:str, arg2:str) -> None: print(f'{arg1} {arg2}') say_something('Hello', 'World')
Will create this output:
args passed to wrapper:('Hello', 'World') Executed before function "say_something" Hello World Executed after function "say_something"
Closures
Before talking about decorators with arguments, lets take a quick look at closures
def multiply_base(num): def wrapper_multiplication(x): return x*num return wrapper_multiplication multiply_by_6 = multiply_base(6) # this is a function wrapper_multiplication result = multiply_by_6(7) print(result)
result is 42.
Here “num” is still available inside function wrapper_multiplication
because of the closure context.
Decorators that take arguments
So in decorators with arguments the concept is very similar to above closures, the outermost function is just a factory that creates a decorator function. Lets see an example for clarity
def decorator_factory(some_arg_passed): def decorator_function(func): def wrapper(*args, **kwargs): print(f'{some_arg_passed}:args passed to wrapper:{args}') print(f'Executed before function "{func.__name__}"') func(*args, **kwargs) print(f'Executed after function "{func.__name__}"') return wrapper return decorator_function @decorator_factory('passed to decorator') def say_something(arg1:str, arg2:str) -> None: print(f'{arg1} {arg2}') say_something('Hello', 'World')
Will create this output:
passed to decorator:args passed to wrapper:('Hello', 'World') Executed before function "say_something" Hello World Executed after function "say_something"
Here, note the parentheses in the @decorator_factory('passed to decorator')
. In the earlier example, the decorator looked like so @decorator_function
i.e. no parentheses. This is the key when using decorators with arguments. In this case, I am passing argument of ‘passed to decorator’ inside parentheses to the decorator factory. This essentially runs the decorator_factory
function and returns the inner decorator_function
. It is this decorator_function
that will actually do the decoration for the function say_something()
. The argument ‘passed to decorator’ is stored in a closure. This variable some_arg_passed
that contains this argument is still accessible from this closure context from inside the wrapper function.
Decorator Class
We can also have decorators that are implemented using classes. Often times, you will see these decorators when modifying a class interface. But these are filled with gotchas and need very careful considerations. Particularly when decorators that are implemented using classes are used to decorate class methods.
Some issues with Decorators
Losing context with Decorator
Since decorators are a wrapper they cause the wrapped function to lose some context.
def decorator_func(func): def wrapper(*args, **kwargs): func(*args, **kwargs) return wrapper @decorator_func def hello_world(): ''' docstring for hello world''' print('Hello world')
>>>hello_world.__name__ 'wrapper' >>>hello_world.__doc__ #...empty
As you can see that hello_world.__name__
points to ‘wrapper’ and the docstring associated with hello_world
is gone.
To fix this, use wraps library
from contextlib import wraps def decorator_func(func): @wraps(func) def wrapper(*args, **kwargs): func(*args, **kwargs) return wrapper @decorator_func def hello_world(): ''' docstring for hello world''' print('Hello world')
>>>hello_world.__name__ 'hello_world' >>>hello_world.__doc__ ' docstring for hello world'
Common uses for decorators
- logging
- timing
- validation
- rate_limiting
- mocking/patching
Context Decorators
A very cool trick in using the @contextmanager decorator for creating context manager is that the context manager can also be used as a decorator as well as a context manager inside of a with statement. This feature is available in core library as of Python 3.2. Note, this was called ContextDecorator before but the contextmanager ises ContextDecorator so the context managers it creates can be used as decorators as well as in with
statements.
class FeatureFlags: ''' example class which stores Feature Flags and their state. we only have on feature flag which is called SHOOW_BETA and this tells is to show beta version or not. Defalt, flag is True. I have two methods, one to show whether the flag is on or off and another to toggle tha flag value. ''' SHOW_BETA = "Show beta version of the home page" flags = { SHOW_BETA: True } @classmethod def is_on(cls, name): return cls.flags[name] @classmethod def toggle(cls, name, on): cls.flags[name] = on from contextlib import contextmanager @contextmanager def feature_flag(name, on=True): old_value = FeatureFlags.is_on(name) FeatureFlags.toggle(name, on) yield FeatureFlags.toggle(name, old_value) # as context manager with feature_flag(FeatureFlags.SHOW_BETA): assert '/beta' == get_homepage_url() # as decorator @feature_flag(FeatureFlags.SHOW_BETA, on=False) def get_profile_page(): beta_flag_on = FeatureFlags.is_on(FeatureFlags.SHOW_BETA) return 'beta.html' if beta_flag_on else 'profile.html' assert get_profile_page() != 'profile.html'
Notice the parentheses in the decorator. This tells you that this line @feature_flag(FeatureFlags.SHOW_BETA, on=False)
actually executes the code until the first yield, in effect setting the flag off. After the get_profile_page() is ended, it will run the set of lines after yield, in effect, resetting the flag to where it was. This is very good if we just need to test the behavior of get_profile_page().
One more example to finally make it clear:
from contextlib import contextmanager @contextmanager def example_manager(message): print(f'Before decorated function: {message}') yield print(f'after decorated function: {message}') with example_manager('testing') as e: print (f'Hello World') @example_manager('testing') def some_test_function(): print (f'Hello World from test_function') some_test_function()
This will generate the following output:
Before decorated function: testing Hello World after decorated function: testing Before decorated function: testing Hello World from test_function after decorated function: testing
Singleton Pattern
In Python, there are two ways to implement singleton pattern.
Using modules
file: Only_instantiated_once.py