My personal approach to the topic
In 2019 I was working for tribe29 GmbH. The company develops a well-known IT monitoring tool called CheckMK. The software uses Python plugins called checks to implement monitoring functionality. I already knew about functional programming and it’s benefits of potentially improving performance and memory consumption significantly. However Python does not support the functional programming paradigm natively. Python focusses on supporting the imperative and object-oriented programming paradigm. Nevertheless I wanter to learn more about how one can apply functional programming concepts in Python and read the book Functional Python Programming by Steven F. Lott.
Not much later I profited from what I’ve learned while reading the book. In 2019 I have setup the initial version of exploratory data analysis and data processing functionalities in the domain of mass spectrometry at the Plasmion GmbH. The functional programming concepts helped me a lot to write more reliable, memory-efficient and performant Python code.
A complementary note: At some point in time I’ve created a GitHub repo 30-seconds-of-functional-python-programming to experiment with different Python libraries in the context of functional programming (pyrsistent, toolz). You are invited to use the JupyterLab environment on Binder to experiment on your own.
About the content
The book is structured in a manner which promotes to read it from start to end. If you already know a bit about functional programming you can read single chapters in arbitrary order as well. The Table of Content is as follows:
- Chapter 1: Understanding Functional Programming
- Chapter 2: Introducing Essential Functional Concepts
- Chapter 3: Functions, Iterators, and Generators
- Chapter 4: Working with Collections
- Chapter 5: Higher-Order Functions
- Chapter 6: Recursions and Reductions
- Chapter 7: Additional Tuple Techniques
- Chapter 8: The Itertools Module
- Chapter 9: More Itertools Techniques
- Chapter 10: The Functools Module
- Chapter 11: Decorator Design Techniques
- Chapter 12: The Multiprocessing and Threading Modules
- Chapter 13: Conditional Expressions and the Operator Module
- Chapter 14: The PyMonad Library
- Chapter 15: A Functional Approach to Web Services
- Chapter 16: Optimizations and Improvements
This book is not for beginners. The topic is not the easiest one and the content is extensive (overall almost 400 pages). Consequently the chapters are not summarized in an easy to grasp manner. To be fair: This would not have been possible without bloating the summary sections to an unreasonable length. For every chapter I’ll highlight important buzz words in the original summarys bold.
Chapter 1: Introduction Functional Programming
[Functional Python Programming, p. 22]:
We’ve looked at programming paradigms with an eye toward distinguishing the functional paradigm from two common imperative paradigms. Our objective in this book is to explore the functional programming features of Python. We’ve noted that some parts of Python don’t allow purely functional programming; we’ll be using some hybrid techniques that meld the good features of succinct, expressive functional programming with some high-performance optimizations in Python.
Chapter 2: Introducing Some Functional Features
[Functional Python Programming, p. 36]
In this chapter, we’ve identified a number of features that characterize the functional programming paradigm. We started with first-class and higher-order functions. The idea is that a function can be an argument to a function or the result of a function. When functions become the object of additional programming, we can write some extremely flexible and generic algorithms.
The idea of immutable data is sometimes odd in an imperative and object-oriented programming language such as Python. When we start to focus on functional programming, however, we see a number of ways that state changes can be confusing or unhelpful. Using immutable objects can be a helpful simplification.
Python focuses on strict evaluation: all sub-expressions are evaluated from left-to-right through the statement. Python, however, does perform some non-strict evaluation. The or, and, and if-else logical operators are non-strict: all subexpressions are not necessarily evaluated. Similarly, a generator function is also non-strict. We can also call this eager vs. lazy. Python is generally eager but we can leverage generator functions to create lazy evaluation.
While functional programming relies on recursion instead of explicit loop state, Python imposes some limitations here. Because of the stack limitation and the lack of an optimizing compiler, we’re forced to manually optimize recursive functions. We’ll return to this topic in Chapter 6, Recursions and Reductions.
Although many functional languages have sophisticated type systems, we’ll rely on Python’s dynamic type resolution. In some cases, it means we’ll have to write manual coercion among types. It might also mean that we’ll have to create class definitions to handle very complex situations. For the most part, however, Python’s built-in rules will work very elegantly.
Chapter 3: Functions, Iterators, and Generators
[Functional Python Programming, p. 59]
In this chapter, we looked closely at writing pure functions: free of side effects. The bar is low here, since Python forces us to use the
globalstatement to write impure functions. We looked at generator functions and how we can use these as the backbone of functional programming. We also examined the built-in collection classes to show how they’re used in the functional paradigm. While the general ideal behind functional programming is to limit the use of stateful variables, the collection objects are generally stateful and, for many algorithms, also essential. Our goal is to be judicious in our use of Python’s non-functional features.
Chapter 4: Working with collections
[Functional Python Programming, p. 92]
In this chapter, we saw detailed ways to use a number of built-in reductions.
all()to do essential logic processing. These are tidy examples of reductions using a simple operator such as
We’ve also looked at numeric reductions such as
sum(). We’ve applied these functions to create some higher-order statistical processing. We’ll return to these reductions in Chapter 6, Recursions and Reductions.
We’ve also looked at some of the built-in mappings.
zip()function merges multiple sequences. This leads us to look at using this in the context of structuring and flattening more complex data structures. As we’ll see in examples in later chapters, nested data is helpful in some situations and flat data is helpful in others.
enumerate()function maps an iterable to a sequence of two tuples. Each two-tuple has the sequence number at index
and the original value at index
reversed()function iterates over the items in a sequence object with their original order reversed. Some algorithms are more efficient at producing results in one order, but we’d like to present these results in the opposite order.
Chapter 5: Higher-Order Functions
[Functional Python Programming, p. 126]
In this chapter, we have seen two reductions that are higher-order functions:
min(). We also looked at the two central higher-order functions,
filter(). We also looked at
We also looked at how to use a higher-order function to also transform the structure of data. We can perform several common transformations, including wrapping, unwrapping, flattening, and structure sequences of different kinds.
We looked at three ways to define our own higher-order functions, which are as follows:
defstatement. Similar to this is a lambda form that we assign to a variable.
- Defining a Callable class as a kind of function that emits composite functions.
- We can also use decorators to emit composite functions. We’ll return to this in Chapter 11, Decorator Design Techniques.
Chapter 6: Recursions and Reductions
In this chapter, we’ve looked at two significant functional programming topics. We’ve looked at recursions in some detail. Many functional programming language compilers will optimize a recursive function to transform a call in the tail of the function to a loop. In Python, we must do the tail-call optimization manually by using an explicit for loop instead of a purely function recursion.
We’ve also looked at reduction algorithms including
min()functions. We looked at the
collections.Counter()function and related
We’ve also looked at how parsing (and lexical scanning) are similar to reductions since they transform sequences of tokens (or sequences of characters) into higher-order collections with more complex properties. We’ve examined a design pattern that decomposes parsing into a lower level that tries to produce tuples of raw strings and a higher level that creates more useful application objects.
Chapter 7: Additional Tuple Techniques
[Functional Python Programming, p. 179]
In this chapter, we looked at different ways to use
NamedTupleobjects to implement more complex data structures. The essential features of a
NamedTupleare a good fit with functional design. They can be created with a creation function and accessed by position as well as name.
We looked at how to use immutable
NamedTupleinstead of stateful object definitions. The core technique for replacing state changes is to wrap objects in larger
We also looked at ways to handle multiple data types in Python. For most arithmetic operations, Python’s internal method dispatch locates proper implementations. To work with collections, however, we might want to handle iterators and sequences slightly differently.
Chapter 8: The Itertools Module
[Functional Python Programming, p. 205]
In this chapter, we’ve looked at a number of functions in the
itertoolsmodule. This library module provides a number of functions that help us to work with iterators in sophisticated ways.
We’ve looked at the infinite iterators; these repeat without terminating. These include the
repeat()functions. Since they don’t terminate, the consuming function must determine when to stop accepting values.
We’ve also looked at a number of finite iterators. Some of these are built-in and some of these are part of the
itertoolsmodule. These work with a source iterable, so they terminate when that iterable is exhausted. These functions include
map(). These functions allow us to replace possibly complex generator expressions with simpler-looking functions.
Additionally, we looked at the recipes from the documentation, which provide yet more functions we can study and copy for our own applications. The recipes list shows a wealth of common design patterns.
Chapter 9: More Itertools Techniques
[Functional Python Programming, p. 223]
In this chapter, we looked at a number of functions in the
itertoolsmodule. This library module provides a number of functions that help us work with iterators in sophisticated ways.
We looked at the
product()function that will compute all the possible combinations of the elements chosen from two or more collections. The
permutations()function gives us different ways to reorder a given set of values. The
combinations()function returns all the possible subsets of the original set.
We also looked at ways in which the
permuations()functions can be used naïvely to create extremely large result sets. This is an important cautionary note. A succinct and expressive algorithm can also involve a vast amount of computation. We must perform basic complexity analysis to be sure that the code will finish in a reasonable amount of time.
Chapter 10: The Functools Module
[Functional Python Programming, p. 242]
In this chapter, we’ve looked at a number of functions in the
functools()module. This library module provides a number of functions that help us create sophisticated functions and classes.
We’ve looked at the
@lru_cachefunction as a way to boost certain types of applications with frequent re-calculations of the same values. This decorator is of tremendous value for certain kinds of functions that take the
stringargument values. It can reduce processing by simply implementing memoization.
We looked at the
@total_orderingfunction as a decorator to help us build objects that support rich ordering comparisons. This is at the fringe of functional programming, but is very helpful when creating new kinds of numbers.
partial()function creates a new function with the partial application of argument values. As an alternative, we can build a lambda with similar features. The use case for this is ambiguous.
We also looked at the
reduce()function as a higher-order function. This generalizes reductions like the
sum()function. We’ll use this function in several examples in the later chapters. This fits logically with the
map()functions as an important higher-order function.
Chapter 11: Decorator Design Techniques
[Functional Python Programming, p. 262]
In this chapter, we’ve looked at two kinds of decorators: the simple decorator with no arguments and parameterized decorators. We’ve seen how decorators involve an indirect composition between functions: the decorator wraps a function (defined inside the decorator) around another function.
functools.wraps()decorator assures that our decorators will properly copy attributes from the function being wrapped. This should be a piece of every decorator we write.
Chapter 12: The Multiprocessing and Threading Modules
[Functional Python Programming, p. 290]
In this chapter, we’ve looked at two ways to support concurrent processing of multiple pieces of data:
multipcrocessingmodule: Specifically, the
Poolclass and the various kinds of mappings available to a pool of workers.
concurrent.futuresmodule: Specifically the
ThreadPoolExecutorclass. These classes also support a mapping that will distribute work among workers that are threads or processes.
We’ve also noted some alternatives that don’t seem to fit well with functional programming. There are numerous other features of the
multiprocessingmodule, but they’re not a good fit with functional design. Similarly, the
queuemodules can be used to build multithreaded applications, but the features aren’t a good fit with functional programs.
Chapter 13: Conditional Expressions and the Operator Module
[Functional Python Programming, p. 303]
In this chapter, we looked at alternatives to the
elsestatement sequence. Ideally, using a conditional expression allows some optimization to be done. Pragmatically, Python doesn’t optimize, so there’s little tangible benefit to the more exotic ways to handle conditions.
We also looked at how we can use the
operatormodule with higher order functions like
reduce(). Using operators can save us from having to create a number of small lambdas.
Chapter 14: The PyMonad Library
[Functional Python Programming, p. 323]
In this chapter, we looked at how we can use the
PyMonadlibrary to express some functional programming concepts directly in Python. The module shows many important functional programming techniques.
We looked at the idea of currying, a function that allows combinations of arguments to be applied to create new functions. Currying a function also allows us to use functional composition to create more complex functions from simpler pieces. We looked at functors that wrap simple data objects to make them into functions which can also be used with functional composition.
Monads are a way to impose a strict evaluation order when working with an optimizing compiler and lazy evaluation rules. In Python, we don’t have a good use case for monads, because Python is an imperative programming language under the hood. In some cases, imperative Python may be more expressive and succinct than a monad construction.
Chapter 15: A Functional Approach to Web Services
[Functional Python Programming, p. 349]
In this chapter, we looked at ways in which we can apply functional design to the problem of serving content with REST-based web services. We looked at the ways that the WSGI standard leads to somewhat functional overall applications. We also looked at how we can embed a more functional design into a WSGI context by extracting elements from the request for use by our application functions.
For simple services, the problem often decomposes down into three distinct operations: getting the data, searching or filtering, and then serializing the results. We tackled this with three functions:
serialize(). We wrapped these functions in a simple WSGI-compatible application to divorce the web services from the “real” processing around extracting and filtering the data.
We also looked at the way that web services functions can focus on the “happy path” and assume that all of the inputs are valid. If inputs are invalid, the ordinary Python exception handling will raise exceptions. The WSGI wrapper function will catch the errors and return appropriate status codes and error content.
We have not looked at more complex problems associated with uploading data or accepting data from forms to update a persistent data store. These are not significantly more complex than getting data and serializing the results. They are already solved in a better manner.
For simple queries and data sharing, a small web service application can be helpful. We can apply functional design patterns and assure that the website code is succinct and expressive. For more complex web applications, we should consider using a framework that handles the details properly.
Chapter 16: Optimizations and Improvements
[Functional Python Programming, p. 378]
In this chapter, we looked at three optimization techniques. The first technique involves finding the right algorithm and data structure. This has more impact on performance than any other single design or programming decision. Using the right algorithm can easily reduce runtimes from minutes to fractions of a second. Changing a poorly used sequence to a properly used mapping, for example, may reduce run time by a factor of 200.
We should generally optimize all of our recursions to be loops. This will be faster in Python and it won’t be stopped by the call stack limit that Python imposes. There are many examples of how recursions are flattened into loops in other chapters, primarily, Chapter 6, Recursions and Reductions. Additionally, we may be able to improve performance in two other ways. First, we can apply memoization to cache results. For numeric calculations, this can have a large impact; for collections, the impact may be less. Secondly, replacing large materialized data objects with iterables may also improve performance by reducing the amount of memory management required.
In the case study presented in this chapter, we looked at the advantage of using Python for exploratory data analysis — the initial data acquisition including a little bit of parsing and filtering. In some cases, a significant amount of effort is required to normalize data from various sources. This is a task at which Python excels.
The calculation of a X² value involved three
sum()functions: two intermediate generator expressions, and a final generator expression to create a dictionary with expected values. A final
sum()function created the statistic. In under a dozen expressions, we created a sophisticated analysis of data that will help us accept or reject the null hypothesis.
We also evaluated some complex statistical functions: the incomplete and the complete
gammafunction. The incomplete
gammafunction involves a potentially infinite series; we truncated this and summed the values. The complete
gammafunction has some potential complexity, but it doesn’t happen to apply in our case.
Using a functional approach, we can write succinct and expressive programs that accomplish a great deal of processing. Python isn’t a properly functional programming language. For example, we’re required to use some imperative programming techniques. This limitation forces away from purely functional recursions. We gain some performance advantage, since we’re forced to optimize tail recursions into explicit loops.
We also saw numerous advantages of adopting Python’s hybrid style of functional programming. In particular, the use of Python’s higher order functions and generator expressions give us a number of ways to write high performance programs that are often quite clear and simple.
- [Functional Python Programming]: Lott, Steven F., ed. (2018) Functional Python Programming. 2nd ed. Birmingham: Packt Publishing
Happy reading :)