Struggling to structure code in larger programs? Great resources a beginner might not find so easily
March 2021 ∙ four minute read ∙
So, you're an advanced beginner – you've learned your way past Python basics and can solve real problems.
Maybe you're about to embark on your first larger project, but feel at a loss about where to start from, and how to structure it, and don't want to make any mistakes.
Or you're midway through a large project already (not even your first), but don't know how to make the modules work together; what started as a pythonic script ended up as not-so-beautiful modules and packages, and it all gets messy, so quickly.
Maybe you feel most tutorials and blog posts go straight to one solution to a simple, made-up problem, without an in-depth look at alternatives.
You've tried reading about architecture and design patterns, but they seem too abstract, and you can't see how they apply to your code.
You may have heard that you have to develop an eye by reading and writing a lot of code. That there are so many Python projects on GitHub – "just pick one you like and see how they did it".
But they're too large and you don't know where to start, and even if you did, from the code itself it's not obvious what the design philosophy was and what choices they considered before settling on an implementation.
If any of this sounds familiar, here's a few resources that might help.
The Architecture of Open Source Applications #
Remember reading code and the design decisions not being obvious?
The Architecture of Open Source Applications is a whole book about exactly that (two actually, since there are two volumes) – and it's free!
From their website:
Architects look at thousands of buildings during their training, and study critiques of those buildings written by masters. In contrast, most software developers only ever get to know a handful of large programs well—usually programs they wrote themselves—and never study the great programs of history. As a result, they repeat one another's mistakes rather than building on one another's successes.
Our goal is to change that. In these two books, the authors of four dozen open source applications explain how their software is structured, and why. What are each program's major components? How do they interact? And what did their builders learn during their development? In answering these questions, the contributors to these books provide unique insights into how they think.
About a quarter of the chapters are about Python projects; you will encounter familiar names like SQLAlchemy, PyPy, Twisted, matplotlib, and Mercurial.
The discussions can be a bit high level, since they're all mature and relatively large projects (and there's only so much you can fit in a book chapter), although some of them are more in-depth than others.
You should have a look at the non-Python chapters as well – there's a lot of wisdom in there. Also, at least part of the success of my last job interview was due to having read the nginx chapter :)
500 Lines or Less #
500 Lines or Less is another book from the AOSA series.
It aims to address the initial books being too high-level by looking at purpose-made small projects (again, focusing on design decisions). From the introduction:
The first three volumes in the series were about big problems that big programs have to solve. For an engineer who is early in their career, it may be a challenge to understand and build upon programs that are much bigger than a few thousand lines of code, so, while big problems can be interesting to read about, they can also be challenging to learn from.
And there's more better news: over half of the chapters are in Python.
If you ever wondered how templating engines work, give the A Template Engine chapter a read. It will provide a gentle introduction to the problem and examine a relatively simple solution (but not that simple that it's not useful anymore – that's actually the template engine coverage.py uses to produce its HTML reports).
Jinja #
If the 500 Lines or Less template engine chapter got your interest, you may want to take a look at Jinja.
Jinja is one of the most used template engines for Python, and if you used Flask, you've probably already used it. It's been refined for over 14 years, and like all the Pallets projects, has a great API.
I recommend you dive into the source starting from the Environment class, after reading the Basics and High Level API sections of the API documentation first.
There are two talks given by Jinja's main author, Armin Ronacher, to lead you on your journey:
- Code Generation in Python: Dismantling Jinja (recording) walks through the design of Jinja's compiler infrastructure, why it works the way it works, and how it ended up where it is after many different iterations.
- Let's Talk About Templates (recording) compares Jinja and Django's templates, looking at how their different histories and constraints led to vastly different internal designs.
You can find all of Armin's talks here. Letters from the Battlefield and Good API Design are especially nice higher-level design lessons, with examples from the Flask and Jinja world.
That's it for now.
Learned something new today? Share this with others, it really helps!
This is part of a series: