So, you're using a Python API client to get a bunch of data, and you need to cache results and retry on errors.

You could write a wrapper class, but that's just one more thing to test and maintain.¹

Thankfully, this is Python, and more often than not, there must be a better way.

• But I don't own the client code
• But decorating methods keeps instances around for too long
• But I need to cache a lot of methods
• But I don't instantiate the client
• But the proxy breaks completion
• But the proxy fails isinstance() checks
• But I only need to cache a few methods

Note

In this article, I will focus on caching using functools.lru_cache(). Retrying with a library like tenacity would work more or less the same.

But I don't own the client code #

We'll assume a client that looks like this (but keep in mind it's not our code – we can only use it, not change it):

class Client:

    def one(self, arg):
        print(f"Client.one({arg!r})")
        return arg.upper()

    def two(self, arg):
        print(f"Client.two({arg!r})")
        return arg.title()

If Client were our code, we could just slap @functools.lru_cache on the methods.

As is, we can subclass it, override each method, and decorate the overrides:

class SubclassOverride(Client):

    @functools.lru_cache(maxsize=1000)
    def one(self, arg):
        return super().one(arg)

    @functools.lru_cache(maxsize=1000)
    def two(self, arg):
        return super().two(arg)

Of course, this doesn't really help with maintainability, but it does give us an excuse to look at a potential issue with lru_cache().

But decorating methods keeps instances around for too long #

The lru_cache() docs say,

If a method is cached, the self instance argument is included in the cache. See How do I cache method calls?

... in turn:

The lru_cache approach [...] creates a reference to the instance [...] The disadvantage is that instances are kept alive until they age out of the cache or until the cache is cleared.

Depending on how often Client is instantiated, this might count as a memory and/or resource leak. For a more detailed explanation, see this article.

One solution is to decorate the bound method, after the object is created:

class SubclassShadow(Client):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.one = functools.lru_cache(maxsize=1000)(self.one)
        self.two = functools.lru_cache(maxsize=1000)(self.two)

This way, the instance is not in the cache keys, and we have one cache per instance.

There's still a reference cycle between the cache and the instance (via the bound method), but the garbage collector will take care of that – normally, all three go away at the same time. Before, the cache had the same lifetime as the unbound method (that is, the same lifetime as the class).

Important

You might be tempted to say, "but if I call one(arg) on two different instances, the underlying method will be called twice".

True, but this is the opposite of a problem, it's likely the correct behavior. Because each instance has different state, the results of the calls may be different (it helps to think of instance attributes as implicit arguments passed to each method); for example, clients instantiated with different users should not return the same thing.

But I need to cache a lot of methods #

OK, the above is a bit less verbose, but we still have to decorate methods one by one.

Thankfully, we can do it on the fly:

class SubclassDynamic(Client):

    def __getattribute__(self, name):
        # conveniently, also prevents infinite recursion
        if name.startswith('_'):
            return getattr(super(), name)
        try:
            return self.__dict__[name]
        except KeyError:
            pass
        value = getattr(super(), name)
        if not callable(value):
            return value
        cached = functools.lru_cache(maxsize=1000)(value)
        self.__dict__[name] = cached
        return cached

__getattribute__() allows us to intercept all attribute accesses (including methods). When a method is requested, we get it from the parent, decorate it, and store it in the instance dictionary; the second time around, we return the already decorated method.

We don't decorate methods whose names start with an underscore, including any __magic__ or _private ones; additional filtering logic (e.g. only cache methods starting with get_) would go here as well.

Note we can't use __getattr__(), because it's only called if the attribute is not found through the normal mechanism – and looking up the class tree is part of the normal mechanism.

We could just decorate all the methods upfront, in __init__(), but that might not work if the parent does magic stuff without implementing __dir__() properly.

But I don't instantiate the client #

Sometimes, you're not the one instantiating the client; it may be created by a callable factory that doesn't allow passing in a different class,² or you might just get it from a framework.

No worries, subclassing is not required for Python meta­programming – instead, we can wrap the instance in an object proxy (also known as dynamic wrapper).

The code looks pretty much the same, but instead of delegating to the parent, we delegate to the wrapped instance:

class Proxy:

    def __init__(self, wrapped):
        self.__wrapped__ = wrapped

    def __getattr__(self, name):
        value = getattr(self.__wrapped__, name)
        if name.startswith('_'):
            return value
        if not callable(value):
            return value
        cached = functools.lru_cache(maxsize=1000)(value)
        setattr(self, name, cached)
        return cached

Because the proxy is a separate object, now we can use __getattr__(), and we don't have to check the instance dictionary explicitly – on the second call, the decorated method is already there, so __getattr__() isn't called anymore.

But the proxy breaks completion #

Oops, we're guilty of the not implementing __dir__() mentioned before.

Among others, this breaks completion:

>>> client = Proxy(Client())
>>> 'one' in dir(client)
False
>>> 'one' in dir(client.__wrapped__)
True
>>> client.<TAB>
           ... crickets ...

... but it's pretty easy to fix:

    def __dir__(self):
        return dir(self.__wrapped__)

>>> client = Proxy(Client())
>>> 'one' in dir(client)
True

With this, we have a solution that covers almost everything I've met in practice.

But the proxy fails isinstance() checks #

Turns out, writing a perfect proxy is pretty difficult.

For example, although not recommended, code may need to do isinstance() checks:

>>> isinstance(client, Client)
False

We could fix this by implementing __instancecheck__(), but I thought we were past wrapping things one by one. wrapt gives us an almost-perfect proxy:

class WraptProxy(wrapt.ObjectProxy):

    def __init__(self, wrapped):
        super().__init__(wrapped)
        self._self_cached_ones = {}

    def __getattr__(self, name):
        try:
            return self._self_cached_ones[name]
        except KeyError:
            pass
        value = super().__getattr__(name)
        if name.startswith('_'):
            return value
        if not callable(value):
            return value
        cached = functools.lru_cache(maxsize=1000)(value)
        self._self_cached_ones[name] = cached
        return cached

It's not as elegant as ours, but check this out:

>>> client = WraptProxy(Client())
>>> client
<WraptProxy at 0x106c2f5c0 for Client at 0x106c2ec10>
>>> print(client)
<__main__.Client object at 0x10d71ad90>
>>> client.__class__
<class '__main__.Client'>
>>> isinstance(client, Client)
True
>>> 'one' in dir(client)
True

But I only need to cache a few methods #

OK, all that seems a bit excessive if you only need to cache a couple of methods.

If we're the only ones using the client, we could go back to decorating the bound methods ...except, didn't we say the instance already exists?

So what? It's not like you can only set attributes in __init__():

client = Client()

def cache_method_in_place(self, name, **kwargs):
    one = getattr(self, name)
    cached = functools.lru_cache(**kwargs)(one)
    setattr(self, name, cached)

cache_method_in_place(client, 'one', maxsize=1000)
cache_method_in_place(client, 'two', maxsize=1000)

Unexpected? A bit. Unholy? Maybe, depending where you're coming from. But actually, no, not really – after all, it still walks and quacks like a Client, and we've only changed our instance.

In the end, that's one of the reasons people like Python – you're free to bring in the big guns of meta­programming if you need to, but sometimes a tiny bit of monkey patching will do just as well.³

---

Anyway, that's it for now.

Learned something new today? Share this with others, it really helps! PyCoder's Weekly HN Reddit linkedin Twitter

Want to know when new articles come out?

Drop your email in the box below and I'll send new stuff straight to your inbox!

1. I'm mainly dissing static (1:1) wrappers here. As a higher level of abstraction, wrappers can be very useful, both maintainability and readability-wise; for a great example of this, check out Raymond Hettinger's Beyond PEP 8 – Best practices for beautiful intelligible code talk. ^[return]

2. For example, sqlite3.connect() does, via the factory parameter; reader.​make_reader() doesn't (at least, not yet). ^[return]

3. Just like cats and salami, a very small amount of monkey patching is probably fine, though you definitely shouldn't make it a staple of your diet. ^[return]