What’s a durabledict?
Good question. Durabledict is a Python implementation of a persistent dictionary. The dictionary values are cached locally and sync with the datastore whenever a value in the datastore changes.
Disqus provides concrete implementations for Redis, Django, ZooKeeper and in memory. This blog post details an implementation using the App Engine datastore and memcache.
Creating your own durabledict
By following the guide the durabledict
README we can create our own
implementation. We need to subclass
durabledict.base.DurableDict and implement
the following interface methods. Strictly speaking,
not have to be implemented but doing so makes your durabledict behave like a
base dict in all cases.
persist(key, value) - Persist value at key to your data store.
depersist(key) - Delete the value at key from your data store.
durables() - Return a key=val dict of all keys in your data store.
last_updated() - A comparable value of when the data in your data store was last updated.
_pop(key, default=None) - If key is in the dictionary, remove it and return its value, else return default. If default is not given and key is not in the dictionary, a KeyError is raised.
_setdefault(key, default=None) - If key is in the dictionary, return its value. If not, insert key with a value of default and return default. default defaults to None.
Let’s implement these one-by-one.
Persisting a value to the datastore is a relatively simple operation. If the key
already exists we update it’s value. If the key does not already exist we create
it. To aid with this operation we create a
get_or_create method that will
return an existing entity if one exists or create a new entity if one does not
The last line of this function updates the last time this durabledict was
changed. This is used for caching. We create the
touch_last_updated functions now.
We now have the building blocks to create our initial durabledict. Within the
__init__ method we set a manager and cache instance. The manager is
responsible for ndb datastore operations to decouple the ndb interface from the
durabledict implementation. We decouple our caching method in a similar fashion.
We also set the initial value of the cache whenever we create a new instance of
Depersist implies deleting a key from the dictionary (and datastore). Here we
assume a helper method
delete that, given an ndb model and a string
representing it’s key deletes the model. Since the data has changed we also
update the last touched value to force a cache invalidation and data refresh.
durables() returns the entire dictionary. Since we are all matching entities
from the datastore it is important to keep your dictionary relatively small –
as the dictionary grows in size, resyncing it’s state with the datastore will
get more and more expensive. This function assumes a
get_all method that will
return all instances of a model.
_setdefault() overrides the dictionary built-in
setdefault which allows you
to insert a key into the dictionary, creating the key with the default value if
it does not exist and returning the existing value if it does exist.
For example, the following sequence of code creates a key for
y, which does not
exist, and returns the existing value for
We can implement
_setdefault using the
get_or_create helper method, updating
the cache if we have changed the dictionary.
pop returns the value for a key and deletes the key. This is fairly straight
forward given a
delete helper method.
The previous discussion uses a few helper methods that we haven’t defined yet. Each of these methods takes an arbitrary ndb model and performs an operation on it.
The last item of note is the use of a parent for each DatastoreDict. This common
ancestor forces strong read consistency for the
get_all method, allowing us to
update a dictionary and have a consistent view of the data on subsequent reads.
We use an additional model to provide the strong read consistency.