Durabledict for App Engine

tldr; DatastoreDict. What’s a durabledict? Good question. Durabledict is a Python implementation of a persistent dictionary. The dictionary values are cached locally and sync with the datastore whenever a value in the datastore changes. Disqus provides concrete implementations for Redis, Django, ZooKeeper and in memory. This blog post details an implementation using the App Engine datastore and memcache. Creating your own durabledict By following the guide the durabledict README we can create our own implementation. [Read More]

Composing Asynchronous Functions With Tasklets

Asynchronous functions can provide a boon to application performance by allowing time consuming functions to operate in parallel and without blocking the main execution thread. This article explains how to use the Tasklet API to compose and execute asynchronous functions in Google App Engine.

[Read More]

Converting an ndb model to a BigQuery schema

I have been working on the problem of recording changes to an ndb model. One way to accomplish this is to stream data changes to a BigQuery table corresponding to the ndb model. It would be great to do this in a generic way which gives us the problem of generating a BigQuery table given an ndb model. This article will describe one solution to this problem.

[Read More]

Bypassing ndb hooks with the RawDatastoreInputReader

When doing a MapReduce operation there are times when you want to edit a set of entities without triggering the post or pre put hooks associated with those entities. On such ocassions using the raw datastore entity allows you to process the data without unwanted side effects. This article will show how to use the RawDatastoreInputReader to process datastore entities.

[Read More]

How to bypass the auto_now property option during an ndb put

In App Engine the auto_now option sets a property to the current date/time whenever the entity is created or updated. This is a great feature for tracking the time when an entity was last updated. However, sometimes you may want to put an entity without updating an auto_now timestamp. This article will show you how.

[Read More]