App Engine

I’ve been thinking about the transition of App Engine to Python 3 and have come to the conclusion that it will never happen — App Engine will eventually be deprecated in favour of Managed VMs. Let’s break this apart to see why this is. First, consider the effort required by Google to develop App Engine. The Python runtime environment was modified to enforce the sandbox of the App Engine environment. To provide a Python 3 environment for App Engine as we know it, the Python 3 runtime would need to be modified with the same restrictions. Even imagining that this would happen for Python 3.4, the effort to upgrade to Python 3.5 would require additional effort by Google to modify the runtime. ...

tldr; DatastoreDict. What’s a durabledict? Good question. Durabledict is a Python implementation of a persistent dictionary. The dictionary values are cached locally and sync with the datastore whenever a value in the datastore changes. Disqus provides concrete implementations for Redis, Django, ZooKeeper and in memory. This blog post details an implementation using the App Engine datastore and memcache. Creating your own durabledict By following the guide the durabledict README we can create our own implementation. We need to subclass durabledict.base.DurableDict and implement the following interface methods. Strictly speaking, _pop and _setdefault do not have to be implemented but doing so makes your durabledict behave like a base dict in all cases. ...

At Vendasta the App Engine Datastore serves as the single point of truth for most operational data and the majority of interactions are against this single point of truth. However, a piece of required functionality in many of our products is to provide a searchable view of the data in the App Engine Datastore. Search is difficult using the Datastore and so we have moved to using the Search API as a managed solution for searching datastore entities. In this use case, every edit to an entity in the Datastore is reflected as a change to a Search Document. This article details an architecture for keeping Datastore entities and Search Documents in sync throughout failure and race conditions. ...

I’ve been working on a simple App Engine application that offers upload and download functionality to and from Google Cloud Storage. When it came time to actually download the content I needed to write a webapp2 RequestHandler that will retrieve the file from Cloud Storage and return it to the client. ...

From a DevOps perspective having a historical record of application logs can aid immensely in tracking down bugs, responding to customer questions, or finding out when and why that critical piece of data was updated to the wrong value. One of the biggest grievances with the built-in log handling of Google App Engine is that historical logs are only available for the previous three days. We wanted to do a little bit better and have logs available for a 30 day time period. This article outlines a method we’ve developed for pushing App Engine logs to an elasticsearch cluster. ...

One unfortunate difficulty when working with App Engine is managing your local dependencies. You don’t have access to your Python environment so all libraries you wish to use must be vendored with your installation. That is, you need to copy all of your library code into a local folder to ship along with your app. ...

View all articles in the MapReduce API Series. The MapReduce library supports a number of default output writers. You can also write your own that implements the output writer interface. This article examines how to write a custom output writer that pushes data from the App Engine datastore to an elasticsearch cluster. A similar pattern can be followed to push the output from your MapReduce job to any number of places. ...

View all articles in the MapReduce API Series. One of the great things about the MapReduce library is the abilitiy to write a cutom InputReader to process data from any data source. In this post we will explore how to write an InputReader the leases tasks from an AppEngine pull queue by implementing the InputReader interface. ...

At VendAsta we have a few APIs that are backed by search documents built using the App Engine Search API. These APIs are queried using a search string entered in a text box. One way to improve the user experience of this text box is to offer the user suggestions of popular searches to use as their query. This article describes how to achieve this. ...

Asynchronous functions can provide a boon to application performance by allowing time consuming functions to operate in parallel and without blocking the main execution thread. This article explains how to use the Tasklet API to compose and execute asynchronous functions in Google App Engine. ...