App Engine

I’ve been thinking about the transition of App Engine to Python 3 and have come to the conclusion that it will never happen — App Engine will eventually be deprecated in favour of Managed VMs. Let’s break this apart to see why this is. First, consider the effort required by Google to develop App Engine. The Python runtime environment was modified to enforce the sandbox of the App Engine environment. To provide a Python 3 environment for App Engine as we know it, the Python 3 runtime would need to be modified with the same restrictions....

View all articles in the Pipeline API Series. This article will serve as a reminder of the Pipeline UI as much for the writer as for the reader. The Pipeline UI requires the MapMeduce library to be installed. If you are not familiar with MapReduce please refer to the MapReduce API Series of articles. Once MapReduce is installed you will need to add a few indices to index.yaml to properly query for pipeline records for display in the UI....

View all articles in the Pipeline API Series. This article will cover fully asynchronous pipelines. The term ‘asynchronous’ is misleading here — all piplines are asynchronous in the sense that yielding a pipeline is a non-blocking operation. An asynchronous refers to a pipeline that remains in a RUN state until outside action is taken, for example, a button is clicked or a task is executed. Marking a pipeline as an asynchronous pipeline is as simple as setting the async class property to True....

View all articles in the Pipeline API Series. We’ve learned how to execute and chain together pipelines, now let’s take a look at how pipelines execute under the hood. If necessary, you can refer to the source code of the pipelines project to clarify any details. The Pipeline Data Model Let’s start with the pipeline data model. Note that each Kind defined by the pipelines API is prefixed by _AE_Pipeline, making it easy to view individual pipeline details by viewing the datastore entity....

View all articles in the Pipeline API Series. Last time, we studied how to connect two pipelines together. In this post, we expand on this topic, exploring how to fan-out to do multiple tasks in parallel, fan-in to combine multiple tasks into one, and how to do sequential work. ...

View all articles in the Pipeline API Series. Last time, we discussed basic pipeline instantiation and execution. This time, we will cover sequential pipelines, answering the question “How do I connect the output of one pipeline with the input of another pipeline”? ...

View all articles in the Pipeline API Series. The Pipelines API is a general purpose workflow engine for App Engine applications. With the Pipelines API we can connect together complex workflows into a coherent run time backed by the Datastore. This article provides a basic overview of the Pipelines API and how it can be used for abritrary computational workflows. In the most basic sense a Pipeline is an object that takes input, performs some logic or computation on that input, and produces output....

tldr; DatastoreDict. What’s a durabledict? Good question. Durabledict is a Python implementation of a persistent dictionary. The dictionary values are cached locally and sync with the datastore whenever a value in the datastore changes. Disqus provides concrete implementations for Redis, Django, ZooKeeper and in memory. This blog post details an implementation using the App Engine datastore and memcache. Creating your own durabledict By following the guide the durabledict README we can create our own implementation....

At Vendasta the App Engine Datastore serves as the single point of truth for most operational data and the majority of interactions are against this single point of truth. However, a piece of required functionality in many of our products is to provide a searchable view of the data in the App Engine Datastore. Search is difficult using the Datastore and so we have moved to using the Search API as a managed solution for searching datastore entities....

I’ve been working on a simple App Engine application that offers upload and download functionality to and from Google Cloud Storage. When it came time to actually download the content I needed to write a webapp2 RequestHandler that will retrieve the file from Cloud Storage and return it to the client. ...