View all articles in the Pipeline API Series. This article will serve as a reminder of the Pipeline UI as much for the writer as for the reader. The Pipeline UI requires the MapMeduce library to be installed. If you are not familiar with MapReduce please refer to the MapReduce API Series of articles. Once MapReduce is installed you will need to add a few indices to index.yaml to properly query for pipeline records for display in the UI....
View all articles in the Pipeline API Series. This article will cover fully asynchronous pipelines. The term ‘asynchronous’ is misleading here — all piplines are asynchronous in the sense that yielding a pipeline is a non-blocking operation. An asynchronous refers to a pipeline that remains in a RUN state until outside action is taken, for example, a button is clicked or a task is executed. Marking a pipeline as an asynchronous pipeline is as simple as setting the async class property to True....
View all articles in the Pipeline API Series. We’ve learned how to execute and chain together pipelines, now let’s take a look at how pipelines execute under the hood. If necessary, you can refer to the source code of the pipelines project to clarify any details. The Pipeline Data Model Let’s start with the pipeline data model. Note that each Kind defined by the pipelines API is prefixed by _AE_Pipeline, making it easy to view individual pipeline details by viewing the datastore entity....
View all articles in the Pipeline API Series. Last time, we studied how to connect two pipelines together. In this post, we expand on this topic, exploring how to fan-out to do multiple tasks in parallel, fan-in to combine multiple tasks into one, and how to do sequential work. ...
View all articles in the Pipeline API Series. Last time, we discussed basic pipeline instantiation and execution. This time, we will cover sequential pipelines, answering the question “How do I connect the output of one pipeline with the input of another pipeline”? ...
View all articles in the Pipeline API Series. The Pipelines API is a general purpose workflow engine for App Engine applications. With the Pipelines API we can connect together complex workflows into a coherent run time backed by the Datastore. This article provides a basic overview of the Pipelines API and how it can be used for abritrary computational workflows. In the most basic sense a Pipeline is an object that takes input, performs some logic or computation on that input, and produces output....