Querying App Engine Logs with Elasticsearch

From a DevOps perspective having a historical record of application logs can aid immensely in tracking down bugs, responding to customer questions, or finding out when and why that critical piece of data was updated to the wrong value. One of the biggest grievances with the built-in log handling of Google App Engine is that historical logs are only available for the previous three days. We wanted to do a little bit better and have logs available for a 30 day time period....

January 23, 2015 · 2 min · Kevin Sookocheff

Managing App Engine Dependencies Using pip

One unfortunate difficulty when working with App Engine is managing your local dependencies. You don’t have access to your Python environment so all libraries you wish to use must be vendored with your installation. That is, you need to copy all of your library code into a local folder to ship along with your app. ...

December 30, 2014 · 2 min · Kevin Sookocheff

App Engine MapReduce API - Part 7: Writing a Custom Output Writer

View all articles in the MapReduce API Series. The MapReduce library supports a number of default output writers. You can also write your own that implements the output writer interface. This article examines how to write a custom output writer that pushes data from the App Engine datastore to an elasticsearch cluster. A similar pattern can be followed to push the output from your MapReduce job to any number of places. ...

December 22, 2014 · 4 min · Kevin Sookocheff

App Engine MapReduce API - Part 6: Writing a Custom Input Reader

View all articles in the MapReduce API Series. One of the great things about the MapReduce library is the abilitiy to write a cutom InputReader to process data from any data source. In this post we will explore how to write an InputReader the leases tasks from an AppEngine pull queue by implementing the InputReader interface. ...

December 4, 2014 · 7 min · Kevin Sookocheff

Suggested Searches with Google App Engine

At VendAsta we have a few APIs that are backed by search documents built using the App Engine Search API. These APIs are queried using a search string entered in a text box. One way to improve the user experience of this text box is to offer the user suggestions of popular searches to use as their query. This article describes how to achieve this. ...

October 6, 2014 · 3 min · Kevin Sookocheff

Composing Asynchronous Functions With Tasklets

Asynchronous functions can provide a boon to application performance by allowing time consuming functions to operate in parallel and without blocking the main execution thread. This article explains how to use the Tasklet API to compose and execute asynchronous functions in Google App Engine. ...

September 27, 2014 · 4 min · Kevin Sookocheff

Converting an ndb model to a BigQuery schema

I have been working on the problem of recording changes to an ndb model. One way to accomplish this is to stream data changes to a BigQuery table corresponding to the ndb model. It would be great to do this in a generic way which gives us the problem of generating a BigQuery table given an ndb model. This article will describe one solution to this problem. ...

August 14, 2014 · 3 min · Kevin Sookocheff

Restoring an App Engine backup into a Big Query table

An unfortunate DevOps task for any team running App Engine is restoring data from backups. One way to do this is by accessing the Google Cloud Storage URL for a given App Engine backup and importing that backup into BigQuery. This article will show you to get the Cloud Storage URL for an App Engine backup and manually perform that import. ...

August 4, 2014 · 2 min · Kevin Sookocheff

Bypassing ndb hooks with the RawDatastoreInputReader

When doing a MapReduce operation there are times when you want to edit a set of entities without triggering the post or pre put hooks associated with those entities. On such ocassions using the raw datastore entity allows you to process the data without unwanted side effects. This article will show how to use the RawDatastoreInputReader to process datastore entities. ...

July 29, 2014 · 2 min · Kevin Sookocheff

Generating a C# client for an App Engine Cloud Endpoints API

The Cloud Endpoints API comes packaged with endpointscfg.py to generate client libraries in JavaScript, Objective-C (for iOS) and Java (for Android). You can also generate a few additional client libraries using the Google APIs client generator. This article will show you how to use the generator to create a C# client library. ...

July 22, 2014 · 2 min · Kevin Sookocheff