One unfortunate difficulty when working with App Engine is managing your local dependencies. You don’t have access to your Python environment so all libraries you wish to use must be vendored with your installation. That is, you need to copy all of your library code into a local folder to ship along with your app. ...
View all articles in the MapReduce API Series. The MapReduce library supports a number of default output writers. You can also write your own that implements the output writer interface. This article examines how to write a custom output writer that pushes data from the App Engine datastore to an elasticsearch cluster. A similar pattern can be followed to push the output from your MapReduce job to any number of places. ...
A common task in bash programming is to manipulate portions of a string and return the result. bash provides rich support for these manipulations via string operators. The syntax is not always intuitive so I wanted to use this blog post to serve as a permanent reminder of the operators. The string operators are signified with the ${} notation. The operations can be grouped in to a few classes. Each heading in this article describes a class of operation. ...
View all articles in the MapReduce API Series. One of the great things about the MapReduce library is the abilitiy to write a cutom InputReader to process data from any data source. In this post we will explore how to write an InputReader the leases tasks from an AppEngine pull queue by implementing the InputReader interface. ...
Installing the MySQL-Python package requires a few steps. In an effort to aid future Internet travellers, this post will document how to install the MySQL-Python package on OS X Yosemite. First, install MariaDB, the drop-in replacement for MySQL. I chose MacPorts for this task, though Homebrew would work just fine. Second, update your PATH to include the mariadb executables. Third, install the Python MySQL connector. sudo port install mariadb PATH=/opt/local/lib/mariadb/bin:$PATH pip install MySQL-Python That’s it! You should be able to import MySQLdb in your Python code and interact with your MariaDB database.
A recurring issue when working with Compute Engine is that newly created Instances have only 10GB of free space available. To take advantage of the full disk size you need to manually partition and resize it. This article shows one method of accomplishing this task. ...
Overview The Google Compute Engine core concepts video provides a great overview of the technology. Please take a moment to watch it. As the video shows, Compute Engine is defined by Resources, and each of these Resources is available through the Compute Engine API. You can access this API through the Google API Explorer or by installing the gcloud sdk. ...
fdisk is a wonderful little utility for managing partitions. I recently had to script a series of fdisk commands for resizing a partition and needed to extract the start sector from the existing disk to do so. I ended up using this combination of grep and awk to do the job. ...
Google Compute Engine allows you to make custom images from a running virtual machine. The documentation provides a sufficient example but is a little bit scattered. This article collects and presents all the steps necessary to create your own Compute Engine images that you can use as a base for virtual machines. ...
The Google Prediction API offers the power of Google’s machine learning algorithms over a RESTful API interface. The machine learning algorithms themselves are a complete black box. As a user you upload the training data and, once it has been analyzed, start classifying new observations based on the analysis of the training data. I recently spent some time investigating how to use the API to determine the sentiment of a tweet. This article collects my thoughts on the experience and a few recommendations for future work. ...