I’ve recently been working with Java via the Google Cloud Dataflow SDK. One problem I’ve had is working with the BigQuery Java Client. It was never entirely clear how to create a repeated record. This article explains how it works and how you can accomplish the same thing. First, you need to create a new TableRow. For this example, let’s assume we are logging events using a guid and a timestamp. ...
I haven’t been able to find great documentation on creating a BigQuery TableSchema using the Java Client Library. This blog post hopes to rectify that :). You can use the BigQuery sample code for an idea of how to create a client connection to BigQuery. Assuming you have the connection set up you can start by creating a new TableSchema. The TableSchema provides a method for setting the list of fields that make up the columns of your BigQuery Table. Those columns are defined as an Array of TableFieldSchema objects. ...
I have been working on the problem of recording changes to an ndb model. One way to accomplish this is to stream data changes to a BigQuery table corresponding to the ndb model. It would be great to do this in a generic way which gives us the problem of generating a BigQuery table given an ndb model. This article will describe one solution to this problem. ...
An unfortunate DevOps task for any team running App Engine is restoring data from backups. One way to do this is by accessing the Google Cloud Storage URL for a given App Engine backup and importing that backup into BigQuery. This article will show you to get the Cloud Storage URL for an App Engine backup and manually perform that import. ...