So you want to use Thrift?

You’ve come here because you want to use Apache Thrift and you don’t know where to start. Good. You’re in the right spot.

Throughout this document we will develop a simple service that communicates using Thrift. This will introduce you to the workflow for generating client and server code using Thrift and how to Thrift works to separate your application’s business logic from it’s transport methods.

Introductions aside, let’s start writing our first Thrift service.

The Task Manager Service

Alice is the boss. She keeps a running list of tasks that need to be done. Being the boss, she doesn’t actually do any tasks herself — she delegates that work to her employee, Bob. When the company was small, she just told Bob face-to-face the tasks to do, but as the company grew she decided that developing an internal task management system would be an ideal use of company resources. Thus, the “Task Manager” was born. Micro-services and messaging being all the rage, this new system must use be deployed as a service using Thrift.

Defining the Task Manager API Using Thrift

One of the first steps when creating a service is defining your API. The Task Manager’s API, will expose a single endpoint for adding tasks. In effect, this operation transfers a task from Alice’s todo list to Bob’s. As a REST end-point the API would look something like the following.

POST /api/v1/tasks/create

with payload:

{
    "taskId": "TK-2190809",
    "description": "Close Jira ticket OPS-12345"
}

Upon receipt of a task, Bob responds with an acknowledgement containing the taskId received. The response would look something like this:

200 SUCCESS

with payload:

{
    "taskId": "TK-2190809"
}

Although REST is a great way to expose a service, we have a hard requirement to use Thrift.

Thrift Basics

Thrift provides a language-agnostic way of defining a services API using an interface definition language or IDL. Thrift was originally developed at Facebook as a software library and set of code-generation tools meant to “expedite development and implementation of efficient and scalable backend services”. Thrift provides a way to separate the interface of your service from the implementation; the interface defines your service’s API, while the implementation provides the serialization and transport necessary to interact with your service over-the-wire.

In more detail, Thrift divides a service API into four key components:

Types: Types define a common type system that is mapped to language implementations. For example, a developer should be able to transparently exchange a Python dictionary with a Go map without additional serialization effort.
Transport: Transports specify how data is moved from one service to another — the specifics of which should not matter to an application developer. The same service can use TCP sockets, HTTP requests, or even files on disk as a transport mechanism.
Protocol: Protocols define the encoding/decoding of data types. For example, the same service can communicate using a binary protocol, JSON, or XML.
Processors: Processing incoming requests is separated from the details of transport, data types, and protocol, allowing the client and server to focus solely on business logic.

These four components provide the interface and implementation of your service. For now, we can focus solely on the interface which is part of the Types component. Thrift’s IDL defines services using base types, structs, containers, and exceptions, each of which is discussed below.

base types: base types such as boolean, byte, double, and string are mapped to language specific types during code generation.
structs: structs serve as a common implementation of a generic object. A struct has a set of fields and syntactically is very similar to a C struct.
containers: containers map to containers or collections available in most programming languages. For example, a list<int> defines an ordered list of elements that is translated to a Java ArrayList or a Python list. Similar constructs are available for set and map containers.
exceptions: exceptions generate objects that inherit from the base exception class for your language of choice. This allows for native integration with the languages exception handling facilities.
service: Semantically, services are equivalent to defining an interface in object-oriented programming. Given a service definition, the Thrift compiler generates fully functional client and server code that implements the interface.

For more details on the IDL consult Thrift: The Missing Guide and the Thrift whitepaper.

Defining a Thrift Service

To define the Task Manager service, begin by creating a Thrift definition file taskmanager.thrift in a new Go project directory. The file should look like the one below. Inline comments have been added to help describe the IDL syntax.

# Single-line comments use a hashtag
// C style comments work too
/*
 * And multi-line comments are just like C.
 */

# Namespaces are language-specific. Generated code uses the namespace
# to separate generated code into packages or modules.
namespace java com.soofaloofa.taskmanager

/**
 * Define some common types for our service.
 * Thrift ensures that different languages serialize these type definitions to appropriate
 * language-specific types.
 */
typedef i64 id
typedef i32 int

/**
 * Define a constant using the `const` keyword.
 */
const id DEFAULT_ID = -1

/**
 * structs are mapped by Thrift to classes or structs in your language of
 * choice. This struct has two fields, an Identifier of type `id` and
 * a Description of type `string`. The Identifier defaults to DEFAULT_ID.
 */
struct Task {
    /** A unique identifier for this task. */
    1: id Identifier = DEFAULT_ID,

    /** A description of the task. */
    2: string Description
}

/**
 * Exceptions inherit from language-specific base exceptions.
 */
exception TaskException {
    /**@ The reason for this exception. */
    1: string Reason
}

/**
 * A service defines the API that is exposed to clients.
 *
 * This TaskManager service has one available endpoint for creating a task.
 */
service TaskManager {
    /**@
     * Create a new task.
     *
     * This method accepts a Task struct and returns an i64.
     * It may throw a TaskException.
     */
    i64 create(1:Task task) throws (1:TaskException e)
}

This Thrift definition file can be used to generate client and server code for the language of your choice, but first you need to install Thrift. On OS X, it is easiest to do this using Homebrew. On other platforms, consult the Thrift documentation.

brew install thrift

Generating Thrift clients and servers

With Thrift installed, you can generate client and server code from your Thrift IDL file. For example, to generate Go code, run the following command:

$ thrift --gen=go taskmanager.thrift

or generate Java code with:

$ thrift --gen=java taskmanager.thrift

By default, generated code will be placed in the gen-go or gen-java directories. This can be changed using the optional --out option to the thrift command.

Now that we have our generated code, we can write our client and server applications to send and receive messages in the language of your choice.

Implementing a Thrift Service

We will start with a Go implementation. To follow along, create a new Go project called taskmanager in your workspace and copy the taskmanager.thrift file defined above to that directory. After installing Thrift, generate your go code using the command

thrift --gen go taskmanager.thrift

After completing the above steps, you should have a folder structure like the following.

$GOPATH/src/github.com/soofaloofa/taskmanager/
├── gen-go
│   └── taskmanager
│       ├── constants.go
│       ├── task_manager-remote
│       │   └── task_manager-remote.go
│       ├── taskmanager.go
│       └── ttypes.go
└── taskmanager.thrift

What’s lacking is an entry point to our service that uses the generated code. For that purpose, create a main.go file that imports the generated code and processes messages with it.

package main

import (
	"fmt"

	"git.apache.org/thrift.git/lib/go/thrift"
	"github.com/soofaloofa/taskmanager/gen-go/taskmanager"
)

// Maintain a global list of tasks to work on
var tasks []taskmanager.Task

// TaskManagerHandler processes incoming tasks
type TaskManagerHandler struct{}

// Create adds tasks to our global list
func (t *TaskManagerHandler) Create(task *taskmanager.Task) (r int64, err error) {
	// task is a struct as defined by our Thrift IDL definition.
	// here we simply add it to our list of work
	tasks = append(tasks, *task)

	// Return the response
	return int64(task.GetIdentifier()), nil
}

func main() {
	addr := "localhost:9090"

	// Declare the serialization protocol
	protocolFactory := thrift.NewTSimpleJSONProtocolFactory()

	// Declare the transport method to use
	transportFactory := thrift.NewTBufferedTransportFactory(8192)

	// Get a tansport
	transport, _ := thrift.NewTServerSocket(addr)

	// Implements the interface to our service
	handler := &TaskManagerHandler{}

	// Tell the Thrift processor which interface implementation to use
	processor := taskmanager.NewTaskManagerProcessor(handler)

	// Start the server
	server := thrift.NewTSimpleServer4(processor, transport, transportFactory, protocolFactory)

	fmt.Println("Starting the simple server... on ", addr)
	server.Serve()
}

In this simplified example, we declare an implementation for our services interface with TaskManagerHandler. Then tell Thrift to use this implementation to handle incoming requests. This basic application can be installed and ran. Go ahead and do that now.

$ go install
$ taskmanager
Starting the Thrift server on localhost:9090

Implementing a Thrift Client

With our task manager service up and running, we can write the client application that sends requests to the service. Alice is an enterprise Java developer, so we begin by creating a new directory alice in $HOME to house the Java code for sending messages to the server. The client uses the same Thrift definition as the server, so copy the taskmanager.thrift file into your new Java directory and generate the Java code for the Thrift definition. This time, generate your code into the src/main/java directory of your new project using Thrift’s --out option.

$ thrift -gen java -out src/main/java taskmanager.thrift

At the end of these steps, you should end up with a directory structure looking like the structure below. Note that Thrift respected the namespace in taskmanager.thrift and packaged your Java code in com.soofaloofa.taskmanager.

$HOME/alice/
├── pom.xml
├── src
│   └── main
│       └── java
│           └── com
│               └── soofaloofa
│                   └── taskmanager
│                       ├── Task.java
│                       ├── TaskException.java
│                       ├── TaskManager.java
│                       └── taskmanagerConstants.java
└── taskmanager.thrift

Much like with the server, it’s time to write our client entry point by creating a Main.java that uses Thrift to send messages. If you don’t have Maven installed already, you can install it with Homebrew.

brew install maven

With Maven installed, create a pom.xml file in your new project with the following contents.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.soofaloofa</groupId>
    <artifactId>alice</artifactId>
    <version>1.0</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.thrift</groupId>
            <artifactId>libthrift</artifactId>
            <version>0.9.3</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <finalName>${project.artifactId}-bundled-${project.version}</finalName>
                            <artifactSet>
                                <includes>
                                    <include>*:*</include>
                                </includes>
                            </artifactSet>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

This POM file adds the Thrift dependency and configures Maven to use Java 1.8 and include all dependencies in the generated Jar file in the <build> section. It’s really unimportant for the purposes of this tutorial — just copy and paste it to pom.xml.

You should be able to test your new application by running mvn package and watching for a BUILD SUCCESS message.

$ mvn package
... stuff happening ...
... BUILD SUCCESS ...

With the Maven project setup complete, create a new file src/main/java/com/soofaloofa/alice/Main.java as the entry point for our client application.

package com.soofaloofa.alice;

import com.soofaloofa.taskmanager.Task;
import com.soofaloofa.taskmanager.TaskManager;
import org.apache.thrift.TException;
import org.apache.thrift.protocol.TProtocol;
import org.apache.thrift.protocol.TSimpleJSONProtocol;
import org.apache.thrift.transport.TSocket;
import org.apache.thrift.transport.TTransport;

public class Main {

  public static void main(String[] args) throws TException {
    // Open a transport to the server
    TTransport transport = new TSocket("localhost", 9090);
    transport.open();

    // Declare the protocol to use
    TProtocol protocol = new TSimpleJSONProtocol(transport);

    // Create a client using the protocol
    TaskManager.Client client = new TaskManager.Client(protocol);

    // Create a task
    Task task = new Task();
    task.setIdentifier(42);
    task.setDescription("Close Jira ticket PR-12345");

    // Send it to the server
    long result = client.create(task);
    System.out.println("Processed task " + result);

    transport.close();
  }
}

You should be able to run this application to send tasks to the task manager Go application we started earlier.

$ mvn package
...
$ java -cp target/alice-bundled-1.0.jar com.soofaloofa.alice.Main
Processed task 42

You can see the response from our server printed to the console.

Recap

Let’s recap what we’ve learned. First, we defined the service that we wish to implement using Thrift’s Interface Definition Language or IDL. Then, we used the thrift application to generate client and server code for our service, allowing us to decouple the serialization protocol, transport method, and programming language from the application’s business logic.

So you want to use Thrift?#

The Task Manager Service#

Defining the Task Manager API Using Thrift#

Thrift Basics#

Defining a Thrift Service#

Generating Thrift clients and servers#

Implementing a Thrift Service#

Implementing a Thrift Client#

Recap#