Skip to content

Building microservices in Go and Python using gRPC and TLS/SSL authentication

Posted on by Gustavo Henrique

The microservice architecture is more than a technical decision, is mainly a business decision because it possibility separated parts of the system to be improved in parallel without impact the other parts. The gRPC is a protocol used to implement a communication between microservices.

Remote Procedure Calls (RPCs) provide a useful abstraction for building distributed applications and services. Developers using gRPC start with a language agnostic description of an RPC service (a collection of methods). From this description, gRPC will generate client and server-side interfaces in any of the supported languages. The server implements the service interface, which can be remotely invoked by the client interface.

By default, gRPC uses Protocol Buffers as the Interface Definition Language (IDL) for describing both the service interface and the structure of the payload messages. Starting from an interface definition in a .proto file, gRPC provides Protocol Compiler plugins that generate Client and Server-side APIs. gRPC users call into these APIs on the Client side and implement the corresponding API on the server side. According to the official documentation, the main usage scenarios:

  • Low latency, highly scalable, distributed systems.
  • Developing mobile clients which are communicating to a cloud server.
  • Designing a new protocol that needs to be accurate, efficient and language independent.
  • Layered design to enable extension eg. authentication, load balancing, logging, and monitoring etc.

The proof of concept that I will show up is based in a fictional e-commerce platform. There are two microservices:

  1. Catalog: It is written in Go and exposes a REST API that returns a product list.
  2. Discount: It is consumed by Catalog, written in Python and returns the received product with 10% discount applied.

In order to generated code from protocol buffers definition files, do the following:

  • Download and install protoc compiler from here: https://github.com/google/protobuf. Add the location of protoc binary file into PATH environment variable so that you can invoke protoc compiler from any location.
git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
./configure
make
sudo make install
sudo ldconfig
  • Install the protoc plugin for Go. Run the go get command to install the protoc plugin for Go and ensure that the protoc-gen-go binary is in the PATH:
go get -u google.golang.org/grpc
go get -u github.com/golang/protobuf/proto
go get -u github.com/golang/protobuf/protoc-gen-go
sudo ln -snf $GOROOT/bin/protoc-gen-go /usr/local/bin
  • Install the protoc plugin for Python3. I recommend you to use a virtual environment for this:
pip install virtualenv
virtualenv venv
source venv/bin/activate
pip install grpcio grpcio-tools

Getting Started

First, we need to create the project structure. In the following example, the microservices-grpc-go-python directory is created in $HOME:

mkdir -p ~/microservices-grpc-go-python/{keys,catalog,discount}

Then we need to generate the self-signed certificates used by authentication. The Catalog client will use the cert.pem to be authenticated in the Discount server.

cd ~/microservices-grpc-go-python/keys
openssl req -x509 -newkey rsa:4096 -keyout private.key -out cert.pem -days 365 -nodes -subj '/CN=localhost'

And finally, we define the service interface and the structure of the payload messages in a Protocol Buffer file. The .proto file extension is used for creating a Protocol Buffer file. Here is the source of ecommerce.proto file from the root directory of the application:

syntax = "proto3";
package ecommerce;

service Discount {
  rpc ApplyDiscount (DiscountRequest) returns (DiscountResponse) {}
}

message Customer {
  int32 id = 1;
  string first_name = 2;
  string last_name = 3;
}

message Product {
  int32 id = 1;
  string slug = 2;
  string description = 3;
  int32 price_in_cents = 4;
  DiscountValue discount_value = 5;
}

message DiscountValue {
  float pct = 1;
  int32 value_in_cents = 2;
}

message DiscountRequest {
  Customer customer = 1;
  Product product = 2;
}

message DiscountResponse {
  Product product = 1;
}

The .proto file starts with the version of Protocol Buffer and a package declaration. We use the latest proto3 version of the Protocol Buffers language. The package is declared with a name “ecommerce”. When you generate Go source code from the proto file, it will add Go package name as “ecommerce”.

Inside a .proto file, we define message types and service interface. Standard data types such as int32, float, double, and string are available as field types for declaring elements in message types. The user-defined message types can also be used as the field types. The number after the “=” marker on each element of the message types specifies the unique identifier that field uses in the binary encoding. The Protocol Buffers language guide is available from here: https://developers.google.com/protocol-buffers/docs/proto3.

A named service “Discount” is used to define a service that has the RPC method named “ApplyDiscount”. This method works with a typical Request/Response model where the client sends a request to the RPC server using the stub and waits for a response.

Generating the Python code for Discount gRPC server

Once we’ve defined the proto file, the next step is to generate source code for the gRPC server interface to write your server implementation and making client calls based on the messages types and service interface defined in the proto file. The protocol buffer compiler protoc is used with a gRPC Python plugin to generate the server code. From the discount directory of the application, run the compiler as a Python module:

cd ~/microservices-grpc-go-python/discount
python -m grpc_tools.protoc -I=.. --python_out=. --grpc_python_out=. ../ecommerce.proto

This will generate the Python source files named ecommerce_pb2_grpc.py and ecommerce_pb2.py. We will use these files to create our implementation of the “ApplyDiscount” method. The business logic to apply the discount is very simple: if the customer ID is equal to 1 and the product price is greater than zero, there will be applied a discount of 10%.

The below server.py file in the discount directory implements the “ApplyDiscount” method:

import sys
import time
import os
import grpc
import decimal
import ecommerce_pb2
import ecommerce_pb2_grpc
from concurrent import futures


class Ecommerce(ecommerce_pb2_grpc.DiscountServicer):
    def ApplyDiscount(self, request, content):
        customer = request.customer
        product = request.product
        discount = ecommerce_pb2.DiscountValue()
        if customer.id == 1 and product.price_in_cents > 0:
            percentual = decimal.Decimal(10) / 100  # 10%
            price = decimal.Decimal(product.price_in_cents) / 100
            new_price = price - (price * percentual)
            value_in_cents = int(new_price * 100)
            discount = ecommerce_pb2.DiscountValue(pct=percentual, value_in_cents=value_in_cents)

        product_with_discount = ecommerce_pb2.Product(id=product.id,
            slug=product.slug,
            description=product.description,
            price_in_cents=product.price_in_cents,
            discount_value=discount)
        return ecommerce_pb2.DiscountResponse(product=product_with_discount)


if __name__ == '__main__':
    port = sys.argv[1] if len(sys.argv) > 1 else 443
    host = '[::]:%s' % port
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=5))
    keys_dir = os.path.abspath(os.path.join('.', os.pardir, 'keys'))
    with open('%s/private.key' % keys_dir, 'rb') as f:
        private_key = f.read()
    with open('%s/cert.pem' % keys_dir, 'rb') as f:
        certificate_chain = f.read()
    server_credentials = grpc.ssl_server_credentials(((private_key, certificate_chain),))
    server.add_secure_port(host, server_credentials)
    ecommerce_pb2_grpc.add_DiscountServicer_to_server(Ecommerce(), server)
    try:
        server.start()
        print('Running Discount service on %s' % host)
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        server.stop(0)

The script should receive as an argument the port number to accepts incoming connections or the default port 443 will be used. The server uses the self-signed certificates stored in the keys directory to create a secure connection.

Generating the Go code for Catalog gRPC client/HTTP Server

From the catalog directory, run the protoc compiler with gRPC Go plugin:

mkdir ecommerce
protoc -I=.. --go_out=plugins=grpc:ecommerce ../ecommerce.proto

This will generate a Go source file named ecommerce.pb.go inside the ecommerce directory. This file provides the essential code to make RPC calls from client applications. The below main.go file in the catalog directory shows the source for creating the gRPC client by providing an implementation for the RPC methods defined in the service definition and the HTTP server exposing the /products API.

package main

import (
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"log"
	"net/http"
	"os"
	"path/filepath"
	"strconv"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"

	pb "microservices-grpc-go-python/catalog/ecommerce"
)

func getDiscountConnection(host string) (*grpc.ClientConn, error) {
	wd, _ := os.Getwd()
	parentDir := filepath.Dir(wd)
	certFile := filepath.Join(parentDir, "keys", "cert.pem")
	creds, _ := credentials.NewClientTLSFromFile(certFile, "")
	return grpc.Dial(host, grpc.WithTransportCredentials(creds))
}

func findCustomerByID(id int) (pb.Customer, error) {
	c1 := pb.Customer{Id: 1, FirstName: "John", LastName: "Snow"}
	c2 := pb.Customer{Id: 2, FirstName: "Daenerys", LastName: "Targaryen"}
	customers := map[int]pb.Customer{
		1: c1,
		2: c2,
	}
	found, ok := customers[id]
	if ok {
		return found, nil
	}
	return found, errors.New("Customer not found.")
}

func getFakeProducts() []*pb.Product {
	p1 := pb.Product{Id: 1, Slug: "iphone-x", Description: "64GB, black and iOS 12", PriceInCents: 99999}
	p2 := pb.Product{Id: 2, Slug: "notebook-avell-g1511", Description: "Notebook Gamer Intel Core i7", PriceInCents: 150000}
	p3 := pb.Product{Id: 3, Slug: "playstation-4-slim", Description: "1TB Console", PriceInCents: 32999}
	return []*pb.Product{&p1, &p2, &p3}
}

func getProductsWithDiscountApplied(customer pb.Customer, products []*pb.Product) []*pb.Product {
	host := os.Getenv("DISCOUNT_SERVICE_HOST")
	if len(host) == 0 {
		host = "localhost:11443"
	}
	conn, err := getDiscountConnection(host)
	if err != nil {
		log.Fatalf("did not connect: %v", err)
	}
	defer conn.Close()

	c := pb.NewDiscountClient(conn)
	ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
	defer cancel()

	productsWithDiscountApplied := make([]*pb.Product, 0)
	for _, product := range products {
		r, err := c.ApplyDiscount(ctx, &pb.DiscountRequest{Customer: &customer, Product: product})
		if err == nil {
			productsWithDiscountApplied = append(productsWithDiscountApplied, r.GetProduct())
		} else {
			log.Println("Failed to apply discount.", err)
		}
	}

	if len(productsWithDiscountApplied) > 0 {
		return productsWithDiscountApplied
	}
	return products
}

func handleGetProducts(w http.ResponseWriter, req *http.Request) {
	products := getFakeProducts()
	w.Header().Set("Content-Type", "application/json")

	customerID := req.Header.Get("X-USER-ID")
	if customerID == "" {
		json.NewEncoder(w).Encode(products)
		return
	}
	id, err := strconv.Atoi(customerID)
	if err != nil {
		http.Error(w, "Customer ID is not a number.", http.StatusBadRequest)
		return
	}

	customer, err := findCustomerByID(id)
	if err != nil {
		json.NewEncoder(w).Encode(products)
		return
	}

	productsWithDiscountApplied := getProductsWithDiscountApplied(customer, products)
	json.NewEncoder(w).Encode(productsWithDiscountApplied)
}

func main() {
	port := "11080"
	if len(os.Args) > 1 {
		port = os.Args[1]
	}

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "It is working.")
	})
	http.HandleFunc("/products", handleGetProducts)

	fmt.Println("Server running on", port)
	http.ListenAndServe(":"+port, nil)
}

A gRPC channel is created to communicate with the server in order to call RPC methods. Function grpc.Dial is used to communicate with the RPC server. When you call grpc.Dial, you can pass a DialOptions to set the authentication credentials. In order to call RPC methods, we need to create a client stub which is created using function NewDiscountClient that returns an object of DiscountClient. Function NewDiscountClient and type DiscountClient are defined in the generated code file ecommerce.pb.go. The main function should receive as an argument the port number to accepts incoming connections or the default port 11080 will be used. The gRPC server address is defined by an environment variable DISCOUNT_SERVICE_HOST or the default localhost:11443 will be used.

Running and testing the application

We should open three terminal windows (or tabs) to realize this test. Let’s run the Discount gRPC server in the first terminal window:

cd ~/microservices-grpc-go-python/discount
source venv/bin/activate
python server.py 11443

Then we run the Catalog gRPC client in the second terminal window:

cd ~/microservices-grpc-go-python/catalog
go run main.go

Now we have two services running at the same time. The Discount service is running a gRPC server on localhost:11443 and the Catalog service is running an HTTP server on localhost:11080.

We should test the RPC method by making a request with the customer ID in the HTTP header to the Catalog service:

curl -H 'X-USER-ID: 1' http://localhost:11080/products

You should see the output similar to the following:

[
    {
        "id": 1,
        "slug": "iphone-x",
        "description": "64GB, black and iOS 12",
        "price_in_cents": 99999,
        "discount_value": {
            "pct": 0.1,
            "value_in_cents": 89999
        }
    },
    {
        "id": 2,
        "slug": "notebook-avell-g1511",
        "description": "Notebook Gamer Intel Core i7",
        "price_in_cents": 150000,
        "discount_value": {
            "pct": 0.1,
            "value_in_cents": 135000
        }
    },
    {
        "id": 3,
        "slug": "playstation-4-slim",
        "description": "1TB Console",
        "price_in_cents": 32999,
        "discount_value": {
            "pct": 0.1,
            "value_in_cents": 29699
        }
    }
]

Source Code

The source code of the example is available in https://github.com/gustavohenrique/microservices-grpc-go-python.