gRPC – quick talk between C# Python

gRPC is a high-performance remote procedure call (RPC) framework. RPC used to be great at some points, but this type of communication always used to be heavily coupled to a specific language and framework. Contrary, gRPC uses Protocol Buffers as its Interface Definition Language which offers efficient, reliable and fast communication between distributed systems independently of a framework those are built upon.

In this post I want to show how easy it is to establish a gRPC communication between a C# and Python services.

tldr; whole example is available here

C# server

First let’s create a simple C# worker service and install Grpc.AspNetCore library

dotnet add package Grpc.AspNetCore --version 2.49.0

Then let’s create somo proto definitions. Create a folder Proto and add a calculator.proto to it with the following:

syntax = "proto3";
import "google/protobuf/wrappers.proto";

option csharp_namespace = "GrpcService";

package calculator;

service Calculator {
  rpc Add (AddRequest) returns (AddReply);
  rpc Random (RandomRequest) returns (RandomReply);
}

message AddRequest {
  double a = 1;
  double b = 2;
}

message AddReply {
  double sum = 1;
}

message RandomRequest{
  optional google.protobuf.Int32Value min = 1;
  optional google.protobuf.Int32Value max = 2;
}

message RandomReply {
  int32 result = 1;
}

To auto generate c# classes from proto files you may need to install relevant plugin to the IDE that you are using. For example Rider should automatically suggest you to install one. Anyway it’s always worth to check your csproj after adding proto file if it’s correctly added.

<ItemGroup>
    <Protobuf Include="Protos\calculator.proto" GrpcServices="Server" />
</ItemGroup>

The proto defines a service that exposes to methods. Now we need to implement those.

// CalculatorService.cs
public class CalculatorService : Calculator.CalculatorBase
{
  private static readonly Random _random = new Random();
    public override Task<AddReply> Add(AddRequest request, ServerCallContext context)
    {
      var reply = new AddReply
      {
        Sum = request.A + request.B
      };
      return Task.FromResult(reply);
    }

    public override Task<RandomReply> Random(RandomRequest request, ServerCallContext context)
    {
      var reply = new RandomReply
      {
        Result =
          request.Max.HasValue && request.Min.HasValue ? _random.Next(request.Min.Value, request.Max.Value) :
          request.Max.HasValue ? _random.Next(request.Max.Value) : _random.Next()
      };
      return Task.FromResult(reply);
    }
}

Finally we’ll start a service when the app is started using MapGrpcService

// Program.cs
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddGrpc();

var app = builder.Build();

app.MapGrpcService<CalculatorService>();

app.Run();

Make sure to update appsettings.json with http2 protocol support

  "Kestrel": {
    "EndpointDefaults": {
      "Protocols": "Http1AndHttp2"
    }
  }

We still want to keep http1 support for the last case in this post.

Python client

We’ll write a simple python script that will send requests to the C#-based service. First make sure you have installed grpcio-tools

pip install grpcio-tools

Can copy or link the proto file from C# project so you don’t have to edit it in multiple places

# make sure to have valid paths
ln ../../calculator.proto ./

To generate the proto files you can use the following command which I suggest separating into some shell script so it can be reused later (unless you have an IDE that build it)

#build-protos.sh
#!/bin/bash
python -m grpc_tools.protoc -I./protos --python_out=. --pyi_out=./ --grpc_python_out=./ ./protos/calculator.proto

The above runs grpc_tools to build python services and models from proto files. It requires to set a folder where to look for proto, where to put grpc python services, last but not least pyi files contain output for models defining requests & replies. The last parameter is the proto file. It may contain multiple files or you can set it to include all protos from a folder like ./protos/*.proto

Server script is always we can write the python script

import grpc
import calculator_pb2
import calculator_pb2_grpc
import random
from concurrent import futures
from calculator_pb2_grpc import CalculatorServicer


class CalculatorServicerImpl:
    CalculatorServicer

    def Add(self, request, context):
        return calculator_pb2.AddReply(sum=request.a + request.b)

    def Random(self, request, context):
        result = random.randint(request.min.value, request.max.value)
        return calculator_pb2.RandomReply(result=result)


server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
calculator_pb2_grpc.add_CalculatorServicer_to_server(
    CalculatorServicerImpl(), server)
server.add_insecure_port('[::]:5029')
server.start()
server.wait_for_termination()

CalculatorServiceImpl is the implementation of generated CalulcatorServier from proto files. This is where you put the logic that gets executed when the request to the server is made.

The rest of the code is creating a gRPC server with mentioned service enabled. The above calls are synchronous, but each has also an option to get future representation.

I used the type wrappers_pb2.Int32Value which is converted into a nullable int in C#.

Reverse time: Python server <- C# Client

Ok, so the communication works fine in one-way. Let’s make it a little complex and make a Python into server as well. For simplicity let’s use the same proto definitions. Create a server python file that replies to two gRPC calls.

# server.py
import grpc
import calculator_pb2
import calculator_pb2_grpc
import random
from concurrent import futures
from calculator_pb2_grpc import CalculatorServicer


class CalculatorServicerImpl:
    CalculatorServicer

    def Add(self, request, context):
        return calculator_pb2.AddReply(sum=request.a + request.b)

    def Random(self, request, context):
        result = random.randint(request.min.value, request.max.value)
        return calculator_pb2.RandomReply(result=result)


server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
calculator_pb2_grpc.add_CalculatorServicer_to_server(
    CalculatorServicerImpl(), server)
server.add_insecure_port('[::]:5029')
server.start()
server.wait_for_termination()

Similar to the C# server, this one starts a grpc server exposed at 5029, that supports CalculatorService.

Python side seems finish, head back to C# project. Here we can support a client for the same proto file and a port where it should send request to. We also add two endpoints to easily test the communication. First add the client packages to project:

dotnet add package Grpc.Net.Client --version 2.52.0
dotnet add package Grpc.Net.ClientFactory --version 2.52.0

Now update csproj so that it will not only create server classes, but client ones as well.

<Protobuf Include="Protos\calculator.proto" GrpcServices="Server,Client" />

In the program setup we can now add GrpcClient and some endpoints to test communication with Python based server

// Program.cs
using GrpcService;
using GrpcService.Services;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddGrpc();
builder.Services.AddGrpcClient<GrpcService.Calculator.CalculatorClient>(o =>
{
  o.Address = new Uri("http://localhost:5029");
});

var app = builder.Build();
app.MapGrpcService<CalculatorService>();
app.MapPost("/add", async (httpContext) =>
{
  var request = await httpContext.Request.ReadFromJsonAsync<AddRequest>();
  var client = httpContext.RequestServices.GetService<GrpcService.Calculator.CalculatorClient>();
  var result = await client.AddAsync(request);

  await httpContext.Response.WriteAsJsonAsync(new
  {
    Sum = result.Sum
  });
});
app.MapPost("/random", async (httpContext) =>
{
  var request = await httpContext.Request.ReadFromJsonAsync<RandomRequest>();
  var client = httpContext.RequestServices.GetService<GrpcService.Calculator.CalculatorClient>();
  var result = await client.RandomAsync(request);
  await httpContext.Response.WriteAsJsonAsync(new
  {
    Random = result.Result
  });
});

app.Run();

AddGrpcClient works similar to adding a HttpClient, making client classes available for DI later in code.

Now run python server and c# client. Use any rest client to test it or simply curl

curl -X POST http://localhost:5028/random -H 'Content-Type: application/json' -d '{"min":"1","max":"3"}'

curl -X POST http://localhost:5028/add -H 'Content-Type: application/json' -d '{"a":"1","b":"3"}' --http2

Summary

gRPC may seems a little complicated at first, since you have to learn a new type of defining data using proto files. Except for that the communication works very smoothly and you can use it using many popular languages.

While using plain data may be easy, there are few popular types that have a problem with communication for example: Date/DateTime and UUID.

UUID issue – the safest possible way to use it is to represent it as a string and do encoding/decoding in the app. For example C# (Guid.ToBytesArray) and Java (Uuid.toRawBytes) will offer slightly different results:

cfd64db1-8784-40fb-83ef-60dd0bb7d88e ->

cf:d6:4d:b1:87:84:40:fb:83:ef:60:dd:0b:b7:d8:8e
vs
b1:4d:d6:cf:84:87:fb:40:83:ef:60:dd:0b:b7:d8:8e

Datetime is another good example when you have to be cautious. There are a lot of ISO formats supported. My favorite and I think the most secure way is to use an Unix timestamp and defined a proto field as integer.

gRPC – quick talk between C# <-> Python

C# server

Python client

Reverse time: Python server <- C# Client

Summary

readdarek.com