Demystify Proto

20.02.2024 — proto, grpc, buf — 3 min read

Embarking on the journey of maintaining Protocol Buffers (proto) at Flyte led us through a lot's of challenges, pushing us to find innovative solutions. Our initial script, designed to generate code in Java, Python, Go, JS, and C, proved functional but lacked the maintenance and user-friendliness we needed. Every Flyte contributor shared the struggle of dealing with this script.

Code consumption posed another hurdle, especially when it came to publishing packages to a package registry. While Flyte successfully published Python, JS, and Go packages, the Java package remained neglected. Enter the Flytekit-Java engineers from Spotify, who took matters into their own hands, manually consuming proto and crafting a distinct build script for code generation.

Witnessing similar struggles, At UnionAI, heavy reliance on flyteidl for our UnionAI Cloud became apparent. Despite using the Flyteidl submodule, the implementation within the code was less than ideal, involving manual copy-paste or bash scripts. My first experience handling cross-project proto usage left much to be desired.

Despite being a gRPC advocate, the intricacies of proto production and consumption led me to question its early stage adoption. To address distribution challenges, I discovered Buf CLI, a powerful tool streamlining proto code generation through a plugin marketplace, eliminating the need for hacky bash scripts.

Buf also introduced a proto registry, enabling the publication and cross-project utilization of proto definitions. This established a best practice where producers publish proto definitions, and consumers are responsible for code generation based on their specific needs.

To address our own challenges, we published Flyteidl proto in the Buf registry, using it as a dependency in the Union proto. This eliminated the need for the submodule and the bash script, enabling us to create generation templates tailored to our needs

version: v1
name: buf.build/flyteorg/flyteidl
lint:
  use:
    - DEFAULT
breaking:
  use:
    - FILE
deps:
  - buf.build/googleapis/googleapis:62f35d8aed1149c291d606d958a7ce32
  - buf.build/unionai/protoc-gen-swagger

Buf simplifies package management for NPM, Go, Java, and Python, eliminating the need for manual package publishing. To publish your proto definition, use Buf registry with the following command:

buf push

If consumers require custom code generation, they can create their own generation template, as shown in the example below:

# buf.gen.yaml generation template
version: v1
managed:
  enabled: true
plugins:
  - plugin: buf.build/grpc/python:v1.53.0
    out: gen/pb_python
  - plugin: buf.build/protocolbuffers/python:v22.2
    out: gen/pb_python
  - plugin: buf.build/protocolbuffers/pyi:v22.2
    out: gen/pb_python
  - plugin: buf.build/community/neoeinstein-prost
    out: gen/pb_rust

Now, code generation is straightforward using the specified template:

buf generation buf.build/flyteorg/flyteidl --include-imports

You can also use Flyteidl as a dependency in your proto definition:

version: v1
name: buf.build/<ORG>/<REGISTRY>
lint:
  use:
    - DEFAULT
breaking:
  use:
    - FILE
deps:
  - buf.build/googleapis/googleapis:62f35d8aed1149c291d606d958a7ce32
  - buf.build/unionai/protoc-gen-swagger
  - buf.build/flyteorg/flyteidl

Challenges persisted in our ecosystem, with engineers navigating the clash between GRPC and REST preferences. In response, a grpc gateway emerged, enabling developers to write in GRPC while consumers interacted with it as REST. Despite its complexity, the grpc gateway gained popularity across various projects.

In the numerous challenges, I came across an grpc implementation presented by Akshay Shah at GopherCon 2022. In just seven minutes, he showcased how effortlessly anyone could create a grpc server implementation, challenging the conventional understanding of GRPC. This revelation underscored the complexities within the GRPC community and laid the foundation for Buf's announcement of ConnectRPC. ConnectRPC emerges as the optimal approach for seamlessly constructing REST servers, and I'm contemplating writing another blog post to delve deeper into the realm of ConnectRPC.