Why Protobuf

Protobuf is an open-source cross-language and cross-platform serialization protocol developed by Google, widely used for inter-service communication and data archiving on disks. For example, the common gRPC framework uses the Protobuf protocol by default for inter-service communication.

The advantage of Protobuf is its support for multiple languages, such as Go, Dart, Java, Kotlin, and various other languages...

Developers using different languages and frameworks can use the same standard to describe and call interfaces without worrying about the specific code details, thus greatly improving collaboration efficiency.

In teams collaborating with Protobuf, the development process will become as follows:

Invalid mermaid header: "graph TD;". Expected "graph TD", "flowchart LR", "stateDiagram-v2", etc.

CodeBlock Loading...

As you can see, when the source of development collaboration is converged into a unified entry point, team collaboration naturally becomes unified and smooth. When there are interface changes, the change logic is also in a closed loop, making it possible for interface changes to become an atomic change. (If it's a Monorepo, this can be easily achieved, and the cost of changes will be lower.)

From a team collaboration perspective, the complexity of interface docking between personnel is transferred to writing .proto files. Although complexity does not disappear, it only shifts, the complexity brought by personnel collaboration is uncontrollable compared to the complexity of writing .proto.

Why Not Swagger

When we want to generate interface type definitions, the common choice is Swagger, which has good RESTful API support, is easy to read and test, and has strong community support, such as Swagger-based interface testing.

The downside is that developers usually need additional annotations to generate documentation, and from Swagger's design purpose, it is more designed for API documentation. This purpose gives it several inherent drawbacks:

Allows Swagger documents and actual interfaces to be inconsistent, Swagger annotations do not participate in runtime, so it's hard to detect if developers forget to update interface documentation;
Interface changes cannot be perceived, especially during the initial development phase when interface changes are frequent. It's hard for the web to detect interface changes from long documents.

Both of these shortcomings rely on developers to supplement and improve, often causing practical problems in daily collaboration.

Protobuf can perfectly solve these two problems:

Because the .proto file is the source, if the interface is changed without changing the implementation, the code cannot be compiled;
Similarly, edits to the .proto file will be synchronized to code on various ends, and issues can be detected during code compilation.

Note

This section mainly discusses using Protobuf and corresponding tools to solve team collaboration issues, which are essentially benefits brought by Protobuf's design philosophy.

How

First, our client code runs in a browser environment, so HTTP API is certainly a more common choice. Although there is grpc-web that can run in a browser, it ultimately adds another runtime, which is not as straightforward as XHR/Fetch Json.

Since we want to obtain interface agreements and field hints, TypeScript is obviously a better choice than JavaScript.

So our question becomes, how to generate a TypeScript HttpClient from .proto.

After some searching, we found the following tools available:

ts-proto -- Generates TypeScript from .proto
protobuf-ts -- Generates TypeScript from .proto
protobuf.js -- Parses .proto AST with js
ts-morph -- Usually used to manipulate TypeScript AST

You can see this discussion for the differences between ts-proto and protobuf-js

Clearly, there are no tools that directly meet our expectations; the first two are more designed for RPC clients.

So we break the problem into 2 steps:

Generate TS type agreements;
Generate the corresponding HTTP Client code;

Only Types

From the perspective of generating only types, ts-proto clearly meets our expectations better. It supports many Type Only options to help generate types more concisely.

onlyTypes=true will automatically generate only types and exclude dependencies on long and protobufjs/minimal. Equivalent to outputJsonMethods=false,outputEncodeMethods=false,outputClientImpl=false,nestJs=false

Referencing ts-proto documentation, we have the following command:

BASH

protoc --plugin=./node_modules/.bin/protoc-gen-ts_proto --proto_path=$PROTO_PATH --proto_path=./node_modules/google-proto-files --ts_proto_out=$DIST_PATH --ts_proto_opt=useSnakeTypeName=false,outputIndex=true,outputClientImpl=false,lowerCaseServiceMethods=true,outputServices=default,onlyTypes=true,outputJsonMethods=false,outputEncodeMethods=false,useExactTypes=true,useOptionals=messages,esModuleInterop=true $PROTO_PATH/apis/**/**/**.proto

Let's break down the corresponding configuration:

We need to pass the proto_path to be parsed, which includes the dependency path. For example, if we use Google's Proto dependency, we need to add google-proto-files.
The final ts output directory: DIST_PATH
useSnakeTypeName=false, Box_Element_Image_Alignment => BoxElementImageAlignment
outputIndex=true, will output index.ts, and export through namespaces, like export * from './api/book'
lowerCaseServiceMethods=true, the resulting Service will be service.findFoo instead of service.FindFoo
useOptionals=messages, Message fields in proto3 are always optional
esModuleInterop=true, keep this consistent with tsconfig, no further explanation
outputServices=default We need to retain the Service field to determine which interfaces exist; otherwise, the output code will not have interface agreements, only entity type agreements (Messages in Proto).

If your .proto file is:

PROTO

service Example {
  rpc Foo(FooRequest) returns (FooResponse) {
    option (google.api.http) = {get: "/example/foo"};
  }
}

message FooRequest {
  string foo = 1;
}

message FooResponse {
  string bar   = 1;
}

Then you will get a type description with a lowercase service method name and optional fields for each interface:

export interface FooRequest {
  foo?: string | undefined;
}

export interface FooResponse {
  bar?: string | undefined;
}

export interface Example {
  foo(request: FooRequest): Promise<FooResponse>;
}

At this point, our first goal is completed, So Easy.

Generate Method

Here is a problem: the google.api.http declared in the original service is gone, and we cannot get the HTTP Method and Path fields.

In fact, ts-proto documentation mentions:

We also don't support clients for google.api.http -based Google Cloud APIs, see #948 if you'd like to submit a PR.

Interested parties can follow this PR:

https://github.com/stephenh/ts-proto/pull/1075

However, even if google.api.http is supported, it still doesn't meet my requirements. I need a more customized interface type to be compatible with existing types while meeting business requirements. Therefore, I choose to modify on top of the Only Types.

Generate Api With Path

Next, we need to use ts-proto and protobuf.js to generate the Api types we need. The code here is for reference only.

import { Project, VariableDeclarationKind } from "ts-morph";
import protobuf from "protobufjs";

// Check if an array is valid
function isValidArray(val) {
  return val && Array.isArray(val) && val.length > 0;
}

// Capitalize the first letter
function getFirstUpperCase(str) {
  return str.charAt(0).toUpperCase() + str.slice(1);
}

function generateApi() {
  // Build a project corresponding to morph, you can pass in tsconfig.json as needed
  const project = new Project({});
  const sourceFiles = project.addSourceFilesAtPaths(`src/**/*.ts`);
  // Because it includes asynchronous file writing operations, hence Promise.all to ensure state completion
  return Promise.all([
    // Spell corresponding methods according to google.api.http
    ...sourceFiles.map((sourceFile) => {
      const filePath = sourceFile.getFilePath();
      // Get all Interfaces including Service and Message
      const interfaceDeclarations = sourceFile.getInterfaces();
      interfaceDeclarations.forEach((interfaceDeclaration) => {
        if (!interfaceDeclaration) {
          return;
        }
        const methods = interfaceDeclaration.getMethods();
        // Only process Services, those without methods will be filtered out
        if (!isValidArray(methods)) {
          return;
        }
        // Get service name
        const serviceName = interfaceDeclaration.getName();
        // Find the corresponding proto file
        const protoFilePath = filePath
          .replace(tsDir, protoDir)
          .replace(".ts", ".proto");
        if (existsSync(protoFilePath)) {
          const content = readFileSync(protoFilePath, "utf-8");
          // Parse proto's AST
          const result = protobuf.parse(content);
          const packageName = result.package.split(".")[1];
          const protoServiceMethods = result.root.lookupService(
            `${result.package}.${serviceName}`
          ).methods;
          if (!isValidArray(Object.keys(protoServiceMethods))) {
            return;
          }
          methods.forEach((method) => {
            const methodName = method.getName();
            const comments = method.getLeadingCommentRanges();
            // Get comments
            const comment =
              comments?.[0]
                ?.getText()
                .replace(/(\/\*\*|\*\/|\s+?\*|\/)/g, "") || " ";
            // The method name in ts starts with a lowercase letter, in proto it starts with an uppercase letter
            const protoMethodName = getFirstUpperCase(methodName);
            const protoServiceMethod = protoServiceMethods[protoMethodName];
            // Parse options
            const protoServiceMethodOptions = protoServiceMethod?.parsedOptions;
            if (protoServiceMethodOptions) {
              // Get api related information
              const httpOption = protoServiceMethodOptions.find(
                (option) => option["(google.api.http)"]
              )?.["(google.api.http)"];
              if (!httpOption) {
                return;
              }
              const httpMethod = Object.keys(httpOption)[0];
              const httpPath = httpOption[httpMethod];
              let data = null;
              let query = null;
              if (httpMethod === "get") {
                // For get requests, parameters are passed with query
                query = protoServiceMethod.requestType;
              } else {
                // For post requests, parameters are passed with body
                data = protoServiceMethod.requestType;
              }
              const methodName = `${httpMethod.toLowerCase()}${serviceName}${protoMethodName}`;
              if (data) {
                sourceFile
                  .addVariableStatement({
                    declarationKind: VariableDeclarationKind.Const,
                    isExported: true,
                    declarations: [
                      {
                        name: methodName,
                        initializer: `(query : {}, data: ${data}, config: {} = {})=> ({
                    resp: ${protoServiceMethod.responseType},
                    path: '${httpPath}',
                    method: '${httpMethod.toUpperCase()}',
                    query: query,
                    body: data,
                    config
                    }) as const`,
                      },
                    ],
                  })
                  .addJsDoc(comment);
              }
              if (query) {
                sourceFile
                  .addVariableStatement({
                    declarationKind: VariableDeclarationKind.Const,
                    isExported: true,
                    declarations: [
                      {
                        name: methodName,
                        initializer: `(query : ${query}, config: {} = {})=> ({
                    resp: ${protoServiceMethod.responseType},
                    path: '${httpPath}',
                    method: '${httpMethod.toUpperCase()}',
                    query: query,
                    config
                  }) as const`,
                      },
                    ],
                  })
                  .addJsDoc(comment);
              }
            }
          });
        }
      });
      sourceFile.formatText();
      // Save ts file, this is an asynchronous method, so we use promise.all
      return sourceFile.save();
    }),
  ]);
}

At this point, we will get an interface type agreement like this:

export const getExampleFoo = (query: FooRequest, params: {} = {}) =>
  ({
    resp: FooResponse,
    path: "/example/foo",
    method: "GET",
    query: query,
    params,
  } as const);

Doesn't it look very close to Swagger's generated code, with the method name written at the beginning of the function, and parameters like Path included? This method can be conveniently used for request libraries like axios. Since I have my own encapsulation, there is no specific properties mapping for axios.

At this point, we have achieved our initial expectation by generating HttpClientApi type agreements by parsing .proto files.

Function Name With Namespace

Smart friends quickly realized that the type names above can easily duplicate between different services, causing many prompts for the same type import when writing code on the front end. This is the difference between ServiceApi and WebApi.

ServiceApi is very clear about which service it wants to call, what methods it needs, and is more database-oriented, naturally having nested features. So server-side code is generally like BookService.getDetailByName.

WebApi's interface is more user-oriented and business-oriented, often involving multiple Services, and return parameters often need to come from multiple entity classes, which makes it very easy to have naming conflicts if we need a flat index.

We can easily think that if I attach the service name to Interface, it can solve this problem well, such as ExampleFooRequest and ExampleFooResponse. ServiceName often directly corresponds to a table, making conflicts less likely, and methods within a single service are not allowed to conflict.

So we can have code like this:

Promise.all(
  project.addSourceFilesAtPaths(`src/**/**/*.ts`).map((sourceFile) => {
    const pathArr = sourceFile.getFilePath().replace(src, "").split("/");
    // Only process the last layer
    if (pathArr.length <= 3) {
      return;
    }
    const interfaceDeclarations = [
      ...sourceFile.getInterfaces(),
      ...sourceFile.getEnums(),
    ];
    interfaceDeclarations.forEach((interfaceDeclaration) => {
      if (!interfaceDeclaration) {
        return;
      }
      const oldInterfaceName = interfaceDeclaration.getName();
      if (!oldInterfaceName) {
        return;
      }
      const newServiceName = getFirstUpperCase(
        sourceFile.getBaseName()
      ).replace(".ts", "");
      const newInterfaceName = `${newServiceName}${getFirstUpperCase(
        oldInterfaceName
      )}`;
      interfaceDeclaration.rename(newInterfaceName);
    });
    sourceFile.formatText();
    return sourceFile.save();
  })
);

After completing this step, we can get code like the following:

// example.ts
export interface ExampleFooRequest {
  foo?: string | undefined;
}

export interface ExampleFooResponse {
  bar?: string | undefined;
}

// If necessary, you can decide whether to skip the service based on whether a method exists
export interface ExampleExample {
  foo(request: FooRequest): Promise<FooResponse>;
}

// This is a variable and will not be retrieved by getInterfaces
export const getExampleFoo = (query: ExampleFooRequest, params: {} = {}) =>
  ({
    resp: ExampleFooResponse,
    path: "/example/foo",
    method: "GET",
    query: query,
    params,
  } as const);

Export Declaration

If we feel that it's troublesome to import the methods we need from multi-layer files each time, we can consider exporting named exports in the root directory's index.ts for convenience.

const indexFilePaths = glob.sync(`${DIST_PATH}/*.ts`);
// Prioritize processing the lower layers, i.e., those with longer lengths, so that the last processed is index.ts
indexFilePaths.sort((a, b) => {
  const aLength = a.replace(DIST_PATH, "").split(".").length;
  const bLength = b.replace(DIST_PATH, "").split(".").length;
  return bLength - aLength;
});
Promise.all(
  indexFiles.map((sourceFile) => {
    const exportDeclarations = sourceFile.getExportDeclarations();
    const moduleExportsMap = new Map();
    exportDeclarations.forEach((exportDeclaration) => {
      // Check if it is an export * from statement
      if (exportDeclaration.isNamespaceExport()) {
        const moduleSpecifier = exportDeclaration.getModuleSpecifierValue();
        const exportedDeclarations = project
          .getSourceFileOrThrow(join(DIST_PATH, moduleSpecifier) + ".ts")
          .getExportedDeclarations();
        // Replace with specific exports
        if (!moduleExportsMap.has(moduleSpecifier)) {
          moduleExportsMap.set(moduleSpecifier, []);
        }
        for (const [name] of exportedDeclarations) {
          moduleExportsMap.get(moduleSpecifier).push(name);
        }
        exportDeclaration.remove();
      }
    });
    moduleExportsMap.forEach((exports, moduleSpecifier) => {
      if (exports.length > 0) {
        sourceFile.addExportDeclaration({
          namedExports: exports,
          moduleSpecifier: moduleSpecifier,
        });
      }
    });
    sourceFile.formatText();
    return sourceFile.save();
  })
);

In this way, we can directly export methods and variable names with uniqueness through index.ts.

Protobuf To Http Client Api