The recent hot open source project go-zero is an integrated variety of engineering practices including Web and RPC protocols for a full-featured microservices framework, today we will analyze the RPC part of zRPC github.com/zeromicro/go-zero/tree/master/zrpc).
The underlying dependency of zRPC on gRPC has built-in modules for service registration, load balancing, and interceptors, which also include adaptive load dropping, adaptive fusion, flow limiting, and other microservice governance schemes, making it an easy-to-use enterprise-class RPC framework that can be used directly in production.
zRPC supports both direct connection and etcd-based service discovery. We use etcd-based service discovery as an example to demonstrate the basic use of zRPC.
Create the hello.yaml configuration file with the following configuration.
Create the hello.proto file and generate the corresponding go code
Start the service and see if the service is registered:
Run the client to see the output.
This example demonstrates the basic use of zRPC, and you can see that building an RPC service with zRPC is very simple and requires only a few lines of code, so let's continue our exploration
The following figure shows the architecture diagram and main components of zRPC
zRPC consists of the following main modules.
- discov: service discovery module, which implements the service discovery function based on etcd
- resolver: service registration module, which implements the resolver.Builder interface of gRPC and registers to gRPC
- interceptor: interceptor, intercept request and response processing
- balancer: load balancer module that implements the p2c load balancing algorithm and is registered to gRPC
- client: zRPC client, responsible for initiating requests
- server: zRPC server, responsible for processing requests
The main components of zRPC and the main functions of each module are introduced here, among which the resolver and balancer modules implement the open interfaces of gRPC and realize custom resolver and balancer. The interceptor module is the focus of the entire zRPC functionality.
gRPC provides interceptor function, mainly for additional processing before and after the request interception operation, which interceptor contains client-side interceptor and server-side interceptor, and is divided into a unary interceptor and stream (Stream) interceptor, here we mainly explain the unary interceptor, stream interceptor the same.
The client-side interceptor is defined as follows:
where method is the method name, req, reply are the request and response parameters, cc is the client connection object, invoker parameter is the real execution of the rpc method handler is actually called in the interceptor execution
The server-side interceptor is defined as follows:
req is the request parameter, info contains the request method properties, and handler is the wrapper for the server-side method, which is also called in the interceptor.
zRPC has a rich set of built-in interceptors, including adaptive dowload, adaptive fusion, permission validation, prometheus metrics collection, etc. Due to the large number of interceptors, space is limited to analyze all the interceptors one by one, here we mainly analyze two, adaptive fusion and prometheus service monitoring metrics collection.
When the client initiates a request to the server, the client will record the error returned by the server, and when the error reaches a certain percentage, the client will fuse the process itself, discarding a certain percentage of requests to protect downstream dependencies, and can automatically recover. zRPC adaptive fusion follows the [Google SRE](https://landing.google. com/sre/sre-book/chapters/handling-overload) with the following algorithm for overload protection.
requests: total number of requests
accepts: number of normal requests
K: multiplier (Google SRE recommends 2)
The aggressiveness of the fusion can be modified by changing the value of K. Decreasing the value of K will make the adaptive fusion algorithm more aggressive, and increasing the value of K will make it less aggressive.
The accept method implements the Google SRE overload protection algorithm to determine whether to fuse
doReq method first determine whether the fuse, meet the conditions directly return error (circuit breaker is open), does not meet the conditions of the request count to accumulate
Service monitoring is an important tool to understand the current operation status of the service and the trend of changes, monitoring relies on the collection of service metrics, the collection of monitoring metrics through prometheus is the mainstream solution in the industry, zRPC also uses prometheus to collect the metrics
This interceptor mainly collects the monitoring metrics of the service, here it mainly collects the time consumption and invocation errors of RPC methods, here it mainly uses Prometheus' Histogram and Counter data types
In addition to the rich built-in interceptors, zRPC also supports the addition of custom interceptors
The Client side adds a one-dimensional interceptor via the AddInterceptor method.
The Server side adds monadic interceptors via the AddUnaryInterceptors method:
zRPC service registration architecture diagram. ：
The resolver module is customized in zRPC to implement the service registration function. zRPC relies on gRPC at the bottom, and to customize resolver in gRPC you need to implement the resolver.Builder interface.
Where the Build method returns the Resolver, the Resolver is defined as follows:
There are two types of resolver defined in zRPC, direct and discov, here we mainly analyze discov based on etcd to do service discovery, custom resolver needs to be registered through the Register method provided by gRPC code as follows:
When we start our zRPC Server, we call the Start method, which registers the corresponding service address in etcd as follows.
When we start the zRPC client, the Build method of our custom resolver is called inside gRPC. zRPC executes the UpdateState method of resolver.ClientConn by calling within the Build method, which registers the service address inside the gRPC client.
In discov, all addresses of the specified service are retrieved from etcd by calling the load method at:
and listens for changes in the service address via watch:
This part mainly introduces how to customize the resolver in zRPC, as well as the principle of service discovery based on etcd, through this part of the introduction you can understand the principle of service registration and discovery inside zRPC, the source code is more just a rough analysis of the entire process, if you are more interested in the source code of zRPC can learn on their own
Load balancing schematic.
Avoiding overload is an important indicator of a load balancing strategy, and a good load balancing algorithm can balance server-side resources well. The commonly used load balancing algorithms are Rotation, Random, Hash, Weighted Rotation, etc. However, in order to cope with various complex scenarios, simple load balancing algorithms often do not perform well enough, such as the round robin algorithm when the service response time becomes longer, it is easy to cause the load to stop balancing, so the default load balancing algorithm P2C (Power of Two Choices) is customized in zRPC, similar to resolver, in order to customize balancer also needs to Builder interface defined by gRPC. Since it is similar to resolver, we will not take you through the analysis of how to customize the balancer.
Note that zRPC is a client-side load balancing, common and through the nginx intermediate proxy way
The default load balancing algorithm in the zRPC framework is P2C, and the main idea of this algorithm is to
- do two random selection operations from the list of available nodes to get nodes A and B
- compare the two nodes A and B and select the node with the lowest load as the selected node
The pseudo code is as follows.
The main algorithm logic is implemented in the Pick method.
The choose method compares the loads of randomly selected nodes to determine which node to choose.
The above mainly introduces the design idea and code implementation of the default load balancing algorithm of zRPC, how the custom balancer is registered to gRPC, the resolver provides the Register method to register, the same balancer also provides the Register method to register.
How does gRPC know which balancer to use after registering a balancer? Here we need to use the configuration item to configure, through the grpc.WithBalancerName method at the time of NewClient.
This part mainly introduces the implementation principle of the load balancing algorithm in zRPC and the specific implementation method, and then introduces how zRPC registers the custom balancer and how to choose the custom balancer, through this part you should have a further understanding of load balancing
First, the basic usage of zRPC is introduced. You can see that zRPC is very simple to use, and only a few lines of code are needed to build a high-performance RPC service with service governance capabilities.
Next, we introduce several important modules of zRPC and their implementation principles, and analyze some to the source code. The interceptor module is the focus of zRPC, which has a rich set of built-in functions, such as fusion, monitoring, load reduction, etc., which are essential for building highly available microservices. The customization of the load balancing algorithm is no longer a mystery.
Finally, zRPC is an RPC framework that has undergone various engineering practices, and is a rare open source project, whether you want to use it for production or learn the design patterns. We hope you can learn more about zRPC through this article.