Microservices on Go with the Go kit: Introduction
- Transfer
In this article I will describe the use of the Go kit, a set of tools and libraries for creating microservices on Go. This article is an introduction to the Go kit. The first part in my blog, the source code of the examples is available here .
Go is increasingly being chosen to develop modern distributed systems. When you develop a cloud-based distributed system, you may need support for various specific functionalities in your services, such as: various transport protocols ( eg. HTTP lane, gRPC, etc. ) and message encoding formats for them, RPC reliability, logging , tracing, metrics and profiling, interrupting queries, limiting the number of queries, integrating into the infrastructure, and even describing the architecture. Go is a popular language due to its simplicity and “no magic” approaches, so Go packages, for example, the standard library, are already suitable for developing distributed systems more than using a full-fledged framework with a lot of “magic under the hood”. I personally [approx. per. Shiju Varghese ] do not support the use of full frameworks, I prefer to use libraries that give more freedom to the developer. Go kit filled the gap in the Go ecosystem, making it possible to use a set of libraries and packages when creating microservices, which in turn allow the use of good principles for designing individual services in distributed systems.
Introduction to go kit
Go kit is a set of Go packages that facilitate the creation of reliable and supported microservices. Go kit provides libraries for the implementation of various components of a transparent and reliable application architecture, using such layers as: logging, metrics, tracing, restriction and interruption of requests that are required to run microservices on the server. Go kit is good because it is well implemented tools for interacting with different infrastructures, message encoding formats and different transport levels.
In addition to a set of libraries for the development of mircoservices, it provides and encourages the use of good design principles for the architecture of your services. The go kit helps to adhere to the principles of SOLID, the subject-oriented approach (DDD) and the hexagonal architecture proposed by Alistair Cockburn or any other approaches from the architectural principles known as the onion architecture by Jeffrey Palermo and the pure architecture by Robert C. Martin . Although the Go kit was designed as a set of packages for developing microservices, it is also suitable for developing elegant monoliths.
Go kit architecture
Three main levels in the architecture of the application developed using the Go kit are:
- transport level
- endpoint level
- service level
Transport level
When you write microservices for distributed systems, services in them often have to communicate with each other using different transport protocols, such as: HTTP or gRPC, or use pub / sub systems, such as NATS. The transport level in the Go kit is bound to a specific transport protocol (hereinafter referred to as transport). Go kit supports various transport for your service, such as: HTTP, gRPC, NATS, AMQP and Thirft ( you can also develop your own transport protocol for your protocol).). Therefore, services written using the Go kit often emphasize the implementation of a specific business logic, which knows nothing about the transport used, you are free to use different transports for the same service. As an example, one service written on the Go kit can simultaneously provide access to it via HTTP and gRPC.
Endpoints
The endpoint or endpoint is a fundamental building block for services and customers. In the Go kit, the main communication pattern is RPC. Endpoint is presented as a separate RPC method. Each service method in the Go kit is converted to an endpoint, allowing communication between the server and the client in the RCP style. Each endpoint exposes the service method using the Transport layer, which in turn uses various transport protocols, such as HTTP or gRPC. A separate endpoint can be placed outside the service simultaneously with the help of several transports ( approx. HTTP and gRPC at different ports ).
Services
Business logic is implemented in the service layer. Services written with the Go kit are designed as interfaces. The business logic in the service layer contains the main business logic core, which does not need to know anything about the endpoints used or a particular transport protocol, like HTTP or gRPC, or about encoding or decoding requests and responses of various types of messages. This will allow you to stick to pure architecture in services written using the Go kit. Each service method is converted to endpoint using an adapter and exhibited using a specific transport. Thanks to the use of pure architecture, a separate method can be exposed using several transports simultaneously.
Examples
And now let's look at the layers described above on the example of a simple application.
Business logic in the service
Business logic in the service is designed using interfaces. We look at the example of an order in e-commerce:
// Service describes the Order service.type Service interface {
Create(ctx context.Context, order Order) (string, error)
GetByID(ctx context.Context, id string) (Order, error)
ChangeStatus(ctx context.Context, id string, status string) error
}The Order service interface works with the entity of the subject domain Order:
// Order represents an ordertype Order struct {
ID string`json:"id,omitempty"`
CustomerID string`json:"customer_id"`
Status string`json:"status"`
CreatedOn int64`json:"created_on,omitempty"`
RestaurantId string`json:"restaurant_id"`
OrderItems []OrderItem `json:"order_items,omitempty"`
}
// OrderItem represents items in an ordertype OrderItem struct {
ProductCode string`json:"product_code"`
Name string`json:"name"`
UnitPrice float32`json:"unit_price"`
Quantity int32`json:"quantity"`
}
// Repository describes the persistence on order modeltype Repository interface {
CreateOrder(ctx context.Context, order Order) error
GetOrderByID(ctx context.Context, id string) (Order, error)
ChangeOrderStatus(ctx context.Context, id string, status string) error
}Here we implement the Order service interface:
package implementation
import (
"context""database/sql""time""github.com/go-kit/kit/log""github.com/go-kit/kit/log/level""github.com/gofrs/uuid"
ordersvc "github.com/shijuvar/gokit-examples/services/order"
)
// service implements the Order Servicetype service struct {
repository ordersvc.Repository
logger log.Logger
}
// NewService creates and returns a new Order service instancefuncNewService(rep ordersvc.Repository, logger log.Logger)ordersvc.Service {
return &service{
repository: rep,
logger: logger,
}
}
// Create makes an orderfunc(s *service)Create(ctx context.Context, order ordersvc.Order)(string, error) {
logger := log.With(s.logger, "method", "Create")
uuid, _ := uuid.NewV4()
id := uuid.String()
order.ID = id
order.Status = "Pending"
order.CreatedOn = time.Now().Unix()
if err := s.repository.CreateOrder(ctx, order); err != nil {
level.Error(logger).Log("err", err)
return"", ordersvc.ErrCmdRepository
}
return id, nil
}
// GetByID returns an order given by idfunc(s *service)GetByID(ctx context.Context, id string)(ordersvc.Order, error) {
logger := log.With(s.logger, "method", "GetByID")
order, err := s.repository.GetOrderByID(ctx, id)
if err != nil {
level.Error(logger).Log("err", err)
if err == sql.ErrNoRows {
return order, ordersvc.ErrOrderNotFound
}
return order, ordersvc.ErrQueryRepository
}
return order, nil
}
// ChangeStatus changes the status of an orderfunc(s *service)ChangeStatus(ctx context.Context, id string, status string)error {
logger := log.With(s.logger, "method", "ChangeStatus")
if err := s.repository.ChangeOrderStatus(ctx, id, status); err != nil {
level.Error(logger).Log("err", err)
return ordersvc.ErrCmdRepository
}
returnnil
}RPC Endpoint Queries and Answers
Service methods are exposed as RPC endpoints. So we need to determine the types of messages ( note. DTO - data transfer object ) that will be used to send and receive messages through RPC endpoints. Let's now define the structure for the types of requests and responses for RPC endpoints in the Order service:
// CreateRequest holds the request parameters for the Create method.type CreateRequest struct {
Order order.Order
}
// CreateResponse holds the response values for the Create method.type CreateResponse struct {
ID string`json:"id"`
Err error `json:"error,omitempty"`
}
// GetByIDRequest holds the request parameters for the GetByID method.type GetByIDRequest struct {
ID string
}
// GetByIDResponse holds the response values for the GetByID method.type GetByIDResponse struct {
Order order.Order `json:"order"`
Err error `json:"error,omitempty"`
}
// ChangeStatusRequest holds the request parameters for the ChangeStatus method.type ChangeStatusRequest struct {
ID string`json:"id"`
Status string`json:"status"`
}
// ChangeStatusResponse holds the response values for the ChangeStatus method.type ChangeStatusResponse struct {
Err error `json:"error,omitempty"`
}Endpoint Go kit for service methods like RPC endpoint
The core of our business logic is separated from the rest of the code and rendered into the service layer, which is exposed using RPC endpoints, which use the Go kit abstraction called Endpoint.
This is how the endpoint from the go kit looks like:
type Endpoint func(ctx context.Context, request interface{})(response interface{}, err error)As we said above, endpoint is a separate RPC method. Each service method is converted to endpoint.Endpointusing adapters. Let's make the Go kit endpoints for the Order service methods:
import (
"context""github.com/go-kit/kit/endpoint""github.com/shijuvar/gokit-examples/services/order"
)
// Endpoints holds all Go kit endpoints for the Order service.type Endpoints struct {
Create endpoint.Endpoint
GetByID endpoint.Endpoint
ChangeStatus endpoint.Endpoint
}
// MakeEndpoints initializes all Go kit endpoints for the Order service.funcMakeEndpoints(s order.Service)Endpoints {
return Endpoints{
Create: makeCreateEndpoint(s),
GetByID: makeGetByIDEndpoint(s),
ChangeStatus: makeChangeStatusEndpoint(s),
}
}
funcmakeCreateEndpoint(s order.Service)endpoint.Endpoint {
returnfunc(ctx context.Context, request interface{})(interface{}, error) {
req := request.(CreateRequest)
id, err := s.Create(ctx, req.Order)
return CreateResponse{ID: id, Err: err}, nil
}
}
funcmakeGetByIDEndpoint(s order.Service)endpoint.Endpoint {
returnfunc(ctx context.Context, request interface{})(interface{}, error) {
req := request.(GetByIDRequest)
orderRes, err := s.GetByID(ctx, req.ID)
return GetByIDResponse{Order: orderRes, Err: err}, nil
}
}
funcmakeChangeStatusEndpoint(s order.Service)endpoint.Endpoint {
returnfunc(ctx context.Context, request interface{})(interface{}, error) {
req := request.(ChangeStatusRequest)
err := s.ChangeStatus(ctx, req.ID, req.Status)
return ChangeStatusResponse{Err: err}, nil
}
}The endpoint adapter accepts an interface as a parameter for input and converts it into an Go kit abstraction, endpoint.Enpointmaking each individual service method an endpoint. This adapter function does the comparison and type conversion for requests, calls the service method, and returns a response message.
funcmakeCreateEndpoint(s order.Service)endpoint.Endpoint {
returnfunc(ctx context.Context, request interface{})(interface{}, error) {
req := request.(CreateRequest)
id, err := s.Create(ctx, req.Order)
return CreateResponse{ID: id, Err: err}, nil
}
}Setting the service out using HTTP
We created our service and described RPC endpoints for exposing our service methods to the outside. Now we need to publish our service outwards so that other services can call RCP endpoints. To place our service outside, we need to decide on the transport protocol for our service, according to which it will receive requests. Go kit supports various transports, for example HTTP, gRPC, NATS, AMQP and Thrift out of the box.
For example, we use HTTP transport for our service. Go kit package github.com/go-kit/kit/transport/http provides the ability to serve HTTP requests. And the function NewServerfrom the package transport/httpwill create a new http server that will implement http.Handlerand wrap the endpoints provided.
Below is the code that converts the Go kit endpoints to an HTTP transport that serves HTTP requests:
package http
import (
"context""encoding/json""errors""github.com/shijuvar/gokit-examples/services/order""net/http""github.com/go-kit/kit/log"
kithttp "github.com/go-kit/kit/transport/http""github.com/gorilla/mux""github.com/shijuvar/gokit-examples/services/order/transport"
)
var (
ErrBadRouting = errors.New("bad routing")
)
// NewService wires Go kit endpoints to the HTTP transport.funcNewService(
svcEndpoints transport.Endpoints, logger log.Logger,
)http.Handler {
// set-up router and initialize http endpoints
r := mux.NewRouter()
options := []kithttp.ServerOption{
kithttp.ServerErrorLogger(logger),
kithttp.ServerErrorEncoder(encodeError),
}
// HTTP Post - /orders
r.Methods("POST").Path("/orders").Handler(kithttp.NewServer(
svcEndpoints.Create,
decodeCreateRequest,
encodeResponse,
options...,
))
// HTTP Post - /orders/{id}
r.Methods("GET").Path("/orders/{id}").Handler(kithttp.NewServer(
svcEndpoints.GetByID,
decodeGetByIDRequest,
encodeResponse,
options...,
))
// HTTP Post - /orders/status
r.Methods("POST").Path("/orders/status").Handler(kithttp.NewServer(
svcEndpoints.ChangeStatus,
decodeChangeStausRequest,
encodeResponse,
options...,
))
return r
}
funcdecodeCreateRequest(_ context.Context, r *http.Request)(request interface{}, err error) {
var req transport.CreateRequest
if e := json.NewDecoder(r.Body).Decode(&req.Order); e != nil {
returnnil, e
}
return req, nil
}
funcdecodeGetByIDRequest(_ context.Context, r *http.Request)(request interface{}, err error) {
vars := mux.Vars(r)
id, ok := vars["id"]
if !ok {
returnnil, ErrBadRouting
}
return transport.GetByIDRequest{ID: id}, nil
}
funcdecodeChangeStausRequest(_ context.Context, r *http.Request)(request interface{}, err error) {
var req transport.ChangeStatusRequest
if e := json.NewDecoder(r.Body).Decode(&req); e != nil {
returnnil, e
}
return req, nil
}
funcencodeResponse(ctx context.Context, w http.ResponseWriter, response interface{})error {
if e, ok := response.(errorer); ok && e.error() != nil {
// Not a Go kit transport error, but a business-logic error.// Provide those as HTTP errors.
encodeError(ctx, e.error(), w)
returnnil
}
w.Header().Set("Content-Type", "application/json; charset=utf-8")
return json.NewEncoder(w).Encode(response)
}We create http.Handlerusing a function NewServerfrom a packet transport/httpthat provides us with endpoints and query decoding functions (returns value type DecodeRequestFunc func) and encoding responses (for example type EncodeReponseFunc func).
The following are examples DecodeRequestFuncand EncodeResponseFunc:
// For decoding request type DecodeRequestFunc func(context.Context, *http.Request)(request interface{}, err error)// For encoding responsetype EncodeResponseFunc func(context.Context, http.ResponseWriter, interface{})errorStart HTTP server
Finally, we can run our HTTP server to process requests. The function NewServicedescribed above implements an interface http.Handlerthat allows us to run it as an HTTP server:
funcmain() {
var (
httpAddr = flag.String("http.addr", ":8080", "HTTP listen address")
)
flag.Parse()
var logger log.Logger
{
logger = log.NewLogfmtLogger(os.Stderr)
logger = log.NewSyncLogger(logger)
logger = level.NewFilter(logger, level.AllowDebug())
logger = log.With(logger,
"svc", "order",
"ts", log.DefaultTimestampUTC,
"caller", log.DefaultCaller,
)
}
level.Info(logger).Log("msg", "service started")
defer level.Info(logger).Log("msg", "service ended")
var db *sql.DB
{
var err error
// Connect to the "ordersdb" database
db, err = sql.Open("postgres",
"postgresql://shijuvar@localhost:26257/ordersdb?sslmode=disable")
if err != nil {
level.Error(logger).Log("exit", err)
os.Exit(-1)
}
}
// Create Order Servicevar svc order.Service
{
repository, err := cockroachdb.New(db, logger)
if err != nil {
level.Error(logger).Log("exit", err)
os.Exit(-1)
}
svc = ordersvc.NewService(repository, logger)
}
var h http.Handler
{
endpoints := transport.MakeEndpoints(svc)
h = httptransport.NewService(endpoints, logger)
}
errs := make(chan error)
gofunc() {
c := make(chan os.Signal)
signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
errs <- fmt.Errorf("%s", <-c)
}()
gofunc() {
level.Info(logger).Log("transport", "HTTP", "addr", *httpAddr)
server := &http.Server{
Addr: *httpAddr,
Handler: h,
}
errs <- server.ListenAndServe()
}()
level.Error(logger).Log("exit", <-errs)
}Now our service is running and uses the HTTP protocol at the transport level. The same service can be launched using another transport. For example, the service can be exposed to the outside using gRPC or Apache Thrift.
For the introductory article, we have already sufficiently used the Go kit primitives, but it also provides more functionality for creating systems of transparent, reliable patterns, detection of services, load balancing, etc. We will discuss these and other things in the Go kit in the following articles.
Source
The entire source code of the examples can be viewed on GitHub here.
Middlewares in go kit
Go kit predisposes to the use of good principles of system design, such as separation into layers. Isolating service components and endpoints is possible by using Middlewares ( approx. Lane pattern mediator ). Middlewares in the Go kit provide a powerful mechanism by which you can wrap services and endpoints and add functionality (isolated components), such as logging, interrupting requests, limiting the number of requests, load balancing, or distributed tracing.
Below is a picture from the Go kit website , which is depicted as a typical "onion architecture" using Middlewares in the Go kit:
Beware of Spring Boot Mikroservices
Like Go kit, Spring Boot is a toolkit for creating microservices in the Java world. But, unlike the Go kit, Spring Boot is a quite mature framework. Also, many Java developers use Spring Boot to create mirroservices using Java stack with positive feedback from use, some of them believe that microservices are only about using Spring Boot. I see many development teams who misinterpret the use of microservices, that they can be developed only with the help of Spring Boot and OSS Netflix and do not perceive microservices as a template when developing distributed systems.
So keep in mind that using a toolkit such as a go kit or some kind of framework, you are directing your development towards micro-sevris as a design pattern. Although microservices solve a lot of problems with scaling and commands and systems, but it also creates many problems, because the data in systems based on microservices is scattered across various databases, which sometimes create many problems when creating transactional or data queries. It all depends on the problem of the domain and the context of your system. The cool thing is that the Go kit, designed as a tool for creating microservices, was also suitable for creating elegant monoliths that are created with a good design of the architecture of your systems.
And some Go kit functionality, such as interruption and restriction of requests, are also available in service mesh platforms, for example, Istio. So if you use something like Istio to launch your microsecure, you may not need some things from the Go kit, but not everyone will have enough channel width to use the service mesh to create inter-service communication as it adds one level and extra complexity.
PS
The author of the translation may not share the opinion of the author of the original text , this article is translated only for educational purposes for the Russian language community Go.
UPD This is
also the first article in the translation section and I would appreciate any feedback on the translation.
Only registered users can participate in the survey. Sign in , please.