Log Collection

In order to ensure the stable operation of the business and predict the unhealthy risks of the service, the collection of logs can help us observe the current health of the service. In traditional business development, when there are not many machine deployments, we usually log in directly to the server to view and debug logs. However, as the business increases, services continue to be split.

The maintenance cost of the service will also become more and more complicated. In a distributed system, there are more server machines, and the service is distributed on different servers. When problems are encountered, We can't use traditional methods to log in to the server for log investigation and debugging. The complexity can be imagined.

log-flow

tip

If it is a simple single service system, or the service is too small, it is not recommended to use it directly, otherwise it will be counterproductive.

Prepare#

kafka
elasticsearch
kibana
filebeat、Log-Pilot（k8s）
go-stash

Filebeat#

$ vim xx/filebeat.yaml

filebeat.inputs:
- type: log
  enabled: true
  # Turn on json parsing
  json.keys_under_root: true
  json.add_error_key: true
  # Log file path
  paths:
    - /var/log/order/*.log

setup.template.settings:
  index.number_of_shards: 1

# Define kafka topic field
fields:
  log_topic: log-collection

# Export to kafka
output.kafka:
  hosts: ["127.0.0.1:9092"]
  topic: '%{[fields.log_topic]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  keep_alive: 10s

# ================================= Processors =================================
processors:
  - decode_json_fields:
      fields: ['@timestamp','level','content','trace','span','duration']
      target: ""

tip

xx is the path where filebeat.yaml is located

go-stash configuration#

Create a new config.yaml file
Add configuration content

$ vim config.yaml

Clusters:
- Input:
    Kafka:
      Name: go-stash
      Log:
        Mode: file
      Brokers:
      - "127.0.0.1:9092"
      Topics: 
      - log-collection
      Group: stash
      Conns: 3
      Consumers: 10
      Processors: 60
      MinBytes: 1048576
      MaxBytes: 10485760
      Offset: first
  Filters:
  - Action: drop
    Conditions:
      - Key: status
        Value: "503"
        Type: contains
      - Key: type
        Value: "app"
        Type: match
        Op: and
  - Action: remove_field
    Fields:
    - source
    - _score
    - "@metadata"
    - agent
    - ecs
    - input
    - log
    - fields
  Output:
    ElasticSearch:
      Hosts:
      - "http://127.0.0.1:9200"
      Index: "go-stash-{{yyyy.MM.dd}}"
      MaxChunkBytes: 5242880
      GracePeriod: 10s
      Compress: false
      TimeZone: UTC

Start services (start in order)#

Start kafka
Start elasticsearch
Start kibana
Start go-stash
Start filebeat
Start the order-api service and its dependent services (order-api service in the go-zero-demo project)

Visit kibana#

Enter 127.0.0.1:5601 log

tip

Here we only demonstrate the logs generated by logx in the collection service, and log collection in nginx is the same.

tip

Prepare#

Filebeat#

tip

go-stash configuration#

Start services (start in order)#

Visit kibana#

tip

Reference#