A rule-based log analyzer & filter

Overview

Flog

一个根据规则集来处理文本日志的工具。

前言

在日常开发过程中,由于缺乏必要的日志规范,导致很多人乱打一通,一个日志文件夹解压缩后往往有几十万行。

日志泛滥会导致信息密度骤减,给排查问题带来了不小的麻烦。

以前都是用grep之类的工具先挑选出有用的,再逐条进行排查,费时费力。在忍无可忍之后决定写这个工具,根据规则自动分析日志、剔除垃圾信息。

使用方法

安装

python setup.py install

基础用法

flog -r rules.yaml /path/to/1.log /path/to/2.log /path/to/3.log -o /path/to/filtered.log

其中:

  • rules.yaml是规则文件
  • /path/to/x.log是原始的日志文件,支持一次输入多个日志文件。
  • /path/to/filtered.log是过滤后的日志文件,如果不指定文件名(直接一个-o),会自动生成一个。

如果不需要过滤日志内容,只需显示分析结果,可以直接:

flog -r rules.yaml /path/to/your.log

规则语法

基础

name: Rule Name #规则集名称
patterns: #规则列表
  # 单行模式,如果匹配到 ^Hello,就输出 Match Hello
  - match: "^Hello"
    message: "Match Hello"
    action: bypass #保留此条日志(会输出到-o指定的文件中)
    
  # 多行模式,以^Hello开头,以^End结束,输出 Match Hello to End,并丢弃此条日志
  - start: "^Hello"
    end: "^End"
    message: "Match Hello to End"
    action: drop

  - start: "Start"
    start_message: "Match Start" #匹配开始时显示的信息
    end: "End"
    end_messagee: "Match End" #结束时显示的信息

纯过滤模式

name: Rule Name
patterns:
  - match: "^Hello" #删除日志中以Hello开头的行
  - start: "^Hello" #多行模式,删除从Hello到End中间的所有内容
    end: "^End"

过滤日志内容,并输出信息

name: Rule Name
patterns:
  - match: "^Hello" #删除日志中以Hello开头的行
    message: "Match Hello"
    action: drop #删除此行日志

规则嵌套

仅多行模式支持规则嵌套。

name: Rule
patterns:
  - start: "^Response.*{$"
    end: "^}"
    patterns:
      - match: "username = (.*)"
        message: "Current user: {{ capture[0] }}"

输入:

Login Response {
  username = zorro
  userid = 123456
}

输出:

Current user: zorro

action

action字段主要用于控制是否过滤此条日志,仅在指定 -o 参数后生效。 取值范围:【dropbypass】。

为了简化纯过滤类型规则的书写,action默认值的规则如下:

  • 如果规则中包含messagestart_messageend_message字段,action默认为bypass,即输出到文件中。
  • 如果规则中不包含message相关字段,action默认为drop,变成一条纯过滤规则。

message

message 字段用于在标准输出显示信息,并且支持 Jinja 模版语法来自定义输出信息内容,通过它可以实现一些简单的日志分析功能。

目前支持的参数有:

  • lines: (多行模式下)匹配到的所有行
  • content: 匹配到的日志内容
  • captures: 正则表达式(match/start/end)捕获的内容

例如:

name: Rule Name
patterns:
  - match: "^Hello (.*)"
    message: "Match {{captures[0]}}"

如果遇到:"Hello lilei",则会在终端输出"Match lilei"

context

可以把日志中频繁出现的正则提炼出来,放到context字段下,避免复制粘贴多次,例如:

name: Rule Name

context:
  timestamp: "\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}.\\d{3}"
patterns:
  - match: "hello ([^:]*):"
    message: "{{ timestamp }} - {{ captures[0] }}"

输入:2022-04-08 16:52:37.152 hello world: this is a test message
输出:2022-04-08 16:52:37.152 - world

高亮

内置了一些 Jinjafilter,可以在终端高亮输出结果,目前包含:

black, red, green, yellow, blue, purple, cyan, white, bold, light, italic, underline, blink, reverse, strike

例如:

patterns:
  - match: "Error: (.*)"
    message: "{{ captures[0] | red }}"

输入:Error: file not found
输出:file not found

include

支持引入其它规则文件,例如:

name: Rule
include: base #引入同级目录下的 base.yaml 或 base.yml

include支持引入一个或多个文件,例如:

name: Rule
include:
  - base
  - ../base
  - base.yaml
  - base/base1
  - base/base2.yaml
  - ../base.yaml
  - /usr/etc/rules/base.yml

contextpatterns会按照引用顺序依次合并,如果有同名的context,后面的会替换之前的。

License

MIT

Owner
上山打老虎
专业造工具
上山打老虎
This repository is for Contrastive Embedding Distribution Refinement and Entropy-Aware Attention Network (CEDR)

CEDR This repository is for Contrastive Embedding Distribution Refinement and Entropy-Aware Attention Network (CEDR) introduced in the following paper

phoenix 3 Feb 27, 2022
Using pretrained GROVER to extract the atomic fingerprints from molecule

Extracting atomic fingerprints from molecules using pretrained Graph Neural Network models (GROVER).

Xuan Vu Nguyen 1 Jan 28, 2022
An open source library for face detection in images. The face detection speed can reach 1000FPS.

libfacedetection This is an open source library for CNN-based face detection in images. The CNN model has been converted to static variables in C sour

Shiqi Yu 11.4k Dec 27, 2022
Minimal PyTorch implementation of YOLOv3

A minimal PyTorch implementation of YOLOv3, with support for training, inference and evaluation.

Erik Linder-Norén 6.9k Dec 29, 2022
Evaluating Privacy-Preserving Machine Learning in Critical Infrastructures: A Case Study on Time-Series Classification

PPML-TSA This repository provides all code necessary to reproduce the results reported in our paper Evaluating Privacy-Preserving Machine Learning in

Dominik 1 Mar 08, 2022
Welcome to The Eigensolver Quantum School, a quantum computing crash course designed by students for students.

TEQS Welcome to The Eigensolver Quantum School, a crash course designed by students for students. The aim of this program is to take someone who has n

The Eigensolvers 53 May 18, 2022
Full-featured Decision Trees and Random Forests learner.

CID3 This is a full-featured Decision Trees and Random Forests learner. It can save trees or forests to disk for later use. It is possible to query tr

Alejandro Penate-Diaz 3 Aug 15, 2022
Generative Modelling of BRDF Textures from Flash Images [SIGGRAPH Asia, 2021]

Neural Material Official code repository for the paper: Generative Modelling of BRDF Textures from Flash Images [SIGGRAPH Asia, 2021] Henzler, Deschai

Philipp Henzler 80 Dec 20, 2022
ImageNet Adversarial Image Evaluation

ImageNet Adversarial Image Evaluation This repository contains the code and some materials used in the experimental work presented in the following pa

Utku Ozbulak 11 Dec 26, 2022
Code release for "MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound"

merlot_reserve Code release for "MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound" MERLOT Reserve (in submission) is a mo

Rowan Zellers 92 Dec 11, 2022
Code repo for "FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation" (ICCV 2021)

FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation (ICCV 2021) This repository contains the implementation of th

Yuhang Zang 21 Dec 17, 2022
TensorFlow Tutorial and Examples for Beginners (support TF v1 & v2)

TensorFlow Examples This tutorial was designed for easily diving into TensorFlow, through examples. For readability, it includes both notebooks and so

Aymeric Damien 42.5k Jan 08, 2023
Pytorch and Keras Implementations of Hyperspectral Image Classification -- Traditional to Deep Models: A Survey for Future Prospects.

The repository contains the implementations for Hyperspectral Image Classification -- Traditional to Deep Models: A Survey for Future Prospects. Model

Ankur Deria 115 Jan 06, 2023
Video Instance Segmentation with a Propose-Reduce Paradigm (ICCV 2021)

Propose-Reduce VIS This repo contains the official implementation for the paper: Video Instance Segmentation with a Propose-Reduce Paradigm Huaijia Li

DV Lab 39 Nov 23, 2022
The official implementation of ICCV paper "Box-Aware Feature Enhancement for Single Object Tracking on Point Clouds".

Box-Aware Tracker (BAT) Pytorch-Lightning implementation of the Box-Aware Tracker. Box-Aware Feature Enhancement for Single Object Tracking on Point C

Kangel Zenn 5 Mar 26, 2022
Transformer in Vision

Transformer-in-Vision Recent Transformer-based CV and related works. Welcome to comment/contribute! Keep updated. Resource SCENIC: A JAX Library for C

Yong-Lu Li 1.1k Dec 30, 2022
Code of Adverse Weather Image Translation with Asymmetric and Uncertainty aware GAN

Adverse Weather Image Translation with Asymmetric and Uncertainty-aware GAN (AU-GAN) Official Tensorflow implementation of Adverse Weather Image Trans

Jeong-gi Kwak 36 Dec 26, 2022
An efficient and easy-to-use deep learning model compression framework

TinyNeuralNetwork 简体中文 TinyNeuralNetwork is an efficient and easy-to-use deep learning model compression framework, which contains features like neura

Alibaba 441 Dec 25, 2022
[NeurIPS'20] Multiscale Deep Equilibrium Models

Multiscale Deep Equilibrium Models 💥 💥 💥 💥 This repo is deprecated and we will soon stop actively maintaining it, as a more up-to-date (and simple

CMU Locus Lab 221 Dec 26, 2022
An API-first distributed deployment system of deep learning models using timeseries data to analyze and predict systems behaviour

Gordo Building thousands of models with timeseries data to monitor systems. Table of content About Examples Install Uninstall Developer manual How to

Equinor 26 Dec 27, 2022