V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
Distributions
Ubuntu
Fedora
CentOS
中文资源站
网易开源镜像站
LxnChan
V2EX  ›  Linux

请问有没有离线的 nginx 日志分析的工具

  •  
  •   LxnChan ·
    lxnchan · 2023-05-02 10:05:57 +08:00 · 2676 次点击
    这是一个创建于 575 天前的主题,其中的信息可能已经有所发展或是发生改变。

    由于我的网站访问人数越来越多,nginx 的日志大小也在指数级增长,想问一下大家有没有 nginx 日志分析的工具,即我将 nginx 日志复制出来,然后通过该工具进行数据的相关分析(错误类型、各地区 IP 访问量等)

    14 条回复    2023-05-04 10:50:38 +08:00
    hasdream
        1
    hasdream  
       2023-05-02 10:08:57 +08:00
    goaccess
    julyclyde
        2
    julyclyde  
       2023-05-02 10:36:23 +08:00
    如果真的是指数级那估计是被攻击了
    应该线性增长才对啊
    seers
        3
    seers  
       2023-05-02 10:43:30 +08:00
    一般都是用 prometheus
    bjzhush
        4
    bjzhush  
       2023-05-02 10:46:12 +08:00
    不用这么麻烦,直接 awk sort 几个组合一下,按 IP 排序访问数量就知道了
    tonlmy
        5
    tonlmy  
       2023-05-02 10:46:23 +08:00
    logstalgia
    LxnChan
        6
    LxnChan  
    OP
       2023-05-02 10:51:53 +08:00
    @hasdream @seers @tonlmy 谢谢,稍后我会逐个尝试。
    @julyclyde 就是因为 nginx 日志异常增长且量大没法手动分析了才打算找的分析工具🤣,我现在也不是很清楚是被打了还是真的有那么高的访问量
    PolarBears
        7
    PolarBears  
       2023-05-02 11:50:24 +08:00
    默认格式就 goaccess 吧,如果有条件的话还是建议配置 nginx 日志格式为 json 格式然后丢 elasticsearch 上分析吧
    tiga99
        8
    tiga99  
       2023-05-02 17:32:37 +08:00
    如果资源足够,可以将日志写到 es ,用 grafana 做展示;参考博客: https://www.xiaoleizhang.com/index.php/archives/120/

    如果资源不足或者不想这么复杂,可以将 nginx 重新编译一下,加上 nginx-module-vts 和 geoip2 模块,大概效果如下:
    ```text
    # HELP nginx_vts_filter_bytes_total The request/response bytes
    # TYPE nginx_vts_filter_bytes_total counter
    # HELP nginx_vts_filter_requests_total The requests counter
    # TYPE nginx_vts_filter_requests_total counter
    # HELP nginx_vts_filter_request_seconds_total The request processing time in seconds counter
    # TYPE nginx_vts_filter_request_seconds_total counter
    # HELP nginx_vts_filter_request_seconds The average of request processing times in seconds
    # TYPE nginx_vts_filter_request_seconds gauge
    # HELP nginx_vts_filter_request_duration_seconds The histogram of request processing time
    # TYPE nginx_vts_filter_request_duration_seconds histogram
    # HELP nginx_vts_filter_cache_total The requests cache counter
    # TYPE nginx_vts_filter_cache_total counter
    nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="unknow",direction="in"} 7332314
    nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="unknow",direction="out"} 504487933
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="1xx"} 0
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="2xx"} 25577
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="3xx"} 6
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="4xx"} 4
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="5xx"} 0
    nginx_vts_filter_request_seconds_total{filter="a.example.com",filter_name="unknow"} 0.000
    nginx_vts_filter_request_seconds{filter="a.example.com",filter_name="unknow"} 0.000
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="miss"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="bypass"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="expired"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="stale"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="updating"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="revalidated"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="hit"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="scarce"} 0
    nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="四川省",direction="in"} 3647380
    nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="四川省",direction="out"} 5444493
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="1xx"} 0
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="2xx"} 12939
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="3xx"} 0
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="4xx"} 6441
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="5xx"} 0
    nginx_vts_filter_request_seconds_total{filter="a.example.com",filter_name="四川省"} 0.000
    nginx_vts_filter_request_seconds{filter="a.example.com",filter_name="四川省"} 0.000
    nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="0.100"} 19380
    nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="0.200"} 19380
    nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="0.500"} 19380
    nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="1.000"} 19380
    nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="3.000"} 19380
    nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="5.000"} 19380
    nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="10.000"} 19380
    nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="+Inf"} 19380
    nginx_vts_filter_request_duration_seconds_sum{filter="a.example.com",filter_name="四川省"} 0.000
    nginx_vts_filter_request_duration_seconds_count{filter="a.example.com",filter_name="四川省"} 19380
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="miss"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="bypass"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="expired"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="stale"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="updating"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="revalidated"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="hit"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="scarce"} 0
    nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="overseas",direction="in"} 2160
    nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="overseas",direction="out"} 4680
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="1xx"} 0
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="2xx"} 18
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="3xx"} 0
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="4xx"} 0
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="5xx"} 0
    nginx_vts_filter_request_seconds_total{filter="a.example.com",filter_name="overseas"} 0.000
    nginx_vts_filter_request_seconds{filter="a.example.com",filter_name="overseas"} 0.000
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="miss"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="bypass"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="expired"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="stale"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="updating"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="revalidated"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="hit"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="scarce"} 0
    nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="江西",direction="in"} 14156529
    nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="江西",direction="out"} 19690820
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="1xx"} 0
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="2xx"} 74493
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="3xx"} 0
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="4xx"} 1000
    nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="5xx"} 0
    nginx_vts_filter_request_seconds_total{filter="a.example.com",filter_name="江西"} 0.000
    nginx_vts_filter_request_seconds{filter="a.example.com",filter_name="江西"} 0.000
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="miss"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="bypass"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="expired"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="stale"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="updating"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="revalidated"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="hit"} 0
    nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="scarce"} 0
    ```
    davidyin
        9
    davidyin  
       2023-05-02 18:25:24 +08:00 via Android
    Awstat
    eroko
        10
    eroko  
       2023-05-02 19:21:10 +08:00
    我们用的是 loki
    LxnChan
        11
    LxnChan  
    OP
       2023-05-03 11:49:58 +08:00
    @tiga99 这个我看了一下,资源消耗好像真的有点大🤣,不过还是谢谢了
    LxnChan
        12
    LxnChan  
    OP
       2023-05-03 11:51:30 +08:00
    @davidyin @eroko 谢谢,稍后我会逐个尝试。
    changdig
        13
    changdig  
       2023-05-03 22:04:13 +08:00
    我是简单写了个 py 脚本去处理 nginx 日志然后写库或者写 csv 文件
    killva4624
        14
    killva4624  
       2023-05-04 10:50:38 +08:00
    elk 或者 loki ,把日志结构化入库分析。
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   5521 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 24ms · UTC 07:06 · PVG 15:06 · LAX 23:06 · JFK 02:06
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.