首页 注册 登录
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
广告
ice2016
V2EX  ›  Kubernetes

kube-prometheus-0.7 部署出错问题求助

  •   
  •   ice2016 · 2021-08-31 11:27:23 +08:00 · 2047 次点击
    这是一个创建于 1179 天前的主题，其中的信息可能已经有所发展或是发生改变。

    hUdgY9.png 系统环境是 CentOS7.9 + K8s v1.19.14 ，根据官方兼容性说明下载安装的是 kube-prometheus-release-0.7 版本，老是报错，安装包是 https://github.com/prometheus-operator/kube-prometheus/archive/refs/heads/release-0.7.zip

    cd /root/kube-prometheus-release-0.7/manifests
kubectl create -f setup

    [root@k8s-master manifests]# kubectl get pod -o wide -n monitoring
NAME                                   READY   STATUS             RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
prometheus-operator-7649c7454f-q9m29   1/2     CrashLoopBackOff   6          10m   10.201.1.58   k8s-node1   <none>           <none>

    查看日志：

    [root@k8s-master manifests]# kubectl logs prometheus-operator-7649c7454f-q9m29 --all-containers -n monitoring
level=info ts=2021-08-31T03:16:07.410585761Z caller=main.go:235 msg="Starting Prometheus Operator" version="(version=0.44.1, branch=refs/tags/pkg/apis/monitoring/v0.44.1, revision=d8b7d3766225908d0239fd0d78258892cd0fc384)"
level=info ts=2021-08-31T03:16:07.410684044Z caller=main.go:236 build_context="(go=go1.14.12, user=simonpasquier, date=20201209-17:17:30)"
ts=2021-08-31T03:16:07.415831751Z caller=main.go:107 msg="Starting insecure server on [::]:8080"
ts=2021-08-31T03:16:10.422020132Z caller=main.go:422 msg="Unhandled error received. Exiting..." err="communicating with server failed: Get \"https://10.96.0.1:443/version?timeout=32s\": dial tcp 10.96.0.1:443: connect: connection timed out"
I0831 03:10:09.496650       1 main.go:190] Valid token audiences:
I0831 03:10:09.496764       1 main.go:262] Generating self signed cert as no cert is provided
I0831 03:10:10.447004       1 main.go:311] Starting TCP socket on :8443
I0831 03:10:10.447347       1 main.go:318] Listening securely on :8443

    根据报错测试几个节点上 10.96.0.1:443 这个地址都可以访问

    [root@k8s-master manifests]# curl -k https://10.96.0.1:443/version?timeout=32s
{
  "major": "1",
  "minor": "19",
  "gitVersion": "v1.19.14",
  "gitCommit": "0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c",
  "gitTreeState": "clean",
  "buildDate": "2021-08-11T18:02:17Z",
  "goVersion": "go1.15.15",
  "compiler": "gc",
  "platform": "linux/amd64"

    iptables 信息：

    [root@k8s-master manifests]# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
KUBE-FORWARD  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
DOCKER-USER  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER-ISOLATION-STAGE-1  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  10.244.0.0/16        0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            10.244.0.0/16

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER (1 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
DROP       all  -- !127.0.0.0/8          127.0.0.0/8          /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination

Chain KUBE-PROXY-CANARY (0 references)
target     prot opt source               destination

Chain KUBE-SERVICES (3 references)
target     prot opt source               destination

    环境信息如下：

    [root@k8s-master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.14", GitCommit:"0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c", GitTreeState:"clean", BuildDate:"2021-08-11T18:07:41Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.14", GitCommit:"0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c", GitTreeState:"clean", BuildDate:"2021-08-11T18:02:17Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

[root@k8s-master ~]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

[root@k8s-master ]# kubectl get node
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   10d   v1.19.14
k8s-node1    Ready    <none>   10d   v1.19.14
k8s-node2    Ready    <none>   10d   v1.19.14
k8s-node3    Ready    <none>   10d   v1.19.14
  • all
  • prot
  • opt
  • 0.0.0.0/0
    6 条回复  •  2021-09-03 08:57:29 +08:00
    defunct9
        1
    defunct9  
       2021-08-31 11:34:15 +08:00
    开 ssh，让我上去看看
    vemier
        2
    vemier  
       2021-08-31 12:31:07 +08:00
    @defunct9 老哥又来了
    nino789pzw
        3
    nino789pzw  
       2021-08-31 13:39:46 +08:00
    10.96.0.1 是什么的 ip ？
    你的 curl 是 insecure 的 command，ssh 到 Promethus 的 pod 上 ssh 到 10.96.0.1 试试？
    ice2016
        4
    ice2016  
    OP
       2021-08-31 21:10:10 +08:00
    @nino789pzw 把网络插件由 flannel 换成 calico 以后，正常部署了，也不报错，但是打开 svc
    对外暴露端口以后，一会正常一会就打不开了，过会又正常，，
    感觉哪里不对。。
    SIGEV13
        5
    SIGEV13  
       2021-08-31 22:53:19 +08:00
    老哥去开个 issue， 看看社区有没有懂得。
    ethangogogo
        6
    ethangogogo  
       2021-09-03 08:57:29 +08:00
    暴露端口之后有报错吗？
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   4118 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 26ms · UTC 04:10 · PVG 12:10 · LAX 20:10 · JFK 23:10
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.