【Prometheus】node

Docker安装node-exporter的命令如下：

docker run -d --name node_exporter --restart=always -p 9100:9100 prom/node-exporter

查看启动日志：

docker logs cf

ts=2022-05-10T22:39:29.681Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)"
ts=2022-05-10T22:39:29.681Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)"
ts=2022-05-10T22:39:29.688Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)
ts=2022-05-10T22:39:29.688Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2022-05-10T22:39:29.691Z caller=node_exporter.go:108 level=info msg="Enabled collectors"
ts=2022-05-10T22:39:29.691Z caller=node_exporter.go:115 level=info collector=arp
ts=2022-05-10T22:39:29.691Z caller=node_exporter.go:115 level=info collector=bcache
ts=2022-05-10T22:39:29.691Z caller=node_exporter.go:115 level=info collector=bonding
ts=2022-05-10T22:39:29.691Z caller=node_exporter.go:115 level=info collector=btrfs
ts=2022-05-10T22:39:29.691Z caller=node_exporter.go:115 level=info collector=conntrack
ts=2022-05-10T22:39:29.691Z caller=node_exporter.go:115 level=info collector=cpu
ts=2022-05-10T22:39:29.691Z caller=node_exporter.go:115 level=info collector=cpufreq
ts=2022-05-10T22:39:29.691Z caller=node_exporter.go:115 level=info collector=diskstats
ts=2022-05-10T22:39:29.691Z caller=node_exporter.go:115 level=info collector=dmi
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=edac
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=entropy
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=fibrechannel
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=filefd
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=filesystem
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=hwmon
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=infiniband
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=ipvs
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=loadavg
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=mdadm
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=meminfo
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=netclass
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=netdev
ts=2022-05-10T22:39:29.692Z caller=node_exporter.go:115 level=info collector=netstat
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=nfs
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=nfsd
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=nvme
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=os
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=powersupplyclass
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=pressure
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=rapl
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=schedstat
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=sockstat
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=softnet
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=stat
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=tapestats
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=textfile
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=thermal_zone
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=time
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=timex
ts=2022-05-10T22:39:29.693Z caller=node_exporter.go:115 level=info collector=udp_queues
ts=2022-05-10T22:39:29.694Z caller=node_exporter.go:115 level=info collector=uname
ts=2022-05-10T22:39:29.694Z caller=node_exporter.go:115 level=info collector=vmstat
ts=2022-05-10T22:39:29.694Z caller=node_exporter.go:115 level=info collector=xfs
ts=2022-05-10T22:39:29.694Z caller=node_exporter.go:115 level=info collector=zfs
ts=2022-05-10T22:39:29.694Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100
ts=2022-05-10T22:39:29.697Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false

默认开启的采集项目介绍：黑名单: 关闭某一项默认开启的采集项：

--no-collector.<name>

---------------------------------------------------------------------

docker run -d --name node_exporter --restart=always -p 9100:9100 prom/node-exporter --no-collector.cpu

白名单：关闭默认采集项而只开启某些采集：

--collector.disable-defaults --collector.<name>

---------------------------------------------------------------------

docker run -d --name node_exporter --restart=always -p 9100:9100 prom/node-exporter --web.disable-exporter-metrics --collector.disable-defaults --collector.cpu --collector.meminfo

默认关闭的原因：

太重：High cardinality
太慢：Prolonged runtime that exceeds the Prometheus scrape_interval or scrape_timeout
太多资源开销： Significant resource demands on the host

prometheus sdk指标

promhttp_/metrics

docker run -d --name node_exporter --restart=always -p 9100:9100 prom/node-exporter

go_代表 goruntime 信息等

process_代表进程信息等

禁用golang sdk 指标

使用 --web.disable-exporter-metrics

---------------------------------------------------------------------

docker run -d --name node_exporter --restart=always -p 9100:9100 prom/node-exporter --web.disable-exporter-metrics

节点上自打点数据上报

.prom

docker run -d --name node_exporter --restart=always -p 9100:9100 -v /data/soft/node_exporter/test.prom:/node_exporter/inner_test.prom prom/node-exporter --collector.textfile.directory=/node_exporter

将node_exporter job配置在prometheus中

- job_name: node_exporter
  honor_timestamps: true
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  static_configs:
  - targets:
    - ljxwtl.cn:9100

编辑prometheus配置文件，发送热更新命令

首先赋予/data/soft/prometheus/prometheus.yml这个文件的权限为-rwxr-xrwx.

这样做的目的是为了使容器内对应的文件权限和挂载的文件权限一致。prometheus这个容器默认使的不是root用户，所以要额外给文件加上写权限。

docker run -d --name prometheus --restart=always -p 9090:9090 -v /data/soft/prometheus/prometheus.yml:/prometheus/config/prometheus.yml prom/prometheus --config.file=/prometheus/config/prometheus.yml --storage.tsdb.path=/prometheus --web.console.libraries=/usr/share/prometheus/console_libraries --web.console.templates=/usr/share/prometheus/consoles --web.enable-lifecycle

global:
  scrape_interval:     5s
  evaluation_interval: 5s

scrape_configs:

  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: prometheus
          
  - job_name: node_exporter
    honor_timestamps: true
    scrape_interval: 5s
    scrape_timeout: 5s
    metrics_path: /metrics
    scheme: http
    follow_redirects: true
    static_configs:
    - targets:
      - ljxwtl.cn:9100

curl -vvv -X POST localhost:9090/-/reload

在prometheus中查询 node_exporter的指标

http传入参数，按采集器过滤指标

# 只看cpu采集器的指标
http://ljxwtl.cn:9100/metrics?collect[]=cpu


# 只看cpu和mem采集器的指标
http://ljxwtl.cn:9100/metrics?collect[]=cpu&collect[]=meminfo

prometheus配置参数

  params:
    collect[]:
      - cpu
      - meminfo

总的配置变为：

global:
  scrape_interval:     5s
  evaluation_interval: 5s

scrape_configs:

  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: prometheus

  - job_name: node_exporter
    honor_timestamps: true
    scrape_interval: 5s
    scrape_timeout: 5s
    metrics_path: /metrics
    scheme: http
    follow_redirects: true
    static_configs:
      - targets:
        - ljxwtl.cn:9100
        labels:
          instance: mynode
          mylabel: hello,world
          otherLabel: other
    params:
      collect[]:
        - cpu
        - meminfo