Prometheus和Grafana监控Mysql服务器性能以及不显示Dashboard和No datapoints found解决方案

Prometheus和Grafana监控Mysql服务器性能

Mysql监控这个话题相信DBA和运维是非常熟悉的，监控的方式也分为交互式和非交互式，工具也有非常多，比较常用的有Zabbix，Lepus，Nagios，PMM（需要docker运行）等，各种工具都有自身的特点在这里就不多分析了，具体可以自行了解，这里我们采用的是与PMM类似的监控体系：Prometheus+Grafana

具体的安装过程和配置可以参考： 使用Prometheus和Grafana监控Mysql服务器性能以及prometheus

prometheus：prometheus

grafana-dashboards:grafana-dashboards

mysqld_exporter:mysqld_exporter

node_exporter]:node_exporter

所有内容下载地址：https://prometheus.io/或者百度云盘 密码：nh70

这里说明下本人在安装过程中遇到的问题以及配置文件的说明

1、Error pinging mysqld: Error 1045: Access denied for user ‘prom’@’::1’ (using password: YES) source=”mysqld_exporter.go:268”

这里很明显啊数据库连接失败了，看下数据库该用户的配置修改下就可以，我是按照icyfire创建的用户，但是我的监控监控Client组件和监控Server都在一个机器上所以会有这种问题，要是部署在不同机器上应该不会有这个问题。

2、Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory

按照icyfire的教程安装完Grafana后不显示Dashboard:



点击Dashboard展示出来的只有Home一项（截图是处理好之后的），其他的全部没有也就展示不出来任何数据，明显就是Dashboard没有加载出来，但是按照



这种配置后明显就是哪里配置不对，首先想到确认下是不是配置信息写错了，

#################################### Dashboard JSON files ################
[dashboards.json]
enabled = true
path = /var/lib/grafana/dashboards

发现没错。打开grafana-server开启控制台看是不是报错了，果然报错，信息如下：

[root@localhost bin]# ./grafana-server
INFO[01-25|10:54:59] Starting Grafana                         logger=main version=4.3.1 commit=befc15c compiled=2017-05-23T21:50:22+0800
INFO[01-25|10:54:59] Config loaded from                       logger=settings file=/usr/local/services/grafana/conf/defaults.ini
INFO[01-25|10:54:59] Path Home                                logger=settings path=/usr/local/services/grafana
INFO[01-25|10:54:59] Path Data                                logger=settings path=/usr/local/services/grafana/data
INFO[01-25|10:54:59] Path Logs                                logger=settings path=/usr/local/services/grafana/data/log
INFO[01-25|10:54:59] Path Plugins                             logger=settings path=/usr/local/services/grafana/data/plugins
INFO[01-25|10:54:59] Initializing DB                          logger=sqlstore dbtype=sqlite3
INFO[01-25|10:54:59] Starting DB migration                    logger=migrator
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create migration_log table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create user table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add unique index user.login"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add unique index user.email"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="drop index UQE_user_login - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="drop index UQE_user_email - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Rename table user to user_v1 - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create user table v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index UQE_user_login - v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index UQE_user_email - v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="copy data_source v1 to v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Drop old table user_v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add column help_flags1 to user table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update user table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create temp user table v1-7"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index IDX_temp_user_email - v1-7"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index IDX_temp_user_org_id - v1-7"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index IDX_temp_user_code - v1-7"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index IDX_temp_user_status - v1-7"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update temp_user table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create star table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add unique index star.user_id_dashboard_id"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create org table v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index UQE_org_name - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create org_user table v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index IDX_org_user_org_id - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index UQE_org_user_org_id_user_id - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="copy data account to org"
INFO[01-25|10:54:59] Skipping migration condition not fulfilled logger=migrator id="copy data account to org"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="copy data account_user to org_user"
INFO[01-25|10:54:59] Skipping migration condition not fulfilled logger=migrator id="copy data account_user to org_user"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Drop old table account"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Drop old table account_user"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update org table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update org_user table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create dashboard table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index dashboard.account_id"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add unique index dashboard_account_id_slug"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create dashboard_tag table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add unique index dashboard_tag.dasboard_id_term"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Rename table dashboard to dashboard_v1 - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create dashboard v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index IDX_dashboard_org_id - v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index UQE_dashboard_org_id_slug - v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="copy dashboard v1 to v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="drop table dashboard_v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="alter dashboard.data to mediumtext v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add column updated_by in dashboard - v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add column created_by in dashboard - v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add column gnetId in dashboard"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add index for gnetId in dashboard"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add column plugin_id in dashboard"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add index for plugin_id in dashboard"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add index for dashboard_id in dashboard_tag"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update dashboard table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update dashboard_tag table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create data_source table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index data_source.account_id"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add unique index data_source.account_id_name"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="drop index IDX_data_source_account_id - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="drop index UQE_data_source_account_id_name - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Rename table data_source to data_source_v1 - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create data_source table v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index IDX_data_source_org_id - v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index UQE_data_source_org_id_name - v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="copy data_source v1 to v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Drop old table data_source_v1 #2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add column with_credentials"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add secure json data column"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update data_source table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create api_key table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index api_key.account_id"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index api_key.key"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index api_key.account_id_name"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="drop index IDX_api_key_account_id - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="drop index UQE_api_key_key - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="drop index UQE_api_key_account_id_name - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Rename table api_key to api_key_v1 - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create api_key table v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index IDX_api_key_org_id - v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index UQE_api_key_key - v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index UQE_api_key_org_id_name - v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="copy api_key v1 to v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Drop old table api_key_v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update api_key table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create dashboard_snapshot table v4"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="drop table dashboard_snapshot_v4 #1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create dashboard_snapshot table v5 #2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index UQE_dashboard_snapshot_key - v5"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="alter dashboard_snapshot to mediumtext v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update dashboard_snapshot table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create quota table v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index UQE_quota_org_id_user_id_target - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update quota table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create plugin_setting table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add column plugin_version to plugin_settings"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update plugin_setting table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create session table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Drop old table playlist table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Drop old table playlist_item table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create playlist table v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create playlist item table v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update playlist table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update playlist_item table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="drop preferences table v2"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="drop preferences table v3"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create preferences table v3"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update preferences table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create alert table v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index alert org_id & id "
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index alert state"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index alert dashboard_id"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create alert_notification table v1"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add column is_default"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index alert_notification org_id & name"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update alert table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update alert_notification table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Drop old annotation table v4"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create annotation table v5"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index annotation 0 v3"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index annotation 1 v3"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index annotation 2 v3"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index annotation 3 v3"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="add index annotation 4 v3"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Update annotation table charset"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="Add column region_id to annotation table"
INFO[01-25|10:54:59] Executing migration                      logger=migrator id="create test_data table"
INFO[01-25|10:54:59] Created default admin user: admin
INFO[01-25|10:54:59] Creating json dashboard index for path: /var/lib/grafana/dashboards
INFO[01-25|10:54:59] Starting plugin search                   logger=plugins
WARN[01-25|10:54:59] Plugin dir does not exist                logger=plugins dir=/usr/local/services/grafana/data/plugins
INFO[01-25|10:54:59] Plugin dir created                       logger=plugins dir=/usr/local/services/grafana/data/plugins
INFO[01-25|10:54:59] Initializing Alerting                    logger=alerting.engine
INFO[01-25|10:54:59] Initializing CleanUpService              logger=cleanup
INFO[01-25|10:55:00] Initializing Stream Manager
INFO[01-25|10:55:00] Initializing HTTP Server                 logger=http.server address=0.0.0.0:3000 protocol=http subUrl= socket=
INFO[01-25|10:55:24] Request Completed                        logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=127.0.0.1 time_ms=0 size=29
EROR[01-25|10:55:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|10:56:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|10:57:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|10:58:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|10:59:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|11:00:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|11:01:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|11:02:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|11:03:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|11:04:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|11:05:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|11:06:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|11:07:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory
EROR[01-25|11:08:59] Failed to update dashboard json index lstat /var/lib/grafana/dashboards: no such file or directory

爆了很多error，看了一下大致报错信息，就是没有/var/lib/grafana/dashboards这个目录 no such file or directory，说的白话点就是你配置程序去/var/lib/grafana/dashboards目录下找json文件实际没有这个目录所以找不到，然后检查一看，果然是/var/lib/grafana没有dashboards，json文件就在/var/lib/grafana下，然后将配置信息配置为：

################### Dashboard JSON files ################
[dashboards.json]
enabled = true
path = /var/lib/grafana

顺利加载dashboard，这里就是因为 path = /var/lib/grafana 这个配置导致，大家在安装的时候要注意下，还有就是如果有问题就把所有组件全部用控制台启动，不要在后台启动，这样问题会直接报出来不至于挨个看日志，等没问题了在后台运行，希望对大家有帮助。