ELK Stack生产实践-ES8.8集群与Kibana部署

es可以使用二进制、docker、k8s、rpm方式部署，此处以rpm方式为例。相较于二进制部署，省去了繁琐的用户创建、证书生成、密码设置、启动脚本配置等操作，简化部署流程，可以将更多的精力用于es的使用而不是部署上面。如果资源有限，想体验elk相关功能，可参考文档：https://www.cuiliangblog.cn/detail/section/117075458，后续也会发布docker模式下elk自定义日志采集文章。

系统环境设置

以下操作在所有主机执行

修改主机名添加hosts

# hostnamectl set-hostname es-master

# cat /etc/hosts 
192.168.10.132 es-master
192.168.10.133 es-hot1
192.168.10.134 es-hot2
192.168.10.135 es-hot3
192.168.10.136 es-warm1
192.168.10.137 es-warm2
192.168.10.138 es-cold

关闭防火墙和selinux

systemctl stop firewalld.service
systemctl  disable firewalld
setenforce  0
sed  -i 's/enforcing/disabled/g' /etc/selinux/config
grep  SELINUX= /etc/selinux/config

配置时间同步

dnf -y install chrony
systemctl  start chronyd
systemctl  enable chronyd
timedatectl set-timezone Asia/Shanghai
chronyc sourcestats -v 
date

也可以在内网环境其中一台主机启动chronyd服务，其他主机配置chronyd服务地址，参考文档：https://www.cuiliangblog.cn/detail/section/31516177

系统参数优化

修改文件描述符数目

修改原因

原因1： Elasticsearch 在节点和 HTTP 客户端之间进行通信也使用了大量的套接字（sockets）。所有这一切都需要足够的文件描述符。
原因2：linux系统对每个用户、进程、或整个系统的可打开文件描述符数量都有一个限制，一般默认为1024。这对一个小的 Elasticsearch 节点来说实在是太低了，更不用说一个处理数以百计索引的节点。

设置环境变量

# 修改环境变量文件
vim /etc/profile
ulimit -n 65535
# 使配置生效
source /etc/profile

修改limits.conf配置文件

# 修改limits.conf配置
vim /etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535

验证

# ulimit -n
65535

修改虚拟内存数大小

修改原因

Elasticsearch 对各种文件混合使用了 NioFs（非阻塞文件系统）和 MMapFs （内存映射文件系统）。
请确保配置的最大映射数量，以便有足够的虚拟内存可用于 mmapped 文件

临时设置

# sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144

永久设置

cat >> /etc/sysctl.conf << EOF
vm.max_map_count=262144
EOF
# sysctl -p 
vm.max_map_count = 262144

关闭swap分区

修改原因

关闭 Swap 分区可以提高服务器的性能，因为 Swap 分区通常会在物理内存不足时被使用，这会导致额外的 I/O 操作和延迟。当系统使用物理内存满足所有应用程序的需求时，关闭 Swap 分区可以避免这种情况的发生。

设置

swapoff -a  
sed -i '/ swap / s/^(.*)$/#1/g' /etc/fstab

数据盘挂载

将数据盘格式化后挂载至/data目录。

具体参考文章：https://www.cuiliangblog.cn/detail/section/31508181

ES部署

以下操作除非特殊说明，所有节点均操作。

下载rpm包

国内下载地址：https://elasticsearch.cn/download/

官网下载地址：https://www.elastic.co/cn/downloads/past-releases#elasticsearch

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.8.2-x86_64.rpm

安装elasticsearch

安装 Elasticsearch 时，默认情况下会启用和配置安全功能。安装 Elasticsearch 时，会自动进行以下配置：

启用身份验证和授权，并生成超级用户elastic的密码。
启用TLS 的证书和密钥，为传输层和 HTTP 层加密生成的。

密码、证书和密钥将会显示在终端。

[root@es-master ~]# rpm -ivh elasticsearch-8.8.2-x86_64.rpm 
warning: elasticsearch-8.8.2-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Updating / installing...
   1:elasticsearch-0:8.8.2-1          ################################# [100%]
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : enu7CEdICqQ5oXp5U0BC

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token '
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with 
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with 
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with 
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
/usr/lib/tmpfiles.d/elasticsearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/elasticsearch   /run/elasticsearch; please update the tmpfiles.d/ drop-in file accordingly.

设置为开机自启动

[root@es-master ~]# systemctl daemon-reload
[root@es-master ~]# systemctl enable elasticsearch.service
Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch.service   /usr/lib/systemd/system/elasticsearch.service.

创建数据与日志目录

[root@es-master ~]# mkdir /data/es-log /data/es-data
[root@es-master ~]# chown -R elasticsearch:elasticsearch /data

修改jvm配置

实际生产环境中计算公式：min（机器内存的一半，32GB内存）。也就是说：取机器环境内存的一半和32GB内存之间的小值，jvm配置如下

[root@es-master ~]# cat > /etc/elasticsearch/jvm.options.d/es.options << EOF
-Xms2g
-Xmx2g
EOF

修改es配置

修改es配置时，先拷贝默认配置文件到其他目录下，使用默认配置文件加入集群后，再替换新的配置文件启动es服务，从es8开始需要使用token注册集群才能正常启动。

在配置master节点时，先不要设置node.roles，因为在集群初始化过程中需要写入集群元数据索引，如果集群节点为master无法写入数据，集群状态为red，无法生成token，其他节点无法加入集群。

配置文件说明

配置项	说明
cluster.name	集群名称，唯一确定一个集群。
cluster.initial_master_nodes	用于初始化第一次时选举 master 节点，填写 node name或者node ip
node.name	节点名称，一个集群中的节点名称是唯一固定的，不同节点不能同名。
node.master	主节点属性值
node.data	数据节点属性值
network.host	本节点的ip
http.port	本节点的http端口
transport.port	集群之间通信的端口，若不指定默认：9300
path.logs	修改日志目录
path.data	修改数据目录
discovery.seed_hosts	节点发现需要配置一些种子节点，与7.X之前老版本：disvoery.zen.ping.unicast.hosts类似，一般配置集群中的全部节点

master节点配置

[root@es-master ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: es-cluster
node.name: es-master
# node.roles: [ master, ingest ] # 先不设置集群角色，待集群全部加入后再设置
path.data: /data/es-data
path.logs: /data/es-log
network.host: 192.168.10.100 # master节点IP
# 填写集群ip或主机名列表
discovery.seed_hosts: ["es-master", "es-hot1", "es-hot2", "es-hot2", "es-warm1", "es-warm2", "es-cold"]

xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

cluster.initial_master_nodes: ["es-master"]  # master节点IP或主机名
http.host: 0.0.0.0

启动master节点服务并生成token

以下操作在es-master节点执行

[root@es-master ~]# systemctl start elasticsearch

生成集群注册token

[root@es-master ~]# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node
eyJ2ZXIiOiI4LjguMiIsImFkciI6WyIxOTIuMTY4LjguNTo5MjAwIl0sImZnciI6IjdmMTlkZjk5OGQ1NTRjOTJlY2I1MDVkYjZkYWUwOTU0ZDc0ZjNjNDdlZjU2Yjc4YjQ2YjY1NjYyNTU4Yjk4OGMiLCJrZXkiOiIxSG5iZDRrQmRoRkRQY2VFY0xLcjpMeTVQZ3dlZVNtT1BYTnZWS0hqWG53In0=

其他节点加入集群

以下操作在其他剩余节点执行

使用默认配置文件执行加入集群，token为上面master节点生成的

[root@es-hot1 ~]# /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjguMiIsImFkciI6WyIxOTIuMTY4LjguNTo5MjAwIl0sImZnciI6IjdmMTlkZjk5OGQ1NTRjOTJlY2I1MDVkYjZkYWUwOTU0ZDc0ZjNjNDdlZjU2Yjc4YjQ2YjY1NjYyNTU4Yjk4OGMiLCJrZXkiOiIxSG5iZDRrQmRoRkRQY2VFY0xLcjpMeTVQZ3dlZVNtT1BYTnZWS0hqWG53In0=

This node will be reconfigured to join an existing cluster, using the enrollment token that you provided.
This operation will overwrite the existing configuration. Specifically: 
  - Security auto configuration will be removed from elasticsearch.yml
  - The [certs] config directory will be removed
  - Security auto configuration related secure settings will be removed from the elasticsearch.keystore
Do you want to continue with the reconfiguration process [y/N]y

上述操作一定要使用es默认配置文件执行，待reconfigure-node完成后，再修改节点yml配置文件

hot节点配置

[root@es-hot1 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: es-cluster
node.name: es-hot1 # 修改节点名称
node.roles: [ data_content, data_hot ]
path.data: /data/es-data
path.logs: /data/es-log
network.host: 192.168.10.133 # 修改节点ip
discovery.seed_hosts: ["es-master", "es-hot1", "es-hot2", "es-hot2", "es-warm1", "es-warm2", "es-cold"]

xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

cluster.initial_master_nodes: ["es-master"]
http.host: 0.0.0.0

warm节点配置

[root@es-warm1 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: es-cluster
node.name: es-warm1 # 修改节点名称
node.roles: [ data_content, data_warm ]
path.data: /data/es-data
path.logs: /data/es-log
network.host: 192.168.10.136 # 修改节点ip
discovery.seed_hosts: ["es-master", "es-hot1", "es-hot2", "es-hot2", "es-warm1", "es-warm2", "es-cold"]

xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

cluster.initial_master_nodes: ["es-master"]
http.host: 0.0.0.0

cold节点配置

[root@es-cold ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: es-cluster
node.name: es-cold
node.roles: [ data_content, data_cold ]
path.data: /data/es-data
path.logs: /data/es-log
network.host: 192.168.10.138
discovery.seed_hosts: ["es-master", "es-hot1", "es-hot2", "es-hot2", "es-warm1", "es-warm2", "es-cold"]

xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

cluster.initial_master_nodes: ["192.168.10.100"]
http.host: 0.0.0.0

启动es服务

[root@es-hot1 ~]# systemctl start elasticsearch

剩余节点依次重复执行上述操作。

重置elastic用户密码

以下操作在master节点执行

[root@es-master ~]# /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
This tool will reset the password of the [elastic] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y
Password for the [elastic] user successfully reset.
New value: _21FDs+tGRRSaxg=q=4P

修改master节点属性

由于在集群初始化过程中，master节点写入了集群元数据索引，需要先将元数据迁移至其他数据节点。我们可以使用cluster.routing.allocation.exclude._ip，将指定节点ip上面的分片分配到其他节点上。

[root@es-master ~]# curl -X PUT -H 'content-type:application/json' -d '{"transient":{"cluster.routing.allocation.exclude._ip":"192.168.10.132"}}' --cacert /etc/elasticsearch/certs/http_ca.crt https://elastic:_21FDs+tGRRSaxg=q=4P@127.0.0.1:9200/_cluster/settings
{"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":{"allocation":{"exclude":{"_ip":"192.168.10.132"}}}}}}

等待分片迁移完成后，查看分片信息验证，保证master节点无分片数据

[root@es-master ~]# curl --cacert /etc/elasticsearch/certs/http_ca.crt https://elastic:_21FDs+tGRRSaxg=q=4P@127.0.0.1:9200/_cat/shards?
vindex       shard prirep state   docs  store ip             node
.security-7 0     p      STARTED    2 12.4kb 192.168.10.134 es-hot2
.security-7 0     r      STARTED    2 12.4kb 192.168.10.133 es-hot1

修改es配置，重启es服务

[root@es-master ~]# vim /etc/elasticsearch/elasticsearch.yml 
# 指定节点角色
node.roles: [ master, ingest ]
[root@es-master ~]# systemctl restart elasticsearch

访问验证

查看集群状态

[root@es-master ~]# curl --cacert /etc/elasticsearch/certs/http_ca.crt https://elastic:_21FDs+tGRRSaxg=q=4P@127.0.0.1:9200/_cat/health?
vepoch      timestamp cluster    status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1689561374 02:36:14  es-cluster green           6         5      2   1    0    0        0             0                  -                100.0%

查看节点状态

[root@es-master ~]# curl --cacert /etc/elasticsearch/certs/http_ca.crt https://elastic:_21FDs+tGRRSaxg=q=4P@127.0.0.1:9200/_cat/nodes?v
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.10.132           18          62  20    1.29    0.96     0.56 im        *      es-master
192.168.10.135            6          62   2    0.39    0.20     0.15 hs        -      es-hot3
192.168.10.134           43          63   4    0.01    0.07     0.08 hs        -      es-hot2
192.168.10.133           16          62   4    0.87    0.52     0.26 hs        -      es-hot1
192.168.10.138           33          62   4    0.09    0.63     0.45 cs        -      es-cold
192.168.10.136           18          62   4    0.05    0.07     0.12 sw        -      es-warm1
192.168.10.137           11          62   4    0.07    0.15     0.26 sw        -      es-warm2

kibana安装配置

以下操作在es-master节点执行

下载安装rpm包

[root@es-master ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-8.8.2-x86_64.rpm
[root@es-master ~]# rpm -ivh kibana-8.8.2-x86_64.rpm
warning: kibana-8.8.2-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:kibana-8.8.2-1                   ################################# [100%]
Creating kibana group... OK
Creating kibana user... OK
Created Kibana keystore in /etc/kibana/kibana.keystore
/usr/lib/tmpfiles.d/elasticsearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/elasticsearch   /run/elasticsearch; please update the tmpfiles.d/ drop-in file accordingly.

为Kibana生成注册令牌

[root@es-master ~]# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
eyJ2ZXIiOiI4LjguMiIsImFkciI6WyIxOTIuMTY4LjguNTo5MjAwIl0sImZnciI6IjdmMTlkZjk5OGQ1NTRjOTJlY2I1MDVkYjZkYWUwOTU0ZDc0ZjNjNDdlZjU2Yjc4YjQ2YjY1NjYyNTU4Yjk4OGMiLCJrZXkiOiJ2TEgtZDRrQllLU1JQelFQSXBGVTpwVzZEUmtTV1NJMmg4QmVlMjdpMkFRIn0=

kibana注册集群

[root@es-master ~]# /usr/share/kibana/bin/kibana-setup --enrollment-token eyJ2ZXIiOiI4LjguMiIsImFkciI6WyIxOTIuMTY4LjguNTo5MjAwIl0sImZnciI6IjdmMTlkZjk5OGQ1NTRjOTJlY2I1MDVkYjZkYWUwOTU0ZDc0ZjNjNDdlZjU2Yjc4YjQ2YjY1NjYyNTU4Yjk4OGMiLCJrZXkiOiJ2TEgtZDRrQllLU1JQelFQSXBGVTpwVzZEUmtTV1NJMmg4QmVlMjdpMkFRIn0=

  Kibana configured successfully.

To start Kibana run:
  bin/kibana

修改kibana配置

从es8开始，kibana使用token注册连接es，kibana通过解析token可以获取到es的地址用户名和密码信息，因此在kibana配置文件中无需设置es地址与账号密码信息。

[root@es-master ~]# vim /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
# kibana中文界面
i18n.locale: "zh-CN"

启动服务

[root@es-master ~]# systemctl enable kibana
Created symlink /etc/systemd/system/multi-user.target.wants/kibana.service   /usr/lib/systemd/system/kibana.service.
[root@es-master ~]# systemctl start kibana

访问验证

登录kibana http://192.168.10.132:5601，用户名为elasic，密码为上述步骤中重置的elastic用户密码

导入样例数据

登录进kibana后，我们选择使用样例数据

然后导入kibana提供的样例数据

monitor异常处理

访问kibana的monitor时，会提示以下报错。

修改kibana配置并重启。

[root@es-master ~]# vim /etc/kibana/kibana.yml
monitoring.ui.ccs.enabled: false
[root@es-master ~]# systemctl restart kibana

访问monitor页面验证

官方文档

rpm安装es服务：https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html

es配置文件：https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html

es节点角色配置：https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-roles

jvm配置：https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options

kibana配置文件：https://www.elastic.co/guide/en/kibana/current/settings.html

展开阅读全文

页面更新：2024-04-18

标签：集群节点初始化主机名名称原因操作文件数据系统

1 2 3 4 5