OpenShift自带的日志搜索引擎ES服务的扩容

有以下两种扩容方式：

1. 存储扩容
2. 节点扩容

存储扩容

1. 查看集群的状态

$ oc exec $es-pod-name -c elasticsearch -- health

2. 停止ES节点分片平衡

$ oc exec $es-pod-name -c elasticsearch -- curl --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key --cacert /etc/elasticsearch/secret/admin-ca -XPUT https://localhost:9200/_cluster/settings -d '{"transient": {"cluster.routing.allocation.enable":"none"}}'

3. 为每个ES节点的DC执行以下操作
   3.1 获取需要暂停的ES pod所在的Node节点

$ oc get pods -o wide 

3.2 暂停第一个ES pod

$ oc scale dc <es-dc-01> --replicas=0

3.3 将节点的数据复制到更大的存储盘目录中
3.4 更新该ES节点的DC配置，将数据盘挂载到新的目录
3.5 恢复ES节点

$ oc scale dc <es-dc-01> --replicas=1

4. 按照3的步骤为剩下的ES节点进行扩容
5. 恢复ES节点分片平衡

$ oc exec $es-pod-name -c elasticsearch -- curl --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key --cacert /etc/elasticsearch/secret/admin-ca -XPUT https://localhost:9200/_cluster/settings -d '{"transient": {"cluster.routing.allocation.enable":"all"}}'

节点扩容

1. 添加infra-logging节点，配置保持与之前的logging节点一致，并准备好数据盘目录
2. 停止ES节点分片平衡

$ oc exec $es-pod-name -c elasticsearch -- curl --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key --cacert /etc/elasticsearch/secret/admin-ca -XPUT https://localhost:9200/_cluster/settings -d '{"transient": {"cluster.routing.allocation.enable":"none"}}'

3. 修改部署配置文件ansible hosts文件，增大ES节点大小

openshift_logging_es_cluster_size=5
openshift_logging_es_number_of_shards=3
openshift_logging_es_number_of_replicas=1
openshift_logging_elasticsearch_storage_type=hostmount
openshift_logging_elasticsearch_hostmount_path=/es-data

4. 运行logging的部署安装playbook

$ ansible-playbook openshift-ansible/playbooks/openshift-logging/config.yml

5.恢复ES节点分片平衡

$ oc exec $es-pod-name -c elasticsearch -- curl --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key --cacert /etc/elasticsearch/secret/admin-ca -XPUT https://localhost:9200/_cluster/settings -d '{"transient": {"cluster.routing.allocation.enable":"all"}}'