前言：最近玩ceph ,发现关于ceph的安装配置文档还是太少，且不同版的坑也是不一样。经常参照的文档无法进行下一步安装操作，并有各种各样的坑。介于开源精神，分享下centos8.5环境下的ceph-P版的安装步骤。
一、环境准备：
操作系统：
hostnamectl
Operating System: CentOS Stream 8
CPE OS Name: cpe:/o:centos:centos:8
Kernel: Linux 4.18.0-408.el8.x86_64
ansible --version
ansible [core 2.13.5]
python version = 3.9.13 (main, Jun 24 2022, 15:32:51) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]
jinja version = 3.1.2
ceph --version
ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
python3 --version
Python 3.6.8
网络环境
主机名 |
public_ip |
cluster_ip |
角色 |
备注 |
ceph-admin |
10.255.208.1 |
- |
ceph-ansible |
|
ceph-mon01 |
10.255.208.2 |
- |
mon |
|
ceph-mon02 |
10.255.208.4 |
- |
mon |
|
ceph-mon03 |
10.255.208.9 |
- |
mon |
|
ceph-osd01 |
10.255.208.11 |
192.168.2.1 |
osd |
双网卡，3*100GB |
ceph-osd02 |
10.255.208.12 |
192.168.2.2 |
osd |
双网卡，3*100GB |
ceph-osd03 |
10.255.208.14 |
192.168.2.3 |
osd |
双网卡，3*100GB |
ceph-osd04 |
10.255.208.16 |
192.168.2.4 |
osd |
双网卡，3*100GB |
ceph-client01 |
10.255.208.17 |
- |
client |
|
ceph-client02 |
10.255.208.18 |
- |
client |
二、初始化系统
注：如未另行注明，下面配置到所有主机上
1、设置主机名
hostnamectl set-hostname ceph-admin
2、配置IP地址
nmcli connection modify System\ ens192 ipv4.method manual ipv4.addresses 10.255.208.11/24 ipv4.gateway 10.255.208.254 ipv4.dns 10.0.1.1 connection.autoconnect yes
OSD节点配置cluster_IP地址
修改网卡的连接名称（con-name）
nmcli connection modify Wired\ connection\ 1 con-name ens224
nmcli connection show
nmcli connection modify ens224 ipv4.method manual ipv4.addresses 192.168.2.1/24 connection.autoconnect yes
3、关闭防火墙，selinux
systemctl stop firewalld && systemctl disable firewalld
setenforce 0 && sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config && getenforce
4、修改 /etc/hosts文件，进行本地解析。
注： 解析的IP是public_IP
cp /etc/hosts /etc/hosts.back
cat > /etc/hosts <<EOF
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.255.208.1 ceph-admin
10.255.208.2 ceph-mon01
10.255.208.4 ceph-mon02
10.255.208.9 ceph-mon03
10.255.208.11 ceph-osd01
10.255.208.12 ceph-osd02
10.255.208.14 ceph-osd03
10.255.208.16 ceph-osd04
10.255.208.17 ceph-client01
10.255.208.18 ceph-client02
EOF
5、修改DNS文件
echo "nameserver 10.0.1.1" >> /etc/resolv.conf
systemctl restart network
ping -c 3 www.baidu.com
6、配置主机时间同步
配置ntp服务，保证集群服务器时间统一
统一时间非常重要，必须要做
大前提：chrony服务端客户端配置完后，重启chronyd服务即可快速完成时间同步，在这之后就不要再手动去修改时间了，一切让时间服务器自己去同步
chrony服务端：ceph-admin节点
# 1、安装
yum -y install chrony
# 2、修改配置文件
mv /etc/chrony.conf /etc/chrony.conf.bak
cat > /etc/chrony.conf << EOF
server ntp1.aliyun.com iburst minpoll 4 maxpoll 10
server ntp2.aliyun.com iburst minpoll 4 maxpoll 10
server ntp3.aliyun.com iburst minpoll 4 maxpoll 10
server ntp4.aliyun.com iburst minpoll 4 maxpoll 10
server ntp5.aliyun.com iburst minpoll 4 maxpoll 10
server ntp6.aliyun.com iburst minpoll 4 maxpoll 10
server ntp7.aliyun.com iburst minpoll 4 maxpoll 10
driftfile /var/lib/chrony/drift
makestep 10 3
rtcsync
allow 0.0.0.0/0
local stratum 10
keyfile /etc/chrony.keys
logdir /var/log/chrony
stratumweight 0.05
noclientlog
logchange 0.5
EOF
# 4、启动chronyd服务
# 最好重启，这样无论原来是否启动都可以重新加载配置
systemctl restart chronyd.service && systemctl enable chronyd.service && systemctl status chronyd.service
chrony客户端：其他节点，完全一样的配置与操作
chrony客户端：除ceph-admin节点的其他节点
# 下述步骤一次性粘贴到每个客户端执行即可
# 1、安装chrony
yum -y install chrony
# 2、需改客户端配置文件
mv /etc/chrony.conf /etc/chrony.conf.bak
cat > /etc/chrony.conf << EOF
server ceph-admin iburst
driftfile /var/lib/chrony/drift
makestep 10 3
rtcsync
local stratum 10
keyfile /etc/chrony.key
logdir /var/log/chrony
stratumweight 0.05
noclientlog
logchange 0.5
EOF
# 3、启动chronyd
systemctl restart chronyd.service && systemctl enable chronyd.service &&systemctl status chronyd.service
7、添加节点信任关系,免密连接
ssh-keygen -t rsa
for i in {1..3}; do ssh-copy-id ceph-client0$i; done
三、 安装ceph-ansible
1、ceph-admin节点安装 pip和ansible、git
注：ceph版本与ansible版本对应关系
ceph-ansible — ceph-ansible documentation
Releases
stable-6.0 Supports Ceph version pacific. This branch requires Ansible version 2.10.
stable-7.0 Supports Ceph version quincy. This branch requires Ansible version 2.12.
main Supports the main (devel) branch of Ceph. This branch requires Ansible version 2.12.
ansible官网安装文档
Installing Ansible on specific operating systems — Ansible Documentation
yum install -y git epel-release
yum install -y ansible
ansible --version
2、部署ceph集群
这里我选择安装的是ceph quincy版本
git clone https://github.com/ceph/ceph-ansible.git
cd ceph-ansible/
git checkout stable-6.0
pip install -r requirements.txt
###if you see an error like No module named pip, you’ll need to install pip under your chosen Python interpreter before proceeding. This may mean installing an additional OS package (for example, python3-pip), or installing the latest pip directly from the Python Packaging Authority by running the following:
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py --user
python3 -m pip install -r requirements.txt
###Successfully installed ansible-2.10.7 ansible-base-2.10.17 importlib-resources-5.4.0 netaddr-0.8.0 packaging-21.3 pyparsing-3.0.9 zipp-3.6.0
4、备份修改/etc/ansible/hosts文件，
cd /etc/ansible/
cp hosts hosts.back
cat >> /etc/ansible/hosts <<EOF
[mons]
ceph-mon0[1:3]
[osds]
ceph-osd0[1:4]
[mgrs]
ceph-mon0[1:3]
[mdss]
ceph-mon0[1:3]
[clients]
ceph-client0[1:2]
[rgws]
ceph-mon0[1:3]
[grafana-server]
ceph-mon01
EOF
5、检查ansible连接服务器
ansible -m ping all
6、备份group_vars下的yml文件
cd ceph-ansible/group_vars
for file in *;do cp $file ${file%.*};done
7、修改group_vars/all.yml配置
cat > all.yml <<EOF
---
dummy:
mon_group_name: mons
osd_group_name: osds
rgw_group_name: rgws
mds_group_name: mdss
client_group_name: clients
mgr_group_name: mgrs
grafana_server_group_name: grafana-server
configure_firewall: False
ceph_origin: repository
ceph_repository: community
ceph_mirror: ceph安装包下载_开源镜像站-阿里云
ceph_stable_key: http://mirrors.aliyun.com/ceph/keys/release.asc
ceph_stable_release: pacific
ceph_stable_repo: "{{ ceph_mirror }}/rpm-{{ ceph_stable_release }}"
public_network: "10.255.208.0/24"
cluster_network: "192.168.2.0/24"
monitor_interface: ens192
monitor_address_block: 10.255.208.0/24
ip_version: ipv4
osd_auto_discovery: true
osd_objectstore: bluestroe
radosgw_interface: ens192
dashboard_admin_password: asd123456
grafana_admin_password: admin
pg_autoscale_mode: True
dashboard_enabled: false
EOF
8、修改group_vars/osdo.yml配置
devices:
- /dev/sdb
- /dev/sdc
- /dev/sdd
9、修改site.yml配置
[root@ceph-admin ceph-ansible]# pwd
/root/ceph-ansible
cp site.yml.sample site.yml
[root@ceph-admin ceph-ansible]# cat site.yml
- hosts:
- mons
- osds
- mdss
- rgws
# - nfss
#- rbdmirrors
- clients
- mgrs
#- iscsigws
- monitoring
#- rgwloadbalancers
10、开始进行安装
ansible-playbook -i /etc/ansible/hosts site.yml
11、检查ceph状态
root@ceph-mon01 ~]# ceph -s
cluster:
id: 74cd55a1-69c2-4114-b73b-a90cf504eda9
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 (age 4h)
mgr: ceph-mon01(active, since 2d), standbys: ceph-mon03, ceph-mon02
mds: 1/1 daemons up, 2 standby
osd: 12 osds: 12 up (since 99m), 12 in (since 2d)
rgw: 3 daemons active (3 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 7 pools, 169 pgs
objects: 247 objects, 9.6 KiB
usage: 2.0 GiB used, 1.2 TiB / 1.2 TiB avail
pgs: 169 active+clean
io:
client: 95 KiB/s rd, 0 B/s wr, 95 op/s rd, 63 op/s wr
progress:
Global Recovery Event (10s)
[===========================.]