本次演示环境如下：
hostname |
IP |
roles |
node01.srv.world |
192.168.10.101 |
Object Storage;Monitor Daemon;Manager Daemon |
node02.srv.world |
192.168.10.102 |
Object Storage |
node03.srv.world |
192.168.10.103 |
Object Storage |
dlp.srv.world |
192.168.10.142 |
client |
1）在node01为dlp做互信
[root@node01 ~]# ssh-copy-id dlp
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@dlp's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'dlp'"
and check to make sure that only the key(s) you wanted were added.
[root@node01 ~]#2）为dlp安装所需的软件包
[root@node01 ~]# ssh dlp "dnf -y install centos-release-ceph-reef epel-release; dnf -y install ceph-common"
CentOS Stream 9 - BaseOS 716 B/s | 4.0 kB 00:05
CentOS Stream 9 - BaseOS 333 kB/s | 7.8 MB 00:24
CentOS Stream 9 - AppStream 8.6 kB/s | 4.0 kB 00:00
CentOS Stream 9 - AppStream 2.8 MB/s | 18 MB 00:06
CentOS Stream 9 - Extras packages 779 B/s | 4.3 kB 00:05
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
centos-release-ceph-reef noarch 1.0-1.el9s extras-common 7.3 k
epel-release noarch 9-7.el9 extras-common 19 k
Installing weak dependencies:
epel-next-release noarch 9-7.el9 extras-common 8.1 k
Transaction Summary
================================================================================
Install 3 Packages
Total download size: 34 k
Installed size: 30 k
Downloading Packages:
(1/3): epel-next-release-9-7.el9.noarch.rpm 50 kB/s | 8.1 kB 00:00
(2/3): centos-release-ceph-reef-1.0-1.el9s.noar 44 kB/s | 7.3 kB 00:00
(3/3): epel-release-9-7.el9.noarch.rpm 95 kB/s | 19 kB 00:00
--------------------------------------------------------------------------------
Total 20 kB/s | 34 kB 00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : epel-release-9-7.el9.noarch 1/3
Running scriptlet: epel-release-9-7.el9.noarch 1/3
Many EPEL packages require the CodeReady Builder (CRB) repository.
It is recommended that you run /usr/bin/crb enable to enable the CRB repository.
Installing : epel-next-release-9-7.el9.noarch 2/3
Installing : centos-release-ceph-reef-1.0-1.el9s.noarch 3/3
Running scriptlet: centos-release-ceph-reef-1.0-1.el9s.noarch 3/3
Verifying : centos-release-ceph-reef-1.0-1.el9s.noarch 1/3
Verifying : epel-next-release-9-7.el9.noarch 2/3
Verifying : epel-release-9-7.el9.noarch 3/3
Installed:
centos-release-ceph-reef-1.0-1.el9s.noarch epel-next-release-9-7.el9.noarch
epel-release-9-7.el9.noarch
Complete!
CentOS-9-stream - Ceph Reef 13 kB/s | 32 kB 00:02
Extra Packages for Enterprise Linux 9 - x86_64 2.5 MB/s | 19 MB 00:07
Extra Packages for Enterprise Linux 9 openh264 731 B/s | 2.5 kB 00:03
Extra Packages for Enterprise Linux 9 - Next - 99 kB/s | 1.3 MB 00:13
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
ceph-common x86_64 2:18.2.0-1.el9s centos-ceph-reef 18 M
Installing dependencies:
boost-program-options x86_64 1.75.0-8.el9 appstream 107 k
daxctl-libs x86_64 71.1-8.el9 baseos 42 k
libarrow x86_64 9.0.0-9.el9 epel 4.4 M
libarrow-doc noarch 9.0.0-9.el9 epel 24 k
libbabeltrace x86_64 1.5.8-10.el9 appstream 190 k
libcephfs2 x86_64 2:18.2.0-1.el9s centos-ceph-reef 638 k
libicu x86_64 67.1-9.el9 baseos 9.6 M
liboath x86_64 2.6.7-2.el9 epel 49 k
libpmem x86_64 1.12.1-1.el9 appstream 112 k
libpmemobj x86_64 1.12.1-1.el9 appstream 160 k
librabbitmq x86_64 0.11.0-7.el9 appstream 45 k
librados2 x86_64 2:18.2.0-1.el9s centos-ceph-reef 3.2 M
libradosstriper1 x86_64 2:18.2.0-1.el9s centos-ceph-reef 459 k
librbd1 x86_64 2:18.2.0-1.el9s centos-ceph-reef 2.9 M
librdkafka x86_64 1.6.1-102.el9 appstream 662 k
librdmacm x86_64 46.0-1.el9 baseos 71 k
librgw2 x86_64 2:18.2.0-1.el9s centos-ceph-reef 4.4 M
liburing x86_64 2.3-2.el9 appstream 27 k
lttng-ust x86_64 2.12.0-6.el9 appstream 292 k
ndctl-libs x86_64 71.1-8.el9 baseos 89 k
parquet-libs x86_64 9.0.0-9.el9 epel 837 k
python3-ceph-argparse x86_64 2:18.2.0-1.el9s centos-ceph-reef 47 k
python3-ceph-common x86_64 2:18.2.0-1.el9s centos-ceph-reef 122 k
python3-cephfs x86_64 2:18.2.0-1.el9s centos-ceph-reef 158 k
python3-prettytable noarch 0.7.2-27.el9 appstream 42 k
python3-rados x86_64 2:18.2.0-1.el9s centos-ceph-reef 321 k
python3-rbd x86_64 2:18.2.0-1.el9s centos-ceph-reef 300 k
python3-rgw x86_64 2:18.2.0-1.el9s centos-ceph-reef 101 k
re2 x86_64 1:20211101-3.el9 epel 192 k
thrift x86_64 0.15.0-2.el9 epel 1.6 M
Transaction Summary
================================================================================
Install 31 Packages
Total download size: 49 M
Installed size: 177 M
Downloading Packages:
(1/31): libcephfs2-18.2.0-1.el9s.x86_64.rpm 949 kB/s | 638 kB 00:00
(2/31): libradosstriper1-18.2.0-1.el9s.x86_64.r 379 kB/s | 459 kB 00:01
(3/31): librados2-18.2.0-1.el9s.x86_64.rpm 1.1 MB/s | 3.2 MB 00:03
(4/31): librbd1-18.2.0-1.el9s.x86_64.rpm 1.8 MB/s | 2.9 MB 00:01
(5/31): python3-ceph-argparse-18.2.0-1.el9s.x86 487 kB/s | 47 kB 00:00
(6/31): python3-ceph-common-18.2.0-1.el9s.x86_6 1.1 MB/s | 122 kB 00:00
(7/31): python3-cephfs-18.2.0-1.el9s.x86_64.rpm 907 kB/s | 158 kB 00:00
(8/31): python3-rados-18.2.0-1.el9s.x86_64.rpm 1.1 MB/s | 321 kB 00:00
(9/31): python3-rbd-18.2.0-1.el9s.x86_64.rpm 1.1 MB/s | 300 kB 00:00
(10/31): python3-rgw-18.2.0-1.el9s.x86_64.rpm 893 kB/s | 101 kB 00:00
(11/31): librgw2-18.2.0-1.el9s.x86_64.rpm 1.4 MB/s | 4.4 MB 00:03
(12/31): daxctl-libs-71.1-8.el9.x86_64.rpm 4.2 kB/s | 42 kB 00:10
(13/31): librdmacm-46.0-1.el9.x86_64.rpm 214 kB/s | 71 kB 00:00
(14/31): ndctl-libs-71.1-8.el9.x86_64.rpm 587 kB/s | 89 kB 00:00
(15/31): boost-program-options-1.75.0-8.el9.x86 92 kB/s | 107 kB 00:01
(16/31): libbabeltrace-1.5.8-10.el9.x86_64.rpm 306 kB/s | 190 kB 00:00
(17/31): ceph-common-18.2.0-1.el9s.x86_64.rpm 1.0 MB/s | 18 MB 00:17
(18/31): libpmem-1.12.1-1.el9.x86_64.rpm 288 kB/s | 112 kB 00:00
(19/31): librabbitmq-0.11.0-7.el9.x86_64.rpm 306 kB/s | 45 kB 00:00
(20/31): libpmemobj-1.12.1-1.el9.x86_64.rpm 349 kB/s | 160 kB 00:00
(21/31): liburing-2.3-2.el9.x86_64.rpm 393 kB/s | 27 kB 00:00
(22/31): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 292 kB 00:00
(23/31): python3-prettytable-0.7.2-27.el9.noarc 378 kB/s | 42 kB 00:00
(24/31): libicu-67.1-9.el9.x86_64.rpm 810 kB/s | 9.6 MB 00:12
(25/31): librdkafka-1.6.1-102.el9.x86_64.rpm 768 kB/s | 662 kB 00:00
(26/31): libarrow-doc-9.0.0-9.el9.noarch.rpm 62 kB/s | 24 kB 00:00
(27/31): liboath-2.6.7-2.el9.x86_64.rpm 111 kB/s | 49 kB 00:00
(28/31): re2-20211101-3.el9.x86_64.rpm 319 kB/s | 192 kB 00:00
(29/31): parquet-libs-9.0.0-9.el9.x86_64.rpm 719 kB/s | 837 kB 00:01
(30/31): libarrow-9.0.0-9.el9.x86_64.rpm 2.2 MB/s | 4.4 MB 00:01
(31/31): thrift-0.15.0-2.el9.x86_64.rpm 715 kB/s | 1.6 MB 00:02
--------------------------------------------------------------------------------
Total 2.0 MB/s | 49 MB 00:24
Extra Packages for Enterprise Linux 9 - x86_64 1.6 MB/s | 1.6 kB 00:00
Importing GPG key 0x3228467C:
Userid : "Fedora (epel9) <epel@fedoraproject.org>"
Fingerprint: FF8A D134 4597 106E CE81 3B91 8A38 72BF 3228 467C
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-9
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31
Installing : daxctl-libs-71.1-8.el9.x86_64 2/31
Installing : ndctl-libs-71.1-8.el9.x86_64 3/31
Installing : thrift-0.15.0-2.el9.x86_64 4/31
Installing : librdkafka-1.6.1-102.el9.x86_64 5/31
Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31
Installing : libicu-67.1-9.el9.x86_64 7/31
Installing : python3-ceph-argparse-2:18.2.0-1.el9s.x86_64 8/31
Installing : libpmem-1.12.1-1.el9.x86_64 9/31
Installing : libpmemobj-1.12.1-1.el9.x86_64 10/31
Installing : re2-1:20211101-3.el9.x86_64 11/31
Installing : libarrow-9.0.0-9.el9.x86_64 12/31
Installing : liboath-2.6.7-2.el9.x86_64 13/31
Installing : libarrow-doc-9.0.0-9.el9.noarch 14/31
Installing : python3-prettytable-0.7.2-27.el9.noarch 15/31
Installing : liburing-2.3-2.el9.x86_64 16/31
Installing : libbabeltrace-1.5.8-10.el9.x86_64 17/31
Installing : boost-program-options-1.75.0-8.el9.x86_64 18/31
Installing : parquet-libs-9.0.0-9.el9.x86_64 19/31
Installing : librdmacm-46.0-1.el9.x86_64 20/31
Installing : librados2-2:18.2.0-1.el9s.x86_64 21/31
Running scriptlet: librados2-2:18.2.0-1.el9s.x86_64 21/31
Installing : python3-rados-2:18.2.0-1.el9s.x86_64 22/31
Installing : libcephfs2-2:18.2.0-1.el9s.x86_64 23/31
Running scriptlet: libcephfs2-2:18.2.0-1.el9s.x86_64 23/31
Installing : librbd1-2:18.2.0-1.el9s.x86_64 24/31
Running scriptlet: librbd1-2:18.2.0-1.el9s.x86_64 24/31
Installing : python3-rbd-2:18.2.0-1.el9s.x86_64 25/31
Installing : python3-cephfs-2:18.2.0-1.el9s.x86_64 26/31
Installing : libradosstriper1-2:18.2.0-1.el9s.x86_64 27/31
Running scriptlet: libradosstriper1-2:18.2.0-1.el9s.x86_64 27/31
Installing : librgw2-2:18.2.0-1.el9s.x86_64 28/31
Running scriptlet: librgw2-2:18.2.0-1.el9s.x86_64 28/31
Installing : python3-rgw-2:18.2.0-1.el9s.x86_64 29/31
Installing : python3-ceph-common-2:18.2.0-1.el9s.x86_64 30/31
Running scriptlet: ceph-common-2:18.2.0-1.el9s.x86_64 31/31
Installing : ceph-common-2:18.2.0-1.el9s.x86_64 31/31
Running scriptlet: ceph-common-2:18.2.0-1.el9s.x86_64 31/31
Verifying : ceph-common-2:18.2.0-1.el9s.x86_64 1/31
Verifying : libcephfs2-2:18.2.0-1.el9s.x86_64 2/31
Verifying : librados2-2:18.2.0-1.el9s.x86_64 3/31
Verifying : libradosstriper1-2:18.2.0-1.el9s.x86_64 4/31
Verifying : librbd1-2:18.2.0-1.el9s.x86_64 5/31
Verifying : librgw2-2:18.2.0-1.el9s.x86_64 6/31
Verifying : python3-ceph-argparse-2:18.2.0-1.el9s.x86_64 7/31
Verifying : python3-ceph-common-2:18.2.0-1.el9s.x86_64 8/31
Verifying : python3-cephfs-2:18.2.0-1.el9s.x86_64 9/31
Verifying : python3-rados-2:18.2.0-1.el9s.x86_64 10/31
Verifying : python3-rbd-2:18.2.0-1.el9s.x86_64 11/31
Verifying : python3-rgw-2:18.2.0-1.el9s.x86_64 12/31
Verifying : daxctl-libs-71.1-8.el9.x86_64 13/31
Verifying : libicu-67.1-9.el9.x86_64 14/31
Verifying : librdmacm-46.0-1.el9.x86_64 15/31
Verifying : ndctl-libs-71.1-8.el9.x86_64 16/31
Verifying : boost-program-options-1.75.0-8.el9.x86_64 17/31
Verifying : libbabeltrace-1.5.8-10.el9.x86_64 18/31
Verifying : libpmem-1.12.1-1.el9.x86_64 19/31
Verifying : libpmemobj-1.12.1-1.el9.x86_64 20/31
Verifying : librabbitmq-0.11.0-7.el9.x86_64 21/31
Verifying : librdkafka-1.6.1-102.el9.x86_64 22/31
Verifying : liburing-2.3-2.el9.x86_64 23/31
Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/31
Verifying : python3-prettytable-0.7.2-27.el9.noarch 25/31
Verifying : libarrow-9.0.0-9.el9.x86_64 26/31
Verifying : libarrow-doc-9.0.0-9.el9.noarch 27/31
Verifying : liboath-2.6.7-2.el9.x86_64 28/31
Verifying : parquet-libs-9.0.0-9.el9.x86_64 29/31
Verifying : re2-1:20211101-3.el9.x86_64 30/31
Verifying : thrift-0.15.0-2.el9.x86_64 31/31
Installed:
boost-program-options-1.75.0-8.el9.x86_64
ceph-common-2:18.2.0-1.el9s.x86_64
daxctl-libs-71.1-8.el9.x86_64
libarrow-9.0.0-9.el9.x86_64
libarrow-doc-9.0.0-9.el9.noarch
libbabeltrace-1.5.8-10.el9.x86_64
libcephfs2-2:18.2.0-1.el9s.x86_64
libicu-67.1-9.el9.x86_64
liboath-2.6.7-2.el9.x86_64
libpmem-1.12.1-1.el9.x86_64
libpmemobj-1.12.1-1.el9.x86_64
librabbitmq-0.11.0-7.el9.x86_64
librados2-2:18.2.0-1.el9s.x86_64
libradosstriper1-2:18.2.0-1.el9s.x86_64
librbd1-2:18.2.0-1.el9s.x86_64
librdkafka-1.6.1-102.el9.x86_64
librdmacm-46.0-1.el9.x86_64
librgw2-2:18.2.0-1.el9s.x86_64
liburing-2.3-2.el9.x86_64
lttng-ust-2.12.0-6.el9.x86_64
ndctl-libs-71.1-8.el9.x86_64
parquet-libs-9.0.0-9.el9.x86_64
python3-ceph-argparse-2:18.2.0-1.el9s.x86_64
python3-ceph-common-2:18.2.0-1.el9s.x86_64
python3-cephfs-2:18.2.0-1.el9s.x86_64
python3-prettytable-0.7.2-27.el9.noarch
python3-rados-2:18.2.0-1.el9s.x86_64
python3-rbd-2:18.2.0-1.el9s.x86_64
python3-rgw-2:18.2.0-1.el9s.x86_64
re2-1:20211101-3.el9.x86_64
thrift-0.15.0-2.el9.x86_64
Complete!3）从node01拷贝所需的文件至dlp
[root@node01 ~]# scp /etc/ceph/ceph.conf dlp:/etc/ceph/
ceph.conf 100% 582 273.3KB/s 00:00
[root@node01 ~]# scp /etc/ceph/ceph.client.admin.keyring dlp:/etc/ceph/
ceph.client.admin.keyring 100% 151 178.7KB/s 00:00
[root@node01 ~]# ssh dlp "chown ceph:ceph /etc/ceph/ceph.*"
[root@node01 ~]#4)创建块存储，并将该设备挂载至dlp a.在dlp创建rbd池
[root@dlp ~]# ceph osd pool create rbd 32
pool 'rbd' createdb.启用Placement Groups auto scale模式
[root@dlp ~]# ceph osd pool set rbd pg_autoscale_mode on
set pool 2 pg_autoscale_mode to onc.初始化池
[root@dlp ~]# rbd pool init rbd
[root@dlp ~]# ceph osd pool autoscale-status
POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK
.mgr 448.5k 3.0 61428M 0.0000 1.0 1 on False
rbd 19 3.0 61428M 0.0000 1.0 32 on False
[root@dlp ~]#d.创建10GB的块存储，并验证是否成功
[root@dlp ~]# rbd create --size 10G --pool rbd rbd01
[root@dlp ~]# rbd ls -l
NAME SIZE PARENT FMT PROT LOCK
rbd01 10 GiB 2e.块设备映射并确认
[root@dlp ~]# rbd map rbd01
/dev/rbd0
[root@dlp ~]# rbd showmapped
id pool namespace image snap device
0 rbd rbd01 - /dev/rbd0
[root@dlp ~]#f.块设备格式化，并挂载验证
[root@dlp ~]# mkfs.xfs /dev/rbd0
meta-data=/dev/rbd0 isize=512 agcount=16, agsize=163840 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=1 inobtcount=1 nrext64=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=16 swidth=16 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=16384, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Discarding blocks...Done.
[root@dlp ~]# mount /dev/rbd0 /mnt
[root@dlp ~]# df -hT
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs tmpfs 467M 0 467M 0% /dev/shm
tmpfs tmpfs 187M 3.9M 183M 3% /run
/dev/mapper/cs-root xfs 17G 1.5G 16G 9% /
/dev/nvme0n1p1 xfs 960M 197M 764M 21% /boot
tmpfs tmpfs 94M 0 94M 0% /run/user/0
/dev/rbd0 xfs 10G 104M 9.9G 2% /mnt
[root@dlp ~]#g.取消映射块设备
[root@dlp ~]# rbd unmap /dev/rbd/rbd/rbd01
rbd: sysfs write failed
rbd: unmap failed: (16) Device or resource busy
[root@dlp ~]# rbd unmap -o force /dev/rbd/rbd/rbd01
[root@dlp ~]# rbd rm rbd01 -p rbd
Removing image: 100% complete...done.h.删除池
[root@dlp ~]# ceph osd pool delete rbd rbd --yes-i-really-really-mean-it
pool 'rbd' removed下一小节将介绍使用文件系统，欢迎持续关注！