OpenHarmony 3.2使用fio进行I/O性能测试-摩杜云开发者社区

一、环境配置

源码环境：OpenHarmony 3.2 Release
平台硬件： RK3568
平台软件：OpenHarmony 3.2 Release 64bit
fio版本：3.35

二、fio简介

fio是一个I/O测试工具，可以运行在Linux、Windows等多种系统之上，可以用来测试本地磁盘、网络存储等的I/O性能。

fio官网：https://fio.readthedocs.io/en/latest/fio_doc.html

fio源码下载：https://brick.kernel.dk/snaps/

三、交叉编译fio源码

参考资料：Openharmony 交叉编译haveged

将下载的fio源码fio-3.35.tar.gz放入OpenHarmony项目源码third_party/目录下进行解压

$ cd third_party
$ tar -zxvf fio-3.35.tar.gz

3.1、配置交叉编译环境

在third_party/fio-3.35/目录下创建交叉编译环境的配置脚本fio_env.sh

#!/bin/bash
#aarch64-linux-ohos
SDK_PATH=../../out/sdk/ohos-sdk/linux
CUR_PATH=$(pwd)
cd $SDK_PATH
export OHOS_SDK=$(pwd)
cd $CUR_PATH
export AS=${OHOS_SDK}/native/llvm/bin/llvm-as
export CC="${OHOS_SDK}/native/llvm/bin/clang --target=aarch64-linux-ohos"
export CXX="${OHOS_SDK}/native/llvm/bin/clang++ --target=aarch64-linux-ohos"
export LD="${OHOS_SDK}/native/llvm/bin/lld --target=aarch64-linux-ohos"
export STRIP=${OHOS_SDK}/native/llvm/bin/llvm-strip
export RANLIB=${OHOS_SDK}/native/llvm/bin/llvm-ranlib
export OBJDUMP=${OHOS_SDK}/native/llvm/bin/llvm-objdump
export OBJCOPY=${OHOS_SDK}/native/llvm/bin/llvm-objcopy
export NM=${OHOS_SDK}/native/llvm/bin/llvm-nm
export AR=${OHOS_SDK}/native/llvm/bin/llvm-ar
export CFLAGS="-fPIC -D__MUSL__=1"
export CXXFLAGS="-fPIC -D__MUSL__=1"

注意:交叉编译环境与设备CPU架构和软件系统架构有关，需根据实际情况配置和修改。

RK3568 cpu架构为aarch64，查看方法如下：

$ hdc shell uname -a
Linux localhost 5.10.97 #1 SMP Thu Aug 31 13:59:32 CST 2023 aarch64

设备系统软件架构为64bit,查看方法如下：

$ hdc shell getconf LONG_BIT
64

在third_party/fio-3.35/目录下运行fio_env.sh

$ cd third_party/fio-3.35/
$ source fio_env.sh

3.2、编译fio源码

接着上面步骤，在third_party/fio-3.35/目录下执行下面的命令

#配置fio编译环境，cpu架构为aarch64; 安装文件生成路径为./install
$ ./configure --cpu=aarch64 --prefix=$PWD/install
...
Lib-based ioengines dynamic   no
TCMalloc support              no
seed_buckets                  4

#执行编译
$ make && make install
...
install -m 755 -d /home/xxx/third_party/fio-3.35/install/bin
install fio t/fio-genzipf t/fio-btrace2fio t/fio-verify-state ./tools/fio_generate_plots ./tools/plot/fio2gnuplot ./tools/genfio ./tools/fiologparser.py ./tools/hist/fiologparser_hist.py ./tools/hist/fio-histo-log-pctiles.py ./tools/fio_jsonplus_clat2csv /home/xxx/third_party/fio-3.35/install/bin
install -m 755 -d /home/xxx/third_party/fio-3.35/install/man/man1
install -m 644 ./fio.1 /home/xxx/third_party/fio-3.35/install/man/man1
install -m 644 ./tools/fio_generate_plots.1 /home/xxx/third_party/fio-3.35/install/man/man1
install -m 644 ./tools/plot/fio2gnuplot.1 /home/xxx/third_party/fio-3.35/install/man/man1
install -m 644 ./tools/hist/fiologparser_hist.py.1 /home/xxx/third_party/fio-3.35/install/man/man1
install -m 755 -d /home/xxx/third_party/fio-3.35/install/share/fio
install -m 644 ./tools/plot/*gpm /home/xxx/third_party/fio-3.35/install/share/fio/

编译通过后在third_party/fio-3.35/install/bin目录下生成fio可执行文件

fio可执行文件网盘下载

third_party/fio-3.35$ tree install/bin
install/bin
├── fio
├── fio2gnuplot
├── fio-btrace2fio
├── fio_generate_plots
├── fio-genzipf
├── fio-histo-log-pctiles.py
├── fio_jsonplus_clat2csv
├── fiologparser_hist.py
├── fiologparser.py
├── fio-verify-state
└── genfio

使用file检测fio文件属性

third_party/fio-3.35$ file install/bin/fio
install/bin/fio: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, with debug_info, not stripped

3.3、验证fio可执行文件

将上述生成的fio 放入开发板/system/bin下

$ hdc shell "mount -o remount,rw /";hdc file send xxx\third_party\fio-3.35\install\bin\fio /system/bin

#修改/system/bin/fio 文件权限和用户组，与同目录文件保持属性一致。
$ hdc shell
# cd system/bin
# chmod 755 fio && chown shell:root fio

# 输出fio版本信息，检测fio运行正常与否
# fio -v
fio-3/35

四、测试RK3568 I/O性能

在进行I/O测试前，有必要先了解下fio 常用参数

fio的简单介绍及部分参数翻译

fio 命令行参数详解

参数	举例	说明
filename	filename=/data/fio_read_test	测试文件名，通常选择data分区
direct	direct=1	1表示不使用I/O缓存，反之则使用。为了使测试结果更真实，通常设为1。
rw	rw=read / write / randread / randwrite / randrw	I/O类型，可设置为顺序读、顺序写、随机读、随机写、混合随机读写
bs	bs=4k	单次I/O的块文件大小，通常为2k、4k、6k、8k等
bsrange	bsrange=512-2048	设定单次I/O块文件大小范围
size	size=5G	设定本次测试文件大小为5GB，每次以bs=4k大小进行I/O测试
numjobs	numjobs=30	测试线程数为30
runtime	runtime=120	测试时长为120秒，未设置默认写满size=5GB大小
ioengine	ioegine=psync	I/O引擎类型，包括libaio、sync、psync等
rwmixwrite	rwmixwrite=30	混合读写模式下，写占比30%
group_reporting		按组展示结果
output	output=/data/fio_test_result	输出测试结果至/data/fio_test_result文件中
output-format	output-format=json	设置fio输出结果的格式
lockmem	lockmem=1g	只使用1g内存进行测试
iodepth	iodepth 1	文件上I/O模块的数量(大于1的iodepth对同步io无意义)
thread		使用pthread_create创建线程代替使用fork创建进程,可在一定程度上节省系统开销

4.1、测试实例

顺序写 fio -filename=/data/fio_test_ordwrite -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=1M -size=1G -numjobs=5 -group_reporting -name=mytest

# fio -filename=/data/fio_test_ordwrite -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=1M -size=1G -numjobs=5 -group_reporting -name=mytest
mytest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
...
fio-3.35
Starting 5 threads	#启用5个线程，对应“numjobs=5”
mytest: Laying out IO file (1 file / 1024MiB)
Jobs: 5 (f=5): [W(5)][98.7%][w=65.0MiB/s][w=65 IOPS][eta 00m:01s]
mytest: (groupid=0, jobs=5): err= 0: pid=1688: Mon Aug  7 18:46:21 2017
  write: IOPS=66, BW=66.1MiB/s (69.3MB/s)(5120MiB/77482msec); 0 zone resets
    clat (msec): min=8, max=285, avg=75.35, stdev=56.61
     lat (msec): min=9, max=285, avg=75.61, stdev=56.57
    clat percentiles (msec):
     |  1.00th=[   37],  5.00th=[   42], 10.00th=[   44], 20.00th=[   45],
     | 30.00th=[   45], 40.00th=[   45], 50.00th=[   53], 60.00th=[   54],
     | 70.00th=[   55], 80.00th=[   85], 90.00th=[  186], 95.00th=[  220],
     | 99.00th=[  253], 99.50th=[  255], 99.90th=[  266], 99.95th=[  271],
     | 99.99th=[  288]
   bw (  KiB/s): min=20462, max=114818, per=99.81%, avg=67534.76, stdev=7052.44, samples=770
   iops        : min=   18, max=  111, avg=64.64, stdev= 6.72, samples=770
  lat (msec)   : 10=0.04%, 20=0.02%, 50=44.75%, 100=39.92%, 250=13.12%
  lat (msec)   : 500=2.15%
  cpu          : usr=0.42%, sys=2.60%, ctx=10327, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5120,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):	#顺序写速率为66.1MiB/s ，共I/O操作5120MiB数据，运行77.482秒。
  WRITE: bw=66.1MiB/s (69.3MB/s), 66.1MiB/s-66.1MiB/s (69.3MB/s-69.3MB/s), io=5120MiB (5369MB), run=77482-77482msec

Disk stats (read/write):
  mmcblk0: ios=0/10232, merge=0/4, ticks=0/106644, in_queue=106769, util=100.00%

顺序读 fio -filename=/data/fio_test_ordread -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=1M -size=1G -numjobs=5 -group_reporting -name=mytest

# fio -filename=/data/fio_test_ordread -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=1M -size=1G -numjobs=5 -group_reporting -name=mytest
mytest: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
...
fio-3.35
Starting 5 threads
mytest: Laying out IO file (1 file / 1024MiB)
Jobs: 5 (f=5): [R(5)][100.0%][r=167MiB/s][r=167 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=5): err= 0: pid=1718: Mon Aug  7 18:53:10 2017
  read: IOPS=167, BW=167MiB/s (175MB/s)(5120MiB/30634msec)
    clat (usec): min=19427, max=38153, avg=29892.50, stdev=4558.43
     lat (usec): min=19430, max=38156, avg=29893.75, stdev=4558.32
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[23987], 10.00th=[23987], 20.00th=[23987],
     | 30.00th=[24773], 40.00th=[29754], 50.00th=[29754], 60.00th=[30016],
     | 70.00th=[33817], 80.00th=[35914], 90.00th=[35914], 95.00th=[35914],
     | 99.00th=[35914], 99.50th=[35914], 99.90th=[35914], 99.95th=[35914],
     | 99.99th=[38011]
   bw (  KiB/s): min=163510, max=174080, per=100.00%, avg=171331.97, stdev=902.72, samples=305
   iops        : min=  155, max=  170, avg=166.48, stdev= 0.97, samples=305
  lat (msec)   : 20=0.02%, 50=99.98%
  cpu          : usr=0.09%, sys=0.76%, ctx=5127, majf=0, minf=1280
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5120,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):	#顺序读速率为167MiB/s ，共I/O操作5120MiB数据，运行30.634秒。
   READ: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=5120MiB (5369MB), run=30634-30634msec

Disk stats (read/write):
  mmcblk0: ios=10227/0, merge=0/0, ticks=257561/0, in_queue=257561, util=99.83%

随机写 fio -filename=/data/fio_test_randw -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=1M -size=5G -numjobs=5 -group_reporting -name=mytest

# fio -filename=/data/fio_test_randw -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=1M -size=5G -numjobs=5 -group_reporting -name=mytest
mytest: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
...
fio-3.35
Starting 5 threads    #启用5个线程，对应“numjobs=5”
mytest: Laying out IO file (1 file / 5120MiB)
Jobs: 5 (f=5): [w(5)][99.8%][w=23.0MiB/s][w=23 IOPS][eta 00m:01s]
mytest: (groupid=0, jobs=5): err= 0: pid=1651: Mon Aug  7 18:34:55 2017
  write: IOPS=49, BW=49.4MiB/s (51.8MB/s)(25.0GiB/517785msec); 0 zone resets
    clat (msec): min=7, max=336, avg=100.86, stdev=71.81
     lat (msec): min=7, max=337, avg=101.09, stdev=71.77
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   42], 10.00th=[   43], 20.00th=[   46],
     | 30.00th=[   52], 40.00th=[   57], 50.00th=[   82], 60.00th=[   83],
     | 70.00th=[   90], 80.00th=[  184], 90.00th=[  239], 95.00th=[  255],
     | 99.00th=[  266], 99.50th=[  271], 99.90th=[  284], 99.95th=[  288],
     | 99.99th=[  338]
   bw (  KiB/s): min=10215, max=118734, per=100.00%, avg=50645.50, stdev=6021.63, samples=5167
   iops        : min=    5, max=  115, avg=47.89, stdev= 5.89, samples=5167
  lat (msec)   : 10=0.03%, 20=0.02%, 50=28.41%, 100=45.87%, 250=19.45%
  lat (msec)   : 500=6.23%
  cpu          : usr=0.30%, sys=1.63%, ctx=51655, majf=0, minf=0    #cpu使用情况
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%	#I/O深度情况
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25600,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):	#随机写 速率为49.4MiB/s，共I/O操作25GiB数据，运行517.785秒。
  WRITE: bw=49.4MiB/s (51.8MB/s), 49.4MiB/s-49.4MiB/s (51.8MB/s-51.8MB/s), io=25.0GiB (26.8GB), run=517785-517785msec

Disk stats (read/write):
  mmcblk0: ios=0/51248, merge=0/5, ticks=0/743625, in_queue=743778, util=100.00%

随机读

# fio -filename=/data/fio_test_randr -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=1M -size=5G -numjobs=5 -group_reporting -name=mytest
mytest: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
...
fio-3.35
Starting 5 threads
mytest: Laying out IO file (1 file / 5120MiB)
Jobs: 4 (f=4): [r(2),_(1),r(2)][99.4%][r=167MiB/s][r=167 IOPS][eta 00m:01s]
mytest: (groupid=0, jobs=5): err= 0: pid=1736: Mon Aug  7 19:00:02 2017
  read: IOPS=166, BW=167MiB/s (175MB/s)(25.0GiB/153732msec)
    clat (usec): min=6391, max=47451, avg=29965.34, stdev=5073.05
     lat (usec): min=6394, max=47454, avg=29966.54, stdev=5073.05
    clat percentiles (usec):
     |  1.00th=[17957],  5.00th=[17957], 10.00th=[23987], 20.00th=[29754],
     | 30.00th=[30016], 40.00th=[30016], 50.00th=[30016], 60.00th=[30016],
     | 70.00th=[30016], 80.00th=[35914], 90.00th=[35914], 95.00th=[35914],
     | 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min=153414, max=207179, per=100.00%, avg=170873.53, stdev=1759.68, samples=1531
   iops        : min=  147, max=  202, avg=166.46, stdev= 1.73, samples=1531
  lat (msec)   : 10=0.01%, 20=6.49%, 50=93.50%
  cpu          : usr=0.08%, sys=0.77%, ctx=25649, majf=0, minf=1280
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=25600,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):	#随机读 速率为167MiB/s，共I/O操作25GiB数据，运行153.732秒。
   READ: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=25.0GiB (26.8GB), run=153732-153732msec

Disk stats (read/write):
  mmcblk0: ios=51145/0, merge=0/0, ticks=1445831/0, in_queue=1445831, util=100.00%

随机读&写 50% fio -filename=/data/fio_test_randrw -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=50 -ioengine=psync -bs=1M -size=2G -numjobs=5 -group_reporting -name=mytest

# fio -filename=/data/fio_test_randrw -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=50 -ioengine=psync -bs=1M -size=2G -numjobs=5 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
...
fio-3.35
Starting 5 threads
mytest: Laying out IO file (1 file / 2048MiB)
Jobs: 4 (f=4): [m(4),_(1)][98.6%][r=67.1MiB/s,w=74.1MiB/s][r=67,w=74 IOPS][eta 00m:01s]
mytest: (groupid=0, jobs=5): err= 0: pid=1756: Mon Aug  7 19:06:00 2017
  read: IOPS=68, BW=68.5MiB/s (71.8MB/s)(5014MiB/73211msec)
    clat (usec): min=6113, max=47540, avg=12604.83, stdev=5299.80
     lat (usec): min=6114, max=47541, avg=12606.44, stdev=5299.79
    clat percentiles (usec):
     |  1.00th=[ 6521],  5.00th=[ 6587], 10.00th=[ 6587], 20.00th=[ 6718],
     | 30.00th=[ 7242], 40.00th=[12125], 50.00th=[12649], 60.00th=[12911],
     | 70.00th=[13042], 80.00th=[18220], 90.00th=[19006], 95.00th=[23987],
     | 99.00th=[30278], 99.50th=[30540], 99.90th=[31065], 99.95th=[31065],
     | 99.99th=[47449]
   bw (  KiB/s): min=10240, max=157749, per=100.00%, avg=70549.60, stdev=5970.51, samples=721
   iops        : min=   10, max=  154, avg=67.92, stdev= 5.87, samples=721
  write: IOPS=71, BW=71.4MiB/s (74.8MB/s)(5226MiB/73211msec); 0 zone resets
    clat (msec): min=7, max=114, avg=57.44, stdev=13.11
     lat (msec): min=7, max=114, avg=57.60, stdev=13.11
    clat percentiles (msec):
     |  1.00th=[   34],  5.00th=[   39], 10.00th=[   42], 20.00th=[   47],
     | 30.00th=[   50], 40.00th=[   54], 50.00th=[   57], 60.00th=[   60],
     | 70.00th=[   64], 80.00th=[   69], 90.00th=[   75], 95.00th=[   82],
     | 99.00th=[   90], 99.50th=[   95], 99.90th=[  108], 99.95th=[  112],
     | 99.99th=[  115]
   bw (  KiB/s): min=49131, max=92552, per=100.00%, avg=73276.53, stdev=1939.47, samples=724
   iops        : min=   47, max=   90, avg=70.59, stdev= 1.88, samples=724
  lat (msec)   : 10=15.42%, 20=31.24%, 50=17.71%, 100=35.50%, 250=0.13%
  cpu          : usr=0.32%, sys=1.14%, ctx=16027, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5014,5226,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
	#随机读 速率为68.5MiB/s，共I/O操作5014MiB数据，运行73.211秒。
   READ: bw=68.5MiB/s (71.8MB/s), 68.5MiB/s-68.5MiB/s (71.8MB/s-71.8MB/s), io=5014MiB (5258MB), run=73211-73211msec
   #随机写 速率为71.4MiB/s，共I/O操作5226MiB数据，运行73.211秒。
  WRITE: bw=71.4MiB/s (74.8MB/s), 71.4MiB/s-71.4MiB/s (74.8MB/s-74.8MB/s), io=5226MiB (5480MB), run=73211-73211msec
Disk stats (read/write):
  mmcblk0: ios=10071/10464, merge=0/0, ticks=109367/124484, in_queue=233859, util=100.00%