Methodology
/mnt/tmpfs/ contains trimmed linux sources. Large files where removed to reduce the size of the total storage to 5GB. /mnt/tmpfs/ is a tmpfs filesystem.
The following are the benchmarks done --
Copy operation --
time cp -a /mnt/tmpfs/* /mnt/temp/
Cold search --
time find /mnt/temp/ -iname '*a*' > /dev/null
Warm search --
time for i in {a..j}; do find /mnt/temp/ -iname "*$i*" > /dev/null; done
read all files in an alphabetic way (cold) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
read all files in an alphabetic way (warm) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
Write a certain small value to all files alphabetically (check for CPU utilization too of the script) --
cd /mnt/temp/
find /mnt/temp/ -type f > /tmp/flist.txt
dd if=/dev/urandom of=/tmp/write_data bs=1K count=6
time write_mulitple_files.rb /tmp/flist.txt /tmp/write_data
Delete dir tree --
time rm -rf /mnt/temp/*
time cp -a /mnt/tmpfs/* /mnt/temp/
Cold search --
time find /mnt/temp/ -iname '*a*' > /dev/null
Warm search --
time for i in {a..j}; do find /mnt/temp/ -iname "*$i*" > /dev/null; done
read all files in an alphabetic way (cold) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
read all files in an alphabetic way (warm) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
Write a certain small value to all files alphabetically (check for CPU utilization too of the script) --
cd /mnt/temp/
find /mnt/temp/ -type f > /tmp/flist.txt
dd if=/dev/urandom of=/tmp/write_data bs=1K count=6
time write_mulitple_files.rb /tmp/flist.txt /tmp/write_data
Delete dir tree --
time rm -rf /mnt/temp/*
HDD benchmarks
mount and mkfs options
mount paramters for xfs - -
mount -o logbufs=8,logbsize=256k,noquota,noatime
mount parameters for ext4 --
mount -o noatime,data=writeback,journal_async_commit,inode_readahead_blks=32768,max_batch_time=10000000,i_version,noquota,delalloc
nodelalloc was removed since bigalloc was removed.
ext4 is optimized for small + large files. It shouldnt make a difference in performance.
format parameters for xfs and ext4 --
mkfs.ext4 -g 256 -G 4 -J size=100 -m 1 -O none,extent,flex_bg,has_journal,large_file,^uninit_bg,dir_index,dir_nlink,^sparse_super,^sparse_super2 -i 4096
bigalloc had to be removed because of large no. of inodes (Expect worst performance with larger files, which this benchmark does not cover).
mkfs.xfs -f -m rmapbt=0,reflink=0
Results --
ext4 --
Create/copy --
0m27.925s
Cold search --
0m0.157s
Warm search --
0m1.509s
read all files in an alphabetic way (cold) (parallel) --
0m0.253s
read all files in an alphabetic way (warm) (parallel) --
0m0.252s
Write a certain small value to all files alphabetically in parallel --
11m41.727s
Delete dir tree --
0m1.161s
xfs --
Create/copy --
0m21.857s
Cold search --
0m0.081s
Warm search --
0m0.752s
read all files in an alphabetic way (cold) (parallel) --
0m0.239s
read all files in an alphabetic way (warm) (parallel) --
0m0.238s
Write a certain small value to all files alphabetically in parallel --
11m43.711s
Delete dir tree --
0m1.086s
Create/copy --
0m27.925s
Cold search --
0m0.157s
Warm search --
0m1.509s
read all files in an alphabetic way (cold) (parallel) --
0m0.253s
read all files in an alphabetic way (warm) (parallel) --
0m0.252s
Write a certain small value to all files alphabetically in parallel --
11m41.727s
Delete dir tree --
0m1.161s
xfs --
Create/copy --
0m21.857s
Cold search --
0m0.081s
Warm search --
0m0.752s
read all files in an alphabetic way (cold) (parallel) --
0m0.239s
read all files in an alphabetic way (warm) (parallel) --
0m0.238s
Write a certain small value to all files alphabetically in parallel --
11m43.711s
Delete dir tree --
0m1.086s
Conclusion --
Despite rmapbt being disabled in XFS (which improves performance with small files), XFS is faster than ext4 in most tests. If this ext4 FS (which is optimized for large files) is used for operations on large files, expect lower performance.
SSD benchmarks
mount and mkfs options
blkdiscard done before each benchmark.
mount paramters for xfs - -
mount -o logbufs=8,logbsize=256k,noquota,noatime
mount parameters for ext4 --
mount
-o
noatime,data=writeback,journal_async_commit,inode_readahead_blks=32768,max_batch_time=10000000,i_version,noquota,delalloc
nodelalloc was removed since bigalloc was removed.
ext4 is optimized for small + large files. It shouldnt make a difference in performance.
format parameters for xfs and ext4 --
mkfs.ext4
-g 256 -G 4 -J size=100 -m 1 -O
none,extent,flex_bg,has_journal,large_file,^uninit_bg,dir_index,dir_nlink,^sparse_super,^sparse_super2
-i 4096
bigalloc
had to be removed because of large no. of inodes (Expect worst
performance with larger files, which this benchmark does not cover).
xfs with no rmapbt --
mkfs.xfs -f -m rmapbt=0,reflink=0
xfs with rmapbt --
mkfs.xfs -f -m rmapbt=1,reflink=0
Results --
ext4 --
Copy operation --
time cp -a /mnt/tmpfs/* /mnt/temp/
real 0m48.826s
user 0m0.204s
sys 0m3.005s
real 0m48.290s
user 0m0.246s
sys 0m2.898s
Cold search --
time find /mnt/temp/ -iname '*a*' > /dev/null
real 0m0.172s
user 0m0.074s
sys 0m0.097s
real 0m0.169s
user 0m0.064s
sys 0m0.105s
Warm search --
time for i in {a..j}; do find /mnt/temp/ -iname "*$i*" > /dev/null; done
real 0m1.616s
user 0m0.536s
sys 0m1.075s
real 0m1.651s
user 0m0.615s
sys 0m1.031s
read all files in an alphabetic way (cold) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
real 0m0.444s
user 0m0.227s
sys 0m2.850s
real 0m0.402s
user 0m0.271s
sys 0m2.793s
read all files in an alphabetic way (warm) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
real 0m0.407s
user 0m0.230s
sys 0m2.851s
real 0m0.402s
user 0m0.223s
sys 0m2.845s
Write a certain small value to all files alphabetically (check for CPU utilization too of the script) --
cd /mnt/temp/
find -type f > /tmp/flist.txt
dd if=/dev/urandom of=/tmp/write_data bs=1K count=6
time /home/de/small/docs/Practice/Software/ruby/write_mulitple_files.rb /tmp/flist.txt /tmp/write_data
real 9m59.305s
user 9m53.748s
sys 0m51.903s
real 9m38.867s
user 9m33.476s
sys 0m49.930s
Delete dir tree --
time rm -rf /mnt/temp/*
real 0m0.824s
user 0m0.021s
sys 0m0.743s
real 0m0.820s
user 0m0.038s
sys 0m0.718s
xfs rmapbt=0
Copy operation --
time cp -a /mnt/tmpfs/* /mnt/temp/
real 0m14.851s
user 0m0.298s
sys 0m3.860s
Cold search --
time find /mnt/temp/ -iname '*a*' > /dev/null
real 0m0.082s
user 0m0.054s
sys 0m0.027s
Warm search --
time for i in {a..j}; do find /mnt/temp/ -iname "*$i*" > /dev/null; done
real 0m0.694s
user 0m0.511s
sys 0m0.179s
read all files in an alphabetic way (cold) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
real 0m0.389s
user 0m0.277s
sys 0m2.680s
read all files in an alphabetic way (warm) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
real 0m0.388s
user 0m0.256s
sys 0m2.705s
Write a certain small value to all files alphabetically (check for CPU utilization too of the script) --
cd /mnt/temp/
find /mnt/temp/ -type f > /tmp/flist.txt
dd if=/dev/urandom of=/tmp/write_data bs=1K count=6
time /home/de/small/docs/Practice/Software/ruby/write_mulitple_files.rb /tmp/flist.txt /tmp/write_data
real 10m45.878s
user 10m40.476s
sys 0m7.636s
Delete dir tree --
time rm -rf /mnt/temp/*
real 0m1.181s
user 0m0.030s
sys 0m0.482s
xfs rmapbt=1
Copy operation --
time cp -a /mnt/tmpfs/* /mnt/temp/
real 0m2.883s
user 0m0.159s
sys 0m2.556s
Cold search --
time find /mnt/temp/ -iname '*a*' > /dev/null
real 0m0.082s
user 0m0.049s
sys 0m0.033s
Warm search --
time for i in {a..j}; do find /mnt/temp/ -iname "*$i*" > /dev/null; done
real 0m0.700s
user 0m0.480s
sys 0m0.216s
read all files in an alphabetic way (cold) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
real 0m0.389s
user 0m0.218s
sys 0m2.752s
read all files in an alphabetic way (warm) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
real 0m0.389s
user 0m0.229s
sys 0m2.739s
Write a certain small value to all files alphabetically (check for CPU utilization too of the script) --
cd /mnt/temp/
find /mnt/temp/ -type f > /tmp/flist.txt
dd if=/dev/urandom of=/tmp/write_data bs=1K count=6
time /home/de/small/docs/Practice/Software/ruby/write_mulitple_files.rb /tmp/flist.txt /tmp/write_data
real 8m53.297s
user 8m48.394s
sys 0m9.786s
Delete dir tree --
time rm -rf /mnt/temp/*
real 0m2.373s
user 0m0.024s
sys 0m0.498s
Copy operation --
time cp -a /mnt/tmpfs/* /mnt/temp/
real 0m48.826s
user 0m0.204s
sys 0m3.005s
real 0m48.290s
user 0m0.246s
sys 0m2.898s
Cold search --
time find /mnt/temp/ -iname '*a*' > /dev/null
real 0m0.172s
user 0m0.074s
sys 0m0.097s
real 0m0.169s
user 0m0.064s
sys 0m0.105s
Warm search --
time for i in {a..j}; do find /mnt/temp/ -iname "*$i*" > /dev/null; done
real 0m1.616s
user 0m0.536s
sys 0m1.075s
real 0m1.651s
user 0m0.615s
sys 0m1.031s
read all files in an alphabetic way (cold) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
real 0m0.444s
user 0m0.227s
sys 0m2.850s
real 0m0.402s
user 0m0.271s
sys 0m2.793s
read all files in an alphabetic way (warm) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
real 0m0.407s
user 0m0.230s
sys 0m2.851s
real 0m0.402s
user 0m0.223s
sys 0m2.845s
Write a certain small value to all files alphabetically (check for CPU utilization too of the script) --
cd /mnt/temp/
find -type f > /tmp/flist.txt
dd if=/dev/urandom of=/tmp/write_data bs=1K count=6
time /home/de/small/docs/Practice/Software/ruby/write_mulitple_files.rb /tmp/flist.txt /tmp/write_data
real 9m59.305s
user 9m53.748s
sys 0m51.903s
real 9m38.867s
user 9m33.476s
sys 0m49.930s
Delete dir tree --
time rm -rf /mnt/temp/*
real 0m0.824s
user 0m0.021s
sys 0m0.743s
real 0m0.820s
user 0m0.038s
sys 0m0.718s
xfs rmapbt=0
Copy operation --
time cp -a /mnt/tmpfs/* /mnt/temp/
real 0m14.851s
user 0m0.298s
sys 0m3.860s
Cold search --
time find /mnt/temp/ -iname '*a*' > /dev/null
real 0m0.082s
user 0m0.054s
sys 0m0.027s
Warm search --
time for i in {a..j}; do find /mnt/temp/ -iname "*$i*" > /dev/null; done
real 0m0.694s
user 0m0.511s
sys 0m0.179s
read all files in an alphabetic way (cold) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
real 0m0.389s
user 0m0.277s
sys 0m2.680s
read all files in an alphabetic way (warm) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
real 0m0.388s
user 0m0.256s
sys 0m2.705s
Write a certain small value to all files alphabetically (check for CPU utilization too of the script) --
cd /mnt/temp/
find /mnt/temp/ -type f > /tmp/flist.txt
dd if=/dev/urandom of=/tmp/write_data bs=1K count=6
time /home/de/small/docs/Practice/Software/ruby/write_mulitple_files.rb /tmp/flist.txt /tmp/write_data
real 10m45.878s
user 10m40.476s
sys 0m7.636s
Delete dir tree --
time rm -rf /mnt/temp/*
real 0m1.181s
user 0m0.030s
sys 0m0.482s
xfs rmapbt=1
Copy operation --
time cp -a /mnt/tmpfs/* /mnt/temp/
real 0m2.883s
user 0m0.159s
sys 0m2.556s
Cold search --
time find /mnt/temp/ -iname '*a*' > /dev/null
real 0m0.082s
user 0m0.049s
sys 0m0.033s
Warm search --
time for i in {a..j}; do find /mnt/temp/ -iname "*$i*" > /dev/null; done
real 0m0.700s
user 0m0.480s
sys 0m0.216s
read all files in an alphabetic way (cold) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
real 0m0.389s
user 0m0.218s
sys 0m2.752s
read all files in an alphabetic way (warm) --
time find /mnt/temp/ -type f | xargs -d $'\n' -r -P 100 -n 300 -L 300 cat > /dev/null
real 0m0.389s
user 0m0.229s
sys 0m2.739s
Write a certain small value to all files alphabetically (check for CPU utilization too of the script) --
cd /mnt/temp/
find /mnt/temp/ -type f > /tmp/flist.txt
dd if=/dev/urandom of=/tmp/write_data bs=1K count=6
time /home/de/small/docs/Practice/Software/ruby/write_mulitple_files.rb /tmp/flist.txt /tmp/write_data
real 8m53.297s
user 8m48.394s
sys 0m9.786s
Delete dir tree --
time rm -rf /mnt/temp/*
real 0m2.373s
user 0m0.024s
sys 0m0.498s
Conclusion --
When comparing xfs rmapbt=1 and xfs rmapbt=0, rmapbt=1 wins on average (but not by a large margin).
When comparing xfs rmapbt=1 and ext4, xfs wins by a large margin
