Tonight I received a TOSHIBA DT01ACA300 3TB HDD. This is a 7200 RPM SATA III drive. Tests were run with FreeBSD 9.1 in the hardware listed below.
The hardware
We are testing on the following hardware:
- motherboard – SUPERMICRO MBD-H8SGL-O ATX Server Motherboard (Supermicro link): $224.99
- CPU – AMD Opteron 6128 Magny-Cours 2.0GHz 8 x 512KB L2 Cache 12MB L3 Cache Socket G34 115W 8-Core Server : $284.99
- RAM – Kingston 8GB 240-Pin DDR3 SDRAM ECC Registered DDR3 1600 Server Memory : 4 x $64.99 = $259.96
- PSU – PC Power and Cooling Silencer MK III 600W power supply : $99.99
- SATA card – LSI Internal SATA/SAS 9211-8i 6Gb/s PCI-Express 2.0 RAID Controller Card, Kit (LSI page): $319.99
- HDD for ZFS – Seagate Barracuda ST2000DM001 2TB 7200 RPM 64MB : 8 x $109.99 = $879.92
The drive being tested is not part of the base OS.
The devices
The LSI card:
mps0: <LSI SAS2008> port 0x8000-0x80ff mem 0xfde3c000-0xfde3ffff,0xfde40000-0xfde7ffff irq 28 at device 0.0 on pci1 mps0: Firmware: 14.00.01.00, Driver: 14.00.00.01-fbsd mps0: IOCCapabilities: 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>
The drives:
da0 at mps0 bus 0 scbus0 target 4 lun 0 da0: <ATA TOSHIBA DT01ACA3 ABB0< Fixed Direct Access SCSI-6 device da0: 600.000MB/s transfers da0: Command Queueing enabled da0: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
dd to raw device
root@heckler:/home/dan # dd if=/dev/zero of=/dev/da0 bs=32k count=300000 300000+0 records in 300000+0 records out 9830400000 bytes transferred in 53.709288 secs (183029796 bytes/sec) root@heckler:/home/dan # dd if=/dev/zero of=/dev/da0 bs=64k count=300000 300000+0 records in 300000+0 records out 19660800000 bytes transferred in 106.232741 secs (185072887 bytes/sec) root@heckler:/home/dan # dd if=/dev/zero of=/dev/da0 bs=128k count=300000 300000+0 records in 300000+0 records out 39321600000 bytes transferred in 212.234953 secs (185273912 bytes/sec) root@heckler:/home/dan # dd if=/dev/zero of=/dev/da0 bs=256k count=300000 300000+0 records in 300000+0 records out 78643200000 bytes transferred in 423.508192 secs (185694637 bytes/sec) root@heckler:/home/dan # dd if=/dev/zero of=/dev/da0 bs=1024k count=300000 300000+0 records in 300000+0 records out 314572800000 bytes transferred in 1695.812390 secs (185499765 bytes/sec)
That’s 177 MB/s.
The diskinfo
Here’s diskinfo:
# diskinfo -tv /dev/da0 /dev/da0 512 # sectorsize 3000592982016 # mediasize in bytes (2.7T) 5860533168 # mediasize in sectors 4096 # stripesize 0 # stripeoffset 364801 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. Z2T5TSSAS # Disk ident. Seek times: Full stroke: 250 iter in 6.302236 sec = 25.209 msec Half stroke: 250 iter in 4.395477 sec = 17.582 msec Quarter stroke: 500 iter in 7.240961 sec = 14.482 msec Short forward: 400 iter in 1.999334 sec = 4.998 msec Short backward: 400 iter in 2.338969 sec = 5.847 msec Seq outer: 2048 iter in 0.163964 sec = 0.080 msec Seq inner: 2048 iter in 0.172619 sec = 0.084 msec Transfer rates: outside: 102400 kbytes in 0.580453 sec = 176414 kbytes/sec middle: 102400 kbytes in 0.652515 sec = 156931 kbytes/sec inside: 102400 kbytes in 1.098502 sec = 93218 kbytes/sec
phybs
Next, we run phybs:
# ./phybs -rw -l 1024 /dev/da0 count size offset step msec tps kBps 131072 1024 0 4096 59527 2201 2201 131072 1024 512 4096 59320 2209 2209 65536 2048 0 8192 31212 2099 4199 65536 2048 512 8192 31780 2062 4124 65536 2048 1024 8192 31896 2054 4109 32768 4096 0 16384 11575 2830 11322 32768 4096 512 16384 26017 1259 5037 32768 4096 1024 16384 26197 1250 5003 32768 4096 2048 16384 26188 1251 5004 16384 8192 0 32768 9464 1731 13849 16384 8192 512 32768 21142 774 6199 16384 8192 1024 32768 23422 699 5595 16384 8192 2048 32768 22764 719 5757 16384 8192 4096 32768 10493 1561 12491
dd to the filesystem
After partitioning and newfs’ing, we have:
# gpart show da0 da0s1
=> 63 4294967232 da0 MBR (2.7T)
63 945 - free - (472k)
1008 4294966221 1 freebsd (2T)
4294967229 66 - free - (33k)
=> 0 4294966221 da0s1 BSD (2T)
0 4294966216 1 freebsd-ufs (2T)
4294966216 5 - free - (2.5k)
You’ll notice that this is a 2TB slice. I suspect that it the maximum for MBR.
Next, the dd:
[dan@heckler:/mnt/dan] $ dd if=/dev/zero of=testing32 bs=32k count=300000 300000+0 records in 300000+0 records out 9830400000 bytes transferred in 61.175199 secs (160692571 bytes/sec) [dan@heckler:/mnt/dan] $ dd if=/dev/zero of=testing64 bs=64k count=300000 300000+0 records in 300000+0 records out 19660800000 bytes transferred in 122.130356 secs (160982091 bytes/sec) [dan@heckler:/mnt/dan] $ dd if=/dev/zero of=testing128 bs=128k count=300000 300000+0 records in 300000+0 records out 39321600000 bytes transferred in 253.067559 secs (155379853 bytes/sec) [dan@heckler:/mnt/dan] $ dd if=/dev/zero of=testing256 bs=256k count=300000 300000+0 records in 300000+0 records out 78643200000 bytes transferred in 536.249214 secs (146654201 bytes/sec) [dan@heckler:/mnt/dan] $ dd if=/dev/zero of=testing512 bs=512k count=300000 300000+0 records in 300000+0 records out 157286400000 bytes transferred in 989.688996 secs (158925077 bytes/sec) $ dd if=/dev/zero of=testing1024 bs=1024k count=300000 300000+0 records in 300000+0 records out 314572800000 bytes transferred in 1923.917453 secs (163506391 bytes/sec)
That ranges from 139-153 MB/s.
bonnie++
And finally, a quick bonnie++:
[dan@heckler:/mnt/dan] $ bonnie++ -s 66000
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heckler.unix 66000M 530 99 152464 29 58137 45 1036 98 154698 24 197.2 12
Latency 39965us 284ms 8428ms 67529us 883ms 2959ms
Version 1.97 ------Sequential Create------ --------Random Create--------
heckler.unixathome. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 43000us 58us 41us 46987us 36us 41us
1.97,1.97,heckler.unixathome.org,1,1360161593,66000M,,530,99,152464,29,58137,45,1036,98,154698,24,197.2,12,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,39965us,284ms,8428ms,67529us,883ms,2959ms,43000us,58us,41us,46987us,36us,41us
Trying a different partitioning
After running those tests, and not being happy with them, I tried again.
I created a small script to automate these tests:
$ cat ~/ddtest
#!/bin/sh
SIZES="32 64 128 256 1024 2048"
for size in ${SIZES}
do
CMD="dd if=/dev/zero of=testing${size} bs=${size}k count=10k"
echo ${CMD}
`${CMD}`
done
The disk looks like this:
# gpart show da0
=> 34 5860533101 da0 GPT (2.7T)
34 6 - free - (3.0k)
40 5860533088 1 freebsd (2.7T)
5860533128 7 - free - (3.5k)
Now for the first set of dd test?
$ time ~/ddtest dd if=/dev/zero of=testing32 bs=32k count=10k 10240+0 records in 10240+0 records out 335544320 bytes transferred in 1.804411 secs (185957793 bytes/sec) dd if=/dev/zero of=testing64 bs=64k count=10k 10240+0 records in 10240+0 records out 671088640 bytes transferred in 3.559059 secs (188557878 bytes/sec) dd if=/dev/zero of=testing128 bs=128k count=10k 10240+0 records in 10240+0 records out 1342177280 bytes transferred in 7.622590 secs (176078906 bytes/sec) dd if=/dev/zero of=testing256 bs=256k count=10k 10240+0 records in 10240+0 records out 2684354560 bytes transferred in 15.043288 secs (178442011 bytes/sec) dd if=/dev/zero of=testing1024 bs=1024k count=10k 10240+0 records in 10240+0 records out 10737418240 bytes transferred in 61.112390 secs (175699531 bytes/sec) dd if=/dev/zero of=testing2048 bs=2048k count=10k 10240+0 records in 10240+0 records out 21474836480 bytes transferred in 122.675087 secs (175054585 bytes/sec) real 3m32.239s user 0m0.072s sys 1m1.035s
That is 166-180 MB/s and much faster than the previous partitioning scheme.
And for completeness:
$ df -h /mnt Filesystem Size Used Avail Capacity Mounted on /dev/da0 2.7T 69G 2.4T 3% /mnt
You might want to add to your benchmarks the tests I did with dd, use ‘iseek’ and read from 1TB and 2TB in to the disk, disks are faster at the start of the drive, and slower at the end of the drive
Does that come out in the diskinfo output? Not quite….?
After the fact, I realized, I’m doing this in the file system, not to a raw device.