I’m still seeing a 50% performance hit when I try bonnie++. It’s pretty clear. One system is getting 456MB/s sequential read. While the other is doing 265MB/s.
Both machines are using the same type of HDD: The 3TB Toshiba DT01ACA300.
The faster server
This is on bulldog, the faster machine but destined to have fewer HDD. All HDD are directly connected to the motherboard.
$ bonnie++
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
floater.unixath 32G 207 99 467039 55 255923 33 511 97 600814 27 281.3 2
Latency 40235us 190ms 1007ms 87779us 139ms 207ms
Version 1.96 ------Sequential Create------ --------Random Create--------
floater.unixathome. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ 32634 82 +++++ +++ +++++ +++ +++++ +++
Latency 10107us 76us 82us 20472us 25us 51us
1.96,1.96,floater.unixathome.org,1,1363988277,32G,,207,99,467039,55
The above server:
- motherboard – SUPERMICRO MBD-X9SCL+-F Micro ATX Server Motherboard LGA 1155 Intel C202 DDR3 1333 (manufacturer page) – $179.99
- CPU – Intel Xeon E3-1230 V2 Ivy Bridge 3.3GHz (3.7GHz Turbo) 4 x 256KB L2 Cache 8MB L3 Cache LGA 1155 69W Quad-Core Server Processor – $239.99
- RAM – Samsung DDR3-1600 8GB/1Gx72 ECC Samsung Chip Server Memory – M391B1G73BH0-CK0 (16GB for $145.40)
Here is the dmesg output:
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.1-STABLE #0 r247249: Mon Feb 25 01:57:17 UTC 2013
root@floater.unixathome.org:/usr/obj/usr/src/sys/FLOATER amd64
CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (3300.10-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x306a9 Family = 0x6 Model = 0x3a Stepping = 9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
Standard Extended Features=0x281<GSFSBASE,SMEP,ENHMOVSB>
TSC: P-state invariant, performance statistics
real memory = 17188257792 (16392 MB)
avail memory = 16495628288 (15731 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <SUPERM SMCI--MB>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
cpu4 (AP): APIC ID: 4
cpu5 (AP): APIC ID: 5
cpu6 (AP): APIC ID: 6
cpu7 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
ctl: CAM Target Layer loaded
acpi0: <SUPERM SMCI--MB> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 67, 1 (4) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xdfc04000-0xdfc043ff irq 16 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
ahci0: <Marvell 88SE912x AHCI SATA controller> port 0xe090-0xe097,0xe080-0xe083,0xe070-0xe077,0xe060-0xe063,0xe050-0xe05f mem 0xdfb21000-0xdfb217ff irq 16 at device 0.0 on pci1
ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
atapci0: <Marvell 88SE912x UDMA133 controller> port 0xe040-0xe047,0xe030-0xe033,0xe020-0xe027,0xe010-0xe013,0xe000-0xe00f mem 0xdfb20000-0xdfb2000f irq 17 at device 0.1 on pci1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xd000-0xd01f mem 0xdfa00000-0xdfa1ffff,0xdfa20000-0xdfa23fff irq 16 at device 0.0 on pci2
em0: Using MSIX interrupts with 3 vectors
em0: Ethernet address: 00:25:90:a4:cd:d4
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci3: <ACPI PCI bus> on pcib3
em1: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xc000-0xc01f mem 0xdf900000-0xdf91ffff,0xdf920000-0xdf923fff irq 18 at device 0.0 on pci3
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: 00:25:90:a4:cd:d5
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xdfc03000-0xdfc033ff irq 23 at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci1
pcib4: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci4: <ACPI PCI bus> on pcib4
vgapci0: <VGA-compatible display> mem 0xde000000-0xdeffffff,0xdf800000-0xdf803fff,0xdf000000-0xdf7fffff irq 23 at device 3.0 on pci4
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci1: <Intel Cougar Point AHCI SATA controller> port 0xf050-0xf057,0xf040-0xf043,0xf030-0xf037,0xf020-0xf023,0xf000-0xf01f mem 0xdfc02000-0xdfc027ff irq 19 at device 31.2 on pci0
ahci1: AHCI v1.30 with 6 3Gbps ports, Port Multiplier not supported
ahcich2: <AHCI channel> at channel 0 on ahci1
ahcich3: <AHCI channel> at channel 1 on ahci1
ahcich4: <AHCI channel> at channel 2 on ahci1
ahcich5: <AHCI channel> at channel 3 on ahci1
ahcich6: <AHCI channel> at channel 4 on ahci1
ahcich7: <AHCI channel> at channel 5 on ahci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 10 on acpi0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ada0 at ahcich2 bus 0 scbus2 target 0 lun 0
ada0: <WDC WD5000AAKS-00YGA0 12.01C02> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad8
ada1 at ahcich3 bus 0 scbus3 target 0 lun 0
ada1: <Hitachi HDS723030BLE640 MX6OAAB0> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad10
ada2 at ahcich4 bus 0 scbus4 target 0 lun 0
ada2: <Hitachi HDS723030BLE640 MX6OAAB0> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad12
ada3 at ahcich5 bus 0 scbus5 target 0 lun 0
ada3: <Hitachi HDS723030BLE640 MX6OAAB0> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad14
ada4 at ahcich6 bus 0 scbus6 target 0 lun 0
ada4: <Hitachi HDS723030BLE640 MX6OAAB0> ATA-8 SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada4: Previously was known as ad16
ada5 at ahcich7 bus 0 scbus7 target 0 lun 0
ada5: <Hitachi HDS723030BLE640 MX6OAAB0> ATA-8 SATA 3.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada5: Previously was known as ad18
SMP: AP CPU #1 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #7 Launched!
Timecounter "TSC-low" frequency 1650049314 Hz quality 1000
Root mount waiting for: usbus1 usbus0
uhub1: 2 ports with 2 removable, self powered
uhub0: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1 usbus0
ugen0.2: <vendor 0x8087> at usbus0
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
Root mount waiting for: usbus1 usbus0
uhub3: 6 ports with 6 removable, self powered
uhub2: 6 ports with 6 removable, self powered
ugen0.3: <Winbond Electronics Corp> at usbus0
ums0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
ums0: error reading report description
device_attach: ums0 attach returned 12
ukbd0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
kbd2 at ukbd0
Trying to mount root from ufs:/dev/ada0p2 [rw]...
The slower server
The following is the slower server, heckler:
$ bonnie++
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heckler.unixath 64G 100 99 272276 89 173252 61 236 98 517897 83 226.4 12
Latency 82766us 359ms 1171ms 100ms 308ms 223ms
Version 1.97 ------Sequential Create------ --------Random Create--------
heckler.unixathome. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 17783 91 +++++ +++ 16989 91 17895 92 +++++ +++ 15928 92
Latency 20653us 154us 216us 26320us 71us 184us
1.97,1.97,heckler.unixathome.org,1,1363990862,64G,,100,99,272276,89,173252,61,236,98,517897,83,226.4,12,16,,,,,17783,91,+++++,+++,16989,91,17895,92,+++++,+++,15928,92,82766us,359ms,1171ms,100ms,308ms,223ms,20653us,154us,216us,26320us,71us,184us
This server contains:
- motherboard – SUPERMICRO MBD-H8SGL-O ATX Server Motherboard (Supermicro link): $224.99
- CPU – AMD Opteron 6128 Magny-Cours 2.0GHz 8 x 512KB L2 Cache 12MB L3 Cache Socket G34 115W 8-Core Server : $284.99
- CPU Cooler – Noctua NH-U9DO A3 AMD Opteron, 4 Dual Heat-pipe SSO Bearing Quiet CPU Cooler : $76.99
- RAM – Kingston 8GB 240-Pin DDR3 SDRAM ECC Registered DDR3 1600 Server Memory : 4 x $64.99 = $259.96
- SATA card – LSI Internal SATA/SAS 9211-8i 6Gb/s PCI-Express 2.0 RAID Controller Card, Kit (LSI page): $319.99
All drives in this server are connected to the SATA card. The card has been flashed with IT firmware (vs IR firmware).
I think my next test should be without the SATA card on this server.
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.1-STABLE #0 r246949: Tue Feb 19 01:26:27 UTC 2013
root@heckler.unixathome.org:/usr/obj/usr/src/sys/HECKLER amd64
CPU: AMD Opteron(tm) Processor 6128 (2000.05-MHz K8-class CPU)
Origin = "AuthenticAMD" Id = 0x100f91 Family = 0x10 Model = 0x9 Stepping = 1
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x802009<SSE3,MON,CX16,POPCNT>
AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
AMD Features2=0x837ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,NodeId>
TSC: P-state invariant
real memory = 34359738368 (32768 MB)
avail memory = 33105543168 (31571 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <083112 APIC1535>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 core(s)
cpu0 (BSP): APIC ID: 16
cpu1 (AP): APIC ID: 17
cpu2 (AP): APIC ID: 18
cpu3 (AP): APIC ID: 19
cpu4 (AP): APIC ID: 20
cpu5 (AP): APIC ID: 21
cpu6 (AP): APIC ID: 22
cpu7 (AP): APIC ID: 23
ioapic0 <Version 2.1> irqs 0-23 on motherboard
ioapic1 <Version 2.1> irqs 24-55 on motherboard
kbd1 at kbdmux0
acpi0: <SMCI > on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of fec00000, 1000 (3) failed
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of ffb80000, 80000 (3) failed
acpi0: reservation of fec10000, 20 (3) failed
acpi0: reservation of ffe50000, e060 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dff00000 (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 52 at device 3.0 on pci0
pci1: <ACPI PCI bus> on pcib1
mps0: <LSI SAS2008> port 0x8000-0x80ff mem 0xfde3c000-0xfde3ffff,0xfde40000-0xfde7ffff irq 28 at device 0.0 on pci1
mps0: Firmware: 15.00.00.00, Driver: 14.00.00.01-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib2: <ACPI PCI-PCI bridge> irq 52 at device 4.0 on pci0
pci2: <ACPI PCI bus> on pcib2
ahci0: <Marvell 88SE912x AHCI SATA controller> port 0xa000-0xa007,0x9800-0x9803,0xb000-0xb007,0xa800-0xa803,0xa400-0xa40f mem 0xfe8cf800-0xfe8cffff irq 44 at device 0.0 on pci2
ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
atapci0: <Marvell 88SE912x UDMA133 controller> port 0xb400-0xb407,0xc800-0xc803,0xc400-0xc407,0xc000-0xc003,0xb800-0xb80f mem 0xfe8efc00-0xfe8efc0f irq 45 at device 0.1 on pci2
pcib3: <ACPI PCI-PCI bridge> irq 53 at device 9.0 on pci0
pci3: <ACPI PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xd800-0xd81f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 48 at device 0.0 on pci3
em0: Using MSIX interrupts with 3 vectors
em0: Ethernet address: 00:25:90:82:21:5a
pcib4: <ACPI PCI-PCI bridge> irq 54 at device 10.0 on pci0
pci4: <ACPI PCI bus> on pcib4
em1: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xe800-0xe81f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 47 at device 0.0 on pci4
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: 00:25:90:82:21:5b
ahci1: <ATI IXP700 AHCI SATA controller> port 0x7000-0x7007,0x6000-0x6003,0x5000-0x5007,0x3000-0x3003,0x2000-0x200f mem 0xfebfbc00-0xfebfbfff irq 22 at device 17.0 on pci0
ahci1: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported
ahcich2: <AHCI channel> at channel 0 on ahci1
ahcich3: <AHCI channel> at channel 1 on ahci1
ahcich4: <AHCI channel> at channel 2 on ahci1
ahcich5: <AHCI channel> at channel 3 on ahci1
ohci0: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfebf9000-0xfebf9fff irq 16 at device 18.0 on pci0
usbus0 on ohci0
ohci1: <OHCI (generic) USB controller> mem 0xfebfa000-0xfebfafff irq 16 at device 18.1 on pci0
usbus1 on ohci1
ehci0: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfebfb800-0xfebfb8ff irq 17 at device 18.2 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci0
ohci2: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfebf7000-0xfebf7fff irq 18 at device 19.0 on pci0
usbus3 on ohci2
ohci3: <OHCI (generic) USB controller> mem 0xfebf8000-0xfebf8fff irq 18 at device 19.1 on pci0
usbus4 on ohci3
ehci1: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfebfb400-0xfebfb4ff irq 19 at device 19.2 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci1
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
atapci1: <ATI IXP700/800 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0
ata0: <ATA channel> at channel 0 on atapci1
ata1: <ATA channel> at channel 1 on atapci1
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib5: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci5: <ACPI PCI bus> on pcib5
vgapci0: <VGA-compatible display> mem 0xfc000000-0xfcffffff,0xfdffc000-0xfdffffff,0xfe000000-0xfe7fffff irq 20 at device 4.0 on pci5
ohci4: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfebf6000-0xfebf6fff irq 18 at device 20.5 on pci0
usbus6 on ohci4
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xcc800-0xcd7ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
ctl: CAM Target Layer loaded
acpi_throttle0: <ACPI CPU Throttling> on cpu0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 480Mbps High Speed USB v2.0
usbus6: 12Mbps Full Speed USB v1.0
ugen0.1: <ATI> at usbus0
uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <ATI> at usbus1
uhub1: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <ATI> at usbus2
uhub2: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ugen3.1: <ATI> at usbus3
uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <ATI> at usbus4
uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <ATI> at usbus5
uhub5: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
ugen6.1: <ATI> at usbus6
uhub6: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
uhub6: 2 ports with 2 removable, self powered
uhub0: 3 ports with 3 removable, self powered
uhub1: 3 ports with 3 removable, self powered
uhub3: 3 ports with 3 removable, self powered
uhub4: 3 ports with 3 removable, self powered
ada0 at ahcich4 bus 0 scbus5 target 0 lun 0
ada0: <ST3000DM001-1CH166 CC24> ATA-8 SATA 3.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad12
ada1 at ahcich5 bus 0 scbus6 target 0 lun 0
ada1: <TOSHIBA DT01ACA300 MX6OABB0> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad14
ada2 at ata0 bus 0 scbus7 target 0 lun 0
ada2: <ST250DM000-1BD141 KC45> ATA-8 SATA 3.x device
ada2: 150.000MB/s transfers (SATA, UDMA6, PIO 8192bytes)
ada2: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad0
ada3 at ata0 bus 0 scbus7 target 1 lun 0
ada3: <WDC WD2500AAKX-08ERMA0 19.01H19> ATA-8 SATA 3.x device
ada3: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
ada3: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad1
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #5 Launched!
Timecounter "TSC" frequency 2000047250 Hz quality 800
da3 at mps0 bus 0 scbus0 target 12 lun 0
da3: <ATA TOSHIBA DT01ACA3 ABB0> Fixed Direct Access SCSI-6 device
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da1 at mps0 bus 0 scbus0 target 10 lun 0
da1: <ATA TOSHIBA DT01ACA3 ABB0> Fixed Direct Access SCSI-6 device
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da2 at mps0 bus 0 scbus0 target 11 lun 0
da2: <ATA TOSHIBA DT01ACA3 ABB0> Fixed Direct Access SCSI-6 device
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da4 at mps0 bus 0 scbus0 target 13 lun 0
da4: <ATA TOSHIBA DT01ACA3 ABB0> Fixed Direct Access SCSI-6 device
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da0 at mps0 bus 0 scbus0 target 2 lun 0
da0: <ATA TOSHIBA DT01ACA3 ABB0> Fixed Direct Access SCSI-6 device
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
GEOM_MIRROR: Device mirror/gm0 launched (2/2).
uhub5: 6 ports with 6 removable, self powered
uhub2: 6 ports with 6 removable, self powered
Trying to mount root from ufs:/dev/mirror/gm0s1a [rw]...
Update 25 Mar 2013
I tried bonnie on a drive directly attached to the motherboard:
$ bonnie++
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heckler.unixath 64G 99 99 123784 36 61262 19 235 98 173133 24 210.6 8
Latency 84273us 2553ms 3523ms 209ms 275ms 263ms
Version 1.97 ------Sequential Create------ --------Random Create--------
heckler.unixathome. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 17075 96 +++++ +++ 16721 97 16091 96 +++++ +++ 14822 94
Latency 20138us 154us 200us 41522us 52us 14055us
1.97,1.97,heckler.unixathome.org,1,1364215751,64G,,99,99,123784,36,61262,19,235,98,173133,24,210.6,8,16,,,,,17075,96,+++++,+++,16721,97,16091,96,+++++,+++,14822,94,84273us,2553ms,3523ms,209ms,275ms,263ms,20138us,154us,200us,41522us,52us,14055us
Yes folks, that’s a mere 266MB/s.
What is the disk controller? got the dmesg somewhere?
I just added dmesg output for both boxes.
Typical hardware raid controllers start by turning off write-cache on all the attached drives in the firmware unless they have a backup battery unit attached to them. Makes it safer in a power fail situation, but at the expense of performance. Freebsd seems to have write-cache on by default for directly connected devices, but I would bet that doesn’t include your drives connected to the LSI board.
Might be worth checking.
Can write-cache be checked with smartctl?
write cache is disabled on the drives and enabled on the controller… i.e. on a 3ware with BBU (backup battery unit) if you do a BBU test it tells you that cache is disabled while that is running.
i.e. normal operation is that the os does a write… the controller puts it into cache… immediately reports the write complete to the OS and then sometime in the future it flushes the cache to the disk. When the BBU isn’t present or is drained ..
example here from a 3ware controller I am having the BBU replaced on…. when this dies the cache on the controller will go off and my performance drops by 40% or so…
/c0/bbu> show all
/c0/bbu Firmware Version = BBU: 2.00.00.003
/c0/bbu Serial Number = xxxxxxxxx
/c0/bbu Online State = On
/c0/bbu BBU Ready = Yes
/c0/bbu BBU Status = Weak Battery
/c0/bbu Battery Voltage = OK
/c0/bbu Battery Temperature = OK
/c0/bbu Estimated Backup Capacity = 255 Hours
/c0/bbu Last Capacity Test = 19-Oct-2011
/c0/bbu Battery Installation Date = 18-Jul-2006
/c0/bbu Bootloader Version = BBU 0.02.02.001
/c0/bbu PCB Revision = 65
On Solaris you check drive cache with ‘format -e’ then you can drop through the menus and do a display on write cache…
on freebsd 9.1
> camcontrol identify ada0 | grep cache
write cache yes yes
flush cache yes yes
I don’t have an LSI controller to play with, but mptutil is supposed to be able to tell you details of the setup with one of those…
I fear I cannot use mputil because the LSI 9211 shows up as an mps(4) device. And from man mps I see: No userland utility available (e.g. mptutil(8)).
For what it’s worth, I have one Toshiba drive not on the LSI card:
$ sudo camcontrol identify ada1 | grep cache
write cache yes yes
flush cache yes yes
I don’t suppose there is any control that tells you the cache settings for the LSI card?
I know on the 3ware you can get to it from tw_cli, but you can also get to it at boot through the 3ware bios.
From the chipset reported in your dmesg it seems that you should be able to use ‘sas2ircu’ which is a LSI equivalent of the tw_cli and it is released for FreeBSD?
Anyways, that is my 2 cents worth…
According to post #2 in this FreeBSD Forums thread, the card has no cache settings.
I located the sas2ircu utility at http://www.lsi.com/support/Pages/Download-Results.aspx?keyword=sas2ircu and downloaded it. Sadly, it does nothing for me:
$ ./sas2ircu list
LSI Corporation SAS2 IR Configuration Utility.
Version 15.00.00.00 (2012.11.08)
Copyright (c) 2009-2012 LSI Corporation. All rights reserved.
SAS2IRCU: No Controllers Found.
I only have one final thought then… have you tried the LSI card in a different PCI slot?
From what I can see in the dmesg you have it in slot 1… which is a standard PCI slot from the motherboard manual …
mps0: <LSI SAS2008> port 0x8000-0x80ff mem 0xfde3c000-0xfde3ffff,0xfde40000-0xfde7ffff irq 28 at device 0.0 on pci1
from the specs it is a pcie2 x8 lane card… which means it technically should go in PCI slot 5 to match up. Looks like you have a generic VGA adapter in that slot that could be moved to a different position.