raidz2: 10 disks vs 8 disks

Yesterday, I mentioned I wanted to compare raidz2 when using 10 disks versus using 8 disks. Along the way, I noticed that my ZFS-setup script needed a change to cope with mixed devices. Once I got past that hurdle, I started adding packages and setting up the system for some very simple tests.

The test

My test of preference is loading a database, and then dumping it.

$ createdb -E SQL_ASCII -T template0 freshports.org.testing
$ time pg_restore -j 8 -d freshports.org.testing freshports.org.dump.For.Testing
$ time pg_restore -j 8 -d freshports.org.testing freshports.org.dump.For.Testing
$ time pg_dump freshports.org.testing > freshports.org.dump

Each pg_restore was done into a different database.

Testing configurations

I used three configurations, all involving Toshiba 3TB DT01ACA300 HDD:

1. 8 HDD on the LSI card
2. 8 HDD on the LSI card, two on the mother board
3. 8 HDD on the LSI card, two on the smaller SATA card

NOTE: This differs from my initial decision to also test 8 HDD on the LSI card, one on the mother board, one on the smaller SATA card.

The LSI card is an 8-port LSI SATA/SAS 9211-8i 6Gb/s card. The smaller SATA card is a 2-port SATA card (HighPoint Rocket 620 2 SATA Port PCI-Express).

The results

The results are pretty simple. I did three (or more) database loads and three database dumps. Then averaged the times.

8 HDD on the LSI card

With raidz-2 with 8 HDD, all on the LSI card, the values for database load were:

1. 18m15.395s
2. 18m22.754s
3. 18m26.974s
4. 18m26.843s

The pg_dump times for each of these databases were:

1. 8m48.504s
2. 8m42.781s
3. 8m42.781s
4. 8m42.938s

8 HDD, two on the mother board

For raidz-2 with 10 HDD, eight on the LSI card, and two on the motherboard:

1. 18m13.562s
2. 18m26.107s
3. 18m24.061s

For pg_dump:

1. 9m6.544s
2. 9m7.623s
3. 9m8.501s

8 HDD, two on the smaller SATA card

For raidz-2 with 10 HDD, eight on the LSI card, and two on the smaller SATA card:

1. 18m17.284s
2. 18m32.162s
3. 18m28.050s

The pg_dump times for each of these databases were:

1. 9m16.837s
2. 9m13.830s
3. 9m15.795s

Comparing the results

The average times (pg_restore, pg_dump) for each configuration are:

1. 8 HDD: 0:18:22, 0:08:43
2. 8 HDD, two on the mother board: 0:18:21, 0:09:07
3. 8 HDD, two on the smaller SATA card: 0:18:25, 0:09:14

Conclusion

It looks like I should put the two extra HDD on the m/b.