FYI: This server has since gone to a new home.
After powering off the server about 8 months ago, I took the first steps to selling it. I opened it up and took out
- 2x NVMe sticks (1TB each, ZFS mirrored, giving a 930G zpool)
- INTEL fiber NIC (Intel X540-AT2)
I’ll be keeping those items.
I also took an inventory of the drives still in the system:
- 10 x 5TB drives – all Toshiba – in drive sleds
- 2x 250GB SSDs (SSD 860 EVO 250GB RVT02B6Q) – mounted inside the chassis – acting as boot drives connected directly to the M/B
- 2x 500SSD Samsung SSD 850 2B6Q – in drive sleds – I am considering retaining these and removing them from the chassis
The chassis is a Supermicro 846
Inside the chassis is:
- M/B – Supermicro X9DRE-TF+
- RAM – 128GB composed of 8x 16GB DDR3 1600Mhz 21300 ECC/REG
- SAS9300-8i card connects the backplane of the chassis with the M/B
Redeploying the removed hardware
The goals for the 2x NVMe sticks is to add space used in either the r730-01 or r730-03 hosts. Because they use PCIe slots, they should be able to go into either hosts, regardless of drive-bay availability.
On r730-03, the data02 zpool is at 83%:
[16:28 r730-03 dvl ~] % zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data01 21.8T 18.2T 3.58T - - 14% 83% 1.00x ONLINE - zroot 412G 4.72G 407G - - 6% 1% 1.00x ONLINE -
That zpool is a 4x12TB HDD array:
[16:43 r730-03 dvl ~] % zpool status data01 pool: data01 state: ONLINE scan: scrub repaired 0B in 21:21:21 with 0 errors on Mon Oct 16 00:59:01 2023 config: NAME STATE READ WRITE CKSUM data01 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/HGST_8CJVT8YE ONLINE 0 0 0 gpt/SG_ZHZ16KEX ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gpt/SG_ZHZ03BAT ONLINE 0 0 0 gpt/HGST_8CJW1G4E ONLINE 0 0 0 errors: No known data errors [16:43 r730-03 dvl ~] %
It looks like this:
data01 18.2T 3.45T 96K none data01/bacula-volumes 9.76T 3.45T 96K /jails/bacula-sd-04/usr/local/bacula/volumes data01/bacula-volumes/DiffFile 96.1G 928G 80.9G /jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile data01/bacula-volumes/FullFile 3.77T 2.23T 3.69T /jails/bacula-sd-04/usr/local/bacula/volumes/FullFile data01/bacula-volumes/FullFileNoNextPool 5.70T 3.45T 5.70T /jails/bacula-sd-04/usr/local/bacula/volumes/FullFileNoNextPool data01/bacula-volumes/IncrFile 205G 2.30T 103G /jails/bacula-sd-04/usr/local/bacula/volumes/IncrFile data01/dbclone.backups.rsyncer 3.10T 411G 121G /jails/dbclone/usr/home/rsyncer/backups data01/dbclone.postgres 273G 3.45T 1005M /jails/dbclone/var/db/postgres data01/jails 270G 3.45T 160K /jails data01/jails/ansible 14.3G 3.45T 13.1G /jails/ansible data01/jails/bacula-sd-01 10.3G 3.45T 10.3G /jails/bacula-sd-01 data01/jails/bacula-sd-04 2.00G 3.45T 1.72G /jails/bacula-sd-04 data01/jails/cliff1 11.2G 3.45T 10.8G /jails/cliff1 data01/jails/dbclone 20.4G 3.45T 18.1G /jails/dbclone data01/jails/empty 9.07G 3.45T 8.28G /jails/empty data01/jails/fruity-ext 10.9G 3.45T 10.5G /jails/fruity-ext data01/jails/fruity-int 11.5G 3.45T 11.1G /jails/fruity-int data01/jails/mysql57 14.4G 3.45T 14.4G /jails/mysql57 data01/jails/mysql80 9.14G 3.45T 9.12G /jails/mysql80 data01/jails/pg11 14.8G 3.45T 14.8G /jails/pg11 data01/jails/pg12 14.4G 3.45T 14.4G /jails/pg12 data01/jails/pg13 57.5G 3.45T 57.4G /jails/pg13 data01/jails/pg14 52.7G 3.45T 52.7G /jails/pg14 data01/jails/tc 1.21G 3.45T 1.13G /jails/tc data01/jails/testing-bacula9 612M 3.45T 597M /jails/testing-bacula9 data01/jails/timecapsule 1.07G 3.45T 1.04G /jails/timecapsule data01/jails/toiler 14.0G 3.45T 11.4G /jails/toiler data01/mkjail 3.02G 3.45T 96K /var/db/mkjail data01/mkjail/12.4-RELEASE 750M 3.45T 750M /var/db/mkjail/12.4-RELEASE data01/mkjail/13.2-RELEASE 1.50G 3.45T 1.50G /var/db/mkjail/13.2-RELEASE data01/mkjail/releases 801M 3.45T 801M /var/db/mkjail/releases data01/snapshots 286G 3.45T 96K none data01/snapshots/homeassistant-r730-01 286G 3.45T 21.8G none data01/timecapsule 4.56T 3.45T 96K /jails/timecapsule/usr/local/timecapsule data01/timecapsule/dvl-air01 1.26T 3.45T 949G /jails/timecapsule/usr/local/timecapsule/dvl-air01 data01/timecapsule/dvl-dent 641G 3.45T 641G /jails/timecapsule/usr/local/timecapsule/dvl-dent data01/timecapsule/dvl-dent-sparse 176G 3.45T 176G /jails/timecapsule/usr/local/timecapsule/dvl-dent-sparse data01/timecapsule/dvl-pro02 1.42T 3.45T 1.10T /jails/timecapsule/usr/local/timecapsule/dvl-pro02 data01/timecapsule/dvl-pro03 1.08T 3.45T 934G /jails/timecapsule/usr/local/timecapsule/dvl-pro03
I could use the 2x 500GB HDD for the data01/dbclone.postgres dataset shown above. Oh wait. I need to review that. The dataset is used to test restore databases from backups. At present, that dataset uses 273G – which would be 273G/398G or about 69% full on that pair of drives (the resulting zpool is 398G – see the tank_fast zpool shown in the server URL at the top of this post). However, the PostgreSQL database server is wipe clean before every test. The current data usage represents only the last test.
The database tests
My crontab:
[16:51 dbclone dan ~] % crontab -l
# use /bin/sh to run commands, overriding the default set by cron
SHELL=/bin/sh
# mail any output to `dan', no matter whose crontab this is
MAILTO=dan@langille.org
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:$HOME/bin; export PATH
# let's do that database testing during the night, except on First Sun
3 6 * * * /usr/bin/lockf -t 0 /tmp/.dan.lockf.test-backups-all.sh ${HOME}/bin/test-backups-all.sh | grep -v 'ERROR: language "plpgsql" already exists'
# it does not matter if the above database testing overlaps with this
# regression testing. The DB tests occurs on this host. The Bacula testing
# occurs on other hosts (e.g. pg10, pg12, etc)
#18 2 * * * ${HOME}/bin/bacula-regression-testing.sh > ${HOME}/logs/bacula-regression-testing.log
#11 3 * * * ${HOME}/bin/bacula-regression-testing-1.sh > ${HOME}/logs/bacula-regression-testing-1.log 2>&1
#11 3 * * * ${HOME}/bin/bacula-regression-testing-2.sh > ${HOME}/logs/bacula-regression-testing-2.log 2>&1
Here’s that script:
[16:52 dbclone dan ~] % cat ${HOME}/bin/test-backups-all.sh
#!/bin/sh
#
# test both sets of backups, serially
#
${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/aws-1 ${HOME}/logs/aws-1.dbtest.log
${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/bacula-database ${HOME}/logs/bacula-database.dbtest.log
${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/gelt/database-backup ${HOME}/logs/gelt.dbtest.log
# we get ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)
# let's try a sleep
sleep 200
${HOME}/bin/run-db-tests-mysql ~rsyncer/backups/mysql01/database-backup/ ${HOME}/logs/mysql01.dbtest.log
#${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/papers/database-backup ${HOME}/logs/papers.dbtest.log
${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/pg01/database-backup ${HOME}/logs/pg01.dbtest.log
${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/pg02/database-backup ${HOME}/logs/pg02.dbtest.log
${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/pg03/database-backup ${HOME}/logs/pg03.dbtest.log
#${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/supernews/database-backup ${HOME}/logs/supernews.dbtest.log
${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/tallboy/database-backup ${HOME}/logs/tallboy.dbtest.log
${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/x8dtu-pg01/database-backup ${HOME}/logs/x8dtu-pg01.dbtest.log
${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/x8dtu-pg02/database-backup ${HOME}/logs/x8dtu-pg02.dbtest.log
${HOME}/bin/run-db-tests-mysql ~rsyncer/backups/zuul-mysql/database-backup/ ${HOME}/logs/zuul-mysql.dbtest.log
${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/zuul-pg01/database-backup ${HOME}/logs/zuul-pg01.dbtest.log
${HOME}/bin/run-db-tests-postgresql ~rsyncer/backups/zuul-pg02/database-backup ${HOME}/logs/zuul-pg02.dbtest.log
[16:52 dbclone dan ~] %
Meaning, the database server contains only the databases from zuul-pg02. Specifically:
[16:58 dbclone dan ~] % psql postgres
psql (12.16)
Type "help" for help.
postgres=# \l+
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges | Size | Tablespace | Description
---------------------+----------+-----------+---------+---------+-----------------------+---------+------------+--------------------------------------------
glwd_bsdcan | postgres | SQL_ASCII | C.UTF-8 | C.UTF-8 | | 12 MB | pg_default |
postgres | postgres | UTF8 | C.UTF-8 | C.UTF-8 | | 8105 kB | pg_default | default administrative connection database
postgresqleu | postgres | SQL_ASCII | C.UTF-8 | C.UTF-8 | | 273 MB | pg_default |
postgresqleu_bsdcan | postgres | SQL_ASCII | C.UTF-8 | C.UTF-8 | | 410 MB | pg_default |
template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +| 7961 kB | pg_default | unmodifiable empty database
| | | | | postgres=CTc/postgres | | |
template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +| 8105 kB | pg_default | default template for new databases
| | | | | postgres=CTc/postgres | | |
(6 rows)
postgres=#
The \l+ command includes the size. Useful. However, there’s not much disk space in there. That’s less than a GB of space. What is taking up 273G?
Snapshots?
[16:57 r730-03 dvl ~] % zfs list -r -t snapshot data01/dbclone.postgres NAME USED AVAIL REFER MOUNTPOINT data01/dbclone.postgres@autosnap_2023-10-20_04:30:02_frequently 0B - 1003M - data01/dbclone.postgres@autosnap_2023-10-20_04:45:00_frequently 0B - 1003M - data01/dbclone.postgres@autosnap_2023-10-20_05:00:05_frequently 0B - 1003M - data01/dbclone.postgres@autosnap_2023-10-20_05:15:02_frequently 0B - 1003M - data01/dbclone.postgres@autosnap_2023-10-20_05:30:00_frequently 0B - 1003M - data01/dbclone.postgres@autosnap_2023-10-20_05:45:02_frequently 0B - 1003M - data01/dbclone.postgres@autosnap_2023-10-20_06:00:00_frequently 0B - 1003M - data01/dbclone.postgres@autosnap_2023-10-20_06:15:32_frequently 4.68G - 8.00G - data01/dbclone.postgres@autosnap_2023-10-20_06:30:03_frequently 14.6G - 17.9G - data01/dbclone.postgres@autosnap_2023-10-20_06:45:08_frequently 452M - 10.0G - data01/dbclone.postgres@autosnap_2023-10-20_07:00:13_frequently 418M - 20.4G - data01/dbclone.postgres@autosnap_2023-10-20_07:15:05_frequently 405M - 29.3G - data01/dbclone.postgres@autosnap_2023-10-20_07:30:07_frequently 438M - 39.7G - data01/dbclone.postgres@autosnap_2023-10-20_07:45:09_frequently 460M - 50.0G - data01/dbclone.postgres@autosnap_2023-10-20_08:00:08_frequently 420M - 60.2G - data01/dbclone.postgres@autosnap_2023-10-20_08:16:28_frequently 12.3G - 73.3G - data01/dbclone.postgres@autosnap_2023-10-20_08:32:34_frequently 17.3G - 86.3G - data01/dbclone.postgres@autosnap_2023-10-20_08:45:13_frequently 342M - 89.6G - data01/dbclone.postgres@autosnap_2023-10-20_09:00:36_frequently 348M - 101G - data01/dbclone.postgres@autosnap_2023-10-20_09:15:10_frequently 8.89G - 94.9G - data01/dbclone.postgres@autosnap_2023-10-20_09:30:03_frequently 16.5G - 102G - data01/dbclone.postgres@autosnap_2023-10-20_09:45:05_frequently 539M - 2.55G - data01/dbclone.postgres@autosnap_2023-10-20_10:00:07_frequently 7.85G - 9.88G - data01/dbclone.postgres@autosnap_2023-10-20_10:15:27_frequently 3.30G - 3.30G - data01/dbclone.postgres@autosnap_2023-10-20_10:30:06_frequently 1.62G - 3.40G - data01/dbclone.postgres@autosnap_2023-10-20_10:45:01_frequently 10.1G - 11.9G - data01/dbclone.postgres@autosnap_2023-10-20_11:00:02_hourly 0B - 921M - data01/dbclone.postgres@autosnap_2023-10-20_11:00:02_frequently 0B - 921M - data01/dbclone.postgres@autosnap_2023-10-20_11:15:01_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_11:30:03_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_11:45:01_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_12:00:01_hourly 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_12:00:01_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_12:15:01_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_12:30:02_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_12:45:02_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_13:00:00_hourly 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_13:00:00_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_13:15:02_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_13:30:03_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_13:45:01_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_14:00:04_hourly 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_14:00:04_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_14:15:02_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_14:30:00_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_14:45:01_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_15:00:02_hourly 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_15:00:02_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_15:15:03_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_15:30:01_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_15:45:02_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_16:00:02_hourly 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_16:00:02_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_16:15:00_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_16:30:00_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_16:45:02_frequently 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_17:00:00_hourly 0B - 1005M - data01/dbclone.postgres@autosnap_2023-10-20_17:00:00_frequently 0B - 1005M - [17:03 r730-03 dvl ~] %
This dataset does not need snapshots. This data is transitory. It comes. It goes. It is not required to stay around, except for the test duration. Let’s turn off snapshots and remove the existing items. Not shown: I modified /usr/local/etc/sanoid/sanoid.conf. This is how I deleted the snapshots:
[17:10 r730-03 dvl ~] % zfs list -r -t snapshot data01/dbclone.postgres | cut -f 1 -w | grep data01/dbclone.postgres@autosnap | xargs -n 1 zfs destroy -nrv would destroy data01/dbclone.postgres@autosnap_2023-10-20_04:30:02_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_04:45:00_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_05:00:05_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_05:15:02_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_05:30:00_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_05:45:02_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_06:00:00_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_06:15:32_frequently would reclaim 4.68G would destroy data01/dbclone.postgres@autosnap_2023-10-20_06:30:03_frequently would reclaim 14.6G would destroy data01/dbclone.postgres@autosnap_2023-10-20_06:45:08_frequently would reclaim 452M would destroy data01/dbclone.postgres@autosnap_2023-10-20_07:00:13_frequently would reclaim 418M would destroy data01/dbclone.postgres@autosnap_2023-10-20_07:15:05_frequently would reclaim 405M would destroy data01/dbclone.postgres@autosnap_2023-10-20_07:30:07_frequently would reclaim 438M would destroy data01/dbclone.postgres@autosnap_2023-10-20_07:45:09_frequently would reclaim 460M would destroy data01/dbclone.postgres@autosnap_2023-10-20_08:00:08_frequently would reclaim 420M would destroy data01/dbclone.postgres@autosnap_2023-10-20_08:16:28_frequently would reclaim 12.3G would destroy data01/dbclone.postgres@autosnap_2023-10-20_08:32:34_frequently would reclaim 17.3G would destroy data01/dbclone.postgres@autosnap_2023-10-20_08:45:13_frequently would reclaim 342M would destroy data01/dbclone.postgres@autosnap_2023-10-20_09:00:36_frequently would reclaim 348M would destroy data01/dbclone.postgres@autosnap_2023-10-20_09:15:10_frequently would reclaim 8.89G would destroy data01/dbclone.postgres@autosnap_2023-10-20_09:30:03_frequently would reclaim 16.5G would destroy data01/dbclone.postgres@autosnap_2023-10-20_09:45:05_frequently would reclaim 539M would destroy data01/dbclone.postgres@autosnap_2023-10-20_10:00:07_frequently would reclaim 7.85G would destroy data01/dbclone.postgres@autosnap_2023-10-20_10:15:27_frequently would reclaim 3.30G would destroy data01/dbclone.postgres@autosnap_2023-10-20_10:30:06_frequently would reclaim 1.62G would destroy data01/dbclone.postgres@autosnap_2023-10-20_10:45:01_frequently would reclaim 10.1G would destroy data01/dbclone.postgres@autosnap_2023-10-20_11:00:02_hourly would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_11:00:02_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_11:15:01_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_11:30:03_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_11:45:01_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_12:00:01_hourly would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_12:00:01_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_12:15:01_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_12:30:02_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_12:45:02_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_13:00:00_hourly would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_13:00:00_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_13:15:02_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_13:30:03_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_13:45:01_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_14:00:04_hourly would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_14:00:04_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_14:15:02_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_14:30:00_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_14:45:01_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_15:00:02_hourly would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_15:00:02_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_15:15:03_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_15:30:01_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_15:45:02_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_16:00:02_hourly would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_16:00:02_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_16:15:00_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_16:30:00_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_16:45:02_frequently would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_17:00:00_hourly would reclaim 0B would destroy data01/dbclone.postgres@autosnap_2023-10-20_17:00:00_frequently would reclaim 0B
That was a dry run. Changing the zfs command to the following deleted the snapshots: sudo zfs destroy -rv
Now how much space do we have:
[17:13 r730-03 dvl ~] % zfs list -r data01/dbclone.postgres NAME USED AVAIL REFER MOUNTPOINT data01/dbclone.postgres 1005M 3.72T 1005M /jails/dbclone/var/db/postgres
Now we’re using only 1GB – which is way closer to what the psql command shows. And to what this command shows too:
[17:01 dbclone dan /var/db/postgres] % sudo du -ch -d 1 . 1.0G ./data12 1.0G . 1.0G total
But how much space is needed for this?
The above was an interesting exercise, but how much space does the database testing need?
Here are the largest database backups:
[17:16 dbclone dan ~rsyncer/backups] % ls -lhS `find . -type f ` -rw-r----- 1 rsyncer rsyncer 148G 2023.10.20 04:04 ./bacula-database/postgresql/bacula.dump -rw-r----- 1 rsyncer rsyncer 13G 2023.03.17 09:37 ./tallboy/database-backup/postgresql/pentabarf_pgcon.dump -rw-r--r-- 1 rsyncer rsyncer 13G 2023.04.12 00:37 ./papers/database-backup/postgresql/pentabarf_pgcon.dump -rw-r----- 1 rsyncer rsyncer 8.9G 2023.03.17 09:28 ./tallboy/database-backup/postgresql/pentabarf_bsdcan.dump -rw-r--r-- 1 rsyncer rsyncer 8.9G 2023.04.12 00:26 ./papers/database-backup/postgresql/pentabarf_bsdcan.dump -rw-r--r-- 1 rsyncer rsyncer 3.3G 2023.10.20 02:18 ./aws-1/postgresql/freshports.org.dump -rw-r--r-- 1 rsyncer rsyncer 3.3G 2023.10.16 14:08 ./aws-1/postgresql/archive/freshports.org.dump -rw-r--r-- 1 rsyncer rsyncer 3.3G 2023.10.20 02:14 ./pg02/database-backup/postgresql/freshports.devgit.dump -rw-r--r-- 1 rsyncer rsyncer 3.2G 2020.08.21 02:14 ./x8dtu-pg02/database-backup/postgresql/freshports.org.dump -rw-r--r-- 1 rsyncer rsyncer 713M 2023.10.20 02:02 ./zuul-pg02/database-backup/postgresql/postgresqleu_bsdcan.dump -rw-r--r-- 1 rsyncer rsyncer 374M 2023.10.20 02:02 ./zuul-pg02/database-backup/postgresql/postgresqleu.dump -rw-r--r-- 1 rsyncer rsyncer 69M 2023.10.20 02:02 ./zuul-mysql/database-backup/mysql/wordpress_danlangilleorg.sql -rw-r--r-- 1 rsyncer rsyncer 36M 2023.10.20 02:02 ./mysql01/database-backup/mysql/librenms.sql -rw-r--r-- 1 rsyncer rsyncer 31M 2023.10.20 02:14 ./pg02/database-backup/postgresql/samdrucker.dump -rw-r--r-- 1 rsyncer rsyncer 21M 2023.10.20 02:02 ./zuul-mysql/database-backup/mysql/wordpress_newsfreshportsorg.sql -rw-r--r-- 1 rsyncer rsyncer 16M 2023.10.20 02:02 ./mysql01/database-backup/mysql/db_nagiosql_v34.sql
By far, Bacula is the largest database. I keep a lot of backup history in there. Let’s go see how much space is used on the database server which hosts that database:
[16:56 pg02 dan ~] % psql bacula
psql (12.16)
Type "help" for help.
bacula=# \l+
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges | Size | Tablespace | Description
------------------------+--------------+-----------+---------+-------+-----------------------+---------+------------+--------------------------------------------
bacula | bacula | SQL_ASCII | C | C | | 309 GB | pg_default |
empty | dan | UTF8 | C | C | | 7969 kB | pg_default |
fpphorum | postgres | SQL_ASCII | C | C | | 12 MB | pg_default |
freebsddiary.old | postgres | SQL_ASCII | C | C | | 35 MB | pg_default |
freebsddiary.org | postgres | SQL_ASCII | C | C | =Tc/postgres +| 35 MB | pg_default |
| | | | | postgres=CTc/postgres+| | |
| | | | | rsyncer=c/postgres | | |
freshports.devgit | postgres | SQL_ASCII | C | C | | 51 GB | pg_default |
freshports.mobile | dan | SQL_ASCII | C | C | | 35 GB | pg_default |
freshports.stagegit | postgres | SQL_ASCII | C | C | | 52 GB | pg_default |
freshports.testgit | postgres | SQL_ASCII | C | C | | 48 GB | pg_default |
freshports.testgit.old | postgres | SQL_ASCII | C | C | | 50 GB | pg_default |
fsphorum | postgres | SQL_ASCII | C | C | | 8921 kB | pg_default |
gitea | git | UTF8 | C | C | | 21 MB | pg_default |
keycloak | keycloak | UTF8 | C | C | | 12 MB | pg_default |
nagiostest | dan | UTF8 | C | C | | 8089 kB | pg_default |
postgres | postgres | UTF8 | C | C | | 8153 kB | pg_default | default administrative connection database
postgresqleu | postgresqleu | UTF8 | C | C | | 20 MB | pg_default |
samdrucker | dan | UTF8 | C | C | | 245 MB | pg_default |
template0 | postgres | UTF8 | C | C | =c/postgres +| 7937 kB | pg_default | unmodifiable empty database
| | | | | postgres=CTc/postgres | | |
template1 | postgres | UTF8 | C | C | =c/postgres +| 8089 kB | pg_default | default template for new databases
| | | | | postgres=CTc/postgres | | |
(19 rows)
bacula=#
Line 9 shows about 310 GB for Bacula. Let’s look at the dataset used for that database server:
[16:39 r730-01 dvl ~] % zfs list data03/pg02/postgres NAME USED AVAIL REFER MOUNTPOINT data03/pg02/postgres 761G 5.52T 553G /jails/pg02/var/db/postgres [17:20 r730-01 dvl ~] %
That’s about 552GB. Which matches up with this:
[17:20 r730-01 dvl ~] % sudo du -ch -d 1 /jails/pg02/var/db/postgres 553G /jails/pg02/var/db/postgres/data12 553G /jails/pg02/var/db/postgres 553G total
But that’s all the database on pg02, not just Bacula. Let’s check the math: 310/398 = 78% full. That’s pretty close to the magic 80% free which I try to achieve. I think this dataset is unique though. It fills up, it empties out, repeatedly.
First, I need to find out IF the two PCIe cards will fit into r730-03. If not, then it’s the 2x 500GB SSDs.
BONUS!
I also need to dispose of another server. I was looking through the contents. I found a pair of 1TB SSDs. I could combine those with the 2 1TB NVMe and create a 2TB pool. That’d be a nice addition to r730-03. I would combine 1 NVMe and one SSD into a mirror, repeat with the other pair, and then stripe across the two. That should be maximize resilience across two different brands of storage devices. Yes, one device in each mirror will be vastly faster than the other. I’m OK with that.
Inserting the 2x 1TB SSDs
Late today, I mounted those two 2.5″ SSDs in a pair of 3.5″ drive sleds. Here they are:
Oct 20 20:53:17 r730-03 kernel: mrsas0: System PD created target ID: 0x7 Oct 20 20:53:17 r730-03 kernel: da7 at mrsas0 bus 1 scbus1 target 7 lun 0 Oct 20 20:53:17 r730-03 kernel: da7:Fixed Direct Access SPC-4 SCSI device Oct 20 20:53:17 r730-03 kernel: da7: Serial Number S3Z8NB0KB11776R Oct 20 20:53:17 r730-03 kernel: da7: 150.000MB/s transfers Oct 20 20:53:17 r730-03 kernel: da7: 953869MB (1953525168 512 byte sectors) Oct 20 21:00:23 r730-03 kernel: mrsas0: System PD created target ID: 0x6 Oct 20 21:00:23 r730-03 kernel: da8 at mrsas0 bus 1 scbus1 target 6 lun 0 Oct 20 21:00:23 r730-03 kernel: da8: Fixed Direct Access SPC-4 SCSI device Oct 20 21:00:23 r730-03 kernel: da8: Serial Number S3Z8NB0KB11784L Oct 20 21:00:23 r730-03 kernel: da8: 150.000MB/s transfers Oct 20 21:00:23 r730-03 kernel: da8: 953869MB (1953525168 512 byte sectors)
Interesting, yet disappointing that these are connected to a 150.000MB/s bus.
Perhaps I should move dbclone over to r730-01. No, they are all 150.000MB/s too. :(
They are clearly still “in use”:
[21:02 r730-03 dvl ~] % gpart show da7 da8
=> 40 1953525088 da7 GPT (932G)
40 1953525088 1 freebsd-zfs (932G)
=> 40 1953525088 da8 GPT (932G)
40 1953525088 1 freebsd-zfs (932G)
[21:03 r730-03 dvl ~] %
Let’s try zpool labelclear:
[21:06 r730-03 dvl ~] % sudo zpool labelclear /dev/da7 failed to clear label for /dev/da7
Oh, by device, the man page means this:
[21:06 r730-03 dvl ~] % sudo zpool labelclear /dev/da7p1 use '-f' to override the following error: /dev/da7p1 is a member of potentially active pool "tank_fast01" [21:06 r730-03 dvl ~] % zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data01 21.8T 15.1T 6.68T - - 11% 69% 1.00x ONLINE - zroot 412G 4.73G 407G - - 6% 1% 1.00x ONLINE -
Nice warning. Thanks. It’s not imported. I know it’s from another host which I am decommissioning. Let’s do this:
[21:06 r730-03 dvl ~] % sudo zpool labelclear -f /dev/da7p1 [21:06 r730-03 dvl ~] %
On to the next:
[21:06 r730-03 dvl ~] % sudo zpool labelclear /dev/da8p1
use '-f' to override the following error:
/dev/da8p1 is a member of potentially active pool "tank_fast01"
[21:06 r730-03 dvl ~] % sudo zpool labelclear -f /dev/da8p1
[21:06 r730-03 dvl ~] % gpart show da7 da8
=> 40 1953525088 da7 GPT (932G)
40 1953525088 1 freebsd-zfs (932G)
=> 40 1953525088 da8 GPT (932G)
40 1953525088 1 freebsd-zfs (932G)
[21:07 r730-03 dvl ~] %
OK, good. That’s expected. This works on the labels, not on the partitions. These are ready to go.
Tomorrow, I’ll install the NVMe3 drives. I think I can move some drive cages from one host to another. We shall see.
