x8dtu

Leave a Comment / hardware / By Dan Langille

This is x8dtu (named after the SuperMicro motherboard). This is the old but new FreshPorts server, which currently running on AWS, not this host.

The previous post for this is still available

In short:

FreeBSD 13.0 booting off a mirrored pair of zfsroot SSDs
4.5TB of mirrored ZFS
196612 MB of RAM (yeah, that’s 196GB of RAM)
SuperMicro X8DTU motherboard
Intel Xeon E5620 @ 2.40GHz (two of those, giving 16 CPUs)

NOTE: this post replaces an older version.

camcontrol devlist

$ sudo camcontrol devlist
  at scbus0 target 0 lun 0 (ada0,pass0)
  at scbus1 target 0 lun 0 (ada1,pass1)
         at scbus2 target 0 lun 0 (ada2,pass2)
         at scbus3 target 0 lun 0 (ada3,pass3)
               at scbus4 target 0 lun 0 (cd0,pass4)
   at scbus6 target 0 lun 0 (ses0,pass5)
$

/etc/fstab

$ cat /etc/fstab
# Device		Mountpoint	FStype	Options		Dump	Pass#
/dev/ada0p2		none	swap	sw		0	0
/dev/ada1p2		none	swap	sw		0	0

fdescfs /dev/fd  fdescfs  rw  0  0
$

gpart show

$ gpart show
=>       40  488397088  ada0  GPT  (233G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048   41943040     2  freebsd-swap  (20G)
   41945088  446451712     3  freebsd-zfs  (213G)
  488396800        328        - free -  (164K)

=>       40  488397088  ada1  GPT  (233G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048   41943040     2  freebsd-swap  (20G)
   41945088  446451712     3  freebsd-zfs  (213G)
  488396800        328        - free -  (164K)

=>         6  1220942635  ada2  GPT  (4.5T)
           6  1220280320     1  freebsd-zfs  (4.5T)
  1220280326      662315        - free -  (2.5G)

=>         6  1220942635  ada3  GPT  (4.5T)
           6  1220280320     1  freebsd-zfs  (4.5T)
  1220280326      662315        - free -  (2.5G)

$

zpool list

$ zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
main_tank  4.53T  26.3G  4.51T        -         -    19%     0%  1.00x    ONLINE  -
zroot       212G  29.1G   183G        -         -     7%    13%  1.00x    ONLINE  -
$

zpool status

$ zpool status
  pool: main_tank
 state: ONLINE
  scan: scrub repaired 0B in 00:05:44 with 0 errors on Mon Sep 13 03:08:34 2021
config:

	NAME        STATE     READ WRITE CKSUM
	main_tank   ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    ada2p1  ONLINE       0     0     0
	    ada3p1  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:00:43 with 0 errors on Mon Sep 13 03:03:39 2021
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    ada1p3  ONLINE       0     0     0
	    ada0p3  ONLINE       0     0     0

errors: No known data errors
$

zpool get all zroot

$ zpool get all zroot
NAME   PROPERTY                       VALUE                          SOURCE
zroot  size                           212G                           -
zroot  capacity                       13%                            -
zroot  altroot                        -                              default
zroot  health                         ONLINE                         -
zroot  guid                           18320603570228782289           -
zroot  version                        -                              default
zroot  bootfs                         zroot/ROOT/default             local
zroot  delegation                     on                             default
zroot  autoreplace                    off                            default
zroot  cachefile                      -                              default
zroot  failmode                       wait                           default
zroot  listsnapshots                  off                            default
zroot  autoexpand                     off                            default
zroot  dedupratio                     1.00x                          -
zroot  free                           183G                           -
zroot  allocated                      29.1G                          -
zroot  readonly                       off                            -
zroot  ashift                         0                              default
zroot  comment                        -                              default
zroot  expandsize                     -                              -
zroot  freeing                        0                              -
zroot  fragmentation                  7%                             -
zroot  leaked                         0                              -
zroot  multihost                      off                            default
zroot  checkpoint                     -                              -
zroot  load_guid                      374103634417328703             -
zroot  autotrim                       off                            default
zroot  feature@async_destroy          enabled                        local
zroot  feature@empty_bpobj            active                         local
zroot  feature@lz4_compress           active                         local
zroot  feature@multi_vdev_crash_dump  enabled                        local
zroot  feature@spacemap_histogram     active                         local
zroot  feature@enabled_txg            active                         local
zroot  feature@hole_birth             active                         local
zroot  feature@extensible_dataset     active                         local
zroot  feature@embedded_data          active                         local
zroot  feature@bookmarks              enabled                        local
zroot  feature@filesystem_limits      enabled                        local
zroot  feature@large_blocks           active                         local
zroot  feature@large_dnode            enabled                        local
zroot  feature@sha512                 enabled                        local
zroot  feature@skein                  enabled                        local
zroot  feature@userobj_accounting     active                         local
zroot  feature@encryption             enabled                        local
zroot  feature@project_quota          active                         local
zroot  feature@device_removal         enabled                        local
zroot  feature@obsolete_counts        enabled                        local
zroot  feature@zpool_checkpoint       enabled                        local
zroot  feature@spacemap_v2            active                         local
zroot  feature@allocation_classes     enabled                        local
zroot  feature@resilver_defer         enabled                        local
zroot  feature@bookmark_v2            enabled                        local
zroot  feature@redaction_bookmarks    enabled                        local
zroot  feature@redacted_datasets      enabled                        local
zroot  feature@bookmark_written       enabled                        local
zroot  feature@log_spacemap           active                         local
zroot  feature@livelist               enabled                        local
zroot  feature@device_rebuild         enabled                        local
zroot  feature@zstd_compress          enabled                        local
zroot  feature@draid                  enabled                        local
$

zpool get all main_tank

$ zpool get all main_tank
NAME       PROPERTY                       VALUE                          SOURCE
main_tank  size                           4.53T                          -
main_tank  capacity                       0%                             -
main_tank  altroot                        -                              default
main_tank  health                         ONLINE                         -
main_tank  guid                           376274468418206170             -
main_tank  version                        -                              default
main_tank  bootfs                         -                              default
main_tank  delegation                     on                             default
main_tank  autoreplace                    off                            default
main_tank  cachefile                      -                              default
main_tank  failmode                       wait                           default
main_tank  listsnapshots                  off                            default
main_tank  autoexpand                     off                            default
main_tank  dedupratio                     1.00x                          -
main_tank  free                           4.51T                          -
main_tank  allocated                      26.3G                          -
main_tank  readonly                       off                            -
main_tank  ashift                         0                              default
main_tank  comment                        -                              default
main_tank  expandsize                     -                              -
main_tank  freeing                        0                              -
main_tank  fragmentation                  19%                            -
main_tank  leaked                         0                              -
main_tank  multihost                      off                            default
main_tank  checkpoint                     -                              -
main_tank  load_guid                      17692580810121648431           -
main_tank  autotrim                       off                            default
main_tank  feature@async_destroy          enabled                        local
main_tank  feature@empty_bpobj            active                         local
main_tank  feature@lz4_compress           active                         local
main_tank  feature@multi_vdev_crash_dump  enabled                        local
main_tank  feature@spacemap_histogram     active                         local
main_tank  feature@enabled_txg            active                         local
main_tank  feature@hole_birth             active                         local
main_tank  feature@extensible_dataset     active                         local
main_tank  feature@embedded_data          active                         local
main_tank  feature@bookmarks              enabled                        local
main_tank  feature@filesystem_limits      enabled                        local
main_tank  feature@large_blocks           enabled                        local
main_tank  feature@large_dnode            enabled                        local
main_tank  feature@sha512                 enabled                        local
main_tank  feature@skein                  active                         local
main_tank  feature@userobj_accounting     active                         local
main_tank  feature@encryption             enabled                        local
main_tank  feature@project_quota          active                         local
main_tank  feature@device_removal         enabled                        local
main_tank  feature@obsolete_counts        enabled                        local
main_tank  feature@zpool_checkpoint       enabled                        local
main_tank  feature@spacemap_v2            active                         local
main_tank  feature@allocation_classes     enabled                        local
main_tank  feature@resilver_defer         enabled                        local
main_tank  feature@bookmark_v2            enabled                        local
main_tank  feature@redaction_bookmarks    enabled                        local
main_tank  feature@redacted_datasets      enabled                        local
main_tank  feature@bookmark_written       enabled                        local
main_tank  feature@log_spacemap           active                         local
main_tank  feature@livelist               enabled                        local
main_tank  feature@device_rebuild         enabled                        local
main_tank  feature@zstd_compress          enabled                        local
main_tank  feature@draid                  enabled                        local
$

/var/run/dmesg.boot

$ cat /var/run/dmesg.boot
---<>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-RELEASE-p4 #0: Tue Aug 24 07:33:27 UTC 2021
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe)
VT(vga): resolution 640x480
CPU microcode: updated from 0x15 to 0x1f
CPU: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz (2400.14-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
  Features=0xbfebfbff
  Features2=0x29ee3ff
  AMD Features=0x2c100800
  AMD Features2=0x1
  Structured Extended Features3=0x9c000000
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 171802886144 (163844 MB)
avail memory = 167267131392 (159518 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <080312 APIC1521>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 hardware threads
random: unblocking device.
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 128/64 (20201113/tbfadt-748)
ioapic0: MADT APIC ID 6 != hw id 4
ioapic1: MADT APIC ID 7 != hw id 6
ioapic0  irqs 0-23
ioapic1  irqs 24-47
Launching APs: 1 12 5 3 13 9 4 7 6 14 11 15 10 2 8
Timecounter "TSC-low" frequency 1200067710 Hz quality 1000
KTLS: Initialized 16 threads
random: entropy device external interface
[ath_hal] loaded
WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 14.0.
kbd1 at kbdmux0
000.000070 [4354] netmap_init               netmap: loaded module
mlx5en: Mellanox Ethernet driver 3.6.0 (December 2020)
nexus0
vtvga0: 
cryptosoft0: 
aesni0: 
acpi0: 
acpi0: Power Button (fixed)
cpu0:  on acpi0
attimer0:  port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0:  port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0:  iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer

zfs list

$ zfs list
NAME                                                                     USED  AVAIL     REFER  MOUNTPOINT
main_tank                                                               26.3G  4.36T       96K  none
main_tank/freshports                                                    6.62G  4.36T       96K  none
main_tank/freshports/ingress01                                          4.16G  4.36T       96K  none
main_tank/freshports/ingress01/var                                      4.16G  4.36T       96K  none
main_tank/freshports/ingress01/var/db                                   4.16G  4.36T       96K  none
main_tank/freshports/ingress01/var/db/freshports                         656K  4.36T       96K  /jails/ingress01/var/db/freshports
main_tank/freshports/ingress01/var/db/freshports/cache                   344K  4.36T       96K  /jails/ingress01/var/db/freshports/cache
main_tank/freshports/ingress01/var/db/freshports/cache/html              152K  4.36T       96K  /jails/ingress01/var/db/freshports/cache/html
main_tank/freshports/ingress01/var/db/freshports/cache/spooling           96K  4.36T       96K  /jails/ingress01/var/db/freshports/cache/spooling
main_tank/freshports/ingress01/var/db/freshports/message-queues          120K  4.36T      120K  /jails/ingress01/var/db/freshports/message-queues
main_tank/freshports/ingress01/var/db/freshports/repos                    96K  4.36T       96K  /jails/ingress01/var/db/freshports/repos
main_tank/freshports/ingress01/var/db/ingress                           4.16G  4.36T       96K  /jails/ingress01/var/db/ingress
main_tank/freshports/ingress01/var/db/ingress/message-queues             104K  4.36T      104K  /jails/ingress01/var/db/ingress/message-queues
main_tank/freshports/ingress01/var/db/ingress/repos                     4.16G  4.36T     4.16G  /jails/ingress01/var/db/ingress/repos
main_tank/freshports/jailed                                             2.46G  4.36T       96K  none
main_tank/freshports/jailed/ingress01                                   2.46G  4.36T       96K  none
main_tank/freshports/jailed/ingress01/jails                             2.13G  4.36T     1.64G  /jails
main_tank/freshports/jailed/ingress01/jails/freshports                   505M  4.36T      505M  /jails/freshports
main_tank/freshports/jailed/ingress01/mkjail                             333M  4.36T      333M  /var/db/mkjail
main_tank/freshports/jailed/nginx01                                     1.38M  4.36T       96K  none
main_tank/freshports/jailed/nginx01/var                                 1.28M  4.36T       96K  none
main_tank/freshports/jailed/nginx01/var/db                              1.19M  4.36T       96K  none
main_tank/freshports/jailed/nginx01/var/db/freshports                   1.09M  4.36T       96K  none
main_tank/freshports/jailed/nginx01/var/db/freshports/cache                1M  4.36T      160K  /var/db/freshports/cache
main_tank/freshports/jailed/nginx01/var/db/freshports/cache/categories    96K  4.36T       96K  /var/db/freshports/cache/categories
main_tank/freshports/jailed/nginx01/var/db/freshports/cache/commits       96K  4.36T       96K  /var/db/freshports/cache/commits
main_tank/freshports/jailed/nginx01/var/db/freshports/cache/daily         96K  4.36T       96K  /var/db/freshports/cache/daily
main_tank/freshports/jailed/nginx01/var/db/freshports/cache/general       96K  4.36T       96K  /var/db/freshports/cache/general
main_tank/freshports/jailed/nginx01/var/db/freshports/cache/news          96K  4.36T       96K  /var/db/freshports/cache/news
main_tank/freshports/jailed/nginx01/var/db/freshports/cache/packages      96K  4.36T       96K  /var/db/freshports/cache/packages
main_tank/freshports/jailed/nginx01/var/db/freshports/cache/pages         96K  4.36T       96K  /var/db/freshports/cache/pages
main_tank/freshports/jailed/nginx01/var/db/freshports/cache/ports         96K  4.36T       96K  /var/db/freshports/cache/ports
main_tank/freshports/jailed/nginx01/var/db/freshports/cache/spooling      96K  4.36T       96K  /var/db/freshports/cache/spooling
main_tank/freshports/nginx01                                             480K  4.36T       96K  none
main_tank/freshports/nginx01/var                                         384K  4.36T       96K  none
main_tank/freshports/nginx01/var/db                                      288K  4.36T       96K  none
main_tank/freshports/nginx01/var/db/freshports                           192K  4.36T       96K  none
main_tank/freshports/nginx01/var/db/freshports/cache                      96K  4.36T       96K  none
main_tank/home                                                          13.7G  4.36T     13.7G  /usr/home
main_tank/jails                                                         5.27G  4.36T      104K  /jails
main_tank/jails/ingress01                                               1.71G  4.36T     1.71G  /jails/ingress01
main_tank/jails/nginx01                                                 1.58G  4.36T     1.58G  /jails/nginx01
main_tank/jails/pg01                                                    1.98G  4.36T     1.98G  /jails/pg01
zroot                                                                   29.1G   176G       96K  /zroot
zroot/ROOT                                                              9.46G   176G       88K  none
zroot/ROOT/default                                                      9.46G   176G     6.16G  /
zroot/freebsd_releases                                                   334M   176G      333M  /var/db/mkjail
zroot/freshports                                                        19.1G   176G       96K  /zroot/freshports
zroot/freshports/ingress01                                               288K   176G       96K  /zroot/freshports/ingress01
zroot/freshports/ingress01/ports                                          96K   176G       96K  /jails/ingress01/jails/freshports/usr/ports
zroot/freshports/ingress01/repos                                          96K   176G       96K  /jails/ingress01/var/db/ingress/repos
zroot/freshports/pg01                                                   19.1G   176G       96K  /zroot/freshports/pg01
zroot/freshports/pg01/postgres                                          19.1G   176G     17.8G  /jails/pg01/var/db/postgres
zroot/tmp                                                               5.42M   176G      144K  /tmp
zroot/usr                                                                176K   176G       88K  /usr
zroot/usr/src                                                             88K   176G       88K  /usr/src
zroot/var                                                               35.6M   176G       88K  /var
zroot/var/audit                                                           88K   176G       88K  /var/audit
zroot/var/crash                                                           88K   176G       88K  /var/crash
zroot/var/log                                                           33.9M   176G     11.0M  /var/log
zroot/var/mail                                                           224K   176G      112K  /var/mail
zroot/var/tmp                                                           1.16M   176G      308K  /var/tmp
$

Leave a Comment Cancel Reply

You must be logged in to post a comment.