data04: DEGRADED – oh and have I mentioned WatchGoose?

Leave a Comment / nvme, ZFS / By /

I saw this today in my logs on r730-01. It looks similar to a previous DEGRADED even back on July 2. In this case, the device is S7KGNU0Y915666E, not S7KGNU0Y912955D as it was back then.

Sep 12 03:46:54 r730-01 kernel: nvme1: Resetting controller due to a timeout.
Sep 12 03:46:54 r730-01 kernel: nvme1: event="start"
Sep 12 03:46:54 r730-01 kernel: nvme1: Waiting for reset to complete
Sep 12 03:47:14 r730-01 syslogd: last message repeated 197 times
Sep 12 03:47:14 r730-01 ZFS[96518]: vdev I/O failure, zpool=data04 path=/dev/gpt/S7KGNU0Y915666E offset=270336 size=8192 error=6
Sep 12 03:47:14 r730-01 kernel: nvme1: Waiting for reset to complete
Sep 12 03:47:14 r730-01 kernel: nvme1: controller ready did not become 0 within 20500 ms
Sep 12 03:47:14 r730-01 kernel: nvme1: event="timed_out"
Sep 12 03:47:14 r730-01 kernel: nvme1: failing outstanding i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: FLUSH (00) sqid:1 cid:112 nsid:1
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:1 cid:112 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: failing outstanding i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:120 nsid:1 lba:2320185448 len:2048
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): FLUSH (00). NCB: opc=0 fuse=0 nsid=1 prp1=0 prp2=0 cdw=0 0 0 0 0 0
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:120 cdw0:0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 5, Retries exhausted
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:121 nsid:1 lba:2320187496 len:2048
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:121 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:123 nsid:1 lba:2320189544 len:1368
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:123 cdw0:0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b3868 0 7ff 0 0 0
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:118 nsid:1 lba:2320190912 len:2048
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 5, Retries exhausted
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b4068 0 7ff 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 5, Retries exhausted
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b4868 0 557 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:118 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:125 nsid:1 lba:2320192960 len:2048
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 5, Retries exhausted
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b4dc0 0 7ff 0 0 0
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:125 cdw0:0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 5, Retries exhausted
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:113 nsid:1 lba:2320195008 len:1368
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b55c0 0 7ff 0 0 0
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:113 cdw0:0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 5, Retries exhausted
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:115 nsid:1 lba:2320196376 len:2048
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:115 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:114 nsid:1 lba:2320198424 len:2048
Sep 12 03:47:14 r730-01 kernel: nda1 at nvme1 bus 0 scbus17 target 0 lun 1
Sep 12 03:47:14 r730-01 kernel: nda1: <Samsung SSD 990 PRO 4TB 4B2QJXD7 S7KGNU0Y915666E> s/n S7KGNU0Y915666E detached
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:114 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:122 nsid:1 lba:2320200472 len:1368
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:122 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: failing queued i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:7 cid:0 nsid:1 lba:4317358168 len:8
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b5dc0 0 557 0 0 0
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:7 cid:0 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:7 cid:0 cdw0:0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: nvme1: failing outstanding i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: WRITE (01) sqid:11 cid:125 nsid:1 lba:4127749432 len:8
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:11 cid:125 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: WRITE (01) sqid:11 cid:119 nsid:1 lba:5324663584 len:8
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:11 cid:119 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: failing queued i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:13 cid:0 nsid:1 lba:5554129904 len:8
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:13 cid:0 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: failing outstanding i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:14 cid:115 nsid:1 lba:4123885016 len:32
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:14 cid:115 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:14 cid:125 nsid:1 lba:4123885456 len:8
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:14 cid:125 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: failing outstanding i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: WRITE (01) sqid:15 cid:104 nsid:1 lba:4127749424 len:8
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:15 cid:104 cdw0:0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b6318 0 7ff 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b6b18 0 7ff 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b7318 0 557 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=155a858 1 7 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=f6087538 0 7 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=3d5feb20 1 7 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=4b0d4bf0 1 7 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=f5cd7dd8 0 1f 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=f5cd7f90 0 7 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=f6087530 0 7 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: nvme1: Failed controller, stopping watchdog timeout.
Sep 12 03:47:15 r730-01 ZFS[96546]: vdev state changed, pool_guid=1760090623298529734 vdev_guid=10692465753147695115
Sep 12 03:47:15 r730-01 ZFS[96550]: vdev is removed, pool_guid=1760090623298529734 vdev_guid=10692465753147695115
Sep 12 03:47:15 r730-01 kernel: nvme1: Failed controller, stopping watchdog timeout.
Sep 12 03:47:15 r730-01 syslogd: last message repeated 1 times
Sep 12 03:47:15 r730-01 syslogd: last message repeated 1 times
Sep 12 03:47:15 r730-01 kernel: (nda1:nvme1:0:0:1): Periph destroyed
Sep 12 03:47:15 r730-01 kernel: nvme1: Failed controller, stopping watchdog timeout.
Sep 12 03:47:15 r730-01 syslogd: last message repeated 1 times

The zpool status is:

[12:16 r730-01 dvl ~] % zpool status data04
  pool: data04
 state: DEGRADED
status: One or more devices have been removed.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using zpool online' or replace the device with
	'zpool replace'.
  scan: scrub repaired 0B in 01:20:36 with 0 errors on Sat Sep 12 04:29:12 2026
config:

	NAME                     STATE     READ WRITE CKSUM
	data04                   DEGRADED     0     0     0
	  raidz2-0               DEGRADED     0     0     0
	    gpt/S7KGNU0Y722875X  ONLINE       0     0     0
	    gpt/S7KGNU0Y915666E  REMOVED      0     0     0
	    gpt/S7KGNU0Y912937J  ONLINE       0     0     0
	    gpt/S7KGNU0Y912955D  ONLINE       0     0     0
	    gpt/S7U8NJ0Y716854P  ONLINE       0     0     0
	    gpt/S7U8NJ0Y716801F  ONLINE       0     0     0
	    gpt/S757NS0Y700758M  ONLINE       0     0     0
	    gpt/S757NS0Y700760R  ONLINE       0     0     0

errors: No known data errors

What’s next?

I will reboot this host after dbclone finishes restoring database backups

.

I’ll take this opportunity to mention WatchGoose – a third party website for monitoring tasks, such as cronjobs. If I’d checked here first, I’d have seen dbclone had not finished yet today. There is a free tier for this service.

WatchGoose
WatchGoose

Full disclosure: They offered free accounts to FreeBSD Developers. I got one. And I plan to add more monitoring such as:

  • has the Bacula database backup completed? (one host)
  • has the Bacula database backup been rsynced? (three hosts)
  • has the Bacula configuration been rsynced? (three hosts)
  • have the zfs snapshots been created? (several hosts)
  • have the data backup rsyncs run? (several hosts)
  • have the FreshPorts package checks been pulled? (several hosts)
  • have the database backups been rsynced to dbclone? (several hosts)

It’s mainly a matter of time for implementing that.

I plan to post this, then create another post as I resolve the issue.

After the reboot

After the reboot, the missing device was not seen in /var/run/dmesg.boot.

I powered off the host, opened it up, and pulled the NVMe card, and removed the heatsink.

The 4x NVMe sticks
The 4x NVMe sticks

The stick in question is the 2nd slot from the fan. In the previous post, the problem stick was described as: “It was in the slot farthest from the fan. Now it’s one slot closer to the fan”.

That tells me this incident is not related to the stick or to the slot on the card.

This time, I removed the screw holding down the NVMe stick, removed the stick, reseated it, and reinstalled the screw.

I reinstalled the card, closed the server, hooked up all the cables, and powered on.

My plan: go into single user modem run service jail disable and proceed into multi user mode.

That didn’t stop the jails from starting up….

However, the zpool status is fine:

[16:18 r730-01 dvl ~] % zpool status data04
  pool: data04
 state: ONLINE
  scan: resilvered 3.03G in 00:00:31 with 0 errors on Sat Sep 12 16:18:20 2026
config:

	NAME                     STATE     READ WRITE CKSUM
	data04                   ONLINE       0     0     0
	  raidz2-0               ONLINE       0     0     0
	    gpt/S7KGNU0Y722875X  ONLINE       0     0     0
	    gpt/S7KGNU0Y915666E  ONLINE       0     0     0
	    gpt/S7KGNU0Y912937J  ONLINE       0     0     0
	    gpt/S7KGNU0Y912955D  ONLINE       0     0     0
	    gpt/S7U8NJ0Y716854P  ONLINE       0     0     0
	    gpt/S7U8NJ0Y716801F  ONLINE       0     0     0
	    gpt/S757NS0Y700758M  ONLINE       0     0     0
	    gpt/S757NS0Y700760R  ONLINE       0     0     0

errors: No known data errors
[16:18 r730-01 dvl ~] %

Why did the jails start?

Based on this, my command didn’t take, and I don’t know why.

[16:19 r730-01 dvl ~] % grep jail /etc/rc.conf
jail_enable="YES"
jail_reverse_stop="YES"
jail_list="logs dns1 cliff2 mysql01 mysql02 pg01 pg02 pg03 pkg01 dev-ingress01             dvl-ingress01             test-ingress01              stage-ingress01                bacula besser certs-rsync certs git svn webserver mqtt01 bacula-sd-02 talos samdrucker bacula-sd-03 mydev jail_within_jail serpico dns-hidden-master nsnotify unifi01 dev-nginx01 dvl-nginx01 test-nginx01 stage-nginx01"

The file has not been touched in months:

[16:22 r730-01 dvl /etc/rc.conf.d] % cd
[16:22 r730-01 dvl ~] % date
Sat Sep 12 16:22:04 UTC 2026
[16:22 r730-01 dvl ~] % ls -l /etc/rc.conf
-rw-r--r--  1 root wheel 3410 2026.06.29 18:06 /etc/rc.conf

User error? I tried scrolling back to confirm my command, but it has scrolled off my console terminal.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Related Posts

Leave a Comment

Scroll to Top