I’m writing some scripts to periodically inspect ZFS pools and parse the output from zpool status, and squawk if something’s unhappy.

But what exactly does an unhappy zpool look like anyway?

To find out, I did a fresh install of SmartOS with a freshly initialized zpool, with a couple of disks that I knew were buggy as members. Fill it up with several copies of a handy dataset (my Apple Lossless library suited fine), took some errors when writing and faulted a drive. Voila. Why stop there? Run some scrubs and look for more bugginess. I was not disappointed. Even lost some data.

Without further comment (except “thanks drupal for stomping on indentation”), here’s some “zpool status” output from an unhappy zpool.

[root@a0-b3-cc-e8-95-9a ~]# zpool status
  pool: zones
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
    repaired.
  scan: scrub in progress since Thu Jun 11 11:31:31 2015
387G scanned out of 5.13T at 204M/s, 6h46m to go
0 repaired, 7.37% done
config:

    NAME        STATE     READ WRITE CKSUM
    zones       DEGRADED     0     0     0
      raidz1-0  DEGRADED     0     0     0
	c0t0d0  FAULTED      0 3.88K     0  too many errors
	c0t1d0  ONLINE       0     0     0
	c0t2d0  ONLINE       0     0     0
	c0t3d0  ONLINE       0     0     0

errors: No known data errors
[root@a0-b3-cc-e8-95-9a ~]# 


[root@a0-b3-cc-e8-95-9a ~]# zpool status
  pool: zones
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub in progress since Thu Jun 11 11:31:31 2015
440G scanned out of 5.13T at 191M/s, 7h10m to go
0 repaired, 8.38% done
config:

    NAME        STATE     READ WRITE CKSUM
    zones       DEGRADED    11     0     0
      raidz1-0  DEGRADED    11     0     0
	c0t0d0  FAULTED      0 3.88K     0  too many errors
	c0t1d0  ONLINE       0     0     0
	c0t2d0  ONLINE       0     0     0
	c0t3d0  DEGRADED    11     0     0  too many errors

errors: 4 data errors, use '-v' for a list
[root@a0-b3-cc-e8-95-9a ~]# 

[root@a0-b3-cc-e8-95-9a ~]# zpool status -v
  pool: zones
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub in progress since Thu Jun 11 11:31:31 2015
451G scanned out of 5.13T at 192M/s, 7h7m to go
0 repaired, 8.60% done
config:

    NAME        STATE     READ WRITE CKSUM
    zones       DEGRADED    11     0     0
      raidz1-0  DEGRADED    11     0     0
	c0t0d0  FAULTED      0 3.88K     0  too many errors
	c0t1d0  ONLINE       0     0     0
	c0t2d0  ONLINE       0     0     0
	c0t3d0  DEGRADED    11     0     0  too many errors

errors: Permanent errors have been detected in the following files:

    zones/rs-applelossless24@migrate:/Music/iTunes/iTunes Music/E. Power Biggs/J.S. Bach_ Organ Works/04 Passacaglia & Fugue in C minor, BWV 582.m4a
    zones/rs-applelossless24@migrate:/Music/iTunes/iTunes Music/E. Power Biggs/J.S. Bach_ Organ Works/05 Pastorale in F major, BWV 590.m4a
    zones/rs-applelossless24@migrate:/Music/iTunes/iTunes Music/E. Power Biggs/J.S. Bach_ Organ Works/06 Pastorale in F major, BWV 590.m4a
    zones/rs-applelossless24@migrate:/Music/iTunes/iTunes Music/E. Power Biggs/J.S. Bach_ Organ Works/07 Pastorale in F major, BWV 590.m4a
[root@a0-b3-cc-e8-95-9a ~]#