- Tue 16 June 2015
- misc
I'm writing some scripts to periodically inspect ZFS pools and parse the output from zpool status, and squawk if something's unhappy.
But what exactly does an unhappy zpool look like anyway?
To find out, I did a fresh install of SmartOS with a freshly initialized zpool, with a couple of disks that I knew were buggy as members. Fill it up with several copies of a handy dataset (my Apple Lossless library suited fine), took some errors when writing and faulted a drive. Voila. Why stop there? Run some scrubs and look for more bugginess. I was not disappointed. Even lost some data.
Without further comment (except "thanks drupal for stomping on indentation"), here's some "zpool status" output from an unhappy zpool.
[root@a0-b3-cc-e8-95-9a ~]# zpool status
pool: zones
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub in progress since Thu Jun 11 11:31:31 2015
387G scanned out of 5.13T at 204M/s, 6h46m to go
0 repaired, 7.37% done
config:
NAME STATE READ WRITE CKSUM
zones DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
c0t0d0 FAULTED 0 3.88K 0 too many errors
c0t1d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
errors: No known data errors
[root@a0-b3-cc-e8-95-9a ~]#
[root@a0-b3-cc-e8-95-9a ~]# zpool status
pool: zones
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub in progress since Thu Jun 11 11:31:31 2015
440G scanned out of 5.13T at 191M/s, 7h10m to go
0 repaired, 8.38% done
config:
NAME STATE READ WRITE CKSUM
zones DEGRADED 11 0 0
raidz1-0 DEGRADED 11 0 0
c0t0d0 FAULTED 0 3.88K 0 too many errors
c0t1d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c0t3d0 DEGRADED 11 0 0 too many errors
errors: 4 data errors, use '-v' for a list
[root@a0-b3-cc-e8-95-9a ~]#
[root@a0-b3-cc-e8-95-9a ~]# zpool status -v
pool: zones
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub in progress since Thu Jun 11 11:31:31 2015
451G scanned out of 5.13T at 192M/s, 7h7m to go
0 repaired, 8.60% done
config:
NAME STATE READ WRITE CKSUM
zones DEGRADED 11 0 0
raidz1-0 DEGRADED 11 0 0
c0t0d0 FAULTED 0 3.88K 0 too many errors
c0t1d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c0t3d0 DEGRADED 11 0 0 too many errors
errors: Permanent errors have been detected in the following files:
zones/rs-applelossless24@migrate:/Music/iTunes/iTunes Music/E. Power Biggs/J.S. Bach_ Organ Works/04 Passacaglia & Fugue in C minor, BWV 582.m4a
zones/rs-applelossless24@migrate:/Music/iTunes/iTunes Music/E. Power Biggs/J.S. Bach_ Organ Works/05 Pastorale in F major, BWV 590.m4a
zones/rs-applelossless24@migrate:/Music/iTunes/iTunes Music/E. Power Biggs/J.S. Bach_ Organ Works/06 Pastorale in F major, BWV 590.m4a
zones/rs-applelossless24@migrate:/Music/iTunes/iTunes Music/E. Power Biggs/J.S. Bach_ Organ Works/07 Pastorale in F major, BWV 590.m4a
[root@a0-b3-cc-e8-95-9a ~]#