Device add/removal
~~~~~~~~~~~~~~~~~~

The following subcommands exist for adding and removing devices from a
mounted filesystem:

-  ``bcachefs device add``: Formats and adds a new device to an existing
   filesystem.

-  ``bcachefs device remove``: Permenantly removes a device from an
   existing filesystem.

-  ``bcachefs device online``: Connects a device to a running filesystem
   that was mounted without it (i.e. in degraded mode)

-  ``bcachefs device offline``: Disconnects a device from a mounted
   filesystem without removing it.

-  ``bcachefs device evacuate``: Migrates data off of a particular
   device to prepare for removal, setting it read-only if necessary.

-  ``bcachefs device set-state``: Changes the state of a member device:
   one of rw (readwrite), ro (readonly), failed, or spare.

   A failed device is considered to have 0 durability, and replicas on
   that device won’t be counted towards the number of replicas an extent
   should have by rereplicate - however, bcachefs will still attempt to
   read from devices marked as failed.

The ``bcachefs device remove``, ``bcachefs device offline`` and
``bcachefs device set-state`` commands take force options for when they
would leave the filesystem degraded or with data missing. Todo:
regularize and improve those options.