You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Erik Hollensbe f75abc80a1 send/recv support and some tests 1 year ago
ansible Extremely crude ansible to deploy a zpool to nodes 1 year ago
api/v1alpha1 first stab at integrating kubebuilder 1 year ago
cmd/k8zfs Refactoring log into the util package 1 year ago
config Setup affinity for stateful sets that would run the driver. 1 year ago
hack Fix the inventory so it appropriately configures across the network 1 year ago
pkg send/recv support and some tests 1 year ago
.gitignore Fix the inventory so it appropriately configures across the network 1 year ago
.gitmodules Vagrantfile, kubespray and ansible inventory 1 year ago
Dockerfile Dockerfile to deploy the CSI driver 1 year ago Licensed 1 year ago
Makefile Vagrant environment integration into makefile 1 year ago
Makefile.custom Oops 1 year ago
PROJECT first stab at integrating kubebuilder 1 year ago more additions 1 year ago
Vagrantfile Vagrantfile, kubespray and ansible inventory 1 year ago
go.mod Support string units in allocator (because kubernetes) 1 year ago
go.sum Support string units in allocator (because kubernetes) 1 year ago

k8zfs - kubernetes with more zfs.

some experiments I am performing around making a CSI driver + operator on k8s.


  • Support for the following features:
    • Controller (global namespace) support
      • peer-to-peer node transfers of ZFS datasets for publishing rules (on demand storage transfer)
        • grpc / redirect to tls-capable port for stream probably
      • sharenfs support for the above, in lieu of moving the datasets in particular (optional)
      • migration CRD for controlling this behavior manually (drain a node, migrate its datasets, etc)
      • storage allocator so we don't overfill the disks migrating
    • Snapshots
    • ZVol (block device) support
  • Deployment
    • Better kustomization support
    • RedHat/SuSE kernels + Ansible (need to investigate)
    • Some kind of Vagrant thing or kubevirt to show it off


  • Kubevirt support

Installation and Usage

You'll probably want to do this in a VM; you are playing directly with the kernel. Also, most tests/operations run as root as a result, and have destructive potential. For development, I am using Ubuntu 20.04.

Install golang; any version after 1.14 should work.

apt install -y libzfslinux-dev zfsutils-linux

Installing the Driver + CSI framework

NOTE: if you are reading this, the driver only works in a single node configuration right now.

To install the driver from source:

  1. IMG=image/tag make docker-build docker-push. It will require your sudo password to run the tests, after which it will build the image and push it to a registry.
    • If you do not set the IMG parameter above, it will use erikh/k8zfs.
  2. If you set IMG above, edit config/csi/bases/k8zfs-plugin.yaml to reflect it (yes, kustomization support is on the way for this).
  3. KUBECONFIG=~/.kube/myconfig make install will install the CRDs, RBAC rules and stateful sets required to run the driver. To uninstall, simply make uninstall with the same KUBECONFIG set.

Running tests

Only tests really work right now. Note, this will mess with any live ZFS installation on the host. It impacts the pool testpool by default by removing and re-creating it a number of times, as well as datasets beneath it. To run them:

Note that you may run into go.mod issues; be sure to go clean -modcache as root and your user account if you're running into resolution issues.

make test

Other tasks

As noted in the examples, most makefile commands come from the Makefile.custom file; specify -f with make to invoke tasks in it.

The tests create several files in /tmp that it uses as zfs zpools. The datasets are created on top of the zpools. The default pool name is testpool, and if your tests break, you can correct them by destroying this pool with this make task:

sudo -E make -f Makefile.custom testclean

If you'd like to mount a ZFS zpool for playing around with:

sudo -E make -f Makefile.custom dummy

If you'd like to create a testbed running the daemon along with the zpool already setup:

make -f Makefile.custom testbed # no root here, sudo will be called
make -f Makefile.custom clean-testbed # clean up

If you'd like to install the latest csi-sanity suite:

make -f Makefile.custom install-csi-sanity
make -f Makefile.custom csi-sanity # to run it

If you'd like to adjust parameters of the test suite, such as which pool it affects, adjust these environment variables:

  • ZFS_TESTPOOL: used in makefiles and tests to provide the pool to affect.
  • ZFS_TESTPOOLSIZE: Size (bytes) of the test pool. The pools are created from sparse images and should not affect your filesystem, but default to 5GiB which may be a problem on smaller systems. This setting does not affect the Makefile.
  • DEBUG: When set, this does not perform any storage cleanup. You will want to manually invoke one test at a time (with go test -run) with this option enabled, then trigger make testclean to fix your system before running tests again.
  • CSI_SANITY_DEBUG: When set, this does not clean up the test pool after the CSI sanity run.
  • LOGRUS_DEBUG: when set non-empty, it will trigger logrus request logging.
  • ITERATIONS: set to a number, this configures the number of iterations for certain battery tests