some zfs experimentation around k8s and CSI.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
Erik Hollensbe f75abc80a1 send/recv support and some tests 2 months ago
ansible Extremely crude ansible to deploy a zpool to nodes 2 months ago
api/v1alpha1 first stab at integrating kubebuilder 2 months ago
cmd/k8zfs Refactoring log into the util package 2 months ago
config Setup affinity for stateful sets that would run the driver. 2 months ago
hack Fix the inventory so it appropriately configures across the network 2 months ago
pkg send/recv support and some tests 2 months ago
.gitignore Fix the inventory so it appropriately configures across the network 2 months ago
.gitmodules Vagrantfile, kubespray and ansible inventory 2 months ago
Dockerfile Dockerfile to deploy the CSI driver 2 months ago
LICENSE.md Licensed 2 months ago
Makefile Vagrant environment integration into makefile 2 months ago
Makefile.custom Oops 2 months ago
PROJECT first stab at integrating kubebuilder 2 months ago
README.md more additions 2 months ago
Vagrantfile Vagrantfile, kubespray and ansible inventory 2 months ago
go.mod Support string units in allocator (because kubernetes) 2 months ago
go.sum Support string units in allocator (because kubernetes) 2 months ago

README.md

k8zfs - kubernetes with more zfs.

some experiments I am performing around making a CSI driver + operator on k8s.

backlog:

  • Support for the following features:
  • Deployment

ideas:

  • Kubevirt support

Installation and Usage

You’ll probably want to do this in a VM; you are playing directly with the kernel. Also, most tests/operations run as root as a result, and have destructive potential. For development, I am using Ubuntu 20.04.

Install golang; any version after 1.14 should work.

apt install -y libzfslinux-dev zfsutils-linux

Installing the Driver + CSI framework

NOTE: if you are reading this, the driver only works in a single node configuration right now.

To install the driver from source:

  1. IMG=image/tag make docker-build docker-push. It will require your sudo password to run the tests, after which it will build the image and push it to a registry.
    • If you do not set the IMG parameter above, it will use erikh/k8zfs.
  2. If you set IMG above, edit config/csi/bases/k8zfs-plugin.yaml to reflect it (yes, kustomization support is on the way for this).
  3. KUBECONFIG=~/.kube/myconfig make install will install the CRDs, RBAC rules and stateful sets required to run the driver. To uninstall, simply make uninstall with the same KUBECONFIG set.

Running tests

Only tests really work right now. Note, this will mess with any live ZFS installation on the host. It impacts the pool testpool by default by removing and re-creating it a number of times, as well as datasets beneath it. To run them:

Note that you may run into go.mod issues; be sure to go clean -modcache as root and your user account if you’re running into resolution issues.

make test

Other tasks

As noted in the examples, most makefile commands come from the Makefile.custom file; specify -f with make to invoke tasks in it.

The tests create several files in /tmp that it uses as zfs zpools. The datasets are created on top of the zpools. The default pool name is testpool, and if your tests break, you can correct them by destroying this pool with this make task:

sudo -E make -f Makefile.custom testclean

If you’d like to mount a ZFS zpool for playing around with:

sudo -E make -f Makefile.custom dummy

If you’d like to create a testbed running the daemon along with the zpool already setup:

make -f Makefile.custom testbed # no root here, sudo will be called
make -f Makefile.custom clean-testbed # clean up

If you’d like to install the latest csi-sanity suite:

make -f Makefile.custom install-csi-sanity
make -f Makefile.custom csi-sanity # to run it

If you’d like to adjust parameters of the test suite, such as which pool it affects, adjust these environment variables:

  • ZFS_TESTPOOL: used in makefiles and tests to provide the pool to affect.
  • ZFS_TESTPOOLSIZE: Size (bytes) of the test pool. The pools are created from sparse images and should not affect your filesystem, but default to 5GiB which may be a problem on smaller systems. This setting does not affect the Makefile.
  • DEBUG: When set, this does not perform any storage cleanup. You will want to manually invoke one test at a time (with go test -run) with this option enabled, then trigger make testclean to fix your system before running tests again.
  • CSI_SANITY_DEBUG: When set, this does not clean up the test pool after the CSI sanity run.
  • LOGRUS_DEBUG: when set non-empty, it will trigger logrus request logging.
  • ITERATIONS: set to a number, this configures the number of iterations for certain battery tests