You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
Erik Hollensbe 2fa9d32a20 initial commit 4 months ago
box-builds initial commit 4 months ago
cmd initial commit 4 months ago
git initial commit 4 months ago
gp initial commit 4 months ago
info initial commit 4 months ago
rq initial commit 4 months ago
.gitignore initial commit 4 months ago
.golangci.yml initial commit 4 months ago
.wwhrd.yml initial commit 4 months ago
LICENSE initial commit 4 months ago
Makefile initial commit 4 months ago
README.md initial commit 4 months ago
go.mod initial commit 4 months ago
go.sum initial commit 4 months ago
todo.txt initial commit 4 months ago

README.md

gp: a proxy for remote git services. Make git fast.

gp allows you to localize many clones of the same SHA by way of marshalling git’s operations through a queue. The idea is principally for CI/CD solutions which may need to make many full clones of a large remote git repository.

To use gp, there is a GRPC client and commandline tool, gc, which can trigger the proxy to cache a certain SHA if it exists. From there, you can clone anew, or fetch an existing repository with the remote pointed at any git server and fetch the contents.

gp only makes one promise at this point, that it’ll never run git twice at the same time for the same repository, avoiding corruption.

gp writes off the base directory (provided by the -d argument, but is /tmp/git in the binary, and /var/lib/git in the container by default). This means if you start git daemon, line it up with this directory so you can just clone off of it.

Demo

$ docker run -itd --name gp-demo -p 127.0.0.1:9418:9418 erikh/gp
# feel free to replace with your own repos.
$ docker exec -it gp-demo gc trigger \
    https://github.com/tinyci/ci-agents b0e39d2cbd0b95d1cb79b7760cafc1d71358da61
$ git clone git://127.0.0.1/tinyci/ci-agents 
$ cd ci-agents && git reflog show b0e39d2cbd0b95d1cb79b7760cafc1d71358da61

Current state of development

This is at the “proven concept” phase. A similar solution has been deployed at a company to great success. However, this specific code has not been tested very hard. You may run into issues specifically with long-term storage of the cache, or potentially (but unlikely) with force-pushes and time of use.

Always check that you got your SHA in your clone. We simply cannot guarantee the atomicity of the cache. Others may update or replace it as it changes upstream.

Notes

RAM is a big deal with large Git repositories. If you plan to use it with large packs or a lot of heavily compressed objects, you may want to check your resource limits before blaming gp, which spawns git with impunity.

Proxy Authentication

Since all gp does is run git, it should be trivial to inject automated auth of any kind into a container with a $HOME directory set properly in the environment and a .gitconfig to provide the credentials. However, as of this writing I have not tried this.

Problems

  • It’s fast enough, but it probably shouldn’t shell out to git.
  • It doesn’t have queue limits.
  • It doesn’t have auth.
  • It doesn’t have https or SSH support in the container.

Author

Erik Hollensbe erik+git@hollensbe.org

License

MIT (c) 2020