tinyci 2.0
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
Erik Hollensbe 6736e307ad fix links 5 months ago
boot move to code.hollensbe.org 6 months ago
box-builds Fix up the build a bit 5 months ago
client phew 5 months ago
cmd move to code.hollensbe.org 6 months ago
config move to code.hollensbe.org 6 months ago
doc fix links 5 months ago
docker-compose.d Integrate prometheus scraping for GRPC endpoints 8 months ago
lib move to code.hollensbe.org 6 months ago
runner phew 5 months ago
sd move to code.hollensbe.org 6 months ago
service Fix a potential bug managing the oauth state and crypto/rand 5 months ago
test move to code.hollensbe.org 6 months ago
vendor phew 5 months ago
.gitignore Fix up the build a bit 5 months ago
.golangci.yml golangci-lint integration 10 months ago
.wwhrd.yml Update wwhrd for another fucking license 8 months ago
LICENSE Here we go. 5 months ago
Makefile Fix up the build a bit 5 months ago
README.md Add a readme 5 months ago
docker-compose.yml Fix up the build a bit 5 months ago
go.mod phew 5 months ago
go.sum phew 5 months ago
oauth.yml.example Initial commit of tinyci-runner binary 8 months ago
task.yml move to code.hollensbe.org 6 months ago
tinyci-runner.yml.example Initial commit of tinyci-runner binary 8 months ago
tinyci.yml tinyci fixes 11 months ago

README.md

About the service framework

This is the TinyCI 0.3.0 framework. It has a lot of useful things you may be interested in, but it is nowhere near complete at this point in time.

tinyCI’s framework is about 95% GRPC and JWT with a dash of NATS. It has a full RBAC layer and some support for multi-site federation / untrusted API access.

TinyCI’s framework is not really a high performance framework. The intent is not to make something ultra fast, but flexible enough to fit into most environments as CI systems typically don’t beat up anything other than the runners themselves. As a result, you may see trades in here that I have tried to note below.

Setup

Docker and docker-compose are required prerequisites for playing around. make start will install my toolchain (which requires sudo to install at one point, but only then) and boot a docker-compose instance with the CI stack, runner, and all dependencies. Jaeger is used for local tracing, and prometheus is tied to consul so all services are automatically discovered and scraped.

You will want to copy the tinyci-runner.yml.example to tinyci-runner.yml too. After doing all that and starting the stack, docker-compose exec tinyci bash will get you near a place where you can play around. go install -v ./... will be necessary to install the tinyci-dev binary which can do some data faking and queue filling; otherwise the tinyci admin command should already be available for playing around a bit.

make test outside of the containerized environment also works, and validates the system.

About

In most cases tinyCI is optimized for portability. All services are bundled into a single binary. Currently, the tinyci boot command boots all of them at once. This is intentional at this stage to keep it simple, but as you can see if you trawl through the cmd directory’s imports you will see the capability for an easy split-out. The idea here was that you can bundle multiple services onto hosts depending on need. Do a lot of logging? Fan out logging only, etc.

It does not use DNS for service lookup at all, it instead uses the consul API directly. This incurs a ~100ns hit on RTT in local testing with a lot of genuine advantages, a large one being that DNS isn’t necessary to use. You can find this work in the sd directory.

In the lib directory is a large portion of the framework; inside you will find work around linux-style capabilities to evoke an RBAC-like system with JWT and all the GRPC glue to support that; there’s also a tiny smidgeon of work around NATS.

In the service directory are the actual services. They are short applications used to talk to and provide resources. client contains custom client wrappers for the services, intended for external consumption.

Most services have a simple protocol and are easy to replace to support other forms of resource management. authsvc writes to disk, for example; it should be trivial to make it target S3 with a functionality replacement and WITHOUT compromising the rest of the system.

  • authsvc is a oauth-compatible service suitable for linking oauth services to your user database, through usersvc.
  • usersvc is the user store. Uses (hopefully) not terrible crypto practices.
  • reposvc is a service for managing github repositories
  • queuesvc is still new, and a method for communicating with runners (also new, see runner/ for more)
  • logsvc is a graph-oriented logging system that leverages facebook’s ent to log to postgresql.
  • assetsvc is a CI log management system currently, but is intended to manage all stored assets.

I hope you like it, or if nothing else, find it entertaining. I’ll be building on this more.

Author

Erik Hollensbe erik+git@hollensbe.org

License

MIT, (c) 2020