You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
Erik Hollensbe b21c31eafe README.md: update for updated API 3 years ago
storage async storage support 3 years ago
transport async storage support 3 years ago
.travis.yml travis shit 3 years ago
Makefile async storage support 3 years ago
README.md README.md: update for updated API 3 years ago
box.rb box.rb: remove redis dep 3 years ago
go.mod storage: preliminary storage interface 3 years ago
go.sum storage: preliminary storage interface 3 years ago
item.go adjust types 3 years ago
qshift.go async storage support 3 years ago
qshift_test.go async storage support 3 years ago
sort.go sort by tag and manage tags in test 3 years ago
storage.go adjust types 3 years ago

README.md

qshift: a small queueing library with some cool features

Build Status

qshift provides a unified queue with basic tag and id indexes for lookup efficiency. It takes a lazy approach to expiring queue items (although there is a manual step you can take to clean up).

qshift also provides some basic transport functionality via https://github.com/gorilla/mux if you'd prefer to use it than roll your own. This makes it easy to implement basic queues as a service. More transports will be added soon.

Performance

qshift is currently handling about 1mil qps on an 8 thread i7 (4 cores) with 32GB ram with GOMAXPROCS set to the number of threads both for tags and the linear queue. There are 20 consumers in our load test and 20 fillers. Please see qshift_test.go for more.

Example:

package main

import (
	"fmt"

	"github.com/erikh/qshift"
)

func main() {
	q := qshift.NewQueue(0)

	for i := 0; i < 5; i++ {
		q.NewItem(nil, i) // no tags
	}

	for i := 0; i < 5; i++ {
		item, err := q.Shift()
		if err != nil {
			panic(err)
		}
		item.Shifted()           // true, scheduled for GC
		fmt.Println(item.Data()) // value of i
		fmt.Println(item.Tags()) // empty array
	}

	for i := 0; i < 5; i++ {
		q.NewItem([]string{"tag1", "tag2"}, i) // with tags
	}

	// tags share the queue, so shifting one will remove that item from the other.
	for i := 0; i < 3; i++ {
		item, err := q.ShiftTag("tag1")
		if err != nil {
			panic(err)
		}
		fmt.Println(item.Data()) // i
	}

	for i := 3; i < 5; i++ {
		item, err := q.ShiftTag("tag2")
		if err != nil {
			panic(err)
		}
		fmt.Println(item.Data()) // i
	}

	fmt.Println(q.Shift()) // nil
}

Misfeatures / Caveats

Currently, the queue data suffers from an opportunity to be mutated. It's imperative that you take copies of the data you pull off the queue if you intend to re-use it inside the queue system in any way. While this is unlikely, it may be necessary/possible to do so with the Create/Add interface in particular. Just be aware you may be modifying data that currently exists in the queue if you do that.

License

Copyright 2018 Erik Hollensbe

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Author

Erik Hollensbe h-e@hollensbe.org