Browse Source

* First commit.

Erik Hollensbe 12 years ago
  1. 13
  2. 139


@ -0,0 +1,13 @@
@texi2pdf gap-protocol.texi
all: build
view: build
@if [ `uname -s` = "Darwin" ]; then open gap-protocol.pdf; else evince gap-protocol.pdf; fi
find . -maxdepth 1 -name 'gap-protocol.*' | grep -vE '\.(pdf|texi)$$' | xargs rm
clean: buildclean
rm gap-protocol.pdf


@ -0,0 +1,139 @@
\input texinfo
@setfilename gap-protocol.texi
@settitle General Assertion Protocol 1.0
Copyright @copyright{} 2010 Erik Hollensbe. My dad can distribute this work
better than your dad.
@end copying
@title General Assertion Protocol 1.0
@author Erik Hollensbe <>
@vskip 0pt plus 1filll
@end titlepage
@node Top
@chapter Introduction
Testing applications and libraries has been a battle-tested way of mitigating
problems in our software, and many organizations and projects have mandated
testing as a result.
However, the way we accomplish testing a piece of software is not so concrete.
We must evaluate many systems to find one that fits our needs. The vast
majority of these systems provide their own way of demonstrating results;
results that are varied enough that tools used to evaluate this data in an
automated fashion cannot be agnostic to the systems they support.
The author of this protocol description feels that this is an unnecessary
disadvantage that causes the user of the test Producer and Consumer to suffer,
locking them into choices that may determine the Producer or Consumers used
based on the perceived advantage of one or the other, but not both.
Therefore, the @i{General Assertion Protocol} (or @i{GAP})'s goals are to
create a unified format of test output in the hopes that the user is not maimed
by bad/poorly-applied Producer implementations or bad Consumers for analysis:
users sticking with GAP-based tools are more likely to find a wide array of
choices on both ends.
@chapter Terms
Terminology used throughout this document.
@section Assertion
An assertion in our context is a single boolean test. "A equals B" is a single
assertion, but in most cases the code that is used to express "A equals B, C,
and D" is generally 3 assertions.
@section Consumer
A Consumer is an analysis tool which may provide specialized output or takes
action upon its input. It reads, or @i{consumes} the output from the Producer.
@section Producer
A Producer provides the protocol output as a part of its reaction to
assertions. It writes, or @i{produces} output for consumption.
@section Test Suite
The application-specific tests that make up a comprehensive set of tests for
the application. This is not a @i{Producer}, but instead is something that uses
the @i{Producer} to evaluate the results of its tests.
@section Unit
A collection of assertions that meet a common testing goal.
@chapter Influences
Influences which have both shaped our protocol, and the decision to define our
own, are listed below.
@section TAP
TAP, or the @i{Test Anything Protocol} is originally a Perl-centric method of
communication which abstracts the Producer from the Consumer. It is
very similar in goal to @i{GAP}, and is the foundation for almost all testing
in the Perl ecosystem.
TAP as an idea is not poor at all, but the implementation itself is much harder
to parse than need be, resorting to traditional grammatical parsing even for
the simplest of analysis tasks. The authors feel that TAP is a poor fit even
for some more advanced Perl Producers, such as @code{Test::Class} which uses
namespaces and resembles @code{SUnit} in function. The authors also feel
this is largely due to TAP's legacy of being borne from original Perl
testing frameworks, which did not support features such as namespacing,
exception output, or multiple units inside a single file.
@section Fit
Fit, or the @i{Framework for Integrated Test} is a system built by Ward
Cunningham for cross-communication between customers and developers. The
developer processes a set of descriptions provided by the customer, and these
descriptions are then used to produce a spreadsheet with results.
Fit, on the parsing end, is a much better fit than TAP, but Fit differs largely
in scope to our project. However, using one-row, one-assertion output was a
large influence for the current design of GAP.
@chapter Goals
@section Easy to Generate and Parse
Our protocol must be easy to generate. It should be easy for the generator to
find a consistent analogue to their system that would serve the basic needs of
a Consumer, to implement switfly and with a minimal amount of unexpected
parsing errors.
It should also be easy to parse; any program that has basic string operations
should be able to handle the output with a minimal amount of effort, making the
analysis itself the focus of a program dealing with the output of a test suite.
@section Extensible
No one here claims that we're the end-all-be-all of test output, and as a
result wish to keep the protocol extensible.
This means that the protocol's Consumers should be able to trivially ignore
output that is not consequential to their analysis goals, and that Producers
that wish to extend the protocol to better fit what the Producer provides are
not hampered by a limited offering in the protocol or a subset of Consumers
that support it.
@section Portable
Our output should not be formed in the shape of one language or testing system,
nor should it be hampered by platform deficiencies, system/network
accessibility, or the like. It should be trivial to use a Producer in one
language and a Consumer in another, perhaps even on a wildly different
operating system.