Beyond TAP: polyglot testing with exeter
Room B | Tue 21 Jan 3:45 p.m.–4:30 p.m.
Presented by
-
David Gibson
https://ozlabs.org/~dgibson/
David has had a 20+ year IT career working almost entirely on open source projects. He made his first kernel contributions in 2000 with
some work on ramfs, and then the "orinoco" wireless driver while at LinuxCare.
From there he moved to IBM where he worked on the kernel for embedded PowerPC systems. He wrote, and still maintains, the device tree compiler "dtc" to assist with this work, and it has since become a standard tool for ARM as well as PowerPC embedded kernels. From there he moved on to kernel code for POWER server machines and then virtualization. He wrote the "pseries" machine for qemu and was PowerPC target maintainer in qemu for around 5 years.
In 2013 he moved to Red Hat where he again worked on virtualization in qemu and the kernel before drifting into networking as the second major contributor to passt and pasta, a modern userspace networking implementation with applications for virtual machines, containers and running both together in the cloud.
David Gibson
https://ozlabs.org/~dgibson/
Abstract
Automated testing and has become very common in the open source world over the last two decades. Modern languages like Go and Rust have
built in support making it easy to start write unit tests and other external test frameworks abound.
But if you're not using a fancy new language, or your implementation language isn't convenient for writing testsor your project uses
multiple langauges, the language centric systems don't work.
Meanwhile, some of the external frameworks impose a language for writing tests which might be inconvenient for you. Many of them
impose a style of writing tests that could be unnatural for your language. In particular JUnit's influence often brings Java-isms to
languages where they don't belong. Building arrays or matrices of similar tests might require framework specific approaches syntax which
can be awkward or limited.
Then there's TAP, the "Test Anything Protocol" from the Perl world. It lets you write test programs in any language and collate the
results. However, TAP is showing its age: the protocol structure implies sequential test execution; with test's identified only by a
number which can change if tests are added or removed. That makes makes running tests in parallel hard. It encourages interdependencies
between tests which makes it hard to run just one test to debug a failure.
Exeter[0] is a set of tiny libraries for several languages (C, sh and Python so far) with a new approach to TAP's goals. You write tests in
a way that's natural for the language, then register them with exeter. Exeter turns this into an executable which can be invoked either to
run a single test or to produce a manifest of tests. You can programmatically construct complex matrices in your language of
choice.
Exeter doesn't run a testsuite or collate results itself. Instead you feed the exeter manifest into an external test runner instructing it
how to run all the defined tests. We explicitly support Avocado[1] as a test runner and there's limited support for Meson[2] which we hope
to improve.
This talk discusses why I created Exeter, what it can do, and as a case study, how we're using it to improve testing for the
passt/pasta[3] project.
[0] https://gitlab.com/dgibson/exeter
[1] https://avocado-framework.github.io/
[2] https://mesonbuild.com/
[3] https://passt.top
Automated testing and has become very common in the open source world over the last two decades. Modern languages like Go and Rust have built in support making it easy to start write unit tests and other external test frameworks abound. But if you're not using a fancy new language, or your implementation language isn't convenient for writing testsor your project uses multiple langauges, the language centric systems don't work. Meanwhile, some of the external frameworks impose a language for writing tests which might be inconvenient for you. Many of them impose a style of writing tests that could be unnatural for your language. In particular JUnit's influence often brings Java-isms to languages where they don't belong. Building arrays or matrices of similar tests might require framework specific approaches syntax which can be awkward or limited. Then there's TAP, the "Test Anything Protocol" from the Perl world. It lets you write test programs in any language and collate the results. However, TAP is showing its age: the protocol structure implies sequential test execution; with test's identified only by a number which can change if tests are added or removed. That makes makes running tests in parallel hard. It encourages interdependencies between tests which makes it hard to run just one test to debug a failure. Exeter[0] is a set of tiny libraries for several languages (C, sh and Python so far) with a new approach to TAP's goals. You write tests in a way that's natural for the language, then register them with exeter. Exeter turns this into an executable which can be invoked either to run a single test or to produce a manifest of tests. You can programmatically construct complex matrices in your language of choice. Exeter doesn't run a testsuite or collate results itself. Instead you feed the exeter manifest into an external test runner instructing it how to run all the defined tests. We explicitly support Avocado[1] as a test runner and there's limited support for Meson[2] which we hope to improve. This talk discusses why I created Exeter, what it can do, and as a case study, how we're using it to improve testing for the passt/pasta[3] project. [0] https://gitlab.com/dgibson/exeter [1] https://avocado-framework.github.io/ [2] https://mesonbuild.com/ [3] https://passt.top