terriko: I am a serious academic (Twlight Sparkle looking confused) (Serious Academic)
[personal profile] terriko
This was originally posted on But Grace, but I don't want my regular blog to wind up devoid of technical content so I'll be crostposting all my posts from there in their entirety, I suspect.

One of the cool things going on at work is some software we have that automates the creation of small bug fixes. We're looking to try it on some more active projects with real bugs, but we need projects with reasonable test case coverage so that the automated system can also ensure that it isn't causing other things to break in making the fix. Basically, we're potentially offering up a bunch of free bugfixes if your open source project has decent test cases. Pretty good deal, I hope.

Open source projects with good test suites

But how do we find software with good test cases? Here's a few I know of off the top of my head:

Open source projects with good test suites

Name/URLTest suite?Notes
Firefox A brief search turns up some automated tests My fuzzy memory suggests there was more than these...
Gnumeric Extensive regression tests for each function The function tests are in .xls spreadsheets, so we could potentially apply them to other spreadsheet software.
SQLite They claim extensive test coverage Very promising!
Webkit (Chrome, Safari) a brief web search turns up regression tests for javascript I believe the Chromium project has even more tests

Can anyone suggest other software or more details (and better links) on the things I have mentioned already?

Open source routing software

For various reasons, I've been encouraged to try experiments on open source routing software. There's some existing academic literature on the types of bugs found in open source routers, and it seems like our automated patch creation system would be a good fit especially since router bugs often cause huge outages or security problems and having a temporary patch to solve the problem right away could be a huge boon.

My query on twitter generated a nice list of open source router software, but no one seems to know anything about test suites. Here's a table summarizing what I've found thus far:

Open source router software test suite information

Name/URLTest suite?Notes
Click Unknown Nothing obvious, and given that it's on a university website, I'll be shocked if it has testing. ;)
dd-WRT Unknown Nothing obvious in the wiki, but there were lots of hits I haven't investigated.
OpenWRT Unknown Clearly there was interest in automated test suites in Jan 2011 but it's unclear to me if these are now around somewhere. Need to look more.
pfSense No evidence of a test suite Searching the dev wiki for "test" yields nothing likely, so I'm guessing there isn't one.
Quagga There is a tests/ directory, but it looks unsuitable "make test" doesn't work and "make check" pokes a bunch of directories but doesn't seem to do what I need. There's a directory called tests/ in the repository, but I'm not sure what it does. I can run the tests manually, but the output is currently meaningless to me. No one answered my question on #quagga, although another open source friend on #kernel.org suggested that the test suite may have been abandoned.
Tomato No evidence of a test suite The web site contains nothing useful, so if there is a test suite, it's likely being provided by someone else.
XORP There is a tests/ directory, unsure if suitable but look promising These look promising, but I'm having some build errors and haven't been able to run them yet

You'd think, perhaps, that reasonable test suites for routers would already exist. A generic test suite would be totally sufficient for my needs at the moment. And in fact, I've found a set of routing tests from the University of New Hampshire InterOperability Laboratory, but while their tests are well-described, it doesn't look like something we can run locally and repeatedly as we'd need to in order to test the auto-generated patches. I haven't yet found others.

Let's be clear: I don't really care what the router supports in great detail. The important thing for these tests are that there be a good test suite, and preferably a good bug queue so we can grab candidate bugs and bug test cases to try to solve them. Generally speaking, the bugs have been easier to find than the regression tests.

In summary...

I am looking for:

1. More information about router test suites.
2. Updates to my current tables of information. This represents a morning's work, so I'd be shocked if they're perfectly correct.
3. Any open source/free software projects with good test suites (and preferably good bug queues).

Again, the key here is that I need good test suites. I'm most interested in routing software at the moment, but I'm building up a list of alternative ideas if that doesn't pan out, so anything with good automated tests I'll be able to run repeatedly is of potential interest. We've got access to a reasonable amount of computing power, so heavier weight tests are fine as long as they aren't going to take all month to run.

We would love to contribute any fixes we find back to the community, so if you think your project might qualify please get in touch! I think the end result is going to be awesome for all involved: free bug fixes for the project, more impressive real-world validation of our automated patch creation system, and maybe even an academic paper out of it for some of the folk around here.

CPAN modules have great tests

Date: December 5th, 2011 10:23 pm (UTC)
ext_112544: (Default)
From: [identity profile] schwern.myopenid.com
I can point out some Perl CPAN modules with excellent test suites. Usually something like mid 90% code coverage, and they are run with every install. All CPAN modules have a bug tracker on rt.cpan.org (http://rt.cpan.org) and some have an additional one on Github (http://github.com). All CPAN modules use the same system for testing and it can all be run automatically. I presume this uniformity, strange in the Perl world, will be very useful to you.

Results are gathered and indexed by the CPAN Testers (http://cpantesters.org/). They even have rudimentary automatic analysis (http://analysis.cpantesters.org/) by looking at what Perl configurations, operating systems and prerequisite modules correlate to failures.

On to the projects with good test suites.

Fittingly, Test::More (https://github.com/schwern/test-more) has an excellent test suite (https://github.com/schwern/test-more/tree/Test-Builder1.5/t) and nearly full coverage. Testing a test library is "special". It's used by pretty much everyone. I'm completing a near total rewrite (that's what the Test-Builder1.5 branch is), and having an excellent test suite has proven most useful.

perl5i (https://github.com/schwern/perl5i) is another with very good test coverage. Don't be thrown off by perl5i::1, which is an older version shipped for compatibility.

If you want something very small, Object::ID (https://github.com/schwern/Object-ID) and CLASS (http://search.cpan.org/dist/CLASS) are good.

Method::Signatures (https://github.com/schwern/method-signatures), URI::Find (https://github.com/schwern/uri-find), Moose (http://search.cpan.org/dist/Moose), Mouse (http://search.cpan.org/dist/Mouse), DateTime (http://search.cpan.org/dist/DateTime) and DBI (http://search.cpan.org/dist/DBI) are good candidates for larger and more complicated suites.

Date: December 6th, 2011 09:12 pm (UTC)
From: (Anonymous)
Hi there. Your request sounds quite interesting.

The NetBSD project has been working steadily on what will eventually (hopefully) become a big comprehensive testsuite of the entire operating system with the ATF (Automated Testing Framework) http://www.netbsd.org/~jmmv/atf/ - the tests that have been written so far are in a "tests" dir under the "src" tree. http://cvsweb.netbsd.org/bsdweb.cgi/src/tests

PostgreSQL is another high-quality code project with an impressive testsuite: http://www.postgresql.org/docs/current/static/regress-run.html


terriko: (Default)

October 2014

5678 91011
1920 2122232425

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Oct. 23rd, 2014 06:30 pm
Powered by Dreamwidth Studios