![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Entry tags:
Looking for open source projects with good test suites
This was originally posted on But Grace, but I don't want my regular blog to wind up devoid of technical content so I'll be crostposting all my posts from there in their entirety, I suspect.
One of the cool things going on at work is some software we have that automates the creation of small bug fixes. We're looking to try it on some more active projects with real bugs, but we need projects with reasonable test case coverage so that the automated system can also ensure that it isn't causing other things to break in making the fix. Basically, we're potentially offering up a bunch of free bugfixes if your open source project has decent test cases. Pretty good deal, I hope.
But how do we find software with good test cases? Here's a few I know of off the top of my head:
Can anyone suggest other software or more details (and better links) on the things I have mentioned already?
For various reasons, I've been encouraged to try experiments on open source routing software. There's some existing academic literature on the types of bugs found in open source routers, and it seems like our automated patch creation system would be a good fit especially since router bugs often cause huge outages or security problems and having a temporary patch to solve the problem right away could be a huge boon.
My query on twitter generated a nice list of open source router software, but no one seems to know anything about test suites. Here's a table summarizing what I've found thus far:
You'd think, perhaps, that reasonable test suites for routers would already exist. A generic test suite would be totally sufficient for my needs at the moment. And in fact, I've found a set of routing tests from the University of New Hampshire InterOperability Laboratory, but while their tests are well-described, it doesn't look like something we can run locally and repeatedly as we'd need to in order to test the auto-generated patches. I haven't yet found others.
Let's be clear: I don't really care what the router supports in great detail. The important thing for these tests are that there be a good test suite, and preferably a good bug queue so we can grab candidate bugs and bug test cases to try to solve them. Generally speaking, the bugs have been easier to find than the regression tests.
I am looking for:
1. More information about router test suites.
2. Updates to my current tables of information. This represents a morning's work, so I'd be shocked if they're perfectly correct.
3. Any open source/free software projects with good test suites (and preferably good bug queues).
Again, the key here is that I need good test suites. I'm most interested in routing software at the moment, but I'm building up a list of alternative ideas if that doesn't pan out, so anything with good automated tests I'll be able to run repeatedly is of potential interest. We've got access to a reasonable amount of computing power, so heavier weight tests are fine as long as they aren't going to take all month to run.
We would love to contribute any fixes we find back to the community, so if you think your project might qualify please get in touch! I think the end result is going to be awesome for all involved: free bug fixes for the project, more impressive real-world validation of our automated patch creation system, and maybe even an academic paper out of it for some of the folk around here.
One of the cool things going on at work is some software we have that automates the creation of small bug fixes. We're looking to try it on some more active projects with real bugs, but we need projects with reasonable test case coverage so that the automated system can also ensure that it isn't causing other things to break in making the fix. Basically, we're potentially offering up a bunch of free bugfixes if your open source project has decent test cases. Pretty good deal, I hope.
Open source projects with good test suites
But how do we find software with good test cases? Here's a few I know of off the top of my head:
Open source projects with good test suites
| Name/URL | Test suite? | Notes |
| Firefox | A brief search turns up some automated tests | My fuzzy memory suggests there was more than these... |
| Gnumeric | Extensive regression tests for each function | The function tests are in .xls spreadsheets, so we could potentially apply them to other spreadsheet software. |
| SQLite | They claim extensive test coverage | Very promising! |
| Webkit (Chrome, Safari) | a brief web search turns up regression tests for javascript | I believe the Chromium project has even more tests |
Can anyone suggest other software or more details (and better links) on the things I have mentioned already?
Open source routing software
For various reasons, I've been encouraged to try experiments on open source routing software. There's some existing academic literature on the types of bugs found in open source routers, and it seems like our automated patch creation system would be a good fit especially since router bugs often cause huge outages or security problems and having a temporary patch to solve the problem right away could be a huge boon.
My query on twitter generated a nice list of open source router software, but no one seems to know anything about test suites. Here's a table summarizing what I've found thus far:
Open source router software test suite information
| Name/URL | Test suite? | Notes |
| Click | Unknown | Nothing obvious, and given that it's on a university website, I'll be shocked if it has testing. ;) |
| dd-WRT | Unknown | Nothing obvious in the wiki, but there were lots of hits I haven't investigated. |
| OpenWRT | Unknown | Clearly there was interest in automated test suites in Jan 2011 but it's unclear to me if these are now around somewhere. Need to look more. |
| pfSense | No evidence of a test suite | Searching the dev wiki for "test" yields nothing likely, so I'm guessing there isn't one. |
| Quagga | There is a tests/ directory, but it looks unsuitable | "make test" doesn't work and "make check" pokes a bunch of directories but doesn't seem to do what I need. There's a directory called tests/ in the repository, but I'm not sure what it does. I can run the tests manually, but the output is currently meaningless to me. No one answered my question on #quagga, although another open source friend on #kernel.org suggested that the test suite may have been abandoned. |
| Tomato | No evidence of a test suite | The web site contains nothing useful, so if there is a test suite, it's likely being provided by someone else. |
| XORP | There is a tests/ directory, unsure if suitable but look promising | These look promising, but I'm having some build errors and haven't been able to run them yet |
You'd think, perhaps, that reasonable test suites for routers would already exist. A generic test suite would be totally sufficient for my needs at the moment. And in fact, I've found a set of routing tests from the University of New Hampshire InterOperability Laboratory, but while their tests are well-described, it doesn't look like something we can run locally and repeatedly as we'd need to in order to test the auto-generated patches. I haven't yet found others.
Let's be clear: I don't really care what the router supports in great detail. The important thing for these tests are that there be a good test suite, and preferably a good bug queue so we can grab candidate bugs and bug test cases to try to solve them. Generally speaking, the bugs have been easier to find than the regression tests.
In summary...
I am looking for:
1. More information about router test suites.
2. Updates to my current tables of information. This represents a morning's work, so I'd be shocked if they're perfectly correct.
3. Any open source/free software projects with good test suites (and preferably good bug queues).
Again, the key here is that I need good test suites. I'm most interested in routing software at the moment, but I'm building up a list of alternative ideas if that doesn't pan out, so anything with good automated tests I'll be able to run repeatedly is of potential interest. We've got access to a reasonable amount of computing power, so heavier weight tests are fine as long as they aren't going to take all month to run.
We would love to contribute any fixes we find back to the community, so if you think your project might qualify please get in touch! I think the end result is going to be awesome for all involved: free bug fixes for the project, more impressive real-world validation of our automated patch creation system, and maybe even an academic paper out of it for some of the folk around here.