Automated testing is useful and good. But I really feel it's reached a lever of fetishisation that is quite concerning.
Testing code is code which needs to be written, read, maintained, refactored. Very often nowadays I have to wade through tests which test nothing useful, except syntax. Even worse, with developers who adopt the mock-everything approach, I often find tests which only verify that the implementation is exactly the one they wrote, which is even worse: it makes refactoring a pain, because, even if you rewrote a method in a better way which produces exactly the results you wanted, the test will fail.
So, the ratio of testing code vs implementation code is a completely wrong proxy for code quality.
EDIT: I'm not criticising SQLite and their code quality - which I never studie - but the idea that you can judge code quality for a project just by the ratio of test code vs implementation code.
They actually have to test to that degree to follow aviation standards (DO-178b [0]) because they're used in aviation equipment.
Dr. Hipp said he started really following it when Android came out and included SQLite and suddenly there were 200M mobile SQLite users finding edge cases: https://youtu.be/Jib2AmRb_rk?t=3413
Lightly edited transcript here:
> It made a huge difference. That that was when Android was just kicking off. In fact Android might not have been publicly announced, but we had been called in to help with getting Android going with SQLite. [Actually], they had been publicly announced and there were a bunch of Android phones out and we were getting flooded with problems coming in from Android.
> I mean it worked great in the lab it worked great in all the testing and then [...] you give it to 200 million people and let them start clicking on their phone all day and suddenly bugs come up. And this is a big problem for us.
> So I started doing following this DO-178b process and it took a good solid year to get us there. Good solid year of 12 hour days, six days a week, I mean we really really pushed but we got it there. And you know, once we got SQLite to the point where it was at that DO-178b level, standard, we still get bugs but you know they're very manageable. They're infrequent and they don't affect nearly as many people.
> So it's been a huge huge thing. If you're writing an application deal ones, you know a website, a DO-178b/a is way overkill, okay? It's just because it's very expensive and very time-consuming, but if you're running an infrastructure thing like SQL, it's the only way to do it.
SQlite is very high quality software, but they use DO-178b "inspired" testing process. As far as I know they don't have version of software that is or can be used in safety critical parts despite their boasting.
They say in their site that:
> Airbus confirms that SQLite is being used in the flight software for the A350 XWB family of aircraft.
Flight software does not imply safety critical parts of avionics. It can be the entertainment system or some logging that is not critical.
Correct. The key word is "inspired". Multiple companies have run a DO-178B cert on SQLite, I am told, but the core developers did not get to participate, and I think the result was level-C or -D.
While all that was happening 10+ years ago, I learned about DO-178B. I have a copy of the DO-178B spec within arms reach. And I found that, unlike most other "quality" standards I have encountered, DO-178B is actually useful for improving quality.
I originally developed the TH3 test suite for SQLite with the idea that I could sell it to companies interested in using SQLite in safety-critical applications, and thereby help pay for the open-source side of SQLite. That plan didn't work out as nobody ever bought it. But TH3 and the discipline of 100% MC/DC testing was and continues to be enormously helpful in keeping bugs out of SQLite, and so TH3 and all the other DO-178B-inspired testing and refactoring of SQLite has turned out to be well worth the thousands of hours of effort invested.
The SQLite project is not 100% DO-178B compliant. We have gotten slack on some of the more mundane paperwork aspects. Also, we aggressively optimize the SQLite code base for performance, whereas in a real safety-critical application the focus would be on extreme simplicity at the cost of reduced performance.
However, if some company does call us tomorrow and says that they want to purchase a complete set of DO-178B/C Level-A certification artifacts from us, I think we could deliver that with a few months of focused effort.
Yeah DO-178B gives several levels for software from DALA (highest) to DALE (lowest). If DALA software fails the results are catastrophic if DALE fails there is no effect on the aircraft. Since DALE is usually just test equipment and such they might be at a DALD level. So still requires a lot of testing but not nearly to the level that DALA requires.
I think it's possible that parts of SQLite, for example file format in read only mode and few constant queries are certified as part of some safety critical software.
Hipp's Hwaci consulting company would probably help to do the work, but it has no relation to the SQLite as a library.
Good point. The video I linked to merely says that he was contacted by someone in the aviation space about the standard, which I took to mean that it was used in avionics.
While I agree in general I disagree here. If you read about the Sqlite tests you will find that they do test sensibly.
One suite I'm particularly impressed with will run tests from zero bytes with slowly increasing available memory until the program passes. The tests verify that at no point the DB is corrupted by an OOM event.
Just to clarify, I wasn't criticising sqlite, I was criticising the idea of judging their code quality "for this reason alone!" - ie that they have so much test code vs implementation.
As a heuristic the code versus test-code ratio serves well as an indicator of quality. Just like consistent indentation does. You don't know whether a well-indented program is good. But if the indentation is inconsistent you'll expect worse.
Yes, bad tests are bad. Yes, mocks are bad. Good tests, however, are good.
To expound on that, designing for testability allows you to sidestep the need for mocks almost entirely, and forces you into easier, more reliable and more consistent code. Then when you choose to test it, the tests are simple, straightforward and valuable.
Oh boy, I would give your comment an infinite number of up-votes. Yes, testing has reached fetish-like levels.
Some of the test code I've encountered recently has been more voluminous, complex and has taken more man hours to develop and maintain than the application or library it's assigned to.
For the love of God, develop the damned software! It's either going to work or it's not.
Yeah, yeah. Just venting. The products I work on aren't the most important, but they certainly are quite important and most of the testing infrastructure that has been built thus far has a lot of goofy sh@t in it.
I just don't have a high tolerance for needless complexity and gee-whiz-look-what-I-can-do while the clock is running.
Testing code is code which needs to be written, read, maintained, refactored. Very often nowadays I have to wade through tests which test nothing useful, except syntax. Even worse, with developers who adopt the mock-everything approach, I often find tests which only verify that the implementation is exactly the one they wrote, which is even worse: it makes refactoring a pain, because, even if you rewrote a method in a better way which produces exactly the results you wanted, the test will fail.
So, the ratio of testing code vs implementation code is a completely wrong proxy for code quality.
EDIT: I'm not criticising SQLite and their code quality - which I never studie - but the idea that you can judge code quality for a project just by the ratio of test code vs implementation code.