Actually, the experiments that need to be conducted wouldn't be that hard to do. In fact I'd be surprised if research of this kind has not already been done.
The problem of poor performance under scrutiny, where there is pressure for superior performance (as in a job interview) is well-described as a form of social anxiety disorder (or social phobia), a common condition affecting ~10% of adults in the US (lifetime prevalence). Of course, there's a range of mild to severe symptoms, nonetheless it affects a significant population.
The implication is that in a not-so-pressured setting candidates might perform very differently. Furthermore, writing code, solving software problems are generally incremental processes more akin to watching paint dry than putting out fires for all the externally visible action there is to see. A "whiteboard" exam likely isn't a good model of the real requirements of the job.
There's an enormous amount of research on testing methodology, testing is a huge industry. Ironically enough, one that is extremely reliant on software for analyzing test data in order to determine what is a good test of sets of knowledge or actual abilities. Seems like there's a clue in there somewhere about how software enterprises could find out who is really good at creating software.
The problem of poor performance under scrutiny, where there is pressure for superior performance (as in a job interview) is well-described as a form of social anxiety disorder (or social phobia), a common condition affecting ~10% of adults in the US (lifetime prevalence). Of course, there's a range of mild to severe symptoms, nonetheless it affects a significant population.
The implication is that in a not-so-pressured setting candidates might perform very differently. Furthermore, writing code, solving software problems are generally incremental processes more akin to watching paint dry than putting out fires for all the externally visible action there is to see. A "whiteboard" exam likely isn't a good model of the real requirements of the job.
There's an enormous amount of research on testing methodology, testing is a huge industry. Ironically enough, one that is extremely reliant on software for analyzing test data in order to determine what is a good test of sets of knowledge or actual abilities. Seems like there's a clue in there somewhere about how software enterprises could find out who is really good at creating software.