"Journal of the Association for Consumer Research" and vague implied effects on mental state both pretty good indicators that this will not replicate. About half of psychology papers don't replicate anyway, the field has been doing some house cleaning, which is awesome, but needs to do a lot more.
Oh you can add that all the researchers are marketing professors at business schools (one now at Snap). Good job on the marketing though, perfectly designed to get headlines.
I find this response overly dismissive. Did you read the paper or just the abstract?
Dismissing the work of marketing professors out of hand isn't the right approach. What if this is one of the half of psychology papers that do replicate?
While I definitely agree that one shouldn't be overly dismissive without reading the underlying paper, the abstract can actually be surprisingly predictive.
"Beyond statistical issues, it strikes me that several of the studies that didn’t replicate have another quality in common: newsworthiness. They reported cute, attention-grabbing, whoa-if-true results that conform to the biases of at least some parts of society. One purportedly showed that reading literary fiction improves our ability to understand other people’s beliefs and desires. Another said that thinking analytically weakens belief in religion. Yet another said that people who think about computers are worse at recalling old information—a phenomenon that the authors billed as “the Google effect.” All of these were widely covered in the media."
Scanned the paper too with sci hub. Nothing remarkable.
But we should be dismissive of any new results from psychology, it just doesn't have systems in place to validate claims. There is some cool stuff in psychology that has been replicated 20 times, across different cultures, and over time. But the chances of a headline psychology paper being true are, generously, 5%.
To be far to the authors, they are in a bit of a bind. In order to get their Phd, and progress in their academic career they have to do "original" research.
For psychology for the last 40 years, this means do stuff like this. Get cohorts together and test claims. When one is statistically significant, publish. They really didn't have much of a choice other than drop their career. They are probably nice people who just want to teach college classes. Misinforming people is an unintended side effect and more an indictment of academia than of them.
I think skepticism is warranted, but dismissiveness is not. Honestly it's a little offensive.
Also, in this case their findings aren't even counter intuitive or that surprising. They are just measuring something that most of use believe already (judging by the rest of these comment threads).
The fact that most of us already believe it is reason for increased skepticism. We have a bias towards accepting things that mesh with our existing beliefs.
That doesn't mean their results are untrue. We just have to be careful not to overestimate the strength of this evidence.
Being a little offensive is fine. The question and argument should be whether it's a corre t assessment about the field, not whether it will hurt their feelings to read this. If it's false, argue that. It can be offensive to priests to say there's no god. It doesn't by itself make it false.
That’s fair. But you are just proving my point. This research was dismissed out of hand, it wasn’t actually engaged with to see if it’s a correct assessment or not. That seems to happen whenever any sort of soft science research is posted here. There is a contingent of folks that seems to believe soft science is an oxymoron and therefore shouldn’t even be tried.
It's not necessarily bad habit to kind of purge HN of these random "paper announcements". Wait until there's a meta-analysis and let's submit that and talk about that.
Regarding the folks who are militantly ignorant about the science of soft science, alas I have to agree, they are a bit of a problem.
I read part of the paper. Note the graphs where the y axis is as short as possible to make the effect size look larger. The title is explicitly non-serious. They also represent that they can measure cognitive capacity and fluid intelligence reliably and meaningfully, instead of presenting the data more clearly in the form of... data, eg, raw text scores. It also lacks the sort of control group that has a non-smartphone distraction present, so there isn't conclusive evidence that its the smartphone itself. What if it was a chessboard or a sandwich?
They need to hold themselves to more serious standards, this makes science itself look bad
In the physics lab the T.A. will yell at you if choose a scale that starts at zero instead of a scale that shows only the relevant section. When the T.A. is tired, he/she will ask another T.A. to continue the yelling. (Or sometimes just make you redo the report until you choose the correct scale.)
The research journal have a similar policy.
For some reason people in the Internet don't like it, so a solution for a blog post is to show both graphs. One that starts at zero and other with the relevant section.
It's worse than half. In some areas it's 2/3rds of the best!
No one is actually replicating the mass of research, they are trying to replicate highly cited results. Of those its half.
The vast majority of social science research is unreplicable, primarily because it uses dramatically under-powered association modelling to make causal claims. It's dressed-up astrology.
I share your sentiment, but to make it more concrete, here are a few quickly glanced things that give good/bad indications:
+ >500 participants
+ Report not only aggregated values, but also bar charts with confidence interval (you can compare for yourself)
- Charts don't start at zero (gives reader wrong impression about effect size)
Need to read it more carefully though, because I would like to agree with the paper, since I notice this effect myself a lot. Also very related to ego depletion, which doesn't seem to be even mentioned in the paper, weirdly enough. Maybe they wanted to coin their own term.
Oh you can add that all the researchers are marketing professors at business schools (one now at Snap). Good job on the marketing though, perfectly designed to get headlines.
https://www.colorado.edu/business/adrian-f-ward
https://www.rotman.utoronto.ca/FacultyAndResearch/Faculty/Fa...
https://rady.ucsd.edu/people/faculty/ayelet-gneezy/
https://www.linkedin.com/in/maartenbos/