The dot-guessing game: A ‘fruit fly’ for human computation research
Status: Working paper
Last updated: 2012
Summary
I propose a human computation task that has a number of properties that make it useful for empirical research. The task itself is simple: subjects are asked to guess the number of dots in an image. The task is useful because: (1) it is simple to generate examples; (2) it is a CAPTCHA; and (3) it has an objective solution that allows us to finely grade performances, yet subjects cannot simply look up the answer. To demonstrate its value, I conducted two experiments using the dot-guessing task. Across both experiments, I found that the “crowd” performed well, with the arithmetic mean guess besting most individual guesses. However, I found that subjects displayed well-known behavioral biases relevant to the design of human computation systems. In the first experiment, subjects reported whether the number of dots in an image was larger or smaller than a randomly chosen number. The randomly chosen reference number strongly anchored subjects’ subsequent guesses. In the second experiment, subjects were asked to choose between an incentive payment contract that rewarded good work (and self-knowledge about relative ability) and a fixed payment contract. I found that selection of the incentive contract did not improve performance, nor did it reveal information about relative ability. However, subjects who chose the incentive contract were more likely to be male, suggesting that their choice in contracts was determined by differences in risk-aversion.
Paper links
Cite
@unpublished{horton2012thedotguessinggameafruitflyforhumancomputationresearch,
title = {{The dot-guessing game: A ‘fruit fly’ for human computation research}},
author = {Horton, John J.},
year = {2012},
url = {https://john-joseph-horton.com/papers/the-dot-guessing-game-a-fruit-fly-for-human-computation-research/}
}