Recent years have seen an increased interest in crowdsourcing as a way
of obtaining information from a potentially large group of workers at
a reduced cost. The crowdsourcing process, as we consider in this
paper, is as follows: a requester hires a number of workers to work on
a set of similar tasks. After completing the tasks, each worker
reports back outputs. The requester then aggregates the reported
outputs to obtain aggregate outputs. A crucial question that arises
during this process is: how many crowd workers should a requester
hire? In this paper, we investigate from an empirical perspective the
optimal number of workers a requester should hire when crowdsourcing
tasks, with a particular focus on the crowdsourcing platform Amazon
Mechanical Turk. Specifically, we report the results of three studies
involving different tasks and payment schemes. We find that both the
expected error in the aggregate outputs as well as the risk of a poor
combination of workers decrease as the number of workers
increases. Surprisingly, we find that the optimal number of workers a
requester should hire for each task is around 10 to 11, no matter the
underlying task and payment scheme. To derive such a result, we employ
a principled analysis based on bootstrapping and segmented linear
regression. Besides the above result, we also find that overall
top-performing workers are more consistent across multiple tasks than
other workers. Our results thus contribute to a better understanding
of, and provide new insights into, how to design more effective
crowdsourcing processes.