The automation of hiring decisions is a well-studied topic in
crowdsourcing. Existing hiring algorithms make a common assumption -
that each worker has a level of task competence that is static and
does not vary over time. In this work, we explore the question of how
to hire workers who can learn over time. Using a medical time series
classification task as a case study, we conducted experiments to show
that workers' performance does improve with experience and that it is
possible to model and predict their learning rate. Furthermore, we
propose a dynamic hiring mechanism that accounts for workers' learning
potential. Through both simulation and real-world crowdsourcing data,
we show that our hiring procedure can lead to high-accuracy outcomes
at lower cost compared to other mechanisms.