[Back]

T. Hammerer:
"A Simulation Framework for Task-based Crowdsourcing with Auctions";
Supervisor: S. Dustdar, B. Satzger; Institut für Informationssysteme, AB Verteilte Systeme, 2012; final examination: 06-18-2012.

Crowdsourcing describes a model for distributed online problem solving harnessing the creative solutions as well as the scalability and availability of the distributed network of individuals. A
popular way of crowdsourcing where one can outsource a task to a crowd and receives a result in reasonable time is referred to as task-based crowdsourcing (TBCS). A multitude of different TBCS platforms is currently available. However, they have to cope with certain challenges. Firstly, they have to address the issue of the varying quality of user-processed tasks and secondly, they have to find solutions for distributing tasks in a way that all stakeholders get the highest attainable benefit. For the latter, the use of auctions as a means of task distribution seems to be an interesting approach, which has not been discussed in related literature yet. Another challenge in TBCS is the difficulty of testing and evaluating new approaches due to the scarcity of "real" data for evaluations.
The contribution of this thesis is twofold. Firstly, we address the lack of an appropriate simulation framework for task-based crowdsourcing and come up with a solution. We introduce a highly
configurable, modular and extensible framework which is grounded on an agent-based modeling approach. The framework supports state of the art quality management methods based on a skill profiling module and various auction mechanisms to simulate task distribution methods.
Secondly, we discuss the suitability of different auction types for various crowdsourcing scenarios. By means of the simulation framework, we simulate five different scenarios and compare three reversed standard auction types (sealed-bid, Dutch, English) and one double auction type (continuous double auction) with each other. Based on the analysis of quality, task distribution and payoff we find that among the standard auction types, the sealed-bid auction provides the most balanced results, whereas in comparison to the double auction market, the latter provides slightly better results in most categories. We have further analyzed three different quality management methods and come to the conclusion that the best quality can be achieved by using an auction mechanism that matches the task to the best suited worker. The most balanced approach, however, is to use a qualification policy.