Uriel Cohen Priva



Easier Amazon Mechanical Turk worker selection

In some of the experiments I am currently running I have to throw away some 80% of the data because I am interested only in workers that have very specific characteristics. Because it is in the turkers’ best interest to lie to me if I require these characteristics, I have to ask them rather that require that they have the required properties, and accept their HITs regardless.

However, Amazon Mechanical Turk have just introduced a new interface that allows defining and assigning “qualifications” to workers that performed well in the past:

http://www.slideshare.net/mechanicalturk/manage-your-workforce-fall-2010-requester-product-update

All I have to do in order to use this feature is create a new qualification type for the workers that have reported that they have the right characteristics in the past (on the new “Qualification Types” tab under “manage”), assign the new qualification to those workers (under the new “Workers” tab under “manage”), and modify my HITs so that only workers that have that qualification can do the HIT. Letting the verified workers make more money on such HITs is not a bad idea either.

Instructions:

  • Create a new qualification under the qualification tab.
  • Wait a few minutes for the rest of the system to know about the new qualification. Let’s suppose the name of the qualification is “myqual”.
  • Download the workers file under the workers tab.
  • There will be a column name “UPDATE-myqual”.
  • Update “UPDATE-myqual” with a numeric value between 0 and 100.
  • Upload the file under the workers tab and approve the changes.
  • Modify the HITs to require a sufficient level of myqual.