CAREER: Improving Work Quality and Worker Safety in AI-Supported Crowdwork

University Of Tulsa, Tulsa OK

Investigators

Abstract

Crowdsourcing is a technique for breaking large problems that require human input into bite-sized tasks that can be divided among a number of "crowd workers" who complete the tasks for pay. Crowdsourcing is used in many areas: artificial intelligence (AI) tasks require large-scale datasets with human-created labels; research studies with people benefit from crowd workers both as study participants and as data analysts; and many companies use crowd work for their own business needs. As with many other areas of work and life, the rapid growth of AI is also poised to transform crowdsourcing: workers may be able to collaborate with AI-based tools to work more accurately and efficiently. However, risks also arise from workers misusing AI-based systems, errors in the AI systems themselves, and privacy risks that AI systems might pose to workers. This project aims to address these risks, and in the process advance understanding of human-AI collaboration, create safer AI-based systems, and develop more ethical crowdwork systems. The project team will also develop educational modules that bring more disciplinary perspectives to cybersecurity education and that integrate cybersecurity concepts into courses across disciplines. These modules, along with the theories, datasets, and tools developed in the research, will be widely shared to support educational, research, and practical impacts. This project addresses emerging security and ethical challenges in AI-integrated crowdsourcing through three main research objectives. First, it seeks to develop trustworthy human-AI collaborative annotation systems by improving robustness against adversarial inputs, including optimization-based white-box attacks and black-box attacks. Second, it proposes signature-based and behavior-based detection strategies to identify the misuse of generative AI in crowdsourcing. Third, it aims to design and implement worker-centric and security-aware AI assistants to help workers navigate crowdsourcing platforms more effectively while protecting their privacy and security. The project's contributions include novel technical solutions, such as robust human-AI collaboration pipelines and misuse detection algorithms, grounded in empirical studies of real-world crowdsourcing systems. The research builds on recent progress in human-AI collaboration but goes further by focusing on security threats and worker protection. The project will also yield tangible deliverables: open-source tools and methodologies, educational modules, and security-aware interfaces for crowdsourcing platforms. These innovations will be widely disseminated and integrated into education and research infrastructure. This project advances understanding of AI's impact on the crowdsourcing ecosystem and establishes a foundation for developing secure, trustworthy, and ethical human-AI collaborative systems. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.