Tali WeinsteinAbstract
This talk was delivered at an AI South Africa meetup, examining why technically accurate AI systems can still fail the people they are meant to serve. The talk drew on three case studies (a hiring tool that systematically rejected applicants over 55, the COMPAS recidivism algorithm’s disparate risk scores for defendants based on race proxies, and a medical imaging system that learned to detect rulers) to illustrate how systems not properly evaluated can lead to societal harm. Using South Africa’s now-retracted draft AI policy as an opening for how we can start thinking about AI regulation, I outlined the policy’s strengths and its gaps. The talk closed with practical tools for AI developers, including ImpactBench, a benchmark for evaluating models on human flourishing rather than task accuracy alone, and ESG-based checks that can be built into MLOps pipelines to catch fairness and accountability issues before deployment.