IBM is set to release world’s largest annotation dataset for studying bias in facial analysis.
According to IBM fellows Aleksandra Mojsilovic and John Smith, society is paying more attention than ever to the question of bias in artificial intelligence systems, and particularly those used to recognize and analyze images of faces.
IBM is taking a number of actions to help ensure facial recognition technology is built and trained responsibly.
“One of the biggest issues causing bias in the area of facial analysis is the lack of diverse data to train systems on,” the two fellows write. “So, this fall, we intend to make publicly available a dataset as a tool for the technology industry and research community.”
The dataset includes annotations for over 1-million images to improve the understanding of bias in facial analysis being built by IBM Research scientists.
Images will be annotated with attributes, leveraging geo-tags from Flickr images to balance data from multiple countries and active learning tools to reduce sample selection bias.
Currently, the largest facial attribute dataset available is 200,000 images, so a dataset with a million images will be a major improvement.
An annotation dataset for up to 36 000 images, equally distributed across skin tones, genders and ages, annotated by IBM Research, will provide a more diverse dataset for people to use in the evaluation of their technologies.
“This will specifically help algorithm designers to identify and address bias in their facial analysis systems,” according to Mojsilovic and Smith. “The first step in addressing bias is to know there is a bias — and that is what this dataset will enable.”
Earlier this year, the accuracy of IBM’s Watson Visual Recognition service for facial analysis was improves, which demonstrated a nearly 10-fold decrease in error-rate for facial analysis.
“Our researchers continue to work with a broad range of stakeholders, users and experts to understand other biases and vulnerabilities that can affect AI decision-making, so that we can continue to make our systems better,” writes Mojsilovic and Smith.
“AI holds significant power to improve the way we live and work, but only if AI systems are developed and trained responsibly, and produce outcomes we trust. Making sure that the system is trained on balanced data, and rid of biases is critical to achieving such trust.
“As the adoption of AI increases, the issue of preventing bias from entering into AI systems is rising to the forefront. We believe no technology — no matter how accurate — can or should replace human judgement, intuition and expertise.
“The power of advanced innovations, like AI, lies in their ability to augment, not replace, human decision-making. It is therefore critical that any organization using AI — including visual recognition or video analysis capabilities — train the teams working with it to understand bias, including implicit and unconscious bias, monitor for it, and know how to address it.”