OpenAI has conceded that it bungled its response after it discovered that its models had access Australian government sites.
The breaches happened in June, during internal training and evaluation, and were discovered by Open AI in August. But they were only communicated to the Australian authorities in September, through a general email address.
The company has now released a statement acknowledging not only the breach but its poor response.
“We are sorry and working to do better in the future,” according to a statement from Open AI.
In that statement, the company sets out what it knows, what it has changed, and what it will do to rebuild trust with the Australian people.
“This is a new kind of cyber incident which represents an emerging global challenge.”
One of the ways OpenAI intends to take accountability for the situation is to be intentional in working with Australia to help develop practical approaches to how AI developers and governments identify, disclose, and respond to AI cyber behaviour, whether malicious or unintentional.
After the Hugging Face incident in July, OpenAI began reviewing earlier training and evaluation activity to identify other affected organisations. In mid-August, that review identified activity affecting the Australian government websites below.
- Services Australia: An OpenAI model discovered a way to gain non-public access to the service, and ran commands, retrieved internal files, credentials and aggregate statistics, and wrote files. However, individual patient or client records were not accessed.
- NSW Bureau of Crime Statistics and Research (BOCSAR): An OpenAI model accessed BOCSAR’s public Crime Mapping Tool to research public crime statistics (we explain further below why models carry out various information research tasks). The model made API and website metadata requests via the public BOCSAR tool, which supplies credentials for browser API requests. The BOCSAR system returned application configuration, operational jobs and logs, and website metadata. Crime records of individuals were not accessed.
- Victorian Department of Health: OpenAI agents discovered an exposed access key to query the Victorian Agency for Health Information’s reporting system and retrieve reporting configuration and aggregate survey statistics. The extent to which this information should have been accessible is unclear, and depends on VAHI’s access policies. Individual medical records or identifiable survey responses were not accessed.
- Australian Institute of Health and Welfare: OpenAI agents retrieved aggregate statistics using third-party browsing and download services, including from AIHW’s website, and queried chart data directly. Separate attempts to bypass access controls were unsuccessful. The downloaded material appears to have been publicly available. There was no system compromise. Individual medical records were not accessed.
Australia Medicare Statistics Reporting Service
During internal training and evaluation in June, OpenAI was running an experimental, internal-only OpenAI model that was not intended for public release and without the full set of safeguards used in it publicly available products.
In the course of this training and evaluation, it accessed Services Australia’s Medicare Statistics Reporting Service. The company’s review to date has found no evidence that anyone’s medical records were accessed.
“When we do internal training and evaluation on our models, we assign them tasks drawn from a broad collection of research questions spanning many subjects, reflecting the kinds of detailed questions users might ask,” according to the statement.
“This trains a model to find, interpret and analyse publicly available information so the model can be more useful to people. Our models are supposed to answer these questions using publicly published statistics.
“In this case, one of the tasks assigned to the model was to research government spending per person on medicines for skin conditions in Victorian communities. The model had difficulty obtaining that information, and it took actions that we had not authorised it to take.
“In the course of looking for this information at Services Australia’s Medicare Statistics Reporting Service, it discovered a way to gain non-public access to the service. It then used this access to review technical system information and source code related to the service – all still with the objective of trying to find the information it was originally looking for. We did not intend for this activity to occur, and the access to the service and follow-on activity should not have happened.”
What is changing
OpenAI has strengthened research safeguards, including additional network restrictions and expanded monitoring. It has implemented controls to block live internet access in these research environments, with web access served through cached content.
As an additional layer of security, current monitoring systems would have detected this activity and paged the team for urgent human review.
OpenAI has also paused training and evaluation involving tool use for its most capable models and will resume training them only when it is confident additional safeguards are in place.
The company has also joined organisations across technology, cybersecurity and critical infrastructure in a call for collective action on cyber defence.