Warnings from within the industry deserve attention. In a September 6 essay, OpenAI chief scientist Jakub Pachocki called for mandatory safety thresholds for continued development, enforced through outside auditors, governments or international bodies. On September 10, Anthropic reported that it had disrupted malicious uses of its models, including cyber operations, surveillance and fraud.
AI can help improve medical diagnosis, make knowledge more accessible and streamline public services. But processing data and identifying patterns do not guarantee a correct or fair decision. People affected by those decisions need explanations, protection and the ability to appeal.
When an algorithm gets it wrong
One central risk is a flawed decision that appears objective. Systems trained on historical data can reproduce discrimination, errors and gaps in representation. Those weaknesses matter when the output influences who receives a loan, gets a job or qualifies for public assistance.
In January 2020, Detroit police wrongfully arrested Robert Williams after a false facial-recognition match. According to the American Civil Liberties Union, which represented him, he spent about 30 hours in detention. His case showed how police reliance on an incorrect match could turn a technological failure into a deprivation of liberty.
In 2024, British Columbia’s Civil Resolution Tribunal ordered Air Canada to compensate a passenger after its chatbot gave him incorrect information about bereavement fares. The airline remained responsible for the information provided on its website.
The Netherlands offered another warning. Its childcare benefits scandal exposed the dangers of discriminatory screening and failures of institutional accountability. The Dutch data protection authority found that the tax authority had processed applicants’ nationality data unlawfully and discriminatorily, as documented in the European Data Protection Board’s 2021 annual report. Describing such failures simply as technical errors obscures the responsibility of the institutions using the systems.
Generative AI also makes convincing false text, images and voices easier to produce. That expands the opportunities for impersonation, fraud and manipulation of public debate. People need practical ways to verify what they see and hear.
Human oversight in classrooms and computer systems
In education, AI can tailor exercises to a student’s pace, explain concepts and broaden access to knowledge. Overreliance can also allow a convincing answer to replace the work of understanding a problem. Students should learn to question the system, check its answers and disclose how they used it.
Children need protection from excessive data collection and dependence on automated tools. Schools must also address unequal access. Teachers and lecturers remain essential for discussion, ethical guidance and assessing how a student reached an answer.
AI agents, which can carry out tasks using software and other tools, raise a further concern: an instruction alone cannot guarantee that a system will stay within its intended limits.
On September 16, OpenAI published six reports of unexpected or concerning behavior observed during model training or evaluation, including unauthorized actions. The company cautioned that these individual cases did not establish how frequently such behavior occurred across its models. They nevertheless reinforce the need for safeguards outside the model itself.
Organizations should restrict agents to the permissions each task requires, separate testing systems from live services and require human authorization for destructive actions. Independent backups, records of actions and tested recovery procedures are basic protections. Someone must also have the authority and ability to halt an agent when it behaves unexpectedly.
The lesson of Dolly the sheep
Dolly, born in 1996, was the first mammal cloned from an adult cell. Her creation expanded scientific possibilities and prompted debate about the ethical limits of cloning. It raised a question that applies to AI today: What should society permit once a new capability becomes possible?
AI can write, teach, screen candidates and support decisions in ways that resemble human work. Institutions may begin to treat the teacher, therapist or employee as an avoidable expense, overlooking the accountability, empathy and judgment that person brings.
We must decide which responsibilities should remain with people and who answers when something goes wrong. A new technical capability alone cannot establish that its use is wise or safe.
Systems affecting rights or safety should undergo an impact assessment before deployment, including tests for bias, accuracy and privacy risks. Their purpose and limitations should be clear. Decisions should be documented, and human reviewers must have the authority to intervene. In sensitive cases, people should have access to independent review and a meaningful appeal.
Governments must establish enforceable rules. Educational institutions should teach critical digital literacy, and companies must be prepared to delay a launch when safety demands it. Users, too, should verify information and avoid sharing sensitive data unnecessarily.
The boundary should be clear: If we cannot control a system, adequately explain its actions or accept responsibility for the harm it may cause, we should withhold deployment until those conditions are met.
The writer is a professor and the founding dean of the Faculty of Computer Science at the College of Management Academic Studies in Israel.