AI misuse is the use of artificial intelligence systems to cause harm, deceive people or break the law. It differs from accidents and design flaws, which are failures of the system itself, because the harm comes from a person who deliberately applies the technology for bad ends. Misuse is a leading concern in AI safety, AI ethics and AI security, and it shapes debates on AI governance and AI regulation.
Why AI changes the picture
AI can lower the skill, cost and time needed for some harmful activities, allow them to scale to thousands of targets, and make them more convincing. A scammer can write fluent messages in many languages, produce a cloned voice from a short recording or generate fake images and video. Most capabilities are dual-use, meaning that the same tool serves helpful and harmful purposes, which makes simple bans difficult.
Main categories
A 2018 multi-institution report grouped malicious uses into digital, physical and political security threats [1]. Common categories today include:
- Fraud and impersonation. Voice cloning for emergency scams, fake video calls, fake reviews and synthetic identities. A reported 2024 case involved a Hong Kong employee who transferred a large sum after a video call with deepfaked colleagues [2]; see AI phishing.
- Disinformation and manipulation. Mass-produced false stories, fake accounts and persuasive messages aimed at elections or public opinion.
- Non-consensual and abusive imagery. Synthetic sexual images of real people, including minors, are a serious and growing harm and are illegal in many places.
- Cyberattacks. Help with writing malicious code, finding vulnerabilities or crafting convincing phishing; see AI cybersecurity.
- Harassment and defamation. Fake media used to target individuals.
- Surveillance and repression. Facial recognition and behavior analysis used to track dissidents or minorities.
- Academic and professional dishonesty. Passing off generated work as one’s own.
- Weapons and dangerous knowledge. Concern that models could assist with biological, chemical or other dangerous capabilities, a focus of frontier model testing.
- Intellectual property abuse. Copying styles or content in ways that breach rights.
Who misuses AI
Misuse ranges from individual hobbyists and scammers to organized crime and state actors. Capabilities that were once limited to well-funded groups can spread as tools become cheap and easy to use, while sophisticated actors may combine AI with traditional methods.
Safeguards
### Technical
- Training and tuning. Teach models to refuse clearly harmful requests; see AI alignment.
- Filters and monitoring. Detect abusive prompts and outputs and flag suspicious accounts.
- Provenance and watermarking. Mark synthetic media so it can be identified; see AI watermarking and content authentication.
- Detection tools. Classifiers for deepfakes and synthetic text, which remain imperfect.
- Access controls. Identity checks, rate limits and staged release for powerful models.
- Red teaming. Test for misuse before launch; see red teaming and jailbreaking AI.
### Organizational
Usage policies, trust and safety teams, abuse reporting, cooperation with law enforcement and information sharing among companies.
### Legal and policy
Laws on fraud, harassment, election interference and non-consensual imagery apply to AI-enabled acts, and new rules require disclosure of synthetic media or ban specific uses. The EU AI Act bans some manipulative practices and requires labeling of synthetic content. Law enforcement agencies and consumer regulators have issued warnings and run challenges, such as one on detecting voice cloning [3].
### Social
Media literacy, public warnings, family code words to confirm emergencies and verification habits such as calling back on a known number.
Debates
- Open versus closed models. Open release supports research and competition but makes it harder to prevent misuse.
- Dual-use trade-offs. Strict filters may block legitimate research, journalism or education.
- Responsibility. Whether developers, platforms or users should bear liability.
- Evidence. The scale of harm is sometimes hard to measure, and early fears have at times been overstated or understated.
- Free expression. Rules on synthetic media must avoid suppressing satire and lawful speech.
What individuals can do
Treat unexpected urgent requests for money with suspicion, verify by a second channel, be careful with sharing voice and images, use strong authentication and report abuse to platforms and authorities. Organizations can add approval steps for payments, train staff on deepfake scams and keep clear incident plans.
A worked example
A small company’s finance officer receives a voice message that sounds like the chief executive asking for an urgent transfer to a new supplier. The voice was cloned from a few seconds of a public interview, and the request arrives late on a Friday when senior staff are traveling. Because the company has a payment policy, the officer does not act on the message. She calls the executive on a known number, learns that no request was made and reports the attempt. The company then adds a rule that any new bank account or change above a set limit needs approval from two people, trains staff to expect cloned voices and sets up a code word for emergencies. The case shows how misuse works, by exploiting trust and urgency, and why simple process controls often defeat sophisticated technology.
Misuse of generative models by stage
Researchers often look at where in the chain a harm can be stopped. At the model stage, developers choose training data, add safety training and test for dangerous capabilities before release. At the access stage, they decide whether to offer a model by a controlled interface, which allows monitoring and removal of abusive accounts, or to release the weights, which allows broad use and modification but removes central controls. At the distribution stage, platforms and search engines can limit the spread of harmful content, label synthetic media and slow viral sharing. At the target stage, banks, schools, employers and individuals can add checks and education. No single layer is enough, so most experts recommend several together; see AI risk management.
Measuring and reporting incidents
Public incident databases and company threat reports describe how actors tried to use AI tools, which accounts were removed and what patterns appeared. These sources help researchers compare risks, but they capture only detected cases, and definitions of harm differ. Governments and companies increasingly share information on abuse trends, while civil society groups document cases involving elections, harassment and fraud. Clear reporting channels for victims, and quick takedown routes for abusive content, are an important part of response, and they give regulators evidence for deciding when new rules are needed.
FAQs
What is AI misuse? It is the deliberate use of AI for harmful, deceptive or unlawful purposes.
What is the difference between AI misuse and AI failure? Misuse comes from a person’s harmful intent. Failure is a fault in the system itself.
What are examples of AI misuse? Deepfake fraud, voice-cloning scams, disinformation campaigns, abusive synthetic images and AI-assisted cyberattacks.
How can AI misuse be reduced? Through model safeguards, provenance tools, platform enforcement, laws, and public awareness.
Why is misuse hard to prevent? Many AI tools are dual-use and widely available, so the same capability can help or harm.
More in AI Ethics, Safety and Governance
Key facts
- The 2018 report The Malicious Use of Artificial Intelligence, by researchers from several institutions, grouped threats into digital, physical and political security1.
- In 2024, a finance employee in Hong Kong was reported to have transferred about 25 million U.S. dollars after a video call with deepfaked colleagues2.
- The U.S. Federal Trade Commission reported in 2023 on voice cloning used in family emergency scams and began a challenge to detect such cloning3.
Related concepts
Quick checkWhat is AI misuse?Show answer
AI misuse is the use of artificial intelligence systems for harmful, deceptive or unlawful purposes, such as fraud, disinformation, cyberattacks, harassment or surveillance, by people who deliberately apply them against their intended or ethical use.
Sources
- Miles Brundage et al., The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation, https://arxiv.org/abs/1802.07228
- CNN, Finance worker pays out 25 million after video call with deepfake chief financial officer, https://edition.cnn.com/2024/02/04/asia/deepfake-cfo-scam-hong-kong-intl-hnk/index.html
- Federal Trade Commission, FTC Announces Exploratory Challenge to Prevent the Harms of AI-enabled Voice Cloning, https://www.ftc.gov/news-events/news/press-releases/2024/11/ftc-announces-winners-voice-cloning-challenge
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
