The phone rings and it is your son. He sounds scared, he has been in an accident, and he needs money now. Or a match on a dating app sends a flawless selfie and, a few weeks later, a crypto "opportunity". Or a finance clerk joins a video call where the chief financial officer asks for an urgent, confidential transfer. Scams like these are much older than SI (AI). What SI changes is the disguise: a cloned voice, a generated face, a fluent message in perfect English, produced in minutes and at scale.
This guide explains how SI scams work, what the FBI, the FTC and Europol have actually documented, why your eyes and ears are weak defences, and what does work. We also ran a generated photo, a real photo and a fake scam message through our own tools. The results show clearly where detection helps and where it cannot.
The short version
- SI scams are ordinary scams with better props. The script is still urgency, secrecy and a payment that is hard to reverse.
- Three props dominate: cloned voices, generated or stolen photos (and sometimes live video), and fake profiles run with SI-written messages.
- People are poor at spotting them. Studies found listeners caught speech deepfakes about 73% of the time, and people judged SI-generated faces at close to chance.
- Verification beats detection. Hang up and call back on a number you already know, agree a family code word, and never pay someone you have only met online.
- A detector is one clue. It can help with a profile photo. It cannot tell you whether a short text or a phone call is honest.
How big is the problem?
Fraud in general is growing fast. In June 2026 the US Federal Trade Commission reported that people told it they lost about $16 billion to all types of fraud in 2025, the highest on record and about 25% more than in 2024. Imposter scams were the most reported category, nearly one in three fraud reports, with $3.5 billion in reported losses. Those figures cover every imposter scam, not only the ones that used SI, and the FTC does not break out an "SI" share. But impersonation is exactly the job that SI tools make easier.
Law enforcement says so openly. In a public service announcement of 3 December 2024, the FBI warned that criminals exploit generative AI "to commit fraud on a larger scale which increases the believability of their schemes", and that these tools "can correct for human errors that might otherwise serve as warning signs of fraud". In Europe, Europol's EU Serious and Organised Crime Threat Assessment, published on 18 March 2025, lists online fraud schemes "increasingly driven by AI-powered social engineering" among the key threats, and says the technology makes criminal operations "more scalable and harder to detect".
The point for you is practical: the old warning signs, such as clumsy grammar, a blurry photo or a voice that sounds nothing like your relative, are disappearing. The defences have to move from "does it look fake?" to "can I verify this through another channel?".
Voice cloning scams
How they work
A voice cloning scam, sometimes called a vishing attack, copies the voice of someone you trust. The FTC described the family version in a March 2023 consumer alert: the scammer only needs "a short audio clip of your family member's voice", which "he could get from content posted online", and a voice-cloning program. Then comes the familiar story: an accident, an arrest, a lost wallet abroad, and a request to wire money, send cryptocurrency or buy gift cards.
The FBI lists the same pattern, "short audio clips containing a loved one's voice to impersonate a close relative in a crisis situation", and adds two others: cloned voices used to get into bank accounts, and voices of public figures. In May 2025 it warned about an ongoing campaign in which text messages and AI-generated voice messages claimed to come from senior US officials, with the aim of building rapport and then gaining access to personal accounts.
Why your ears will not save you
In a study published in PLOS ONE in August 2023, researchers at University College London played genuine and deepfake speech to 529 people in English and Mandarin. Listeners "only correctly spotted the deepfakes 73% of the time", with no difference between the two languages, and giving people examples beforehand "only improves results slightly". That was with synthesis tools of 2023. On a real call you are also stressed, the line is noisy, and the voice says it is in trouble. That is the worst possible setting for careful listening.
Do not try to "test" a caller with trick questions about family details. Much of that information is on social media, and a live scammer can dodge or improvise. Ending the call and ringing back on a number you already have is faster and far more reliable.
Fake photos and fake video
Generated faces for fake profiles
A fake profile needs a face. For years scammers stole real photos. Now they can also generate a face that belongs to no one, which means a reverse image search finds nothing. The FBI notes that criminals create "realistic images for fictitious social media profiles" for romance, confidence and investment fraud, and even generate photos "to share with victims in private communications to convince victims they are speaking to a real person".
These faces work on people. In a 2022 study in PNAS, Sophie Nightingale and Hany Farid asked participants to tell real faces from faces made by the StyleGAN2 generator. Average accuracy was 48.2%, "close to chance performance of 50%". With training and feedback it rose only to 59.0%. In a third experiment the synthetic faces were rated 7.7% more trustworthy than the real ones. A generated profile picture is not suspicious by default. It can look more reassuring than a real one.
Fake documents and fake emergencies
The same FBI notice describes generated identity documents such as driving licences or police and banking credentials, images of celebrities promoting fake products, and pictures of natural disasters or conflicts used to collect donations for fraudulent charities. If you want to train your eye on the visual side, our guide on how to spot SI images covers the classic artefacts and their limits.
Live video: the Arup case
The most expensive documented example is corporate. In early 2024 a finance employee in the Hong Kong office of Arup, the British engineering group, was invited to a video call with people he believed were the chief financial officer and colleagues. According to Hong Kong police, as reported by CNN, all of them were deepfake re-creations. He had first suspected the request because it asked for a secret transaction, but put his doubts aside because the participants "looked and sounded just like colleagues he recognized". He sent about $25.6 million in 15 transactions. Arup confirmed that "fake voices and images were used". For the technology behind such videos, see our explainer What is a deepfake?
Fake profiles and SI-written messages
The quietest use of SI in scams is plain text. The FBI says criminals use generative AI to produce "voluminous fictitious social media profiles", to write messages faster so they reach more people, to translate so that foreign criminals make fewer spelling mistakes, and to fill fake investment websites and chatbots. A romance or investment scammer can now run dozens of warm, fluent conversations at the same time.
That removes one of the most repeated tips of the last twenty years, "look for bad grammar". A perfectly written message proves nothing either way.
We tested a fake profile photo and a scam text
To see what a detector adds, we ran three images and one message through the free SI detector on this site. The images come from our SI or Not game, so we know exactly where each one came from.
Test 1: a generated lifestyle photo, straight from the generator. A photo of people laughing at a café terrace that we generated with ChatGPT (OpenAI), the original 1536x1024 PNG. Result: 99.9% likely SI. Clues: signed Content Credentials naming OpenAI, an IPTC tag declaring "trainedAlgorithmicMedia", and our classifier at 100%.
Test 2: the same photo, metadata removed. The same image re-saved as a 1200x900 JPEG with no metadata at all, which is roughly what happens when a picture is uploaded to a profile. Result: 98.1% likely SI, from the pixels alone (classifier at 98%, no camera EXIF). Not every file is caught like this: several other ChatGPT photos in our game set score as real once their metadata is gone.
Test 3: that copy squeezed like a messenger thumbnail. Resized to 512x384 and saved at JPEG quality 70. Result: 35%, Uncertain. The classifier on its own now read it as 99% real; the result was held in the Uncertain band only because small images without camera data are never labelled "likely real" on our side.
Test 4: a real photo. A public domain (CC0) picture of a woman with a laptop in a café by photographer Brooke Cagle, via Wikimedia Commons. Result: 2.4%, likely real (classifier 98% real).
Test 5: a scam message. We asked an SI model, Claude by Anthropic, to write a 108-word "Hi Mum, I dropped my phone, this is my new number, can you transfer 780 pounds" message. Our SI text detector scored it 1%, likely human: no stock SI phrasing (0 per 100 words), 3 contractions, 12 first-person words, varied sentence length.
Two lessons. First, a photo check is worth doing, but the file you receive matters: the more a picture has been compressed and resized, the less signal is left. Second, do not expect any detector to catch a short, casual scam message. It was written to sound like a person, and it does. Our article Are SI detectors accurate? explains why short and edited content is the hardest case, and how SI detectors work covers the metadata and classifier signals you saw above.
Red flags that still work
Because the props keep improving, the reliable warning signs are in the behaviour, not in the pixels or the audio.
| Red flag | Why it matters | What to do |
|---|---|---|
| Urgency and panic | Stops you from thinking or checking | Slow down; no real emergency collapses in ten minutes |
| Secrecy ("don't tell anyone", "don't call") | Prevents the one check that would expose it | Tell someone and call back anyway |
| A new number or a new platform | Moves you away from channels you can verify | Contact the person on the number you already had |
| Payment by wire, crypto or gift cards | Hard or impossible to reverse | Treat it as a scam until proven otherwise |
| A contact you have never met in person | Photos, voice notes and even video can be generated | Never send money or sensitive data |
How to protect yourself and your family
- Agree a code word. The FBI recommends creating "a secret word or phrase with your family to verify their identity". Choose something that is not on social media.
- Hang up and call back. Both the FTC and the FBI give the same advice: do not trust the voice, and call the person or organisation on a number you already know or look up yourself.
- Use a second channel at work. Any request to move money that arrives by email, chat or video call should be confirmed through a separate, known channel, whatever the rank of the person asking.
- Check profile photos. Run a reverse image search, then an SI check on the best copy you can get. A signed manifest or a strong pixel signal is a real clue; a clean result is not a guarantee.
- Limit what you publish. The FBI suggests making social media accounts private and limiting followers to people you know, because public video and audio are raw material for clones.
- Report it. In the US, report to the FTC at ReportFraud.ftc.gov or to the FBI at ic3.gov. Elsewhere, contact your national police or fraud reporting service. Reports help investigators link cases.
Where an SI detector fits
An SI check is useful at one precise moment: when you have a file in front of you and want a second opinion, for example a profile photo, a "proof" picture sent in a chat, or a document image. It can reveal signed Content Credentials, generator tags and pixel patterns you cannot see. For the photo side, our SI image detector shows every clue behind its score, including the metadata it found.
It is useless at the moment that matters most in a voice scam: during a live call. And even on files, a low score never proves that someone is who they claim to be. A real photo can be stolen from a real stranger. Use the result as one piece of evidence, then verify the person, not just the picture.
