—
Recent advances in generative AI models like GPT-4, DALL-E 3, and Google’s Imagen have demonstrated the rapid pace of progress in systems that can create human-like text, images, video, and audio from short text prompts. While much of the public focus has been on benign applications like using these systems to generate website copy or artwork, experts warn the same technologies could be misused to spread false and misleading information at scale.
The fact that AI can now write novels, generate code, hold conversations, and create original artwork shows that these systems have a very sophisticated understanding of language, concepts, and our world. Unfortunately, this also means they can plausibly fabricate content that seems authentic and aligned with an author’s style.
Unlike previous generations of AI systems that focused solely on pattern recognition, modern generative models can not only identify complex structures in data but also actively imagine and create original examples without human involvement. With systems like GPT-4 now able to generate output that rivals professional human writers in terms of quality, experts say synthetic text will become increasingly difficult to detect.
Where deepfakes used to be obviously fake, now they are becoming good enough to fool most people if created carefully. Text is, in some ways, more dangerous than images or video since there are no visual artifacts for detection systems to analyze, making the role of an AI Detector even more crucial in identifying AI-generated misinformation.
Emerging Techniques for AI-Generated Misinformation
While most discussion around synthetic media focuses on fake celebrity porn videos or Photoshopped images, experts warn that text-based misinformation poses a serious and complex threat. Some emerging techniques identified by researchers include:
AI-Assisted Article Writing. Using generative language models like GPT-4 or Google’s LaMDA, anyone can now generate long-form articles on arbitrary topics with minimal human input. This lowers the barriers to fabricating professional-looking news stories or journal articles that spread false claims.
Automated Misattribution. Language models can mimic the writing style of public figures based on analyzing samples of their real output. This raises the risk of fabricating tweets, speeches, or other statements in someone’s name.
False Expert Identities. Since generative AI does not have a concept of personal identity, it is trivial to invent fictitious credentials, such as academic bios or work histories, for nonexistent doctors, professors, or industry experts to lend credibility to misinformation.
Synthetic Evidence Fabrication. From fake research abstracts to generated legal documents or financial reports, the same systems that can fabricate free-form text can also produce structured outputs like tables, charts, and citations that appear authentic to support false claims.
Targeted Content Generation. Generative AI allows the creation of fully customized text content tailored to specific individuals, informed by their browsing history, interests, and psychological vulnerabilities identified through analytics. This enables more persuasive and compelling misinformation for each target.
When you combine language models that can mimic anyone’s writing style with the ability to fabricate documents or social media histories, essentially any form of text-based evidence can now be faked at scale. And that represents an unprecedented risk for manipulating public opinion and discourse.
Emerging Startups Focused on AI-Generated Content
While tech giants like Google and Microsoft have received the most attention for their investments in generative AI research, a growing number of startups have also formed to productize synthetic text capabilities:
Anthropic
This San Francisco startup, founded by a former OpenAI researcher,s is focused on developing “constitutional AI” with strong safety constraints. Still, their proprietary language mode,l Claude, is capable of generating high-quality text.
You.com
Founded by former Salesforce executives, You.com offers a search engine powered by a large language model capable of answering queries in natural language. Its generative AI can summarize web pages and write new text on arbitrary topics, which could be misused to fabricate content.
Jasper
Jasper offers a conversational AI assistant for content creators who can write original text and code and generate creative ideas for articles, product names, domain names, and taglines. However, bad actors could leverage these same tools to generate misinformation automatically.
Rytr
Rytr provides an AI writing assistant to generate blog posts, social media captions, emails, and more for marketers. The platform is capable of fully automated content creation once configured with a basic content strategy. Similar functionality could enable mass production of AI-generated propaganda if misused.
Copy.ai
Copy.ai offers AI copywriting models for industries like e-commerce, crypto, SaaS, and more to generate product descriptions and other marketing copy. These domain-specific language models lower barriers for startups or niche sites to create convincing-looking content spreading false claims.
Emerging Detection Methods for AI-Generated Text
In response to the growing threat of use cases like automated disinformation campaigns, natural language processing experts have shifted focus toward developing new techniques for detecting if the text was written by an AI system instead of a human:
- Stylistic Analysis. Researchers are compiling linguistic signatures of patterns exposed in AI-generated text that expose it as non-human, such as overusing filler words, repetitive phrasing, or unnatural sentence structure. Automated classifiers can flag text containing these generative markers.
- Semantic Coherence. Humans intuitively relate concepts when writing in ways subtle enough to evade AI training data. By analyzing clusters of entities and meaning in generated samples, incoherent or logically inconsistent connections can detect synthetic text.
- Contextual Analysis. When language models fabricate text without broader context, warning signs, such as contradicting statements or entities appearing without proper introduction, emerge. Models that ingest samples and assess against external data can expose such anomalies.
- Native Detectors. Some AI groups, including Anthropic, intentionally train “self-informed” models that can recognize their own weaknesses and alert users if prompted to generate harmful, biased, or misleading text. However, adoption among startups remains limited so far.
- Immunization Training. Researchers have proposed retraining language models to “immunize” them against certain types of rules-based triggers for generating misinformation. For example, removing ideological biases or blocking fabricated experiential claims.
- Cryptographic Authentication. Emerging techniques such as AI watermarking will tag genuinely human-generated text with encrypted authorship markers, enabling automated systems or human reviewers to verify the authenticity of sources before spreading unverified content.
However, experts caution that the rapid advancement of generative AI means detection systems are already struggling to keep pace. New techniques, such as micro-targeted text generation informed by psychographic profiles, allow for crafting more persuasive outputs uniquely tailored to each reader.
The Role of Social Platforms in Combatting AI Misinformation
As the shift toward AI-generated misinformation gathers steam in areas like political influence campaigns, public health, and financial fraud, social platforms face growing pressure to defend the integrity of content spread through their networks.
Major platforms like Meta and Twitter have already struggled to combat much simpler forms of misinformation with human fact-checkers and content moderators, but the challenge has now grown exponentially.
In response to recent advances in generative AI, companies like Meta, Reddit, and Medium have clarified policies prohibiting sharing AI-generated content on their platforms without proper disclosures about the automated origin.
YouTube and TikTok have also invested heavily in media forensics teams to analyze uploads for manipulated video, audio, and now text. Twitter recently acquired a startup called Alethea AI, which specializes in synthetic media detection. Meta has developed one of the largest datasets for deepfake research, comprising over 200,000 samples.
However, most platforms have stopped short of outright bans on synthetic media, given the challenges of defining enforceable policies that don’t also restrict many benign applications of AI generation that advance creativity, free expression, and accessibility.
The Outlook for Policymakers and Regulators
As awareness grows around emerging misuse cases for generative AI, policymakers have just begun considering responses that balance countermeasures against harmful applications while avoiding restrictive bans that would have major economic consequences.
In their 2021Â report on AI and disinformation, the National Security Commission on AI warned that “the disinformation crisis is going to get exponentially worse with deepfakes and machine-generated text.” The report called for establishing definitions and a framework for monitoring and responding to AI-powered influence campaigns from foreign adversaries.
Similarly, in Europe, a recent report from the EU High-Level Expert Group on AI warned that advances in generative models “enable the production of hyper-realistic media that will be almost impossible to identify as fake” while allowing malicious actors to “automate the production of disinformation at unprecedented scale.” They advised investments into forensic detection capabilities and content authentication techniques as a top priority.
However, given the issue’s recency and complexity, tangible steps toward regulation so far remain limited. Without clear legal frameworks in place, monitoring and enforcement are left largely to the discretion of social platforms themselves.
What Comes Next?
The public is drawn to recent headlines about Turing Test achievements by chatbots. Still, experts believe deep learning supervised and diffusion models are just starting to expose their deep industrial disruption potential. AI safety experts monitor the problem of unchecked AI harm because they predict such advanced systems will move faster than protective measures can be implemented.
The key question now is whether there is enough momentum to build momentum behind proactive initiatives like transparency requirements, audits for the generative start, training workers in ethics, and other constraints before misuse cases tip into mainstream adoption and become impossible to contain. Once the technological genie is out of the bottle, history shows that our ability to put it back in or direct its course tends to be tragically limited.
—
