How Publishers Can Protect Content From AI
TECHNOLOGY · LAW
Key Facts
- The basic tool robots.txt is a public file that asks crawlers to stay out, not a lock.
- AI controls Google, ChatGPT maker OpenAI, Apple and Facebook owner Meta each document an AI rule in robots.txt.
- Search Blocking Google’s and Apple’s AI training rules does not remove a site from search.
- Brazil and Norway Oslo and São Paulo courts issued provisional orders against Meta’s new commercial terms.
- The limit robots.txt is not access control, so server-side blocks add protection.
Publishers can tell artificial intelligence (AI) companies not to use their pages through a public text file called robots.txt, which is a request and not a lock. This guide explains the controls the companies document and the licensing routes.
It also covers what the Brazilian and Norwegian court orders against Meta, the owner of Facebook and Instagram, change, as the plaintiffs describe them. Both orders concern Meta’s new commercial terms.
It is general information drawn from the operators’ own documentation and from court filings, not legal advice and not a recommendation of any product. It follows our explainer on the Brazilian ruling and our reading of Meta’s new terms.
What Robots.txt Can and Cannot Do
The Robots Exclusion Protocol, published as RFC 9309 by the Internet Engineering Task Force (the body that writes internet standards), sets out the format. The file must be named robots.txt, sit at the top level of a site and use UTF-8, the standard text encoding.
The standard states plainly that these rules are not a form of access authorisation, so they ask crawlers to behave but cannot force them. Because anyone can read the file, it also reveals the folders a publisher wants kept out of view.
Crawlers match rules by their user-agent name, the label a crawler gives itself. A crawler with no matching group follows the general rules for all robots.
AI Controls Operators Document
Google says its Google-Extended token lets a site manage whether content Google crawls is used to train future Gemini AI models, Google’s family of AI systems. Google also says the token does not affect inclusion in Google Search.
OpenAI, the maker of the ChatGPT chatbot, says disallowing GPTBot, its training crawler, signals that content should not be used for training. It adds that the settings for GPTBot and OAI-SearchBot (search) are independent and that user-started ChatGPT-User visits may not follow robots.txt.
Meta documents five crawlers. One, meta-externalagent, is used for purposes such as training AI models or improving products by indexing content.
Another, meta-webindexer, supports search in Meta AI, Meta’s chatbot, while facebookexternalhit builds link previews and may bypass robots.txt for security checks. Meta also says meta-externalfetcher, which fetches links at a user’s request, may bypass robots.txt.
Apple says a Disallow rule for Applebot-Extended opts a site out of training its foundation models, the large AI systems behind Apple Intelligence, its suite of AI features. Common Crawl, a non-profit web archive, says its CCBot crawler can be blocked the same way.
Example robots.txt: blocks three AI training controls
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
The example blocks three AI training controls from a whole site, and a rule can name a folder instead of the slash. Operators say changes are not instant, since Meta says crawlers may cache the file for up to 24 hours and OpenAI says search systems need about 24 hours to adjust.
These AI rules are documented as separate from search. Google says Google-Extended does not affect inclusion in Search, and Apple says pages blocking Applebot-Extended can still appear in results.
Controls Beyond Robots.txt
Robots.txt cannot stop a crawler that ignores it, so publishers who need a hard block can refuse requests at the server. They can do it by user agent or IP address, a machine’s network address, which is a technical measure and not a request.
Google and Common Crawl document how to verify their crawlers by reverse DNS lookup, a check of which network name an address belongs to. Common Crawl also warns that other crawlers sometimes falsely use the CCBot name.
Apple says a nosnippet meta tag, a line of page code that bars quoting, keeps content out of its broad answers in Siri, its voice assistant, and in Search. This is a different control from the training opt-out.
Google says it intends to support a new AI training-preference signal in early 2027. That is a plan and not yet a feature.
In US court papers filed on Friday 4 September 2026, OpenAI argued that copying by Browse, ChatGPT’s web-reading tool, is impliedly licensed unless a site opts out. That is one party’s argument in a pending case.
We found no ruling on it in the filings we reviewed. The question matters because it ties permission to what a site’s robots.txt file says.
Licensing Routes
The US Copyright Office, the federal agency that registers copyrights, said in May 2025 that AI training licences, both individual and collective, are emerging in some sectors but unevenly. It called government intervention premature and said extended collective licensing, where one group licenses on behalf of many owners, could be considered where gaps remain.
On Tuesday 29 September 2026 the US Court of Appeals for the Third Circuit decided a case between Thomson Reuters and ROSS Intelligence, an AI legal-research start-up. Thomson Reuters is the information group that owns the Westlaw legal database.
The court found that the market for licensing headnotes, short summaries of legal rulings, as AI training data is rapidly developing, and it counted harm to that market against ROSS. That makes the licensing market part of the court’s fair use analysis, the legal test of whether copying without permission is allowed.
The case concerned legal headnotes and not news. A California federal court rejected a similar licensing-market argument in Kadrey v. Meta, a suit by authors.
What the Brazil and Norway Orders Change
On Tuesday 3 November 2026 the 44th Civil Court of São Paulo granted a preliminary injunction, a temporary order issued before a full trial, according to three Brazilian media associations. They are ANJ for newspapers, Abert for broadcasters and Aner for magazines.
They say it bars Meta from collecting or using members’ content without authorisation, including to train AI. They add that it requires Meta to keep their accounts active without penalty.
The associations put the fine at R$1 million a day (about US$182,000), capped at R$100 million (about US$18.2 million). Those conversions use 5.4935 reais per US dollar, the central bank rate of 3 November.
On Wednesday 28 October 2026 the Oslo District Court temporarily barred Meta from bringing in the same terms, according to a statement by the Norwegian media trade body. The statement says ten Norwegian media groups asked the court that week.
Both orders are provisional. The Brazilian associations say theirs protects their members, and the Norwegian trade body says Meta may not start the terms until a court rules on their lawfulness.
Nothing in the plaintiffs’ descriptions of the orders changes how robots.txt works. Meta’s own crawler page says robots.txt can block meta-externalagent.
What It Means for You
If you run a site or a publication, start by reading your robots.txt file and deciding which AI crawlers to allow. The operators document separate controls for training and for search.
The choice is yours. Blocking training rules does not remove a page from Google Search or Apple search, according to those companies.
If you have a Facebook or Instagram business account, check any notice about Meta’s new commercial terms and consider legal advice before accepting. Readers will not see these controls, because they work between a site and the crawlers that visit it.
What Is Not Known
It is not known whether every AI crawler honours robots.txt, since operators document exceptions such as user-started visits. It is also not known whether US courts will treat a site’s silence, or its robots.txt rules, as consent to AI use.
Meta scheduled its new commercial terms to start on Friday 20 November 2026, and whether they will start then has not been confirmed. The plaintiffs say the terms would let Meta use publishers’ content for AI.
We have not seen either court decision, so how far each order reaches is unconfirmed. The plaintiffs’ statements are our only source for what each order says.
More: Meta, AI and news: our full dossier, with the explainer on Brazil’s court ruling.
Frequently Asked Questions
Does Blocking AI Crawlers Remove My Site From Search?
Not for the controls that Google and Apple document: Google says Google-Extended does not impact inclusion in Search, and Apple says such pages can still appear in search results. OpenAI treats its search crawler and its training crawler as separate settings.
Is Robots.txt Legally Binding?
The standard itself says its rules are not a form of access authorisation. In December 2025 a New York federal judge ruled in a suit against OpenAI by Ziff Davis, a US digital publisher. The judge found robots.txt is not a technological measure that effectively controls access.
That means ignoring it is not circumvention under the US anti-circumvention law in the Digital Millennium Copyright Act. We found no US ruling that settles whether ignoring it breaks copyright or contract law.
Do the Brazil and Norway Rulings Cover Every Publisher?
Not necessarily: the Brazilian associations say their order protects their members, while the Norwegian statement describes a ban on bringing in the terms. We have not seen either court file.
Sources: Internet Engineering Task Force, RFC 9309, Robots Exclusion Protocol, 2022; Google Search Central, documentation on Google’s common crawlers (Google-Extended); OpenAI, crawler documentation (GPTBot, OAI-SearchBot, ChatGPT-User); Meta for Developers, web crawlers documentation; Apple Support, Applebot documentation; Common Crawl, CCBot documentation; US Copyright Office, Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version), 9 May 2025; US Court of Appeals for the Third Circuit, Thomson Reuters Enterprise Centre v. ROSS Intelligence, No. 25-2153, 29 September 2026; US District Court, Southern District of New York, In re OpenAI, Inc. Copyright Infringement Litigation, No. 25-md-3143, OpenAI defendants’ summary judgment filing in the news cases (public redacted version), 4 September 2026; US District Court, Northern District of California, Kadrey v. Meta Platforms, No. 3:23-cv-03417, order of 25 June 2025; US District Court, Southern District of New York, Ziff Davis v. OpenAI, opinion and order of 15 December 2025; Central Bank of Brazil, PTAX reference exchange rate of 3 November 2026, 5.4935 reais per US dollar; joint statement of ANJ, Abert and Aner on the São Paulo injunction, 3 November 2026; statement of the Norwegian media trade body on the Oslo District Court order, 28 October 2026.
This article was produced by The Rio Times’ automated newsroom system. How we use AI · Report an error · Editorial responsibility: Matthias Camenzind, Editor-in-Chief
Read More from The Rio Times