18Ar

Crawler policy

What ArgonMediaAnalyzer does

If you've seen ArgonMediaAnalyzer/1.0 in your logs, this page tells you what we're doing and how to opt out if you want to.

What Argon is

Argon is an editorial meta-analysis tool. It reads AI-related news across a broad range of sources and characterises how each outlet covers the same story — substance vs hype, stance (critical / neutral / favorable), and which sources are missing from a given thread.

What Argon explicitly is NOT

  • Not training data. No text we fetch is used to train, fine-tune, or otherwise improve any AI model — ours or anyone else's. The LLM calls we make are inference-only.
  • Not a content aggregator that republishes. We cache article bodies locally (see "What we store" below) but we never serve them back to readers. Reader traffic always links out to your URL.
  • Not an answer engine. Argon doesn't pretend to substitute for the original reporting — it surfaces which outlets cover what, then sends readers to them.

How to identify the crawler

We fetch with a clearly identifying User-Agent string. Look for this in your logs:
ArgonMediaAnalyzer/1.0 (+https://argon.news/about-crawler;
  media-stance-analysis; link-out-first; not for training)

How to block us — hard opt-out

If you'd rather not be included at all, add this to your robots.txt:
User-agent: ArgonMediaAnalyzer
Disallow: /
We respect this on every fetch — both robots.txt itself and the per-page check before reading article bodies. Pages we've already read stay cached but stop being re-fetched on the next cycle.

Generic anti-AI directives

We honour User-agent: GPTBot and similar generic anti-AI-training directives as advisory signals rather than hard blocks — because those directives are about training, and we don't train. If you want a hard block, use the explicit ArgonMediaAnalyzer stanza above; that's the unambiguous one.

What we store about your pages

  • The URL, headline, and publication time (from your RSS feed if you have one, otherwise extracted from the page).
  • A short LLM-generated stance + substance score (an opinion about the article — not the article text).
  • Cluster membership: which other articles across other sources cover the same underlying story.
  • The article body itself, cached locally. We do this so we don't have to re-fetch from your servers every time we want to re-score or summarise — being a polite client matters more to us than minimising what we keep.

What we do — and don't do — with the cached body

The cached body powers three things:

  • Stance + substance scoring. One LLM inference call per article, producing the short stance label and 5-axis substance score you see on the card.
  • Translation for non-English articles. Cached as the English version so we serve readers a short card summary that links back to your original.
  • Cross-source synthesis. When 2+ outlets cover the same story, we use the bodies to write a neutral cross-source paragraph that surfaces all the angles.

What we explicitly don't do:

  • Republish. Reader traffic always links to your URL. We never serve your article text as content.
  • Include in AI training data. None of the cached text is used to train, fine-tune, or otherwise improve any AI model — ours or anyone else's. The LLM calls we make are inference-only.
  • Share with third parties. The cache lives on our own infrastructure; nothing about your articles is sold or licensed onward.