AI-generated artifacts are trusted more than source material
Summaries, synthesized docs, or generated analyses start becoming the operational truth instead of pointers back to real sources.
- Where you see this
documentationincident reviewmeeting summariesdesign review
- Not necessarily a problem when
- the source is trivial and the summary is purely convenience for low-risk use
- Often mistaken for
- the summary is probably accurate enough
- Time horizon
- near-term
- Best placed to act
team leadAI policy ownerdocumentation owner
The signal
What you would actually notice
Compression improves convenience while increasing the chance of subtle but important distortion.
Field observation
Teams quote the summary rather than checking the doc, the code, the log, or the transcript it came from.
Also observed
- The summary said that was already decided.
- I did not read the source doc; the AI recap was enough.
Primary reading
What it usually indicates
Most likely underlying patterns when this signal shows up. Not a diagnosis, a starting hypothesis.
Usually indicates
Most likely underlying patterns when this signal shows up.
- source fatigue
- over-trust in synthesis
- weak source discipline
Stakes
Why it matters
Compression improves convenience while increasing the chance of subtle but important distortion.
Inspection
What to check next
Deliberate steps to confirm or disconfirm the primary reading above. Not a checklist. An order of inspection.
- citation quality
- source accessibility
- decision records tied to evidence
Diagnostic questions
Questions to ask the team, or yourself, before concluding anything.
- What is the source of this claim?
- Was the underlying material checked by a human?
- Would the conclusion change if we read the source directly?
Progression
Under the signal
Where this pattern tends to come from, what's holding it up, and where it goes if nothing changes.
Leading indicators
What tends to show up first.
- people cannot point to source evidence behind important claims
- generated summaries get reused without verification
- nuance disappears from decisions
Common root causes
What is usually sitting under the signal.
- time pressure
- over-trust in AI synthesis
- low source discipline
Likely consequences
What happens if nothing changes.
- decision distortion
- false confidence
- knowledge drift
Look-alikes
Not what it looks like
Patterns that can be mistaken for this signal, and 'fix' attempts that make it worse.
- the summary is probably accurate enough
Anti-patterns when responding
Responses that feel sensible and usually make the underlying pattern worse.
- replacing primary-source review with AI digest review
- using summaries in place of logs, code, or data inspection
Context
Context and ownership
Where this signal surfaces, who sees it first, who can actually act, and how much runway there usually is before escalation.
Where it shows up
- documentation
- incident review
- meeting summaries
- design review
Who sees it first
Before it escalates.
- careful reviewers
- staff engineers
- incident leads
Who can move on it
Not always the same as who notices it.
- team lead
- AI policy owner
- documentation owner
near-term
How much runway there usually is before the signal hardens into the underlying pattern.
AI impact
AI effects on this signal
How AI-assisted and AI-driven workflows tend to amplify or hide this signal.
AI amplifies
Ways AI tooling tends to make this signal louder or more common.
- This red flag is directly AI-amplified.
AI masks
Ways AI tooling tends to hide this signal, so it keeps growing under the surface.
- High-quality prose can make weak synthesis feel authoritative.
Relationships
Connected signals
Related failure modes, decisions behind the signal, response playbooks, and neighboring red flags.