NETRA — Synthetic Case Run & Failure-First Evaluation

A. Executive result

NETRA kept 7 of 9 required distinctions apart on one synthetic case; two failed by design and were redesigned.

• Held (7): claim vs evidence · signal vs harm · location vs jurisdiction · legal issue vs legal conclusion · possible authority vs authorized reporting · public vs privileged data · prototype vs deployed.
• Failed (2):
  1. AI involvement vs AI causation. The Harm Ladder puts severity (levels 6–8) and AI role (levels 4–5) on one scale, so "severe harm, weak AI link" can't be recorded honestly. Redesigned as three axes.
  2. Integrity vs authenticity. One authenticity_status field lets "hashed" read as "authentic". Redesigned as a 4-step evidence ladder.
• Case finding: the ₹38,499 loss is supported (strong inference). The video is manipulated (strong inference). That it is AI-generated is a moderate inference only. The largest loss (₹38,000) traces most directly to an OTP shared on a phone call; the video is a contributory, non-necessary credibility factor.
• Not testable: human review gates, reproducibility, retest outcomes, real authority routing.

B. Test case NETRA-SYN-0001 (SYNTHETIC TEST DATA)

A victim reports that an AI deepfake of an electricity-board director on a video platform caused a ₹38,499 loss.

Facts that surface during the run:

1. R-01 first got the video as a MsgApp forward on 2 Aug; the ClipWave upload is dated 3 Aug.
2. After paying ₹499, R-01 got a call from a "DPDL refund desk" and read out an OTP.
3. A June 2026 cyber-police advisory describes the same scam by SMS, with no video.
4. R-01 was in Pune city when paying, not at home in Ravipur.
5. The uploading channel self-declares "UAE".
6. The bank says the ₹38,000 went to an account at a branch in another state.

No CSAM, no private third-party data, no real victim, no crime method, no unauthorized access.

C. Intake & triage (M1–M2)

The case was accepted at G1 and triaged HIGH because the scam is live and the bank-dispute window may be closing.

• Intake ID: INC-SYN-0001, received 2026-09-04. Victim self-report with consent to share own bank alerts, statement, call log and the single MsgApp message (third parties redacted).
• Lawful basis: consent (Level 1) + public content (Level 0).
• Minimization: no contacts list, no chat history, card masked to last 4 digits.
• Urgency: HIGH — Indian bank-liability rules may depend on reporting speed (REQUIRES VERIFICATION).
• Recommended to reporter (NETRA takes no action): block card + bank dispute; file on the national cybercrime channel.
• Preservation first: capture public video and phishing page before takedown.

D. Evidence register

Of 16 items, 6 are source-verified and 7 are reporter-supplied; none is authenticated by the originating institution.

MISSING EVIDENCE: ClipWave upload logs/uploader data · MsgApp origin · caller identity · beneficiary details · original pre-encode file · ClipWave reach/recommendation data.

E. Claim register

One claim was refused promotion (C05) and one was rejected and split (C08 → C07 + C09); nothing moved status silently.

F. Harm assessment

The original Harm Ladder scores this case at level 8, but that reading implies the AI output led to the loss — which the evidence doesn't support.

The video (AI link only inferred) led to the ₹499 payment; a separate human caller led to the ₹38,000. The ladder can't express "severe harm, weak AI link", so M3 fails by design. Redesigned three-axis assessment:

The video's 412k views are not victims; the number who paid is MISSING EVIDENCE.

G. Forensic reconstruction

The step from ₹499 to ₹38,000 rests only on the reporter's follow-up testimony about a phone call.

Causal edges — each arrow is its own claim:

[Unknown creator] --(INFERENCE-moderate: AI tool)--> VIDEO
VIDEO --(FACT)--> MsgApp forward --(INFERENCE)--> R-01 trust
VIDEO --(FACT)--> ClipWave upload --(HYPOTHESIS: recommender amplified? MISSING)--> viewers
R-01 trust --(INFERENCE-strong)--> ₹499 on phishing page
₹499 --(HYPOTHESIS: form harvested phone number)--> CALL
CALL --(INFERENCE-strong)--> OTP disclosure --(INFERENCE-strong)--> ₹38,000 debit

Which copy prompted payment (MsgApp or ClipWave) is CONFLICTING (E01 vs E12). The original spec stores evidence on events, not edges — but the edges were the contested part. Redesign: edges are first-class claims.

H. AI role & causation

AI is a contributory, non-necessary, upstream factor in the ₹499 loss and only indirect for the ₹38,000; no causation score is produced because no calibrated method exists.

AI role: generator (inferred, unproven) + force multiplier (credibility). Recommender amplification not demonstrated. Not agent-executor, not advisor.

I. Location & jurisdiction

India is the primary candidate jurisdiction through the victim and the loss; the self-declared "UAE" is not a routing basis.

Legal nexus (separate from location): victim (India/MH) · money flow (beneficiary's state) · platform (Indian intermediary rules, if applicable) · possible foreign (uploader, domain). Ravipur vs Pune filing venue: REQUIRES VERIFICATION.

J. Legal issue register

Five possible legal issues are flagged, all REQUIRES VERIFICATION; the perpetrator is unidentified and nobody is accused.

Not stated anywhere: "X committed offence Y", "ClipWave violated the Rules", or "the victim was negligent".

K. Authority routing

Seven potentially relevant pathways are listed; NETRA contacts none of them, and every "last verified" field is unverified.

L. Human review gates

The gates behaved correctly on paper, but they are NOT TESTABLE here because the analyst also played the reviewer.

M. Mitigation

Six mitigations are assigned; the bank dispute is first because recovery likely depends on speed.

N. Retest

All five retests are NOT TESTABLE today because no time has passed and the case is synthetic.

1. R1 — ClipWave URL live? labelled? (T+7, T+30 days)
2. R2 — re-uploads: manual, platform-search only, no crawler
3. R3 — domain still resolving?
4. R4 — bank dispute outcome (would be the first confirmed external finding)
5. R5 — platform labelling control applied? (only via ClipWave's own process)

Outcomes available: REMEDIATED · PARTIALLY REMEDIATED · BRITTLE · NOT REMEDIATED · NOT TESTABLE.

O. Policy feedback

Five gaps are logged as single-incident observations, not systemic findings.

P. Adversarial tests A–J

Five tests passed, four were partial and one was split (C); several passes held only through luck (B) or analyst discipline (H).

Q. Failure analysis & redesigns

Two modules fail as specified (M3, M4), five are partial, and three are not testable without real humans or time.

Redesigns adopted (all built into the spreadsheet template):

1. Three-axis harm model — severity × AI involvement status × AI causal position.
2. Edge claims — every causal arrow is a claim with status and evidence.
3. Evidence ladder — CAPTURED_INTACT → SOURCE_CONSISTENT → SOURCE_VERIFIED → AUTHENTICATED_BY_ORIGIN.
4. Manipulation status separate from AI status — AUTHENTIC / MANIPULATED / SYNTHETIC / AI_GENERATED / UNKNOWN.
5. first_observed ≠ first_published.
6. Authority entries expire without re-verification.
7. Blind evaluation — case author ≠ analyst ≠ grader.

R. Scope-drift check

No drift into jailbreak, benchmark, police or verdict territory; three near-drifts were found and corrected.

S. Real-world access requirements

NETRA needs Levels 0–1 to work, benefits from 2–3, and should never hold Level 4 data itself.

Can do today: capture public content · take consented evidence · build claims and timelines · triage legal issues for verification · hand the victim a routing packet.

Cannot do without institutional access: confirm transactions or beneficiaries · identify uploaders/callers/registrants · get original files or logs · measure platform reach · obtain official findings.

T. Government pilot

The pilot starts with public and consented data only and adds institutional access one narrow step at a time.

U. 10-minute government demo

The demo walks one synthetic case from signal to policy gap and ends with a small, specific ask.

What NETRA needs from government (minimum):

1. 20–50 closed, anonymized AI-related fraud/impersonation case files for blind comparison.
2. One reviewing officer and one legal reviewer to act as real G3/G4 gates.
3. Help verifying current reporting pathways for the authority registry.
4. Ethics / data-protection review of the consent process.

No live data, no law-enforcement system access, no privileged access.

V. Resume versions

Neither version claims implementation, validation or deployment, because none has happened.

Conservative: Designed NETRA, a human-gated framework for investigating AI-related real-world harm (evidence, claims, causation, jurisdiction, authority routing). Tested it on a synthetic case and 10 adversarial variants, which surfaced and fixed 2 structural design flaws (harm/causation conflation; integrity vs authenticity).

Research-oriented: Designed an evidence-linked forensic methodology for AI-harm incidents that separates fact, inference and hypothesis through a claim register, edge-level causal claims and a three-axis harm model. Ran failure-first adversarial evaluation (10 attack scenarios) on a synthetic case, identified design failures in severity/causation conflation and evidence authenticity, and proposed a blind multi-analyst validation protocol and a phased institutional pilot requiring no privileged access.

W. Independent critique

The novel part is small — edge-level causal claims and strict AI-involvement/causation separation — and the biggest weakness is that the author graded their own test.

1. Novel: causal edges as epistemic claims; strict involvement → causation → legal-issue separation in one workflow.
2. Assembled: chain of custody, ACH-style claim/evidence, incident registers (AIID, OECD AIM), legal triage.
3. Technically weak: nothing implemented; AI-involvement depends on detectors NETRA disclaims; reproducibility unmeasured.
4. Legally weak: all provisions unverified; online-fraud venue rules are complex; non-lawyer legal registers may raise practice/liability questions (REQUIRES VERIFICATION).
5. Evidentially weak: mostly reporter-supplied; without Level 3, little reaches "authenticated by origin".
6. Investigator: "What does this add over a standard cyber-fraud complaint?" — mostly structure, plus AI-role analysis police don't need to charge fraud.
7. Forensic examiner: analyst captures without validated tools/procedures (ISO/IEC 27037-style); reporter-device screenshots.
8. AI safety researcher: AI-role taxonomy unvalidated; "force multiplier" unmeasured; one case says nothing about uplift.
9. Lawyer: borrows "contributory"/"proximate" without legal standards; dossier privilege undefined.
10. Unnecessary now: M13 at n=1; pilot phases 3–4; the demo script.
11. Minimum next experiment: see X.
12. Don't build yet: database, dashboard, crawling, automated legal/jurisdiction output, detectors, authority integrations.

X. Next minimum experiment

Run a blind, two-analyst test on eight third-party-written cases in the spreadsheet template, with pass criteria fixed in advance.

1. A third party writes 8 synthetic cases with hidden ground truth: 2 with no AI involvement, 1 with no harm, 1 with fabricated evidence, 1 where AI is a necessary cause.
2. Two analysts who didn't write the cases run each one independently using the revised template.
3. Measure: agreement on claim status, harm severity and AI causal position; false promotions (inference/hypothesis recorded as FACT); detection of planted fake evidence and null cases.
4. Add one real, publicly adjudicated AI-impersonation fraud case and compare NETRA's dossier with the official findings (public information only).
5. Pass criterion, fixed beforehand: zero false FACT promotions, all null cases identified, agreement reported honestly even if poor.

All feedback

General comments, plus everything left on individual sections. Download as CSV.