Analysis & Methodology

1. The "Other" Category (262 deficiencies, 22.7%)

262 of 1,152 deficiencies were categorized as "Other" — meaning the AI couldn't cleanly map them to a standard QSR deficiency category. These include premarket authorization violations (510(k)/PMA issues), FD&C Act violations, and MDR-adjacent findings. We did not force-fit them into categories to make the numbers look cleaner.

2. Severity Classification Is Subjective

The AI classified 966 of 1,152 deficiencies as "Critical" (83.9%). This skew likely reflects the nature of warning letters themselves — FDA only sends warning letters for serious violations, so the baseline severity is high. A different AI model or a human reviewer might classify the same deficiency differently. We used a low temperature (0.2) for consistency, but acknowledge this is a judgment call.

3. Risk Scores Are Relative, Not Absolute

The tight clustering at the top of the scale reflects that these are warning letters — the worst of the worst by definition. A company receiving a warning letter is already in serious trouble. The risk score is useful for ranking severity within the dataset, not for comparing against companies that haven't received warning letters.

4. QMSR Clause Mapping Consistency

The AI occasionally produced slightly different formatting for the same clause:

• ISO 13485:2016 Clause 8.2.2 vs. ISO 13485:2016 Clause 8.2.2 (Complaint Handling)
• ISO 13485:2016 Clause 8.5.2 / 8.5.3 vs. ISO 13485:2016 Clause 8.5.2 (Corrective Action) and 8.5.3 (Preventive Action)

We aggregated using primary clause numbers (8.2.2, 8.5.2, 7.3, etc.) for the co-occurrence analysis. The top-cited clause list in the report shows them as the AI labeled them, which is why some appear as separate line items. This slightly undercounts clause totals in the ranked list but does not affect the category-level analysis.

5. 820-to-13485 Mapping Validation

The clause mappings (e.g., 820.100 → 8.5.2/8.5.3, 820.30 → 7.3, 820.198 → 8.2.2) follow the FDA's published QMSR crosswalk and are consistent with industry-standard references from MasterControl, PQE Group, and The FDA Group. These are not proprietary interpretations — they're the same mappings every RA/QA professional uses.

6. What This Dataset Does Not Include

• 483 observations — We analyzed warning letters only, not Form 483 inspectional observations. Warning letters represent escalated enforcement actions.
• Consent decrees or injunctions — Not included.
• Warning letters to drug, biologic, or food companies — Filtered to medical devices only.
• International enforcement — FDA only. No EU notified body actions, Health Canada, or TGA.

Check the category math

This matches our reported total. Each deficiency lands in exactly one category under the strict non-overlapping definitions.

Check the risk score math

This matches our reported total.

Check the severity math

This matches our reported total.

Check the co-occurrence claims

56 letters cite both CAPA and Design Controls. 69 cite both Design Controls and Complaint Files. 41 cite both CAPA and Complaint Files. These were computed by counting letters where both categories appear — a straightforward set intersection under the strict definitions (CAPA = 820.100, Complaint Files = 820.198, Design Controls = 820.30).

Check the percentage claims

Spot-check the source data

Pick any company from the FDA warning letter database. Read their letter. Compare what we extracted against what's in the letter. We will happily provide our extracted data for any specific letter upon request.

1. Exact AI Prompt — Verbatim

Each of the 223 warning letters was analyzed individually (not batched). The following prompt was sent to Claude Opus 4.6 for every letter:

System Message

You are an FDA regulatory compliance expert. Always respond with valid JSON only, no markdown code blocks.

User Prompt Template

You are an FDA regulatory compliance expert specializing in medical device quality systems.
Analyze this FDA Warning Letter and extract structured findings.

Company: {company}
Device: {device}
CMS Number: {cms_number}
Date: {date}

Warning Letter Content:
{first 12,000 characters of letter content}

Respond ONLY with valid JSON (no markdown, no code blocks) in this exact structure:
{
    "company": "{company}",
    "device": "{device}",
    "deficiencies": [
        {
            "clause": "the specific CFR/ISO clause cited (e.g., 21 CFR 820.30, ISO 13485:2016 Clause 7.3)",
            "qmsr_mapping": "the equivalent QMSR/ISO 13485 clause number (e.g., 7.3, 8.5, 8.2.1)",
            "title": "short title of the deficiency",
            "severity": "critical|major|minor",
            "description": "brief description of what was found deficient",
            "qmsr_impact": "how this maps to QMSR requirements"
        }
    ],
    "root_causes": ["list of systemic root causes identified"],
    "risk_score": 0-100 (higher = more serious compliance risk),
    "qmsr_readiness_impact": "paragraph explaining implications for QMSR transition"
}

Content window: The first 12,000 characters of each letter's full text were passed to the model. Most warning letters fall well within this window. Letters exceeding 12,000 characters had their content truncated — the AI analyzed only what was provided.

2. AI Parameters

3. Data Integrity: ALCOA+ Audit Trail

Every AI analysis is logged with a full ALCOA+ compliance trail (Attributable, Legible, Contemporaneous, Original, Accurate, Complete, Consistent, Enduring, Available). This is the same data integrity framework required by 21 CFR Part 11 for electronic records in regulated environments.

Per-Letter Metadata

Each analyzed letter stores the following integrity fields:

Centralized Audit Log Entry

In addition to per-letter metadata, every analysis creates a record in the AI audit log table containing:

• prompt (first 2,000 chars) — what was sent to the AI
• response (first 2,000 chars) — what the AI returned
• model — exact model used
• feature — warning_letter_analyzer
• request_type — fda_warning_letter_analysis
• regulatory_standards — ["21 CFR 820", "ISO 13485:2016", "QMSR"]
• latency_ms, success (boolean), error_message (if failed)
• response_hash_full — full SHA-256 hash
• alcoa_plus_compliant — boolean flag

Aggregate Trail

{
    "total_analyses": 224,
    "audit_logged": 224,
    "compliance_status": "compliant",
    "data_integrity_verified": true
}

If any analysis failed to log an audit entry, compliance_status would show "partial" instead of "compliant".

4. Deficiency Category Classification

Deficiencies were classified into categories using deterministic keyword matching — not AI. The classifier concatenates each deficiency's clause, qmsr_mapping, and title fields into a single string, then checks for the first matching keyword from this static map:

5. Co-occurrence Algorithm

Co-occurrence patterns identify which clause pairs appear together in the same warning letter most frequently. The algorithm:

1. For each letter, extract the set of unique QMSR clause mappings from all its deficiencies
2. Generate all sorted pairwise combinations from that set (order-independent)
3. Increment a counter for each pair across all 224 letters
4. Report the top 15 pairs by frequency

Example

Letter A has deficiencies mapped to: [7.3, 8.2.2, 8.5.2]
Pairs generated: (7.3, 8.2.2), (7.3, 8.5.2), (8.2.2, 8.5.2)

Letter B has deficiencies mapped to: [7.3, 8.2.2]
Pairs generated: (7.3, 8.2.2)

Result: (7.3, 8.2.2) = 2 letters, (7.3, 8.5.2) = 1 letter, (8.2.2, 8.5.2) = 1 letter

6. Report Generation Pipeline

The analysis runs in three sequential phases:

Key design decision: Each letter is analyzed individually, not batched. This means the AI sees only one letter per API call and cannot be influenced by content from other letters. It also means each analysis is independently cacheable, retriable, and auditable.

7. Risk Score Distribution

Interpretation: Risk scores are assigned by the AI based on the severity, number, and systemic nature of deficiencies in each letter. The tight clustering at the high end (75.8% scoring 90+) reflects the nature of warning letters — FDA only issues them for serious, sustained violations. A risk score of 62 (the minimum in our dataset) still represents a significant compliance failure; it simply had fewer individual deficiencies or less systemic impact than the median letter.

8. Data Enrichment: openFDA API

After fetching and analyzing warning letters, device classification data was enriched using the openFDA Device Enforcement API:

This enrichment is supplementary — it adds device classification context but does not affect deficiency extraction, clause mapping, severity scoring, or any of the numbers reported in the analysis.