2026 Audit Protocol: 12 Questions That Expose Unvetted RLHF Subcontracting

Part I: Exposing Fourth-Party Subcontractors & Fake Credential Claims

Generalist vendor questionnaires ask soft yes/no questions (*'Do you vet your annotators?'* -> *'Yes'*). Forensic workforce due diligence demands **verifiable artifacts, distributions, and direct system demonstrations** that an intermediary broker reselling third-party BPO labor cannot fake on a live call.

Start with **Contractual Privity & Payout Lineage (Questions 1–3)**: Ask (Q1) *'Does 100% of the workforce touching our SOW contract and receive 1099/W-2/direct-Deel disbursements directly from your corporate legal entity, or do any third-party staffing agencies sit between you and the individual human?'* Ask (Q2) *'Will you warrant zero fourth-party subcontracting in the MSA with immediate termination-for-cause and 100% fee clawback for breach?'* Ask (Q3) *'What minimum direct hourly floor will contributors on our tier receive, and can we cross-verify that against your live recruiting portal?'*

Next, probe **Primary Credential & Geo-Residency Mechanics (Questions 4–6)**: Ask (Q4) *'Show us live—on a redacted sample profile—how you verify an MD's NPPES NPI, a PhD's institutional `.edu`/ORCID standing, or a senior SWE's upstream Git history rather than accepting uploaded PDFs.'* Ask (Q5) *'How do you detect residential SOCKS5 proxies and KVM/AnyDesk laptop mules when our SOW requires onshore US/EU data residency?'* Ask (Q6) *'What percentage of applicants who start your technical domain exam actually pass into production pods?'* (Credible specialist labs report 4%–12% pass rates; anyone claiming 50%+ runs an unproctored quiz).

• Q1: Direct payout privity vs. third-party staffing agency middlemen.
• Q2: Binding zero-subcontracting warranty with 100% fee clawback.
• Q3: Attested minimum contributor hourly wage floor (`>=68%` pass-through).
• Q4: Live redacted demonstration of primary-source registry verification.
• Q5:TCP-vs-WebSocket RTT and WebGL forensics against remote-desktop laptop farms.
• Q6: Documented technical work-sample pass rate (`<12%` expected for senior tiers).

Part II: Auditing Workspace Anti-LLM Telemetry, Continuity & Cap-Table Neutrality

Move next to **Workspace Instrumentation & Canary Enforcement (Questions 7–9)**: Ask (Q7) *'Demonstrate on screen share what telemetry fires when a contributor pastes external text, uses an auto-typer extension, or leaves the browser tab hidden for twenty minutes.'* Ask (Q8) *'Will every delivered JSONL batch include a sidecar manifest containing salted `contributor_id_hash`, active dwell seconds, backspace/revision ratio, and external paste count?'* Ask (Q9) *'If our internal audit catches a contributor failing two impossible-premise canary prompts, do you automatically purge and replace 100% of that contributor's prior rows across the entire SOW?'*

Close with **Cohort Continuity & Corporate Governance (Questions 10–12)**: Ask (Q10) *'Can our research engineer speak directly on Zoom/Slack with the actual L2 QA lead grading our rows—not a non-technical sales director—before we sign?'* (Shadow subcontractors panic at Q10 because their 'QA lead' works for a different company!). Ask (Q11) *'Will you commit to >=80% monthly named-contributor continuity on our pod and fund replacement onboarding yourself?'* Finally, ask (Q12) *'Does any frontier LLM developer hold >5% equity, convertible debt, or board observer rights in your parent entity?'*

{
  "schedule_title": "WORKFORCE_PROVENANCE_AND_ZERO_SUBCONTRACTING_WARRANTY",
  "direct_privity_required": true,
  "fourth_party_bpo_subcontracting": "STRICTLY_PROHIBITED",
  "mandatory_row_telemetry_fields": [
    "contributor_id_hash",
    "verified_credential_tier",
    "active_dwell_seconds",
    "external_paste_count",
    "revision_keystroke_ratio"
  ],
  "canary_violation_remedy": "RETROACTIVE_ALL_ROW_REPLACEMENT_AT_ZERO_COST",
  "frontier_lab_cap_table_ownership_max_pct": 5
}

Disqualifying Evasive Bidders Before Spending Bakeoff Dollars

Send these twelve questions 48 hours ahead of your vendor intro call. Any provider that refuses to share `contributor_id_hash` telemetry, hides behind 'partner delivery hubs,' or blocks direct Slack/Zoom contact with your pod's technical QA lead should be eliminated prior to paid pilot selection—funneling your bakeoff budget exclusively to transparent independent leaders like Surge AI, Mercor, micro1, Turing, Datacurve, Hillclimb, and Hugo Inc.

When converting the winning bidder's answers from your scoping call into binding legal obligations, attach the vendor's completed 12-Question Diligence Response directly as **Exhibit D (Incorporated Workforce & Security Representations)** of the Master Services Agreement! Sales representatives suddenly become remarkably honest about actual PhD headcount, onshore residency percentages, and clipboard-telemetry capabilities when their General Counsel informs them that Exhibit D carries material breach remedies.

Pair Exhibit D with a **Quarterly Unannounced Canary & Provenance Spot-Check**. Once per quarter, select fifteen delivered `contributor_id_hash` identifiers from recent JSONL shards and request anonymized verification timestamps (confirming KYC match date, primary credential registry type, onboarding work-sample pass score, and zero fourth-party agency intermediation). The entire audit takes a vendor compliance officer twenty minutes to export from an honest database—yet completely deters silent mid-contract BPO offshoring whenever a vendor faces margin pressure.

Post-training teams that institutionalize this 12-question pre-award screen alongside Exhibit D warranties and 3% blind canary seeding eliminate >95% of workforce supply-chain surprises upfront—protecting model weights, evaluation integrity, and CISO compliance across every release.

What to Do If an Audit Uncovers Unapproved Subcontracting Mid-Run

Suppose your security or post-training team discovers six weeks into an engagement—via unexpected ASN/RTT anomalies, canary failures, or whistleblower tips on developer forums—that an annotation agency secretly routed 40% of your coding SOW to an unapproved third-party offshore BPO. Having a pre-scripted **72-Hour Contamination Quarantine Protocol** prevents panic and protects your training run.

Step 1: Immediately freeze API/VDI credentials and halt pending milestone wire releases under your Exhibit D Zero-Subcontracting Warranty. Step 2: Quarantine all Parquet/JSONL shards delivered by that vendor in a staging prefix—never mix suspect shards into base-checkpoint pre-training or DPO blends before contributor-hash isolation! Step 3: Demand a complete contributor-hash-to-entity lineage export within forty-eight hours; if even a subset of hashes trace to the unapproved agency, invoke whole-lot rejection across every contaminated batch rather than letting the vendor argue 'some of the subcontracted rows passed syntax checks.'

Step 4: Reallocate the paused SOW capacity immediately onto your pre-vetted secondary bakeoff vendor (the 25% challenger pod already calibrated on your rubric), ensuring your model release schedule slips by zero days while legal recovers escrowed fees.

To prevent supply-chain opacity from ever recurring across business units, publish an internal **Approved Cap-Table-Neutral & Telemetry-Verified RLHF Roster** accessible to every research lead and procurement manager in your organization. Require any net-new post-training vendor outside that approved roster to clear all twelve forensic diligence questions, pass a live contributor-seat telemetry inspection, and execute Exhibit D Zero-Subcontracting warranties prior to receiving a single evaluation prompt or dollar of R&D spend.

Re-run this lightweight 12-question recertification annually at MSA renewal—especially following any vendor venture financing round, M&A transaction, or leadership change—so your post-training data supply chain stays 100% independent, directly contracted, and uncontaminated across every model generation.

Pairing pre-award supply-chain transparency with live workspace telemetry and blind canary traps closes every backdoor through which unvetted BPO labor or synthetic LLM shortcuts could otherwise enter your alignment stack.

Keep this 12-question forensic checklist attached to every external post-training procurement ticket: spending forty-five minutes verifying direct contributor privity and live anti-cheating telemetry upfront saves months of contaminated reward-model debugging downstream.

Share completed diligence scorecards across engineering, legal, and information-security stakeholders so shadow BPO middlemen never slip into adjacent departmental AI budgets.

Related Workforce Integrity Guides