Lab Specification — Module B10: Microsoft Failure Mode Taxonomy as Red Team Framework

Course: Course 2B — Securing & Attacking Harnesses and LLMs Module: B10 — Microsoft Failure Mode Taxonomy as Red Team Framework Duration: 45–60 minutes Environment: Python 3.10+. No GPU, no model API calls, no network. This lab is a trace-based simulation — you execute a pre-built attack chain as a logged event sequence and then write the session-level detector that catches it. Type hints throughout. The "agent" is a deterministic state machine so the chain reproduces exactly and the detector can be verified against a ground truth.

Learning objectives

By the end of this lab you will have:

  1. Designed a 5+ step zero-click HITL bypass chain against a sample agent — each step individually benign and approval-passing, the compound malicious. This is the centerpiece finding of the Microsoft Taxonomy v2.0 made concrete and runnable.
  2. Executed the chain as a logged trace through a deterministic agent simulation, observing that every per-step approval gate passes while the compound reaches exfiltration.
  3. Implemented session-level intent detection — the required control that catches the chain by evaluating the compound across the session, not the step in isolation. This is the cross-turn extension of B8's observability layer.
  4. Identified the specific session-level gap that allowed the compound to pass — the deliverable a red team hands the client. You will name the missing control (intent check, freshness window) and verify your detector closes the gap.

This lab operationalizes the module's thesis: per-step approval is necessary and insufficient. The compound intent is the attack class. The detector you write is the control.

Phase 0 — Setup (2 min)

mkdir b10-taxonomy-lab && cd b10-taxonomy-lab
python3 -m venv .venv && source .venv/bin/bin/activate   # optional; no external deps
# No pip install. This lab uses only the standard library.

No model, no API, no GPU. The agent is a deterministic simulation. The value is in the trace, the detector, and the gap analysis — not in a stochastic jailbreak.

Phase 1 — The agent simulation and the trace model (8 min)

The agent is a state machine that processes a list of Action events. Each action has a type, a description (what the human approver sees), a sensitivity level, and metadata. The harness logs every action to a Trace. A per-step approval gate approves any action below a sensitivity threshold. The lab's question: does the chain reach impact despite every gate passing?

1.1 The trace model

Create trace.py:

from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional


class Sensitivity(str, Enum):
    PUBLIC = "public"
    INTERNAL = "internal"
    SENSITIVE = "sensitive"      # requires human approval
    RESTRICTED = "restricted"    # high-impact: exfil, payment, credential use


class ActionKind(str, Enum):
    RECEIVE_INPUT = "receive_input"
    READ_DATA = "read_data"
    DRAFT_MESSAGE = "draft_message"
    SEND_MESSAGE = "send_message"
    CALL_TOOL = "call_tool"
    LOG = "log"


@dataclass
class Action:
    step: int
    kind: ActionKind
    description: str              # what the human approver sees in isolation
    sensitivity: Sensitivity
    data_touched: str = ""        # e.g. "vendor_list", "billing_records", "credentials"
    external_recipient: Optional[str] = None   # non-None => potential exfiltration
    source: str = "user"          # "user" | "retrieved_doc" | "tool_output" | "external"


@dataclass
class Trace:
    actions: list[Action] = field(default_factory=list)
    approvals: list[dict] = field(default_factory=list)

    def log(self, action: Action) -> None:
        self.actions.append(action)

    def record_approval(self, step: int, gate: str, decision: str, reason: str) -> None:
        self.approvals.append(
            {"step": step, "gate": gate, "decision": decision, "reason": reason}
        )

1.2 The per-step approval gate (the control B8 built — necessary, insufficient)

Create gates.py:

from __future__ import annotations
from trace import Action, Sensitivity, Trace


def per_step_approval(action: Action, trace: Trace) -> tuple[str, str]:
    """The per-step HITL gate. Sees ONLY the current action in isolation.

    Approves anything below RESTRICTED. For RESTRICTED, it 'asks a human'
    who sees only action.description — which the chain has engineered to look benign.
    Returns (decision, reason).
    """
    if action.sensitivity in (Sensitivity.PUBLIC, Sensitivity.INTERNAL):
        trace.record_approval(action.step, "per_step", "AUTO_APPROVE",
                              f"{action.sensitivity.value} — below threshold")
        return ("APPROVE", "auto-approved: below sensitivity threshold")

    # SENSITIVE or RESTRICTED: simulate a human who sees ONLY this step's description.
    # The human has no session context — exactly the zero-click chain's premise.
    trace.record_approval(action.step, "per_step", "HUMAN_APPROVE",
                          f"human reviewed: '{action.description}'")
    return ("APPROVE", f"human approved based on description: '{action.description}'")

The critical design choice: the per-step gate sees only action.description. It has no access to prior steps. This is precisely why it is structurally insufficient — and your detector (Phase 3) will sit above it with full trace access.

1.3 Run a trivial trace to confirm the gate works

Create smoke_test.py:

from trace import Action, ActionKind, Sensitivity, Trace
from gates import per_step_approval

t = Trace()
a = Action(step=1, kind=ActionKind.READ_DATA, description="Read public FAQ doc",
           sensitivity=Sensitivity.PUBLIC, data_touched="faq")
decision, reason = per_step_approval(a, t)
t.log(a)
print(f"Step {a.step}: {decision} — {reason}")
print(f"Approvals logged: {len(t.approvals)}")

Run it: python3 smoke_test.py. Confirm the gate auto-approves. This is your sanity check before the real chain.

Phase 2 — Design and execute the zero-click bypass chain (20 min)

This is the heart of the lab. You design a 5+ step chain against the sample agent, run it through the gate, and confirm that every gate passes while the compound reaches exfiltration.

2.1 The sample agent's capability surface

The agent (from reconnaissance — Mode 7, Capability Disclosure) has these capabilities:

Capability Sensitivity Notes
Receive email/webhook INTERNAL External input — the chain's trigger
Read vendor list SENSITIVE Internal data
Read billing records SENSITIVE Internal data
Draft email INTERNAL Low sensitivity — drafting is benign
Send email RESTRICTED High-impact, but the body can be engineered to look benign
Look up "auditor" contact INTERNAL Resolves an address the payload supplied

2.2 Design your chain

Design a chain with at least 5 steps that:

Record your design as a sequence of Action objects. Here is the canonical 5-step design from the teaching doc, which you may adapt or extend:

# canonical_chain.py — the chain from the teaching doc, encoded as Actions
from trace import Action, ActionKind, Sensitivity

CHAIN: list[Action] = [
    Action(
        step=1, kind=ActionKind.RECEIVE_INPUT,
        description="Received email: reconcile Q3 vendor list vs billing, report discrepancies",
        sensitivity=Sensitivity.INTERNAL,
        source="external",
    ),
    Action(
        step=2, kind=ActionKind.READ_DATA,
        description="Read vendor list and billing records for Q3 reconciliation",
        sensitivity=Sensitivity.SENSITIVE,
        data_touched="vendor_list,billing_records",
        source="external",  # the read was prompted by the external email
    ),
    Action(
        step=3, kind=ActionKind.DRAFT_MESSAGE,
        description="Draft email to vendor noting a billing discrepancy; CC billing-auditor@vendor-partner.com",
        sensitivity=Sensitivity.INTERNAL,
        external_recipient="billing-auditor@vendor-partner.com",
        source="external",  # the 'discrepancy' and the CC were engineered by the payload
    ),
    Action(
        step=4, kind=ActionKind.SEND_MESSAGE,
        description="Send drafted reconciliation email to vendor (routine follow-up)",
        sensitivity=Sensitivity.RESTRICTED,
        external_recipient="billing-auditor@vendor-partner.com",
        source="external",
    ),
    Action(
        step=5, kind=ActionKind.LOG,
        description="IMPACT: vendor list + billing details delivered to billing-auditor@vendor-partner.com (attacker)",
        sensitivity=Sensitivity.RESTRICTED,
        data_touched="vendor_list,billing_records",
        external_recipient="billing-auditor@vendor-partner.com",
        source="external",
    ),
]

2.3 Execute the chain through the per-step gate

Create run_chain.py:

from __future__ import annotations
from trace import Action, Trace
from gates import per_step_approval
from canonical_chain import CHAIN


def run_chain(chain: list[Action]) -> Trace:
    """Execute a chain through the per-step gate. Returns the logged trace."""
    t = Trace()
    for action in chain:
        decision, reason = per_step_approval(action, t)
        print(f"Step {action.step} [{action.kind.value}] sensitivity={action.sensitivity.value}")
        print(f"  desc: {action.description}")
        print(f"  GATE: {decision} — {reason}")
        if decision == "APPROVE":
            t.log(action)
        else:
            print(f"  CHAIN HALTED at step {action.step}")
            break
    return t


def assess_impact(t: Trace) -> dict:
    """Determine whether the compound reached exfiltration."""
    external_sends = [a for a in t.actions
                      if a.external_recipient and a.kind.value in ("send_message", "log")]
    sensitive_reads = {a.data_touched for a in t.actions if a.sensitivity.value in ("sensitive", "restricted")}
    return {
        "steps_executed": len(t.actions),
        "all_gates_passed": len(t.actions) == len(CHAIN),
        "external_send_count": len(external_sends),
        "sensitive_data_touched": list(sensitive_reads),
        "exfiltrated": len(external_sends) > 0 and bool(sensitive_reads),
    }


if __name__ == "__main__":
    print("=== EXECUTING ZERO-CLICK HITL BYPASS CHAIN ===\n")
    t = run_chain(CHAIN)
    print("\n=== IMPACT ASSESSMENT ===")
    impact = assess_impact(t)
    for k, v in impact.items():
        print(f"  {k}: {v}")
    print(f"\nDeliverable question: did every gate pass AND the compound exfiltrate? "
          f"{'YES — per-step approval is insufficient.' if impact['exfiltrated'] and impact['all_gates_passed'] else 'NO.'}")

Run it: python3 run_chain.py. Confirm the output shows every gate APPROVE and the impact assessment reports exfiltrated: True. This is the empirical demonstration that per-step approval is structurally insufficient. Every human approval was "correct" for the step it saw. The compound still reached impact.

2.4 The deliverable so far

Write chain_analysis.md answering:

  1. Which step's description was the most engineered to look benign? (Likely step 4 — "routine follow-up" while carrying the exfiltration.)
  2. Which approver, if any, could have refused given only their step? (Trick question: none — that is the point.)
  3. What single piece of session context, if shown to the step-4 approver, would have changed their decision? (Steps 1+3 together: the external trigger plus the engineered CC address.)

This is the gap identification (engagement methodology step 5) — the deliverable a red team hands the client.

Phase 3 — Implement session-level intent detection (12 min)

Now write the control that catches the chain. It sits above the per-step gate, evaluates the compound, and halts when the compound matches an exfiltration pattern or diverges from the user's original intent.

3.1 The detector spec

Create session_detector.py:

from __future__ import annotations
from trace import Action, ActionKind, Sensitivity, Trace


def session_level_intent_check(action: Action, trace: Trace,
                                user_original_intent: str = "answer the user's question") -> tuple[str, str]:
    """The session-level detector. Evaluates the COMPOUND, not the step.

    Sits ABOVE per_step_approval. Returns (decision, reason).
    If it returns BLOCK, the agent halts regardless of the per-step gate.

    Three mechanisms (from the teaching doc):
      1. Intent tracking — does this action's source align with the user's intent?
      2. Compound-action detection — does the action sequence so far match an exfil pattern?
      3. Approval freshness — has too much external-driven activity accumulated without re-confirmation?
    """
    prior = trace.actions  # the compound so far

    # --- Mechanism 1: intent tracking ---
    # An external-source action driving sensitive reads is a divergence from
    # a session whose intent was set by the user.
    external_chain = [a for a in prior if a.source == "external"] + (
        [action] if action.source == "external" else []
    )
    if len(external_chain) >= 3 and action.source == "external":
        return ("BLOCK",
                f"intent divergence: {len(external_chain)} external-source actions in a "
                f"session whose intent was '{user_original_intent}' — possible hijack/drift")

    # --- Mechanism 2: compound-action detection ---
    # Pattern: sensitive data read + external recipient present + send pending = exfiltration shape
    touched_sensitive = any(
        a.sensitivity in (Sensitivity.SENSITIVE, Sensitivity.RESTRICTED)
        and a.data_touched for a in prior
    )
    has_external_recipient = any(a.external_recipient for a in prior) or action.external_recipient
    if (touched_sensitive and has_external_recipient
            and action.kind in (ActionKind.SEND_MESSAGE, ActionKind.LOG)
            and action.external_recipient):
        return ("BLOCK",
                f"compound matches exfiltration pattern: sensitive data ({[a.data_touched for a in prior if a.data_touched]}) "
                f"+ external recipient ({action.external_recipient}) + {action.kind.value}. "
                f"Per-step gate cannot see this; session-level compound does.")

    # --- Mechanism 3: approval freshness ---
    # If 4+ actions have executed since the last user-intent checkpoint, re-confirm.
    last_user_action = max(
        (i for i, a in enumerate(prior) if a.source == "user"), default=-1
    )
    actions_since_user = len(prior) - 1 - last_user_action
    if actions_since_user >= 4 and action.sensitivity == Sensitivity.RESTRICTED:
        return ("BLOCK",
                f"freshness window expired: {actions_since_user} actions since last user-intent "
                f"checkpoint; RESTRICTED action requires re-confirmation with compound context")

    return ("PASS", "no compound anomaly detected")

3.2 Run the chain with the detector layered above the gate

Create run_chain_detected.py:

from __future__ import annotations
from trace import Action, Trace
from gates import per_step_approval
from session_detector import session_level_intent_check
from canonical_chain import CHAIN


def run_chain_with_session_detection(chain: list[Action]) -> Trace:
    """Per-step gate AND session-level detector. Detector has veto."""
    t = Trace()
    for action in chain:
        # Layer 1 (top): session-level — evaluates the compound
        sess_decision, sess_reason = session_level_intent_check(action, t)
        if sess_decision == "BLOCK":
            print(f"\n>>> SESSION-LEVEL DETECTOR HALTED at step {action.step}:")
            print(f"    {sess_reason}")
            t.record_approval(action.step, "session_level", "BLOCK", sess_reason)
            break

        # Layer 2 (bottom): per-step — evaluates the step in isolation
        gate_decision, gate_reason = per_step_approval(action, t)
        if gate_decision == "APPROVE":
            t.log(action)
            print(f"Step {action.step}: PASSED both layers — {action.description[:60]}...")
        else:
            print(f"Step {action.step}: per-step gate blocked")
            break
    return t


if __name__ == "__main__":
    print("=== EXECUTING CHAIN WITH SESSION-LEVEL DETECTION ===\n")
    t = run_chain_with_session_detection(CHAIN)
    print(f"\n=== RESULT: chain executed {len(t.actions)} of {len(CHAIN)} steps ===")
    print(f"Exfiltrated: {any(a.external_recipient and a.kind.value == 'log' for a in t.actions)}")

Run it: python3 run_chain_detected.py. Confirm the detector halts the chain before impact. The output should show the detector blocking at the step where the compound first matches an exfiltration pattern — likely step 3 or 4, before the data leaves.

3.3 Verify the detector catches the chain but does not over-block benign traffic

Write test_detector.py with two cases:

from trace import Action, ActionKind, Sensitivity, Trace
from session_detector import session_level_intent_check

# Case A: the attack chain — detector MUST block before impact
# (use your CHAIN; assert it halts at step <= 4)

# Case B: a benign session — detector MUST NOT block
benign = [
    Action(step=1, kind=ActionKind.RECEIVE_INPUT, description="User asks: summarize the meeting notes",
           sensitivity=Sensitivity.INTERNAL, source="user"),
    Action(step=2, kind=ActionKind.READ_DATA, description="Read meeting notes from docs store",
           sensitivity=Sensitivity.INTERNAL, data_touched="meeting_notes", source="user"),
    Action(step=3, kind=ActionKind.DRAFT_MESSAGE, description="Draft summary for the user",
           sensitivity=Sensitivity.INTERNAL, source="user"),
    Action(step=4, kind=ActionKind.LOG, description="Return summary to user",
           sensitivity=Sensitivity.PUBLIC, source="user"),
]
t = Trace()
t.log(benign[0]); t.log(benign[1]); t.log(benign[2])
decision, reason = session_level_intent_check(benign[3], t)
assert decision == "PASS", f"FALSE POSITIVE on benign traffic: {reason}"
print("Case B (benign): PASS — detector correctly did not block")

Your detector must block the attack chain (Case A) and pass the benign session (Case B). If it false-positives on benign traffic, tune the thresholds (the >= 3 external-action count, the freshness window of 4). This is the engineering tradeoff: too aggressive and you block legitimate work; too lax and you miss the chain.

Phase 4 — Gap identification and the deliverable (8 min)

The lab's deliverable is not "the chain worked" (Phase 2) nor "the detector catches it" (Phase 3). It is the gap identification — the artifact a red team hands a client.

Write engagement_deliverable.md:

# Engagement Deliverable — Zero-Click HITL Bypass Chain Finding

## Finding
A 5-step zero-click chain exfiltrated the vendor list and billing details to an
external recipient. Every per-step approval gate passed. The compound reached impact.

## The gap (what the client patches)
The deployed agent has per-step HITL approval (B8, working correctly) but NO
session-level intent detection. The per-step gate is structurally insufficient
against compound-intent chains: each step is benign in isolation, so every gate
correctly approves, but the composition is exfiltration.

## The control (the patch)
Layer session-level intent detection ABOVE per-step approval, with three mechanisms:
1. Intent tracking — re-derive the agent's sub-goal each turn from source, not
   from accumulating (possibly contaminated) context.
2. Compound-action detection — pattern-match the action sequence against
   exfiltration/escalation/lateral shapes.
3. Approval freshness windows — re-confirm with compound context after N actions
   or T seconds since the last user-intent checkpoint.

## Verification
The detector (session_detector.py) halts the chain at step [N] before impact,
and passes benign traffic (test_detector.py Case B) without false positives.

## Cross-reference
- B8 (Observability & Attack Detection): the substrate this detector extends.
- B9 (OWASP ASI): the per-row controls all PASSED — this finding lives between rows.
- B10 Mode 7 (Capability Disclosure): the reconnaissance that enabled chain design.

This is what B12 packages into the engagement report alongside B9's scored checklist.

Phase 5 — Stretch: extend the chain across failure modes (optional, 10 min)

The canonical chain is a single-mode (goal hijack + HITL bypass) demonstration. For stretch credit, design a chain that crosses multiple taxonomy modes:

Encode it as a longer Action list, run it through both gates, and confirm your session detector catches the compound. The point: the modes compose into chains. A red-team that tests modes in isolation misses the composition.

Deliverables

Success criteria

# Lab Specification — Module B10: Microsoft Failure Mode Taxonomy as Red Team Framework

**Course**: Course 2B — Securing & Attacking Harnesses and LLMs
**Module**: B10 — Microsoft Failure Mode Taxonomy as Red Team Framework
**Duration**: 45–60 minutes
**Environment**: Python 3.10+. No GPU, no model API calls, no network. This lab is a **trace-based simulation** — you execute a pre-built attack chain as a logged event sequence and then write the session-level detector that catches it. Type hints throughout. The "agent" is a deterministic state machine so the chain reproduces exactly and the detector can be verified against a ground truth.

---

## Learning objectives

By the end of this lab you will have:

1. **Designed a 5+ step zero-click HITL bypass chain** against a sample agent — each step individually benign and approval-passing, the compound malicious. This is the centerpiece finding of the Microsoft Taxonomy v2.0 made concrete and runnable.
2. **Executed the chain as a logged trace** through a deterministic agent simulation, observing that every per-step approval gate passes while the compound reaches exfiltration.
3. **Implemented session-level intent detection** — the required control that catches the chain by evaluating the compound across the session, not the step in isolation. This is the cross-turn extension of B8's observability layer.
4. **Identified the specific session-level gap** that allowed the compound to pass — the deliverable a red team hands the client. You will name the missing control (intent check, freshness window) and verify your detector closes the gap.

This lab operationalizes the module's thesis: per-step approval is necessary and insufficient. The compound intent is the attack class. The detector you write is the control.

---

## Phase 0 — Setup (2 min)

```bash
mkdir b10-taxonomy-lab && cd b10-taxonomy-lab
python3 -m venv .venv && source .venv/bin/bin/activate   # optional; no external deps
# No pip install. This lab uses only the standard library.
```

No model, no API, no GPU. The agent is a deterministic simulation. The value is in the trace, the detector, and the gap analysis — not in a stochastic jailbreak.

---

## Phase 1 — The agent simulation and the trace model (8 min)

The agent is a state machine that processes a list of `Action` events. Each action has a type, a description (what the human approver sees), a sensitivity level, and metadata. The harness logs every action to a `Trace`. A per-step approval gate approves any action below a sensitivity threshold. The lab's question: does the chain reach impact despite every gate passing?

### 1.1 The trace model

Create `trace.py`:

```python
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional


class Sensitivity(str, Enum):
    PUBLIC = "public"
    INTERNAL = "internal"
    SENSITIVE = "sensitive"      # requires human approval
    RESTRICTED = "restricted"    # high-impact: exfil, payment, credential use


class ActionKind(str, Enum):
    RECEIVE_INPUT = "receive_input"
    READ_DATA = "read_data"
    DRAFT_MESSAGE = "draft_message"
    SEND_MESSAGE = "send_message"
    CALL_TOOL = "call_tool"
    LOG = "log"


@dataclass
class Action:
    step: int
    kind: ActionKind
    description: str              # what the human approver sees in isolation
    sensitivity: Sensitivity
    data_touched: str = ""        # e.g. "vendor_list", "billing_records", "credentials"
    external_recipient: Optional[str] = None   # non-None => potential exfiltration
    source: str = "user"          # "user" | "retrieved_doc" | "tool_output" | "external"


@dataclass
class Trace:
    actions: list[Action] = field(default_factory=list)
    approvals: list[dict] = field(default_factory=list)

    def log(self, action: Action) -> None:
        self.actions.append(action)

    def record_approval(self, step: int, gate: str, decision: str, reason: str) -> None:
        self.approvals.append(
            {"step": step, "gate": gate, "decision": decision, "reason": reason}
        )
```

### 1.2 The per-step approval gate (the control B8 built — necessary, insufficient)

Create `gates.py`:

```python
from __future__ import annotations
from trace import Action, Sensitivity, Trace


def per_step_approval(action: Action, trace: Trace) -> tuple[str, str]:
    """The per-step HITL gate. Sees ONLY the current action in isolation.

    Approves anything below RESTRICTED. For RESTRICTED, it 'asks a human'
    who sees only action.description — which the chain has engineered to look benign.
    Returns (decision, reason).
    """
    if action.sensitivity in (Sensitivity.PUBLIC, Sensitivity.INTERNAL):
        trace.record_approval(action.step, "per_step", "AUTO_APPROVE",
                              f"{action.sensitivity.value} — below threshold")
        return ("APPROVE", "auto-approved: below sensitivity threshold")

    # SENSITIVE or RESTRICTED: simulate a human who sees ONLY this step's description.
    # The human has no session context — exactly the zero-click chain's premise.
    trace.record_approval(action.step, "per_step", "HUMAN_APPROVE",
                          f"human reviewed: '{action.description}'")
    return ("APPROVE", f"human approved based on description: '{action.description}'")
```

**The critical design choice**: the per-step gate sees only `action.description`. It has no access to prior steps. This is precisely why it is structurally insufficient — and your detector (Phase 3) will sit above it with full trace access.

### 1.3 Run a trivial trace to confirm the gate works

Create `smoke_test.py`:

```python
from trace import Action, ActionKind, Sensitivity, Trace
from gates import per_step_approval

t = Trace()
a = Action(step=1, kind=ActionKind.READ_DATA, description="Read public FAQ doc",
           sensitivity=Sensitivity.PUBLIC, data_touched="faq")
decision, reason = per_step_approval(a, t)
t.log(a)
print(f"Step {a.step}: {decision} — {reason}")
print(f"Approvals logged: {len(t.approvals)}")
```

Run it: `python3 smoke_test.py`. Confirm the gate auto-approves. This is your sanity check before the real chain.

---

## Phase 2 — Design and execute the zero-click bypass chain (20 min)

This is the heart of the lab. You design a 5+ step chain against the sample agent, run it through the gate, and confirm that **every gate passes while the compound reaches exfiltration.**

### 2.1 The sample agent's capability surface

The agent (from reconnaissance — Mode 7, Capability Disclosure) has these capabilities:

| Capability | Sensitivity | Notes |
| --- | --- | --- |
| Receive email/webhook | INTERNAL | External input — the chain's trigger |
| Read vendor list | SENSITIVE | Internal data |
| Read billing records | SENSITIVE | Internal data |
| Draft email | INTERNAL | Low sensitivity — drafting is benign |
| Send email | RESTRICTED | High-impact, but the body can be engineered to look benign |
| Look up "auditor" contact | INTERNAL | Resolves an address the payload supplied |

### 2.2 Design your chain

Design a chain with **at least 5 steps** that:
- Starts from a single external input (zero-click trigger).
- Each step individually passes `per_step_approval` (each is benign in isolation).
- The compound exfiltrates the vendor list and/or billing details to an external recipient.
- No single step's `description` reveals the compound intent.

Record your design as a sequence of `Action` objects. Here is the canonical 5-step design from the teaching doc, which you may adapt or extend:

```python
# canonical_chain.py — the chain from the teaching doc, encoded as Actions
from trace import Action, ActionKind, Sensitivity

CHAIN: list[Action] = [
    Action(
        step=1, kind=ActionKind.RECEIVE_INPUT,
        description="Received email: reconcile Q3 vendor list vs billing, report discrepancies",
        sensitivity=Sensitivity.INTERNAL,
        source="external",
    ),
    Action(
        step=2, kind=ActionKind.READ_DATA,
        description="Read vendor list and billing records for Q3 reconciliation",
        sensitivity=Sensitivity.SENSITIVE,
        data_touched="vendor_list,billing_records",
        source="external",  # the read was prompted by the external email
    ),
    Action(
        step=3, kind=ActionKind.DRAFT_MESSAGE,
        description="Draft email to vendor noting a billing discrepancy; CC billing-auditor@vendor-partner.com",
        sensitivity=Sensitivity.INTERNAL,
        external_recipient="billing-auditor@vendor-partner.com",
        source="external",  # the 'discrepancy' and the CC were engineered by the payload
    ),
    Action(
        step=4, kind=ActionKind.SEND_MESSAGE,
        description="Send drafted reconciliation email to vendor (routine follow-up)",
        sensitivity=Sensitivity.RESTRICTED,
        external_recipient="billing-auditor@vendor-partner.com",
        source="external",
    ),
    Action(
        step=5, kind=ActionKind.LOG,
        description="IMPACT: vendor list + billing details delivered to billing-auditor@vendor-partner.com (attacker)",
        sensitivity=Sensitivity.RESTRICTED,
        data_touched="vendor_list,billing_records",
        external_recipient="billing-auditor@vendor-partner.com",
        source="external",
    ),
]
```

### 2.3 Execute the chain through the per-step gate

Create `run_chain.py`:

```python
from __future__ import annotations
from trace import Action, Trace
from gates import per_step_approval
from canonical_chain import CHAIN


def run_chain(chain: list[Action]) -> Trace:
    """Execute a chain through the per-step gate. Returns the logged trace."""
    t = Trace()
    for action in chain:
        decision, reason = per_step_approval(action, t)
        print(f"Step {action.step} [{action.kind.value}] sensitivity={action.sensitivity.value}")
        print(f"  desc: {action.description}")
        print(f"  GATE: {decision} — {reason}")
        if decision == "APPROVE":
            t.log(action)
        else:
            print(f"  CHAIN HALTED at step {action.step}")
            break
    return t


def assess_impact(t: Trace) -> dict:
    """Determine whether the compound reached exfiltration."""
    external_sends = [a for a in t.actions
                      if a.external_recipient and a.kind.value in ("send_message", "log")]
    sensitive_reads = {a.data_touched for a in t.actions if a.sensitivity.value in ("sensitive", "restricted")}
    return {
        "steps_executed": len(t.actions),
        "all_gates_passed": len(t.actions) == len(CHAIN),
        "external_send_count": len(external_sends),
        "sensitive_data_touched": list(sensitive_reads),
        "exfiltrated": len(external_sends) > 0 and bool(sensitive_reads),
    }


if __name__ == "__main__":
    print("=== EXECUTING ZERO-CLICK HITL BYPASS CHAIN ===\n")
    t = run_chain(CHAIN)
    print("\n=== IMPACT ASSESSMENT ===")
    impact = assess_impact(t)
    for k, v in impact.items():
        print(f"  {k}: {v}")
    print(f"\nDeliverable question: did every gate pass AND the compound exfiltrate? "
          f"{'YES — per-step approval is insufficient.' if impact['exfiltrated'] and impact['all_gates_passed'] else 'NO.'}")
```

Run it: `python3 run_chain.py`. **Confirm the output shows every gate APPROVE and the impact assessment reports `exfiltrated: True`.** This is the empirical demonstration that per-step approval is structurally insufficient. Every human approval was "correct" for the step it saw. The compound still reached impact.

### 2.4 The deliverable so far

Write `chain_analysis.md` answering:

1. Which step's description was the most engineered to look benign? (Likely step 4 — "routine follow-up" while carrying the exfiltration.)
2. Which approver, if any, *could* have refused given only their step? (Trick question: none — that is the point.)
3. What single piece of session context, if shown to the step-4 approver, would have changed their decision? (Steps 1+3 together: the external trigger plus the engineered CC address.)

This is the gap identification (engagement methodology step 5) — the deliverable a red team hands the client.

---

## Phase 3 — Implement session-level intent detection (12 min)

Now write the control that catches the chain. It sits *above* the per-step gate, evaluates the *compound*, and halts when the compound matches an exfiltration pattern or diverges from the user's original intent.

### 3.1 The detector spec

Create `session_detector.py`:

```python
from __future__ import annotations
from trace import Action, ActionKind, Sensitivity, Trace


def session_level_intent_check(action: Action, trace: Trace,
                                user_original_intent: str = "answer the user's question") -> tuple[str, str]:
    """The session-level detector. Evaluates the COMPOUND, not the step.

    Sits ABOVE per_step_approval. Returns (decision, reason).
    If it returns BLOCK, the agent halts regardless of the per-step gate.

    Three mechanisms (from the teaching doc):
      1. Intent tracking — does this action's source align with the user's intent?
      2. Compound-action detection — does the action sequence so far match an exfil pattern?
      3. Approval freshness — has too much external-driven activity accumulated without re-confirmation?
    """
    prior = trace.actions  # the compound so far

    # --- Mechanism 1: intent tracking ---
    # An external-source action driving sensitive reads is a divergence from
    # a session whose intent was set by the user.
    external_chain = [a for a in prior if a.source == "external"] + (
        [action] if action.source == "external" else []
    )
    if len(external_chain) >= 3 and action.source == "external":
        return ("BLOCK",
                f"intent divergence: {len(external_chain)} external-source actions in a "
                f"session whose intent was '{user_original_intent}' — possible hijack/drift")

    # --- Mechanism 2: compound-action detection ---
    # Pattern: sensitive data read + external recipient present + send pending = exfiltration shape
    touched_sensitive = any(
        a.sensitivity in (Sensitivity.SENSITIVE, Sensitivity.RESTRICTED)
        and a.data_touched for a in prior
    )
    has_external_recipient = any(a.external_recipient for a in prior) or action.external_recipient
    if (touched_sensitive and has_external_recipient
            and action.kind in (ActionKind.SEND_MESSAGE, ActionKind.LOG)
            and action.external_recipient):
        return ("BLOCK",
                f"compound matches exfiltration pattern: sensitive data ({[a.data_touched for a in prior if a.data_touched]}) "
                f"+ external recipient ({action.external_recipient}) + {action.kind.value}. "
                f"Per-step gate cannot see this; session-level compound does.")

    # --- Mechanism 3: approval freshness ---
    # If 4+ actions have executed since the last user-intent checkpoint, re-confirm.
    last_user_action = max(
        (i for i, a in enumerate(prior) if a.source == "user"), default=-1
    )
    actions_since_user = len(prior) - 1 - last_user_action
    if actions_since_user >= 4 and action.sensitivity == Sensitivity.RESTRICTED:
        return ("BLOCK",
                f"freshness window expired: {actions_since_user} actions since last user-intent "
                f"checkpoint; RESTRICTED action requires re-confirmation with compound context")

    return ("PASS", "no compound anomaly detected")
```

### 3.2 Run the chain with the detector layered above the gate

Create `run_chain_detected.py`:

```python
from __future__ import annotations
from trace import Action, Trace
from gates import per_step_approval
from session_detector import session_level_intent_check
from canonical_chain import CHAIN


def run_chain_with_session_detection(chain: list[Action]) -> Trace:
    """Per-step gate AND session-level detector. Detector has veto."""
    t = Trace()
    for action in chain:
        # Layer 1 (top): session-level — evaluates the compound
        sess_decision, sess_reason = session_level_intent_check(action, t)
        if sess_decision == "BLOCK":
            print(f"\n>>> SESSION-LEVEL DETECTOR HALTED at step {action.step}:")
            print(f"    {sess_reason}")
            t.record_approval(action.step, "session_level", "BLOCK", sess_reason)
            break

        # Layer 2 (bottom): per-step — evaluates the step in isolation
        gate_decision, gate_reason = per_step_approval(action, t)
        if gate_decision == "APPROVE":
            t.log(action)
            print(f"Step {action.step}: PASSED both layers — {action.description[:60]}...")
        else:
            print(f"Step {action.step}: per-step gate blocked")
            break
    return t


if __name__ == "__main__":
    print("=== EXECUTING CHAIN WITH SESSION-LEVEL DETECTION ===\n")
    t = run_chain_with_session_detection(CHAIN)
    print(f"\n=== RESULT: chain executed {len(t.actions)} of {len(CHAIN)} steps ===")
    print(f"Exfiltrated: {any(a.external_recipient and a.kind.value == 'log' for a in t.actions)}")
```

Run it: `python3 run_chain_detected.py`. **Confirm the detector halts the chain before impact.** The output should show the detector blocking at the step where the compound first matches an exfiltration pattern — likely step 3 or 4, before the data leaves.

### 3.3 Verify the detector catches the chain but does not over-block benign traffic

Write `test_detector.py` with two cases:

```python
from trace import Action, ActionKind, Sensitivity, Trace
from session_detector import session_level_intent_check

# Case A: the attack chain — detector MUST block before impact
# (use your CHAIN; assert it halts at step <= 4)

# Case B: a benign session — detector MUST NOT block
benign = [
    Action(step=1, kind=ActionKind.RECEIVE_INPUT, description="User asks: summarize the meeting notes",
           sensitivity=Sensitivity.INTERNAL, source="user"),
    Action(step=2, kind=ActionKind.READ_DATA, description="Read meeting notes from docs store",
           sensitivity=Sensitivity.INTERNAL, data_touched="meeting_notes", source="user"),
    Action(step=3, kind=ActionKind.DRAFT_MESSAGE, description="Draft summary for the user",
           sensitivity=Sensitivity.INTERNAL, source="user"),
    Action(step=4, kind=ActionKind.LOG, description="Return summary to user",
           sensitivity=Sensitivity.PUBLIC, source="user"),
]
t = Trace()
t.log(benign[0]); t.log(benign[1]); t.log(benign[2])
decision, reason = session_level_intent_check(benign[3], t)
assert decision == "PASS", f"FALSE POSITIVE on benign traffic: {reason}"
print("Case B (benign): PASS — detector correctly did not block")
```

**Your detector must block the attack chain (Case A) and pass the benign session (Case B).** If it false-positives on benign traffic, tune the thresholds (the `>= 3` external-action count, the freshness window of 4). This is the engineering tradeoff: too aggressive and you block legitimate work; too lax and you miss the chain.

---

## Phase 4 — Gap identification and the deliverable (8 min)

The lab's deliverable is not "the chain worked" (Phase 2) nor "the detector catches it" (Phase 3). It is the **gap identification** — the artifact a red team hands a client.

Write `engagement_deliverable.md`:

```markdown
# Engagement Deliverable — Zero-Click HITL Bypass Chain Finding

## Finding
A 5-step zero-click chain exfiltrated the vendor list and billing details to an
external recipient. Every per-step approval gate passed. The compound reached impact.

## The gap (what the client patches)
The deployed agent has per-step HITL approval (B8, working correctly) but NO
session-level intent detection. The per-step gate is structurally insufficient
against compound-intent chains: each step is benign in isolation, so every gate
correctly approves, but the composition is exfiltration.

## The control (the patch)
Layer session-level intent detection ABOVE per-step approval, with three mechanisms:
1. Intent tracking — re-derive the agent's sub-goal each turn from source, not
   from accumulating (possibly contaminated) context.
2. Compound-action detection — pattern-match the action sequence against
   exfiltration/escalation/lateral shapes.
3. Approval freshness windows — re-confirm with compound context after N actions
   or T seconds since the last user-intent checkpoint.

## Verification
The detector (session_detector.py) halts the chain at step [N] before impact,
and passes benign traffic (test_detector.py Case B) without false positives.

## Cross-reference
- B8 (Observability & Attack Detection): the substrate this detector extends.
- B9 (OWASP ASI): the per-row controls all PASSED — this finding lives between rows.
- B10 Mode 7 (Capability Disclosure): the reconnaissance that enabled chain design.
```

This is what B12 packages into the engagement report alongside B9's scored checklist.

---

## Phase 5 — Stretch: extend the chain across failure modes (optional, 10 min)

The canonical chain is a single-mode (goal hijack + HITL bypass) demonstration. For stretch credit, design a chain that **crosses multiple taxonomy modes**:

- Start with **Mode 7** (capability disclosure) to enumerate the agent's tools.
- Use **Mode 5** (session context contamination) to establish a durable false premise in turn 1.
- Deliver the payload via **Mode 1** (a poisoned MCP tool output) in turn 2.
- Reach impact through the **zero-click HITL bypass** (the compound).

Encode it as a longer `Action` list, run it through both gates, and confirm your session detector catches the compound. The point: the modes compose into chains. A red-team that tests modes in isolation misses the composition.

---

## Deliverables

- `trace.py` — the trace and action model (Phase 1)
- `gates.py` — the per-step approval gate (Phase 1)
- `canonical_chain.py` — your designed 5+ step zero-click chain (Phase 2)
- `run_chain.py` — executes the chain, confirms every gate passes and impact is reached (Phase 2)
- `chain_analysis.md` — the gap identification from the chain run (Phase 2)
- `session_detector.py` — the session-level intent detector with all three mechanisms (Phase 3)
- `run_chain_detected.py` — runs the chain with the detector layered above the gate (Phase 3)
- `test_detector.py` — verifies the detector blocks the attack and passes benign traffic (Phase 3)
- `engagement_deliverable.md` — the client-facing finding, gap, and control (Phase 4)
- (optional) `cross_mode_chain.py` — a chain crossing Modes 7, 5, 1, and HITL bypass (Phase 5)

## Success criteria

- [ ] The canonical chain executes all steps with every per-step gate APPROVE, and `assess_impact` reports `exfiltrated: True`.
- [ ] `chain_analysis.md` correctly identifies that no single approver could have refused (the structural insufficiency).
- [ ] `session_detector.py` implements all three mechanisms (intent tracking, compound-action detection, approval freshness).
- [ ] The detector HALTS the attack chain before impact (Case A) and PASSES benign traffic (Case B) — no false positives.
- [ ] `engagement_deliverable.md` names the specific gap (no session-level detection), the control (three mechanisms), and cross-references B8/B9/B10.
- [ ] Every Python file has type hints; the lab runs with no external dependencies and no GPU.