Debugging with Provenance

Overview

Something went wrong. A booking was confirmed when it shouldn’t have been. An intent fired for a cancelled request. A fact you expected is nowhere in the world state.

In a traditional system, you’d open the code and start reading. In JacqOS, you don’t read the generated rules. You trace the evidence.

This guide walks through debugging with provenance — from spotting the problem to understanding exactly why it happened to fixing it. All examples use the appointment-booking app.

The Debugging Mindset

JacqOS debugging is backwards from what you’re used to. Instead of reading code top-down and simulating what should happen, you start from what did happen and follow the evidence backward.

The chain is always the same:

Bad fact or missing fact
  → Which rule derived it (or failed to fire)
    → Which atoms satisfied the rule body (or didn't)
      → Which observations produced those atoms

Every step is concrete. No mental simulation, no guessing, no “I think this rule might match.” The evaluator already computed everything — you’re just inspecting the result.

Starting Point: Spot the Problem

Problems surface in three ways:

jacqos verify fails — a fixture expectation doesn’t match, or an invariant is violated
jacqos replay shows unexpected state — a fact exists that shouldn’t, or is missing when it should be present
Studio’s Activity surface shows an unexpected row — an action lands in Done, Blocked, or Waiting that shouldn’t be there

Let’s work through each scenario with the appointment-booking app.

Scenario 1: An Unexpected Fact Exists

You run jacqos verify and get:

$ jacqos verify
Replaying fixtures...
  double-booking-path.jsonl    FAIL

  Unexpected fact: booking_confirmed("req-2", "slot-42")
  Expected:        (not derived)

  Hint: rule rules.dh:58 fired unexpectedly.

Two bookings were confirmed for the same slot. The no_double_booking invariant should have caught this, but the fact itself is the immediate problem. Let’s trace it.

Step 1: Open the Drill Inspector

Launch Studio and load the double-booking fixture:

$ jacqos replay fixtures/double-booking-path.jsonl
$ jacqos studio

On the Activity tab, find the two booking_confirmed rows:

booking_confirmed("req-1", "slot-42")
booking_confirmed("req-2", "slot-42")

Click the row for booking_confirmed("req-2", "slot-42") to open its drill inspector.

Step 2: Read the Provenance Chain

The Decision, Facts, and Atoms / Observations sections show the derivation chain in text form. The Timeline section walks the same chain in reverse chronological order:

booking_confirmed("req-2", "slot-42")
  ← rule: assert booking_confirmed (rules.dh:58)
    ← confirmation_sent("req-2")
      ← rule: assert confirmation_sent (rules.dh:39)
        ← atom(obs-4, "confirmation.result", "sent")
          ← Observation obs-4: confirmation.result
        ← atom(obs-4, "confirmation.request_id", "req-2")
    ← slot_hold_active("req-2", "slot-42")
      ← rule: assert slot_hold_active (rules.dh:17)
        ← atom(obs-3, "reservation.result", "succeeded")
          ← Observation obs-3: reservation.result
        ← atom(obs-3, "reservation.request_id", "req-2")
        ← atom(obs-3, "reservation.slot_id", "slot-42")

Now you can see exactly what happened:

booking_confirmed fired because both confirmation_sent("req-2") and slot_hold_active("req-2", "slot-42") were true
slot_hold_active was asserted because observation obs-3 reported a successful reservation
confirmation_sent was asserted because observation obs-4 reported a sent confirmation

The problem is clear: the fixture includes a successful reservation for req-2 that shouldn’t have succeeded (the slot was already held by req-1). The fixture itself is wrong, or the rules need a guard.

Step 3: Check the Rule’s Position in the Ontology

Open the Ontology destination and look up booking_confirmed in the strata tree. Its inspector shows the stratum index and prefix kind. The rule at rules.dh:58 derives booking_confirmed whenever there’s a hold and a confirmation, without checking if another booking already exists. That’s the gap.

Step 4: Fix It

You now know exactly what to fix: the rule needs a negation guard, or the invariant no_double_booking needs to be exercised in property testing to catch this. Add the guard to the fixture expectations and have the AI regenerate:

$ jacqos verify
# After AI updates rules.dh:58 to include: not booking_confirmed(_, slot)
All checks passed.

At no point did you read the rule syntax to understand the bug. You saw what the rule produced, traced the evidence, and identified the missing condition.

Scenario 2: An Expected Fact Is Missing

You run jacqos verify and get:

$ jacqos verify
Replaying fixtures...
  happy-path.jsonl             FAIL

  Expected: booking_confirmed("req-1", "slot-42")
  Got:      (not derived)

  Hint: rule rules.dh:58 did not fire.

The booking should have been confirmed, but it wasn’t. Something in the derivation chain broke.

Step 1: Find the Closest Activity Row

In Studio, the missing booking means no booking_confirmed("req-1", …) row appears in Done. The closest action is whatever made the row land in Waiting — typically a confirmation.required or slot_hold_active event. Open its drill inspector and read the Decision and Facts sections to see how far the chain progressed.

The jacqos verify failure already named the rule that didn’t fire: rules.dh:58 (booking_confirmed). That rule’s body checks confirmation_sent("req-1") and slot_hold_active("req-1", "slot-42"). The verification bundle records which clause failed:

rule: assert booking_confirmed (rules.dh:58)
  ✓ confirmation_sent("req-1") — exists
  ✗ slot_hold_active("req-1", "slot-42") — NOT FOUND

  Derivation blocked: slot_hold_active not asserted for req-1.

Step 2: Trace the Missing Dependency

The bundle also records why slot_hold_active("req-1", "slot-42") is missing:

rule: assert slot_hold_active (rules.dh:17)
  ✗ atom(_, "reservation.result", "succeeded") — no matching atom

  No observation produced a reservation.result atom
  with result "succeeded" for request_id "req-1".

Now you know the root cause: the mapper didn’t produce a reservation.result atom with value succeeded for req-1. Either the observation is missing from the fixture, or the mapper has a bug.

Step 3: Check the Observation Log

In the drill inspector’s Timeline section, the observations are listed alongside the receipt:

obs-0: slot.status      { slot_id: "slot-42", state: "listed" }
obs-1: booking.request   { request_id: "req-1", email: "pat@example.com", slot_id: "slot-42" }
obs-2: reservation.result { result: "suceeded", request_id: "req-1", slot_id: "slot-42" }
obs-3: confirmation.result { result: "sent", request_id: "req-1" }

There it is: "suceeded" — a typo in the fixture. The atom extracted reservation.result = "suceeded", which doesn’t match the rule’s expected "succeeded".

Fix the fixture, rerun, and the fact derives correctly.

Step 4: Atom Bindings Tell You Everything

The key insight: you didn’t need to guess. The provenance system showed you the exact atom that failed to match, and the timeline showed you the raw payload. The gap between "suceeded" and "succeeded" is visible in the data — you just needed to follow the chain.

Scenario 3: An Invariant Violation

jacqos verify finds a counterexample through property testing:

$ jacqos verify
Property testing invariants...
  one_terminal_outcome         FAIL

  Counterexample found for one_terminal_outcome:
    Shrunk to 3 observations (from 31):
      1. booking.request    { request_id: "req-1", slot_id: "slot-42" }
      2. reservation.result { result: "succeeded", request_id: "req-1", slot_id: "slot-42" }
      3. booking.cancelled  { request_id: "req-1", slot_id: "slot-42" }

    Violation: booking_terminal("req-1") derived twice
      via booking_confirmed("req-1", "slot-42")
      via booking_cancelled("req-1", "slot-42")

The request reached two terminal states: confirmed and cancelled. The one_terminal_outcome invariant says this must never happen.

Step 1: Save and Replay the Counterexample

$ jacqos shrink-fixture fixtures/generated-one_terminal_outcome.jsonl \
    --output fixtures/counter-one_terminal_outcome-001.jsonl

$ jacqos replay fixtures/counter-one_terminal_outcome-001.jsonl
$ jacqos studio

Step 2: Trace Both Terminal Paths

In Studio, find the booking_terminal("req-1") row in Activity. Its drill inspector shows the chain in text form:

booking_terminal("req-1")
  Path A:
    ← rule: booking_terminal (rules.dh:62) [confirmed branch]
      ← booking_confirmed("req-1", "slot-42")
        ← confirmation_sent("req-1") — BUT this doesn't exist!

  Path B:
    ← rule: booking_terminal (rules.dh:68) [cancelled branch]
      ← booking_cancelled("req-1", "slot-42")
        ← atom(obs-3, "booking.cancelled.request_id", "req-1")

Wait — booking_confirmed shouldn’t exist if there’s no confirmation_sent. Let’s check.

Step 3: Follow the Unexpected Confirmation

Open the drill inspector for the booking_confirmed("req-1", "slot-42") row:

booking_confirmed("req-1", "slot-42")
  ← rule: assert booking_confirmed (rules.dh:58)
    ← confirmation_sent("req-1") — asserted, then NOT retracted
    ← slot_hold_active("req-1", "slot-42") — asserted, then retracted

The hold was retracted (due to cancellation), but booking_confirmed was asserted before the retraction happened. Once asserted, booking_confirmed persists because there’s no retraction rule for it when a cancellation arrives after confirmation.

Step 4: Understand the Timing

The Timeline section walks the events in reverse chronological order:

obs-2: reservation succeeded → slot_hold_active asserted
(no confirmation observation, but the shrunk example must have included one — check the timeline)
obs-3: booking cancelled → booking_cancelled asserted, slot_hold_active retracted

The issue: the rules don’t retract booking_confirmed when a cancellation arrives. The AI needs to add:

rule retract booking_confirmed(req, slot) :-
  booking_cancelled(req, slot).

Step 5: Fix and Verify

Save the counterexample as a permanent fixture, have the AI update the rules, and verify:

$ jacqos verify
All checks passed. Digest: sha256:b7c8d9e0f1a2...

The counterexample is now a regression test. If the rules ever regress, this fixture catches it.

Understanding Atom Bindings

Atoms are the bridge between raw observations and the logic layer. When debugging, atom bindings tell you exactly what data flowed from an observation into a rule.

Each atom has three parts:

atom(observation_id, key, value)

In the drill inspector’s Atoms / Observations section, atoms appear as leaf entries:

slot_hold_active("req-1", "slot-42")
  ← atom(obs-2, "reservation.result", "succeeded")
  ← atom(obs-2, "reservation.request_id", "req-1")
  ← atom(obs-2, "reservation.slot_id", "slot-42")

All three atoms came from observation obs-2. The mapper for reservation.result observations extracted three key-value pairs from the payload.

Common Atom Issues

Missing atom: The mapper didn’t extract a value from the observation. Check the mapper’s Rhai code — it may not handle a payload field, or the field name may be different from what the rule expects.

Wrong atom value: The mapper extracted the field but with an unexpected value. This often means the observation payload has a different shape than expected (a string instead of a boolean, a nested object where a flat value was expected).

Wrong observation ID: Atoms from different observations shouldn’t satisfy a rule body that requires them from the same observation. If atom(obs-1, ...) and atom(obs-2, ...) both appear in a rule match, the rule may be missing an observation-identity join.

Understanding Negation in Provenance

Negation is where provenance gets subtle. A rule might fire because something is absent.

Negation Witnesses

When a rule body includes not some_relation(...), the verification bundle records a negation witness for the derivation:

slot_available("slot-42")
  ← rule: slot_available (rules.dh:12)
    ← slot_listed("slot-42") — exists
    ← NOT slot_hold_active(_, "slot-42") — no matching fact
    ← NOT booking_confirmed(_, "slot-42") — no matching fact

The negation witnesses confirm: at the point this rule was evaluated, no slot_hold_active or booking_confirmed fact existed for slot-42. That’s why slot_available was derived.

Negation Failures

When an expected fact is missing because a negation blocked it:

slot_available("slot-42") — not derived

rule: slot_available (rules.dh:12)
  ✓ slot_listed("slot-42") — exists
  ✗ NOT slot_hold_active(_, "slot-42") — BLOCKED
    slot_hold_active("req-1", "slot-42") exists

The rule would have fired, but the negation check found slot_hold_active("req-1", "slot-42"), so derivation was blocked. This is correct behavior — the slot isn’t available because someone holds it.

Stratification and Negation Order

The Ontology destination groups relations by stratum so you can see how the evaluator layers its fixed-point computation. The evaluator guarantees that all facts in lower strata are fully computed before evaluating negation in higher strata. If a negation check seems wrong, check whether the relations involved are in the right strata:

Stratum 0: slot_listed, booking_request, slot_hold_active, ...
Stratum 1: slot_available (negates slot_hold_active from stratum 0)
Stratum 2: booking_status (negates multiple stratum 0/1 relations)

If a fact appears to be negated before it’s derived, the stratification may be wrong. This is rare with AI-generated rules (the .dh loader rejects unstratified negation), but understanding strata helps you read the Ontology surface.

The Ontology Surface as a Debugging Tool

The Ontology destination shows you the shape of your ontology at a glance — every relation grouped by stratum, color-coded by reserved prefix, with a relation-detail inspector on the right.

Reading the Strata

Relations are grouped by stratum index. Lower strata are evaluated first.
Reserved-prefix accent colors call out atom, candidate., proposal., intent., observation..
Selecting a relation shows its stratum and prefix kind in the inspector.

Finding Gaps

A coverage overlay ribbon appears whenever a fixture lens is active in Activity. The ribbon highlights which relations are exercised by the active fixture, helping you see uncovered surface area.

Understanding Derivation Depth

The strata browser implicitly shows derivation depth. Higher-stratum relations sit on top of lower-stratum ones:

atom → booking_request → confirmation_pending → (retracted)
atom → slot_hold_active → booking_confirmed → booking_terminal → booking_status

Deep derivation chains are more likely to have subtle bugs. The drill inspector traces the full chain in text; the V1.1 visual graph will let you see the chain’s reach at a glance.

Worked Example: Why Did the Confirmation Intent Not Fire?

Let’s put it all together with a complete debugging session.

Problem: After replaying the happy-path fixture, intent.send_confirmation is not derived. The booking gets stuck at “reserved” and never advances to “confirmed.”

1. Check the Activity Surface

$ jacqos replay fixtures/happy-path.jsonl
$ jacqos studio

In Activity, the Done tab shows booking_confirmed, but intent.send_confirmation never lands as an action receipt — there’s no row for it in any tab.

2. Find the Closest Row and Read the Verify Output

jacqos verify reports which intent failed to fire and which rule body clause blocked it:

intent.send_confirmation("req-1", "pat@example.com", "slot-42") — not derived

rule: intent.send_confirmation (intents.dh:8)
  ✓ booking_request("req-1", "pat@example.com", "slot-42") — exists
  ✓ slot_hold_active("req-1", "slot-42") — exists
  ✗ NOT confirmation_pending("req-1", "pat@example.com", "slot-42") — BLOCKED
    confirmation_pending("req-1", "pat@example.com", "slot-42") exists

The intent rule checks not confirmation_pending(...) — it only fires if confirmation is not already pending. But confirmation_pending is asserted before the intent can fire.

3. Understand the Rule Logic

Open the Ontology destination and inspect confirmation_pending and intent.send_confirmation. They share a stratum, so the pending fact blocks the intent. In your .dh source, the rules read:

rule confirmation_pending(req, email, slot) :-
  booking_request(req, email, slot),
  slot_hold_active(req, slot).

rule intent.send_confirmation(req, email, slot) :-
  booking_request(req, email, slot),
  slot_hold_active(req, slot),
  not confirmation_pending(req, email, slot).

The intent fires when a booking is ready and confirmation isn’t already pending. But the rules assert confirmation_pending from the same conditions that should trigger the intent. Same stratum, so the pending fact blocks the intent. The V1.1 visual rule graph will surface this collision as a negation edge between same-stratum siblings.

4. Identify the Fix

The AI needs to restructure: either the intent should fire without the negation check (and confirmation_pending becomes a tracking fact rather than a guard), or the intent and the pending fact should be in different strata.

5. Add a Fixture Expectation

Add the intent expectation to the happy-path fixture so this regression is caught:

{"expect_intent":"intent.send_confirmation","args":["req-1","pat@example.com","slot-42"]}

Run jacqos verify, have the AI fix the rules, verify again. The fixture locks in the correct behavior.

Debugging Checklist

When something goes wrong, follow this sequence:

Identify the symptom: unexpected fact, missing fact, or invariant violation
Open Studio: replay the relevant fixture and launch jacqos studio
Select the closest Activity row: open its drill inspector, or read jacqos verify output for missing facts
Read the provenance chain: walk the inspector’s Decision, Facts, and Atoms / Observations sections
Check the verification bundle for negation witnesses (V1) or read them inline in the drill inspector (V1.1)
Cross-reference the Ontology surface: confirm stratum and prefix kind for any rule you’re investigating
Check the Timeline section: verify the raw observation payload matches expectations
Fix via invariant or fixture: encode the correct behavior, let the AI regenerate rules
Verify: run jacqos verify to confirm the fix and lock it in with a digest

Next Steps

Debug, Verify, Ship — the rung-8 workflow page that walks one verify failure end-to-end through every CLI command and every Studio view
Visual Provenance — concept deep-dive on the V1 drill inspector and what V1.1 adds
Fixtures and Invariants — defining verification surfaces
Invariant Review — why invariants replace code review
.dh Language Reference — rule syntax, negation, and stratification
CLI Reference — jacqos replay and jacqos verify commands
Lineages and Worldviews — comparing evaluator outputs