Release 2026.10.01: treat error-string results as soft errors

README: the error-string check, its default and the behavior change on
upgrade; event keys used for joining must be unique across restarts; the
SQLite and Postgres join from calls to events; ids are UUIDs, so order
by occurred_at.

CHANGELOG.md

@@ -5,6 +5,42 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Calendar Versioning](https://calver.org/) (YYYY.MM.DD).
 
+## [2026.10.01] - 2026-10-01
+
+Results that report failure only as an `error` string now count as soft
+errors, and the docs cover joining events to calls.
+
+### Changed
+- `soft_error_classifier(..., error_keys=("error",))`: a dict result whose
+  value at any of `error_keys` is a non-empty string (after stripping) is a
+  soft error, with message `<key>: <string>`, redacted and truncated as
+  before. Checked after the status, flag and `isError` checks, so results
+  those already flag keep their message and are counted once. None, empty or
+  blank strings, numbers, booleans, dicts and lists do not count, so
+  `{"error": None}` and `{"error": 0.02}` stay ok. `error_keys=()` turns it
+  off. `DEFAULT_RESULT_CLASSIFIER` has it on.
+- **Behavior change:** results such as `{"played": False, "error": "..."}`,
+  `{"error": "..."}` and `{"status": "unknown_target", "error": "..."}` were
+  recorded as `ok` and are now `soft_error`, so `NOT ok` and soft-error
+  counts can rise after upgrading. This includes the package's own
+  `list_feedback`, `get_feedback_statistics` and analytics tools, whose
+  failure results carry an `error` string and no flag key. Pass
+  `result_classifier=soft_error_classifier(error_keys=())` for the old
+  behavior.
+
+### Docs
+- `record_event` / `events_for`: a `key` used for joining must be unique for
+  at least the retention period and across restarts; a per-boot counter
+  collides. Use a UUID or a startup token plus a counter.
+- The join from a call to its events through an enricher-written `extra`
+  field, for SQLite (`json_extract(c.extra, '$.speech_id') = e.key`) and
+  Postgres (`c.extra->>'speech_id' = e.key`).
+- Schema notes: `ffb_tool_calls` has primary key `id` and no `call_id`
+  column; `ffb_events.call_id` holds a call's `id`. Ids are UUIDs, so order
+  events by `occurred_at`.
+- The soft-error section documents `error_keys`, its default and the
+  upgrade note.
+
 ## [2026.09.30.1] - 2026-09-30
 
 Embed what was stored before embedding was switched on, and embed list and

README.md

@@ -224,6 +224,26 @@ The default classifier flags a dict result when `ok` or `success` is exactly
 `False`, when `status` or `state` is `error`, `failed` or `failure` (any case),
 or when the tool marked the result as an MCP error (`isError`) without raising.
 The message comes from the first of `error`, `message`, `detail` and `reason`.
+
+Failing those, a result whose `error` holds a non-empty string is a soft
+error too, with that string as the message: `{"played": False, "error":
+"audio output stalled"}` is recorded as `error: audio output stalled`. Only a
+string with something other than whitespace counts, so `{"error": None}`,
+`{"error": ""}` and a statistic such as `{"error": 0.02}` stay ok. A dict or
+list there (`{"error": {"code": 1}}`) is not flagged either; catch that shape
+with a flag key, a status or your own classifier. `error_keys` names the keys
+to check, and `error_keys=()` turns the check off:
+
+```python
+from fastmcp_feedback.instrumentation import instrument, soft_error_classifier
+
+# Tools that report {"failure_reason": "..."} instead
+instrument(app, result_classifier=soft_error_classifier(error_keys=("failure_reason",)))
+
+# As before 2026.10.01: an error string alone is not a soft error
+instrument(app, result_classifier=soft_error_classifier(error_keys=()))
+```
+
 Add your own status values, or pass any `(tool, result) -> str | None`:
 
 ```python
@@ -260,6 +280,15 @@ table made earlier by `create_tables=True` needs
 tables are not altered. Until the column exists, database writes fail and
 those records are dropped (logged).
 
+**Upgrading to 2026.10.01:** the `error` key check is on by default, so a
+result that carries a non-empty `error` string with no failing status or flag
+(such as `{"status": "unknown_target", "error": "no object named Cube"}`, or
+`{"error": "..."}` alone) is now a soft error where earlier releases recorded
+it as `ok`. Counts of `NOT ok` and `outcome = 'soft_error'` can rise after
+upgrading. Results that also have a failing status or flag keep their old
+message, such as `success=False: ...`, and are counted once. Pass
+`soft_error_classifier(error_keys=())` to keep the old behavior.
+
 ### Identity and enrichment hooks
 
 ```python
@@ -443,7 +472,12 @@ background queue, the same drop-and-count policy when the queue is full.
 - `kind` is a dotted name (`job.submitted`, `llm.call`), a non-empty string of
   at most 128 characters.
 - `key` is what you join on, such as a job id. It is stored as a string, at
-  most 255 characters.
+  most 255 characters. A key used to join events to calls or to each other
+  must be unique for at least the retention period, **including across
+  restarts**. A per-boot counter (`speech-1`, `speech-2`, ...) starts over
+  on every restart, and the second boot's `speech-1` joins to the first
+  boot's events. Use a UUID, or a token chosen at startup plus a counter
+  (`speech-<boot token>-<n>`).
 - `attrs` is any JSON-like dict. It is redacted like call arguments, then made
   JSON-safe: `NaN` and infinities become the strings `"nan"`/`"inf"` (Postgres
   JSONB rejects them), datetimes become ISO strings, anything else unknown
@@ -468,6 +502,23 @@ for ev in await mw.events_for("job-wing"):
 steps = await mw.events_for("job-wing", kinds=["step.started", "step.finished"], limit=100)
 ```
 
+To join a call to its events in SQL, have the enricher write the key into
+`extra` and match on it. This is how mcspeak reads the phases of one `speak`
+call, with SQLite:
+
+```sql
+SELECT e.occurred_at, e.attrs FROM ffb_events e
+JOIN ffb_tool_calls c ON json_extract(c.extra, '$.speech_id') = e.key
+WHERE e.kind = 'speech.phase' AND c.id = ?
+ORDER BY e.occurred_at;
+```
+
+On Postgres the join condition is `c.extra->>'speech_id' = e.key`. Events
+recorded inside the call also carry its id, so `e.call_id = c.id` joins those
+without a key. `ffb_tool_calls` has no `call_id` column; its primary key is
+`id`, and `ffb_events.call_id` holds that value. Order events by
+`occurred_at`: ids are random UUIDs and do not sort in recording order.
+
 `events_for` reads the `DatabaseSink` and adds events still waiting in the
 queue, so an event is visible as soon as `record_event` returns. With more
 than `limit` (1000) matches it returns the newest `limit`. Without a
@@ -533,8 +584,10 @@ With the default `capture_llm_text=False` both are discarded, and so are
 Events go to a `ffb_events` table: `id`, `occurred_at`, `kind`, `key`,
 `call_id`, `session_id`, `user_sub`, `caller_kind`, `client_id`,
 `server_version`, `attrs` (JSON; JSONB on Postgres), indexed on
-`occurred_at`, `kind`, `key`, `call_id` and `user_sub`. It has no foreign key
-to `ffb_tool_calls`, since calls are pruned and may be written later.
+`occurred_at`, `kind`, `key`, `call_id` and `user_sub`. `id` is a UUID
+string in both tables, so sort by `occurred_at` (events) or `started_at`
+(calls), never by `id`. `call_id` refers to `ffb_tool_calls.id` but has no
+foreign key, since calls are pruned and may be written later.
 
 Sinks get calls and events in the same batches. `DatabaseSink` writes each to
 its table, calls first in their own transaction, so a database without

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "fastmcp-feedback"
-version = "2026.09.30.1"  # YYYYMMDD.patch versioning - update for each release
+version = "2026.10.01"  # YYYYMMDD.patch versioning - update for each release
 description = "Production-ready feedback collection system for FastMCP servers"
 readme = "README.md"
 license = "MIT"

src/fastmcp_feedback/__init__.py

@@ -44,7 +44,7 @@ Quick Start:
 import logging
 
 # Version information - CalVer YYYY.MM.DD
-__version__ = "2026.09.30.1"
+__version__ = "2026.10.01"
 __author__ = "Ryan Malloy"
 __email__ = "ryan@supported.systems"
 

src/fastmcp_feedback/instrumentation/classify.py

@@ -58,6 +58,7 @@ def soft_error_classifier(
     status_keys: Iterable[str] = ("status", "state"),
     flag_keys: Iterable[str] = ("ok", "success"),
     extra_statuses: Iterable[str] = (),
+    error_keys: Iterable[str] = ("error",),
 ) -> ResultClassifier:
     """Build a classifier for dict results that describe their own failure.
 
@@ -67,13 +68,23 @@ def soft_error_classifier(
     ``isError``. The message is the first of ``error``, ``message``,
     ``detail`` and ``reason`` present, e.g. ``status=failed: disk full``.
 
+    Failing those, it is a soft error when any of ``error_keys`` holds a
+    non-empty string, as in ``{"played": False, "error": "output stalled"}``,
+    with that string as the message (``error: output stalled``). None, empty
+    or blank strings, numbers, booleans, dicts and lists there do not count,
+    so ``{"error": None}`` and ``{"error": 0.02}`` stay ok. ``error_keys=()``
+    turns this off.
+
     >>> classify = soft_error_classifier(extra_statuses=("unknown_target",))
     >>> classify("render", {"status": "unknown_target", "error": "no such object"})
     'status=unknown_target: no such object'
+    >>> classify("speak", {"played": False, "error": "output stalled"})
+    'error: output stalled'
     """
     bad = frozenset(s.lower() for s in (*statuses, *extra_statuses))
     status_keys = tuple(status_keys)
     flag_keys = tuple(flag_keys)
+    error_keys = tuple(error_keys)
 
     def classify(tool: str, result: Any) -> str | None:
         if not isinstance(result, dict):
@@ -89,6 +100,10 @@ def soft_error_classifier(
             if result.get(key) is True:
                 detail = _message(result) or _content_text(result.get("content"))
                 return _describe("isError", detail)
+        for key in error_keys:
+            value = result.get(key)
+            if isinstance(value, str) and value.strip():
+                return _describe(key, value.strip())
         return None
 
     return classify

src/fastmcp_feedback/instrumentation/middleware.py

@@ -430,6 +430,9 @@ class InstrumentationMiddleware(Middleware):
         and session; ``call_id`` given here wins. ``attrs`` is redacted like
         call arguments and made JSON-safe. ``kind`` must be a non-empty string
         of at most 128 characters and ``key`` at most 255 once made a string.
+        A ``key`` used for joining must stay unique for the retention period
+        and across restarts, so not a per-boot counter: use a UUID or a
+        startup token plus a counter.
 
         Returns False when the event was not recorded: mode ``off``, an invalid
         ``kind``, ``key`` or ``attrs``, a full queue, or no event loop to hand
@@ -548,6 +551,7 @@ class InstrumentationMiddleware(Middleware):
         memory, so an event is visible as soon as ``record_event`` returns (on
         the loop thread). Without a DatabaseSink, memory only. ``occurred_at``
         is a UTC datetime. A database failure is logged and memory is used.
+        Events are ordered by ``occurred_at``; their ids are random UUIDs.
         """
         key = str(key)
         kinds = None if kinds is None else frozenset(kinds)

tests/unit/test_retention_sampling.py

@@ -57,7 +57,7 @@ def make_app(sinks=None, **kwargs):
 
     @app.tool
     def render(status: str) -> dict:
-        return {"status": status, "error": f"render said {status}"}
+        return {"status": status, "detail": f"render said {status}"}
 
     @app.tool
     def crash() -> dict:

tests/unit/test_soft_errors.py

@@ -59,7 +59,7 @@ def make_app(sinks=None, **kwargs):
 
     @app.tool
     def render(status: str) -> dict:
-        return {"status": status, "error": f"render said {status}"}
+        return {"status": status, "detail": f"render said {status}"}
 
     @app.tool
     def respond(payload: dict) -> dict:
@@ -119,6 +119,19 @@ class TestClassifier:
                 {"is_error": True, "content": [{"type": "text", "text": "a"}]},
                 "isError: a",
             ),
+            # mcspeak's failures: an error string and no status or flag key.
+            (
+                {"played": False, "error": "audio output stalled after 30s"},
+                "error: audio output stalled after 30s",
+            ),
+            (
+                {"error": "Not listening: microphone stopped"},
+                "error: Not listening: microphone stopped",
+            ),
+            ({"error": "  padded  "}, "error: padded"),
+            ({"status": "unhealthy", "error": "db gone"}, "error: db gone"),
+            # Earlier checks still win, with their own message.
+            ({"success": False, "error": "x"}, "success=False: x"),
         ],
     )
     def test_flagged(self, result, expected):
@@ -135,6 +148,16 @@ class TestClassifier:
             {"result": False},
             {"state": 3},
             {"isError": False},
+            {"played": True, "error": None},
+            {"error": ""},
+            {"error": "   "},
+            {"error": 0.02},
+            {"error": 0},
+            {"error": False},
+            # A structured error is left to custom classifiers.
+            {"error": {"code": 1}},
+            {"error": ["a"]},
+            {"failure_reason": "x"},
             [{"status": "failed"}],
             "failed",
             None,
@@ -159,6 +182,24 @@ class TestClassifier:
         assert classify("t", {"passed": False}) == "passed=False"
         assert classify("t", {"status": "failed", "ok": False}) is None
 
+    def test_unknown_status_with_error_string(self):
+        """blender-mcp's shape: flagged by default, labelled by status when known."""
+        result = {"status": "unknown_target", "error": "no object named Cube"}
+        assert DEFAULT_RESULT_CLASSIFIER("t", result) == "error: no object named Cube"
+        classify = soft_error_classifier(extra_statuses=("unknown_target",))
+        assert classify("t", result) == "status=unknown_target: no object named Cube"
+
+    def test_custom_error_keys(self):
+        classify = soft_error_classifier(error_keys=("failure_reason",))
+        assert classify("t", {"failure_reason": "no gpu"}) == "failure_reason: no gpu"
+        assert classify("t", {"error": "ignored"}) is None
+        assert classify("t", {"failure_reason": ""}) is None
+
+    def test_empty_error_keys_disable(self):
+        classify = soft_error_classifier(error_keys=())
+        assert classify("t", {"played": False, "error": "stalled"}) is None
+        assert classify("t", {"ok": False, "error": "x"}) == "ok=False: x"
+
 
 class TestMiddleware:
     async def test_status_failed_is_soft_error(self):
@@ -186,6 +227,33 @@ class TestMiddleware:
         assert REDACTED in record.error_message
         assert len(record.error_message) == 60
 
+    async def test_error_string_is_soft_error(self):
+        record = await record_for(
+            "respond",
+            {"payload": {"played": False, "error": f"stalled, key {BMCP_TOKEN}"}},
+        )
+        assert (record.outcome, record.ok, record.error_type) == (
+            "soft_error",
+            False,
+            "SoftError",
+        )
+        assert record.error_message == f"error: stalled, key {REDACTED}"
+
+    @pytest.mark.parametrize("value", [None, "", 0.02])
+    async def test_empty_or_numeric_error_stays_ok(self, value):
+        record = await record_for(
+            "respond", {"payload": {"played": True, "error": value}}
+        )
+        assert (record.outcome, record.ok, record.error_message) == ("ok", True, None)
+
+    async def test_error_keys_disabled(self):
+        record = await record_for(
+            "respond",
+            {"payload": {"error": "stalled"}},
+            result_classifier=soft_error_classifier(error_keys=()),
+        )
+        assert (record.outcome, record.ok) == ("ok", True)
+
     async def test_extra_statuses(self):
         record = await record_for(
             "render",
@@ -393,3 +461,27 @@ class TestStorageAndCorrelation:
         assert record.error_message == (
             f"success=False: database down, password={REDACTED}"
         )
+
+    async def test_list_feedback_failure_is_soft_error(self, monkeypatch):
+        """list_feedback reports failure as {"feedback": [], "error": ...}."""
+        app = FastMCP("fb")
+        sink = MemorySink()
+        mw = instrument(app, [sink])
+        add_feedback_tools(app, database_url="sqlite:///:memory:", instrumentation=mw)
+
+        def broken(*args, **kwargs):
+            raise RuntimeError("database down")
+
+        from fastmcp_feedback.database import FeedbackDatabase
+
+        monkeypatch.setattr(FeedbackDatabase, "session", broken)
+        async with Client(app) as client:
+            result = await client.call_tool("list_feedback", {})
+        await mw.flush()
+        assert result.data["error"] == "database down"
+        [record] = sink.records
+        assert (record.tool, record.outcome, record.error_message) == (
+            "list_feedback",
+            "soft_error",
+            "error: database down",
+        )

uv.lock

@@ -601,7 +601,7 @@ wheels = [
 
 [[package]]
 name = "fastmcp-feedback"
-version = "2026.9.30.1"
+version = "2026.10.1"
 source = { editable = "." }
 dependencies = [
     { name = "fastmcp" },