Compare commits

...
Author SHA1 Message Date
rsp2k 433a003d09 Point the docs links at the public site
Tests / test (==2.12.2) (push) Canceled after 0s
Tests / test (==2.14.3) (push) Canceled after 0s
Tests / test (>=3.3,<4) (push) Canceled after 0s
Tests / test (>=4,<5) (push) Canceled after 0s
Tests / test-postgres (==2.12.2) (push) Canceled after 0s
Tests / test-postgres (>=4,<5) (push) Canceled after 0s
Tests / canary (push) Canceled after 0s
fastmcp-feedback.supported.systems replaces the private dev address in the
README and the PyPI Documentation URL. Version 2026.10.01.3.
2026-10-01 22:13:34 -06:00
rsp2k e4da62d0df Merge fix/lazy-storage-import: instrumentation imports without the database extras
Tests / test (==2.12.2) (push) Canceled after 0s
Tests / test (==2.14.3) (push) Canceled after 0s
Tests / test (>=3.3,<4) (push) Canceled after 0s
Tests / test (>=4,<5) (push) Canceled after 0s
Tests / test-postgres (==2.12.2) (push) Canceled after 0s
Tests / test-postgres (>=4,<5) (push) Canceled after 0s
Tests / canary (push) Canceled after 0s
2026-10-01 20:56:05 -06:00
rsp2k 5e9a9ca037 Release 2026.10.01.2: instrumentation imports without the database extras
Tests / test (==2.12.2) (push) Canceled after 0s
Tests / test (==2.14.3) (push) Canceled after 0s
Tests / test (>=3.3,<4) (push) Canceled after 0s
Tests / test (>=4,<5) (push) Canceled after 0s
Tests / test-postgres (==2.12.2) (push) Canceled after 0s
Tests / test-postgres (>=4,<5) (push) Canceled after 0s
Tests / canary (push) Canceled after 0s
2026-10-01 20:56:02 -06:00
rsp2k 76999c942c Model SQLAlchemy 2.0 and 2.1 in the base-install test 2026-10-01 20:53:34 -06:00
rsp2k 4995143f8f Import instrumentation without the database extras
storage, embeddings and backfill need sqlalchemy[asyncio] (greenlet). The
package __init__ now loads their names on first access and raises an
ImportError naming the extra when they are missing. The middleware finds a
DatabaseSink or EmbeddingSink through sys.modules instead of importing them.
Table prefix, embedding dim, sources and event text keys move to constants.py
so they import on a base install.
2026-10-01 20:52:33 -06:00
rsp2k 28816477ff Merge fix/explicit-success-wins: an explicit success flag beats error keys
Tests / test (==2.12.2) (push) Canceled after 0s
Tests / test (==2.14.3) (push) Canceled after 0s
Tests / test (>=3.3,<4) (push) Canceled after 0s
Tests / test (>=4,<5) (push) Canceled after 0s
Tests / test-postgres (==2.12.2) (push) Canceled after 0s
Tests / test-postgres (>=4,<5) (push) Canceled after 0s
Tests / canary (push) Canceled after 0s
2026-10-01 18:05:52 -06:00
rsp2k c81dc3bf8c Release 2026.10.01.1: an explicit success flag beats error keys
Tests / test (==2.12.2) (push) Canceled after 0s
Tests / canary (push) Canceled after 0s
Tests / test (==2.14.3) (push) Canceled after 0s
Tests / test (>=3.3,<4) (push) Canceled after 0s
Tests / test (>=4,<5) (push) Canceled after 0s
Tests / test-postgres (==2.12.2) (push) Canceled after 0s
Tests / test-postgres (>=4,<5) (push) Canceled after 0s
2026-10-01 18:05:50 -06:00
rsp2k 4922eaecbd Let an explicit success flag win over error keys
A result with ok or success exactly True skips the error-key check, so
{"ok": True, "error": "deprecated arg"} is ok and the string reads as a
warning. A failing status, a False flag and isError still flag it, and
truthy non-bools are not an explicit success.
2026-10-01 18:04:15 -06:00
rsp2k 36fed0267c Move the dev Postgres port to 55447
Tests / test (==2.12.2) (push) Canceled after 0s
Tests / test (==2.14.3) (push) Canceled after 0s
Tests / test (>=3.3,<4) (push) Canceled after 0s
Tests / test (>=4,<5) (push) Canceled after 0s
Tests / test-postgres (==2.12.2) (push) Canceled after 0s
Tests / test-postgres (>=4,<5) (push) Canceled after 0s
Tests / canary (push) Canceled after 0s
55432 and 55433 are taken by other projects' containers on the dev host,
so make pg-up failed. The test database URL carries the port too.
2026-10-01 18:02:49 -06:00
rsp2k ae52e7a0f9 Merge feature/error-key-classifier: treat error-string results as soft errors
Tests / test (==2.12.2) (push) Canceled after 0s
Tests / test (==2.14.3) (push) Canceled after 0s
Tests / test (>=3.3,<4) (push) Canceled after 0s
Tests / test (>=4,<5) (push) Canceled after 0s
Tests / test-postgres (==2.12.2) (push) Canceled after 0s
Tests / test-postgres (>=4,<5) (push) Canceled after 0s
Tests / canary (push) Canceled after 0s
2026-10-01 16:44:37 -06:00
17 changed files with 501 additions and 55 deletions
+2 -2
View File
@@ -6,8 +6,8 @@ FFB_PG_USER=ffb
FFB_PG_PASSWORD=change-me
FFB_PG_DB=ffb_test
# Published on 127.0.0.1 only. 55432 is commonly taken by other local stacks.
FFB_PG_PORT=55433
FFB_PG_PORT=55447
# Read by the test suite. Unset, the Postgres test cases skip.
# Each test gets its own database created on this server and dropped after.
FFB_TEST_DATABASE_URL=postgresql+asyncpg://ffb:change-me@127.0.0.1:55433/ffb_test
FFB_TEST_DATABASE_URL=postgresql+asyncpg://ffb:change-me@127.0.0.1:55447/ffb_test
+51
View File
@@ -5,6 +5,57 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Calendar Versioning](https://calver.org/) (YYYY.MM.DD).
## [2026.10.01.3] - 2026-10-01
### Documentation
- The documentation site is public at https://fastmcp-feedback.supported.systems,
rewritten against this package with every example executed. The README
links and the PyPI "Documentation" URL point there instead of the private
dev address.
## [2026.10.01.2] - 2026-10-01
The instrumentation imports on a base install, without the database extras.
### Fixed
- `import fastmcp_feedback.instrumentation` failed on `pip install
fastmcp-feedback` without the `instrumentation` extra: the package imported
`storage.py` eagerly, which imports `sqlalchemy.ext.asyncio`, and SQLAlchemy
2.1 raises `ImportError` there when `greenlet` is missing. `instrument(app)`
with `JsonLinesSink`, `CallbackSink` or `MemorySink`, events, redaction and
result classification now need only the base dependencies.
- `DatabaseSink`, `EmbeddingVector`, `build_metadata`, `Embedder`,
`EmbeddingSink`, `EmbeddingText`, `OpenAIEmbedder`, `extract_texts`,
`backfill_embeddings` and `backfill_feedback` load on first access. Without
the extra, accessing one raises `ImportError` naming the missing package and
the extra to install (`[instrumentation]`, or `[instrumentation,embeddings]`
for the embedding names), chained to the original error. Every access
raises, not only the first, although SQLAlchemy itself reports a missing
`greenlet` only once. Import paths are unchanged.
- The middleware finds a `DatabaseSink` or `EmbeddingSink` among its sinks
without importing their modules, so `database_sink`, `embedding_sink`,
`capture_recent_calls`, `link_feedback`, `feedback_context`, `events_for`,
`similar` and `similar_feedback` return their usual empty results on a base
install.
- `DEFAULT_TABLE_PREFIX`, `DEFAULT_EMBEDDING_DIM`, `EMBEDDING_SOURCES` and
`DEFAULT_EVENT_TEXT_KEYS` moved to `instrumentation/constants.py` and import
without the extras; `storage` and `embeddings` still export them.
## [2026.10.01.1] - 2026-10-01
A result that says it succeeded is no longer flagged for its own error string.
### Changed
- Explicit success flag wins over error keys: when any of the classifier's
`flag_keys` (`ok`, `success`) is exactly `True`, the `error_keys` check is
skipped and the string is treated as a warning, so
`{"ok": True, "error": "deprecated arg"}` is recorded as `ok` again. Only
that check is skipped: a failing status (`{"success": True, "status":
"failed"}`), any flag key exactly `False` and `isError` still flag the
result. Truthy values that are not `True`, such as `1` or `"yes"`, are not
an explicit success. Results with no flag key are classified as in
2026.10.01.
## [2026.10.01] - 2026-10-01
Results that report failure only as an `error` string now count as soft
+1 -1
View File
@@ -113,7 +113,7 @@ make pg-down # stop it; the data volume is kept (pg-reset deletes it)
```
The server listens on `127.0.0.1` only, on `FFB_PG_PORT` from `.env`
(55433 by default; change it if that port is taken). `make pg-psql` opens a
(55447 by default; change it if that port is taken). `make pg-psql` opens a
shell on it. Every Postgres test gets its own freshly created database with
the `vector` extension, dropped when the test ends, so tests never see each
other's rows. To write a test that runs on both backends, take the `db_url`
+50 -9
View File
@@ -30,6 +30,20 @@ pip install fastmcp-feedback
uv add "fastmcp-feedback[postgresql]"
```
What each install covers:
| Install | Adds | Covers |
|---------|------|--------|
| `fastmcp-feedback` | FastMCP, SQLAlchemy, Pydantic | The feedback tools (sync SQLAlchemy, no greenlet), and `instrument(app)` with `JsonLinesSink`, `CallbackSink` and `MemorySink`, events, redaction and result classification |
| `fastmcp-feedback[instrumentation]` | `sqlalchemy[asyncio]` (greenlet), `aiosqlite` | `DatabaseSink`, `build_metadata`, call history for linking feedback (`capture_recent_calls` across workers, `link_feedback`, `feedback_context`) |
| `fastmcp-feedback[instrumentation,embeddings]` | `pgvector` | `EmbeddingSink`, `OpenAIEmbedder`, `similar()`, the backfill helpers; pgvector is only needed on PostgreSQL |
Postgres hosts add `asyncpg` for the async sinks. Without the
`instrumentation` extra, `from fastmcp_feedback.instrumentation import
DatabaseSink` raises an `ImportError` that names the missing package and the
extra to install, and the middleware's database helpers return their empty
defaults (`[]`, `False`).
### From Git (Latest Development)
```bash
@@ -181,8 +195,7 @@ instrument(app, [
`DatabaseSink` needs the `instrumentation` extra
(`uv add "fastmcp-feedback[instrumentation]"`), plus `asyncpg` for Postgres.
Install the extra even if you already have SQLAlchemy: SQLAlchemy 2.1 no longer
pulls in `greenlet` by itself, and its async engine cannot run without it.
The other sinks work on the base install (see [Installation](#installation)).
### Using your own database and migrations
@@ -232,7 +245,30 @@ string with something other than whitespace counts, so `{"error": None}`,
`{"error": ""}` and a statistic such as `{"error": 0.02}` stay ok. A dict or
list there (`{"error": {"code": 1}}`) is not flagged either; catch that shape
with a flag key, a status or your own classifier. `error_keys` names the keys
to check, and `error_keys=()` turns the check off:
to check, and `error_keys=()` turns the check off.
A result that says it succeeded wins over its own error string: when `ok` or
`success` (the `flag_keys`) is exactly `True`, the `error` check is skipped and
the string is read as a warning, so `{"ok": True, "error": "deprecated arg"}`
is recorded as `ok`. Only that check is skipped. In order of precedence:
1. A failing `status` or `state` flags the result, even next to `ok: True`,
since `{"success": True, "status": "failed"}` contradicts itself and the
status is the more specific of the two.
2. Any flag key exactly `False` flags it, even if another is `True`.
3. `isError` flags it.
4. A flag key exactly `True` makes it ok. Truthy values that are not `True`,
such as `1` or `"yes"`, do not count.
5. A non-empty string at an `error_keys` key flags it.
```python
from fastmcp_feedback.instrumentation import DEFAULT_RESULT_CLASSIFIER as classify
assert classify("t", {"ok": True, "error": "deprecated arg"}) is None
assert classify("t", {"ok": 1, "error": "x"}) == "error: x"
assert classify("t", {"success": True, "status": "failed"}) == "status=failed"
assert classify("t", {"played": False, "error": "stalled"}) == "error: stalled"
```
```python
from fastmcp_feedback.instrumentation import instrument, soft_error_classifier
@@ -289,6 +325,10 @@ upgrading. Results that also have a failing status or flag keep their old
message, such as `success=False: ...`, and are counted once. Pass
`soft_error_classifier(error_keys=())` to keep the old behavior.
**Upgrading to 2026.10.01.1:** a result with `ok` or `success` exactly `True`
and an `error` string, which 2026.10.01 recorded as `soft_error`, is `ok` again.
Nothing else changes.
### Identity and enrichment hooks
```python
@@ -928,12 +968,13 @@ Add all five feedback tools to a FastMCP server.
## Documentation
Full documentation: https://fastmcp-feedback.l.supported.systems
Full documentation: https://fastmcp-feedback.supported.systems
- [Quick Start Tutorial](https://fastmcp-feedback.l.supported.systems/tutorials/quickstart/)
- [Architecture](https://fastmcp-feedback.l.supported.systems/explanation/architecture/)
- [API Reference](https://fastmcp-feedback.l.supported.systems/reference/api/)
- [Integration Patterns](https://fastmcp-feedback.l.supported.systems/how-to/integration-patterns/)
- [Instrument a server in five minutes](https://fastmcp-feedback.supported.systems/tutorials/instrument-a-server/)
- [Architecture](https://fastmcp-feedback.supported.systems/explanation/architecture/)
- [Middleware reference](https://fastmcp-feedback.supported.systems/reference/middleware/)
- [Feedback linked to the calls behind it](https://fastmcp-feedback.supported.systems/tutorials/feedback-with-context/)
- [Embeddings and similarity search](https://fastmcp-feedback.supported.systems/how-to/embeddings/)
## Examples
@@ -992,7 +1033,7 @@ MIT License - see [LICENSE](LICENSE) for details.
## Links
- **Documentation**: https://fastmcp-feedback.l.supported.systems
- **Documentation**: https://fastmcp-feedback.supported.systems
- **Repository**: https://git.supported.systems/fastmcp-feedback/fastmcp-feedback
- **Issues**: https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/issues
- **FastMCP**: https://gofastmcp.com
+1 -1
View File
@@ -11,7 +11,7 @@ services:
POSTGRES_PASSWORD: ${FFB_PG_PASSWORD}
POSTGRES_DB: ${FFB_PG_DB}
ports:
- "127.0.0.1:${FFB_PG_PORT:-55433}:5432"
- "127.0.0.1:${FFB_PG_PORT:-55447}:5432"
volumes:
- ffb-postgres-data:/var/lib/postgresql/data
- ./docker/postgres-init:/docker-entrypoint-initdb.d:ro
+2 -2
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "fastmcp-feedback"
version = "2026.10.01" # YYYYMMDD.patch versioning - update for each release
version = "2026.10.01.3" # YYYYMMDD.patch versioning - update for each release
description = "Production-ready feedback collection system for FastMCP servers"
readme = "README.md"
license = "MIT"
@@ -83,7 +83,7 @@ all = ["fastmcp-feedback[postgresql,mysql,dev]"]
[project.urls]
Homepage = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback"
Documentation = "https://fastmcp-feedback.l.supported.systems"
Documentation = "https://fastmcp-feedback.supported.systems"
Repository = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback.git"
Issues = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/issues"
Changelog = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/blob/main/CHANGELOG.md"
+1 -1
View File
@@ -44,7 +44,7 @@ Quick Start:
import logging
# Version information - CalVer YYYY.MM.DD
__version__ = "2026.10.01"
__version__ = "2026.10.01.3"
__author__ = "Ryan Malloy"
__email__ = "ryan@supported.systems"
@@ -13,21 +13,21 @@ await mw.similar("render timed out")
await backfill_embeddings(mw.embedding_sink) # rows stored before embedding
"""
from .backfill import backfill_embeddings, backfill_feedback
import importlib
from typing import TYPE_CHECKING, Any
from .classify import (
DEFAULT_RESULT_CLASSIFIER,
ResultClassifier,
soft_error_classifier,
)
from .dispatch import RecordDispatcher
from .embeddings import (
from .constants import (
DEFAULT_EMBEDDING_DIM,
DEFAULT_EVENT_TEXT_KEYS,
Embedder,
EmbeddingSink,
EmbeddingText,
OpenAIEmbedder,
extract_texts,
DEFAULT_TABLE_PREFIX,
EMBEDDING_SOURCES,
)
from .dispatch import RecordDispatcher
from .events import EventRecord
from .middleware import MODE_ENV_VAR, MODES, InstrumentationMiddleware, instrument
from .record import ToolCallRecord
@@ -38,14 +38,75 @@ from .redaction import (
Redactor,
)
from .sinks import CallbackSink, JsonLinesSink, MemorySink, Sink
from .storage import (
DEFAULT_EMBEDDING_DIM,
DEFAULT_TABLE_PREFIX,
EMBEDDING_SOURCES,
DatabaseSink,
EmbeddingVector,
build_metadata,
)
if TYPE_CHECKING:
from .backfill import backfill_embeddings, backfill_feedback
from .embeddings import (
Embedder,
EmbeddingSink,
EmbeddingText,
OpenAIEmbedder,
extract_texts,
)
from .storage import DatabaseSink, EmbeddingVector, build_metadata
# Names whose modules need the async database stack (sqlalchemy[asyncio],
# which brings greenlet). Imported on first access, so instrument() and the
# plain sinks work on a base install.
_LAZY = {
"DatabaseSink": "storage",
"EmbeddingVector": "storage",
"build_metadata": "storage",
"Embedder": "embeddings",
"EmbeddingSink": "embeddings",
"EmbeddingText": "embeddings",
"OpenAIEmbedder": "embeddings",
"extract_texts": "embeddings",
"backfill_embeddings": "backfill",
"backfill_feedback": "backfill",
}
_EXTRA = {
"storage": "instrumentation",
"embeddings": "instrumentation,embeddings",
"backfill": "instrumentation,embeddings",
}
def _missing_module(exc: BaseException) -> str:
seen: BaseException | None = exc
while seen is not None:
if isinstance(seen, ImportError) and seen.name:
return seen.name
seen = seen.__cause__ or seen.__context__
return "a dependency"
def __getattr__(name: str) -> Any:
module = _LAZY.get(name)
if module is None:
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
try:
# Checked first because SQLAlchemy reports a missing greenlet only on
# the first import of its asyncio module; a retry would then succeed
# and leave an engine that fails at runtime.
importlib.import_module("greenlet")
value = getattr(importlib.import_module(f".{module}", __name__), name)
except ImportError as exc:
extra = _EXTRA[module]
missing = _missing_module(exc)
raise ImportError(
f"fastmcp_feedback.instrumentation.{name} needs the async database "
f"stack, and {missing} is not installed. Install it with: "
f'uv add "fastmcp-feedback[{extra}]" '
f'(or pip install "fastmcp-feedback[{extra}]")'
) from exc
globals()[name] = value
return value
def __dir__() -> list[str]:
return sorted(set(globals()) | set(_LAZY))
__all__ = [
"CallbackSink",
@@ -75,11 +75,19 @@ def soft_error_classifier(
so ``{"error": None}`` and ``{"error": 0.02}`` stay ok. ``error_keys=()``
turns this off.
A result that says it succeeded, with any of ``flag_keys`` exactly True,
skips only that last check: ``{"ok": True, "error": "deprecated arg"}``
is ok, the string being a warning. A failing status, a False flag and
``isError`` still flag it. Truthy non-bools such as ``1`` or ``"yes"``
are not an explicit success.
>>> classify = soft_error_classifier(extra_statuses=("unknown_target",))
>>> classify("render", {"status": "unknown_target", "error": "no such object"})
'status=unknown_target: no such object'
>>> classify("speak", {"played": False, "error": "output stalled"})
'error: output stalled'
>>> print(classify("speak", {"ok": True, "error": "deprecated arg"}))
None
"""
bad = frozenset(s.lower() for s in (*statuses, *extra_statuses))
status_keys = tuple(status_keys)
@@ -100,6 +108,8 @@ def soft_error_classifier(
if result.get(key) is True:
detail = _message(result) or _content_text(result.get("content"))
return _describe("isError", detail)
if any(result.get(key) is True for key in flag_keys):
return None
for key in error_keys:
value = result.get(key)
if isinstance(value, str) and value.strip():
@@ -0,0 +1,12 @@
"""Defaults shared by the storage and embedding modules.
Kept apart from them so these import without the async database stack.
"""
DEFAULT_TABLE_PREFIX = "ffb_"
DEFAULT_EMBEDDING_DIM = 1024
# source_type values in {prefix}embeddings. Feedback rows survive pruning.
EMBEDDING_SOURCES = ("feedback", "call_error", "event", "llm")
FEEDBACK_SOURCE = "feedback"
# Attribute keys whose text an EmbeddingSink embeds from events.
DEFAULT_EVENT_TEXT_KEYS = ("error_excerpt", "error", "message", "reason")
@@ -20,16 +20,15 @@ from typing import Any, Protocol, runtime_checkable
from sqlalchemy import Float, and_, literal, not_, select, tuple_
from .constants import DEFAULT_EVENT_TEXT_KEYS, EMBEDDING_SOURCES
from .events import EventRecord
from .record import ToolCallRecord
from .redaction import Redactor
from .sinks import Record
from .storage import EMBEDDING_SOURCES, EmbeddingVector
from .storage import EmbeddingVector
logger = logging.getLogger(__name__)
DEFAULT_EVENT_TEXT_KEYS = ("error_excerpt", "error", "message", "reason")
@runtime_checkable
class Embedder(Protocol):
@@ -8,12 +8,13 @@ import json
import logging
import os
import random as _random
import sys
import time
import uuid
from collections import deque
from collections.abc import Awaitable, Callable, Iterable, Mapping, Sequence
from datetime import UTC, datetime, timedelta
from typing import Any
from typing import TYPE_CHECKING, Any
from fastmcp import FastMCP
from fastmcp.server.middleware import Middleware, MiddlewareContext
@@ -27,7 +28,6 @@ from .correlation import (
select_calls,
)
from .dispatch import RecordDispatcher
from .embeddings import EmbeddingSink
from .events import (
MAX_KEY_CHARS,
MAX_KIND_CHARS,
@@ -38,7 +38,10 @@ from .events import (
from .record import ToolCallRecord
from .redaction import Redactor
from .sinks import JsonLinesSink, Sink
from .storage import DatabaseSink
if TYPE_CHECKING:
from .embeddings import EmbeddingSink
from .storage import DatabaseSink
logger = logging.getLogger(__name__)
@@ -237,7 +240,7 @@ class InstrumentationMiddleware(Middleware):
sinks = list(sinks) if sinks is not None else [JsonLinesSink()]
for sink in sinks:
# Embedded text gets this middleware's redaction rules too.
if isinstance(sink, EmbeddingSink) and sink.redactor is None:
if _is_sink(sink, "embeddings", "EmbeddingSink") and sink.redactor is None:
sink.redactor = self.redactor
self.dispatcher = RecordDispatcher(
sinks,
@@ -573,11 +576,9 @@ class InstrumentationMiddleware(Middleware):
return rows[-limit:] if limit > 0 else []
@property
def embedding_sink(self) -> EmbeddingSink | None:
def embedding_sink(self) -> "EmbeddingSink | None":
"""The first EmbeddingSink among the sinks, if any."""
return next(
(s for s in self.dispatcher.sinks if isinstance(s, EmbeddingSink)), None
)
return _first_sink(self.dispatcher.sinks, "embeddings", "EmbeddingSink")
async def similar(
self,
@@ -644,11 +645,9 @@ class InstrumentationMiddleware(Middleware):
return []
@property
def database_sink(self) -> DatabaseSink | None:
def database_sink(self) -> "DatabaseSink | None":
"""The first DatabaseSink among the sinks, if any."""
return next(
(s for s in self.dispatcher.sinks if isinstance(s, DatabaseSink)), None
)
return _first_sink(self.dispatcher.sinks, "storage", "DatabaseSink")
async def capture_recent_calls(
self,
@@ -786,6 +785,21 @@ class InstrumentationMiddleware(Middleware):
await self.dispatcher.aclose()
def _is_sink(sink: Any, module: str, name: str) -> bool:
"""isinstance against a sink class without importing its module.
storage and embeddings need the async database stack. When the module was
never imported, no instance of its class can exist, so the answer is no.
"""
loaded = sys.modules.get(f"{__package__}.{module}")
cls = getattr(loaded, name, None) if loaded is not None else None
return cls is not None and isinstance(sink, cls)
def _first_sink(sinks: Iterable[Any], module: str, name: str) -> Any:
return next((s for s in sinks if _is_sink(s, module, name)), None)
def _current_context() -> Any:
try:
from fastmcp.server.dependencies import get_context
@@ -40,22 +40,21 @@ from sqlalchemy.exc import DBAPIError
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
from sqlalchemy.types import TypeDecorator
from .constants import (
DEFAULT_EMBEDDING_DIM,
DEFAULT_TABLE_PREFIX,
FEEDBACK_SOURCE,
)
from .constants import EMBEDDING_SOURCES as EMBEDDING_SOURCES # re-export
from .events import MAX_KEY_CHARS, MAX_KIND_CHARS, EventRecord
from .record import ToolCallRecord
from .sinks import Record
logger = logging.getLogger(__name__)
DEFAULT_TABLE_PREFIX = "ffb_"
# none_as_null: a missing payload is SQL NULL, so "WHERE args IS NULL" works.
_JSON = JSON(none_as_null=True).with_variant(JSONB(none_as_null=True), "postgresql")
DEFAULT_EMBEDDING_DIM = 1024
# source_type values in {prefix}embeddings. Feedback rows survive pruning.
EMBEDDING_SOURCES = ("feedback", "call_error", "event", "llm")
FEEDBACK_SOURCE = "feedback"
class EmbeddingVector(TypeDecorator):
"""pgvector ``VECTOR(dim)`` on PostgreSQL, a JSON list of floats elsewhere.
+105
View File
@@ -0,0 +1,105 @@
"""Exercise the instrumentation as a base install would, then print JSON.
Run in a fresh interpreter. greenlet and aiosqlite are made unimportable
before anything else loads, as on ``pip install fastmcp-feedback`` without
the ``instrumentation`` extra. Argument: a directory for the feedback database.
"""
import sys
from importlib.metadata import version
# SQLAlchemy 2.1 refuses to import sqlalchemy.ext.asyncio without greenlet,
# once: a second import succeeds. 2.0 imports it and fails on first use.
# On 2.0 the module is blocked too, so both versions fail at import.
_blocked = ["greenlet", "aiosqlite"]
if tuple(int(p) for p in version("sqlalchemy").split(".")[:2]) < (2, 1):
_blocked.append("sqlalchemy.ext.asyncio")
for _name in _blocked:
sys.modules[_name] = None # import raises ModuleNotFoundError
import asyncio # noqa: E402
import io # noqa: E402
import json # noqa: E402
from pathlib import Path # noqa: E402
STORAGE = "fastmcp_feedback.instrumentation.storage"
EMBEDDINGS = "fastmcp_feedback.instrumentation.embeddings"
async def run(db_dir: Path) -> dict:
from fastmcp import Client, FastMCP
import fastmcp_feedback.instrumentation as inst
from fastmcp_feedback import add_feedback_tools
from fastmcp_feedback.instrumentation import (
DEFAULT_TABLE_PREFIX,
JsonLinesSink,
MemorySink,
instrument,
)
out: dict = {"prefix": DEFAULT_TABLE_PREFIX}
app = FastMCP("base-install")
@app.tool
def add(a: int, b: int) -> int:
return a + b
memory = MemorySink()
lines = io.StringIO()
mw = instrument(app, [memory, JsonLinesSink(lines)], mode="full")
add_feedback_tools(
app, database_url=f"sqlite:///{db_dir / 'fb.db'}", instrumentation=mw
)
async with Client(app) as client:
result = await client.call_tool("add", {"a": 2, "b": 3})
out["result"] = result.content[0].text
fb = await client.call_tool(
"submit_feedback",
{
"request": {
"type": "bug",
"title": "base install",
"description": "submitted without the extra",
"submitter": "worker",
}
},
)
out["feedback"] = json.loads(fb.content[0].text)
mw.record_event("job.done", key="j1", attrs={"n": 1})
await mw.flush()
out["tools"] = [r.tool for r in memory.records if hasattr(r, "tool")]
out["event_kinds"] = [r.kind for r in memory.records if hasattr(r, "kind")]
out["jsonl_lines"] = len(lines.getvalue().splitlines())
out["capture"] = await mw.capture_recent_calls()
out["similar"] = await mw.similar("anything")
out["similar_feedback"] = await mw.similar_feedback("bug-1")
out["feedback_context"] = await mw.feedback_context("bug-1")
out["link_feedback"] = await mw.link_feedback("bug-1", [])
out["events_for"] = len(await mw.events_for("j1"))
out["database_sink"] = mw.database_sink is None
out["embedding_sink"] = mw.embedding_sink is None
await mw.aclose()
errors = {}
for name in (
"DatabaseSink",
"EmbeddingSink",
"build_metadata",
"backfill_feedback",
):
for _attempt in range(2): # SQLAlchemy only complains the first time
try:
getattr(inst, name)
errors.setdefault(name, []).append(None)
except ImportError as exc:
errors.setdefault(name, []).append(str(exc))
out["errors"] = errors
out["loaded"] = sorted(m for m in (STORAGE, EMBEDDINGS) if m in sys.modules)
return out
if __name__ == "__main__":
print(json.dumps(asyncio.run(run(Path(sys.argv[1]))), default=str))
+90
View File
@@ -0,0 +1,90 @@
"""The instrumentation works on a base install, without the database extras.
The worker runs in a fresh interpreter with greenlet and aiosqlite made
unimportable, which is what ``pip install fastmcp-feedback`` leaves out.
"""
import json
import subprocess
import sys
from pathlib import Path
import pytest
WORKER = Path(__file__).parent.parent / "fixtures" / "base_install_worker.py"
@pytest.fixture(scope="module")
def report(tmp_path_factory) -> dict:
proc = subprocess.run(
[sys.executable, str(WORKER), str(tmp_path_factory.mktemp("base"))],
capture_output=True,
text=True,
timeout=60,
)
assert proc.returncode == 0, proc.stderr[-4000:]
return json.loads(proc.stdout.strip().splitlines()[-1])
def test_tool_calls_are_recorded(report):
assert report["result"] == "5"
assert report["tools"] == ["add", "submit_feedback"]
assert report["jsonl_lines"] == 4 # two calls, two events
def test_events_are_recorded(report):
assert report["event_kinds"] == ["feedback.submitted", "job.done"]
assert report["events_for"] == 1
def test_feedback_tools_work_alongside(report):
assert report["feedback"]["success"] is True
assert report["feedback"]["linked_calls"] == 0
def test_database_helpers_return_empty_defaults(report):
assert report["capture"] == []
assert report["similar"] == []
assert report["similar_feedback"] == []
assert report["feedback_context"] == []
assert report["link_feedback"] is False
assert report["database_sink"] is True
assert report["embedding_sink"] is True
def test_constants_import_without_the_extra(report):
assert report["prefix"] == "ffb_"
def test_storage_modules_stay_unloaded(report):
assert report["loaded"] == []
@pytest.mark.parametrize(
("name", "extra"),
[
("DatabaseSink", "[instrumentation]"),
("build_metadata", "[instrumentation]"),
("EmbeddingSink", "[instrumentation,embeddings]"),
("backfill_feedback", "[instrumentation,embeddings]"),
],
)
def test_database_names_explain_the_missing_extra(report, name, extra):
# Asked twice: SQLAlchemy reports a missing greenlet only on first import.
for message in report["errors"][name]:
assert message is not None
assert "greenlet" in message
assert f'uv add "fastmcp-feedback{extra}"' in message
def test_lazy_names_resolve_with_the_extra():
import fastmcp_feedback.instrumentation as inst
from fastmcp_feedback.instrumentation import DatabaseSink, storage
assert DatabaseSink is storage.DatabaseSink
assert set(inst._LAZY) <= set(inst.__all__)
assert set(inst.__all__) <= set(dir(inst))
for name in inst.__all__:
assert getattr(inst, name) is not None
with pytest.raises(AttributeError):
_ = inst.not_a_name
+64
View File
@@ -201,6 +201,70 @@ class TestClassifier:
assert classify("t", {"ok": False, "error": "x"}) == "ok=False: x"
class TestExplicitSuccess:
"""A flag key that is exactly True turns an error string into a warning."""
@pytest.mark.parametrize(
"result",
[
{"ok": True, "error": "deprecated arg"},
{"success": True, "error": "x"},
{"ok": True, "status": "unknown_target", "error": "fell back"},
],
)
def test_true_flag_with_error_string_is_ok(self, result):
assert DEFAULT_RESULT_CLASSIFIER("t", result) is None
@pytest.mark.parametrize(
"result, expected",
[
({"ok": False, "error": "x"}, "ok=False: x"),
# Not exactly True, so not an explicit success.
({"ok": 1, "error": "x"}, "error: x"),
({"ok": "yes", "error": "x"}, "error: x"),
({"success": 1.0, "error": "x"}, "error: x"),
# No flag key: mcspeak's shape, as in 2026.10.01.
(
{"played": False, "error": "audio output stalled"},
"error: audio output stalled",
),
# A False flag beats a True one.
({"ok": True, "success": False, "error": "x"}, "success=False: x"),
# A failing status is more specific than the flag, so it still flags.
({"success": True, "status": "failed"}, "status=failed"),
({"ok": True, "state": "error", "error": "x"}, "state=error: x"),
({"ok": True, "isError": True, "content": ["boom"]}, "isError: boom"),
],
)
def test_still_flagged(self, result, expected):
assert DEFAULT_RESULT_CLASSIFIER("t", result) == expected
def test_custom_flag_keys(self):
classify = soft_error_classifier(flag_keys=("passed",))
assert classify("t", {"passed": True, "error": "slow"}) is None
assert classify("t", {"passed": False, "error": "x"}) == "passed=False: x"
# ok is not a flag key here, so it does not suppress the error check.
assert classify("t", {"ok": True, "error": "x"}) == "error: x"
def test_no_flag_keys(self):
classify = soft_error_classifier(flag_keys=())
assert classify("t", {"ok": True, "error": "x"}) == "error: x"
async def test_recorded_as_ok(self):
record = await record_for(
"respond", {"payload": {"ok": True, "error": "deprecated arg"}}
)
assert (record.outcome, record.ok, record.error_message) == ("ok", True, None)
async def test_truthy_flag_recorded_as_soft_error(self):
record = await record_for("respond", {"payload": {"ok": 1, "error": "x"}})
assert (record.outcome, record.ok, record.error_message) == (
"soft_error",
False,
"error: x",
)
class TestMiddleware:
async def test_status_failed_is_soft_error(self):
record = await record_for("render", {"status": "failed"})
Generated
+1 -1
View File
@@ -601,7 +601,7 @@ wheels = [
[[package]]
name = "fastmcp-feedback"
version = "2026.10.1"
version = "2026.10.1.3"
source = { editable = "." }
dependencies = [
{ name = "fastmcp" },