Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
433a003d09 | ||
|
|
e4da62d0df | ||
|
|
5e9a9ca037 | ||
|
|
76999c942c | ||
|
|
4995143f8f | ||
|
|
28816477ff | ||
|
|
c81dc3bf8c | ||
|
|
4922eaecbd | ||
|
|
36fed0267c | ||
|
|
ae52e7a0f9 |
+2
-2
@@ -6,8 +6,8 @@ FFB_PG_USER=ffb
|
||||
FFB_PG_PASSWORD=change-me
|
||||
FFB_PG_DB=ffb_test
|
||||
# Published on 127.0.0.1 only. 55432 is commonly taken by other local stacks.
|
||||
FFB_PG_PORT=55433
|
||||
FFB_PG_PORT=55447
|
||||
|
||||
# Read by the test suite. Unset, the Postgres test cases skip.
|
||||
# Each test gets its own database created on this server and dropped after.
|
||||
FFB_TEST_DATABASE_URL=postgresql+asyncpg://ffb:change-me@127.0.0.1:55433/ffb_test
|
||||
FFB_TEST_DATABASE_URL=postgresql+asyncpg://ffb:change-me@127.0.0.1:55447/ffb_test
|
||||
|
||||
@@ -5,6 +5,57 @@ All notable changes to this project will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Calendar Versioning](https://calver.org/) (YYYY.MM.DD).
|
||||
|
||||
## [2026.10.01.3] - 2026-10-01
|
||||
|
||||
### Documentation
|
||||
- The documentation site is public at https://fastmcp-feedback.supported.systems,
|
||||
rewritten against this package with every example executed. The README
|
||||
links and the PyPI "Documentation" URL point there instead of the private
|
||||
dev address.
|
||||
|
||||
## [2026.10.01.2] - 2026-10-01
|
||||
|
||||
The instrumentation imports on a base install, without the database extras.
|
||||
|
||||
### Fixed
|
||||
- `import fastmcp_feedback.instrumentation` failed on `pip install
|
||||
fastmcp-feedback` without the `instrumentation` extra: the package imported
|
||||
`storage.py` eagerly, which imports `sqlalchemy.ext.asyncio`, and SQLAlchemy
|
||||
2.1 raises `ImportError` there when `greenlet` is missing. `instrument(app)`
|
||||
with `JsonLinesSink`, `CallbackSink` or `MemorySink`, events, redaction and
|
||||
result classification now need only the base dependencies.
|
||||
- `DatabaseSink`, `EmbeddingVector`, `build_metadata`, `Embedder`,
|
||||
`EmbeddingSink`, `EmbeddingText`, `OpenAIEmbedder`, `extract_texts`,
|
||||
`backfill_embeddings` and `backfill_feedback` load on first access. Without
|
||||
the extra, accessing one raises `ImportError` naming the missing package and
|
||||
the extra to install (`[instrumentation]`, or `[instrumentation,embeddings]`
|
||||
for the embedding names), chained to the original error. Every access
|
||||
raises, not only the first, although SQLAlchemy itself reports a missing
|
||||
`greenlet` only once. Import paths are unchanged.
|
||||
- The middleware finds a `DatabaseSink` or `EmbeddingSink` among its sinks
|
||||
without importing their modules, so `database_sink`, `embedding_sink`,
|
||||
`capture_recent_calls`, `link_feedback`, `feedback_context`, `events_for`,
|
||||
`similar` and `similar_feedback` return their usual empty results on a base
|
||||
install.
|
||||
- `DEFAULT_TABLE_PREFIX`, `DEFAULT_EMBEDDING_DIM`, `EMBEDDING_SOURCES` and
|
||||
`DEFAULT_EVENT_TEXT_KEYS` moved to `instrumentation/constants.py` and import
|
||||
without the extras; `storage` and `embeddings` still export them.
|
||||
|
||||
## [2026.10.01.1] - 2026-10-01
|
||||
|
||||
A result that says it succeeded is no longer flagged for its own error string.
|
||||
|
||||
### Changed
|
||||
- Explicit success flag wins over error keys: when any of the classifier's
|
||||
`flag_keys` (`ok`, `success`) is exactly `True`, the `error_keys` check is
|
||||
skipped and the string is treated as a warning, so
|
||||
`{"ok": True, "error": "deprecated arg"}` is recorded as `ok` again. Only
|
||||
that check is skipped: a failing status (`{"success": True, "status":
|
||||
"failed"}`), any flag key exactly `False` and `isError` still flag the
|
||||
result. Truthy values that are not `True`, such as `1` or `"yes"`, are not
|
||||
an explicit success. Results with no flag key are classified as in
|
||||
2026.10.01.
|
||||
|
||||
## [2026.10.01] - 2026-10-01
|
||||
|
||||
Results that report failure only as an `error` string now count as soft
|
||||
|
||||
+1
-1
@@ -113,7 +113,7 @@ make pg-down # stop it; the data volume is kept (pg-reset deletes it)
|
||||
```
|
||||
|
||||
The server listens on `127.0.0.1` only, on `FFB_PG_PORT` from `.env`
|
||||
(55433 by default; change it if that port is taken). `make pg-psql` opens a
|
||||
(55447 by default; change it if that port is taken). `make pg-psql` opens a
|
||||
shell on it. Every Postgres test gets its own freshly created database with
|
||||
the `vector` extension, dropped when the test ends, so tests never see each
|
||||
other's rows. To write a test that runs on both backends, take the `db_url`
|
||||
|
||||
@@ -30,6 +30,20 @@ pip install fastmcp-feedback
|
||||
uv add "fastmcp-feedback[postgresql]"
|
||||
```
|
||||
|
||||
What each install covers:
|
||||
|
||||
| Install | Adds | Covers |
|
||||
|---------|------|--------|
|
||||
| `fastmcp-feedback` | FastMCP, SQLAlchemy, Pydantic | The feedback tools (sync SQLAlchemy, no greenlet), and `instrument(app)` with `JsonLinesSink`, `CallbackSink` and `MemorySink`, events, redaction and result classification |
|
||||
| `fastmcp-feedback[instrumentation]` | `sqlalchemy[asyncio]` (greenlet), `aiosqlite` | `DatabaseSink`, `build_metadata`, call history for linking feedback (`capture_recent_calls` across workers, `link_feedback`, `feedback_context`) |
|
||||
| `fastmcp-feedback[instrumentation,embeddings]` | `pgvector` | `EmbeddingSink`, `OpenAIEmbedder`, `similar()`, the backfill helpers; pgvector is only needed on PostgreSQL |
|
||||
|
||||
Postgres hosts add `asyncpg` for the async sinks. Without the
|
||||
`instrumentation` extra, `from fastmcp_feedback.instrumentation import
|
||||
DatabaseSink` raises an `ImportError` that names the missing package and the
|
||||
extra to install, and the middleware's database helpers return their empty
|
||||
defaults (`[]`, `False`).
|
||||
|
||||
### From Git (Latest Development)
|
||||
|
||||
```bash
|
||||
@@ -181,8 +195,7 @@ instrument(app, [
|
||||
|
||||
`DatabaseSink` needs the `instrumentation` extra
|
||||
(`uv add "fastmcp-feedback[instrumentation]"`), plus `asyncpg` for Postgres.
|
||||
Install the extra even if you already have SQLAlchemy: SQLAlchemy 2.1 no longer
|
||||
pulls in `greenlet` by itself, and its async engine cannot run without it.
|
||||
The other sinks work on the base install (see [Installation](#installation)).
|
||||
|
||||
### Using your own database and migrations
|
||||
|
||||
@@ -232,7 +245,30 @@ string with something other than whitespace counts, so `{"error": None}`,
|
||||
`{"error": ""}` and a statistic such as `{"error": 0.02}` stay ok. A dict or
|
||||
list there (`{"error": {"code": 1}}`) is not flagged either; catch that shape
|
||||
with a flag key, a status or your own classifier. `error_keys` names the keys
|
||||
to check, and `error_keys=()` turns the check off:
|
||||
to check, and `error_keys=()` turns the check off.
|
||||
|
||||
A result that says it succeeded wins over its own error string: when `ok` or
|
||||
`success` (the `flag_keys`) is exactly `True`, the `error` check is skipped and
|
||||
the string is read as a warning, so `{"ok": True, "error": "deprecated arg"}`
|
||||
is recorded as `ok`. Only that check is skipped. In order of precedence:
|
||||
|
||||
1. A failing `status` or `state` flags the result, even next to `ok: True`,
|
||||
since `{"success": True, "status": "failed"}` contradicts itself and the
|
||||
status is the more specific of the two.
|
||||
2. Any flag key exactly `False` flags it, even if another is `True`.
|
||||
3. `isError` flags it.
|
||||
4. A flag key exactly `True` makes it ok. Truthy values that are not `True`,
|
||||
such as `1` or `"yes"`, do not count.
|
||||
5. A non-empty string at an `error_keys` key flags it.
|
||||
|
||||
```python
|
||||
from fastmcp_feedback.instrumentation import DEFAULT_RESULT_CLASSIFIER as classify
|
||||
|
||||
assert classify("t", {"ok": True, "error": "deprecated arg"}) is None
|
||||
assert classify("t", {"ok": 1, "error": "x"}) == "error: x"
|
||||
assert classify("t", {"success": True, "status": "failed"}) == "status=failed"
|
||||
assert classify("t", {"played": False, "error": "stalled"}) == "error: stalled"
|
||||
```
|
||||
|
||||
```python
|
||||
from fastmcp_feedback.instrumentation import instrument, soft_error_classifier
|
||||
@@ -289,6 +325,10 @@ upgrading. Results that also have a failing status or flag keep their old
|
||||
message, such as `success=False: ...`, and are counted once. Pass
|
||||
`soft_error_classifier(error_keys=())` to keep the old behavior.
|
||||
|
||||
**Upgrading to 2026.10.01.1:** a result with `ok` or `success` exactly `True`
|
||||
and an `error` string, which 2026.10.01 recorded as `soft_error`, is `ok` again.
|
||||
Nothing else changes.
|
||||
|
||||
### Identity and enrichment hooks
|
||||
|
||||
```python
|
||||
@@ -928,12 +968,13 @@ Add all five feedback tools to a FastMCP server.
|
||||
|
||||
## Documentation
|
||||
|
||||
Full documentation: https://fastmcp-feedback.l.supported.systems
|
||||
Full documentation: https://fastmcp-feedback.supported.systems
|
||||
|
||||
- [Quick Start Tutorial](https://fastmcp-feedback.l.supported.systems/tutorials/quickstart/)
|
||||
- [Architecture](https://fastmcp-feedback.l.supported.systems/explanation/architecture/)
|
||||
- [API Reference](https://fastmcp-feedback.l.supported.systems/reference/api/)
|
||||
- [Integration Patterns](https://fastmcp-feedback.l.supported.systems/how-to/integration-patterns/)
|
||||
- [Instrument a server in five minutes](https://fastmcp-feedback.supported.systems/tutorials/instrument-a-server/)
|
||||
- [Architecture](https://fastmcp-feedback.supported.systems/explanation/architecture/)
|
||||
- [Middleware reference](https://fastmcp-feedback.supported.systems/reference/middleware/)
|
||||
- [Feedback linked to the calls behind it](https://fastmcp-feedback.supported.systems/tutorials/feedback-with-context/)
|
||||
- [Embeddings and similarity search](https://fastmcp-feedback.supported.systems/how-to/embeddings/)
|
||||
|
||||
## Examples
|
||||
|
||||
@@ -992,7 +1033,7 @@ MIT License - see [LICENSE](LICENSE) for details.
|
||||
|
||||
## Links
|
||||
|
||||
- **Documentation**: https://fastmcp-feedback.l.supported.systems
|
||||
- **Documentation**: https://fastmcp-feedback.supported.systems
|
||||
- **Repository**: https://git.supported.systems/fastmcp-feedback/fastmcp-feedback
|
||||
- **Issues**: https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/issues
|
||||
- **FastMCP**: https://gofastmcp.com
|
||||
|
||||
+1
-1
@@ -11,7 +11,7 @@ services:
|
||||
POSTGRES_PASSWORD: ${FFB_PG_PASSWORD}
|
||||
POSTGRES_DB: ${FFB_PG_DB}
|
||||
ports:
|
||||
- "127.0.0.1:${FFB_PG_PORT:-55433}:5432"
|
||||
- "127.0.0.1:${FFB_PG_PORT:-55447}:5432"
|
||||
volumes:
|
||||
- ffb-postgres-data:/var/lib/postgresql/data
|
||||
- ./docker/postgres-init:/docker-entrypoint-initdb.d:ro
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "fastmcp-feedback"
|
||||
version = "2026.10.01" # YYYYMMDD.patch versioning - update for each release
|
||||
version = "2026.10.01.3" # YYYYMMDD.patch versioning - update for each release
|
||||
description = "Production-ready feedback collection system for FastMCP servers"
|
||||
readme = "README.md"
|
||||
license = "MIT"
|
||||
@@ -83,7 +83,7 @@ all = ["fastmcp-feedback[postgresql,mysql,dev]"]
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback"
|
||||
Documentation = "https://fastmcp-feedback.l.supported.systems"
|
||||
Documentation = "https://fastmcp-feedback.supported.systems"
|
||||
Repository = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback.git"
|
||||
Issues = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/issues"
|
||||
Changelog = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/blob/main/CHANGELOG.md"
|
||||
|
||||
@@ -44,7 +44,7 @@ Quick Start:
|
||||
import logging
|
||||
|
||||
# Version information - CalVer YYYY.MM.DD
|
||||
__version__ = "2026.10.01"
|
||||
__version__ = "2026.10.01.3"
|
||||
__author__ = "Ryan Malloy"
|
||||
__email__ = "ryan@supported.systems"
|
||||
|
||||
|
||||
@@ -13,21 +13,21 @@ await mw.similar("render timed out")
|
||||
await backfill_embeddings(mw.embedding_sink) # rows stored before embedding
|
||||
"""
|
||||
|
||||
from .backfill import backfill_embeddings, backfill_feedback
|
||||
import importlib
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from .classify import (
|
||||
DEFAULT_RESULT_CLASSIFIER,
|
||||
ResultClassifier,
|
||||
soft_error_classifier,
|
||||
)
|
||||
from .dispatch import RecordDispatcher
|
||||
from .embeddings import (
|
||||
from .constants import (
|
||||
DEFAULT_EMBEDDING_DIM,
|
||||
DEFAULT_EVENT_TEXT_KEYS,
|
||||
Embedder,
|
||||
EmbeddingSink,
|
||||
EmbeddingText,
|
||||
OpenAIEmbedder,
|
||||
extract_texts,
|
||||
DEFAULT_TABLE_PREFIX,
|
||||
EMBEDDING_SOURCES,
|
||||
)
|
||||
from .dispatch import RecordDispatcher
|
||||
from .events import EventRecord
|
||||
from .middleware import MODE_ENV_VAR, MODES, InstrumentationMiddleware, instrument
|
||||
from .record import ToolCallRecord
|
||||
@@ -38,14 +38,75 @@ from .redaction import (
|
||||
Redactor,
|
||||
)
|
||||
from .sinks import CallbackSink, JsonLinesSink, MemorySink, Sink
|
||||
from .storage import (
|
||||
DEFAULT_EMBEDDING_DIM,
|
||||
DEFAULT_TABLE_PREFIX,
|
||||
EMBEDDING_SOURCES,
|
||||
DatabaseSink,
|
||||
EmbeddingVector,
|
||||
build_metadata,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .backfill import backfill_embeddings, backfill_feedback
|
||||
from .embeddings import (
|
||||
Embedder,
|
||||
EmbeddingSink,
|
||||
EmbeddingText,
|
||||
OpenAIEmbedder,
|
||||
extract_texts,
|
||||
)
|
||||
from .storage import DatabaseSink, EmbeddingVector, build_metadata
|
||||
|
||||
# Names whose modules need the async database stack (sqlalchemy[asyncio],
|
||||
# which brings greenlet). Imported on first access, so instrument() and the
|
||||
# plain sinks work on a base install.
|
||||
_LAZY = {
|
||||
"DatabaseSink": "storage",
|
||||
"EmbeddingVector": "storage",
|
||||
"build_metadata": "storage",
|
||||
"Embedder": "embeddings",
|
||||
"EmbeddingSink": "embeddings",
|
||||
"EmbeddingText": "embeddings",
|
||||
"OpenAIEmbedder": "embeddings",
|
||||
"extract_texts": "embeddings",
|
||||
"backfill_embeddings": "backfill",
|
||||
"backfill_feedback": "backfill",
|
||||
}
|
||||
_EXTRA = {
|
||||
"storage": "instrumentation",
|
||||
"embeddings": "instrumentation,embeddings",
|
||||
"backfill": "instrumentation,embeddings",
|
||||
}
|
||||
|
||||
|
||||
def _missing_module(exc: BaseException) -> str:
|
||||
seen: BaseException | None = exc
|
||||
while seen is not None:
|
||||
if isinstance(seen, ImportError) and seen.name:
|
||||
return seen.name
|
||||
seen = seen.__cause__ or seen.__context__
|
||||
return "a dependency"
|
||||
|
||||
|
||||
def __getattr__(name: str) -> Any:
|
||||
module = _LAZY.get(name)
|
||||
if module is None:
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
try:
|
||||
# Checked first because SQLAlchemy reports a missing greenlet only on
|
||||
# the first import of its asyncio module; a retry would then succeed
|
||||
# and leave an engine that fails at runtime.
|
||||
importlib.import_module("greenlet")
|
||||
value = getattr(importlib.import_module(f".{module}", __name__), name)
|
||||
except ImportError as exc:
|
||||
extra = _EXTRA[module]
|
||||
missing = _missing_module(exc)
|
||||
raise ImportError(
|
||||
f"fastmcp_feedback.instrumentation.{name} needs the async database "
|
||||
f"stack, and {missing} is not installed. Install it with: "
|
||||
f'uv add "fastmcp-feedback[{extra}]" '
|
||||
f'(or pip install "fastmcp-feedback[{extra}]")'
|
||||
) from exc
|
||||
globals()[name] = value
|
||||
return value
|
||||
|
||||
|
||||
def __dir__() -> list[str]:
|
||||
return sorted(set(globals()) | set(_LAZY))
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CallbackSink",
|
||||
|
||||
@@ -75,11 +75,19 @@ def soft_error_classifier(
|
||||
so ``{"error": None}`` and ``{"error": 0.02}`` stay ok. ``error_keys=()``
|
||||
turns this off.
|
||||
|
||||
A result that says it succeeded, with any of ``flag_keys`` exactly True,
|
||||
skips only that last check: ``{"ok": True, "error": "deprecated arg"}``
|
||||
is ok, the string being a warning. A failing status, a False flag and
|
||||
``isError`` still flag it. Truthy non-bools such as ``1`` or ``"yes"``
|
||||
are not an explicit success.
|
||||
|
||||
>>> classify = soft_error_classifier(extra_statuses=("unknown_target",))
|
||||
>>> classify("render", {"status": "unknown_target", "error": "no such object"})
|
||||
'status=unknown_target: no such object'
|
||||
>>> classify("speak", {"played": False, "error": "output stalled"})
|
||||
'error: output stalled'
|
||||
>>> print(classify("speak", {"ok": True, "error": "deprecated arg"}))
|
||||
None
|
||||
"""
|
||||
bad = frozenset(s.lower() for s in (*statuses, *extra_statuses))
|
||||
status_keys = tuple(status_keys)
|
||||
@@ -100,6 +108,8 @@ def soft_error_classifier(
|
||||
if result.get(key) is True:
|
||||
detail = _message(result) or _content_text(result.get("content"))
|
||||
return _describe("isError", detail)
|
||||
if any(result.get(key) is True for key in flag_keys):
|
||||
return None
|
||||
for key in error_keys:
|
||||
value = result.get(key)
|
||||
if isinstance(value, str) and value.strip():
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
"""Defaults shared by the storage and embedding modules.
|
||||
|
||||
Kept apart from them so these import without the async database stack.
|
||||
"""
|
||||
|
||||
DEFAULT_TABLE_PREFIX = "ffb_"
|
||||
DEFAULT_EMBEDDING_DIM = 1024
|
||||
# source_type values in {prefix}embeddings. Feedback rows survive pruning.
|
||||
EMBEDDING_SOURCES = ("feedback", "call_error", "event", "llm")
|
||||
FEEDBACK_SOURCE = "feedback"
|
||||
# Attribute keys whose text an EmbeddingSink embeds from events.
|
||||
DEFAULT_EVENT_TEXT_KEYS = ("error_excerpt", "error", "message", "reason")
|
||||
@@ -20,16 +20,15 @@ from typing import Any, Protocol, runtime_checkable
|
||||
|
||||
from sqlalchemy import Float, and_, literal, not_, select, tuple_
|
||||
|
||||
from .constants import DEFAULT_EVENT_TEXT_KEYS, EMBEDDING_SOURCES
|
||||
from .events import EventRecord
|
||||
from .record import ToolCallRecord
|
||||
from .redaction import Redactor
|
||||
from .sinks import Record
|
||||
from .storage import EMBEDDING_SOURCES, EmbeddingVector
|
||||
from .storage import EmbeddingVector
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_EVENT_TEXT_KEYS = ("error_excerpt", "error", "message", "reason")
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class Embedder(Protocol):
|
||||
|
||||
@@ -8,12 +8,13 @@ import json
|
||||
import logging
|
||||
import os
|
||||
import random as _random
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from collections import deque
|
||||
from collections.abc import Awaitable, Callable, Iterable, Mapping, Sequence
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from fastmcp import FastMCP
|
||||
from fastmcp.server.middleware import Middleware, MiddlewareContext
|
||||
@@ -27,7 +28,6 @@ from .correlation import (
|
||||
select_calls,
|
||||
)
|
||||
from .dispatch import RecordDispatcher
|
||||
from .embeddings import EmbeddingSink
|
||||
from .events import (
|
||||
MAX_KEY_CHARS,
|
||||
MAX_KIND_CHARS,
|
||||
@@ -38,7 +38,10 @@ from .events import (
|
||||
from .record import ToolCallRecord
|
||||
from .redaction import Redactor
|
||||
from .sinks import JsonLinesSink, Sink
|
||||
from .storage import DatabaseSink
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .embeddings import EmbeddingSink
|
||||
from .storage import DatabaseSink
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -237,7 +240,7 @@ class InstrumentationMiddleware(Middleware):
|
||||
sinks = list(sinks) if sinks is not None else [JsonLinesSink()]
|
||||
for sink in sinks:
|
||||
# Embedded text gets this middleware's redaction rules too.
|
||||
if isinstance(sink, EmbeddingSink) and sink.redactor is None:
|
||||
if _is_sink(sink, "embeddings", "EmbeddingSink") and sink.redactor is None:
|
||||
sink.redactor = self.redactor
|
||||
self.dispatcher = RecordDispatcher(
|
||||
sinks,
|
||||
@@ -573,11 +576,9 @@ class InstrumentationMiddleware(Middleware):
|
||||
return rows[-limit:] if limit > 0 else []
|
||||
|
||||
@property
|
||||
def embedding_sink(self) -> EmbeddingSink | None:
|
||||
def embedding_sink(self) -> "EmbeddingSink | None":
|
||||
"""The first EmbeddingSink among the sinks, if any."""
|
||||
return next(
|
||||
(s for s in self.dispatcher.sinks if isinstance(s, EmbeddingSink)), None
|
||||
)
|
||||
return _first_sink(self.dispatcher.sinks, "embeddings", "EmbeddingSink")
|
||||
|
||||
async def similar(
|
||||
self,
|
||||
@@ -644,11 +645,9 @@ class InstrumentationMiddleware(Middleware):
|
||||
return []
|
||||
|
||||
@property
|
||||
def database_sink(self) -> DatabaseSink | None:
|
||||
def database_sink(self) -> "DatabaseSink | None":
|
||||
"""The first DatabaseSink among the sinks, if any."""
|
||||
return next(
|
||||
(s for s in self.dispatcher.sinks if isinstance(s, DatabaseSink)), None
|
||||
)
|
||||
return _first_sink(self.dispatcher.sinks, "storage", "DatabaseSink")
|
||||
|
||||
async def capture_recent_calls(
|
||||
self,
|
||||
@@ -786,6 +785,21 @@ class InstrumentationMiddleware(Middleware):
|
||||
await self.dispatcher.aclose()
|
||||
|
||||
|
||||
def _is_sink(sink: Any, module: str, name: str) -> bool:
|
||||
"""isinstance against a sink class without importing its module.
|
||||
|
||||
storage and embeddings need the async database stack. When the module was
|
||||
never imported, no instance of its class can exist, so the answer is no.
|
||||
"""
|
||||
loaded = sys.modules.get(f"{__package__}.{module}")
|
||||
cls = getattr(loaded, name, None) if loaded is not None else None
|
||||
return cls is not None and isinstance(sink, cls)
|
||||
|
||||
|
||||
def _first_sink(sinks: Iterable[Any], module: str, name: str) -> Any:
|
||||
return next((s for s in sinks if _is_sink(s, module, name)), None)
|
||||
|
||||
|
||||
def _current_context() -> Any:
|
||||
try:
|
||||
from fastmcp.server.dependencies import get_context
|
||||
|
||||
@@ -40,22 +40,21 @@ from sqlalchemy.exc import DBAPIError
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
|
||||
from sqlalchemy.types import TypeDecorator
|
||||
|
||||
from .constants import (
|
||||
DEFAULT_EMBEDDING_DIM,
|
||||
DEFAULT_TABLE_PREFIX,
|
||||
FEEDBACK_SOURCE,
|
||||
)
|
||||
from .constants import EMBEDDING_SOURCES as EMBEDDING_SOURCES # re-export
|
||||
from .events import MAX_KEY_CHARS, MAX_KIND_CHARS, EventRecord
|
||||
from .record import ToolCallRecord
|
||||
from .sinks import Record
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_TABLE_PREFIX = "ffb_"
|
||||
|
||||
# none_as_null: a missing payload is SQL NULL, so "WHERE args IS NULL" works.
|
||||
_JSON = JSON(none_as_null=True).with_variant(JSONB(none_as_null=True), "postgresql")
|
||||
|
||||
DEFAULT_EMBEDDING_DIM = 1024
|
||||
# source_type values in {prefix}embeddings. Feedback rows survive pruning.
|
||||
EMBEDDING_SOURCES = ("feedback", "call_error", "event", "llm")
|
||||
FEEDBACK_SOURCE = "feedback"
|
||||
|
||||
|
||||
class EmbeddingVector(TypeDecorator):
|
||||
"""pgvector ``VECTOR(dim)`` on PostgreSQL, a JSON list of floats elsewhere.
|
||||
|
||||
+105
@@ -0,0 +1,105 @@
|
||||
"""Exercise the instrumentation as a base install would, then print JSON.
|
||||
|
||||
Run in a fresh interpreter. greenlet and aiosqlite are made unimportable
|
||||
before anything else loads, as on ``pip install fastmcp-feedback`` without
|
||||
the ``instrumentation`` extra. Argument: a directory for the feedback database.
|
||||
"""
|
||||
|
||||
import sys
|
||||
from importlib.metadata import version
|
||||
|
||||
# SQLAlchemy 2.1 refuses to import sqlalchemy.ext.asyncio without greenlet,
|
||||
# once: a second import succeeds. 2.0 imports it and fails on first use.
|
||||
# On 2.0 the module is blocked too, so both versions fail at import.
|
||||
_blocked = ["greenlet", "aiosqlite"]
|
||||
if tuple(int(p) for p in version("sqlalchemy").split(".")[:2]) < (2, 1):
|
||||
_blocked.append("sqlalchemy.ext.asyncio")
|
||||
for _name in _blocked:
|
||||
sys.modules[_name] = None # import raises ModuleNotFoundError
|
||||
|
||||
import asyncio # noqa: E402
|
||||
import io # noqa: E402
|
||||
import json # noqa: E402
|
||||
from pathlib import Path # noqa: E402
|
||||
|
||||
STORAGE = "fastmcp_feedback.instrumentation.storage"
|
||||
EMBEDDINGS = "fastmcp_feedback.instrumentation.embeddings"
|
||||
|
||||
|
||||
async def run(db_dir: Path) -> dict:
|
||||
from fastmcp import Client, FastMCP
|
||||
|
||||
import fastmcp_feedback.instrumentation as inst
|
||||
from fastmcp_feedback import add_feedback_tools
|
||||
from fastmcp_feedback.instrumentation import (
|
||||
DEFAULT_TABLE_PREFIX,
|
||||
JsonLinesSink,
|
||||
MemorySink,
|
||||
instrument,
|
||||
)
|
||||
|
||||
out: dict = {"prefix": DEFAULT_TABLE_PREFIX}
|
||||
app = FastMCP("base-install")
|
||||
|
||||
@app.tool
|
||||
def add(a: int, b: int) -> int:
|
||||
return a + b
|
||||
|
||||
memory = MemorySink()
|
||||
lines = io.StringIO()
|
||||
mw = instrument(app, [memory, JsonLinesSink(lines)], mode="full")
|
||||
add_feedback_tools(
|
||||
app, database_url=f"sqlite:///{db_dir / 'fb.db'}", instrumentation=mw
|
||||
)
|
||||
|
||||
async with Client(app) as client:
|
||||
result = await client.call_tool("add", {"a": 2, "b": 3})
|
||||
out["result"] = result.content[0].text
|
||||
fb = await client.call_tool(
|
||||
"submit_feedback",
|
||||
{
|
||||
"request": {
|
||||
"type": "bug",
|
||||
"title": "base install",
|
||||
"description": "submitted without the extra",
|
||||
"submitter": "worker",
|
||||
}
|
||||
},
|
||||
)
|
||||
out["feedback"] = json.loads(fb.content[0].text)
|
||||
|
||||
mw.record_event("job.done", key="j1", attrs={"n": 1})
|
||||
await mw.flush()
|
||||
out["tools"] = [r.tool for r in memory.records if hasattr(r, "tool")]
|
||||
out["event_kinds"] = [r.kind for r in memory.records if hasattr(r, "kind")]
|
||||
out["jsonl_lines"] = len(lines.getvalue().splitlines())
|
||||
out["capture"] = await mw.capture_recent_calls()
|
||||
out["similar"] = await mw.similar("anything")
|
||||
out["similar_feedback"] = await mw.similar_feedback("bug-1")
|
||||
out["feedback_context"] = await mw.feedback_context("bug-1")
|
||||
out["link_feedback"] = await mw.link_feedback("bug-1", [])
|
||||
out["events_for"] = len(await mw.events_for("j1"))
|
||||
out["database_sink"] = mw.database_sink is None
|
||||
out["embedding_sink"] = mw.embedding_sink is None
|
||||
await mw.aclose()
|
||||
|
||||
errors = {}
|
||||
for name in (
|
||||
"DatabaseSink",
|
||||
"EmbeddingSink",
|
||||
"build_metadata",
|
||||
"backfill_feedback",
|
||||
):
|
||||
for _attempt in range(2): # SQLAlchemy only complains the first time
|
||||
try:
|
||||
getattr(inst, name)
|
||||
errors.setdefault(name, []).append(None)
|
||||
except ImportError as exc:
|
||||
errors.setdefault(name, []).append(str(exc))
|
||||
out["errors"] = errors
|
||||
out["loaded"] = sorted(m for m in (STORAGE, EMBEDDINGS) if m in sys.modules)
|
||||
return out
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(json.dumps(asyncio.run(run(Path(sys.argv[1]))), default=str))
|
||||
@@ -0,0 +1,90 @@
|
||||
"""The instrumentation works on a base install, without the database extras.
|
||||
|
||||
The worker runs in a fresh interpreter with greenlet and aiosqlite made
|
||||
unimportable, which is what ``pip install fastmcp-feedback`` leaves out.
|
||||
"""
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
WORKER = Path(__file__).parent.parent / "fixtures" / "base_install_worker.py"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def report(tmp_path_factory) -> dict:
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(WORKER), str(tmp_path_factory.mktemp("base"))],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=60,
|
||||
)
|
||||
assert proc.returncode == 0, proc.stderr[-4000:]
|
||||
return json.loads(proc.stdout.strip().splitlines()[-1])
|
||||
|
||||
|
||||
def test_tool_calls_are_recorded(report):
|
||||
assert report["result"] == "5"
|
||||
assert report["tools"] == ["add", "submit_feedback"]
|
||||
assert report["jsonl_lines"] == 4 # two calls, two events
|
||||
|
||||
|
||||
def test_events_are_recorded(report):
|
||||
assert report["event_kinds"] == ["feedback.submitted", "job.done"]
|
||||
assert report["events_for"] == 1
|
||||
|
||||
|
||||
def test_feedback_tools_work_alongside(report):
|
||||
assert report["feedback"]["success"] is True
|
||||
assert report["feedback"]["linked_calls"] == 0
|
||||
|
||||
|
||||
def test_database_helpers_return_empty_defaults(report):
|
||||
assert report["capture"] == []
|
||||
assert report["similar"] == []
|
||||
assert report["similar_feedback"] == []
|
||||
assert report["feedback_context"] == []
|
||||
assert report["link_feedback"] is False
|
||||
assert report["database_sink"] is True
|
||||
assert report["embedding_sink"] is True
|
||||
|
||||
|
||||
def test_constants_import_without_the_extra(report):
|
||||
assert report["prefix"] == "ffb_"
|
||||
|
||||
|
||||
def test_storage_modules_stay_unloaded(report):
|
||||
assert report["loaded"] == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("name", "extra"),
|
||||
[
|
||||
("DatabaseSink", "[instrumentation]"),
|
||||
("build_metadata", "[instrumentation]"),
|
||||
("EmbeddingSink", "[instrumentation,embeddings]"),
|
||||
("backfill_feedback", "[instrumentation,embeddings]"),
|
||||
],
|
||||
)
|
||||
def test_database_names_explain_the_missing_extra(report, name, extra):
|
||||
# Asked twice: SQLAlchemy reports a missing greenlet only on first import.
|
||||
for message in report["errors"][name]:
|
||||
assert message is not None
|
||||
assert "greenlet" in message
|
||||
assert f'uv add "fastmcp-feedback{extra}"' in message
|
||||
|
||||
|
||||
def test_lazy_names_resolve_with_the_extra():
|
||||
import fastmcp_feedback.instrumentation as inst
|
||||
from fastmcp_feedback.instrumentation import DatabaseSink, storage
|
||||
|
||||
assert DatabaseSink is storage.DatabaseSink
|
||||
assert set(inst._LAZY) <= set(inst.__all__)
|
||||
assert set(inst.__all__) <= set(dir(inst))
|
||||
for name in inst.__all__:
|
||||
assert getattr(inst, name) is not None
|
||||
with pytest.raises(AttributeError):
|
||||
_ = inst.not_a_name
|
||||
@@ -201,6 +201,70 @@ class TestClassifier:
|
||||
assert classify("t", {"ok": False, "error": "x"}) == "ok=False: x"
|
||||
|
||||
|
||||
class TestExplicitSuccess:
|
||||
"""A flag key that is exactly True turns an error string into a warning."""
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"result",
|
||||
[
|
||||
{"ok": True, "error": "deprecated arg"},
|
||||
{"success": True, "error": "x"},
|
||||
{"ok": True, "status": "unknown_target", "error": "fell back"},
|
||||
],
|
||||
)
|
||||
def test_true_flag_with_error_string_is_ok(self, result):
|
||||
assert DEFAULT_RESULT_CLASSIFIER("t", result) is None
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"result, expected",
|
||||
[
|
||||
({"ok": False, "error": "x"}, "ok=False: x"),
|
||||
# Not exactly True, so not an explicit success.
|
||||
({"ok": 1, "error": "x"}, "error: x"),
|
||||
({"ok": "yes", "error": "x"}, "error: x"),
|
||||
({"success": 1.0, "error": "x"}, "error: x"),
|
||||
# No flag key: mcspeak's shape, as in 2026.10.01.
|
||||
(
|
||||
{"played": False, "error": "audio output stalled"},
|
||||
"error: audio output stalled",
|
||||
),
|
||||
# A False flag beats a True one.
|
||||
({"ok": True, "success": False, "error": "x"}, "success=False: x"),
|
||||
# A failing status is more specific than the flag, so it still flags.
|
||||
({"success": True, "status": "failed"}, "status=failed"),
|
||||
({"ok": True, "state": "error", "error": "x"}, "state=error: x"),
|
||||
({"ok": True, "isError": True, "content": ["boom"]}, "isError: boom"),
|
||||
],
|
||||
)
|
||||
def test_still_flagged(self, result, expected):
|
||||
assert DEFAULT_RESULT_CLASSIFIER("t", result) == expected
|
||||
|
||||
def test_custom_flag_keys(self):
|
||||
classify = soft_error_classifier(flag_keys=("passed",))
|
||||
assert classify("t", {"passed": True, "error": "slow"}) is None
|
||||
assert classify("t", {"passed": False, "error": "x"}) == "passed=False: x"
|
||||
# ok is not a flag key here, so it does not suppress the error check.
|
||||
assert classify("t", {"ok": True, "error": "x"}) == "error: x"
|
||||
|
||||
def test_no_flag_keys(self):
|
||||
classify = soft_error_classifier(flag_keys=())
|
||||
assert classify("t", {"ok": True, "error": "x"}) == "error: x"
|
||||
|
||||
async def test_recorded_as_ok(self):
|
||||
record = await record_for(
|
||||
"respond", {"payload": {"ok": True, "error": "deprecated arg"}}
|
||||
)
|
||||
assert (record.outcome, record.ok, record.error_message) == ("ok", True, None)
|
||||
|
||||
async def test_truthy_flag_recorded_as_soft_error(self):
|
||||
record = await record_for("respond", {"payload": {"ok": 1, "error": "x"}})
|
||||
assert (record.outcome, record.ok, record.error_message) == (
|
||||
"soft_error",
|
||||
False,
|
||||
"error: x",
|
||||
)
|
||||
|
||||
|
||||
class TestMiddleware:
|
||||
async def test_status_failed_is_soft_error(self):
|
||||
record = await record_for("render", {"status": "failed"})
|
||||
|
||||
Reference in New Issue
Block a user