Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
433a003d09 | ||
|
|
e4da62d0df | ||
|
|
5e9a9ca037 | ||
|
|
76999c942c | ||
|
|
4995143f8f | ||
|
|
28816477ff | ||
|
|
c81dc3bf8c | ||
|
|
4922eaecbd | ||
|
|
36fed0267c | ||
|
|
ae52e7a0f9 |
+2
-2
@@ -6,8 +6,8 @@ FFB_PG_USER=ffb
|
|||||||
FFB_PG_PASSWORD=change-me
|
FFB_PG_PASSWORD=change-me
|
||||||
FFB_PG_DB=ffb_test
|
FFB_PG_DB=ffb_test
|
||||||
# Published on 127.0.0.1 only. 55432 is commonly taken by other local stacks.
|
# Published on 127.0.0.1 only. 55432 is commonly taken by other local stacks.
|
||||||
FFB_PG_PORT=55433
|
FFB_PG_PORT=55447
|
||||||
|
|
||||||
# Read by the test suite. Unset, the Postgres test cases skip.
|
# Read by the test suite. Unset, the Postgres test cases skip.
|
||||||
# Each test gets its own database created on this server and dropped after.
|
# Each test gets its own database created on this server and dropped after.
|
||||||
FFB_TEST_DATABASE_URL=postgresql+asyncpg://ffb:change-me@127.0.0.1:55433/ffb_test
|
FFB_TEST_DATABASE_URL=postgresql+asyncpg://ffb:change-me@127.0.0.1:55447/ffb_test
|
||||||
|
|||||||
@@ -5,6 +5,57 @@ All notable changes to this project will be documented in this file.
|
|||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||||
and this project adheres to [Calendar Versioning](https://calver.org/) (YYYY.MM.DD).
|
and this project adheres to [Calendar Versioning](https://calver.org/) (YYYY.MM.DD).
|
||||||
|
|
||||||
|
## [2026.10.01.3] - 2026-10-01
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
- The documentation site is public at https://fastmcp-feedback.supported.systems,
|
||||||
|
rewritten against this package with every example executed. The README
|
||||||
|
links and the PyPI "Documentation" URL point there instead of the private
|
||||||
|
dev address.
|
||||||
|
|
||||||
|
## [2026.10.01.2] - 2026-10-01
|
||||||
|
|
||||||
|
The instrumentation imports on a base install, without the database extras.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- `import fastmcp_feedback.instrumentation` failed on `pip install
|
||||||
|
fastmcp-feedback` without the `instrumentation` extra: the package imported
|
||||||
|
`storage.py` eagerly, which imports `sqlalchemy.ext.asyncio`, and SQLAlchemy
|
||||||
|
2.1 raises `ImportError` there when `greenlet` is missing. `instrument(app)`
|
||||||
|
with `JsonLinesSink`, `CallbackSink` or `MemorySink`, events, redaction and
|
||||||
|
result classification now need only the base dependencies.
|
||||||
|
- `DatabaseSink`, `EmbeddingVector`, `build_metadata`, `Embedder`,
|
||||||
|
`EmbeddingSink`, `EmbeddingText`, `OpenAIEmbedder`, `extract_texts`,
|
||||||
|
`backfill_embeddings` and `backfill_feedback` load on first access. Without
|
||||||
|
the extra, accessing one raises `ImportError` naming the missing package and
|
||||||
|
the extra to install (`[instrumentation]`, or `[instrumentation,embeddings]`
|
||||||
|
for the embedding names), chained to the original error. Every access
|
||||||
|
raises, not only the first, although SQLAlchemy itself reports a missing
|
||||||
|
`greenlet` only once. Import paths are unchanged.
|
||||||
|
- The middleware finds a `DatabaseSink` or `EmbeddingSink` among its sinks
|
||||||
|
without importing their modules, so `database_sink`, `embedding_sink`,
|
||||||
|
`capture_recent_calls`, `link_feedback`, `feedback_context`, `events_for`,
|
||||||
|
`similar` and `similar_feedback` return their usual empty results on a base
|
||||||
|
install.
|
||||||
|
- `DEFAULT_TABLE_PREFIX`, `DEFAULT_EMBEDDING_DIM`, `EMBEDDING_SOURCES` and
|
||||||
|
`DEFAULT_EVENT_TEXT_KEYS` moved to `instrumentation/constants.py` and import
|
||||||
|
without the extras; `storage` and `embeddings` still export them.
|
||||||
|
|
||||||
|
## [2026.10.01.1] - 2026-10-01
|
||||||
|
|
||||||
|
A result that says it succeeded is no longer flagged for its own error string.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Explicit success flag wins over error keys: when any of the classifier's
|
||||||
|
`flag_keys` (`ok`, `success`) is exactly `True`, the `error_keys` check is
|
||||||
|
skipped and the string is treated as a warning, so
|
||||||
|
`{"ok": True, "error": "deprecated arg"}` is recorded as `ok` again. Only
|
||||||
|
that check is skipped: a failing status (`{"success": True, "status":
|
||||||
|
"failed"}`), any flag key exactly `False` and `isError` still flag the
|
||||||
|
result. Truthy values that are not `True`, such as `1` or `"yes"`, are not
|
||||||
|
an explicit success. Results with no flag key are classified as in
|
||||||
|
2026.10.01.
|
||||||
|
|
||||||
## [2026.10.01] - 2026-10-01
|
## [2026.10.01] - 2026-10-01
|
||||||
|
|
||||||
Results that report failure only as an `error` string now count as soft
|
Results that report failure only as an `error` string now count as soft
|
||||||
|
|||||||
+1
-1
@@ -113,7 +113,7 @@ make pg-down # stop it; the data volume is kept (pg-reset deletes it)
|
|||||||
```
|
```
|
||||||
|
|
||||||
The server listens on `127.0.0.1` only, on `FFB_PG_PORT` from `.env`
|
The server listens on `127.0.0.1` only, on `FFB_PG_PORT` from `.env`
|
||||||
(55433 by default; change it if that port is taken). `make pg-psql` opens a
|
(55447 by default; change it if that port is taken). `make pg-psql` opens a
|
||||||
shell on it. Every Postgres test gets its own freshly created database with
|
shell on it. Every Postgres test gets its own freshly created database with
|
||||||
the `vector` extension, dropped when the test ends, so tests never see each
|
the `vector` extension, dropped when the test ends, so tests never see each
|
||||||
other's rows. To write a test that runs on both backends, take the `db_url`
|
other's rows. To write a test that runs on both backends, take the `db_url`
|
||||||
|
|||||||
@@ -30,6 +30,20 @@ pip install fastmcp-feedback
|
|||||||
uv add "fastmcp-feedback[postgresql]"
|
uv add "fastmcp-feedback[postgresql]"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
What each install covers:
|
||||||
|
|
||||||
|
| Install | Adds | Covers |
|
||||||
|
|---------|------|--------|
|
||||||
|
| `fastmcp-feedback` | FastMCP, SQLAlchemy, Pydantic | The feedback tools (sync SQLAlchemy, no greenlet), and `instrument(app)` with `JsonLinesSink`, `CallbackSink` and `MemorySink`, events, redaction and result classification |
|
||||||
|
| `fastmcp-feedback[instrumentation]` | `sqlalchemy[asyncio]` (greenlet), `aiosqlite` | `DatabaseSink`, `build_metadata`, call history for linking feedback (`capture_recent_calls` across workers, `link_feedback`, `feedback_context`) |
|
||||||
|
| `fastmcp-feedback[instrumentation,embeddings]` | `pgvector` | `EmbeddingSink`, `OpenAIEmbedder`, `similar()`, the backfill helpers; pgvector is only needed on PostgreSQL |
|
||||||
|
|
||||||
|
Postgres hosts add `asyncpg` for the async sinks. Without the
|
||||||
|
`instrumentation` extra, `from fastmcp_feedback.instrumentation import
|
||||||
|
DatabaseSink` raises an `ImportError` that names the missing package and the
|
||||||
|
extra to install, and the middleware's database helpers return their empty
|
||||||
|
defaults (`[]`, `False`).
|
||||||
|
|
||||||
### From Git (Latest Development)
|
### From Git (Latest Development)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -181,8 +195,7 @@ instrument(app, [
|
|||||||
|
|
||||||
`DatabaseSink` needs the `instrumentation` extra
|
`DatabaseSink` needs the `instrumentation` extra
|
||||||
(`uv add "fastmcp-feedback[instrumentation]"`), plus `asyncpg` for Postgres.
|
(`uv add "fastmcp-feedback[instrumentation]"`), plus `asyncpg` for Postgres.
|
||||||
Install the extra even if you already have SQLAlchemy: SQLAlchemy 2.1 no longer
|
The other sinks work on the base install (see [Installation](#installation)).
|
||||||
pulls in `greenlet` by itself, and its async engine cannot run without it.
|
|
||||||
|
|
||||||
### Using your own database and migrations
|
### Using your own database and migrations
|
||||||
|
|
||||||
@@ -232,7 +245,30 @@ string with something other than whitespace counts, so `{"error": None}`,
|
|||||||
`{"error": ""}` and a statistic such as `{"error": 0.02}` stay ok. A dict or
|
`{"error": ""}` and a statistic such as `{"error": 0.02}` stay ok. A dict or
|
||||||
list there (`{"error": {"code": 1}}`) is not flagged either; catch that shape
|
list there (`{"error": {"code": 1}}`) is not flagged either; catch that shape
|
||||||
with a flag key, a status or your own classifier. `error_keys` names the keys
|
with a flag key, a status or your own classifier. `error_keys` names the keys
|
||||||
to check, and `error_keys=()` turns the check off:
|
to check, and `error_keys=()` turns the check off.
|
||||||
|
|
||||||
|
A result that says it succeeded wins over its own error string: when `ok` or
|
||||||
|
`success` (the `flag_keys`) is exactly `True`, the `error` check is skipped and
|
||||||
|
the string is read as a warning, so `{"ok": True, "error": "deprecated arg"}`
|
||||||
|
is recorded as `ok`. Only that check is skipped. In order of precedence:
|
||||||
|
|
||||||
|
1. A failing `status` or `state` flags the result, even next to `ok: True`,
|
||||||
|
since `{"success": True, "status": "failed"}` contradicts itself and the
|
||||||
|
status is the more specific of the two.
|
||||||
|
2. Any flag key exactly `False` flags it, even if another is `True`.
|
||||||
|
3. `isError` flags it.
|
||||||
|
4. A flag key exactly `True` makes it ok. Truthy values that are not `True`,
|
||||||
|
such as `1` or `"yes"`, do not count.
|
||||||
|
5. A non-empty string at an `error_keys` key flags it.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from fastmcp_feedback.instrumentation import DEFAULT_RESULT_CLASSIFIER as classify
|
||||||
|
|
||||||
|
assert classify("t", {"ok": True, "error": "deprecated arg"}) is None
|
||||||
|
assert classify("t", {"ok": 1, "error": "x"}) == "error: x"
|
||||||
|
assert classify("t", {"success": True, "status": "failed"}) == "status=failed"
|
||||||
|
assert classify("t", {"played": False, "error": "stalled"}) == "error: stalled"
|
||||||
|
```
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from fastmcp_feedback.instrumentation import instrument, soft_error_classifier
|
from fastmcp_feedback.instrumentation import instrument, soft_error_classifier
|
||||||
@@ -289,6 +325,10 @@ upgrading. Results that also have a failing status or flag keep their old
|
|||||||
message, such as `success=False: ...`, and are counted once. Pass
|
message, such as `success=False: ...`, and are counted once. Pass
|
||||||
`soft_error_classifier(error_keys=())` to keep the old behavior.
|
`soft_error_classifier(error_keys=())` to keep the old behavior.
|
||||||
|
|
||||||
|
**Upgrading to 2026.10.01.1:** a result with `ok` or `success` exactly `True`
|
||||||
|
and an `error` string, which 2026.10.01 recorded as `soft_error`, is `ok` again.
|
||||||
|
Nothing else changes.
|
||||||
|
|
||||||
### Identity and enrichment hooks
|
### Identity and enrichment hooks
|
||||||
|
|
||||||
```python
|
```python
|
||||||
@@ -928,12 +968,13 @@ Add all five feedback tools to a FastMCP server.
|
|||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
Full documentation: https://fastmcp-feedback.l.supported.systems
|
Full documentation: https://fastmcp-feedback.supported.systems
|
||||||
|
|
||||||
- [Quick Start Tutorial](https://fastmcp-feedback.l.supported.systems/tutorials/quickstart/)
|
- [Instrument a server in five minutes](https://fastmcp-feedback.supported.systems/tutorials/instrument-a-server/)
|
||||||
- [Architecture](https://fastmcp-feedback.l.supported.systems/explanation/architecture/)
|
- [Architecture](https://fastmcp-feedback.supported.systems/explanation/architecture/)
|
||||||
- [API Reference](https://fastmcp-feedback.l.supported.systems/reference/api/)
|
- [Middleware reference](https://fastmcp-feedback.supported.systems/reference/middleware/)
|
||||||
- [Integration Patterns](https://fastmcp-feedback.l.supported.systems/how-to/integration-patterns/)
|
- [Feedback linked to the calls behind it](https://fastmcp-feedback.supported.systems/tutorials/feedback-with-context/)
|
||||||
|
- [Embeddings and similarity search](https://fastmcp-feedback.supported.systems/how-to/embeddings/)
|
||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
|
|
||||||
@@ -992,7 +1033,7 @@ MIT License - see [LICENSE](LICENSE) for details.
|
|||||||
|
|
||||||
## Links
|
## Links
|
||||||
|
|
||||||
- **Documentation**: https://fastmcp-feedback.l.supported.systems
|
- **Documentation**: https://fastmcp-feedback.supported.systems
|
||||||
- **Repository**: https://git.supported.systems/fastmcp-feedback/fastmcp-feedback
|
- **Repository**: https://git.supported.systems/fastmcp-feedback/fastmcp-feedback
|
||||||
- **Issues**: https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/issues
|
- **Issues**: https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/issues
|
||||||
- **FastMCP**: https://gofastmcp.com
|
- **FastMCP**: https://gofastmcp.com
|
||||||
|
|||||||
+1
-1
@@ -11,7 +11,7 @@ services:
|
|||||||
POSTGRES_PASSWORD: ${FFB_PG_PASSWORD}
|
POSTGRES_PASSWORD: ${FFB_PG_PASSWORD}
|
||||||
POSTGRES_DB: ${FFB_PG_DB}
|
POSTGRES_DB: ${FFB_PG_DB}
|
||||||
ports:
|
ports:
|
||||||
- "127.0.0.1:${FFB_PG_PORT:-55433}:5432"
|
- "127.0.0.1:${FFB_PG_PORT:-55447}:5432"
|
||||||
volumes:
|
volumes:
|
||||||
- ffb-postgres-data:/var/lib/postgresql/data
|
- ffb-postgres-data:/var/lib/postgresql/data
|
||||||
- ./docker/postgres-init:/docker-entrypoint-initdb.d:ro
|
- ./docker/postgres-init:/docker-entrypoint-initdb.d:ro
|
||||||
|
|||||||
+2
-2
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "fastmcp-feedback"
|
name = "fastmcp-feedback"
|
||||||
version = "2026.10.01" # YYYYMMDD.patch versioning - update for each release
|
version = "2026.10.01.3" # YYYYMMDD.patch versioning - update for each release
|
||||||
description = "Production-ready feedback collection system for FastMCP servers"
|
description = "Production-ready feedback collection system for FastMCP servers"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
license = "MIT"
|
license = "MIT"
|
||||||
@@ -83,7 +83,7 @@ all = ["fastmcp-feedback[postgresql,mysql,dev]"]
|
|||||||
|
|
||||||
[project.urls]
|
[project.urls]
|
||||||
Homepage = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback"
|
Homepage = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback"
|
||||||
Documentation = "https://fastmcp-feedback.l.supported.systems"
|
Documentation = "https://fastmcp-feedback.supported.systems"
|
||||||
Repository = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback.git"
|
Repository = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback.git"
|
||||||
Issues = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/issues"
|
Issues = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/issues"
|
||||||
Changelog = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/blob/main/CHANGELOG.md"
|
Changelog = "https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/blob/main/CHANGELOG.md"
|
||||||
|
|||||||
@@ -44,7 +44,7 @@ Quick Start:
|
|||||||
import logging
|
import logging
|
||||||
|
|
||||||
# Version information - CalVer YYYY.MM.DD
|
# Version information - CalVer YYYY.MM.DD
|
||||||
__version__ = "2026.10.01"
|
__version__ = "2026.10.01.3"
|
||||||
__author__ = "Ryan Malloy"
|
__author__ = "Ryan Malloy"
|
||||||
__email__ = "ryan@supported.systems"
|
__email__ = "ryan@supported.systems"
|
||||||
|
|
||||||
|
|||||||
@@ -13,21 +13,21 @@ await mw.similar("render timed out")
|
|||||||
await backfill_embeddings(mw.embedding_sink) # rows stored before embedding
|
await backfill_embeddings(mw.embedding_sink) # rows stored before embedding
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from .backfill import backfill_embeddings, backfill_feedback
|
import importlib
|
||||||
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
from .classify import (
|
from .classify import (
|
||||||
DEFAULT_RESULT_CLASSIFIER,
|
DEFAULT_RESULT_CLASSIFIER,
|
||||||
ResultClassifier,
|
ResultClassifier,
|
||||||
soft_error_classifier,
|
soft_error_classifier,
|
||||||
)
|
)
|
||||||
from .dispatch import RecordDispatcher
|
from .constants import (
|
||||||
from .embeddings import (
|
DEFAULT_EMBEDDING_DIM,
|
||||||
DEFAULT_EVENT_TEXT_KEYS,
|
DEFAULT_EVENT_TEXT_KEYS,
|
||||||
Embedder,
|
DEFAULT_TABLE_PREFIX,
|
||||||
EmbeddingSink,
|
EMBEDDING_SOURCES,
|
||||||
EmbeddingText,
|
|
||||||
OpenAIEmbedder,
|
|
||||||
extract_texts,
|
|
||||||
)
|
)
|
||||||
|
from .dispatch import RecordDispatcher
|
||||||
from .events import EventRecord
|
from .events import EventRecord
|
||||||
from .middleware import MODE_ENV_VAR, MODES, InstrumentationMiddleware, instrument
|
from .middleware import MODE_ENV_VAR, MODES, InstrumentationMiddleware, instrument
|
||||||
from .record import ToolCallRecord
|
from .record import ToolCallRecord
|
||||||
@@ -38,14 +38,75 @@ from .redaction import (
|
|||||||
Redactor,
|
Redactor,
|
||||||
)
|
)
|
||||||
from .sinks import CallbackSink, JsonLinesSink, MemorySink, Sink
|
from .sinks import CallbackSink, JsonLinesSink, MemorySink, Sink
|
||||||
from .storage import (
|
|
||||||
DEFAULT_EMBEDDING_DIM,
|
if TYPE_CHECKING:
|
||||||
DEFAULT_TABLE_PREFIX,
|
from .backfill import backfill_embeddings, backfill_feedback
|
||||||
EMBEDDING_SOURCES,
|
from .embeddings import (
|
||||||
DatabaseSink,
|
Embedder,
|
||||||
EmbeddingVector,
|
EmbeddingSink,
|
||||||
build_metadata,
|
EmbeddingText,
|
||||||
)
|
OpenAIEmbedder,
|
||||||
|
extract_texts,
|
||||||
|
)
|
||||||
|
from .storage import DatabaseSink, EmbeddingVector, build_metadata
|
||||||
|
|
||||||
|
# Names whose modules need the async database stack (sqlalchemy[asyncio],
|
||||||
|
# which brings greenlet). Imported on first access, so instrument() and the
|
||||||
|
# plain sinks work on a base install.
|
||||||
|
_LAZY = {
|
||||||
|
"DatabaseSink": "storage",
|
||||||
|
"EmbeddingVector": "storage",
|
||||||
|
"build_metadata": "storage",
|
||||||
|
"Embedder": "embeddings",
|
||||||
|
"EmbeddingSink": "embeddings",
|
||||||
|
"EmbeddingText": "embeddings",
|
||||||
|
"OpenAIEmbedder": "embeddings",
|
||||||
|
"extract_texts": "embeddings",
|
||||||
|
"backfill_embeddings": "backfill",
|
||||||
|
"backfill_feedback": "backfill",
|
||||||
|
}
|
||||||
|
_EXTRA = {
|
||||||
|
"storage": "instrumentation",
|
||||||
|
"embeddings": "instrumentation,embeddings",
|
||||||
|
"backfill": "instrumentation,embeddings",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _missing_module(exc: BaseException) -> str:
|
||||||
|
seen: BaseException | None = exc
|
||||||
|
while seen is not None:
|
||||||
|
if isinstance(seen, ImportError) and seen.name:
|
||||||
|
return seen.name
|
||||||
|
seen = seen.__cause__ or seen.__context__
|
||||||
|
return "a dependency"
|
||||||
|
|
||||||
|
|
||||||
|
def __getattr__(name: str) -> Any:
|
||||||
|
module = _LAZY.get(name)
|
||||||
|
if module is None:
|
||||||
|
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||||
|
try:
|
||||||
|
# Checked first because SQLAlchemy reports a missing greenlet only on
|
||||||
|
# the first import of its asyncio module; a retry would then succeed
|
||||||
|
# and leave an engine that fails at runtime.
|
||||||
|
importlib.import_module("greenlet")
|
||||||
|
value = getattr(importlib.import_module(f".{module}", __name__), name)
|
||||||
|
except ImportError as exc:
|
||||||
|
extra = _EXTRA[module]
|
||||||
|
missing = _missing_module(exc)
|
||||||
|
raise ImportError(
|
||||||
|
f"fastmcp_feedback.instrumentation.{name} needs the async database "
|
||||||
|
f"stack, and {missing} is not installed. Install it with: "
|
||||||
|
f'uv add "fastmcp-feedback[{extra}]" '
|
||||||
|
f'(or pip install "fastmcp-feedback[{extra}]")'
|
||||||
|
) from exc
|
||||||
|
globals()[name] = value
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def __dir__() -> list[str]:
|
||||||
|
return sorted(set(globals()) | set(_LAZY))
|
||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"CallbackSink",
|
"CallbackSink",
|
||||||
|
|||||||
@@ -75,11 +75,19 @@ def soft_error_classifier(
|
|||||||
so ``{"error": None}`` and ``{"error": 0.02}`` stay ok. ``error_keys=()``
|
so ``{"error": None}`` and ``{"error": 0.02}`` stay ok. ``error_keys=()``
|
||||||
turns this off.
|
turns this off.
|
||||||
|
|
||||||
|
A result that says it succeeded, with any of ``flag_keys`` exactly True,
|
||||||
|
skips only that last check: ``{"ok": True, "error": "deprecated arg"}``
|
||||||
|
is ok, the string being a warning. A failing status, a False flag and
|
||||||
|
``isError`` still flag it. Truthy non-bools such as ``1`` or ``"yes"``
|
||||||
|
are not an explicit success.
|
||||||
|
|
||||||
>>> classify = soft_error_classifier(extra_statuses=("unknown_target",))
|
>>> classify = soft_error_classifier(extra_statuses=("unknown_target",))
|
||||||
>>> classify("render", {"status": "unknown_target", "error": "no such object"})
|
>>> classify("render", {"status": "unknown_target", "error": "no such object"})
|
||||||
'status=unknown_target: no such object'
|
'status=unknown_target: no such object'
|
||||||
>>> classify("speak", {"played": False, "error": "output stalled"})
|
>>> classify("speak", {"played": False, "error": "output stalled"})
|
||||||
'error: output stalled'
|
'error: output stalled'
|
||||||
|
>>> print(classify("speak", {"ok": True, "error": "deprecated arg"}))
|
||||||
|
None
|
||||||
"""
|
"""
|
||||||
bad = frozenset(s.lower() for s in (*statuses, *extra_statuses))
|
bad = frozenset(s.lower() for s in (*statuses, *extra_statuses))
|
||||||
status_keys = tuple(status_keys)
|
status_keys = tuple(status_keys)
|
||||||
@@ -100,6 +108,8 @@ def soft_error_classifier(
|
|||||||
if result.get(key) is True:
|
if result.get(key) is True:
|
||||||
detail = _message(result) or _content_text(result.get("content"))
|
detail = _message(result) or _content_text(result.get("content"))
|
||||||
return _describe("isError", detail)
|
return _describe("isError", detail)
|
||||||
|
if any(result.get(key) is True for key in flag_keys):
|
||||||
|
return None
|
||||||
for key in error_keys:
|
for key in error_keys:
|
||||||
value = result.get(key)
|
value = result.get(key)
|
||||||
if isinstance(value, str) and value.strip():
|
if isinstance(value, str) and value.strip():
|
||||||
|
|||||||
@@ -0,0 +1,12 @@
|
|||||||
|
"""Defaults shared by the storage and embedding modules.
|
||||||
|
|
||||||
|
Kept apart from them so these import without the async database stack.
|
||||||
|
"""
|
||||||
|
|
||||||
|
DEFAULT_TABLE_PREFIX = "ffb_"
|
||||||
|
DEFAULT_EMBEDDING_DIM = 1024
|
||||||
|
# source_type values in {prefix}embeddings. Feedback rows survive pruning.
|
||||||
|
EMBEDDING_SOURCES = ("feedback", "call_error", "event", "llm")
|
||||||
|
FEEDBACK_SOURCE = "feedback"
|
||||||
|
# Attribute keys whose text an EmbeddingSink embeds from events.
|
||||||
|
DEFAULT_EVENT_TEXT_KEYS = ("error_excerpt", "error", "message", "reason")
|
||||||
@@ -20,16 +20,15 @@ from typing import Any, Protocol, runtime_checkable
|
|||||||
|
|
||||||
from sqlalchemy import Float, and_, literal, not_, select, tuple_
|
from sqlalchemy import Float, and_, literal, not_, select, tuple_
|
||||||
|
|
||||||
|
from .constants import DEFAULT_EVENT_TEXT_KEYS, EMBEDDING_SOURCES
|
||||||
from .events import EventRecord
|
from .events import EventRecord
|
||||||
from .record import ToolCallRecord
|
from .record import ToolCallRecord
|
||||||
from .redaction import Redactor
|
from .redaction import Redactor
|
||||||
from .sinks import Record
|
from .sinks import Record
|
||||||
from .storage import EMBEDDING_SOURCES, EmbeddingVector
|
from .storage import EmbeddingVector
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
DEFAULT_EVENT_TEXT_KEYS = ("error_excerpt", "error", "message", "reason")
|
|
||||||
|
|
||||||
|
|
||||||
@runtime_checkable
|
@runtime_checkable
|
||||||
class Embedder(Protocol):
|
class Embedder(Protocol):
|
||||||
|
|||||||
@@ -8,12 +8,13 @@ import json
|
|||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
import random as _random
|
import random as _random
|
||||||
|
import sys
|
||||||
import time
|
import time
|
||||||
import uuid
|
import uuid
|
||||||
from collections import deque
|
from collections import deque
|
||||||
from collections.abc import Awaitable, Callable, Iterable, Mapping, Sequence
|
from collections.abc import Awaitable, Callable, Iterable, Mapping, Sequence
|
||||||
from datetime import UTC, datetime, timedelta
|
from datetime import UTC, datetime, timedelta
|
||||||
from typing import Any
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
from fastmcp import FastMCP
|
from fastmcp import FastMCP
|
||||||
from fastmcp.server.middleware import Middleware, MiddlewareContext
|
from fastmcp.server.middleware import Middleware, MiddlewareContext
|
||||||
@@ -27,7 +28,6 @@ from .correlation import (
|
|||||||
select_calls,
|
select_calls,
|
||||||
)
|
)
|
||||||
from .dispatch import RecordDispatcher
|
from .dispatch import RecordDispatcher
|
||||||
from .embeddings import EmbeddingSink
|
|
||||||
from .events import (
|
from .events import (
|
||||||
MAX_KEY_CHARS,
|
MAX_KEY_CHARS,
|
||||||
MAX_KIND_CHARS,
|
MAX_KIND_CHARS,
|
||||||
@@ -38,7 +38,10 @@ from .events import (
|
|||||||
from .record import ToolCallRecord
|
from .record import ToolCallRecord
|
||||||
from .redaction import Redactor
|
from .redaction import Redactor
|
||||||
from .sinks import JsonLinesSink, Sink
|
from .sinks import JsonLinesSink, Sink
|
||||||
from .storage import DatabaseSink
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from .embeddings import EmbeddingSink
|
||||||
|
from .storage import DatabaseSink
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -237,7 +240,7 @@ class InstrumentationMiddleware(Middleware):
|
|||||||
sinks = list(sinks) if sinks is not None else [JsonLinesSink()]
|
sinks = list(sinks) if sinks is not None else [JsonLinesSink()]
|
||||||
for sink in sinks:
|
for sink in sinks:
|
||||||
# Embedded text gets this middleware's redaction rules too.
|
# Embedded text gets this middleware's redaction rules too.
|
||||||
if isinstance(sink, EmbeddingSink) and sink.redactor is None:
|
if _is_sink(sink, "embeddings", "EmbeddingSink") and sink.redactor is None:
|
||||||
sink.redactor = self.redactor
|
sink.redactor = self.redactor
|
||||||
self.dispatcher = RecordDispatcher(
|
self.dispatcher = RecordDispatcher(
|
||||||
sinks,
|
sinks,
|
||||||
@@ -573,11 +576,9 @@ class InstrumentationMiddleware(Middleware):
|
|||||||
return rows[-limit:] if limit > 0 else []
|
return rows[-limit:] if limit > 0 else []
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def embedding_sink(self) -> EmbeddingSink | None:
|
def embedding_sink(self) -> "EmbeddingSink | None":
|
||||||
"""The first EmbeddingSink among the sinks, if any."""
|
"""The first EmbeddingSink among the sinks, if any."""
|
||||||
return next(
|
return _first_sink(self.dispatcher.sinks, "embeddings", "EmbeddingSink")
|
||||||
(s for s in self.dispatcher.sinks if isinstance(s, EmbeddingSink)), None
|
|
||||||
)
|
|
||||||
|
|
||||||
async def similar(
|
async def similar(
|
||||||
self,
|
self,
|
||||||
@@ -644,11 +645,9 @@ class InstrumentationMiddleware(Middleware):
|
|||||||
return []
|
return []
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def database_sink(self) -> DatabaseSink | None:
|
def database_sink(self) -> "DatabaseSink | None":
|
||||||
"""The first DatabaseSink among the sinks, if any."""
|
"""The first DatabaseSink among the sinks, if any."""
|
||||||
return next(
|
return _first_sink(self.dispatcher.sinks, "storage", "DatabaseSink")
|
||||||
(s for s in self.dispatcher.sinks if isinstance(s, DatabaseSink)), None
|
|
||||||
)
|
|
||||||
|
|
||||||
async def capture_recent_calls(
|
async def capture_recent_calls(
|
||||||
self,
|
self,
|
||||||
@@ -786,6 +785,21 @@ class InstrumentationMiddleware(Middleware):
|
|||||||
await self.dispatcher.aclose()
|
await self.dispatcher.aclose()
|
||||||
|
|
||||||
|
|
||||||
|
def _is_sink(sink: Any, module: str, name: str) -> bool:
|
||||||
|
"""isinstance against a sink class without importing its module.
|
||||||
|
|
||||||
|
storage and embeddings need the async database stack. When the module was
|
||||||
|
never imported, no instance of its class can exist, so the answer is no.
|
||||||
|
"""
|
||||||
|
loaded = sys.modules.get(f"{__package__}.{module}")
|
||||||
|
cls = getattr(loaded, name, None) if loaded is not None else None
|
||||||
|
return cls is not None and isinstance(sink, cls)
|
||||||
|
|
||||||
|
|
||||||
|
def _first_sink(sinks: Iterable[Any], module: str, name: str) -> Any:
|
||||||
|
return next((s for s in sinks if _is_sink(s, module, name)), None)
|
||||||
|
|
||||||
|
|
||||||
def _current_context() -> Any:
|
def _current_context() -> Any:
|
||||||
try:
|
try:
|
||||||
from fastmcp.server.dependencies import get_context
|
from fastmcp.server.dependencies import get_context
|
||||||
|
|||||||
@@ -40,22 +40,21 @@ from sqlalchemy.exc import DBAPIError
|
|||||||
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
|
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
|
||||||
from sqlalchemy.types import TypeDecorator
|
from sqlalchemy.types import TypeDecorator
|
||||||
|
|
||||||
|
from .constants import (
|
||||||
|
DEFAULT_EMBEDDING_DIM,
|
||||||
|
DEFAULT_TABLE_PREFIX,
|
||||||
|
FEEDBACK_SOURCE,
|
||||||
|
)
|
||||||
|
from .constants import EMBEDDING_SOURCES as EMBEDDING_SOURCES # re-export
|
||||||
from .events import MAX_KEY_CHARS, MAX_KIND_CHARS, EventRecord
|
from .events import MAX_KEY_CHARS, MAX_KIND_CHARS, EventRecord
|
||||||
from .record import ToolCallRecord
|
from .record import ToolCallRecord
|
||||||
from .sinks import Record
|
from .sinks import Record
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
DEFAULT_TABLE_PREFIX = "ffb_"
|
|
||||||
|
|
||||||
# none_as_null: a missing payload is SQL NULL, so "WHERE args IS NULL" works.
|
# none_as_null: a missing payload is SQL NULL, so "WHERE args IS NULL" works.
|
||||||
_JSON = JSON(none_as_null=True).with_variant(JSONB(none_as_null=True), "postgresql")
|
_JSON = JSON(none_as_null=True).with_variant(JSONB(none_as_null=True), "postgresql")
|
||||||
|
|
||||||
DEFAULT_EMBEDDING_DIM = 1024
|
|
||||||
# source_type values in {prefix}embeddings. Feedback rows survive pruning.
|
|
||||||
EMBEDDING_SOURCES = ("feedback", "call_error", "event", "llm")
|
|
||||||
FEEDBACK_SOURCE = "feedback"
|
|
||||||
|
|
||||||
|
|
||||||
class EmbeddingVector(TypeDecorator):
|
class EmbeddingVector(TypeDecorator):
|
||||||
"""pgvector ``VECTOR(dim)`` on PostgreSQL, a JSON list of floats elsewhere.
|
"""pgvector ``VECTOR(dim)`` on PostgreSQL, a JSON list of floats elsewhere.
|
||||||
|
|||||||
+105
@@ -0,0 +1,105 @@
|
|||||||
|
"""Exercise the instrumentation as a base install would, then print JSON.
|
||||||
|
|
||||||
|
Run in a fresh interpreter. greenlet and aiosqlite are made unimportable
|
||||||
|
before anything else loads, as on ``pip install fastmcp-feedback`` without
|
||||||
|
the ``instrumentation`` extra. Argument: a directory for the feedback database.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from importlib.metadata import version
|
||||||
|
|
||||||
|
# SQLAlchemy 2.1 refuses to import sqlalchemy.ext.asyncio without greenlet,
|
||||||
|
# once: a second import succeeds. 2.0 imports it and fails on first use.
|
||||||
|
# On 2.0 the module is blocked too, so both versions fail at import.
|
||||||
|
_blocked = ["greenlet", "aiosqlite"]
|
||||||
|
if tuple(int(p) for p in version("sqlalchemy").split(".")[:2]) < (2, 1):
|
||||||
|
_blocked.append("sqlalchemy.ext.asyncio")
|
||||||
|
for _name in _blocked:
|
||||||
|
sys.modules[_name] = None # import raises ModuleNotFoundError
|
||||||
|
|
||||||
|
import asyncio # noqa: E402
|
||||||
|
import io # noqa: E402
|
||||||
|
import json # noqa: E402
|
||||||
|
from pathlib import Path # noqa: E402
|
||||||
|
|
||||||
|
STORAGE = "fastmcp_feedback.instrumentation.storage"
|
||||||
|
EMBEDDINGS = "fastmcp_feedback.instrumentation.embeddings"
|
||||||
|
|
||||||
|
|
||||||
|
async def run(db_dir: Path) -> dict:
|
||||||
|
from fastmcp import Client, FastMCP
|
||||||
|
|
||||||
|
import fastmcp_feedback.instrumentation as inst
|
||||||
|
from fastmcp_feedback import add_feedback_tools
|
||||||
|
from fastmcp_feedback.instrumentation import (
|
||||||
|
DEFAULT_TABLE_PREFIX,
|
||||||
|
JsonLinesSink,
|
||||||
|
MemorySink,
|
||||||
|
instrument,
|
||||||
|
)
|
||||||
|
|
||||||
|
out: dict = {"prefix": DEFAULT_TABLE_PREFIX}
|
||||||
|
app = FastMCP("base-install")
|
||||||
|
|
||||||
|
@app.tool
|
||||||
|
def add(a: int, b: int) -> int:
|
||||||
|
return a + b
|
||||||
|
|
||||||
|
memory = MemorySink()
|
||||||
|
lines = io.StringIO()
|
||||||
|
mw = instrument(app, [memory, JsonLinesSink(lines)], mode="full")
|
||||||
|
add_feedback_tools(
|
||||||
|
app, database_url=f"sqlite:///{db_dir / 'fb.db'}", instrumentation=mw
|
||||||
|
)
|
||||||
|
|
||||||
|
async with Client(app) as client:
|
||||||
|
result = await client.call_tool("add", {"a": 2, "b": 3})
|
||||||
|
out["result"] = result.content[0].text
|
||||||
|
fb = await client.call_tool(
|
||||||
|
"submit_feedback",
|
||||||
|
{
|
||||||
|
"request": {
|
||||||
|
"type": "bug",
|
||||||
|
"title": "base install",
|
||||||
|
"description": "submitted without the extra",
|
||||||
|
"submitter": "worker",
|
||||||
|
}
|
||||||
|
},
|
||||||
|
)
|
||||||
|
out["feedback"] = json.loads(fb.content[0].text)
|
||||||
|
|
||||||
|
mw.record_event("job.done", key="j1", attrs={"n": 1})
|
||||||
|
await mw.flush()
|
||||||
|
out["tools"] = [r.tool for r in memory.records if hasattr(r, "tool")]
|
||||||
|
out["event_kinds"] = [r.kind for r in memory.records if hasattr(r, "kind")]
|
||||||
|
out["jsonl_lines"] = len(lines.getvalue().splitlines())
|
||||||
|
out["capture"] = await mw.capture_recent_calls()
|
||||||
|
out["similar"] = await mw.similar("anything")
|
||||||
|
out["similar_feedback"] = await mw.similar_feedback("bug-1")
|
||||||
|
out["feedback_context"] = await mw.feedback_context("bug-1")
|
||||||
|
out["link_feedback"] = await mw.link_feedback("bug-1", [])
|
||||||
|
out["events_for"] = len(await mw.events_for("j1"))
|
||||||
|
out["database_sink"] = mw.database_sink is None
|
||||||
|
out["embedding_sink"] = mw.embedding_sink is None
|
||||||
|
await mw.aclose()
|
||||||
|
|
||||||
|
errors = {}
|
||||||
|
for name in (
|
||||||
|
"DatabaseSink",
|
||||||
|
"EmbeddingSink",
|
||||||
|
"build_metadata",
|
||||||
|
"backfill_feedback",
|
||||||
|
):
|
||||||
|
for _attempt in range(2): # SQLAlchemy only complains the first time
|
||||||
|
try:
|
||||||
|
getattr(inst, name)
|
||||||
|
errors.setdefault(name, []).append(None)
|
||||||
|
except ImportError as exc:
|
||||||
|
errors.setdefault(name, []).append(str(exc))
|
||||||
|
out["errors"] = errors
|
||||||
|
out["loaded"] = sorted(m for m in (STORAGE, EMBEDDINGS) if m in sys.modules)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
print(json.dumps(asyncio.run(run(Path(sys.argv[1]))), default=str))
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
"""The instrumentation works on a base install, without the database extras.
|
||||||
|
|
||||||
|
The worker runs in a fresh interpreter with greenlet and aiosqlite made
|
||||||
|
unimportable, which is what ``pip install fastmcp-feedback`` leaves out.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
WORKER = Path(__file__).parent.parent / "fixtures" / "base_install_worker.py"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def report(tmp_path_factory) -> dict:
|
||||||
|
proc = subprocess.run(
|
||||||
|
[sys.executable, str(WORKER), str(tmp_path_factory.mktemp("base"))],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=60,
|
||||||
|
)
|
||||||
|
assert proc.returncode == 0, proc.stderr[-4000:]
|
||||||
|
return json.loads(proc.stdout.strip().splitlines()[-1])
|
||||||
|
|
||||||
|
|
||||||
|
def test_tool_calls_are_recorded(report):
|
||||||
|
assert report["result"] == "5"
|
||||||
|
assert report["tools"] == ["add", "submit_feedback"]
|
||||||
|
assert report["jsonl_lines"] == 4 # two calls, two events
|
||||||
|
|
||||||
|
|
||||||
|
def test_events_are_recorded(report):
|
||||||
|
assert report["event_kinds"] == ["feedback.submitted", "job.done"]
|
||||||
|
assert report["events_for"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_feedback_tools_work_alongside(report):
|
||||||
|
assert report["feedback"]["success"] is True
|
||||||
|
assert report["feedback"]["linked_calls"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_database_helpers_return_empty_defaults(report):
|
||||||
|
assert report["capture"] == []
|
||||||
|
assert report["similar"] == []
|
||||||
|
assert report["similar_feedback"] == []
|
||||||
|
assert report["feedback_context"] == []
|
||||||
|
assert report["link_feedback"] is False
|
||||||
|
assert report["database_sink"] is True
|
||||||
|
assert report["embedding_sink"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_constants_import_without_the_extra(report):
|
||||||
|
assert report["prefix"] == "ffb_"
|
||||||
|
|
||||||
|
|
||||||
|
def test_storage_modules_stay_unloaded(report):
|
||||||
|
assert report["loaded"] == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("name", "extra"),
|
||||||
|
[
|
||||||
|
("DatabaseSink", "[instrumentation]"),
|
||||||
|
("build_metadata", "[instrumentation]"),
|
||||||
|
("EmbeddingSink", "[instrumentation,embeddings]"),
|
||||||
|
("backfill_feedback", "[instrumentation,embeddings]"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_database_names_explain_the_missing_extra(report, name, extra):
|
||||||
|
# Asked twice: SQLAlchemy reports a missing greenlet only on first import.
|
||||||
|
for message in report["errors"][name]:
|
||||||
|
assert message is not None
|
||||||
|
assert "greenlet" in message
|
||||||
|
assert f'uv add "fastmcp-feedback{extra}"' in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_lazy_names_resolve_with_the_extra():
|
||||||
|
import fastmcp_feedback.instrumentation as inst
|
||||||
|
from fastmcp_feedback.instrumentation import DatabaseSink, storage
|
||||||
|
|
||||||
|
assert DatabaseSink is storage.DatabaseSink
|
||||||
|
assert set(inst._LAZY) <= set(inst.__all__)
|
||||||
|
assert set(inst.__all__) <= set(dir(inst))
|
||||||
|
for name in inst.__all__:
|
||||||
|
assert getattr(inst, name) is not None
|
||||||
|
with pytest.raises(AttributeError):
|
||||||
|
_ = inst.not_a_name
|
||||||
@@ -201,6 +201,70 @@ class TestClassifier:
|
|||||||
assert classify("t", {"ok": False, "error": "x"}) == "ok=False: x"
|
assert classify("t", {"ok": False, "error": "x"}) == "ok=False: x"
|
||||||
|
|
||||||
|
|
||||||
|
class TestExplicitSuccess:
|
||||||
|
"""A flag key that is exactly True turns an error string into a warning."""
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"result",
|
||||||
|
[
|
||||||
|
{"ok": True, "error": "deprecated arg"},
|
||||||
|
{"success": True, "error": "x"},
|
||||||
|
{"ok": True, "status": "unknown_target", "error": "fell back"},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_true_flag_with_error_string_is_ok(self, result):
|
||||||
|
assert DEFAULT_RESULT_CLASSIFIER("t", result) is None
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"result, expected",
|
||||||
|
[
|
||||||
|
({"ok": False, "error": "x"}, "ok=False: x"),
|
||||||
|
# Not exactly True, so not an explicit success.
|
||||||
|
({"ok": 1, "error": "x"}, "error: x"),
|
||||||
|
({"ok": "yes", "error": "x"}, "error: x"),
|
||||||
|
({"success": 1.0, "error": "x"}, "error: x"),
|
||||||
|
# No flag key: mcspeak's shape, as in 2026.10.01.
|
||||||
|
(
|
||||||
|
{"played": False, "error": "audio output stalled"},
|
||||||
|
"error: audio output stalled",
|
||||||
|
),
|
||||||
|
# A False flag beats a True one.
|
||||||
|
({"ok": True, "success": False, "error": "x"}, "success=False: x"),
|
||||||
|
# A failing status is more specific than the flag, so it still flags.
|
||||||
|
({"success": True, "status": "failed"}, "status=failed"),
|
||||||
|
({"ok": True, "state": "error", "error": "x"}, "state=error: x"),
|
||||||
|
({"ok": True, "isError": True, "content": ["boom"]}, "isError: boom"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_still_flagged(self, result, expected):
|
||||||
|
assert DEFAULT_RESULT_CLASSIFIER("t", result) == expected
|
||||||
|
|
||||||
|
def test_custom_flag_keys(self):
|
||||||
|
classify = soft_error_classifier(flag_keys=("passed",))
|
||||||
|
assert classify("t", {"passed": True, "error": "slow"}) is None
|
||||||
|
assert classify("t", {"passed": False, "error": "x"}) == "passed=False: x"
|
||||||
|
# ok is not a flag key here, so it does not suppress the error check.
|
||||||
|
assert classify("t", {"ok": True, "error": "x"}) == "error: x"
|
||||||
|
|
||||||
|
def test_no_flag_keys(self):
|
||||||
|
classify = soft_error_classifier(flag_keys=())
|
||||||
|
assert classify("t", {"ok": True, "error": "x"}) == "error: x"
|
||||||
|
|
||||||
|
async def test_recorded_as_ok(self):
|
||||||
|
record = await record_for(
|
||||||
|
"respond", {"payload": {"ok": True, "error": "deprecated arg"}}
|
||||||
|
)
|
||||||
|
assert (record.outcome, record.ok, record.error_message) == ("ok", True, None)
|
||||||
|
|
||||||
|
async def test_truthy_flag_recorded_as_soft_error(self):
|
||||||
|
record = await record_for("respond", {"payload": {"ok": 1, "error": "x"}})
|
||||||
|
assert (record.outcome, record.ok, record.error_message) == (
|
||||||
|
"soft_error",
|
||||||
|
False,
|
||||||
|
"error: x",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class TestMiddleware:
|
class TestMiddleware:
|
||||||
async def test_status_failed_is_soft_error(self):
|
async def test_status_failed_is_soft_error(self):
|
||||||
record = await record_for("render", {"status": "failed"})
|
record = await record_for("render", {"status": "failed"})
|
||||||
|
|||||||
@@ -601,7 +601,7 @@ wheels = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "fastmcp-feedback"
|
name = "fastmcp-feedback"
|
||||||
version = "2026.10.1"
|
version = "2026.10.1.3"
|
||||||
source = { editable = "." }
|
source = { editable = "." }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "fastmcp" },
|
{ name = "fastmcp" },
|
||||||
|
|||||||
Reference in New Issue
Block a user