Compare commits
14
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3abd5c6311 | ||
|
|
472071730e | ||
|
|
9ddd929a13 | ||
|
|
052862f4fc | ||
|
|
975c24c766 | ||
|
|
a530634eb9 | ||
|
|
4b16764faa | ||
|
|
58e531019d | ||
|
|
00fae0b46b | ||
|
|
10b1eea86c | ||
|
|
39d775a0cd | ||
|
|
a052ca3b4b | ||
|
|
2e1278d81f | ||
|
|
ddf23d10e4 |
@@ -0,0 +1,67 @@
|
||||
name: Tests
|
||||
|
||||
# fastmcp-feedback declares fastmcp>=2.12.2,<5, so every version family in
|
||||
# that range gets the full suite. A user on PyPI gets the newest FastMCP the
|
||||
# range allows, not what uv.lock pins, which is how 2026.01.12.1 shipped with
|
||||
# composition broken on 2.14 and 4.x without anyone noticing.
|
||||
|
||||
on:
|
||||
push:
|
||||
pull_request:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
# Weekly, so a new FastMCP release is caught even when nobody pushes
|
||||
- cron: '17 9 * * 1'
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
fastmcp: ['==2.12.2', '==2.14.3', '>=3.3,<4', '>=4,<5']
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
- name: Install against fastmcp${{ matrix.fastmcp }}
|
||||
run: |
|
||||
uv venv
|
||||
uv pip install -e ".[dev]" "fastmcp${{ matrix.fastmcp }}"
|
||||
uv pip show fastmcp | grep -i '^version'
|
||||
|
||||
- name: Lint
|
||||
run: uv run --no-sync ruff check src/ examples/ tests/_compat.py tests/unit/test_instrumentation.py
|
||||
|
||||
- name: Test
|
||||
run: uv run --no-sync pytest -q --no-cov
|
||||
|
||||
# Newest FastMCP with no ceiling. Allowed to fail: it exists to say
|
||||
# "a new major version is out and here is what breaks" before users do.
|
||||
canary:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
continue-on-error: true
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
- name: Install newest FastMCP, ignoring our ceiling
|
||||
run: |
|
||||
uv venv
|
||||
uv pip install fastmcp "sqlalchemy[asyncio]" aiosqlite pydantic \
|
||||
pytest pytest-asyncio pytest-cov pytest-mock pytest-html
|
||||
uv pip install --no-deps -e .
|
||||
uv pip show fastmcp | grep -i '^version'
|
||||
|
||||
- name: Test
|
||||
run: uv run --no-sync pytest -q --no-cov
|
||||
+157
@@ -5,6 +5,163 @@ All notable changes to this project will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Calendar Versioning](https://calver.org/) (YYYY.MM.DD).
|
||||
|
||||
## [2026.09.27.4] - 2026-09-27
|
||||
|
||||
Count errors that tools return as data instead of raising.
|
||||
|
||||
### Added
|
||||
- Records have an `outcome`: `ok`, `soft_error` (the tool returned a result
|
||||
flagged as a failure) or `error` (the tool raised). Soft errors get
|
||||
`error_type="SoftError"` and an `error_message` built from the result, such
|
||||
as `status=failed: disk full`, redacted and truncated to `max_error_chars`.
|
||||
- `soft_error_classifier(statuses=..., status_keys=..., flag_keys=...,
|
||||
extra_statuses=...)` builds a classifier that flags dict results where `ok`
|
||||
or `success` is `False`, where `status` or `state` is `error`, `failed` or
|
||||
`failure`, or that the tool marked `isError` without raising.
|
||||
`DEFAULT_RESULT_CLASSIFIER` is `soft_error_classifier()`.
|
||||
- `InstrumentationMiddleware(..., result_classifier=...)` takes any
|
||||
`(tool, result) -> str | None`; `None` turns classification off. A
|
||||
classifier that raises is logged and the call is recorded as `ok`.
|
||||
- `feedback_context()` rows include `outcome`.
|
||||
|
||||
### Changed
|
||||
- `ok` is now `outcome == "ok"`, so it is also false for soft errors. Existing
|
||||
`WHERE NOT ok` queries and error rates will rise; filter on
|
||||
`outcome = 'error'` for exceptions only. `CallLink.ok` follows.
|
||||
- The package's own feedback tools answer `{"success": False, ...}` on failure,
|
||||
so those calls are now recorded as soft errors.
|
||||
|
||||
### Migration
|
||||
- New nullable, indexed `outcome` column (`String(16)`) on
|
||||
`{prefix}tool_calls`. Hosts that manage the schema with Alembic need a
|
||||
migration adding it. `create_tables=True` does not alter a table that
|
||||
already exists, so add the column by hand there
|
||||
(`ALTER TABLE ffb_tool_calls ADD COLUMN outcome VARCHAR(16)` plus an index).
|
||||
Until the column exists, `DatabaseSink` writes fail and those records are
|
||||
dropped (logged). Rows from earlier versions keep NULL.
|
||||
|
||||
## [2026.09.27.3] - 2026-09-27
|
||||
|
||||
Link a feedback item to the tool calls that preceded it.
|
||||
|
||||
### Added
|
||||
- `InstrumentationMiddleware.capture_recent_calls(ctx, n=20)` returns the
|
||||
calls that led up to now. It matches the same client session first
|
||||
(`session` rule), then fills remaining slots with calls by the same
|
||||
`user_sub` inside `window` (`user_window` rule, 15 minutes by default), and
|
||||
each link records which rule matched.
|
||||
- Calls are kept in an in-memory buffer as they happen, so calls still
|
||||
waiting in the sink queue count. With a `DatabaseSink` the lookup falls back
|
||||
to the database, which covers calls from other workers and from before a
|
||||
restart.
|
||||
- `link_feedback(ref, links)` stores links against a feedback id of any
|
||||
format (`bug-7Q2X`, `42`) in a new `{prefix}feedback_call_links` table.
|
||||
`build_metadata()` includes it; hosts that manage the schema with Alembic
|
||||
need a new migration.
|
||||
- `feedback_context(ref)` returns the linked calls in order with tool,
|
||||
outcome, error type and matching rule, filling in calls not yet written
|
||||
from memory.
|
||||
- `add_feedback_tools(..., instrumentation=mw)`: `submit_feedback` links the
|
||||
preceding calls automatically and returns `linked_calls` in its response.
|
||||
- `session_key(ctx)` gives a per-client key that stays stable across calls:
|
||||
the HTTP `mcp-session-id` header when present, then one key per process for
|
||||
stdio, then `ctx.session_id`.
|
||||
|
||||
### Fixed
|
||||
- With `create_tables=True`, the background writer and `link_feedback` could
|
||||
both run `create_all` at once and fail with "table already exists". Table
|
||||
creation is now serialized by a lock.
|
||||
|
||||
### Note
|
||||
- Clients on MCP protocol 2026-07-28 (Claude Code, FastMCP 4's `Client`)
|
||||
open no HTTP sessions, so their calls link by user only. HTTP servers should
|
||||
configure an `identity_resolver`. Stdio servers link by session with every
|
||||
client.
|
||||
|
||||
## [2026.09.27.2] - 2026-09-27
|
||||
|
||||
### Fixed
|
||||
- `DatabaseSink` stored missing `args`/`result` (every `meta` mode row) as the
|
||||
JSON literal `null`, so `WHERE args IS NULL` matched nothing. They are now
|
||||
SQL `NULL`. No schema change; rows written by 2026.09.27.1 keep JSON `null`
|
||||
and can be backfilled with
|
||||
`UPDATE ffb_tool_calls SET args = NULL WHERE args = 'null'::jsonb` (same for
|
||||
`result`). Reported from production by blender-mcp.
|
||||
|
||||
## [2026.09.27.1] - 2026-09-27
|
||||
|
||||
Per-call instrumentation for any FastMCP server.
|
||||
|
||||
### Added
|
||||
- `fastmcp_feedback.instrumentation`: `InstrumentationMiddleware` and
|
||||
`instrument(app)` record every tool call on the server (tool, timing,
|
||||
outcome, original error type, session and request ids, argument and result
|
||||
sizes). Modes `off`, `meta` (default) and `full`, set by argument or
|
||||
`FEEDBACK_INSTRUMENTATION_MODE`; `meta_only_tools` keeps chosen tools at
|
||||
`meta`.
|
||||
- Recording never slows or breaks a call: records reach sinks from a
|
||||
background task through a bounded queue and are dropped (and counted) when it
|
||||
is full. Sink and hook errors are logged, never raised; async hooks are cut
|
||||
off after `hook_timeout`.
|
||||
- `Redactor`: secrets removed from arguments, results, error text and hook
|
||||
output, by key at any depth and by value pattern (JWT, bearer, `sk-`,
|
||||
GitHub, Slack, AWS, PyPI and `bmcp_` tokens, `password=...` in text).
|
||||
- Sinks: `JsonLinesSink` (stderr), `CallbackSink`, `MemorySink`, and
|
||||
`DatabaseSink` on async SQLAlchemy. `DatabaseSink` accepts the host's
|
||||
`AsyncEngine`, writes to a prefixed `ffb_tool_calls` table, never creates
|
||||
tables unless asked, and `build_metadata()` plugs the table into a host's
|
||||
Alembic `env.py`.
|
||||
- `identity_resolver` and `enricher` hooks fill `user_sub`, `caller_kind`,
|
||||
`client_id`, `server_version` and free-form `identity`/`extra` fields.
|
||||
- `instrumentation` extra: `sqlalchemy[asyncio]` and `aiosqlite`.
|
||||
- CI covers FastMCP 3.x as well as 2.12, 2.14 and 4.x.
|
||||
|
||||
## [2026.09.27] - 2026-09-27
|
||||
|
||||
FastMCP 4 support, and a composition bug that affected every earlier release.
|
||||
|
||||
### Fixed
|
||||
- `compose_feedback_services()` composed nothing on FastMCP 2.14. It called
|
||||
`import_server()`, which became async in 2.14, without awaiting it, then
|
||||
logged success anyway. On FastMCP 4, where `import_server()` no longer
|
||||
exists, it raised `AttributeError`. It now uses `mount()`, which is
|
||||
synchronous and available on every supported version.
|
||||
- The test suite could not start (a pytest-html hook with pytest-html not
|
||||
installed), so it had not run since the package was split out. It runs
|
||||
again: 407 tests.
|
||||
|
||||
### Changed
|
||||
- Dependency is now `fastmcp>=2.12.2,<5`. Earlier releases had no ceiling,
|
||||
so a fresh install picked up FastMCP 4 untested.
|
||||
- `examples/` moved into this repository and ships in the sdist.
|
||||
`create_demo_server()` accepts an optional `database_url`. Examples list
|
||||
and call tools through `fastmcp.Client`, which works on 2.x and 4.x.
|
||||
|
||||
### Added
|
||||
- CI (Gitea Actions) runs the suite against FastMCP 2.12.2, 2.14.3 and 4.x
|
||||
on every push and weekly, plus a non-blocking job against the newest
|
||||
FastMCP with no ceiling.
|
||||
- A composition test using real servers instead of mocks.
|
||||
|
||||
### Documentation
|
||||
- README rewritten against the code and every Python example executed.
|
||||
Previous docs described things that do not exist: four tools (there are
|
||||
five, including `delete_feedback`), `get_feedback_stats` (it is
|
||||
`get_feedback_statistics`), `InsightsMixin`, an `enable_insights`
|
||||
parameter, feedback types `bug_report`/`feature_request`/`general` (they
|
||||
are `bug`/`feature`/`improvement`/`question`), and a
|
||||
`new -> reviewing -> planned` workflow (it is
|
||||
`open -> in_progress -> resolved -> closed`).
|
||||
- The default database is in-memory SQLite (`sqlite:///:memory:`), not
|
||||
`sqlite:///feedback.db` as previously documented. Pass `database_url` to
|
||||
keep feedback across restarts.
|
||||
|
||||
## [2026.01.12.1] - 2026-01-12
|
||||
|
||||
### Fixed
|
||||
- `ImportError` on install: `mixins/base.py` did not re-export the names
|
||||
the other mixin modules import from it.
|
||||
|
||||
## [2026.01.12] - 2026-01-12
|
||||
|
||||
### Added
|
||||
|
||||
@@ -1,17 +1,19 @@
|
||||
# FastMCP Feedback
|
||||
|
||||
Production-ready feedback collection system for [FastMCP](https://gofastmcp.com) servers.
|
||||
Instrumentation and QA feedback for [FastMCP](https://gofastmcp.com) servers.
|
||||
|
||||
Add structured feedback collection (bug reports, feature requests, general feedback) to any FastMCP server with a single line of code.
|
||||
Record every tool call your server handles, and let people and models report problems with it, each with a line or two of code.
|
||||
|
||||
## Features
|
||||
|
||||
- **Per-Call Instrumentation** - Middleware records every tool call (timing, outcome, identity, payload sizes) without slowing or breaking it
|
||||
- **Secret Redaction** - Tokens and passwords removed from recorded arguments, results and errors by default
|
||||
- **One-Line Integration** - Add feedback tools to any FastMCP server instantly
|
||||
- **Modular Mixins** - Selective tool registration for fine-grained control
|
||||
- **Multi-Database Support** - SQLite (dev), PostgreSQL, MySQL (production)
|
||||
- **Privacy-Compliant Analytics** - Optional usage insights without sensitive data
|
||||
- **Type-Safe** - Full Pydantic validation and type hints
|
||||
- **Well-Tested** - 97%+ test coverage
|
||||
- **Well-Tested** - 98% coverage, run against FastMCP 2.12, 2.14, 3.x and 4.x
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -67,59 +69,304 @@ from fastmcp import FastMCP
|
||||
from fastmcp_feedback import add_feedback_tools
|
||||
|
||||
app = FastMCP("My Server")
|
||||
add_feedback_tools(app) # Adds 4 feedback tools instantly!
|
||||
add_feedback_tools(app, database_url="sqlite:///feedback.db")
|
||||
|
||||
# That's it! Your server now has:
|
||||
# - submit_feedback: Collect bug reports, feature requests, general feedback
|
||||
# - list_feedback: Retrieve and filter feedback items
|
||||
# - get_feedback_stats: Aggregate statistics
|
||||
# - update_feedback_status: Workflow management (new → reviewing → resolved)
|
||||
# Your server now has five tools:
|
||||
# - submit_feedback: bug reports, feature requests, improvements, questions
|
||||
# - list_feedback: page through feedback, filtered by type or status
|
||||
# - get_feedback_statistics: counts by type and status, plus the last 7 days
|
||||
# - update_feedback_status: move an item through open -> in_progress -> resolved -> closed
|
||||
# - delete_feedback: remove an item
|
||||
```
|
||||
|
||||
### With Custom Database
|
||||
Pass a `database_url`. Without one the tools use an in-memory SQLite
|
||||
database, which is handy for tests but loses everything when the process exits.
|
||||
|
||||
### PostgreSQL
|
||||
|
||||
```python
|
||||
# SQLite with custom path
|
||||
add_feedback_tools(app, database_url="sqlite:///./data/feedback.db")
|
||||
|
||||
# PostgreSQL for production
|
||||
add_feedback_tools(app, database_url="postgresql://user:pass@localhost/mydb")
|
||||
add_feedback_tools(app, database_url="postgresql://user:pass@db.example.com/feedback")
|
||||
```
|
||||
|
||||
### With Analytics Enabled
|
||||
Install the driver with `uv add "fastmcp-feedback[postgresql]"`.
|
||||
|
||||
### Prefixed Tool Names
|
||||
|
||||
```python
|
||||
add_feedback_tools(app, enable_insights=True)
|
||||
# Enables privacy-compliant usage tracking
|
||||
add_feedback_tools(app, database_url="sqlite:///feedback.db", prefix="support")
|
||||
# support_submit_feedback, support_list_feedback, ...
|
||||
```
|
||||
|
||||
The prefix and tool name are joined with `separator` (default `_`), so leave
|
||||
the trailing underscore off the prefix.
|
||||
|
||||
### Usage Analytics
|
||||
|
||||
```python
|
||||
from fastmcp_feedback import FeedbackInsights
|
||||
|
||||
add_feedback_tools(
|
||||
app,
|
||||
database_url="sqlite:///feedback.db",
|
||||
insights=FeedbackInsights(enabled=True),
|
||||
)
|
||||
```
|
||||
|
||||
Analytics are off unless you pass `FeedbackInsights(enabled=True)` or set
|
||||
`FEEDBACK_INSIGHTS_ENABLED=true`. When enabled they record tool usage and
|
||||
submission metadata (type, lengths, timing), never feedback text or contact details.
|
||||
|
||||
## Instrumentation
|
||||
|
||||
Record every tool call on your server:
|
||||
|
||||
```python
|
||||
from fastmcp import FastMCP
|
||||
from fastmcp_feedback.instrumentation import instrument
|
||||
|
||||
app = FastMCP("My Server")
|
||||
instrument(app) # one JSON line per tool call on stderr
|
||||
```
|
||||
|
||||
Each record has the tool name, start time, duration, outcome and error type,
|
||||
session and request ids, argument and result sizes, and whatever your hooks
|
||||
add. The recording happens in middleware, so it covers every tool on the server,
|
||||
not just the feedback ones.
|
||||
|
||||
It never slows or breaks a tool call. Records go to the sinks from a background
|
||||
task through a bounded queue, and when the queue is full they are dropped and
|
||||
counted instead of waited on. Errors in a sink or a hook are logged and swallowed.
|
||||
|
||||
### Modes
|
||||
|
||||
| Mode | Records |
|
||||
|------|---------|
|
||||
| `off` | Nothing |
|
||||
| `meta` (default) | Tool, timing, outcome, identity, payload sizes |
|
||||
| `full` | Also the arguments and result, redacted |
|
||||
|
||||
Set it with `mode=` or the `FEEDBACK_INSTRUMENTATION_MODE` environment variable.
|
||||
`meta_only_tools={"run_code", "create_token"}` keeps chosen tools at `meta`
|
||||
even in `full` mode.
|
||||
|
||||
### Redaction
|
||||
|
||||
Secrets are removed from arguments, results, error messages and hook output
|
||||
before anything reaches a sink. It matches keys at any depth (`password`,
|
||||
`token`, `authorization`, `client_secret`, `*_api_key`, ...) and secret-shaped
|
||||
values anywhere in text (JWTs, `Bearer ...`, `sk-...`, GitHub, Slack, AWS and
|
||||
PyPI tokens, `password=...` in error text). Add your own:
|
||||
|
||||
```python
|
||||
from fastmcp_feedback.instrumentation import Redactor, instrument
|
||||
|
||||
instrument(
|
||||
app,
|
||||
mode="full",
|
||||
redactor=Redactor(extra_keys=["license"], extra_patterns=[r"acme_[A-Za-z0-9]{32}"]),
|
||||
)
|
||||
```
|
||||
|
||||
### Sinks
|
||||
|
||||
```python
|
||||
from fastmcp_feedback.instrumentation import CallbackSink, DatabaseSink, JsonLinesSink, instrument
|
||||
|
||||
instrument(app, [
|
||||
JsonLinesSink(), # stderr
|
||||
DatabaseSink("sqlite+aiosqlite:///calls.db", create_tables=True),
|
||||
CallbackSink(lambda records: print(len(records))), # sync or async
|
||||
])
|
||||
```
|
||||
|
||||
`DatabaseSink` needs the `instrumentation` extra
|
||||
(`uv add "fastmcp-feedback[instrumentation]"`), plus `asyncpg` for Postgres.
|
||||
|
||||
### Using your own database and migrations
|
||||
|
||||
Pass the `AsyncEngine` you already have. Records go to a `ffb_tool_calls`
|
||||
table (the prefix is configurable), and nothing is created at startup unless you
|
||||
ask. To manage the table with Alembic, add it to your `env.py`:
|
||||
|
||||
```python
|
||||
from fastmcp_feedback.instrumentation import build_metadata
|
||||
|
||||
target_metadata = [Base.metadata, build_metadata(prefix="ffb_")]
|
||||
```
|
||||
|
||||
```python
|
||||
from fastmcp_feedback.instrumentation import DatabaseSink, instrument
|
||||
|
||||
sink = DatabaseSink(engine, prefix="ffb_") # your engine is never disposed
|
||||
instrument(app, [sink])
|
||||
```
|
||||
|
||||
### Errors returned as data
|
||||
|
||||
Many tools report failure in their result instead of raising, for example
|
||||
`{"status": "failed", "error": "disk full"}` or `{"success": False, ...}`. A
|
||||
result classifier catches these, so error rates count them. Each record has an
|
||||
`outcome`:
|
||||
|
||||
| `outcome` | `ok` | Meaning |
|
||||
|-----------|------|---------|
|
||||
| `ok` | true | The call succeeded |
|
||||
| `soft_error` | false | The tool returned a result the classifier flagged |
|
||||
| `error` | false | The tool raised |
|
||||
|
||||
Soft errors get `error_type="SoftError"` and an `error_message` such as
|
||||
`status=failed: disk full`, redacted and truncated like exception text.
|
||||
|
||||
The default classifier flags a dict result when `ok` or `success` is exactly
|
||||
`False`, when `status` or `state` is `error`, `failed` or `failure` (any case),
|
||||
or when the tool marked the result as an MCP error (`isError`) without raising.
|
||||
The message comes from the first of `error`, `message`, `detail` and `reason`.
|
||||
Add your own status values, or pass any `(tool, result) -> str | None`:
|
||||
|
||||
```python
|
||||
from fastmcp_feedback.instrumentation import instrument, soft_error_classifier
|
||||
|
||||
# blender-mcp answers {"status": "unknown_target", ...} for a missing object
|
||||
instrument(app, result_classifier=soft_error_classifier(extra_statuses=("unknown_target",)))
|
||||
```
|
||||
|
||||
```python
|
||||
from fastmcp_feedback.instrumentation import instrument
|
||||
|
||||
def classify(tool, result):
|
||||
"""None means ok; a string marks a soft error and is its message."""
|
||||
if tool == "search" and isinstance(result, dict) and result.get("hits") == []:
|
||||
return "no hits"
|
||||
return None
|
||||
|
||||
instrument(app, result_classifier=classify)
|
||||
```
|
||||
|
||||
`result_classifier=None` turns classification off. The classifier only sees
|
||||
calls that did not raise, gets the same payload the record stores (structured
|
||||
content when present), and if it raises, the error is logged and the call is
|
||||
recorded as `ok`. It never changes what the client receives.
|
||||
|
||||
**Upgrading:** soft errors set `ok` to false, so existing `WHERE NOT ok` counts
|
||||
include them from this release on. Filter on `outcome = 'error'` for exceptions
|
||||
only. The `outcome` column is new in 2026.09.27.4; it is nullable, and rows
|
||||
written by earlier versions have NULL there. Hosts that manage the schema with
|
||||
Alembic need a migration adding it (with its index) to `ffb_tool_calls`, and a
|
||||
table made earlier by `create_tables=True` needs
|
||||
`ALTER TABLE ffb_tool_calls ADD COLUMN outcome VARCHAR(16)`, since existing
|
||||
tables are not altered. Until the column exists, database writes fail and
|
||||
those records are dropped (logged).
|
||||
|
||||
### Identity and enrichment hooks
|
||||
|
||||
```python
|
||||
from fastmcp_feedback.instrumentation import instrument
|
||||
|
||||
def who(context):
|
||||
"""Who made this call. May be async."""
|
||||
user = context.fastmcp_context # read your auth state from here
|
||||
return {"user_sub": "user-123", "caller_kind": "pat:7", "client_id": "cli"}
|
||||
|
||||
async def enrich(tool, args, result, context):
|
||||
"""Extra indexed fields. May be async."""
|
||||
return {"job_id": (result or {}).get("job_id"), "server_version": "1.4.0"}
|
||||
|
||||
instrument(app, identity_resolver=who, enricher=enrich)
|
||||
```
|
||||
|
||||
`user_sub`, `caller_kind` and `client_id` become columns; other identity keys
|
||||
go in `identity`. Enricher output goes in `extra`, except `server_version`,
|
||||
which has its own column. Async hooks that take longer than `hook_timeout`
|
||||
(0.25 s by default) are skipped for that call.
|
||||
|
||||
### Linking feedback to the calls behind it
|
||||
|
||||
When a model reports "the render tool hung", the report is far more useful with
|
||||
the calls that led up to it attached. Pass the middleware to
|
||||
`add_feedback_tools` and every `submit_feedback` does this for you:
|
||||
|
||||
```python
|
||||
from fastmcp_feedback import add_feedback_tools
|
||||
from fastmcp_feedback.instrumentation import DatabaseSink, instrument
|
||||
|
||||
mw = instrument(app, [DatabaseSink("sqlite+aiosqlite:///calls.db", create_tables=True)])
|
||||
add_feedback_tools(app, database_url="sqlite:///feedback.db", instrumentation=mw)
|
||||
# submit_feedback now answers {"feedback_id": ..., "linked_calls": 3, ...}
|
||||
```
|
||||
|
||||
Read the linked calls back, in order, with their outcomes:
|
||||
|
||||
```python
|
||||
for call in await mw.feedback_context(feedback_id):
|
||||
print(call["position"], call["tool"], call["outcome"], call["error_type"], call["rule"])
|
||||
```
|
||||
|
||||
If you have your own feedback tools, use the same pieces directly. Feedback ids
|
||||
can be anything (`bug-7Q2X`, `42`):
|
||||
|
||||
```python
|
||||
from fastmcp import Context
|
||||
|
||||
@app.tool
|
||||
async def report_problem(summary: str, ctx: Context) -> dict:
|
||||
ref = save_my_feedback(summary) # your storage, your id
|
||||
links = await mw.capture_recent_calls(ctx, n=20)
|
||||
await mw.link_feedback(ref, links)
|
||||
return {"id": ref, "linked_calls": len(links)}
|
||||
```
|
||||
|
||||
Calls are matched in two ways, and each link records which one (`rule`):
|
||||
|
||||
- `session`: the same MCP client session. Over HTTP this is the
|
||||
`mcp-session-id`; over stdio it is the server process, which serves one client.
|
||||
- `user_window`: the same `user_sub` within the last 15 minutes (`window=`),
|
||||
for clients that open a new session per call (personal access tokens, CI).
|
||||
This needs an `identity_resolver`.
|
||||
|
||||
Session matches come first, and user matches fill any remaining slots. Calls
|
||||
still waiting in the queue are included, and with a `DatabaseSink`, so are
|
||||
calls from other workers or from before a restart.
|
||||
|
||||
Links live in a `ffb_feedback_call_links` table next to `ffb_tool_calls`.
|
||||
`build_metadata()` includes it, so hosts that manage the schema with Alembic
|
||||
need one new migration before calling `link_feedback`.
|
||||
|
||||
**Stateless clients:** clients on MCP protocol 2026-07-28, including Claude
|
||||
Code and FastMCP 4's own `Client`, do not open sessions over HTTP. Their calls
|
||||
link by user only, so configure an `identity_resolver` for HTTP servers. Stdio
|
||||
servers link by session with every client.
|
||||
|
||||
### Middleware order and shutdown
|
||||
|
||||
FastMCP runs the first middleware added as the outermost. Call `instrument()`
|
||||
before adding other middleware to include their time and record calls they
|
||||
reject; call it last to time only the tool. On shutdown, `await mw.aclose()`
|
||||
flushes queued records and closes the sinks.
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Mixin Architecture
|
||||
|
||||
For fine-grained control over which tools are exposed:
|
||||
Pick only the tool groups you want to expose:
|
||||
|
||||
```python
|
||||
from fastmcp_feedback import (
|
||||
SubmissionMixin,
|
||||
RetrievalMixin,
|
||||
ManagementMixin,
|
||||
RetrievalMixin,
|
||||
SubmissionMixin,
|
||||
get_database_session,
|
||||
)
|
||||
|
||||
db_session = get_database_session("sqlite:///feedback.db")
|
||||
db = get_database_session("sqlite:///feedback.db")
|
||||
|
||||
# Only expose submission (public-facing)
|
||||
submission = SubmissionMixin(db_session)
|
||||
submission.register_tools(app)
|
||||
# Public server: submission only
|
||||
SubmissionMixin(db).register_tools(app)
|
||||
|
||||
# Add retrieval with custom prefix
|
||||
retrieval = RetrievalMixin(db_session)
|
||||
retrieval.register_tools(app, prefix="analytics_")
|
||||
# Reporting tools under a prefix: analytics_list_feedback, ...
|
||||
RetrievalMixin(db).register_tools(app, prefix="analytics")
|
||||
|
||||
# Admin-only management tools
|
||||
management = ManagementMixin(db_session)
|
||||
management.register_tools(admin_app, prefix="admin_")
|
||||
# Admin server gets the workflow tools
|
||||
ManagementMixin(db).register_tools(admin_app, prefix="admin")
|
||||
```
|
||||
|
||||
### Available Mixins
|
||||
@@ -127,18 +374,16 @@ management.register_tools(admin_app, prefix="admin_")
|
||||
| Mixin | Tools | Use Case |
|
||||
|-------|-------|----------|
|
||||
| `SubmissionMixin` | `submit_feedback` | Public feedback collection |
|
||||
| `RetrievalMixin` | `list_feedback`, `get_feedback_stats` | Dashboards, reporting |
|
||||
| `ManagementMixin` | `update_feedback_status` | Admin workflow |
|
||||
| `InsightsMixin` | `record_insight`, `get_insights_summary` | Analytics |
|
||||
| `RetrievalMixin` | `list_feedback`, `get_feedback_statistics` | Dashboards, reporting |
|
||||
| `ManagementMixin` | `update_feedback_status`, `delete_feedback` | Admin workflow |
|
||||
|
||||
### Multi-Tenant Pattern
|
||||
|
||||
```python
|
||||
# Each tenant gets isolated database
|
||||
def get_tenant_tools(tenant_id: str):
|
||||
session = get_database_session(f"sqlite:///data/{tenant_id}.db")
|
||||
submission = SubmissionMixin(session)
|
||||
submission.register_tools(app, prefix=f"tenant_{tenant_id}_")
|
||||
# Each tenant gets its own database
|
||||
def add_tenant_tools(tenant_id: str):
|
||||
db = get_database_session(f"sqlite:///data/{tenant_id}.db")
|
||||
SubmissionMixin(db).register_tools(app, prefix=f"tenant_{tenant_id}")
|
||||
```
|
||||
|
||||
### Server Composition
|
||||
@@ -146,49 +391,72 @@ def get_tenant_tools(tenant_id: str):
|
||||
```python
|
||||
from fastmcp_feedback import create_feedback_server
|
||||
|
||||
# Create dedicated feedback server
|
||||
feedback_server = create_feedback_server("Feedback API")
|
||||
feedback_server = create_feedback_server(
|
||||
"Feedback API", database_url="sqlite:///feedback.db"
|
||||
)
|
||||
|
||||
# Compose into main application
|
||||
main_app.import_server(feedback_server, prefix="feedback_")
|
||||
# Tools appear on main_app as feedback_submit_feedback, feedback_list_feedback, ...
|
||||
main_app.mount(feedback_server, "feedback")
|
||||
```
|
||||
|
||||
`mount()` works on every supported FastMCP version. `import_server()` was
|
||||
removed in FastMCP 4.
|
||||
|
||||
## Compatibility
|
||||
|
||||
Tested against FastMCP 2.12, 2.14, 3.x and 4.x on every commit. The dependency is
|
||||
declared as `fastmcp>=2.12.2,<5`, and the ceiling moves up once a new major
|
||||
version passes the suite.
|
||||
|
||||
## API Reference
|
||||
|
||||
### `add_feedback_tools(app, database_url=None, enable_insights=False)`
|
||||
### `add_feedback_tools(mcp, database_url=None, insights=None, prefix="", separator="_", instrumentation=None)`
|
||||
|
||||
Add all feedback tools to a FastMCP server.
|
||||
Add all five feedback tools to a FastMCP server.
|
||||
|
||||
**Parameters:**
|
||||
- `app`: FastMCP server instance
|
||||
- `database_url`: Database connection string (default: `sqlite:///feedback.db`)
|
||||
- `enable_insights`: Enable privacy-compliant analytics (default: `False`)
|
||||
- `mcp`: FastMCP server instance
|
||||
- `database_url`: SQLAlchemy URL. Defaults to in-memory SQLite (`sqlite:///:memory:`)
|
||||
- `insights`: a `FeedbackInsights` instance. Defaults to one with analytics disabled
|
||||
- `prefix`: prepended to every tool name
|
||||
- `separator`: placed between `prefix` and the tool name
|
||||
- `instrumentation`: the middleware returned by `instrument()`. When given,
|
||||
`submit_feedback` links the calls that preceded it to the new item and
|
||||
returns `linked_calls` (see [Linking feedback to the calls behind it](#linking-feedback-to-the-calls-behind-it))
|
||||
|
||||
### Feedback Types
|
||||
|
||||
- `bug_report` - Bug reports with severity levels
|
||||
- `feature_request` - Feature suggestions with priority
|
||||
- `general` - General feedback and comments
|
||||
- `bug` - something is broken
|
||||
- `feature` - a new capability
|
||||
- `improvement` - make an existing capability better
|
||||
- `question` - unclear behavior or usage
|
||||
|
||||
### Status Workflow
|
||||
|
||||
`new` → `reviewing` → `planned` → `in_progress` → `resolved` / `wont_fix`
|
||||
`open` → `in_progress` → `resolved` → `closed`
|
||||
|
||||
## Documentation
|
||||
|
||||
Full documentation: https://fastmcp-feedback.l.supported.systems
|
||||
|
||||
- [Quick Start Guide](https://fastmcp-feedback.l.supported.systems/guides/quickstart/)
|
||||
- [Architecture Overview](https://fastmcp-feedback.l.supported.systems/reference/architecture/)
|
||||
- [Quick Start Tutorial](https://fastmcp-feedback.l.supported.systems/tutorials/quickstart/)
|
||||
- [Architecture](https://fastmcp-feedback.l.supported.systems/explanation/architecture/)
|
||||
- [API Reference](https://fastmcp-feedback.l.supported.systems/reference/api/)
|
||||
- [Examples](https://fastmcp-feedback.l.supported.systems/examples/)
|
||||
- [Integration Patterns](https://fastmcp-feedback.l.supported.systems/how-to/integration-patterns/)
|
||||
|
||||
## Examples
|
||||
|
||||
See the [fastmcp-feedback-showcase](https://git.supported.systems/fastmcp-feedback/fastmcp-example) repository for comprehensive examples demonstrating all capabilities:
|
||||
The [`examples/`](https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/src/branch/main/examples)
|
||||
directory has runnable servers, each with a walkthrough:
|
||||
|
||||
- `simple_integration.py`: the one-line integration
|
||||
- `mcp_server.py`: a production-style server configured from environment variables
|
||||
- `demo_server.py`: every feature, including server composition and sample data
|
||||
|
||||
```bash
|
||||
uvx fastmcp-feedback-showcase # Run full demo
|
||||
git clone https://git.supported.systems/fastmcp-feedback/fastmcp-feedback.git
|
||||
cd fastmcp-feedback
|
||||
uv run python examples/simple_integration.py
|
||||
```
|
||||
|
||||
## Contributing
|
||||
@@ -236,5 +504,4 @@ MIT License - see [LICENSE](LICENSE) for details.
|
||||
- **Documentation**: https://fastmcp-feedback.l.supported.systems
|
||||
- **Repository**: https://git.supported.systems/fastmcp-feedback/fastmcp-feedback
|
||||
- **Issues**: https://git.supported.systems/fastmcp-feedback/fastmcp-feedback/issues
|
||||
- **Showcase**: https://git.supported.systems/fastmcp-feedback/fastmcp-example
|
||||
- **FastMCP**: https://gofastmcp.com
|
||||
|
||||
@@ -0,0 +1,318 @@
|
||||
# FastMCP Feedback Examples
|
||||
|
||||
This directory contains comprehensive examples demonstrating all features of the FastMCP Feedback system.
|
||||
|
||||
## 🚀 Demo Server
|
||||
|
||||
### `demo_server.py`
|
||||
**Complete demonstration server showcasing all features**
|
||||
|
||||
**Features Demonstrated:**
|
||||
- ✅ Complete feedback collection workflow
|
||||
- ✅ Privacy-compliant analytics (GDPR-ready)
|
||||
- ✅ Multi-database support (SQLite, PostgreSQL, MySQL)
|
||||
- ✅ Thread-safe operations with 97% test coverage
|
||||
- ✅ Server composition patterns
|
||||
- ✅ Performance monitoring and health checks
|
||||
- ✅ Sample data for testing
|
||||
|
||||
**Available Tools:**
|
||||
- `submit_feedback` - Submit bug reports, feature requests, improvements, questions
|
||||
- `list_feedback` - Query and filter feedback with pagination
|
||||
- `get_feedback_statistics` - Analytics and aggregate metrics
|
||||
- `update_feedback_status` - Change status (open → in_progress → resolved)
|
||||
- `delete_feedback` - Remove feedback items
|
||||
- `get_system_info` - Version and dependency information
|
||||
- `setup_sample_data` - Populate with realistic demo data
|
||||
|
||||
**Usage:**
|
||||
|
||||
```bash
|
||||
# Run the demo server
|
||||
python examples/demo_server.py
|
||||
|
||||
# Or with uv
|
||||
uv run examples/demo_server.py
|
||||
```
|
||||
|
||||
**Claude Desktop MCP Configuration:**
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"fastmcp-feedback-demo": {
|
||||
"command": "uv",
|
||||
"args": ["run", "examples/demo_server.py"],
|
||||
"env": {
|
||||
"LOG_LEVEL": "INFO"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🏗️ Architecture Examples
|
||||
|
||||
### `simple_integration.py`
|
||||
**Minimal integration example - 3 lines of code**
|
||||
|
||||
```python
|
||||
from fastmcp import FastMCP
|
||||
from fastmcp_feedback import add_feedback_tools
|
||||
|
||||
app = FastMCP("My App")
|
||||
add_feedback_tools(app, database_url="sqlite:///feedback.db")
|
||||
```
|
||||
|
||||
### `advanced_configuration.py`
|
||||
**Production-ready configuration with all features**
|
||||
|
||||
```python
|
||||
from fastmcp_feedback import (
|
||||
add_feedback_tools,
|
||||
FeedbackInsights,
|
||||
FeedbackDatabase
|
||||
)
|
||||
|
||||
# Configure privacy-compliant analytics
|
||||
insights = FeedbackInsights(
|
||||
enabled=True,
|
||||
retention_days=90,
|
||||
privacy_compliant=True
|
||||
)
|
||||
|
||||
# Setup database with connection pooling
|
||||
database = FeedbackDatabase(
|
||||
"postgresql://user:pass@localhost/feedback",
|
||||
pool_size=20,
|
||||
max_overflow=10
|
||||
)
|
||||
|
||||
# Add tools with custom prefix
|
||||
add_feedback_tools(
|
||||
app,
|
||||
database=database,
|
||||
insights=insights,
|
||||
prefix="support_"
|
||||
)
|
||||
```
|
||||
|
||||
### `server_composition.py`
|
||||
**Multi-server composition patterns**
|
||||
|
||||
```python
|
||||
from fastmcp_feedback.feedback import (
|
||||
create_submission_server,
|
||||
create_management_server,
|
||||
create_analytics_server
|
||||
)
|
||||
|
||||
# Create specialized servers
|
||||
public_server = create_submission_server("Public Portal")
|
||||
admin_server = create_management_server("Admin Panel")
|
||||
analytics = create_analytics_server("Dashboard")
|
||||
|
||||
# Compose into main application
|
||||
main_app = FastMCP("Main Application")
|
||||
await main_app.import_server(public_server, prefix="public")
|
||||
await main_app.import_server(admin_server, prefix="admin")
|
||||
await main_app.import_server(analytics, prefix="stats")
|
||||
```
|
||||
|
||||
## 🎯 Usage Patterns
|
||||
|
||||
### Basic Feedback Workflow
|
||||
|
||||
1. **Submit Feedback:**
|
||||
```python
|
||||
# Submit a bug report
|
||||
result = await submit_feedback({
|
||||
"request": {
|
||||
"type": "bug",
|
||||
"title": "Login page not responsive on mobile",
|
||||
"description": "The login form overflows on screens < 768px width",
|
||||
"submitter": "user_123",
|
||||
"contact_info": "user@example.com"
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
2. **List and Filter:**
|
||||
```python
|
||||
# Get all open bug reports
|
||||
bugs = await list_feedback({
|
||||
"type_filter": "bug",
|
||||
"status_filter": "open",
|
||||
"limit": 20
|
||||
})
|
||||
```
|
||||
|
||||
3. **Update Status:**
|
||||
```python
|
||||
# Mark as in progress
|
||||
await update_feedback_status({
|
||||
"feedback_id": "123",
|
||||
"new_status": "in_progress",
|
||||
"note": "Engineering team investigating"
|
||||
})
|
||||
```
|
||||
|
||||
4. **Get Analytics:**
|
||||
```python
|
||||
# View statistics
|
||||
stats = await get_feedback_statistics()
|
||||
# Returns: {"total_count": 45, "by_type": {"BUG": 12, "FEATURE": 20}, ...}
|
||||
```
|
||||
|
||||
## 🔒 Privacy & Security Examples
|
||||
|
||||
### Privacy-Compliant Configuration
|
||||
```python
|
||||
insights = FeedbackInsights(
|
||||
enabled=True, # Enable analytics
|
||||
retention_days=90, # Auto-cleanup after 90 days
|
||||
privacy_compliant=True, # Enforce GDPR compliance
|
||||
anonymize_identifiers=True, # Hash user identifiers
|
||||
exclude_contact_info=True # Don't store in metrics
|
||||
)
|
||||
```
|
||||
|
||||
### Security Features
|
||||
```python
|
||||
# Input validation with Pydantic
|
||||
from fastmcp_feedback.models import SubmitFeedbackRequest
|
||||
|
||||
# SQL injection prevention via SQLAlchemy ORM
|
||||
# Rate limiting support (extend mixins)
|
||||
# Authentication ready (add middleware)
|
||||
```
|
||||
|
||||
## 📊 Performance Examples
|
||||
|
||||
### Connection Pooling
|
||||
```python
|
||||
database = FeedbackDatabase(
|
||||
"postgresql://localhost/feedback",
|
||||
pool_size=20, # Base connections
|
||||
max_overflow=10, # Burst capacity
|
||||
pool_timeout=30, # Connection timeout
|
||||
pool_recycle=3600 # Refresh every hour
|
||||
)
|
||||
```
|
||||
|
||||
### Bulk Operations
|
||||
```python
|
||||
# Bulk status updates
|
||||
result = await bulk_update_status(
|
||||
feedback_ids=["1", "2", "3", "4", "5"],
|
||||
new_status="resolved"
|
||||
)
|
||||
```
|
||||
|
||||
### Performance Monitoring
|
||||
```python
|
||||
# Built-in health checks
|
||||
health = await database.health_check()
|
||||
if not health:
|
||||
# Trigger alerts
|
||||
pass
|
||||
|
||||
# Performance metrics
|
||||
stats = await get_database_stats()
|
||||
# Returns connection pool status, query times, etc.
|
||||
```
|
||||
|
||||
## 🧪 Testing Examples
|
||||
|
||||
### Unit Testing with FastMCP Feedback
|
||||
```python
|
||||
import pytest
|
||||
from fastmcp_feedback import FeedbackDatabase
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_feedback_submission():
|
||||
# Use in-memory database for testing
|
||||
db = FeedbackDatabase("sqlite:///:memory:")
|
||||
await db.initialize()
|
||||
|
||||
# Test feedback submission
|
||||
# ... test code here
|
||||
|
||||
await db.close()
|
||||
```
|
||||
|
||||
### Integration Testing
|
||||
```python
|
||||
# Test complete workflows
|
||||
async def test_complete_feedback_lifecycle():
|
||||
# Submit → List → Update → Verify → Analytics
|
||||
pass
|
||||
```
|
||||
|
||||
## 🚀 Production Deployment Examples
|
||||
|
||||
### Docker Configuration
|
||||
```dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
COPY pyproject.toml .
|
||||
RUN pip install -e .
|
||||
|
||||
COPY src/ src/
|
||||
COPY examples/ examples/
|
||||
|
||||
# Health check
|
||||
HEALTHCHECK --interval=30s --timeout=10s \
|
||||
CMD python -c "import asyncio; from examples.health_check import check; asyncio.run(check())"
|
||||
|
||||
CMD ["python", "examples/demo_server.py"]
|
||||
```
|
||||
|
||||
### Environment Configuration
|
||||
```bash
|
||||
# Database
|
||||
DATABASE_URL=postgresql://user:pass@db:5432/feedback
|
||||
|
||||
# Analytics
|
||||
INSIGHTS_ENABLED=true
|
||||
INSIGHTS_RETENTION_DAYS=90
|
||||
|
||||
# Performance
|
||||
CONNECTION_POOL_SIZE=20
|
||||
MAX_OVERFLOW=10
|
||||
|
||||
# Security
|
||||
RATE_LIMIT_REQUESTS=100
|
||||
RATE_LIMIT_WINDOW=3600
|
||||
```
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
Each example includes:
|
||||
- **Comprehensive comments** explaining every feature
|
||||
- **Error handling** patterns
|
||||
- **Performance considerations**
|
||||
- **Security best practices**
|
||||
- **Testing approaches**
|
||||
- **Production deployment guidance**
|
||||
|
||||
## 🎯 Getting Started
|
||||
|
||||
1. **Start with the demo server** - `python examples/demo_server.py`
|
||||
2. **Connect to Claude Desktop** using the MCP configuration above
|
||||
3. **Try the tools** - submit feedback, query results, update status
|
||||
4. **Explore the code** - see how each feature is implemented
|
||||
5. **Customize for your needs** - adapt patterns to your use case
|
||||
|
||||
## 🤝 Contributing Examples
|
||||
|
||||
We welcome additional examples! Please:
|
||||
- Include comprehensive comments
|
||||
- Add error handling
|
||||
- Provide usage instructions
|
||||
- Update this README
|
||||
- Add tests if applicable
|
||||
|
||||
---
|
||||
|
||||
**These examples demonstrate production-ready FastMCP Feedback integration with 97% test coverage and enterprise-grade reliability.**
|
||||
@@ -0,0 +1,247 @@
|
||||
# FastMCP Feedback Demo Server
|
||||
|
||||
**File:** `demo_server.py`
|
||||
**Purpose:** Comprehensive demonstration of all FastMCP Feedback features
|
||||
**Lines:** 396 lines of production-ready code
|
||||
**Test Coverage:** Validated through 396+ comprehensive tests
|
||||
|
||||
## 🎯 Overview
|
||||
|
||||
The demo server is a complete, production-ready FastMCP server that showcases every feature of the FastMCP Feedback system. It's designed to demonstrate real-world usage patterns and serve as a reference implementation for developers.
|
||||
|
||||
## ✨ Features Demonstrated
|
||||
|
||||
### Core Feedback System
|
||||
- **Complete CRUD Operations** - Submit, retrieve, update, delete feedback
|
||||
- **Type Support** - Bug reports, feature requests, improvements, questions
|
||||
- **Status Management** - Open → In Progress → Resolved → Closed workflow
|
||||
- **Rich Metadata** - Timestamps, submitter info, contact details
|
||||
|
||||
### Advanced Capabilities
|
||||
- **Privacy-Compliant Analytics** - GDPR-ready insights collection
|
||||
- **Multi-Database Support** - SQLite for development, PostgreSQL for production
|
||||
- **Thread-Safe Operations** - Concurrent request handling
|
||||
- **Performance Monitoring** - Health checks and dependency validation
|
||||
- **Server Composition** - Multiple specialized server types
|
||||
|
||||
### Data Integrity
|
||||
- **Input Validation** - Pydantic models with comprehensive validation
|
||||
- **Unicode Support** - Emojis, special characters, international text
|
||||
- **Timezone Handling** - UTC-aware timestamp management
|
||||
- **Whitespace Normalization** - Consistent data cleaning
|
||||
|
||||
## 🛠️ Available Tools (7 Total)
|
||||
|
||||
| Tool | Description | Use Case |
|
||||
|------|-------------|----------|
|
||||
| `submit_feedback` | Submit new feedback items | Users reporting bugs/requesting features |
|
||||
| `list_feedback` | Query and filter feedback | Viewing feedback with pagination/filtering |
|
||||
| `get_feedback_statistics` | Analytics and metrics | Dashboard data and reporting |
|
||||
| `update_feedback_status` | Change feedback status | Admin workflow management |
|
||||
| `delete_feedback` | Remove feedback items | Content moderation |
|
||||
| `get_system_info` | Version/dependency info | System diagnostics |
|
||||
| `setup_sample_data` | Populate demo data | Testing and demonstration |
|
||||
|
||||
## 📊 Sample Data
|
||||
|
||||
The demo server includes realistic sample data:
|
||||
|
||||
```python
|
||||
sample_feedback = [
|
||||
{
|
||||
"type": "bug",
|
||||
"title": "Memory leak in analytics processing",
|
||||
"description": "Analytics worker consumes increasing memory over time...",
|
||||
"submitter": "alice@example.com",
|
||||
"contact_info": "alice@engineering.example.com"
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"title": "Add export functionality for feedback data",
|
||||
"description": "Users need ability to export feedback data as CSV or JSON...",
|
||||
"submitter": "product_manager",
|
||||
"contact_info": "pm@example.com"
|
||||
}
|
||||
# ... 3 more realistic examples
|
||||
]
|
||||
```
|
||||
|
||||
## 🚀 Usage
|
||||
|
||||
### Direct Execution
|
||||
```bash
|
||||
# Run with Python
|
||||
python examples/demo_server.py
|
||||
|
||||
# Run with uv (recommended)
|
||||
uv run examples/demo_server.py
|
||||
```
|
||||
|
||||
### Claude Desktop Integration
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"fastmcp-feedback-demo": {
|
||||
"command": "uv",
|
||||
"args": ["run", "examples/demo_server.py"],
|
||||
"env": {
|
||||
"DATABASE_URL": "sqlite:///feedback_demo.db",
|
||||
"LOG_LEVEL": "INFO"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Docker Deployment
|
||||
```dockerfile
|
||||
FROM python:3.11-slim
|
||||
COPY . /app
|
||||
WORKDIR /app
|
||||
RUN pip install uv && uv sync
|
||||
CMD ["uv", "run", "examples/demo_server.py"]
|
||||
```
|
||||
|
||||
## 🔧 Configuration Options
|
||||
|
||||
### Environment Variables
|
||||
- `DATABASE_URL` - Database connection string (default: temporary SQLite)
|
||||
- `LOG_LEVEL` - Logging verbosity (DEBUG, INFO, WARNING, ERROR)
|
||||
- `INSIGHTS_ENABLED` - Enable/disable analytics collection
|
||||
- `RETENTION_DAYS` - Analytics data retention period
|
||||
|
||||
### Database Support
|
||||
- **SQLite** - `sqlite:///path/to/database.db` (development)
|
||||
- **PostgreSQL** - `postgresql://user:pass@host:port/db` (production)
|
||||
- **In-Memory** - `sqlite:///:memory:` (testing)
|
||||
|
||||
## 📈 Performance Characteristics
|
||||
|
||||
### Throughput
|
||||
- **Concurrent Requests** - Handles 100+ concurrent submissions
|
||||
- **Database Performance** - Optimized queries with indexing
|
||||
- **Memory Usage** - Efficient async/await patterns
|
||||
|
||||
### Scalability
|
||||
- **Horizontal Scaling** - Stateless design supports load balancing
|
||||
- **Database Scaling** - Connection pooling and query optimization
|
||||
- **Analytics Scaling** - Configurable retention and aggregation
|
||||
|
||||
## 🧪 Testing Integration
|
||||
|
||||
The demo server is thoroughly tested through:
|
||||
|
||||
### Unit Tests
|
||||
- Tool registration and configuration
|
||||
- Sample data generation
|
||||
- Error handling scenarios
|
||||
|
||||
### Integration Tests
|
||||
- End-to-end workflow validation
|
||||
- Database operations
|
||||
- Server composition patterns
|
||||
|
||||
### Performance Tests
|
||||
- High-volume operations
|
||||
- Concurrent request handling
|
||||
- Memory usage monitoring
|
||||
|
||||
## 🎓 Learning Examples
|
||||
|
||||
### Basic Feedback Submission
|
||||
```python
|
||||
# Submit a bug report
|
||||
result = await submit_feedback({
|
||||
"type": "bug",
|
||||
"title": "Application crashes on startup",
|
||||
"description": "Detailed error description...",
|
||||
"submitter": "developer@company.com"
|
||||
})
|
||||
```
|
||||
|
||||
### Advanced Filtering
|
||||
```python
|
||||
# Get all open bug reports from last week
|
||||
feedback = await list_feedback({
|
||||
"type_filter": "bug",
|
||||
"status_filter": "open",
|
||||
"limit": 50
|
||||
})
|
||||
```
|
||||
|
||||
### Analytics Query
|
||||
```python
|
||||
# Get comprehensive statistics
|
||||
stats = await get_feedback_statistics()
|
||||
# Returns: total_count, by_type, by_status, recent_count
|
||||
```
|
||||
|
||||
## 🔍 Architecture Insights
|
||||
|
||||
### Server Composition Pattern
|
||||
```python
|
||||
# The demo server demonstrates server composition:
|
||||
main_server = FastMCP("Demo Server")
|
||||
feedback_server = create_feedback_server("Feedback Service")
|
||||
main_server.import_server(feedback_server, prefix="feedback_")
|
||||
```
|
||||
|
||||
### Database Abstraction
|
||||
```python
|
||||
# Flexible database backend:
|
||||
database = FeedbackDatabase(database_url)
|
||||
await database.initialize() # Auto-creates tables
|
||||
```
|
||||
|
||||
### Analytics Integration
|
||||
```python
|
||||
# Privacy-compliant analytics:
|
||||
insights = FeedbackInsights(enabled=True, retention_days=90)
|
||||
insights.record_metric("feedback_submitted", metadata)
|
||||
```
|
||||
|
||||
## 📚 Related Documentation
|
||||
|
||||
- **[Simple Integration](simple_integration.md)** - Minimal 3-line integration
|
||||
- **[Production Server](mcp_server.md)** - Environment-configured deployment
|
||||
- **[Testing Guide](../tests/README.md)** - Comprehensive test documentation
|
||||
- **[Architecture Overview](../docs/architecture.md)** - System design details
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
When modifying the demo server:
|
||||
|
||||
1. **Maintain Compatibility** - Ensure all existing tools continue working
|
||||
2. **Update Documentation** - Keep this file synchronized with code changes
|
||||
3. **Add Tests** - Include test coverage for new features
|
||||
4. **Version Compatibility** - Support multiple FastMCP versions where possible
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Database Errors**
|
||||
```bash
|
||||
# Check database permissions
|
||||
ls -la feedback_demo.db
|
||||
# Reset database
|
||||
rm feedback_demo.db && python examples/demo_server.py
|
||||
```
|
||||
|
||||
**Import Errors**
|
||||
```bash
|
||||
# Ensure development installation
|
||||
uv sync
|
||||
# Check Python path
|
||||
python -c "import fastmcp_feedback; print('OK')"
|
||||
```
|
||||
|
||||
**Performance Issues**
|
||||
```bash
|
||||
# Enable debug logging
|
||||
LOG_LEVEL=DEBUG python examples/demo_server.py
|
||||
# Monitor memory usage
|
||||
ps aux | grep demo_server
|
||||
```
|
||||
|
||||
This demo server represents the gold standard for FastMCP Feedback integration, demonstrating every feature with production-ready code patterns and comprehensive error handling.
|
||||
@@ -0,0 +1,402 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastMCP Feedback Demo Server
|
||||
|
||||
A comprehensive demonstration server that showcases all features of the
|
||||
FastMCP Feedback system including:
|
||||
- Complete feedback collection workflow
|
||||
- Privacy-compliant analytics
|
||||
- Multi-database support
|
||||
- Performance monitoring
|
||||
- Server composition patterns
|
||||
|
||||
Usage:
|
||||
# Run the demo server
|
||||
python examples/demo_server.py
|
||||
|
||||
# Or use uv
|
||||
uv run examples/demo_server.py
|
||||
|
||||
# Connect from Claude Desktop with this MCP server configuration:
|
||||
# {
|
||||
# "feedback-demo": {
|
||||
# "command": "uv",
|
||||
# "args": ["run", "examples/demo_server.py"]
|
||||
# }
|
||||
# }
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import sys
|
||||
import tempfile
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
# Add the src directory to the path for development
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
|
||||
|
||||
from fastmcp import Client, FastMCP
|
||||
|
||||
from fastmcp_feedback import (
|
||||
FeedbackDatabase,
|
||||
FeedbackInsights,
|
||||
add_feedback_tools,
|
||||
check_dependencies,
|
||||
get_version_info,
|
||||
)
|
||||
from fastmcp_feedback.feedback import (
|
||||
create_analytics_server,
|
||||
create_management_server,
|
||||
create_submission_server,
|
||||
)
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
|
||||
handlers=[logging.StreamHandler(), logging.FileHandler("feedback_demo.log")],
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def create_temp_database() -> str:
|
||||
"""Create a temporary SQLite database for the demo."""
|
||||
temp_dir = tempfile.mkdtemp(prefix="fastmcp_feedback_demo_")
|
||||
db_path = Path(temp_dir) / "demo_feedback.db"
|
||||
return f"sqlite:///{db_path}"
|
||||
|
||||
|
||||
async def setup_demo_data(database: FeedbackDatabase) -> None:
|
||||
"""Populate the database with sample feedback for demonstration."""
|
||||
logger.info("Setting up demo data...")
|
||||
|
||||
sample_feedback = [
|
||||
{
|
||||
"type": "bug",
|
||||
"title": "Memory leak in analytics processing",
|
||||
"description": "Analytics worker consumes increasing memory over time, eventually causing OOM errors.",
|
||||
"submitter": "alice@example.com",
|
||||
"contact_info": "alice@engineering.example.com",
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"title": "Add export functionality for feedback data",
|
||||
"description": "Users need ability to export feedback data as CSV or JSON for external analysis.",
|
||||
"submitter": "product_manager",
|
||||
"contact_info": "pm@example.com",
|
||||
},
|
||||
{
|
||||
"type": "improvement",
|
||||
"title": "Optimize database query performance",
|
||||
"description": "List feedback queries are slow with large datasets. Need pagination and indexing improvements.",
|
||||
"submitter": "performance_team",
|
||||
},
|
||||
{
|
||||
"type": "question",
|
||||
"title": "Best practices for rate limiting?",
|
||||
"description": "What are recommended rate limits for feedback submission to prevent abuse?",
|
||||
"submitter": "security_team",
|
||||
"contact_info": "security@example.com",
|
||||
},
|
||||
{
|
||||
"type": "bug",
|
||||
"title": "Unicode handling in feedback titles",
|
||||
"description": "Emojis and special characters cause encoding errors in feedback titles. 🐛",
|
||||
"submitter": "qa_engineer",
|
||||
},
|
||||
]
|
||||
|
||||
try:
|
||||
await database.initialize()
|
||||
|
||||
# Import the model here to avoid circular imports
|
||||
from fastmcp_feedback.models import Feedback, FeedbackStatus, FeedbackType
|
||||
|
||||
async with database.session() as session:
|
||||
for item in sample_feedback:
|
||||
feedback = Feedback(
|
||||
type=FeedbackType(item["type"]),
|
||||
status=FeedbackStatus.OPEN,
|
||||
title=item["title"],
|
||||
description=item["description"],
|
||||
submitter=item["submitter"],
|
||||
contact_info=item.get("contact_info"),
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
)
|
||||
session.add(feedback)
|
||||
|
||||
session.commit()
|
||||
logger.info(f"Added {len(sample_feedback)} sample feedback items")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to setup demo data: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def create_demo_server(database_url: str | None = None) -> FastMCP:
|
||||
"""Create the main demo server with all feedback tools.
|
||||
|
||||
Uses a fresh temporary SQLite database unless database_url is given.
|
||||
"""
|
||||
logger.info("🚀 Creating FastMCP Feedback Demo Server")
|
||||
|
||||
# Create the main server
|
||||
server = FastMCP("FastMCP Feedback Demo")
|
||||
server.description = (
|
||||
"Comprehensive demonstration of FastMCP Feedback system featuring "
|
||||
"complete feedback lifecycle, analytics, and multi-database support."
|
||||
)
|
||||
|
||||
# Create temporary database for demo
|
||||
database_url = database_url or create_temp_database()
|
||||
logger.info(f"Using demo database: {database_url}")
|
||||
|
||||
# Configure insights for demo (privacy-compliant by design)
|
||||
insights = FeedbackInsights(
|
||||
enabled=True,
|
||||
retention_days=30, # Short retention for demo
|
||||
)
|
||||
|
||||
# Add all feedback tools to the main server
|
||||
add_feedback_tools(
|
||||
server,
|
||||
database_url=database_url,
|
||||
insights=insights,
|
||||
prefix="", # No prefix for main tools
|
||||
)
|
||||
|
||||
# Add version info tool
|
||||
@server.tool(description="Get FastMCP Feedback version and dependency information")
|
||||
def get_system_info() -> dict:
|
||||
"""Get comprehensive system information."""
|
||||
version_info = get_version_info()
|
||||
dependency_status = check_dependencies()
|
||||
|
||||
return {
|
||||
"system": "FastMCP Feedback Demo Server",
|
||||
"status": "running",
|
||||
"started_at": datetime.now(UTC).isoformat(),
|
||||
"versions": version_info,
|
||||
"dependencies": dependency_status,
|
||||
"demo_features": [
|
||||
"Complete feedback lifecycle",
|
||||
"Privacy-compliant analytics",
|
||||
"Multi-database support",
|
||||
"Performance monitoring",
|
||||
"Sample data included",
|
||||
],
|
||||
}
|
||||
|
||||
# Add demo data setup tool
|
||||
@server.tool(description="Populate database with sample feedback data")
|
||||
async def setup_sample_data() -> dict:
|
||||
"""Setup sample feedback data for demonstration."""
|
||||
try:
|
||||
database = FeedbackDatabase(database_url)
|
||||
await setup_demo_data(database)
|
||||
await database.close()
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"message": "Sample data created successfully",
|
||||
"items_added": 5,
|
||||
"data_includes": [
|
||||
"Bug reports with technical details",
|
||||
"Feature requests from product team",
|
||||
"Performance improvement suggestions",
|
||||
"Security questions",
|
||||
"Unicode handling issues",
|
||||
],
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to setup sample data: {e}")
|
||||
return {"success": False, "error": str(e)}
|
||||
|
||||
return server
|
||||
|
||||
|
||||
async def create_composed_demo_server() -> FastMCP:
|
||||
"""Create a demo server using server composition patterns."""
|
||||
logger.info("🏗️ Creating Composed Demo Server")
|
||||
|
||||
# Create main application server
|
||||
main_server = FastMCP("Composed Feedback System")
|
||||
main_server.description = "Demonstration of FastMCP server composition patterns"
|
||||
|
||||
# Create temporary databases for different services
|
||||
public_db = create_temp_database()
|
||||
admin_db = create_temp_database()
|
||||
analytics_db = create_temp_database()
|
||||
|
||||
# Create specialized servers
|
||||
public_server = create_submission_server(
|
||||
"Public Feedback Portal",
|
||||
database_url=public_db,
|
||||
insights=FeedbackInsights(enabled=True, retention_days=90),
|
||||
)
|
||||
|
||||
admin_server = create_management_server(
|
||||
"Admin Management Panel",
|
||||
database_url=admin_db,
|
||||
insights=FeedbackInsights(enabled=True, retention_days=365),
|
||||
)
|
||||
|
||||
analytics_server = create_analytics_server(
|
||||
"Analytics Dashboard",
|
||||
database_url=analytics_db,
|
||||
insights=FeedbackInsights(enabled=True, retention_days=30),
|
||||
)
|
||||
|
||||
# Compose servers with prefixes
|
||||
main_server.mount(public_server, "public")
|
||||
main_server.mount(admin_server, "admin")
|
||||
main_server.mount(analytics_server, "analytics")
|
||||
|
||||
# Add composition info tool
|
||||
@main_server.tool(description="Get information about server composition")
|
||||
def get_composition_info() -> dict:
|
||||
"""Get information about the composed server architecture."""
|
||||
return {
|
||||
"architecture": "Multi-server composition",
|
||||
"servers": {
|
||||
"public": {
|
||||
"purpose": "Feedback submission for end users",
|
||||
"tools": ["submit_feedback"],
|
||||
"database": "isolated",
|
||||
},
|
||||
"admin": {
|
||||
"purpose": "Management and moderation",
|
||||
"tools": [
|
||||
"update_feedback_status",
|
||||
"delete_feedback",
|
||||
"bulk_operations",
|
||||
],
|
||||
"database": "isolated",
|
||||
},
|
||||
"analytics": {
|
||||
"purpose": "Reporting and insights",
|
||||
"tools": ["generate_reports", "export_data", "usage_statistics"],
|
||||
"database": "isolated",
|
||||
},
|
||||
},
|
||||
"benefits": [
|
||||
"Separation of concerns",
|
||||
"Independent scaling",
|
||||
"Role-based access control",
|
||||
"Database isolation",
|
||||
"Service modularity",
|
||||
],
|
||||
}
|
||||
|
||||
return main_server
|
||||
|
||||
|
||||
async def run_demo_tests(server: FastMCP) -> dict:
|
||||
"""Run basic functionality tests on the demo server."""
|
||||
logger.info("🧪 Running demo functionality tests...")
|
||||
|
||||
try:
|
||||
# Go through a client so this works the same on every FastMCP version
|
||||
async with Client(server) as client:
|
||||
tool_names = [tool.name for tool in await client.list_tools()]
|
||||
|
||||
system_info = None
|
||||
if "get_system_info" in tool_names:
|
||||
result = await client.call_tool("get_system_info", {})
|
||||
system_info = result.structured_content
|
||||
|
||||
setup_result = None
|
||||
if "setup_sample_data" in tool_names:
|
||||
result = await client.call_tool("setup_sample_data", {})
|
||||
setup_result = result.structured_content
|
||||
|
||||
return {
|
||||
"test_status": "passed",
|
||||
"tools_available": len(tool_names),
|
||||
"tool_names": tool_names,
|
||||
"system_info": system_info,
|
||||
"sample_data_setup": setup_result,
|
||||
"test_timestamp": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Demo tests failed: {e}")
|
||||
return {
|
||||
"test_status": "failed",
|
||||
"error": str(e),
|
||||
"test_timestamp": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
def print_demo_banner():
|
||||
"""Print an informative banner for the demo server."""
|
||||
banner = """
|
||||
╔══════════════════════════════════════════════════════════════════╗
|
||||
║ 🚀 FastMCP Feedback Demo Server ║
|
||||
║ ║
|
||||
║ Features Demonstrated: ║
|
||||
║ • Complete feedback collection system ║
|
||||
║ • Privacy-compliant analytics (GDPR-ready) ║
|
||||
║ • Multi-database support (SQLite, PostgreSQL) ║
|
||||
║ • Thread-safe operations with 97% test coverage ║
|
||||
║ • Server composition patterns ║
|
||||
║ • Performance monitoring and health checks ║
|
||||
║ ║
|
||||
║ Available Tools: ║
|
||||
║ 📝 submit_feedback - Submit bug reports & feature requests║
|
||||
║ 📋 list_feedback - Query and filter feedback ║
|
||||
║ 📊 get_feedback_statistics - Analytics and metrics ║
|
||||
║ 🔧 update_feedback_status - Change status (admin) ║
|
||||
║ 🗑️ delete_feedback - Remove feedback items ║
|
||||
║ ℹ️ get_system_info - Version and dependency info ║
|
||||
║ 🎯 setup_sample_data - Populate with demo data ║
|
||||
║ ║
|
||||
║ Usage: Connect this as an MCP server in Claude Desktop ║
|
||||
║ Configuration: See comments in this file ║
|
||||
╚══════════════════════════════════════════════════════════════════╝
|
||||
"""
|
||||
print(banner)
|
||||
|
||||
|
||||
async def main():
|
||||
"""Main entry point for the demo server."""
|
||||
print_demo_banner()
|
||||
|
||||
try:
|
||||
# Check dependencies first
|
||||
deps = check_dependencies()
|
||||
if not deps["all_available"]:
|
||||
logger.error("Missing required dependencies!")
|
||||
for dep, info in deps["details"].items():
|
||||
if not info["available"]:
|
||||
logger.error(f" Missing: {dep} {info['required']}")
|
||||
sys.exit(1)
|
||||
|
||||
logger.info("All dependencies satisfied ✅")
|
||||
|
||||
# Create the main demo server
|
||||
server = create_demo_server()
|
||||
|
||||
# Run functionality tests
|
||||
test_results = await run_demo_tests(server)
|
||||
logger.info(f"Demo tests: {test_results['test_status']}")
|
||||
|
||||
if test_results["test_status"] == "passed":
|
||||
logger.info(f"Server ready with {test_results['tools_available']} tools")
|
||||
|
||||
# Start the FastMCP server
|
||||
logger.info("🎬 Starting FastMCP Feedback Demo Server...")
|
||||
await server.run()
|
||||
|
||||
except KeyboardInterrupt:
|
||||
logger.info("Demo server stopped by user")
|
||||
except Exception as e:
|
||||
logger.error(f"Demo server failed: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,396 @@
|
||||
# FastMCP Feedback Production Server
|
||||
|
||||
**File:** `mcp_server.py`
|
||||
**Purpose:** Production-ready MCP server with environment configuration
|
||||
**Lines:** 89 lines of deployment-focused code
|
||||
**Target:** Production and staging environments
|
||||
|
||||
## 🎯 Overview
|
||||
|
||||
This is a production-ready MCP server launcher designed for real-world deployments. It demonstrates environment-based configuration, proper logging, and integration patterns suitable for staging and production environments.
|
||||
|
||||
## ✨ Production Features
|
||||
|
||||
### Environment Configuration
|
||||
- **Database URL** - Configurable via `DATABASE_URL` environment variable
|
||||
- **Insights Control** - Enable/disable analytics via `INSIGHTS_ENABLED`
|
||||
- **Logging Levels** - Configurable via `LOG_LEVEL` (DEBUG, INFO, WARNING, ERROR)
|
||||
- **Graceful Defaults** - Sensible fallbacks for missing configuration
|
||||
|
||||
### Security Considerations
|
||||
- **No Hardcoded Secrets** - All sensitive data via environment variables
|
||||
- **Input Validation** - Comprehensive Pydantic validation
|
||||
- **Error Handling** - Graceful failure without exposing internals
|
||||
- **Audit Logging** - Configurable logging for compliance
|
||||
|
||||
### Deployment Ready
|
||||
- **Docker Compatible** - Environment variable based configuration
|
||||
- **Kubernetes Ready** - ConfigMap and Secret integration
|
||||
- **Load Balancer Friendly** - Stateless design
|
||||
- **Health Checks** - Built-in system info endpoint
|
||||
|
||||
## 🚀 Usage
|
||||
|
||||
### Local Development
|
||||
```bash
|
||||
# Run with default settings
|
||||
python examples/mcp_server.py
|
||||
|
||||
# Run with uv (recommended)
|
||||
uv run examples/mcp_server.py
|
||||
|
||||
# Run with custom database
|
||||
DATABASE_URL="postgresql://localhost/feedback" uv run examples/mcp_server.py
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
```bash
|
||||
# Required/Optional Configuration
|
||||
export DATABASE_URL="postgresql://user:pass@host:5432/feedback_prod"
|
||||
export INSIGHTS_ENABLED="true"
|
||||
export LOG_LEVEL="INFO"
|
||||
export RETENTION_DAYS="90"
|
||||
|
||||
# Run the server
|
||||
uv run examples/mcp_server.py
|
||||
```
|
||||
|
||||
## 🔧 Configuration Options
|
||||
|
||||
### Database Configuration
|
||||
```bash
|
||||
# SQLite (development/testing)
|
||||
DATABASE_URL="sqlite:///feedback.db"
|
||||
|
||||
# PostgreSQL (production)
|
||||
DATABASE_URL="postgresql://user:password@host:5432/database"
|
||||
|
||||
# PostgreSQL with SSL (production)
|
||||
DATABASE_URL="postgresql://user:pass@host:5432/db?sslmode=require"
|
||||
|
||||
# Environment-specific databases
|
||||
DATABASE_URL_DEV="sqlite:///feedback_dev.db"
|
||||
DATABASE_URL_STAGING="postgresql://staging_user:pass@staging.db/feedback"
|
||||
DATABASE_URL_PROD="postgresql://prod_user:pass@prod.db/feedback"
|
||||
```
|
||||
|
||||
### Analytics Configuration
|
||||
```bash
|
||||
# Enable analytics with retention
|
||||
INSIGHTS_ENABLED="true"
|
||||
RETENTION_DAYS="90"
|
||||
|
||||
# Disable analytics (GDPR compliance)
|
||||
INSIGHTS_ENABLED="false"
|
||||
|
||||
# Different retention by environment
|
||||
RETENTION_DAYS_DEV="7"
|
||||
RETENTION_DAYS_STAGING="30"
|
||||
RETENTION_DAYS_PROD="365"
|
||||
```
|
||||
|
||||
### Logging Configuration
|
||||
```bash
|
||||
# Production logging
|
||||
LOG_LEVEL="INFO"
|
||||
LOG_FORMAT="json" # For log aggregation
|
||||
|
||||
# Development logging
|
||||
LOG_LEVEL="DEBUG"
|
||||
LOG_FORMAT="standard"
|
||||
|
||||
# Disable logging (not recommended)
|
||||
LOG_LEVEL="CRITICAL"
|
||||
```
|
||||
|
||||
## 🐳 Docker Deployment
|
||||
|
||||
### Dockerfile
|
||||
```dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Install uv for faster dependency management
|
||||
RUN pip install uv
|
||||
|
||||
# Copy dependency files
|
||||
COPY pyproject.toml uv.lock ./
|
||||
|
||||
# Install dependencies
|
||||
RUN uv sync --frozen
|
||||
|
||||
# Copy application code
|
||||
COPY . .
|
||||
|
||||
# Set environment variables
|
||||
ENV DATABASE_URL="postgresql://postgres:password@db:5432/feedback"
|
||||
ENV INSIGHTS_ENABLED="true"
|
||||
ENV LOG_LEVEL="INFO"
|
||||
|
||||
# Expose port (if needed)
|
||||
EXPOSE 8000
|
||||
|
||||
# Run the MCP server
|
||||
CMD ["uv", "run", "examples/mcp_server.py"]
|
||||
```
|
||||
|
||||
### Docker Compose
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
feedback-server:
|
||||
build: .
|
||||
environment:
|
||||
- DATABASE_URL=postgresql://postgres:password@db:5432/feedback
|
||||
- INSIGHTS_ENABLED=true
|
||||
- LOG_LEVEL=INFO
|
||||
- RETENTION_DAYS=90
|
||||
depends_on:
|
||||
- db
|
||||
restart: unless-stopped
|
||||
|
||||
db:
|
||||
image: postgres:15
|
||||
environment:
|
||||
- POSTGRES_DB=feedback
|
||||
- POSTGRES_USER=postgres
|
||||
- POSTGRES_PASSWORD=password
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
```
|
||||
|
||||
## ☸️ Kubernetes Deployment
|
||||
|
||||
### ConfigMap
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: feedback-config
|
||||
data:
|
||||
LOG_LEVEL: "INFO"
|
||||
INSIGHTS_ENABLED: "true"
|
||||
RETENTION_DAYS: "90"
|
||||
```
|
||||
|
||||
### Secret
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: feedback-secrets
|
||||
type: Opaque
|
||||
stringData:
|
||||
DATABASE_URL: "postgresql://user:password@postgres-service:5432/feedback"
|
||||
```
|
||||
|
||||
### Deployment
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: feedback-server
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: feedback-server
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: feedback-server
|
||||
spec:
|
||||
containers:
|
||||
- name: feedback-server
|
||||
image: fastmcp-feedback:latest
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: feedback-config
|
||||
- secretRef:
|
||||
name: feedback-secrets
|
||||
resources:
|
||||
requests:
|
||||
memory: "256Mi"
|
||||
cpu: "250m"
|
||||
limits:
|
||||
memory: "512Mi"
|
||||
cpu: "500m"
|
||||
```
|
||||
|
||||
## 🎛️ Claude Desktop Configuration
|
||||
|
||||
### Development
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"fastmcp-feedback-dev": {
|
||||
"command": "uv",
|
||||
"args": ["run", "examples/mcp_server.py"],
|
||||
"env": {
|
||||
"DATABASE_URL": "sqlite:///feedback_dev.db",
|
||||
"INSIGHTS_ENABLED": "true",
|
||||
"LOG_LEVEL": "DEBUG"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Production
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"fastmcp-feedback-prod": {
|
||||
"command": "uv",
|
||||
"args": ["run", "examples/mcp_server.py"],
|
||||
"env": {
|
||||
"DATABASE_URL": "postgresql://user:pass@prod.db/feedback",
|
||||
"INSIGHTS_ENABLED": "true",
|
||||
"LOG_LEVEL": "INFO",
|
||||
"RETENTION_DAYS": "365"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 📊 Monitoring & Observability
|
||||
|
||||
### Health Checks
|
||||
```python
|
||||
# Built-in health check endpoint
|
||||
await get_system_info()
|
||||
# Returns: version, dependencies, database_status, insights_status
|
||||
```
|
||||
|
||||
### Logging Format
|
||||
```json
|
||||
{
|
||||
"timestamp": "2024-01-15T10:30:00Z",
|
||||
"level": "INFO",
|
||||
"logger": "fastmcp_feedback.tools",
|
||||
"message": "Feedback submitted successfully",
|
||||
"feedback_id": "123",
|
||||
"submitter_hash": "a1b2c3",
|
||||
"type": "bug"
|
||||
}
|
||||
```
|
||||
|
||||
### Metrics Collection
|
||||
- **Request Counts** - Total submissions, retrievals, updates
|
||||
- **Error Rates** - Failed operations and reasons
|
||||
- **Performance** - Response times and database query duration
|
||||
- **Usage Patterns** - Popular feedback types, status distributions
|
||||
|
||||
## 🔍 Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Database Connection Errors**
|
||||
```bash
|
||||
# Check database connectivity
|
||||
DATABASE_URL="postgresql://..." python -c "
|
||||
import asyncio
|
||||
from fastmcp_feedback import FeedbackDatabase
|
||||
async def test():
|
||||
db = FeedbackDatabase('$DATABASE_URL')
|
||||
await db.initialize()
|
||||
print('✅ Database connection successful')
|
||||
asyncio.run(test())
|
||||
"
|
||||
```
|
||||
|
||||
**Environment Variable Issues**
|
||||
```bash
|
||||
# Verify environment variables
|
||||
env | grep -E "(DATABASE_URL|INSIGHTS_ENABLED|LOG_LEVEL)"
|
||||
|
||||
# Test with explicit variables
|
||||
DATABASE_URL="sqlite:///test.db" \
|
||||
INSIGHTS_ENABLED="false" \
|
||||
LOG_LEVEL="DEBUG" \
|
||||
python examples/mcp_server.py
|
||||
```
|
||||
|
||||
**Permission Issues**
|
||||
```bash
|
||||
# Check file permissions
|
||||
ls -la feedback.db
|
||||
|
||||
# Check directory permissions
|
||||
ls -ld $(dirname $(readlink -f feedback.db))
|
||||
|
||||
# Fix permissions
|
||||
chmod 644 feedback.db
|
||||
chmod 755 $(dirname $(readlink -f feedback.db))
|
||||
```
|
||||
|
||||
### Performance Tuning
|
||||
|
||||
**Database Optimization**
|
||||
```bash
|
||||
# PostgreSQL connection pooling
|
||||
DATABASE_URL="postgresql://user:pass@host/db?pool_size=20&max_overflow=30"
|
||||
|
||||
# SQLite optimization
|
||||
DATABASE_URL="sqlite:///feedback.db?journal_mode=WAL&synchronous=NORMAL"
|
||||
```
|
||||
|
||||
**Memory Optimization**
|
||||
```bash
|
||||
# Limit analytics retention
|
||||
RETENTION_DAYS="30"
|
||||
|
||||
# Disable insights in memory-constrained environments
|
||||
INSIGHTS_ENABLED="false"
|
||||
```
|
||||
|
||||
## 🚀 Scaling Considerations
|
||||
|
||||
### Horizontal Scaling
|
||||
- **Stateless Design** - No server-side session storage
|
||||
- **Database Sharing** - Multiple instances can share database
|
||||
- **Load Balancer Compatible** - Standard HTTP/JSON interfaces
|
||||
|
||||
### Database Scaling
|
||||
- **Read Replicas** - Use read-only database URLs for list operations
|
||||
- **Connection Pooling** - Configure appropriate pool sizes
|
||||
- **Query Optimization** - Built-in indexing and pagination
|
||||
|
||||
### Caching Strategy
|
||||
```python
|
||||
# Redis caching for statistics (optional)
|
||||
REDIS_URL="redis://localhost:6379/0"
|
||||
CACHE_TTL="300" # 5 minutes
|
||||
|
||||
# Memcached alternative
|
||||
MEMCACHED_SERVERS="localhost:11211"
|
||||
```
|
||||
|
||||
## 📚 Related Documentation
|
||||
|
||||
- **[Demo Server](demo_server.md)** - Complete feature showcase
|
||||
- **[Simple Integration](simple_integration.md)** - Minimal integration
|
||||
- **[Architecture Guide](../docs/architecture.md)** - System design
|
||||
- **[Security Guide](../docs/security.md)** - Security best practices
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
When modifying the production server:
|
||||
|
||||
1. **Maintain Backward Compatibility** - Don't break existing environment variables
|
||||
2. **Add Configuration Validation** - Validate new environment variables
|
||||
3. **Update Documentation** - Keep this file synchronized with changes
|
||||
4. **Test in Multiple Environments** - Docker, Kubernetes, bare metal
|
||||
5. **Consider Security Implications** - Audit any configuration changes
|
||||
|
||||
This production server demonstrates enterprise-ready deployment patterns while maintaining the simplicity and power of the FastMCP Feedback system.
|
||||
@@ -0,0 +1,110 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastMCP Feedback - MCP Server Launcher
|
||||
|
||||
This script provides a clean MCP server launcher that can be easily
|
||||
configured in Claude Desktop or other MCP clients.
|
||||
|
||||
Usage:
|
||||
# Direct execution
|
||||
python examples/mcp_server.py
|
||||
|
||||
# With uv
|
||||
uv run examples/mcp_server.py
|
||||
|
||||
# Claude Desktop configuration:
|
||||
{
|
||||
"mcpServers": {
|
||||
"fastmcp-feedback": {
|
||||
"command": "uv",
|
||||
"args": ["run", "examples/mcp_server.py"],
|
||||
"env": {
|
||||
"DATABASE_URL": "sqlite:///feedback.db",
|
||||
"INSIGHTS_ENABLED": "true",
|
||||
"LOG_LEVEL": "INFO"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add the src directory to the path for development
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
|
||||
|
||||
from fastmcp import Client, FastMCP
|
||||
|
||||
from fastmcp_feedback import FeedbackInsights, add_feedback_tools
|
||||
|
||||
# Configure logging based on environment
|
||||
log_level = os.getenv("LOG_LEVEL", "INFO").upper()
|
||||
logging.basicConfig(
|
||||
level=getattr(logging, log_level),
|
||||
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def create_production_server() -> FastMCP:
|
||||
"""Create a production-ready FastMCP Feedback server."""
|
||||
# Server configuration
|
||||
server_name = os.getenv("SERVER_NAME", "FastMCP Feedback Server")
|
||||
database_url = os.getenv("DATABASE_URL", "sqlite:///feedback.db")
|
||||
|
||||
# Create server
|
||||
server = FastMCP(server_name)
|
||||
server.description = "Production FastMCP Feedback collection system"
|
||||
|
||||
# Configure insights
|
||||
insights_enabled = os.getenv("INSIGHTS_ENABLED", "true").lower() == "true"
|
||||
retention_days = int(os.getenv("INSIGHTS_RETENTION_DAYS", "90"))
|
||||
|
||||
insights = FeedbackInsights(enabled=insights_enabled, retention_days=retention_days)
|
||||
|
||||
# Add feedback tools
|
||||
tool_prefix = os.getenv("TOOL_PREFIX", "")
|
||||
add_feedback_tools(
|
||||
server, database_url=database_url, insights=insights, prefix=tool_prefix
|
||||
)
|
||||
|
||||
logger.info(f"Server '{server_name}' configured")
|
||||
logger.info(f"Database: {database_url}")
|
||||
logger.info(f"Insights: {'enabled' if insights_enabled else 'disabled'}")
|
||||
logger.info(f"Tool prefix: '{tool_prefix}' (empty = no prefix)")
|
||||
|
||||
return server
|
||||
|
||||
|
||||
async def main():
|
||||
"""Main entry point for the MCP server."""
|
||||
try:
|
||||
logger.info("🚀 Starting FastMCP Feedback Server")
|
||||
|
||||
# Create and start server
|
||||
server = create_production_server()
|
||||
|
||||
# Show available tools
|
||||
async with Client(server) as client:
|
||||
tools = await client.list_tools()
|
||||
logger.info(f"Server ready with {len(tools)} tools:")
|
||||
for tool in tools:
|
||||
logger.info(f" • {tool.name}")
|
||||
|
||||
# Start the server
|
||||
await server.run()
|
||||
|
||||
except KeyboardInterrupt:
|
||||
logger.info("Server stopped by user")
|
||||
except Exception as e:
|
||||
logger.error(f"Server failed: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,274 @@
|
||||
# Simple FastMCP Feedback Integration
|
||||
|
||||
**File:** `simple_integration.py`
|
||||
**Purpose:** Minimal 3-line integration example
|
||||
**Lines:** 48 lines of clean, documented code
|
||||
**Complexity:** Beginner-friendly
|
||||
|
||||
## 🎯 Overview
|
||||
|
||||
This example demonstrates the absolute minimal integration of FastMCP Feedback into any FastMCP server. With just 3 lines of code, you get a complete, production-ready feedback system.
|
||||
|
||||
## ✨ Key Features
|
||||
|
||||
### Minimal Code
|
||||
```python
|
||||
# This is all you need - 3 lines of code!
|
||||
app = FastMCP("Simple Feedback App")
|
||||
add_feedback_tools(app, database_url="sqlite:///simple_feedback.db")
|
||||
```
|
||||
|
||||
### Automatic Tool Registration
|
||||
The integration automatically adds 5 production-ready tools:
|
||||
- ✅ `submit_feedback` - Submit bug reports & feature requests
|
||||
- ✅ `list_feedback` - Query and filter feedback
|
||||
- ✅ `get_feedback_statistics` - Analytics and metrics
|
||||
- ✅ `delete_feedback` - Remove feedback items
|
||||
- ✅ `update_feedback_status` - Change status (admin)
|
||||
|
||||
### Zero Configuration
|
||||
- **Database** - Automatically creates SQLite database
|
||||
- **Schema** - Auto-generates tables on first run
|
||||
- **Validation** - Built-in Pydantic input validation
|
||||
- **Error Handling** - Graceful failure scenarios
|
||||
|
||||
## 🚀 Usage
|
||||
|
||||
### Direct Execution
|
||||
```bash
|
||||
# Run the example
|
||||
python examples/simple_integration.py
|
||||
|
||||
# With uv (recommended)
|
||||
uv run examples/simple_integration.py
|
||||
```
|
||||
|
||||
### Expected Output
|
||||
```
|
||||
🚀 Simple FastMCP Feedback Integration
|
||||
==================================================
|
||||
✅ Feedback system added!
|
||||
📊 Available tools:
|
||||
• submit_feedback - Submit feedback to the system
|
||||
• get_feedback_statistics - Get feedback statistics and analytics
|
||||
• list_feedback - List feedback items with optional filtering
|
||||
• delete_feedback - Delete feedback item
|
||||
• update_feedback_status - Update feedback status
|
||||
|
||||
🎯 Total tools: 5
|
||||
|
||||
🎬 Starting server...
|
||||
Connect this as an MCP server in Claude Desktop
|
||||
```
|
||||
|
||||
## 📋 Integration Steps
|
||||
|
||||
### Step 1: Install Dependencies
|
||||
```bash
|
||||
# Using uv (recommended)
|
||||
uv add fastmcp-feedback
|
||||
|
||||
# Using pip
|
||||
pip install fastmcp-feedback
|
||||
```
|
||||
|
||||
### Step 2: Add to Your Server
|
||||
```python
|
||||
from fastmcp import FastMCP
|
||||
from fastmcp_feedback import add_feedback_tools
|
||||
|
||||
# Your existing FastMCP server
|
||||
app = FastMCP("My Awesome App")
|
||||
|
||||
# Add your existing tools here...
|
||||
# app.add_tool(my_custom_tool)
|
||||
|
||||
# Add complete feedback system in one line
|
||||
add_feedback_tools(app, database_url="sqlite:///feedback.db")
|
||||
```
|
||||
|
||||
### Step 3: Configure Claude Desktop
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"my-app-with-feedback": {
|
||||
"command": "python",
|
||||
"args": ["my_server.py"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🔧 Customization Options
|
||||
|
||||
### Database Configuration
|
||||
```python
|
||||
# SQLite (development)
|
||||
add_feedback_tools(app, database_url="sqlite:///feedback.db")
|
||||
|
||||
# PostgreSQL (production)
|
||||
add_feedback_tools(app, database_url="postgresql://user:pass@host/db")
|
||||
|
||||
# In-memory (testing)
|
||||
add_feedback_tools(app, database_url="sqlite:///:memory:")
|
||||
```
|
||||
|
||||
### Analytics Configuration
|
||||
```python
|
||||
from fastmcp_feedback import add_feedback_tools, FeedbackInsights
|
||||
|
||||
# Enable analytics with custom retention
|
||||
insights = FeedbackInsights(enabled=True, retention_days=30)
|
||||
add_feedback_tools(app, insights=insights)
|
||||
```
|
||||
|
||||
### Tool Prefixing
|
||||
```python
|
||||
# Add with prefix to avoid naming conflicts
|
||||
add_feedback_tools(app, prefix="support_")
|
||||
# Creates: support_submit_feedback, support_list_feedback, etc.
|
||||
```
|
||||
|
||||
## 📊 What You Get
|
||||
|
||||
### Complete CRUD Operations
|
||||
```python
|
||||
# Users can submit feedback
|
||||
await submit_feedback({
|
||||
"type": "bug",
|
||||
"title": "App crashes on login",
|
||||
"description": "Detailed error description",
|
||||
"submitter": "user@example.com"
|
||||
})
|
||||
|
||||
# Admins can manage feedback
|
||||
await update_feedback_status({
|
||||
"feedback_id": "123",
|
||||
"new_status": "resolved"
|
||||
})
|
||||
|
||||
# Analytics for insights
|
||||
stats = await get_feedback_statistics()
|
||||
# Returns: total_count, by_type, by_status, recent_count
|
||||
```
|
||||
|
||||
### Input Validation
|
||||
- **Type Validation** - Must be: bug, feature, improvement, question
|
||||
- **Required Fields** - title, description, submitter cannot be empty
|
||||
- **Length Limits** - title ≤ 255 chars, description ≤ 10,000 chars
|
||||
- **Format Validation** - email validation for contact_info (optional)
|
||||
|
||||
### Database Schema
|
||||
Automatically creates:
|
||||
```sql
|
||||
CREATE TABLE feedback (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
type VARCHAR(50) NOT NULL,
|
||||
title VARCHAR(255) NOT NULL,
|
||||
description TEXT NOT NULL,
|
||||
submitter VARCHAR(255) NOT NULL,
|
||||
contact_info VARCHAR(255),
|
||||
status VARCHAR(50) NOT NULL DEFAULT 'open',
|
||||
created_at DATETIME NOT NULL,
|
||||
updated_at DATETIME NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
## 🎓 Learning Examples
|
||||
|
||||
### Basic Usage Pattern
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
from fastmcp import FastMCP
|
||||
from fastmcp_feedback import add_feedback_tools
|
||||
|
||||
app = FastMCP("My App")
|
||||
|
||||
# Your custom tools
|
||||
@app.tool()
|
||||
def my_custom_feature():
|
||||
return "Custom functionality"
|
||||
|
||||
# Add feedback system
|
||||
add_feedback_tools(app)
|
||||
|
||||
# Run server
|
||||
app.run()
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
```python
|
||||
try:
|
||||
add_feedback_tools(app, database_url="invalid://url")
|
||||
except Exception as e:
|
||||
print(f"Failed to add feedback tools: {e}")
|
||||
# Graceful fallback or alternative configuration
|
||||
```
|
||||
|
||||
## 🔍 Under the Hood
|
||||
|
||||
### What `add_feedback_tools()` Does
|
||||
1. **Creates Database Instance** - Initializes FeedbackDatabase with provided URL
|
||||
2. **Sets Up Insights** - Configures optional analytics collection
|
||||
3. **Registers Tools** - Adds 5 tools to your FastMCP server
|
||||
4. **Validates Configuration** - Ensures database connectivity
|
||||
5. **Logs Success** - Confirms successful integration
|
||||
|
||||
### Tool Implementation Pattern
|
||||
```python
|
||||
# Each tool is implemented as an async function
|
||||
@app.tool(description="Submit feedback to the system")
|
||||
async def submit_feedback(request: dict) -> dict:
|
||||
# Validate input with Pydantic
|
||||
validated_request = SubmitFeedbackRequest(**request)
|
||||
|
||||
# Store in database
|
||||
feedback = await database.create_feedback(validated_request)
|
||||
|
||||
# Record analytics (if enabled)
|
||||
insights.record_metric("feedback_submitted", metadata)
|
||||
|
||||
return {"success": True, "feedback_id": feedback.id}
|
||||
```
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
### For More Features
|
||||
If you need advanced capabilities, see:
|
||||
- **[Demo Server](demo_server.md)** - Complete feature showcase
|
||||
- **[Production Server](mcp_server.md)** - Environment configuration
|
||||
- **[Architecture Guide](../docs/architecture.md)** - System design
|
||||
|
||||
### For Customization
|
||||
```python
|
||||
# Use individual mixins for fine-grained control
|
||||
from fastmcp_feedback.mixins import SubmissionMixin, RetrievalMixin
|
||||
|
||||
submission = SubmissionMixin(database, insights)
|
||||
submission.register_tools(app, prefix="feedback_")
|
||||
```
|
||||
|
||||
### For Testing
|
||||
```python
|
||||
# Test your integration
|
||||
import pytest
|
||||
from fastmcp_feedback import add_feedback_tools
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_feedback_integration():
|
||||
app = FastMCP("Test App")
|
||||
add_feedback_tools(app, database_url="sqlite:///:memory:")
|
||||
|
||||
tools = await app.get_tools()
|
||||
assert "submit_feedback" in tools
|
||||
```
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
When modifying this example:
|
||||
1. **Keep It Simple** - This is the "Hello World" of FastMCP Feedback
|
||||
2. **Document Changes** - Update this file with any modifications
|
||||
3. **Test Integration** - Ensure the 3-line pattern still works
|
||||
4. **Preserve Clarity** - Maintain beginner-friendly explanations
|
||||
|
||||
This simple integration example proves that adding a complete, production-ready feedback system to any FastMCP server requires just 3 lines of code while providing enterprise-grade functionality.
|
||||
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Simple FastMCP Feedback Integration Example
|
||||
|
||||
This demonstrates the absolute minimal integration - just 3 lines of code
|
||||
to add a complete feedback system to any FastMCP server.
|
||||
|
||||
Usage:
|
||||
python examples/simple_integration.py
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add the src directory to the path for development
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
|
||||
|
||||
from fastmcp import Client, FastMCP
|
||||
|
||||
from fastmcp_feedback import add_feedback_tools
|
||||
|
||||
|
||||
async def main():
|
||||
"""Minimal FastMCP Feedback integration example."""
|
||||
|
||||
print("🚀 Simple FastMCP Feedback Integration")
|
||||
print("=" * 50)
|
||||
|
||||
# This is all you need - 3 lines of code!
|
||||
app = FastMCP("Simple Feedback App")
|
||||
add_feedback_tools(app, database_url="sqlite:///simple_feedback.db")
|
||||
|
||||
print("✅ Feedback system added!")
|
||||
print("📊 Available tools:")
|
||||
|
||||
# Show what tools were added
|
||||
async with Client(app) as client:
|
||||
tools = await client.list_tools()
|
||||
for tool in tools:
|
||||
print(f" • {tool.name} - {tool.description}")
|
||||
|
||||
print(f"\n🎯 Total tools: {len(tools)}")
|
||||
print("\n🎬 Starting server...")
|
||||
print(" Connect this as an MCP server in Claude Desktop")
|
||||
|
||||
# Start the server
|
||||
await app.run()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,90 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script to verify demo server functionality without running the full MCP server.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add the src directory to the path for development
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from demo_server import create_demo_server
|
||||
from fastmcp import Client
|
||||
|
||||
|
||||
async def test_demo_functionality():
|
||||
"""Test the demo server tools without running the MCP server."""
|
||||
print("🧪 Testing FastMCP Feedback Demo Functionality")
|
||||
print("=" * 55)
|
||||
|
||||
try:
|
||||
# Create the demo server
|
||||
server = create_demo_server()
|
||||
|
||||
async with Client(server) as client:
|
||||
# Get available tools
|
||||
tools = {tool.name: tool for tool in await client.list_tools()}
|
||||
print(f"✅ Server created with {len(tools)} tools")
|
||||
|
||||
# List available tools
|
||||
print("\n📋 Available Tools:")
|
||||
for name, tool in tools.items():
|
||||
print(f" • {name}: {tool.description}")
|
||||
|
||||
# Test system info tool
|
||||
if "get_system_info" in tools:
|
||||
print("\n🔍 Testing system info...")
|
||||
result = await client.call_tool("get_system_info", {})
|
||||
info = result.structured_content
|
||||
print(f" Version: {info['versions']['fastmcp_feedback']}")
|
||||
print(f" Python: {info['versions']['python']}")
|
||||
print(f" Features: {len(info['demo_features'])}")
|
||||
|
||||
# Test sample data setup
|
||||
if "setup_sample_data" in tools:
|
||||
print("\n📊 Testing sample data setup...")
|
||||
result = await client.call_tool("setup_sample_data", {})
|
||||
data_result = result.structured_content
|
||||
if data_result["success"]:
|
||||
print(f" ✅ Added {data_result['items_added']} sample items")
|
||||
else:
|
||||
print(f" ❌ Failed: {data_result['error']}")
|
||||
|
||||
# Test feedback listing
|
||||
if "list_feedback" in tools:
|
||||
print("\n📝 Testing feedback listing...")
|
||||
result = await client.call_tool("list_feedback", {})
|
||||
feedback_data = result.structured_content
|
||||
print(f" Found {len(feedback_data['feedback'])} feedback items")
|
||||
|
||||
# Show first item details
|
||||
if feedback_data["feedback"]:
|
||||
first = feedback_data["feedback"][0]
|
||||
print(f" First item: {first['type']} - {first['title']}")
|
||||
|
||||
# Test statistics
|
||||
if "get_feedback_statistics" in tools:
|
||||
print("\n📈 Testing statistics...")
|
||||
result = await client.call_tool("get_feedback_statistics", {})
|
||||
stats = result.structured_content
|
||||
print(f" Total items: {stats['total_count']}")
|
||||
print(f" By type: {stats['by_type']}")
|
||||
print(f" By status: {stats['by_status']}")
|
||||
|
||||
print("\n🎉 All tests passed! Demo server is working correctly.")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Test failed: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = asyncio.run(test_demo_functionality())
|
||||
sys.exit(0 if success else 1)
|
||||
+18
-3
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "fastmcp-feedback"
|
||||
version = "2026.01.12.1" # YYYYMMDD.patch versioning - update for each release
|
||||
version = "2026.09.27.4" # YYYYMMDD.patch versioning - update for each release
|
||||
description = "Production-ready feedback collection system for FastMCP servers"
|
||||
readme = "README.md"
|
||||
license = "MIT"
|
||||
@@ -43,27 +43,35 @@ classifiers = [
|
||||
]
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"fastmcp>=2.12.2",
|
||||
"fastmcp>=2.12.2,<5", # CI tests 2.12, 2.14 and 4.0; raise the ceiling once 5.x passes
|
||||
"sqlalchemy>=2.0.43",
|
||||
"pydantic>=2.11.7"
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
postgresql = ["psycopg2-binary>=2.9.0"]
|
||||
# Async database sink for per-call instrumentation. Postgres hosts add asyncpg.
|
||||
instrumentation = ["sqlalchemy[asyncio]>=2.0.43", "aiosqlite>=0.22.1"]
|
||||
mysql = ["pymysql>=1.1.0"]
|
||||
dev = [
|
||||
"sqlalchemy[asyncio]>=2.0.43",
|
||||
"aiosqlite>=0.22.1",
|
||||
"pytest>=8.2.0",
|
||||
"pytest-asyncio>=0.23.0",
|
||||
"pytest-cov>=5.0.0",
|
||||
"pytest-mock>=3.12.0",
|
||||
"pytest-html>=4.2.0",
|
||||
"ruff>=0.1.0",
|
||||
"mypy>=1.7.0",
|
||||
]
|
||||
test = [
|
||||
"sqlalchemy[asyncio]>=2.0.43",
|
||||
"aiosqlite>=0.22.1",
|
||||
"pytest>=8.2.0",
|
||||
"pytest-asyncio>=0.23.0",
|
||||
"pytest-cov>=5.0.0",
|
||||
"pytest-mock>=3.12.0",
|
||||
"pytest-html>=4.2.0",
|
||||
]
|
||||
all = ["fastmcp-feedback[postgresql,mysql,dev]"]
|
||||
|
||||
@@ -81,6 +89,7 @@ packages = ["src/fastmcp_feedback"]
|
||||
include = [
|
||||
"/src",
|
||||
"/tests",
|
||||
"/examples",
|
||||
"README.md",
|
||||
"LICENSE",
|
||||
"CHANGELOG.md",
|
||||
@@ -142,7 +151,13 @@ src = ["src", "tests"]
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "W", "F", "UP", "B", "SIM", "I", "N", "C4"]
|
||||
ignore = ["E501", "B008"]
|
||||
ignore = [
|
||||
"E501",
|
||||
"B008",
|
||||
# StrEnum changes str()/format() output of FeedbackType/FeedbackStatus,
|
||||
# which reach the database and API responses. Worth doing, not silently.
|
||||
"UP042",
|
||||
]
|
||||
|
||||
[tool.ruff.lint.per-file-ignores]
|
||||
"tests/**/*.py" = ["ARG", "PLR2004"]
|
||||
|
||||
@@ -44,7 +44,7 @@ Quick Start:
|
||||
import logging
|
||||
|
||||
# Version information - CalVer YYYY.MM.DD
|
||||
__version__ = "2026.01.12.1"
|
||||
__version__ = "2026.09.27.4"
|
||||
__author__ = "Ryan Malloy"
|
||||
__email__ = "ryan@supported.systems"
|
||||
|
||||
@@ -259,8 +259,8 @@ from fastmcp_feedback import create_feedback_server
|
||||
main_app = FastMCP("Main Application")
|
||||
feedback_service = create_feedback_server("Feedback Service")
|
||||
|
||||
# Import feedback service with prefix
|
||||
main_app.import_server(feedback_service, prefix="feedback")
|
||||
# Mount feedback service; its tools become feedback_<name>
|
||||
main_app.mount(feedback_service, "feedback")
|
||||
""",
|
||||
"multi_tenant": """
|
||||
from fastmcp import FastMCP
|
||||
|
||||
@@ -260,8 +260,10 @@ def compose_feedback_services(
|
||||
f"{service_name} Service", database_url, shared_insights
|
||||
)
|
||||
|
||||
# Import service with prefix
|
||||
main_server.import_server(service_server, prefix=service_name)
|
||||
# mount() is sync on every supported FastMCP; the second positional
|
||||
# arg is `prefix` on 2.x and `namespace` on 4.x. import_server() is
|
||||
# gone in 4.x and was async (so silently never ran here) on 2.14.
|
||||
main_server.mount(service_server, service_name)
|
||||
|
||||
logger.info(f"Composed {len(services)} feedback services into main server")
|
||||
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Per-call instrumentation for any FastMCP server.
|
||||
|
||||
from fastmcp_feedback.instrumentation import instrument, DatabaseSink
|
||||
|
||||
mw = instrument(app) # every tool call as JSON lines on stderr
|
||||
mw = instrument(app, [DatabaseSink(engine)], identity_resolver=who)
|
||||
mw = instrument(app, result_classifier=soft_error_classifier(extra_statuses=("unknown_target",)))
|
||||
"""
|
||||
|
||||
from .classify import (
|
||||
DEFAULT_RESULT_CLASSIFIER,
|
||||
ResultClassifier,
|
||||
soft_error_classifier,
|
||||
)
|
||||
from .dispatch import RecordDispatcher
|
||||
from .middleware import MODE_ENV_VAR, MODES, InstrumentationMiddleware, instrument
|
||||
from .record import ToolCallRecord
|
||||
from .redaction import (
|
||||
DEFAULT_SECRET_KEYS,
|
||||
DEFAULT_SECRET_PATTERNS,
|
||||
REDACTED,
|
||||
Redactor,
|
||||
)
|
||||
from .sinks import CallbackSink, JsonLinesSink, MemorySink, Sink
|
||||
from .storage import DEFAULT_TABLE_PREFIX, DatabaseSink, build_metadata
|
||||
|
||||
__all__ = [
|
||||
"CallbackSink",
|
||||
"DatabaseSink",
|
||||
"DEFAULT_RESULT_CLASSIFIER",
|
||||
"DEFAULT_SECRET_KEYS",
|
||||
"DEFAULT_SECRET_PATTERNS",
|
||||
"DEFAULT_TABLE_PREFIX",
|
||||
"InstrumentationMiddleware",
|
||||
"JsonLinesSink",
|
||||
"MemorySink",
|
||||
"MODE_ENV_VAR",
|
||||
"MODES",
|
||||
"RecordDispatcher",
|
||||
"REDACTED",
|
||||
"Redactor",
|
||||
"ResultClassifier",
|
||||
"Sink",
|
||||
"ToolCallRecord",
|
||||
"build_metadata",
|
||||
"instrument",
|
||||
"soft_error_classifier",
|
||||
]
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Spotting tools that report failure as data instead of raising.
|
||||
|
||||
A result classifier is ``(tool, result) -> str | None``. ``result`` is the
|
||||
recorded payload: the structured content when there is one, otherwise the
|
||||
content items. None means the call succeeded; a string marks it as a soft
|
||||
error and becomes the record's ``error_message`` (after redaction).
|
||||
|
||||
A result the tool marked as an MCP error (``isError``) without raising reaches
|
||||
the classifier as a dict with ``"isError": True``, alongside the payload's own
|
||||
keys or with the payload under ``"content"``.
|
||||
"""
|
||||
|
||||
from collections.abc import Callable, Iterable
|
||||
from typing import Any
|
||||
|
||||
ResultClassifier = Callable[[str, Any], str | None]
|
||||
|
||||
MESSAGE_KEYS = ("error", "message", "detail", "reason")
|
||||
ERROR_FLAG_KEYS = ("isError", "is_error")
|
||||
|
||||
|
||||
def _message(result: dict, depth: int = 0) -> str | None:
|
||||
for key in MESSAGE_KEYS:
|
||||
value = result.get(key)
|
||||
if value is None or value == "":
|
||||
continue
|
||||
if isinstance(value, dict):
|
||||
# {"error": {"code": 404, "message": "..."}}
|
||||
if depth == 0 and (inner := _message(value, 1)):
|
||||
return inner
|
||||
continue
|
||||
if isinstance(value, list | tuple):
|
||||
continue
|
||||
return str(value)
|
||||
return None
|
||||
|
||||
|
||||
def _content_text(content: Any) -> str | None:
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if not isinstance(content, list | tuple):
|
||||
return None
|
||||
texts = [
|
||||
item if isinstance(item, str) else item.get("text")
|
||||
for item in content
|
||||
if isinstance(item, str | dict)
|
||||
]
|
||||
return " ".join(t for t in texts if t) or None
|
||||
|
||||
|
||||
def _describe(label: str, detail: str | None) -> str:
|
||||
return f"{label}: {detail}" if detail else label
|
||||
|
||||
|
||||
def soft_error_classifier(
|
||||
*,
|
||||
statuses: Iterable[str] = ("error", "failed", "failure"),
|
||||
status_keys: Iterable[str] = ("status", "state"),
|
||||
flag_keys: Iterable[str] = ("ok", "success"),
|
||||
extra_statuses: Iterable[str] = (),
|
||||
) -> ResultClassifier:
|
||||
"""Build a classifier for dict results that describe their own failure.
|
||||
|
||||
A dict result is a soft error when any of ``flag_keys`` is exactly False,
|
||||
when any of ``status_keys`` holds one of ``statuses`` or
|
||||
``extra_statuses`` (case-insensitive), or when the tool marked it
|
||||
``isError``. The message is the first of ``error``, ``message``,
|
||||
``detail`` and ``reason`` present, e.g. ``status=failed: disk full``.
|
||||
|
||||
>>> classify = soft_error_classifier(extra_statuses=("unknown_target",))
|
||||
>>> classify("render", {"status": "unknown_target", "error": "no such object"})
|
||||
'status=unknown_target: no such object'
|
||||
"""
|
||||
bad = frozenset(s.lower() for s in (*statuses, *extra_statuses))
|
||||
status_keys = tuple(status_keys)
|
||||
flag_keys = tuple(flag_keys)
|
||||
|
||||
def classify(tool: str, result: Any) -> str | None:
|
||||
if not isinstance(result, dict):
|
||||
return None
|
||||
for key in status_keys:
|
||||
value = result.get(key)
|
||||
if isinstance(value, str) and value.lower() in bad:
|
||||
return _describe(f"{key}={value}", _message(result))
|
||||
for key in flag_keys:
|
||||
if key in result and result[key] is False:
|
||||
return _describe(f"{key}=False", _message(result))
|
||||
for key in ERROR_FLAG_KEYS:
|
||||
if result.get(key) is True:
|
||||
detail = _message(result) or _content_text(result.get("content"))
|
||||
return _describe("isError", detail)
|
||||
return None
|
||||
|
||||
return classify
|
||||
|
||||
|
||||
DEFAULT_RESULT_CLASSIFIER = soft_error_classifier()
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Linking feedback to the tool calls that came before it."""
|
||||
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import asdict, dataclass
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
RULE_SESSION = "session"
|
||||
RULE_USER_WINDOW = "user_window"
|
||||
|
||||
|
||||
@dataclass
|
||||
class CallLink:
|
||||
"""A tool call linked to a piece of feedback, and why it matched."""
|
||||
|
||||
call_id: str
|
||||
rule: str
|
||||
tool: str
|
||||
started_at: datetime
|
||||
ok: bool
|
||||
position: int = 0
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
data = asdict(self)
|
||||
data["started_at"] = self.started_at.isoformat()
|
||||
return data
|
||||
|
||||
|
||||
@dataclass
|
||||
class Candidate:
|
||||
"""The fields correlation needs, from a buffered record or a database row."""
|
||||
|
||||
id: str
|
||||
tool: str
|
||||
started_at: datetime
|
||||
ok: bool
|
||||
session_id: str | None
|
||||
user_sub: str | None
|
||||
|
||||
|
||||
class ResolverContext:
|
||||
"""What an identity resolver receives when called during correlation.
|
||||
|
||||
Outside a tool call there is no MiddlewareContext, so this carries only the
|
||||
``fastmcp_context`` attribute, which is what resolvers normally read.
|
||||
"""
|
||||
|
||||
def __init__(self, fastmcp_context: Any):
|
||||
self.fastmcp_context = fastmcp_context
|
||||
self.message = None
|
||||
|
||||
|
||||
def as_utc(value: datetime) -> datetime:
|
||||
# SQLite drops the timezone from DateTime(timezone=True) columns.
|
||||
return value.replace(tzinfo=UTC) if value.tzinfo is None else value
|
||||
|
||||
|
||||
def select_calls(
|
||||
candidates: Iterable[Candidate],
|
||||
*,
|
||||
session_id: str | None,
|
||||
user_sub: str | None,
|
||||
since: datetime,
|
||||
limit: int,
|
||||
) -> list[CallLink]:
|
||||
"""Pick up to ``limit`` recent calls for a feedback item.
|
||||
|
||||
Calls from the same MCP session come first. If there are fewer than
|
||||
``limit``, the rest are filled with the same user's calls since ``since``,
|
||||
which covers clients that open a new session per call (PATs, CI). The
|
||||
result is in chronological order, numbered from 0.
|
||||
"""
|
||||
unique: dict[str, Candidate] = {}
|
||||
for c in candidates:
|
||||
unique.setdefault(c.id, c)
|
||||
newest_first = sorted(
|
||||
unique.values(), key=lambda c: as_utc(c.started_at), reverse=True
|
||||
)
|
||||
|
||||
chosen: list[tuple[Candidate, str]] = []
|
||||
if session_id:
|
||||
chosen += [
|
||||
(c, RULE_SESSION) for c in newest_first if c.session_id == session_id
|
||||
][:limit]
|
||||
if user_sub and len(chosen) < limit:
|
||||
taken = {c.id for c, _ in chosen}
|
||||
chosen += [
|
||||
(c, RULE_USER_WINDOW)
|
||||
for c in newest_first
|
||||
if c.id not in taken
|
||||
and c.user_sub == user_sub
|
||||
and as_utc(c.started_at) >= since
|
||||
][: limit - len(chosen)]
|
||||
|
||||
chosen.sort(key=lambda pair: as_utc(pair[0].started_at))
|
||||
return [
|
||||
CallLink(
|
||||
call_id=c.id,
|
||||
rule=rule,
|
||||
tool=c.tool,
|
||||
started_at=as_utc(c.started_at),
|
||||
ok=c.ok,
|
||||
position=i,
|
||||
)
|
||||
for i, (c, rule) in enumerate(chosen)
|
||||
]
|
||||
@@ -0,0 +1,117 @@
|
||||
"""Background delivery of records to sinks, off the tool-call path."""
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import logging
|
||||
from collections.abc import Sequence
|
||||
|
||||
from .record import ToolCallRecord
|
||||
from .sinks import Sink
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class RecordDispatcher:
|
||||
"""Queue records and hand them to sinks from a background task.
|
||||
|
||||
``submit`` never blocks and never raises: when the queue is full the record
|
||||
is dropped and counted, so a slow or broken sink cannot slow down or fail
|
||||
a tool call. Sink errors are logged and counted, never propagated.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
sinks: Sequence[Sink],
|
||||
*,
|
||||
max_queue: int = 1000,
|
||||
batch_size: int = 100,
|
||||
sink_timeout: float = 10.0,
|
||||
):
|
||||
self.sinks = list(sinks)
|
||||
self.max_queue = max_queue
|
||||
self.batch_size = batch_size
|
||||
self.sink_timeout = sink_timeout
|
||||
self.dropped = 0
|
||||
self.sink_errors = 0
|
||||
self._queue: asyncio.Queue[ToolCallRecord] | None = None
|
||||
self._task: asyncio.Task | None = None
|
||||
self._loop: asyncio.AbstractEventLoop | None = None
|
||||
|
||||
def submit(self, record: ToolCallRecord) -> bool:
|
||||
"""Queue a record for delivery. Returns False if it was dropped."""
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
except RuntimeError:
|
||||
self._drop("no running event loop")
|
||||
return False
|
||||
|
||||
# A queue belongs to one loop; tests and some hosts run several.
|
||||
if self._queue is None or self._loop is not loop:
|
||||
self._loop = loop
|
||||
self._queue = asyncio.Queue(maxsize=self.max_queue)
|
||||
self._task = loop.create_task(self._run(self._queue))
|
||||
|
||||
try:
|
||||
self._queue.put_nowait(record)
|
||||
except asyncio.QueueFull:
|
||||
self._drop("queue full")
|
||||
return False
|
||||
return True
|
||||
|
||||
def _drop(self, reason: str) -> None:
|
||||
self.dropped += 1
|
||||
if self.dropped == 1 or self.dropped % 1000 == 0:
|
||||
logger.warning(
|
||||
"Dropped tool-call record (%s); %d dropped so far", reason, self.dropped
|
||||
)
|
||||
|
||||
async def _run(self, queue: asyncio.Queue) -> None:
|
||||
while True:
|
||||
batch = [await queue.get()]
|
||||
while len(batch) < self.batch_size:
|
||||
try:
|
||||
batch.append(queue.get_nowait())
|
||||
except asyncio.QueueEmpty:
|
||||
break
|
||||
try:
|
||||
await self._deliver(batch)
|
||||
finally:
|
||||
for _ in batch:
|
||||
queue.task_done()
|
||||
|
||||
async def _deliver(self, batch: list[ToolCallRecord]) -> None:
|
||||
for sink in self.sinks:
|
||||
try:
|
||||
await asyncio.wait_for(sink.write(batch), self.sink_timeout)
|
||||
except Exception:
|
||||
self.sink_errors += 1
|
||||
logger.warning(
|
||||
"Sink %s failed to write %d records",
|
||||
type(sink).__name__,
|
||||
len(batch),
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
async def flush(self) -> None:
|
||||
"""Wait until every queued record has been handed to the sinks."""
|
||||
if self._queue is not None and self._loop is asyncio.get_running_loop():
|
||||
await self._queue.join()
|
||||
|
||||
async def aclose(self) -> None:
|
||||
"""Flush, stop the background task and close the sinks."""
|
||||
await self.flush()
|
||||
if self._task is not None:
|
||||
self._task.cancel()
|
||||
with contextlib.suppress(asyncio.CancelledError, RuntimeError):
|
||||
await self._task
|
||||
self._task = None
|
||||
self._queue = None
|
||||
for sink in self.sinks:
|
||||
close = getattr(sink, "aclose", None)
|
||||
if close is not None:
|
||||
try:
|
||||
await close()
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"Sink %s failed to close", type(sink).__name__, exc_info=True
|
||||
)
|
||||
@@ -0,0 +1,463 @@
|
||||
"""FastMCP middleware that records every tool call."""
|
||||
|
||||
import asyncio
|
||||
import inspect
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
import uuid
|
||||
from collections import deque
|
||||
from collections.abc import Awaitable, Callable, Iterable, Sequence
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from fastmcp import FastMCP
|
||||
from fastmcp.server.middleware import Middleware, MiddlewareContext
|
||||
|
||||
from .classify import DEFAULT_RESULT_CLASSIFIER, ResultClassifier
|
||||
from .correlation import (
|
||||
CallLink,
|
||||
Candidate,
|
||||
ResolverContext,
|
||||
as_utc,
|
||||
select_calls,
|
||||
)
|
||||
from .dispatch import RecordDispatcher
|
||||
from .record import ToolCallRecord
|
||||
from .redaction import Redactor
|
||||
from .sinks import JsonLinesSink, Sink
|
||||
from .storage import DatabaseSink
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MODES = ("off", "meta", "full")
|
||||
MODE_ENV_VAR = "FEEDBACK_INSTRUMENTATION_MODE"
|
||||
|
||||
IdentityResolver = Callable[
|
||||
[MiddlewareContext], dict[str, Any] | Awaitable[dict[str, Any]] | None
|
||||
]
|
||||
Enricher = Callable[
|
||||
[str, Any, Any, MiddlewareContext],
|
||||
dict[str, Any] | Awaitable[dict[str, Any]] | None,
|
||||
]
|
||||
|
||||
# Keys an identity resolver can return that get their own columns.
|
||||
_IDENTITY_COLUMNS = ("user_sub", "caller_kind", "client_id")
|
||||
|
||||
|
||||
def _attr(obj: Any, name: str) -> Any:
|
||||
try:
|
||||
value = getattr(obj, name, None)
|
||||
except Exception:
|
||||
return None
|
||||
return str(value) if value is not None else None
|
||||
|
||||
|
||||
# stdio serves exactly one client per process, so the process is the session.
|
||||
_STDIO_SESSION = f"stdio:{os.getpid()}:{uuid.uuid4().hex[:8]}"
|
||||
|
||||
|
||||
def session_key(ctx: Any) -> str | None:
|
||||
"""A per-client key that stays the same across that client's calls.
|
||||
|
||||
Clients on MCP protocol 2026-07-28 (Claude Code, FastMCP 4's Client) do
|
||||
not open sessions, and FastMCP then generates a new ``ctx.session_id`` for
|
||||
every request. So stable signals come first: the HTTP ``mcp-session-id``
|
||||
header (older protocol clients), then one key per process for stdio, then
|
||||
``ctx.session_id``. With a stateless HTTP client there is no session, and
|
||||
correlation relies on the same-user rule.
|
||||
"""
|
||||
if ctx is None:
|
||||
return None
|
||||
try:
|
||||
from fastmcp.server.dependencies import get_http_headers
|
||||
|
||||
header = (get_http_headers(include_all=True) or {}).get("mcp-session-id")
|
||||
if header:
|
||||
return header
|
||||
except Exception:
|
||||
pass
|
||||
if _attr(ctx, "transport") == "stdio":
|
||||
return _STDIO_SESSION
|
||||
return _attr(ctx, "session_id")
|
||||
|
||||
|
||||
def _result_payload(result: Any) -> Any:
|
||||
structured = getattr(result, "structured_content", None)
|
||||
if structured is not None:
|
||||
return structured
|
||||
content = getattr(result, "content", None)
|
||||
if content is not None:
|
||||
return [
|
||||
getattr(item, "text", None) or {"type": getattr(item, "type", None)}
|
||||
for item in content
|
||||
]
|
||||
return result
|
||||
|
||||
|
||||
def _json_size(value: Any) -> int | None:
|
||||
try:
|
||||
return len(json.dumps(value, default=str).encode())
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
class InstrumentationMiddleware(Middleware):
|
||||
"""Record every tool call without ever slowing or breaking it.
|
||||
|
||||
Modes:
|
||||
``off``: pass calls straight through.
|
||||
``meta`` (default): tool, timing, outcome, identity and payload sizes.
|
||||
``full``: also the arguments and result, redacted.
|
||||
|
||||
The mode defaults to the ``FEEDBACK_INSTRUMENTATION_MODE`` environment
|
||||
variable, then ``meta``. Records go to the sinks from a background task
|
||||
through a bounded queue; when it fills, records are dropped, not waited on.
|
||||
|
||||
Args:
|
||||
sinks: where records go. Defaults to JSON lines on stderr.
|
||||
mode: see above.
|
||||
redactor: secret redaction for args, results, error text and extras.
|
||||
meta_only_tools: tools that are never recorded above ``meta``.
|
||||
identity_resolver: ``(context) -> dict`` returning ``user_sub``,
|
||||
``caller_kind`` and ``client_id`` (stored as columns) plus any
|
||||
other keys (stored in ``identity``). May be async.
|
||||
enricher: ``(tool, args, result, context) -> dict`` of extra fields,
|
||||
stored in ``extra``. A ``server_version`` key fills that column.
|
||||
May be async.
|
||||
server_version: default for the ``server_version`` column.
|
||||
hook_timeout: seconds an async hook may take before it is skipped.
|
||||
recent_calls: how many recent records to keep in memory for
|
||||
correlation, including ones still waiting to reach the sinks.
|
||||
correlation_timeout: seconds a database lookup during correlation may
|
||||
take before it is skipped.
|
||||
result_classifier: ``(tool, result) -> str | None`` run on results
|
||||
of calls that did not raise. A string marks the call as a soft
|
||||
error (``outcome="soft_error"``, ``ok=False``) and becomes the
|
||||
error message. Defaults to ``DEFAULT_RESULT_CLASSIFIER``; None
|
||||
turns it off. If it raises, the call is recorded as ok.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
sinks: Sequence[Sink] | None = None,
|
||||
*,
|
||||
mode: str | None = None,
|
||||
redactor: Redactor | None = None,
|
||||
meta_only_tools: Iterable[str] = (),
|
||||
identity_resolver: IdentityResolver | None = None,
|
||||
enricher: Enricher | None = None,
|
||||
server_version: str | None = None,
|
||||
hook_timeout: float = 0.25,
|
||||
max_queue: int = 1000,
|
||||
batch_size: int = 100,
|
||||
sink_timeout: float = 10.0,
|
||||
max_error_chars: int = 500,
|
||||
recent_calls: int = 2000,
|
||||
correlation_timeout: float = 1.0,
|
||||
result_classifier: ResultClassifier | None = DEFAULT_RESULT_CLASSIFIER,
|
||||
):
|
||||
mode = (mode or os.getenv(MODE_ENV_VAR) or "meta").lower()
|
||||
if mode not in MODES:
|
||||
raise ValueError(f"mode must be one of {MODES}, got {mode!r}")
|
||||
self.mode = mode
|
||||
self.redactor = redactor or Redactor()
|
||||
self.meta_only_tools = frozenset(meta_only_tools)
|
||||
self.identity_resolver = identity_resolver
|
||||
self.enricher = enricher
|
||||
self.server_version = server_version
|
||||
self.hook_timeout = hook_timeout
|
||||
self.max_error_chars = max_error_chars
|
||||
self.correlation_timeout = correlation_timeout
|
||||
self.result_classifier = result_classifier
|
||||
self.recent: deque[ToolCallRecord] = deque(maxlen=recent_calls)
|
||||
self.dispatcher = RecordDispatcher(
|
||||
sinks if sinks is not None else [JsonLinesSink()],
|
||||
max_queue=max_queue,
|
||||
batch_size=batch_size,
|
||||
sink_timeout=sink_timeout,
|
||||
)
|
||||
|
||||
async def on_call_tool(self, context: MiddlewareContext, call_next):
|
||||
if self.mode == "off":
|
||||
return await call_next(context)
|
||||
|
||||
started_at = datetime.now(UTC)
|
||||
start = time.perf_counter()
|
||||
result = None
|
||||
error: BaseException | None = None
|
||||
try:
|
||||
result = await call_next(context)
|
||||
return result
|
||||
except BaseException as exc:
|
||||
error = exc
|
||||
raise
|
||||
finally:
|
||||
duration_ms = (time.perf_counter() - start) * 1000
|
||||
try:
|
||||
record = await self._build_record(
|
||||
context, started_at, duration_ms, result, error
|
||||
)
|
||||
self.recent.append(record)
|
||||
self.dispatcher.submit(record)
|
||||
except Exception:
|
||||
logger.warning("Failed to record tool call", exc_info=True)
|
||||
|
||||
async def _call_hook(self, hook: Callable | None, *args: Any) -> dict[str, Any]:
|
||||
if hook is None:
|
||||
return {}
|
||||
try:
|
||||
out = hook(*args)
|
||||
if inspect.isawaitable(out):
|
||||
out = await asyncio.wait_for(out, self.hook_timeout)
|
||||
return dict(out) if out else {}
|
||||
except Exception:
|
||||
logger.warning("Instrumentation hook %r failed", hook, exc_info=True)
|
||||
return {}
|
||||
|
||||
def _classify(self, tool: str, result: Any, payload: Any) -> str | None:
|
||||
if getattr(result, "is_error", False) is True:
|
||||
# FastMCP 3.4+ keeps isError on the ToolResult, not in the payload.
|
||||
if isinstance(payload, dict):
|
||||
payload = {**payload, "isError": True}
|
||||
else:
|
||||
payload = {"isError": True, "content": payload}
|
||||
try:
|
||||
message = self.result_classifier(tool, payload)
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"Result classifier %r failed", self.result_classifier, exc_info=True
|
||||
)
|
||||
return None
|
||||
return None if message is None else str(message)
|
||||
|
||||
async def _build_record(
|
||||
self,
|
||||
context: MiddlewareContext,
|
||||
started_at: datetime,
|
||||
duration_ms: float,
|
||||
result: Any,
|
||||
error: BaseException | None,
|
||||
) -> ToolCallRecord:
|
||||
message = context.message
|
||||
tool = getattr(message, "name", None) or "unknown"
|
||||
args = getattr(message, "arguments", None) or {}
|
||||
ctx = context.fastmcp_context
|
||||
payload = _result_payload(result) if result is not None else None
|
||||
|
||||
identity = await self._call_hook(self.identity_resolver, context)
|
||||
extra = await self._call_hook(self.enricher, tool, args, payload, context)
|
||||
identity = self.redactor.redact(identity)
|
||||
extra = self.redactor.redact(extra)
|
||||
columns = {k: identity.pop(k) for k in _IDENTITY_COLUMNS if k in identity}
|
||||
server_version = extra.pop("server_version", None) or self.server_version
|
||||
|
||||
mode = "meta" if tool in self.meta_only_tools else self.mode
|
||||
full = mode == "full"
|
||||
|
||||
outcome = "ok"
|
||||
error_type = error_message = None
|
||||
if error is not None:
|
||||
# FastMCP wraps tool exceptions in ToolError; report the original.
|
||||
cause = error.__cause__ or error
|
||||
outcome = "error"
|
||||
error_type = type(cause).__name__
|
||||
error_message = self.redactor.redact_text(str(cause))[
|
||||
: self.max_error_chars
|
||||
]
|
||||
elif self.result_classifier is not None and payload is not None:
|
||||
soft = self._classify(tool, result, payload)
|
||||
if soft is not None:
|
||||
outcome = "soft_error"
|
||||
error_type = "SoftError"
|
||||
error_message = (
|
||||
self.redactor.redact_text(soft)[: self.max_error_chars] or None
|
||||
)
|
||||
|
||||
return ToolCallRecord(
|
||||
tool=tool,
|
||||
started_at=started_at,
|
||||
duration_ms=duration_ms,
|
||||
ok=outcome == "ok",
|
||||
outcome=outcome,
|
||||
mode=mode,
|
||||
error_type=error_type,
|
||||
error_message=error_message,
|
||||
session_id=session_key(ctx),
|
||||
request_id=_attr(ctx, "request_id"),
|
||||
client_id=columns.get("client_id") or _attr(ctx, "client_id"),
|
||||
user_sub=columns.get("user_sub"),
|
||||
caller_kind=columns.get("caller_kind"),
|
||||
identity=identity,
|
||||
server_version=server_version,
|
||||
args_size=_json_size(args),
|
||||
result_size=_json_size(payload) if payload is not None else None,
|
||||
args=self.redactor.redact(args) if full else None,
|
||||
result=self.redactor.redact(payload)
|
||||
if full and payload is not None
|
||||
else None,
|
||||
extra=extra,
|
||||
)
|
||||
|
||||
@property
|
||||
def database_sink(self) -> DatabaseSink | None:
|
||||
"""The first DatabaseSink among the sinks, if any."""
|
||||
return next(
|
||||
(s for s in self.dispatcher.sinks if isinstance(s, DatabaseSink)), None
|
||||
)
|
||||
|
||||
async def capture_recent_calls(
|
||||
self,
|
||||
ctx: Any = None,
|
||||
n: int = 20,
|
||||
*,
|
||||
user_sub: str | None = None,
|
||||
window: timedelta = timedelta(minutes=15),
|
||||
) -> list[CallLink]:
|
||||
"""The calls that led up to now, for attaching to a feedback item.
|
||||
|
||||
Call from inside a tool. Returns up to ``n`` calls in chronological
|
||||
order: first from the current MCP session, then, if that yields fewer
|
||||
than ``n``, the same user's calls within ``window``. Each link says
|
||||
which rule matched. Never raises; returns [] on any failure.
|
||||
|
||||
Args:
|
||||
ctx: the FastMCP Context. Looked up automatically when omitted.
|
||||
n: maximum number of calls.
|
||||
user_sub: the caller's user id. When omitted, the identity
|
||||
resolver is asked, given an object whose ``fastmcp_context``
|
||||
is ``ctx``.
|
||||
window: how far back the same-user fallback reaches.
|
||||
"""
|
||||
try:
|
||||
if ctx is None:
|
||||
ctx = _current_context()
|
||||
session_id = session_key(ctx)
|
||||
if user_sub is None and self.identity_resolver is not None:
|
||||
identity = await self._call_hook(
|
||||
self.identity_resolver, ResolverContext(ctx)
|
||||
)
|
||||
user_sub = identity.get("user_sub")
|
||||
since = datetime.now(UTC) - window
|
||||
|
||||
candidates = [
|
||||
Candidate(r.id, r.tool, r.started_at, r.ok, r.session_id, r.user_sub)
|
||||
for r in self.recent
|
||||
]
|
||||
links = select_calls(
|
||||
candidates,
|
||||
session_id=session_id,
|
||||
user_sub=user_sub,
|
||||
since=since,
|
||||
limit=n,
|
||||
)
|
||||
sink = self.database_sink
|
||||
if sink is not None and len(links) < n:
|
||||
# Other workers, or calls from before a restart.
|
||||
rows = await asyncio.wait_for(
|
||||
sink.recent_calls(
|
||||
session_id=session_id, user_sub=user_sub, since=since, limit=n
|
||||
),
|
||||
self.correlation_timeout,
|
||||
)
|
||||
candidates += [
|
||||
Candidate(
|
||||
r["id"],
|
||||
r["tool"],
|
||||
as_utc(r["started_at"]),
|
||||
r["ok"],
|
||||
r["session_id"],
|
||||
r["user_sub"],
|
||||
)
|
||||
for r in rows
|
||||
]
|
||||
links = select_calls(
|
||||
candidates,
|
||||
session_id=session_id,
|
||||
user_sub=user_sub,
|
||||
since=since,
|
||||
limit=n,
|
||||
)
|
||||
return links
|
||||
except Exception:
|
||||
logger.warning("Failed to capture recent calls", exc_info=True)
|
||||
return []
|
||||
|
||||
async def link_feedback(self, feedback_ref: str, links: Sequence[CallLink]) -> bool:
|
||||
"""Store ``links`` against a feedback id of any format (``bug-XXXX``, 42).
|
||||
|
||||
Needs a DatabaseSink. Replaces earlier links for the same id. Returns
|
||||
False, without raising, if they could not be stored.
|
||||
"""
|
||||
sink = self.database_sink
|
||||
if sink is None:
|
||||
logger.warning("link_feedback needs a DatabaseSink; links not stored")
|
||||
return False
|
||||
linked_at = datetime.now(UTC)
|
||||
try:
|
||||
await sink.replace_links(
|
||||
str(feedback_ref),
|
||||
[
|
||||
{
|
||||
"feedback_ref": str(feedback_ref),
|
||||
"call_id": link.call_id,
|
||||
"position": link.position,
|
||||
"rule": link.rule,
|
||||
"linked_at": linked_at,
|
||||
}
|
||||
for link in links
|
||||
],
|
||||
)
|
||||
return True
|
||||
except Exception:
|
||||
logger.warning("Failed to link feedback %s", feedback_ref, exc_info=True)
|
||||
return False
|
||||
|
||||
async def feedback_context(self, feedback_ref: str) -> list[dict[str, Any]]:
|
||||
"""The calls linked to a feedback id, in order, with their details.
|
||||
|
||||
Calls not yet written to the database are filled in from memory.
|
||||
"""
|
||||
sink = self.database_sink
|
||||
if sink is None:
|
||||
return []
|
||||
rows = await sink.linked_calls(str(feedback_ref))
|
||||
in_memory = {r.id: r for r in self.recent}
|
||||
for row in rows:
|
||||
if row["tool"] is None and row["call_id"] in in_memory:
|
||||
record = in_memory[row["call_id"]]
|
||||
for key in row:
|
||||
if key not in ("call_id", "position", "rule"):
|
||||
row[key] = getattr(record, key, None)
|
||||
if row["started_at"] is not None:
|
||||
row["started_at"] = as_utc(row["started_at"])
|
||||
return rows
|
||||
|
||||
async def flush(self) -> None:
|
||||
"""Wait until queued records have reached the sinks."""
|
||||
await self.dispatcher.flush()
|
||||
|
||||
async def aclose(self) -> None:
|
||||
"""Flush and close the sinks. Call on server shutdown."""
|
||||
await self.dispatcher.aclose()
|
||||
|
||||
|
||||
def _current_context() -> Any:
|
||||
try:
|
||||
from fastmcp.server.dependencies import get_context
|
||||
|
||||
return get_context()
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def instrument(app: FastMCP, sinks: Sequence[Sink] | None = None, **kwargs: Any):
|
||||
"""Add an InstrumentationMiddleware to ``app`` and return it.
|
||||
|
||||
>>> mw = instrument(app) # JSON lines to stderr
|
||||
>>> mw = instrument(app, [DatabaseSink(engine)], mode="full")
|
||||
"""
|
||||
middleware = InstrumentationMiddleware(sinks, **kwargs)
|
||||
app.add_middleware(middleware)
|
||||
return middleware
|
||||
@@ -0,0 +1,55 @@
|
||||
"""The per-call record produced by InstrumentationMiddleware."""
|
||||
|
||||
import uuid
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolCallRecord:
|
||||
"""One MCP tool call, as seen by the instrumentation middleware.
|
||||
|
||||
``args`` and ``result`` are only populated in ``full`` mode, and are
|
||||
already redacted when they are. Sizes are measured before redaction.
|
||||
|
||||
``outcome`` is ``ok``, ``error`` (the tool raised) or ``soft_error`` (it
|
||||
returned a result the classifier flagged). ``ok`` is ``outcome == "ok"``;
|
||||
when ``outcome`` is not given it is derived from ``ok``.
|
||||
"""
|
||||
|
||||
tool: str
|
||||
started_at: datetime
|
||||
duration_ms: float
|
||||
ok: bool
|
||||
mode: str
|
||||
outcome: str | None = None
|
||||
id: str = field(default_factory=lambda: str(uuid.uuid4()))
|
||||
error_type: str | None = None
|
||||
error_message: str | None = None
|
||||
session_id: str | None = None
|
||||
request_id: str | None = None
|
||||
client_id: str | None = None
|
||||
user_sub: str | None = None
|
||||
caller_kind: str | None = None
|
||||
identity: dict[str, Any] = field(default_factory=dict)
|
||||
server_version: str | None = None
|
||||
args_size: int | None = None
|
||||
result_size: int | None = None
|
||||
args: Any = None
|
||||
result: Any = None
|
||||
extra: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.outcome is None:
|
||||
self.outcome = "ok" if self.ok else "error"
|
||||
|
||||
def to_row(self) -> dict[str, Any]:
|
||||
"""Column values for the tool_calls table."""
|
||||
return asdict(self)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
"""JSON-friendly form, with the timestamp as ISO 8601."""
|
||||
data = asdict(self)
|
||||
data["started_at"] = self.started_at.isoformat()
|
||||
return data
|
||||
@@ -0,0 +1,125 @@
|
||||
"""Secret redaction for recorded tool arguments, results and error text."""
|
||||
|
||||
import re
|
||||
from collections.abc import Iterable
|
||||
from typing import Any
|
||||
|
||||
REDACTED = "[REDACTED]"
|
||||
TRUNCATED = "[TRUNCATED]"
|
||||
|
||||
# Matched after lowercasing and turning "-" into "_", so "Authorization",
|
||||
# "X-Api-Key" style variants are covered too.
|
||||
DEFAULT_SECRET_KEYS = frozenset(
|
||||
{
|
||||
"token",
|
||||
"access_token",
|
||||
"refresh_token",
|
||||
"id_token",
|
||||
"auth_token",
|
||||
"session_token",
|
||||
"csrf_token",
|
||||
"bearer",
|
||||
"authorization",
|
||||
"proxy_authorization",
|
||||
"cookie",
|
||||
"set_cookie",
|
||||
"password",
|
||||
"passwd",
|
||||
"passphrase",
|
||||
"secret",
|
||||
"client_secret",
|
||||
"api_key",
|
||||
"apikey",
|
||||
"x_api_key",
|
||||
"private_key",
|
||||
}
|
||||
)
|
||||
|
||||
# github_token, db_password, stripe_api_key, ...
|
||||
DEFAULT_SECRET_SUFFIXES = (
|
||||
"_token",
|
||||
"_secret",
|
||||
"_password",
|
||||
"_api_key",
|
||||
"_apikey",
|
||||
"_private_key",
|
||||
)
|
||||
|
||||
# Values that are secrets wherever they appear, including inside free text.
|
||||
DEFAULT_SECRET_PATTERNS = (
|
||||
r"\bbmcp_[A-Za-z0-9_-]{8,}", # blender-mcp personal access tokens
|
||||
r"\beyJ[A-Za-z0-9_-]{5,}\.[A-Za-z0-9_-]{5,}\.[A-Za-z0-9_-]{5,}", # JWT
|
||||
r"(?i)\bbearer\s+[A-Za-z0-9._~+/=-]{8,}",
|
||||
r"\bsk-[A-Za-z0-9_-]{16,}", # OpenAI/Anthropic-style API keys
|
||||
r"\bgh[pousr]_[A-Za-z0-9]{20,}", # GitHub tokens
|
||||
r"\bgithub_pat_[A-Za-z0-9_]{20,}",
|
||||
r"\bxox[abposr]-[A-Za-z0-9-]{10,}", # Slack
|
||||
r"\bAKIA[0-9A-Z]{16}\b", # AWS access key id
|
||||
r"\bpypi-[A-Za-z0-9_-]{16,}",
|
||||
)
|
||||
|
||||
# "password=hunter2", "api_key: abc" in error messages and logs. The key is
|
||||
# kept so the text still says what was removed.
|
||||
_KEY_VALUE_TEXT = re.compile(
|
||||
r"(?i)\b(password|passwd|secret|token|api[_-]?key|authorization)(\s*[=:]\s*)[^\s,;&'\"]+"
|
||||
)
|
||||
|
||||
|
||||
def _normalize_key(key: Any) -> str:
|
||||
return str(key).lower().replace("-", "_")
|
||||
|
||||
|
||||
class Redactor:
|
||||
"""Replace secrets in nested JSON-like data with a placeholder.
|
||||
|
||||
Dict values are redacted when their key looks like a secret, at any depth.
|
||||
String values anywhere are scanned for secret-shaped substrings. The input
|
||||
is never modified; a redacted copy is returned.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
keys: Iterable[str] = DEFAULT_SECRET_KEYS,
|
||||
patterns: Iterable[str] = DEFAULT_SECRET_PATTERNS,
|
||||
*,
|
||||
extra_keys: Iterable[str] = (),
|
||||
extra_patterns: Iterable[str] = (),
|
||||
suffixes: Iterable[str] = DEFAULT_SECRET_SUFFIXES,
|
||||
replacement: str = REDACTED,
|
||||
max_depth: int = 32,
|
||||
):
|
||||
self.keys = frozenset(_normalize_key(k) for k in (*keys, *extra_keys))
|
||||
self.suffixes = tuple(_normalize_key(s) for s in suffixes)
|
||||
self.patterns = tuple(re.compile(p) for p in (*patterns, *extra_patterns))
|
||||
self.replacement = replacement
|
||||
self.max_depth = max_depth
|
||||
|
||||
def is_secret_key(self, key: Any) -> bool:
|
||||
normalized = _normalize_key(key)
|
||||
return normalized in self.keys or normalized.endswith(self.suffixes)
|
||||
|
||||
def redact_text(self, text: str) -> str:
|
||||
for pattern in self.patterns:
|
||||
text = pattern.sub(self.replacement, text)
|
||||
return _KEY_VALUE_TEXT.sub(
|
||||
lambda m: f"{m.group(1)}{m.group(2)}{self.replacement}", text
|
||||
)
|
||||
|
||||
def redact(self, value: Any) -> Any:
|
||||
return self._redact(value, 0)
|
||||
|
||||
def _redact(self, value: Any, depth: int) -> Any:
|
||||
if depth > self.max_depth:
|
||||
return TRUNCATED
|
||||
if isinstance(value, str):
|
||||
return self.redact_text(value)
|
||||
if isinstance(value, dict):
|
||||
return {
|
||||
k: self.replacement
|
||||
if self.is_secret_key(k) and v is not None
|
||||
else self._redact(v, depth + 1)
|
||||
for k, v in value.items()
|
||||
}
|
||||
if isinstance(value, list | tuple | set | frozenset):
|
||||
return [self._redact(v, depth + 1) for v in value]
|
||||
return value
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Destinations for tool-call records."""
|
||||
|
||||
import inspect
|
||||
import json
|
||||
import sys
|
||||
from collections.abc import Callable, Sequence
|
||||
from typing import IO, Any, Protocol, runtime_checkable
|
||||
|
||||
from .record import ToolCallRecord
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class Sink(Protocol):
|
||||
"""Anything with ``async write(records)``. ``async aclose()`` is optional."""
|
||||
|
||||
async def write(self, records: Sequence[ToolCallRecord]) -> None: ...
|
||||
|
||||
|
||||
class JsonLinesSink:
|
||||
"""One JSON object per record, to stderr by default."""
|
||||
|
||||
def __init__(self, stream: IO[str] | None = None):
|
||||
self._stream = stream
|
||||
|
||||
async def write(self, records: Sequence[ToolCallRecord]) -> None:
|
||||
# Resolved per write so a stderr swapped by the host or a test is honored.
|
||||
stream = self._stream or sys.stderr
|
||||
for record in records:
|
||||
stream.write(json.dumps(record.to_dict(), default=str) + "\n")
|
||||
stream.flush()
|
||||
|
||||
|
||||
class CallbackSink:
|
||||
"""Pass each batch to a function, sync or async.
|
||||
|
||||
A sync callback runs on the event loop, so keep it quick.
|
||||
"""
|
||||
|
||||
def __init__(self, callback: Callable[[Sequence[ToolCallRecord]], Any]):
|
||||
self.callback = callback
|
||||
|
||||
async def write(self, records: Sequence[ToolCallRecord]) -> None:
|
||||
result = self.callback(records)
|
||||
if inspect.isawaitable(result):
|
||||
await result
|
||||
|
||||
|
||||
class MemorySink:
|
||||
"""Keep records in a list. Meant for tests."""
|
||||
|
||||
def __init__(self):
|
||||
self.records: list[ToolCallRecord] = []
|
||||
|
||||
async def write(self, records: Sequence[ToolCallRecord]) -> None:
|
||||
self.records.extend(records)
|
||||
@@ -0,0 +1,217 @@
|
||||
"""Async SQLAlchemy storage for tool-call records and feedback links.
|
||||
|
||||
Hosts that manage their own schema include ``build_metadata(prefix)`` in their
|
||||
Alembic ``env.py`` (``target_metadata``) and pass their existing AsyncEngine to
|
||||
``DatabaseSink``. Nothing is created unless ``create_tables=True``.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Sequence
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
Boolean,
|
||||
Column,
|
||||
DateTime,
|
||||
Float,
|
||||
Integer,
|
||||
MetaData,
|
||||
String,
|
||||
Table,
|
||||
Text,
|
||||
and_,
|
||||
delete,
|
||||
or_,
|
||||
select,
|
||||
)
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
|
||||
|
||||
from .record import ToolCallRecord
|
||||
|
||||
DEFAULT_TABLE_PREFIX = "ffb_"
|
||||
|
||||
# none_as_null: a missing payload is SQL NULL, so "WHERE args IS NULL" works.
|
||||
_JSON = JSON(none_as_null=True).with_variant(JSONB(none_as_null=True), "postgresql")
|
||||
|
||||
|
||||
def build_metadata(
|
||||
prefix: str = DEFAULT_TABLE_PREFIX, metadata: MetaData | None = None
|
||||
) -> MetaData:
|
||||
"""Return MetaData holding ``{prefix}tool_calls`` and ``{prefix}feedback_call_links``.
|
||||
|
||||
Pass an existing MetaData to add the tables to it instead.
|
||||
"""
|
||||
metadata = metadata if metadata is not None else MetaData()
|
||||
calls = f"{prefix}tool_calls"
|
||||
links = f"{prefix}feedback_call_links"
|
||||
if calls not in metadata.tables:
|
||||
Table(
|
||||
calls,
|
||||
metadata,
|
||||
Column("id", String(36), primary_key=True),
|
||||
Column("started_at", DateTime(timezone=True), nullable=False, index=True),
|
||||
Column("duration_ms", Float, nullable=False),
|
||||
Column("tool", String(255), nullable=False, index=True),
|
||||
Column("mode", String(8), nullable=False),
|
||||
Column("ok", Boolean, nullable=False, index=True),
|
||||
# ok, soft_error or error. NULL on rows from before 2026.09.27.4.
|
||||
Column("outcome", String(16), index=True),
|
||||
Column("error_type", String(255)),
|
||||
Column("error_message", Text),
|
||||
Column("session_id", String(255), index=True),
|
||||
Column("request_id", String(255)),
|
||||
Column("client_id", String(255)),
|
||||
Column("user_sub", String(255), index=True),
|
||||
Column("caller_kind", String(64)),
|
||||
Column("identity", _JSON),
|
||||
Column("server_version", String(64), index=True),
|
||||
Column("args_size", Integer),
|
||||
Column("result_size", Integer),
|
||||
Column("args", _JSON),
|
||||
Column("result", _JSON),
|
||||
Column("extra", _JSON),
|
||||
)
|
||||
if links not in metadata.tables:
|
||||
# No foreign key to tool_calls: a link can be written before its call
|
||||
# record has left the queue, and hosts may keep calls elsewhere.
|
||||
Table(
|
||||
links,
|
||||
metadata,
|
||||
Column("feedback_ref", String(255), primary_key=True),
|
||||
Column("call_id", String(36), primary_key=True, index=True),
|
||||
Column("position", Integer, nullable=False),
|
||||
Column("rule", String(32), nullable=False),
|
||||
Column("linked_at", DateTime(timezone=True), nullable=False),
|
||||
)
|
||||
return metadata
|
||||
|
||||
|
||||
class DatabaseSink:
|
||||
"""Write records to ``{prefix}tool_calls`` through an async engine.
|
||||
|
||||
Args:
|
||||
engine: an AsyncEngine the host already owns, or a URL such as
|
||||
``sqlite+aiosqlite:///calls.db`` or ``postgresql+asyncpg://...``.
|
||||
An engine built from a URL is disposed on ``aclose()``; a passed-in
|
||||
engine is left alone.
|
||||
prefix: table name prefix, so the tables cannot collide with the host's.
|
||||
create_tables: create the tables on first use. Off by default; hosts
|
||||
with Alembic should generate a migration from ``build_metadata``.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
engine: AsyncEngine | str,
|
||||
*,
|
||||
prefix: str = DEFAULT_TABLE_PREFIX,
|
||||
create_tables: bool = False,
|
||||
):
|
||||
self._owns_engine = isinstance(engine, str)
|
||||
self.engine = create_async_engine(engine) if isinstance(engine, str) else engine
|
||||
self.metadata = build_metadata(prefix)
|
||||
self.table = self.metadata.tables[f"{prefix}tool_calls"]
|
||||
self.links = self.metadata.tables[f"{prefix}feedback_call_links"]
|
||||
self.create_tables = create_tables
|
||||
self._tables_ready = False
|
||||
# The background writer and link_feedback can both reach _ready() at
|
||||
# once; without the lock both run create_all and one fails.
|
||||
self._tables_lock = asyncio.Lock()
|
||||
|
||||
async def ensure_tables(self) -> None:
|
||||
"""Create the tables if missing. Only called when create_tables=True."""
|
||||
async with self.engine.begin() as conn:
|
||||
await conn.run_sync(self.metadata.create_all)
|
||||
self._tables_ready = True
|
||||
|
||||
async def _ready(self) -> None:
|
||||
if self.create_tables and not self._tables_ready:
|
||||
async with self._tables_lock:
|
||||
if not self._tables_ready:
|
||||
await self.ensure_tables()
|
||||
|
||||
async def write(self, records: Sequence[ToolCallRecord]) -> None:
|
||||
if not records:
|
||||
return
|
||||
await self._ready()
|
||||
async with self.engine.begin() as conn:
|
||||
await conn.execute(self.table.insert(), [r.to_row() for r in records])
|
||||
|
||||
async def recent_calls(
|
||||
self,
|
||||
*,
|
||||
session_id: str | None,
|
||||
user_sub: str | None,
|
||||
since: datetime,
|
||||
limit: int,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Calls from this session, or from this user since ``since``, newest first."""
|
||||
t = self.table
|
||||
conditions = []
|
||||
if session_id:
|
||||
conditions.append(t.c.session_id == session_id)
|
||||
if user_sub:
|
||||
conditions.append(and_(t.c.user_sub == user_sub, t.c.started_at >= since))
|
||||
if not conditions:
|
||||
return []
|
||||
await self._ready()
|
||||
query = (
|
||||
select(
|
||||
t.c.id, t.c.tool, t.c.started_at, t.c.ok, t.c.session_id, t.c.user_sub
|
||||
)
|
||||
.where(or_(*conditions))
|
||||
.order_by(t.c.started_at.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
async with self.engine.connect() as conn:
|
||||
return [dict(row) for row in (await conn.execute(query)).mappings()]
|
||||
|
||||
async def replace_links(
|
||||
self, feedback_ref: str, links: Sequence[dict[str, Any]]
|
||||
) -> None:
|
||||
"""Set the calls linked to ``feedback_ref``, replacing any earlier set."""
|
||||
await self._ready()
|
||||
async with self.engine.begin() as conn:
|
||||
await conn.execute(
|
||||
delete(self.links).where(self.links.c.feedback_ref == feedback_ref)
|
||||
)
|
||||
if links:
|
||||
await conn.execute(self.links.insert(), list(links))
|
||||
|
||||
async def linked_calls(self, feedback_ref: str) -> list[dict[str, Any]]:
|
||||
"""Links for ``feedback_ref`` in order, joined to their call records.
|
||||
|
||||
Call columns are None for calls not (yet) in this database.
|
||||
"""
|
||||
await self._ready()
|
||||
link, call = self.links, self.table
|
||||
query = (
|
||||
select(
|
||||
link.c.call_id,
|
||||
link.c.position,
|
||||
link.c.rule,
|
||||
call.c.tool,
|
||||
call.c.started_at,
|
||||
call.c.duration_ms,
|
||||
call.c.ok,
|
||||
call.c.outcome,
|
||||
call.c.error_type,
|
||||
call.c.error_message,
|
||||
call.c.session_id,
|
||||
call.c.user_sub,
|
||||
call.c.caller_kind,
|
||||
call.c.server_version,
|
||||
call.c.extra,
|
||||
)
|
||||
.select_from(link.outerjoin(call, link.c.call_id == call.c.id))
|
||||
.where(link.c.feedback_ref == feedback_ref)
|
||||
.order_by(link.c.position)
|
||||
)
|
||||
async with self.engine.connect() as conn:
|
||||
return [dict(row) for row in (await conn.execute(query)).mappings()]
|
||||
|
||||
async def aclose(self) -> None:
|
||||
if self._owns_engine:
|
||||
await self.engine.dispose()
|
||||
@@ -1,15 +1,37 @@
|
||||
"""Submission mixin for feedback collection."""
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from ..insights import record_feedback_submission
|
||||
from ..models import SubmitFeedbackRequest, create_feedback_from_request
|
||||
from .base import BaseFeedbackMixin, logger, mcp_tool
|
||||
from .base import (
|
||||
BaseFeedbackMixin,
|
||||
FeedbackDatabase,
|
||||
FeedbackInsights,
|
||||
logger,
|
||||
mcp_tool,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..instrumentation import InstrumentationMiddleware
|
||||
|
||||
|
||||
class SubmissionMixin(BaseFeedbackMixin):
|
||||
"""Mixin for feedback submission functionality."""
|
||||
"""Mixin for feedback submission functionality.
|
||||
|
||||
Pass an InstrumentationMiddleware to link each submission to the tool
|
||||
calls that led up to it.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
database: FeedbackDatabase,
|
||||
insights: FeedbackInsights | None = None,
|
||||
instrumentation: "InstrumentationMiddleware | None" = None,
|
||||
):
|
||||
super().__init__(database, insights)
|
||||
self.instrumentation = instrumentation
|
||||
|
||||
@mcp_tool(
|
||||
description="Submit feedback about your MCP experience - Use this tool when you encounter issues, have suggestions for improvement, or want to report positive experiences. AI models should proactively use this to help improve future MCP sessions. PRIVACY: Only include information you've already shared via previous MCP calls. Ask human permission before including any new sensitive data."
|
||||
@@ -62,14 +84,23 @@ class SubmissionMixin(BaseFeedbackMixin):
|
||||
source="mcp_tool",
|
||||
)
|
||||
|
||||
linked_calls = 0
|
||||
if self.instrumentation is not None:
|
||||
links = await self.instrumentation.capture_recent_calls()
|
||||
if await self.instrumentation.link_feedback(feedback_id, links):
|
||||
linked_calls = len(links)
|
||||
|
||||
self._record_tool_usage("submit_feedback", start_time, True)
|
||||
logger.info(f"Feedback submitted successfully: {feedback_id}")
|
||||
|
||||
return {
|
||||
response = {
|
||||
"success": True,
|
||||
"feedback_id": feedback_id,
|
||||
"message": "Feedback submitted successfully",
|
||||
}
|
||||
if self.instrumentation is not None:
|
||||
response["linked_calls"] = linked_calls
|
||||
return response
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to submit feedback: {e}")
|
||||
|
||||
@@ -17,6 +17,7 @@ def add_feedback_tools(
|
||||
insights: FeedbackInsights | None = None,
|
||||
prefix: str = "",
|
||||
separator: str = "_",
|
||||
instrumentation=None,
|
||||
) -> None:
|
||||
"""Add all feedback tools to a FastMCP server.
|
||||
|
||||
@@ -29,6 +30,9 @@ def add_feedback_tools(
|
||||
insights: Optional insights instance. Creates default if not provided.
|
||||
prefix: Optional prefix for tool names.
|
||||
separator: Separator between prefix and tool name.
|
||||
instrumentation: an InstrumentationMiddleware (see ``instrument``).
|
||||
When given, each submission is linked to the tool calls that led
|
||||
up to it, and the response reports ``linked_calls``.
|
||||
"""
|
||||
try:
|
||||
# Initialize database
|
||||
@@ -39,7 +43,7 @@ def add_feedback_tools(
|
||||
insights = FeedbackInsights()
|
||||
|
||||
# Create mixin instances
|
||||
submission_mixin = SubmissionMixin(database, insights)
|
||||
submission_mixin = SubmissionMixin(database, insights, instrumentation)
|
||||
retrieval_mixin = RetrievalMixin(database, insights)
|
||||
management_mixin = ManagementMixin(database, insights)
|
||||
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
"""Helpers that paper over FastMCP API differences between 2.x and 4.x."""
|
||||
|
||||
from fastmcp import FastMCP
|
||||
|
||||
|
||||
async def tool_map(server: FastMCP) -> dict:
|
||||
"""Return {name: Tool}. 2.x has get_tools() -> dict, 4.x has list_tools() -> list."""
|
||||
if hasattr(server, "get_tools"):
|
||||
return await server.get_tools()
|
||||
return {tool.name: tool for tool in await server.list_tools()}
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
"""A stdio MCP server used by test_correlation to check session linking end to end."""
|
||||
|
||||
from fastmcp import Context, FastMCP
|
||||
|
||||
from fastmcp_feedback.instrumentation import MemorySink, instrument
|
||||
|
||||
app = FastMCP("stdio-correlation")
|
||||
mw = instrument(app, [MemorySink()])
|
||||
|
||||
|
||||
@app.tool
|
||||
def echo(text: str) -> str:
|
||||
return text
|
||||
|
||||
|
||||
@app.tool
|
||||
async def report(ctx: Context) -> list:
|
||||
return [[link.tool, link.rule] for link in await mw.capture_recent_calls(ctx)]
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
app.run(show_banner=False)
|
||||
@@ -11,11 +11,14 @@ import sys
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
# Add the examples directory to the path for imports
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "examples"))
|
||||
|
||||
from fastmcp import FastMCP
|
||||
from fastmcp_feedback import FeedbackDatabase, FeedbackInsights
|
||||
from tests._compat import tool_map
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -35,7 +38,7 @@ class TestDemoServerIntegration:
|
||||
assert "demonstration" in server.description.lower()
|
||||
|
||||
# Get all tools
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
|
||||
# Verify all expected tools are present
|
||||
expected_tools = {
|
||||
@@ -72,12 +75,12 @@ class TestDemoServerIntegration:
|
||||
from fastmcp_feedback.models import Feedback
|
||||
async with database.session() as session:
|
||||
# Count feedback items
|
||||
result = session.execute("SELECT COUNT(*) FROM feedback")
|
||||
result = session.execute(text("SELECT COUNT(*) FROM feedback"))
|
||||
count = result.scalar()
|
||||
assert count >= 5, f"Expected at least 5 sample items, got {count}"
|
||||
|
||||
# Verify types are present
|
||||
result = session.execute("SELECT DISTINCT type FROM feedback")
|
||||
result = session.execute(text("SELECT DISTINCT type FROM feedback"))
|
||||
types = [row[0] for row in result.fetchall()]
|
||||
expected_types = {"BUG", "FEATURE", "IMPROVEMENT", "QUESTION"}
|
||||
assert expected_types.issubset(set(types)), f"Missing feedback types: {expected_types - set(types)}"
|
||||
@@ -89,21 +92,19 @@ class TestDemoServerIntegration:
|
||||
"""Test actual execution of demo server tools."""
|
||||
from demo_server import create_demo_server, create_temp_database, setup_demo_data
|
||||
|
||||
# Create server with temporary database
|
||||
server = create_demo_server()
|
||||
|
||||
# Override database URL for testing
|
||||
# Seed a temporary database, then point the server at it
|
||||
db_url = create_temp_database()
|
||||
database = FeedbackDatabase(db_url)
|
||||
await setup_demo_data(database)
|
||||
server = create_demo_server(database_url=db_url)
|
||||
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
|
||||
# Test system info tool
|
||||
system_info_tool = tools["get_system_info"]
|
||||
info_result = await system_info_tool.run({})
|
||||
assert info_result.structured_content["success"] is True
|
||||
assert "version" in info_result.structured_content
|
||||
assert info_result.structured_content["status"] == "running"
|
||||
assert "versions" in info_result.structured_content
|
||||
assert "dependencies" in info_result.structured_content
|
||||
|
||||
# Test statistics tool (should work with sample data)
|
||||
@@ -126,7 +127,7 @@ class TestDemoServerIntegration:
|
||||
add_feedback_tools(app, database_url="sqlite:///:memory:")
|
||||
|
||||
# Verify tools were added
|
||||
tools = await app.get_tools()
|
||||
tools = await tool_map(app)
|
||||
expected_tools = {
|
||||
"submit_feedback",
|
||||
"list_feedback",
|
||||
@@ -176,7 +177,7 @@ class TestDemoServerIntegration:
|
||||
from demo_server import create_demo_server
|
||||
|
||||
server = create_demo_server()
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
|
||||
# Test submit_feedback with invalid data
|
||||
submit_tool = tools["submit_feedback"]
|
||||
@@ -199,7 +200,7 @@ class TestDemoServerIntegration:
|
||||
from demo_server import create_demo_server, create_temp_database
|
||||
|
||||
server = create_demo_server()
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
|
||||
# Submit feedback
|
||||
submit_tool = tools["submit_feedback"]
|
||||
@@ -225,7 +226,7 @@ class TestDemoServerIntegration:
|
||||
assert our_feedback is not None, "Submitted feedback not found in list"
|
||||
assert our_feedback["title"] == test_feedback["title"]
|
||||
assert our_feedback["type"] == test_feedback["type"]
|
||||
assert our_feedback["status"] == "OPEN"
|
||||
assert our_feedback["status"] == "open"
|
||||
|
||||
# Update feedback status
|
||||
update_tool = tools["update_feedback_status"]
|
||||
@@ -239,7 +240,7 @@ class TestDemoServerIntegration:
|
||||
list_result = await list_tool.run({})
|
||||
feedback_list = list_result.structured_content["feedback"]
|
||||
updated_feedback = next((f for f in feedback_list if f["id"] == feedback_id), None)
|
||||
assert updated_feedback["status"] == "IN_PROGRESS"
|
||||
assert updated_feedback["status"] == "in_progress"
|
||||
|
||||
# Get statistics (should include our feedback)
|
||||
stats_tool = tools["get_feedback_statistics"]
|
||||
@@ -272,7 +273,7 @@ class TestDemoServerDocumentation:
|
||||
from demo_server import create_demo_server
|
||||
|
||||
server = create_demo_server()
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
|
||||
# According to demo_server.md, should have exactly 7 tools
|
||||
assert len(tools) == 7, f"Expected 7 tools as documented, got {len(tools)}"
|
||||
@@ -296,7 +297,7 @@ class TestDemoServerDocumentation:
|
||||
app = FastMCP("Test App")
|
||||
add_feedback_tools(app, database_url="sqlite:///:memory:")
|
||||
|
||||
tools = await app.get_tools()
|
||||
tools = await tool_map(app)
|
||||
assert len(tools) == 5, f"Expected 5 tools as documented, got {len(tools)}"
|
||||
|
||||
# Verify documented tool names
|
||||
@@ -318,7 +319,7 @@ class TestDemoServerPerformance:
|
||||
from demo_server import create_demo_server
|
||||
|
||||
server = create_demo_server()
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
|
||||
# Submit multiple feedback items concurrently
|
||||
|
||||
@@ -10,6 +10,7 @@ from fastmcp_feedback.feedback import create_feedback_server
|
||||
from fastmcp_feedback.models import SubmitFeedbackRequest, FeedbackStatus
|
||||
from fastmcp_feedback.database import FeedbackDatabase
|
||||
from fastmcp_feedback.insights import FeedbackInsights
|
||||
from tests._compat import tool_map
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@@ -25,7 +26,7 @@ class TestCompleteWorkflow:
|
||||
add_feedback_tools(server, database_url=memory_db_url, insights=insights)
|
||||
|
||||
# Get tools
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
list_tool = tools["list_feedback"]
|
||||
update_tool = tools["update_feedback_status"]
|
||||
@@ -106,7 +107,7 @@ class TestCompleteWorkflow:
|
||||
server = FastMCP("Multi-Type Test Server")
|
||||
add_feedback_tools(server, database_url=memory_db_url)
|
||||
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
list_tool = tools["list_feedback"]
|
||||
|
||||
@@ -179,7 +180,7 @@ class TestCompleteWorkflow:
|
||||
server = FastMCP("Error Test Server")
|
||||
add_feedback_tools(server, database_url=memory_db_url)
|
||||
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
update_tool = tools["update_feedback_status"]
|
||||
delete_tool = tools["delete_feedback"]
|
||||
@@ -233,7 +234,7 @@ class TestCompleteWorkflow:
|
||||
insights = FeedbackInsights(enabled=True)
|
||||
add_feedback_tools(server, database_url=memory_db_url, insights=insights)
|
||||
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
list_tool = tools["list_feedback"]
|
||||
|
||||
@@ -285,7 +286,7 @@ class TestCompleteWorkflow:
|
||||
server = FastMCP("Bulk Test Server")
|
||||
add_feedback_tools(server, database_url=memory_db_url)
|
||||
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
list_tool = tools["list_feedback"]
|
||||
update_tool = tools["update_feedback_status"]
|
||||
@@ -355,7 +356,7 @@ class TestServerCompositionWorkflow:
|
||||
)
|
||||
|
||||
# Get tools from dedicated server
|
||||
tools = await feedback_server.get_tools()
|
||||
tools = await tool_map(feedback_server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
list_tool = tools["list_feedback"]
|
||||
|
||||
@@ -384,11 +385,11 @@ class TestServerCompositionWorkflow:
|
||||
admin_feedback = create_feedback_server("Admin Feedback", database_url="sqlite:///:memory:")
|
||||
|
||||
# Import servers with different prefixes
|
||||
await main_server.import_server(user_feedback, prefix="user")
|
||||
await main_server.import_server(admin_feedback, prefix="admin")
|
||||
main_server.mount(user_feedback, "user")
|
||||
main_server.mount(admin_feedback, "admin")
|
||||
|
||||
# Get tools
|
||||
tools = await main_server.get_tools()
|
||||
tools = await tool_map(main_server)
|
||||
|
||||
user_submit = tools["user_submit_feedback"]
|
||||
admin_submit = tools["admin_submit_feedback"]
|
||||
@@ -439,7 +440,7 @@ class TestServerCompositionWorkflow:
|
||||
add_feedback_tools(server, database_url=memory_db_url, prefix="public")
|
||||
add_feedback_tools(server, database_url=memory_db_url, prefix="internal")
|
||||
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
|
||||
public_submit = tools["public_submit_feedback"]
|
||||
internal_submit = tools["internal_submit_feedback"]
|
||||
@@ -490,7 +491,7 @@ class TestPerformanceWorkflow:
|
||||
insights = FeedbackInsights(enabled=True)
|
||||
add_feedback_tools(server, database_url=memory_db_url, insights=insights)
|
||||
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
list_tool = tools["list_feedback"]
|
||||
stats_tool = tools["get_feedback_statistics"]
|
||||
@@ -537,7 +538,7 @@ class TestPerformanceWorkflow:
|
||||
server = FastMCP("Filter Performance Server")
|
||||
add_feedback_tools(server, database_url=memory_db_url)
|
||||
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
list_tool = tools["list_feedback"]
|
||||
update_tool = tools["update_feedback_status"]
|
||||
@@ -611,7 +612,7 @@ class TestPerformanceWorkflow:
|
||||
server = FastMCP("Concurrent Mixed Server")
|
||||
add_feedback_tools(server, database_url=memory_db_url)
|
||||
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
list_tool = tools["list_feedback"]
|
||||
update_tool = tools["update_feedback_status"]
|
||||
|
||||
@@ -0,0 +1,416 @@
|
||||
"""Tests for linking feedback to the tool calls that preceded it."""
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
import sys
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import fastmcp
|
||||
import pytest
|
||||
from fastmcp import Client, Context, FastMCP
|
||||
from fastmcp.client.transports import PythonStdioTransport
|
||||
from fastmcp.exceptions import ToolError
|
||||
|
||||
from fastmcp_feedback import add_feedback_tools
|
||||
from fastmcp_feedback.instrumentation import (
|
||||
DatabaseSink,
|
||||
MemorySink,
|
||||
instrument,
|
||||
)
|
||||
from fastmcp_feedback.instrumentation import middleware as mw_module
|
||||
from fastmcp_feedback.instrumentation.correlation import (
|
||||
RULE_SESSION,
|
||||
RULE_USER_WINDOW,
|
||||
Candidate,
|
||||
select_calls,
|
||||
)
|
||||
|
||||
NOW = datetime(2026, 9, 27, 12, 0, tzinfo=UTC)
|
||||
|
||||
# FastMCP 4's Client speaks MCP 2026-07-28, which has no sessions, so each
|
||||
# in-memory call gets a new ctx.session_id and there is no header or stdio
|
||||
# signal to fall back on. Calls from one test client link by user instead.
|
||||
FASTMCP_4 = int(fastmcp.__version__.split(".")[0]) >= 4
|
||||
SAME_CLIENT_RULE = RULE_USER_WINDOW if FASTMCP_4 else RULE_SESSION
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def db(tmp_path):
|
||||
"""Make DatabaseSinks whose engines are always disposed, even on failure.
|
||||
|
||||
An undisposed aiosqlite engine keeps a worker thread alive and the test
|
||||
process never exits.
|
||||
"""
|
||||
made = []
|
||||
|
||||
def factory(name="calls", cls=DatabaseSink):
|
||||
sink = cls(f"sqlite+aiosqlite:///{tmp_path}/{name}.db", create_tables=True)
|
||||
made.append(sink)
|
||||
return sink
|
||||
|
||||
yield factory
|
||||
for sink in made:
|
||||
await sink.engine.dispose()
|
||||
|
||||
|
||||
def cand(id, minutes_ago, session="s1", user="u1", tool="t", ok=True):
|
||||
return Candidate(id, tool, NOW - timedelta(minutes=minutes_ago), ok, session, user)
|
||||
|
||||
|
||||
class TestSelectCalls:
|
||||
def test_session_first_then_user_window_chronological(self):
|
||||
links = select_calls(
|
||||
[cand("a", 1), cand("b", 3, session="s2"), cand("c", 2)],
|
||||
session_id="s1",
|
||||
user_sub="u1",
|
||||
since=NOW - timedelta(minutes=15),
|
||||
limit=10,
|
||||
)
|
||||
assert [(link.call_id, link.rule, link.position) for link in links] == [
|
||||
("b", RULE_USER_WINDOW, 0),
|
||||
("c", RULE_SESSION, 1),
|
||||
("a", RULE_SESSION, 2),
|
||||
]
|
||||
|
||||
def test_session_matches_take_priority_over_newer_user_calls(self):
|
||||
links = select_calls(
|
||||
[cand("old", 10), cand("new", 1, session="s2")],
|
||||
session_id="s1",
|
||||
user_sub="u1",
|
||||
since=NOW - timedelta(minutes=15),
|
||||
limit=1,
|
||||
)
|
||||
assert [(link.call_id, link.rule) for link in links] == [("old", RULE_SESSION)]
|
||||
|
||||
def test_user_window_excludes_old_calls_and_other_users(self):
|
||||
links = select_calls(
|
||||
[
|
||||
cand("recent", 5, session="s2"),
|
||||
cand("stale", 30, session="s2"),
|
||||
cand("other", 1, session="s3", user="u2"),
|
||||
],
|
||||
session_id="s9",
|
||||
user_sub="u1",
|
||||
since=NOW - timedelta(minutes=15),
|
||||
limit=10,
|
||||
)
|
||||
assert [link.call_id for link in links] == ["recent"]
|
||||
|
||||
def test_limit_keeps_the_most_recent(self):
|
||||
links = select_calls(
|
||||
[cand(str(i), i) for i in range(10)],
|
||||
session_id="s1",
|
||||
user_sub=None,
|
||||
since=NOW,
|
||||
limit=3,
|
||||
)
|
||||
assert [link.call_id for link in links] == ["2", "1", "0"]
|
||||
|
||||
def test_duplicates_counted_once_and_naive_times_treated_as_utc(self):
|
||||
naive = Candidate(
|
||||
"a",
|
||||
"t",
|
||||
(NOW - timedelta(minutes=1)).replace(tzinfo=None),
|
||||
True,
|
||||
"s1",
|
||||
"u1",
|
||||
)
|
||||
links = select_calls(
|
||||
[naive, cand("a", 1)], session_id="s1", user_sub=None, since=NOW, limit=5
|
||||
)
|
||||
assert len(links) == 1 and links[0].started_at.tzinfo is not None
|
||||
|
||||
def test_no_session_or_user_means_no_links(self):
|
||||
assert (
|
||||
select_calls(
|
||||
[cand("a", 1)], session_id=None, user_sub=None, since=NOW, limit=5
|
||||
)
|
||||
== []
|
||||
)
|
||||
|
||||
|
||||
def build(sinks=None, user="u1", **kwargs):
|
||||
app = FastMCP("correlated")
|
||||
|
||||
@app.tool
|
||||
def echo(text: str) -> str:
|
||||
return text
|
||||
|
||||
@app.tool
|
||||
def render(scene: str) -> str:
|
||||
raise RuntimeError(f"render of {scene} hung")
|
||||
|
||||
captured = {}
|
||||
|
||||
@app.tool
|
||||
async def report(ctx: Context) -> list:
|
||||
links = await mw.capture_recent_calls(ctx)
|
||||
captured["links"] = links
|
||||
return [link.call_id for link in links]
|
||||
|
||||
identity = {"user": user}
|
||||
mw = instrument(
|
||||
app,
|
||||
sinks if sinks is not None else [MemorySink()],
|
||||
identity_resolver=lambda c: {"user_sub": identity["user"]},
|
||||
**kwargs,
|
||||
)
|
||||
return app, mw, captured, identity
|
||||
|
||||
|
||||
class TestCaptureRecentCalls:
|
||||
async def test_same_session_calls_including_failures(self):
|
||||
app, mw, captured, _ = build()
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("echo", {"text": "a"})
|
||||
with contextlib.suppress(ToolError):
|
||||
await client.call_tool("render", {"scene": "castle"})
|
||||
await client.call_tool("report", {})
|
||||
|
||||
links = captured["links"]
|
||||
assert [(link.tool, link.ok, link.rule) for link in links] == [
|
||||
("echo", True, SAME_CLIENT_RULE),
|
||||
("render", False, SAME_CLIENT_RULE),
|
||||
]
|
||||
|
||||
async def test_new_session_per_call_falls_back_to_user_window(self):
|
||||
"""PAT/CI clients open a new MCP session for every call."""
|
||||
app, mw, captured, _ = build()
|
||||
for text in ("one", "two"):
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("echo", {"text": text})
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("report", {})
|
||||
|
||||
assert [link.rule for link in captured["links"]] == [RULE_USER_WINDOW] * 2
|
||||
|
||||
async def test_other_users_calls_are_not_linked(self):
|
||||
app, mw, captured, identity = build()
|
||||
identity["user"] = "someone-else"
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("echo", {"text": "not mine"})
|
||||
identity["user"] = "u1"
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("report", {})
|
||||
assert captured["links"] == []
|
||||
|
||||
async def test_calls_still_queued_are_found(self):
|
||||
release = asyncio.Event()
|
||||
|
||||
class Stalled:
|
||||
async def write(self, records):
|
||||
await release.wait()
|
||||
|
||||
app, mw, captured, _ = build(sinks=[Stalled()])
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("echo", {"text": "queued"})
|
||||
await client.call_tool("report", {})
|
||||
release.set()
|
||||
assert [link.tool for link in captured["links"]] == ["echo"]
|
||||
|
||||
async def test_database_fallback_after_restart(self, db):
|
||||
app1, mw1, _, _ = build(sinks=[db("restart")])
|
||||
async with Client(app1) as client:
|
||||
await client.call_tool("echo", {"text": "before restart"})
|
||||
await mw1.aclose()
|
||||
|
||||
# A new process: empty in-memory buffer, same database.
|
||||
app2, mw2, captured, _ = build(sinks=[db("restart")])
|
||||
async with Client(app2) as client:
|
||||
await client.call_tool("report", {})
|
||||
await mw2.aclose()
|
||||
|
||||
[link] = captured["links"]
|
||||
assert link.tool == "echo" and link.rule == RULE_USER_WINDOW
|
||||
|
||||
@pytest.mark.skipif(
|
||||
FASTMCP_4, reason="FastMCP 4 Client is sessionless (MCP 2026-07-28)"
|
||||
)
|
||||
async def test_failing_resolver_still_links_by_session(self):
|
||||
app, mw, captured, _ = build()
|
||||
mw.identity_resolver = lambda c: 1 / 0
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("echo", {"text": "a"})
|
||||
await client.call_tool("report", {})
|
||||
assert [link.rule for link in captured["links"]] == [RULE_SESSION]
|
||||
|
||||
async def test_slow_database_is_skipped(self, db):
|
||||
class SlowDb(DatabaseSink):
|
||||
async def recent_calls(self, **kwargs):
|
||||
await asyncio.sleep(5)
|
||||
return []
|
||||
|
||||
sink = db("slow", SlowDb)
|
||||
app, mw, captured, _ = build(sinks=[sink], correlation_timeout=0.05)
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("echo", {"text": "a"})
|
||||
await client.call_tool("report", {})
|
||||
await mw.aclose()
|
||||
# The lookup timed out, so capture returned [] rather than hanging.
|
||||
assert captured["links"] == []
|
||||
|
||||
async def test_outside_a_tool_call_returns_empty(self):
|
||||
_, mw, _, _ = build()
|
||||
mw.identity_resolver = None
|
||||
assert await mw.capture_recent_calls() == []
|
||||
|
||||
|
||||
class TestLinksAndContext:
|
||||
async def test_link_and_read_back_with_external_ids(self, db):
|
||||
sink = db("links")
|
||||
app, mw, captured, _ = build(sinks=[sink])
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("echo", {"text": "a"})
|
||||
with contextlib.suppress(ToolError):
|
||||
await client.call_tool("render", {"scene": "castle"})
|
||||
await client.call_tool("report", {})
|
||||
|
||||
assert await mw.link_feedback("bug-7Q2X", captured["links"])
|
||||
await mw.flush()
|
||||
context = await mw.feedback_context("bug-7Q2X")
|
||||
await mw.aclose()
|
||||
|
||||
assert [(c["position"], c["tool"], c["ok"], c["rule"]) for c in context] == [
|
||||
(0, "echo", True, SAME_CLIENT_RULE),
|
||||
(1, "render", False, SAME_CLIENT_RULE),
|
||||
]
|
||||
assert context[1]["error_type"] == "RuntimeError"
|
||||
|
||||
async def test_relinking_replaces_links(self, db):
|
||||
sink = db("relink")
|
||||
app, mw, captured, _ = build(sinks=[sink])
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("echo", {"text": "a"})
|
||||
await client.call_tool("report", {})
|
||||
links = captured["links"]
|
||||
await mw.link_feedback("fb-1", links)
|
||||
await mw.link_feedback("fb-1", links)
|
||||
assert len(await mw.feedback_context("fb-1")) == 1
|
||||
await mw.aclose()
|
||||
|
||||
async def test_context_fills_unflushed_calls_from_memory(self, db):
|
||||
release = asyncio.Event()
|
||||
|
||||
class StalledDb(DatabaseSink):
|
||||
async def write(self, records):
|
||||
await release.wait()
|
||||
|
||||
sink = db("unflushed", StalledDb)
|
||||
app, mw, captured, _ = build(sinks=[sink])
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("echo", {"text": "a"})
|
||||
await client.call_tool("report", {})
|
||||
await mw.link_feedback("fb-2", captured["links"])
|
||||
|
||||
[row] = await mw.feedback_context("fb-2")
|
||||
release.set()
|
||||
await mw.aclose()
|
||||
assert row["tool"] == "echo" and row["ok"] is True
|
||||
|
||||
async def test_link_without_database_sink_is_false(self):
|
||||
_, mw, _, _ = build()
|
||||
assert await mw.link_feedback("x", []) is False
|
||||
assert await mw.feedback_context("x") == []
|
||||
|
||||
|
||||
class TestSubmitFeedbackAutoLinks:
|
||||
async def test_submission_links_preceding_calls(self, db):
|
||||
app = FastMCP("app")
|
||||
|
||||
@app.tool
|
||||
def render(scene: str) -> str:
|
||||
raise RuntimeError("hung")
|
||||
|
||||
sink = db("submit")
|
||||
mw = instrument(app, [sink], identity_resolver=lambda c: {"user_sub": "u1"})
|
||||
add_feedback_tools(app, database_url="sqlite:///:memory:", instrumentation=mw)
|
||||
|
||||
async with Client(app) as client:
|
||||
with contextlib.suppress(ToolError):
|
||||
await client.call_tool("render", {"scene": "castle"})
|
||||
result = await client.call_tool(
|
||||
"submit_feedback",
|
||||
{
|
||||
"request": {
|
||||
"type": "bug",
|
||||
"title": "render hangs",
|
||||
"description": "the render tool hung",
|
||||
"submitter": "model",
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
assert result.data["linked_calls"] == 1
|
||||
await mw.flush()
|
||||
[row] = await mw.feedback_context(result.data["feedback_id"])
|
||||
await mw.aclose()
|
||||
assert (row["tool"], row["ok"], row["error_type"]) == (
|
||||
"render",
|
||||
False,
|
||||
"RuntimeError",
|
||||
)
|
||||
|
||||
async def test_without_instrumentation_response_unchanged(self):
|
||||
app = FastMCP("plain")
|
||||
add_feedback_tools(app, database_url="sqlite:///:memory:")
|
||||
async with Client(app) as client:
|
||||
result = await client.call_tool(
|
||||
"submit_feedback",
|
||||
{
|
||||
"request": {
|
||||
"type": "bug",
|
||||
"title": "t",
|
||||
"description": "no instrumentation",
|
||||
"submitter": "m",
|
||||
}
|
||||
},
|
||||
)
|
||||
assert "linked_calls" not in result.data
|
||||
|
||||
|
||||
class FakeContext:
|
||||
def __init__(self, transport=None, session_id="ctx-session"):
|
||||
self.transport = transport
|
||||
self.session_id = session_id
|
||||
|
||||
|
||||
class TestSessionKey:
|
||||
def test_http_session_header_wins(self, monkeypatch):
|
||||
import fastmcp.server.dependencies as deps
|
||||
|
||||
monkeypatch.setattr(
|
||||
deps,
|
||||
"get_http_headers",
|
||||
lambda include_all=False: {"mcp-session-id": "hdr-1"},
|
||||
)
|
||||
assert (
|
||||
mw_module.session_key(FakeContext(transport="streamable-http")) == "hdr-1"
|
||||
)
|
||||
|
||||
def test_stdio_uses_one_key_per_process(self):
|
||||
a = mw_module.session_key(FakeContext(transport="stdio", session_id="x"))
|
||||
b = mw_module.session_key(FakeContext(transport="stdio", session_id="y"))
|
||||
assert a == b and a.startswith("stdio:")
|
||||
|
||||
def test_falls_back_to_ctx_session_id(self):
|
||||
assert mw_module.session_key(FakeContext()) == "ctx-session"
|
||||
|
||||
def test_no_context(self):
|
||||
assert mw_module.session_key(None) is None
|
||||
|
||||
|
||||
class TestStdioEndToEnd:
|
||||
async def test_same_stdio_client_links_by_session(self):
|
||||
"""On a real stdio transport, one client's calls link by session on every FastMCP version."""
|
||||
server = (
|
||||
Path(__file__).parent.parent / "fixtures" / "stdio_correlation_server.py"
|
||||
)
|
||||
transport = PythonStdioTransport(str(server), python_cmd=sys.executable)
|
||||
async with Client(transport) as client:
|
||||
await client.call_tool("echo", {"text": "a"})
|
||||
await client.call_tool("echo", {"text": "b"})
|
||||
result = await client.call_tool("report", {})
|
||||
pairs = json.loads(result.content[0].text)
|
||||
assert pairs == [["echo", RULE_SESSION], ["echo", RULE_SESSION]]
|
||||
@@ -345,7 +345,7 @@ class TestServerComposition:
|
||||
async def test_compose_feedback_services_success(self):
|
||||
"""Test successful service composition."""
|
||||
main_server = Mock()
|
||||
main_server.import_server = Mock()
|
||||
main_server.mount = Mock()
|
||||
|
||||
services = {
|
||||
"service1": {
|
||||
@@ -376,13 +376,38 @@ class TestServerComposition:
|
||||
mock_submission.assert_called_once()
|
||||
mock_management.assert_called_once()
|
||||
mock_feedback.assert_called_once()
|
||||
assert main_server.import_server.call_count == 3
|
||||
assert main_server.mount.call_count == 3
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compose_feedback_services_real_servers(self, tmp_path):
|
||||
"""Composed tools must actually appear under their prefix.
|
||||
|
||||
The mocked tests above can't catch a missing await: a sync Mock
|
||||
records the call either way. This one uses real FastMCP servers.
|
||||
"""
|
||||
from fastmcp import FastMCP
|
||||
|
||||
from tests._compat import tool_map
|
||||
|
||||
main_server = FastMCP("Main")
|
||||
compose_feedback_services(
|
||||
main_server,
|
||||
{
|
||||
"public": {"database_url": f"sqlite:///{tmp_path}/p.db", "type": "submission"},
|
||||
"admin": {"database_url": f"sqlite:///{tmp_path}/a.db", "type": "full"},
|
||||
},
|
||||
)
|
||||
|
||||
names = set(await tool_map(main_server))
|
||||
assert "public_submit_feedback" in names
|
||||
assert "admin_list_feedback" in names
|
||||
assert "admin_update_feedback_status" in names
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compose_feedback_services_with_shared_insights(self):
|
||||
"""Test service composition with shared insights."""
|
||||
main_server = Mock()
|
||||
main_server.import_server = Mock()
|
||||
main_server.mount = Mock()
|
||||
shared_insights = Mock()
|
||||
|
||||
services = {
|
||||
|
||||
@@ -0,0 +1,484 @@
|
||||
"""Tests for per-call instrumentation: redaction, middleware, dispatch, sinks.
|
||||
|
||||
These run against real FastMCP servers through fastmcp.Client, so they
|
||||
exercise the same middleware path on every supported FastMCP version.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import io
|
||||
import json
|
||||
import time
|
||||
|
||||
import pytest
|
||||
from fastmcp import Client, FastMCP
|
||||
from fastmcp.exceptions import ToolError
|
||||
from sqlalchemy import MetaData, inspect, select
|
||||
from sqlalchemy.ext.asyncio import create_async_engine
|
||||
|
||||
from fastmcp_feedback import add_feedback_tools
|
||||
from fastmcp_feedback.instrumentation import (
|
||||
REDACTED,
|
||||
CallbackSink,
|
||||
DatabaseSink,
|
||||
InstrumentationMiddleware,
|
||||
JsonLinesSink,
|
||||
MemorySink,
|
||||
RecordDispatcher,
|
||||
Redactor,
|
||||
ToolCallRecord,
|
||||
build_metadata,
|
||||
instrument,
|
||||
)
|
||||
|
||||
BMCP_TOKEN = "bmcp_" + "A1b2C3d4E5f6G7h8I9j0"
|
||||
JWT = "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTYifQ.c2lnbmF0dXJlLXZhbHVl"
|
||||
|
||||
|
||||
def make_app(sinks=None, **kwargs):
|
||||
sink = MemorySink()
|
||||
app = FastMCP("instrumented")
|
||||
|
||||
@app.tool
|
||||
def echo(text: str) -> dict:
|
||||
return {"echo": text}
|
||||
|
||||
@app.tool
|
||||
def create_token(label: str) -> dict:
|
||||
return {"label": label, "access_token": BMCP_TOKEN, "note": f"use {BMCP_TOKEN}"}
|
||||
|
||||
@app.tool
|
||||
def login(user: str, password: str) -> str:
|
||||
return f"welcome {user}"
|
||||
|
||||
@app.tool
|
||||
def fail(secret: str) -> str:
|
||||
raise ValueError(f"bad input secret={secret}")
|
||||
|
||||
mw = instrument(app, sinks if sinks is not None else [sink], **kwargs)
|
||||
return app, sink, mw
|
||||
|
||||
|
||||
async def call(app, tool, args=None):
|
||||
async with Client(app) as client:
|
||||
return await client.call_tool(tool, args or {})
|
||||
|
||||
|
||||
class TestRedactor:
|
||||
def test_secret_keys_at_any_depth(self):
|
||||
data = {"a": {"b": [{"password": "hunter2", "keep": 1}]}, "Authorization": "x"}
|
||||
out = Redactor().redact(data)
|
||||
assert out["a"]["b"][0] == {"password": REDACTED, "keep": 1}
|
||||
assert out["Authorization"] == REDACTED
|
||||
|
||||
def test_suffix_and_dash_variants(self):
|
||||
out = Redactor().redact(
|
||||
{
|
||||
"github_token": "t",
|
||||
"DB-Password": "p",
|
||||
"X-Api-Key": "k",
|
||||
"stripe_api_key": "s",
|
||||
}
|
||||
)
|
||||
assert set(out.values()) == {REDACTED}
|
||||
|
||||
def test_non_secret_keys_untouched(self):
|
||||
data = {"max_tokens": 100, "token_count": 3, "tool": "echo", "tokens": [1, 2]}
|
||||
assert Redactor().redact(data) == data
|
||||
|
||||
def test_secret_values_inside_text(self):
|
||||
text = f"pat {BMCP_TOKEN}, jwt {JWT}, header Bearer abcdefghijklmnop, key sk-{'x' * 20}"
|
||||
out = Redactor().redact_text(text)
|
||||
for secret in (BMCP_TOKEN, JWT, "abcdefghijklmnop", "sk-"):
|
||||
assert secret not in out
|
||||
assert out.count(REDACTED) == 4
|
||||
|
||||
def test_key_value_text_keeps_the_key(self):
|
||||
out = Redactor().redact_text("failed: password=hunter2 api_key: abc123, ok")
|
||||
assert out == f"failed: password={REDACTED} api_key: {REDACTED}, ok"
|
||||
|
||||
def test_input_not_mutated(self):
|
||||
data = {"password": "hunter2", "nested": {"token": "t"}}
|
||||
Redactor().redact(data)
|
||||
assert data == {"password": "hunter2", "nested": {"token": "t"}}
|
||||
|
||||
def test_none_secret_value_left_as_none(self):
|
||||
assert Redactor().redact({"password": None}) == {"password": None}
|
||||
|
||||
def test_extra_keys_and_patterns(self):
|
||||
r = Redactor(extra_keys=["license"], extra_patterns=[r"ACME-\d{6}"])
|
||||
assert r.redact({"license": "L", "msg": "code ACME-123456"}) == {
|
||||
"license": REDACTED,
|
||||
"msg": f"code {REDACTED}",
|
||||
}
|
||||
|
||||
def test_depth_limit(self):
|
||||
deep = current = {}
|
||||
for _ in range(50):
|
||||
current["n"] = {}
|
||||
current = current["n"]
|
||||
assert "[TRUNCATED]" in json.dumps(Redactor(max_depth=5).redact(deep))
|
||||
|
||||
|
||||
class TestMiddlewareRecording:
|
||||
async def test_success_in_meta_mode(self):
|
||||
app, sink, mw = make_app()
|
||||
await call(app, "echo", {"text": "hello"})
|
||||
await mw.flush()
|
||||
|
||||
[rec] = sink.records
|
||||
assert rec.tool == "echo" and rec.ok and rec.mode == "meta"
|
||||
assert rec.duration_ms > 0
|
||||
assert rec.args_size == len(json.dumps({"text": "hello"}))
|
||||
assert rec.result_size and rec.result_size > 0
|
||||
assert rec.args is None and rec.result is None
|
||||
assert rec.session_id and rec.request_id
|
||||
|
||||
async def test_full_mode_redacts_args_and_results(self):
|
||||
app, sink, mw = make_app(mode="full")
|
||||
await call(app, "create_token", {"label": "ci"})
|
||||
await call(app, "login", {"user": "ryan", "password": "hunter2"})
|
||||
await mw.flush()
|
||||
|
||||
token_rec, login_rec = sink.records
|
||||
dumped = json.dumps([token_rec.to_dict(), login_rec.to_dict()])
|
||||
assert BMCP_TOKEN not in dumped and "hunter2" not in dumped
|
||||
assert token_rec.result["access_token"] == REDACTED
|
||||
assert token_rec.result["note"] == f"use {REDACTED}"
|
||||
assert token_rec.result["label"] == "ci"
|
||||
assert login_rec.args == {"user": "ryan", "password": REDACTED}
|
||||
|
||||
async def test_failed_call_recorded_and_error_unchanged(self):
|
||||
app, sink, mw = make_app()
|
||||
with pytest.raises(ToolError, match="bad input"):
|
||||
await call(app, "fail", {"secret": "hunter2"})
|
||||
await mw.flush()
|
||||
|
||||
[rec] = sink.records
|
||||
assert not rec.ok
|
||||
assert rec.error_type == "ValueError"
|
||||
assert "hunter2" not in rec.error_message
|
||||
assert f"secret={REDACTED}" in rec.error_message
|
||||
|
||||
async def test_meta_only_tools_override_full_mode(self):
|
||||
app, sink, mw = make_app(mode="full", meta_only_tools={"create_token"})
|
||||
await call(app, "create_token", {"label": "ci"})
|
||||
await call(app, "echo", {"text": "hi"})
|
||||
await mw.flush()
|
||||
|
||||
token_rec, echo_rec = sink.records
|
||||
assert token_rec.mode == "meta" and token_rec.result is None
|
||||
assert token_rec.result_size > 0
|
||||
assert echo_rec.mode == "full" and echo_rec.result == {"echo": "hi"}
|
||||
|
||||
async def test_off_mode_records_nothing(self):
|
||||
app, sink, mw = make_app(mode="off")
|
||||
await call(app, "echo", {"text": "hi"})
|
||||
await mw.flush()
|
||||
assert sink.records == []
|
||||
|
||||
async def test_mode_from_environment(self, monkeypatch):
|
||||
monkeypatch.setenv("FEEDBACK_INSTRUMENTATION_MODE", "full")
|
||||
assert InstrumentationMiddleware([]).mode == "full"
|
||||
|
||||
def test_invalid_mode_rejected(self):
|
||||
with pytest.raises(ValueError, match="mode must be one of"):
|
||||
InstrumentationMiddleware([], mode="everything")
|
||||
|
||||
|
||||
class TestHooks:
|
||||
async def test_identity_resolver_fills_columns(self):
|
||||
def who(context):
|
||||
assert context.fastmcp_context is not None
|
||||
return {
|
||||
"user_sub": "user-1",
|
||||
"caller_kind": "pat:7",
|
||||
"client_id": "blender-addon",
|
||||
"org": "acme",
|
||||
"token": BMCP_TOKEN,
|
||||
}
|
||||
|
||||
app, sink, mw = make_app(identity_resolver=who)
|
||||
await call(app, "echo", {"text": "hi"})
|
||||
await mw.flush()
|
||||
|
||||
[rec] = sink.records
|
||||
assert (rec.user_sub, rec.caller_kind, rec.client_id) == (
|
||||
"user-1",
|
||||
"pat:7",
|
||||
"blender-addon",
|
||||
)
|
||||
assert rec.identity == {"org": "acme", "token": REDACTED}
|
||||
|
||||
async def test_async_enricher_gets_result_and_sets_extra(self):
|
||||
seen = {}
|
||||
|
||||
async def enrich(tool, args, result, context):
|
||||
seen.update(tool=tool, args=args, result=result)
|
||||
return {"job_id": "job-42", "server_version": "2026.9.27", "api_key": "k"}
|
||||
|
||||
app, sink, mw = make_app(enricher=enrich, server_version="default")
|
||||
await call(app, "echo", {"text": "hi"})
|
||||
await mw.flush()
|
||||
|
||||
[rec] = sink.records
|
||||
assert seen == {
|
||||
"tool": "echo",
|
||||
"args": {"text": "hi"},
|
||||
"result": {"echo": "hi"},
|
||||
}
|
||||
assert rec.extra == {"job_id": "job-42", "api_key": REDACTED}
|
||||
assert rec.server_version == "2026.9.27"
|
||||
|
||||
async def test_server_version_default(self):
|
||||
app, sink, mw = make_app(server_version="1.0")
|
||||
await call(app, "echo", {"text": "hi"})
|
||||
await mw.flush()
|
||||
assert sink.records[0].server_version == "1.0"
|
||||
|
||||
async def test_hook_exceptions_do_not_break_calls(self):
|
||||
def boom(*args):
|
||||
raise RuntimeError("hook bug")
|
||||
|
||||
app, sink, mw = make_app(identity_resolver=boom, enricher=boom)
|
||||
result = await call(app, "echo", {"text": "hi"})
|
||||
await mw.flush()
|
||||
|
||||
assert result.data == {"echo": "hi"}
|
||||
[rec] = sink.records
|
||||
assert rec.ok and rec.identity == {} and rec.extra == {}
|
||||
|
||||
async def test_slow_async_hook_is_cut_off(self):
|
||||
async def slow(*args):
|
||||
await asyncio.sleep(5)
|
||||
return {"never": True}
|
||||
|
||||
app, sink, mw = make_app(enricher=slow, hook_timeout=0.05)
|
||||
start = time.perf_counter()
|
||||
await call(app, "echo", {"text": "hi"})
|
||||
assert time.perf_counter() - start < 2
|
||||
await mw.flush()
|
||||
assert sink.records[0].extra == {}
|
||||
|
||||
|
||||
class TestNeverBlocksOrBreaks:
|
||||
async def test_failing_sink_does_not_break_calls_or_other_sinks(self):
|
||||
class Broken:
|
||||
async def write(self, records):
|
||||
raise ConnectionError("database down")
|
||||
|
||||
good = MemorySink()
|
||||
app, _, mw = make_app(sinks=[Broken(), good])
|
||||
result = await call(app, "echo", {"text": "hi"})
|
||||
await mw.flush()
|
||||
|
||||
assert result.data == {"echo": "hi"}
|
||||
assert mw.dispatcher.sink_errors == 1
|
||||
assert len(good.records) == 1
|
||||
|
||||
async def test_stalled_sink_drops_records_instead_of_waiting(self):
|
||||
release = asyncio.Event()
|
||||
|
||||
class Stalled:
|
||||
def __init__(self):
|
||||
self.records = []
|
||||
|
||||
async def write(self, records):
|
||||
await release.wait()
|
||||
self.records.extend(records)
|
||||
|
||||
stalled = Stalled()
|
||||
app, _, mw = make_app(sinks=[stalled], max_queue=2, batch_size=1)
|
||||
|
||||
start = time.perf_counter()
|
||||
async with Client(app) as client:
|
||||
for i in range(6):
|
||||
await client.call_tool("echo", {"text": str(i)})
|
||||
assert time.perf_counter() - start < 3
|
||||
|
||||
# One record is in the stalled write, two fit the queue, the rest drop.
|
||||
assert mw.dispatcher.dropped == 3
|
||||
release.set()
|
||||
await mw.flush()
|
||||
assert len(stalled.records) == 3
|
||||
|
||||
async def test_record_building_failure_does_not_break_calls(self, monkeypatch):
|
||||
app, sink, mw = make_app()
|
||||
|
||||
async def explode(*args, **kwargs):
|
||||
raise RuntimeError("bug in record building")
|
||||
|
||||
monkeypatch.setattr(mw, "_build_record", explode)
|
||||
result = await call(app, "echo", {"text": "hi"})
|
||||
assert result.data == {"echo": "hi"}
|
||||
|
||||
async def test_tool_error_passes_through_even_if_recording_fails(self, monkeypatch):
|
||||
app, sink, mw = make_app()
|
||||
|
||||
async def explode(*args, **kwargs):
|
||||
raise RuntimeError("bug in record building")
|
||||
|
||||
monkeypatch.setattr(mw, "_build_record", explode)
|
||||
with pytest.raises(ToolError, match="bad input"):
|
||||
await call(app, "fail", {"secret": "x"})
|
||||
|
||||
def test_submit_without_event_loop_drops(self):
|
||||
dispatcher = RecordDispatcher([MemorySink()])
|
||||
record = ToolCallRecord(
|
||||
tool="t",
|
||||
started_at=__import__("datetime").datetime.now(),
|
||||
duration_ms=1,
|
||||
ok=True,
|
||||
mode="meta",
|
||||
)
|
||||
assert dispatcher.submit(record) is False
|
||||
assert dispatcher.dropped == 1
|
||||
|
||||
|
||||
class TestSinks:
|
||||
async def test_json_lines_sink(self):
|
||||
stream = io.StringIO()
|
||||
app, _, mw = make_app(sinks=[JsonLinesSink(stream)])
|
||||
await call(app, "echo", {"text": "hi"})
|
||||
await call(app, "echo", {"text": "again"})
|
||||
await mw.flush()
|
||||
|
||||
lines = [json.loads(line) for line in stream.getvalue().splitlines()]
|
||||
assert [line["tool"] for line in lines] == ["echo", "echo"]
|
||||
assert lines[0]["started_at"].endswith("+00:00")
|
||||
|
||||
async def test_callback_sink_sync_and_async(self):
|
||||
got_sync, got_async = [], []
|
||||
|
||||
async def async_cb(records):
|
||||
got_async.extend(records)
|
||||
|
||||
app, _, mw = make_app(
|
||||
sinks=[CallbackSink(got_sync.extend), CallbackSink(async_cb)]
|
||||
)
|
||||
await call(app, "echo", {"text": "hi"})
|
||||
await mw.flush()
|
||||
assert len(got_sync) == len(got_async) == 1
|
||||
|
||||
async def test_aclose_closes_sinks(self):
|
||||
closed = []
|
||||
|
||||
class Closable(MemorySink):
|
||||
async def aclose(self):
|
||||
closed.append(True)
|
||||
|
||||
app, _, mw = make_app(sinks=[Closable()])
|
||||
await call(app, "echo", {"text": "hi"})
|
||||
await mw.aclose()
|
||||
assert closed == [True]
|
||||
|
||||
|
||||
class TestDatabaseSink:
|
||||
async def test_end_to_end_with_created_tables(self, tmp_path):
|
||||
url = f"sqlite+aiosqlite:///{tmp_path}/calls.db"
|
||||
db = DatabaseSink(url, create_tables=True)
|
||||
app, _, mw = make_app(sinks=[db], mode="full", server_version="9.9")
|
||||
await call(app, "create_token", {"label": "ci"})
|
||||
with pytest.raises(ToolError):
|
||||
await call(app, "fail", {"secret": "hunter2"})
|
||||
await mw.flush()
|
||||
|
||||
async with db.engine.connect() as conn:
|
||||
rows = (
|
||||
(await conn.execute(select(db.table).order_by(db.table.c.started_at)))
|
||||
.mappings()
|
||||
.all()
|
||||
)
|
||||
await mw.aclose()
|
||||
|
||||
assert [r["tool"] for r in rows] == ["create_token", "fail"]
|
||||
assert rows[0]["ok"] is True and rows[1]["ok"] is False
|
||||
assert rows[0]["result"]["access_token"] == REDACTED
|
||||
assert rows[1]["error_type"] == "ValueError"
|
||||
assert rows[0]["server_version"] == "9.9"
|
||||
assert "hunter2" not in json.dumps([dict(r) for r in rows], default=str)
|
||||
|
||||
async def test_prefix_and_host_owned_engine(self, tmp_path):
|
||||
engine = create_async_engine(f"sqlite+aiosqlite:///{tmp_path}/host.db")
|
||||
db = DatabaseSink(engine, prefix="qa_", create_tables=True)
|
||||
app, _, mw = make_app(sinks=[db])
|
||||
await call(app, "echo", {"text": "hi"})
|
||||
await mw.aclose()
|
||||
|
||||
# The host's engine is still usable: the sink did not dispose it.
|
||||
async with engine.connect() as conn:
|
||||
tables = await conn.run_sync(lambda c: inspect(c).get_table_names())
|
||||
await engine.dispose()
|
||||
assert sorted(tables) == ["qa_feedback_call_links", "qa_tool_calls"]
|
||||
|
||||
async def test_no_tables_created_by_default(self, tmp_path):
|
||||
engine = create_async_engine(f"sqlite+aiosqlite:///{tmp_path}/host.db")
|
||||
app, _, mw = make_app(sinks=[DatabaseSink(engine)])
|
||||
result = await call(app, "echo", {"text": "hi"})
|
||||
await mw.flush()
|
||||
|
||||
async with engine.connect() as conn:
|
||||
tables = await conn.run_sync(lambda c: inspect(c).get_table_names())
|
||||
await engine.dispose()
|
||||
|
||||
# The insert fails (no table), is counted, and the call still worked.
|
||||
assert result.data == {"echo": "hi"}
|
||||
assert tables == []
|
||||
assert mw.dispatcher.sink_errors == 1
|
||||
|
||||
async def test_absent_payloads_are_sql_null(self, tmp_path):
|
||||
"""Meta mode stores no payloads; they must be SQL NULL, not JSON null."""
|
||||
db = DatabaseSink(f"sqlite+aiosqlite:///{tmp_path}/n.db", create_tables=True)
|
||||
app, _, mw = make_app(sinks=[db])
|
||||
await call(app, "echo", {"text": "hi"})
|
||||
await mw.flush()
|
||||
|
||||
t = db.table
|
||||
async with db.engine.connect() as conn:
|
||||
nulls = (
|
||||
await conn.execute(select(t.c.args.is_(None), t.c.result.is_(None)))
|
||||
).one()
|
||||
await mw.aclose()
|
||||
assert tuple(nulls) == (True, True)
|
||||
|
||||
def test_build_metadata_joins_host_metadata(self):
|
||||
host = MetaData()
|
||||
assert build_metadata("ffb_", host) is host
|
||||
build_metadata("ffb_", host)
|
||||
assert sorted(host.tables) == ["ffb_feedback_call_links", "ffb_tool_calls"]
|
||||
|
||||
|
||||
class TestIntegration:
|
||||
async def test_records_mounted_server_tools_by_prefixed_name(self):
|
||||
main = FastMCP("main")
|
||||
sub = FastMCP("sub")
|
||||
|
||||
@sub.tool
|
||||
def ping() -> str:
|
||||
return "pong"
|
||||
|
||||
main.mount(sub, "svc")
|
||||
sink = MemorySink()
|
||||
mw = instrument(main, [sink])
|
||||
await call(main, "svc_ping")
|
||||
await mw.flush()
|
||||
assert [r.tool for r in sink.records] == ["svc_ping"]
|
||||
|
||||
async def test_records_feedback_tools(self):
|
||||
app = FastMCP("with-feedback")
|
||||
add_feedback_tools(app, database_url="sqlite:///:memory:")
|
||||
sink = MemorySink()
|
||||
mw = instrument(app, [sink])
|
||||
await call(
|
||||
app,
|
||||
"submit_feedback",
|
||||
{
|
||||
"request": {
|
||||
"type": "bug",
|
||||
"title": "t",
|
||||
"description": "instrumented",
|
||||
"submitter": "me",
|
||||
}
|
||||
},
|
||||
)
|
||||
await mw.flush()
|
||||
assert [r.tool for r in sink.records] == ["submit_feedback"]
|
||||
@@ -7,6 +7,7 @@ from fastmcp import FastMCP
|
||||
from fastmcp_feedback.mixins import SubmissionMixin, RetrievalMixin, ManagementMixin
|
||||
from fastmcp_feedback.models import SubmitFeedbackRequest, FeedbackStatus
|
||||
from fastmcp_feedback.database import FeedbackDatabase
|
||||
from tests._compat import tool_map
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@@ -95,7 +96,7 @@ class TestSubmissionMixin:
|
||||
mixin.register_tools(mock_fastmcp_server)
|
||||
|
||||
# Verify tool was registered
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
assert "submit_feedback" in tool_names
|
||||
|
||||
@@ -107,7 +108,7 @@ class TestSubmissionMixin:
|
||||
mixin.register_tools(mock_fastmcp_server, prefix="support", separator="_")
|
||||
|
||||
# Verify tool was registered with prefix
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
assert "support_submit_feedback" in tool_names
|
||||
|
||||
@@ -267,7 +268,7 @@ class TestRetrievalMixin:
|
||||
|
||||
mixin.register_tools(mock_fastmcp_server)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
expected_tools = ["list_feedback", "get_feedback_statistics"]
|
||||
@@ -437,7 +438,7 @@ class TestManagementMixin:
|
||||
|
||||
mixin.register_tools(mock_fastmcp_server)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
expected_tools = ["update_feedback_status", "delete_feedback"]
|
||||
@@ -532,7 +533,7 @@ class TestMixinIntegration:
|
||||
management.register_tools(mock_fastmcp_server)
|
||||
|
||||
# Verify all tools are registered
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
expected_tools = [
|
||||
@@ -555,7 +556,7 @@ class TestMixinIntegration:
|
||||
submission.register_tools(mock_fastmcp_server, prefix="admin")
|
||||
retrieval.register_tools(mock_fastmcp_server, prefix="public")
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
assert "admin_submit_feedback" in tool_names
|
||||
@@ -663,7 +664,7 @@ class TestMixinConfiguration:
|
||||
# Register with dash separator
|
||||
submission.register_tools(mock_fastmcp_server, prefix="api", separator="-")
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
assert "api-submit_feedback" in tool_names
|
||||
@@ -676,7 +677,7 @@ class TestMixinConfiguration:
|
||||
# Test the standard registration instead
|
||||
submission.register_tools(mock_fastmcp_server)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# SubmissionMixin should only register its tools
|
||||
|
||||
@@ -0,0 +1,397 @@
|
||||
"""Tests for classifying error results that tools return instead of raising.
|
||||
|
||||
Real FastMCP servers through fastmcp.Client, as in test_instrumentation.py.
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import logging
|
||||
|
||||
import pytest
|
||||
from fastmcp import Client, FastMCP
|
||||
from fastmcp.exceptions import ToolError
|
||||
from mcp.types import CallToolResult, TextContent
|
||||
from sqlalchemy import select
|
||||
|
||||
from fastmcp_feedback import add_feedback_tools
|
||||
from fastmcp_feedback.instrumentation import (
|
||||
DEFAULT_RESULT_CLASSIFIER,
|
||||
REDACTED,
|
||||
DatabaseSink,
|
||||
MemorySink,
|
||||
ToolCallRecord,
|
||||
instrument,
|
||||
soft_error_classifier,
|
||||
)
|
||||
from fastmcp_feedback.mixins import submission
|
||||
|
||||
BMCP_TOKEN = "bmcp_" + "A1b2C3d4E5f6G7h8I9j0"
|
||||
|
||||
try:
|
||||
from fastmcp.tools.tool import ToolResult
|
||||
except ImportError: # FastMCP 4
|
||||
from fastmcp.tools import ToolResult
|
||||
|
||||
try:
|
||||
ToolResult(content=[TextContent(type="text", text="x")], is_error=True)
|
||||
TOOL_RESULT_IS_ERROR = True
|
||||
except TypeError: # added in FastMCP 3.4
|
||||
TOOL_RESULT_IS_ERROR = False
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def db(tmp_path):
|
||||
"""DatabaseSinks whose engines are always disposed, or the process hangs."""
|
||||
made = []
|
||||
|
||||
def factory(name="calls"):
|
||||
sink = DatabaseSink(
|
||||
f"sqlite+aiosqlite:///{tmp_path}/{name}.db", create_tables=True
|
||||
)
|
||||
made.append(sink)
|
||||
return sink
|
||||
|
||||
yield factory
|
||||
for sink in made:
|
||||
await sink.engine.dispose()
|
||||
|
||||
|
||||
def make_app(sinks=None, **kwargs):
|
||||
sink = MemorySink()
|
||||
app = FastMCP("soft")
|
||||
|
||||
@app.tool
|
||||
def render(status: str) -> dict:
|
||||
return {"status": status, "error": f"render said {status}"}
|
||||
|
||||
@app.tool
|
||||
def respond(payload: dict) -> dict:
|
||||
return payload
|
||||
|
||||
@app.tool
|
||||
def text(value: str) -> str:
|
||||
return value
|
||||
|
||||
@app.tool
|
||||
def items() -> list:
|
||||
return [{"status": "failed"}]
|
||||
|
||||
@app.tool
|
||||
def crash() -> dict:
|
||||
raise RuntimeError("exploded")
|
||||
|
||||
@app.tool
|
||||
def mcp_error() -> CallToolResult:
|
||||
return CallToolResult(
|
||||
content=[TextContent(type="text", text="target missing")], isError=True
|
||||
)
|
||||
|
||||
mw = instrument(app, sinks if sinks is not None else [sink], **kwargs)
|
||||
return app, sink, mw
|
||||
|
||||
|
||||
async def run(app, mw, tool, args=None):
|
||||
async with Client(app) as client:
|
||||
with contextlib.suppress(ToolError):
|
||||
await client.call_tool(tool, args or {})
|
||||
await mw.flush()
|
||||
|
||||
|
||||
async def record_for(tool, args=None, **kwargs) -> ToolCallRecord:
|
||||
app, sink, mw = make_app(**kwargs)
|
||||
await run(app, mw, tool, args)
|
||||
[record] = sink.records
|
||||
return record
|
||||
|
||||
|
||||
class TestClassifier:
|
||||
@pytest.mark.parametrize(
|
||||
"result, expected",
|
||||
[
|
||||
({"status": "failed", "error": "disk full"}, "status=failed: disk full"),
|
||||
({"state": "ERROR"}, "state=ERROR"),
|
||||
({"ok": False, "message": "no scene"}, "ok=False: no scene"),
|
||||
({"success": False, "error": "nope"}, "success=False: nope"),
|
||||
({"success": False, "detail": "d", "reason": "r"}, "success=False: d"),
|
||||
(
|
||||
{"status": "failure", "error": {"code": 7, "message": "inner"}},
|
||||
"status=failure: inner",
|
||||
),
|
||||
({"isError": True, "content": ["boom"]}, "isError: boom"),
|
||||
(
|
||||
{"is_error": True, "content": [{"type": "text", "text": "a"}]},
|
||||
"isError: a",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_flagged(self, result, expected):
|
||||
assert DEFAULT_RESULT_CLASSIFIER("t", result) == expected
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"result",
|
||||
[
|
||||
{"status": "done"},
|
||||
{"status": "unknown_target"},
|
||||
{"ok": True},
|
||||
{"ok": 0},
|
||||
{"success": None},
|
||||
{"result": False},
|
||||
{"state": 3},
|
||||
{"isError": False},
|
||||
[{"status": "failed"}],
|
||||
"failed",
|
||||
None,
|
||||
],
|
||||
)
|
||||
def test_not_flagged(self, result):
|
||||
assert DEFAULT_RESULT_CLASSIFIER("t", result) is None
|
||||
|
||||
def test_extra_statuses(self):
|
||||
classify = soft_error_classifier(extra_statuses=("unknown_target",))
|
||||
assert (
|
||||
classify("t", {"status": "unknown_target", "error": "no cube"})
|
||||
== "status=unknown_target: no cube"
|
||||
)
|
||||
assert classify("t", {"status": "failed"}) == "status=failed"
|
||||
|
||||
def test_custom_keys_replace_defaults(self):
|
||||
classify = soft_error_classifier(
|
||||
statuses=("bad",), status_keys=("phase",), flag_keys=("passed",)
|
||||
)
|
||||
assert classify("t", {"phase": "bad"}) == "phase=bad"
|
||||
assert classify("t", {"passed": False}) == "passed=False"
|
||||
assert classify("t", {"status": "failed", "ok": False}) is None
|
||||
|
||||
|
||||
class TestMiddleware:
|
||||
async def test_status_failed_is_soft_error(self):
|
||||
record = await record_for("render", {"status": "failed"})
|
||||
assert (record.outcome, record.ok, record.error_type) == (
|
||||
"soft_error",
|
||||
False,
|
||||
"SoftError",
|
||||
)
|
||||
assert record.error_message == "status=failed: render said failed"
|
||||
|
||||
@pytest.mark.parametrize("key", ["ok", "success"])
|
||||
async def test_false_flags_are_soft_errors(self, key):
|
||||
record = await record_for("respond", {"payload": {key: False, "error": "x"}})
|
||||
assert (record.outcome, record.ok) == ("soft_error", False)
|
||||
assert record.error_message == f"{key}=False: x"
|
||||
|
||||
async def test_message_is_redacted_and_truncated(self):
|
||||
record = await record_for(
|
||||
"respond",
|
||||
{"payload": {"status": "error", "error": f"bad key {BMCP_TOKEN} " * 50}},
|
||||
max_error_chars=60,
|
||||
)
|
||||
assert BMCP_TOKEN not in record.error_message
|
||||
assert REDACTED in record.error_message
|
||||
assert len(record.error_message) == 60
|
||||
|
||||
async def test_extra_statuses(self):
|
||||
record = await record_for(
|
||||
"render",
|
||||
{"status": "unknown_target"},
|
||||
result_classifier=soft_error_classifier(extra_statuses=("unknown_target",)),
|
||||
)
|
||||
assert record.outcome == "soft_error"
|
||||
assert (
|
||||
record.error_message == "status=unknown_target: render said unknown_target"
|
||||
)
|
||||
|
||||
async def test_unknown_status_is_ok_by_default(self):
|
||||
record = await record_for("render", {"status": "unknown_target"})
|
||||
assert (record.outcome, record.ok, record.error_type) == ("ok", True, None)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"tool, args",
|
||||
[
|
||||
("text", {"value": "failed"}),
|
||||
("items", {}),
|
||||
("respond", {"payload": {"status": "done", "count": 3}}),
|
||||
],
|
||||
)
|
||||
async def test_normal_results_stay_ok(self, tool, args):
|
||||
record = await record_for(tool, args)
|
||||
assert (record.outcome, record.ok, record.error_message) == ("ok", True, None)
|
||||
|
||||
async def test_none_disables(self):
|
||||
record = await record_for(
|
||||
"render", {"status": "failed"}, result_classifier=None
|
||||
)
|
||||
assert (record.outcome, record.ok, record.error_type) == ("ok", True, None)
|
||||
|
||||
async def test_custom_classifier(self):
|
||||
seen = []
|
||||
|
||||
def classify(tool, result):
|
||||
seen.append((tool, result))
|
||||
return "empty" if result.get("count") == 0 else None
|
||||
|
||||
record = await record_for(
|
||||
"respond", {"payload": {"count": 0}}, result_classifier=classify
|
||||
)
|
||||
assert seen == [("respond", {"count": 0})]
|
||||
assert (record.outcome, record.error_message) == ("soft_error", "empty")
|
||||
|
||||
async def test_raising_classifier_records_ok(self, caplog):
|
||||
def classify(tool, result):
|
||||
raise KeyError("classifier bug")
|
||||
|
||||
app, sink, mw = make_app(result_classifier=classify)
|
||||
with caplog.at_level(logging.WARNING):
|
||||
async with Client(app) as client:
|
||||
result = await client.call_tool("render", {"status": "failed"})
|
||||
await mw.flush()
|
||||
assert result.data["status"] == "failed"
|
||||
[record] = sink.records
|
||||
assert (record.outcome, record.ok, record.error_type) == ("ok", True, None)
|
||||
assert "Result classifier" in caplog.text
|
||||
|
||||
async def test_classifier_not_run_on_exceptions(self):
|
||||
calls = []
|
||||
record = await record_for(
|
||||
"crash", result_classifier=lambda t, r: calls.append(t) or "x"
|
||||
)
|
||||
assert calls == []
|
||||
assert (record.outcome, record.ok, record.error_type) == (
|
||||
"error",
|
||||
False,
|
||||
"RuntimeError",
|
||||
)
|
||||
assert record.error_message == "exploded"
|
||||
|
||||
async def test_mcp_error_result_without_raising(self):
|
||||
"""A CallToolResult with isError, on every FastMCP version."""
|
||||
record = await record_for("mcp_error")
|
||||
assert (record.outcome, record.ok) == ("soft_error", False)
|
||||
assert record.error_message == "isError: target missing"
|
||||
|
||||
@pytest.mark.skipif(not TOOL_RESULT_IS_ERROR, reason="ToolResult.is_error is 3.4+")
|
||||
async def test_tool_result_is_error(self):
|
||||
app = FastMCP("is_error")
|
||||
|
||||
@app.tool
|
||||
def lookup():
|
||||
return ToolResult(
|
||||
content=[TextContent(type="text", text="not found")], is_error=True
|
||||
)
|
||||
|
||||
sink = MemorySink()
|
||||
mw = instrument(app, [sink])
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("lookup", {}, raise_on_error=False)
|
||||
await mw.flush()
|
||||
[record] = sink.records
|
||||
assert (record.outcome, record.error_message) == (
|
||||
"soft_error",
|
||||
"isError: not found",
|
||||
)
|
||||
|
||||
def test_record_outcome_defaults_from_ok(self):
|
||||
from datetime import UTC, datetime
|
||||
|
||||
now = datetime.now(UTC)
|
||||
assert ToolCallRecord("t", now, 1.0, True, "meta").outcome == "ok"
|
||||
assert ToolCallRecord("t", now, 1.0, False, "meta").outcome == "error"
|
||||
|
||||
|
||||
class TestStorageAndCorrelation:
|
||||
async def test_database_sink_writes_outcome(self, db):
|
||||
sink = db()
|
||||
app, _, mw = make_app(sinks=[sink])
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("render", {"status": "done"})
|
||||
await client.call_tool("render", {"status": "failed"})
|
||||
with contextlib.suppress(ToolError):
|
||||
await client.call_tool("crash", {})
|
||||
await mw.flush()
|
||||
|
||||
t = sink.table
|
||||
async with sink.engine.connect() as conn:
|
||||
rows = (
|
||||
await conn.execute(
|
||||
select(t.c.outcome, t.c.ok, t.c.error_type).order_by(t.c.started_at)
|
||||
)
|
||||
).all()
|
||||
assert [tuple(r) for r in rows] == [
|
||||
("ok", True, None),
|
||||
("soft_error", False, "SoftError"),
|
||||
("error", False, "RuntimeError"),
|
||||
]
|
||||
async with sink.engine.connect() as conn:
|
||||
[(failing,)] = (
|
||||
await conn.execute(
|
||||
select(t.c.tool).where(~t.c.ok, t.c.outcome != "error")
|
||||
)
|
||||
).all()
|
||||
assert failing == "render"
|
||||
|
||||
async def test_links_and_feedback_context_show_soft_errors(self, db):
|
||||
app, _, mw = make_app(
|
||||
sinks=[db()], identity_resolver=lambda c: {"user_sub": "u1"}
|
||||
)
|
||||
captured = {}
|
||||
|
||||
@app.tool
|
||||
async def report() -> int:
|
||||
captured["links"] = await mw.capture_recent_calls()
|
||||
await mw.link_feedback("bug-1", captured["links"])
|
||||
return len(captured["links"])
|
||||
|
||||
async with Client(app) as client:
|
||||
await client.call_tool("render", {"status": "failed"})
|
||||
await client.call_tool("report", {})
|
||||
|
||||
links = captured["links"]
|
||||
assert [(link.tool, link.ok) for link in links] == [("render", False)]
|
||||
# Filled from memory or read from the database, depending on timing.
|
||||
[row] = await mw.feedback_context("bug-1")
|
||||
assert (row["outcome"], row["ok"], row["error_type"]) == (
|
||||
"soft_error",
|
||||
False,
|
||||
"SoftError",
|
||||
)
|
||||
await mw.flush()
|
||||
[row] = await mw.feedback_context("bug-1")
|
||||
assert (row["tool"], row["outcome"], row["ok"]) == (
|
||||
"render",
|
||||
"soft_error",
|
||||
False,
|
||||
)
|
||||
await mw.aclose()
|
||||
|
||||
async def test_submit_feedback_failure_is_soft_error(self, monkeypatch):
|
||||
def broken(request):
|
||||
raise RuntimeError("database down, password=hunter2")
|
||||
|
||||
monkeypatch.setattr(submission, "create_feedback_from_request", broken)
|
||||
app = FastMCP("fb")
|
||||
sink = MemorySink()
|
||||
mw = instrument(app, [sink])
|
||||
add_feedback_tools(app, database_url="sqlite:///:memory:", instrumentation=mw)
|
||||
|
||||
async with Client(app) as client:
|
||||
result = await client.call_tool(
|
||||
"submit_feedback",
|
||||
{
|
||||
"request": {
|
||||
"type": "bug",
|
||||
"title": "t",
|
||||
"description": "d",
|
||||
"submitter": "m",
|
||||
}
|
||||
},
|
||||
)
|
||||
await mw.flush()
|
||||
assert result.data["success"] is False
|
||||
[record] = sink.records
|
||||
assert (record.tool, record.outcome, record.ok, record.error_type) == (
|
||||
"submit_feedback",
|
||||
"soft_error",
|
||||
False,
|
||||
"SoftError",
|
||||
)
|
||||
assert record.error_message == (
|
||||
f"success=False: database down, password={REDACTED}"
|
||||
)
|
||||
+37
-36
@@ -7,6 +7,7 @@ from fastmcp import FastMCP
|
||||
from fastmcp_feedback.tools import add_feedback_tools
|
||||
from fastmcp_feedback.feedback import create_feedback_server
|
||||
from fastmcp_feedback.models import SubmitFeedbackRequest
|
||||
from tests._compat import tool_map
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@@ -18,7 +19,7 @@ class TestAddFeedbackTools:
|
||||
"""Test adding feedback tools with basic configuration."""
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
expected_tools = [
|
||||
@@ -40,7 +41,7 @@ class TestAddFeedbackTools:
|
||||
prefix="support"
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
expected_tools = [
|
||||
@@ -62,7 +63,7 @@ class TestAddFeedbackTools:
|
||||
insights=test_insights
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
|
||||
# Should have all standard tools - check tool names in the keys
|
||||
assert len([name for name in tools.keys() if "feedback" in name]) >= 5
|
||||
@@ -76,7 +77,7 @@ class TestAddFeedbackTools:
|
||||
# Should use default SQLite database
|
||||
add_feedback_tools(mock_fastmcp_server)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
assert "submit_feedback" in tool_names
|
||||
@@ -91,7 +92,7 @@ class TestAddFeedbackTools:
|
||||
separator="-"
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
assert "api-submit_feedback" in tool_names
|
||||
@@ -105,7 +106,7 @@ class TestAddFeedbackTools:
|
||||
# Second call with different prefix
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url, prefix="user")
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# Should have tools with both prefixes
|
||||
@@ -128,7 +129,7 @@ class TestCreateFeedbackServer:
|
||||
assert isinstance(server, FastMCP)
|
||||
assert server.name == "Test Feedback Server"
|
||||
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
expected_tools = [
|
||||
@@ -150,7 +151,7 @@ class TestCreateFeedbackServer:
|
||||
)
|
||||
|
||||
assert server.name == "Custom DB Server"
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
assert len(tools) >= 5
|
||||
|
||||
async def test_create_feedback_server_with_insights(self, test_insights):
|
||||
@@ -162,7 +163,7 @@ class TestCreateFeedbackServer:
|
||||
|
||||
assert server.name == "Analytics Server"
|
||||
# Verify insights are integrated
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
assert len(tools) >= 5
|
||||
|
||||
async def test_create_feedback_server_with_description(self):
|
||||
@@ -184,8 +185,8 @@ class TestCreateFeedbackServer:
|
||||
assert server1.name != server2.name
|
||||
|
||||
# Each should have independent tool sets
|
||||
tools1 = await server1.get_tools()
|
||||
tools2 = await server2.get_tools()
|
||||
tools1 = await tool_map(server1)
|
||||
tools2 = await tool_map(server2)
|
||||
|
||||
assert len(tools1) >= 5
|
||||
assert len(tools2) >= 5
|
||||
@@ -201,7 +202,7 @@ class TestToolIntegration:
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
# Get the submit_feedback tool
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
|
||||
assert submit_tool is not None
|
||||
@@ -218,7 +219,7 @@ class TestToolIntegration:
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
# Get the list_feedback tool
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
list_tool = tools["list_feedback"]
|
||||
|
||||
assert list_tool is not None
|
||||
@@ -234,7 +235,7 @@ class TestToolIntegration:
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
# Get the statistics tool
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
stats_tool = tools["get_feedback_statistics"]
|
||||
|
||||
assert stats_tool is not None
|
||||
@@ -250,7 +251,7 @@ class TestToolIntegration:
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
# Get the update status tool
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
update_tool = tools["update_feedback_status"]
|
||||
|
||||
assert update_tool is not None
|
||||
@@ -269,7 +270,7 @@ class TestToolIntegration:
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
# Get the delete tool
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
delete_tool = tools["delete_feedback"]
|
||||
|
||||
assert delete_tool is not None
|
||||
@@ -290,7 +291,7 @@ class TestToolValidation:
|
||||
"""Test submit_feedback tool parameter validation."""
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
|
||||
# Test with invalid parameters
|
||||
@@ -301,7 +302,7 @@ class TestToolValidation:
|
||||
"""Test list_feedback tool parameter validation."""
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
list_tool = tools["list_feedback"]
|
||||
|
||||
# Test with valid optional parameters
|
||||
@@ -318,7 +319,7 @@ class TestToolValidation:
|
||||
"""Test update_feedback_status parameter validation."""
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
update_tool = tools["update_feedback_status"]
|
||||
|
||||
# Test with missing required parameters
|
||||
@@ -329,7 +330,7 @@ class TestToolValidation:
|
||||
"""Test delete_feedback parameter validation."""
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
delete_tool = tools["delete_feedback"]
|
||||
|
||||
# Test with missing required parameters
|
||||
@@ -348,9 +349,9 @@ class TestServerComposition:
|
||||
feedback_server = create_feedback_server("Feedback Service")
|
||||
|
||||
# Import feedback server with prefix
|
||||
await main_server.import_server(feedback_server, prefix="feedback")
|
||||
main_server.mount(feedback_server, "feedback")
|
||||
|
||||
tools = await main_server.get_tools()
|
||||
tools = await tool_map(main_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# Should have feedback tools with prefix
|
||||
@@ -364,10 +365,10 @@ class TestServerComposition:
|
||||
user_feedback = create_feedback_server("User Feedback")
|
||||
admin_feedback = create_feedback_server("Admin Feedback")
|
||||
|
||||
await main_server.import_server(user_feedback, prefix="user")
|
||||
await main_server.import_server(admin_feedback, prefix="admin")
|
||||
main_server.mount(user_feedback, "user")
|
||||
main_server.mount(admin_feedback, "admin")
|
||||
|
||||
tools = await main_server.get_tools()
|
||||
tools = await tool_map(main_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# Should have tools from both servers
|
||||
@@ -386,9 +387,9 @@ class TestServerComposition:
|
||||
# Implementation would need to support this feature
|
||||
# FastMCP import_server doesn't support selective tool import
|
||||
# but we can test the full import with prefix
|
||||
await main_server.import_server(feedback_server, prefix="public")
|
||||
main_server.mount(feedback_server, "public")
|
||||
|
||||
tools = await main_server.get_tools()
|
||||
tools = await tool_map(main_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# All tools should be imported with prefix
|
||||
@@ -416,7 +417,7 @@ class TestToolPerformance:
|
||||
assert performance_timer.duration_ms < 1000 # 1 second
|
||||
|
||||
# Verify all tools were registered
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
assert len(tools) >= 5
|
||||
|
||||
async def test_multiple_server_creation_performance(self, performance_timer):
|
||||
@@ -436,14 +437,14 @@ class TestToolPerformance:
|
||||
|
||||
# Each server should be functional
|
||||
for server in servers:
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
assert len(tools) >= 5
|
||||
|
||||
async def test_tool_execution_performance(self, mock_fastmcp_server, memory_db_url, performance_timer):
|
||||
"""Test performance of tool execution."""
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
list_tool = tools["list_feedback"]
|
||||
|
||||
performance_timer.start()
|
||||
@@ -467,7 +468,7 @@ class TestToolConfiguration:
|
||||
"""Test that tools have proper descriptions."""
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
|
||||
for tool_name, tool_obj in tools.items():
|
||||
# Each tool should have a description
|
||||
@@ -480,7 +481,7 @@ class TestToolConfiguration:
|
||||
"""Test tool metadata and categorization."""
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
feedback_tools = [name for name in tools.keys() if "feedback" in name]
|
||||
|
||||
# Should have categorized the tools appropriately
|
||||
@@ -490,7 +491,7 @@ class TestToolConfiguration:
|
||||
"""Test tool parameter schemas."""
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
|
||||
# Tool should have parameter schema defined
|
||||
@@ -510,7 +511,7 @@ class TestToolErrorHandling:
|
||||
# Use invalid database URL
|
||||
add_feedback_tools(server, database_url="sqlite:///nonexistent/path/db.sqlite")
|
||||
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
list_tool = tools["list_feedback"]
|
||||
|
||||
# Tool execution should handle database errors gracefully
|
||||
@@ -523,7 +524,7 @@ class TestToolErrorHandling:
|
||||
"""Test tool handling of invalid parameters."""
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
|
||||
# Test with malformed request
|
||||
@@ -534,7 +535,7 @@ class TestToolErrorHandling:
|
||||
"""Test concurrent execution of tools."""
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
list_tool = tools["list_feedback"]
|
||||
|
||||
import asyncio
|
||||
|
||||
@@ -23,6 +23,7 @@ from fastmcp_feedback.tools import (
|
||||
)
|
||||
from fastmcp_feedback.database import FeedbackDatabase
|
||||
from fastmcp_feedback.insights import FeedbackInsights
|
||||
from tests._compat import tool_map
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@@ -34,7 +35,7 @@ class TestSubmissionTools:
|
||||
"""Test adding submission tools with basic configuration."""
|
||||
add_submission_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# Should have submission-related tools
|
||||
@@ -51,7 +52,7 @@ class TestSubmissionTools:
|
||||
prefix="user"
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
assert "user_submit_feedback" in tool_names
|
||||
@@ -65,14 +66,14 @@ class TestSubmissionTools:
|
||||
insights=test_insights
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
assert len(tools) > 0
|
||||
|
||||
async def test_add_submission_tools_without_database_url(self, mock_fastmcp_server):
|
||||
"""Test adding submission tools without database URL (uses default)."""
|
||||
add_submission_tools(mock_fastmcp_server)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
assert "submit_feedback" in tools
|
||||
|
||||
@patch('fastmcp_feedback.tools.FeedbackDatabase')
|
||||
@@ -105,7 +106,7 @@ class TestSubmissionTools:
|
||||
prefix="api_" # With trailing underscore
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# Should strip trailing underscore
|
||||
@@ -121,7 +122,7 @@ class TestRetrievalTools:
|
||||
"""Test adding retrieval tools with basic configuration."""
|
||||
add_retrieval_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# Should have retrieval-related tools
|
||||
@@ -139,7 +140,7 @@ class TestRetrievalTools:
|
||||
prefix="public"
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
assert "public_list_feedback" in tool_names
|
||||
@@ -154,14 +155,14 @@ class TestRetrievalTools:
|
||||
insights=test_insights
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
assert len(tools) > 0
|
||||
|
||||
async def test_add_retrieval_tools_without_database_url(self, mock_fastmcp_server):
|
||||
"""Test adding retrieval tools without database URL (uses default)."""
|
||||
add_retrieval_tools(mock_fastmcp_server)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
assert "list_feedback" in tools
|
||||
assert "get_feedback_statistics" in tools
|
||||
|
||||
@@ -197,7 +198,7 @@ class TestManagementTools:
|
||||
"""Test adding management tools with basic configuration."""
|
||||
add_management_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# Should have management-related tools
|
||||
@@ -215,7 +216,7 @@ class TestManagementTools:
|
||||
prefix="admin"
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
assert "admin_update_feedback_status" in tool_names
|
||||
@@ -230,14 +231,14 @@ class TestManagementTools:
|
||||
insights=test_insights
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
assert len(tools) > 0
|
||||
|
||||
async def test_add_management_tools_without_database_url(self, mock_fastmcp_server):
|
||||
"""Test adding management tools without database URL (uses default)."""
|
||||
add_management_tools(mock_fastmcp_server)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
assert "update_feedback_status" in tools
|
||||
assert "delete_feedback" in tools
|
||||
|
||||
@@ -347,7 +348,7 @@ class TestAdvancedIntegrationPatterns:
|
||||
|
||||
create_multi_tenant_tools(mock_fastmcp_server, tenants)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# Should have tools for each tenant with prefixes
|
||||
@@ -365,7 +366,7 @@ class TestAdvancedIntegrationPatterns:
|
||||
|
||||
create_multi_tenant_tools(mock_fastmcp_server, tenants, insights=test_insights)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
assert len(tools) >= 10 # At least 5 tools per tenant
|
||||
|
||||
async def test_create_multi_tenant_tools_empty_tenants(self, mock_fastmcp_server):
|
||||
@@ -374,7 +375,7 @@ class TestAdvancedIntegrationPatterns:
|
||||
|
||||
create_multi_tenant_tools(mock_fastmcp_server, tenants)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
# Should complete without error but not add any tools
|
||||
assert len(tools) == 0
|
||||
|
||||
@@ -394,7 +395,7 @@ class TestAdvancedIntegrationPatterns:
|
||||
"""Test creating role-based tools."""
|
||||
create_role_based_tools(mock_fastmcp_server, database_url=memory_db_url)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# Should have public tools (submission and retrieval)
|
||||
@@ -416,14 +417,14 @@ class TestAdvancedIntegrationPatterns:
|
||||
insights=test_insights
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
assert len(tools) >= 7 # Public + admin tools
|
||||
|
||||
async def test_create_role_based_tools_without_database(self, mock_fastmcp_server):
|
||||
"""Test creating role-based tools without database URL."""
|
||||
create_role_based_tools(mock_fastmcp_server)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
assert "public_submit_feedback" in tool_names
|
||||
@@ -465,7 +466,7 @@ class TestMigrationHelpers:
|
||||
mock_fastmcp_server.remove_tool.assert_any_call("old_list_tool")
|
||||
|
||||
# Should have new tools added
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
assert "submit_feedback" in tool_names
|
||||
assert "list_feedback" in tool_names
|
||||
@@ -501,7 +502,7 @@ class TestMigrationHelpers:
|
||||
insights=test_insights
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
assert len(tools) >= 5
|
||||
|
||||
async def test_migrate_tools_to_mixins_empty_list(self, mock_fastmcp_server, memory_db_url):
|
||||
@@ -515,7 +516,7 @@ class TestMigrationHelpers:
|
||||
)
|
||||
|
||||
# Should still add new tools
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
assert len(tools) >= 5
|
||||
|
||||
@patch('fastmcp_feedback.tools.add_feedback_tools')
|
||||
@@ -757,7 +758,7 @@ class TestErrorHandlingAndEdgeCases:
|
||||
|
||||
# Verify all servers have tools
|
||||
for server in servers:
|
||||
tools = await server.get_tools()
|
||||
tools = await tool_map(server)
|
||||
assert "submit_feedback" in tools
|
||||
assert "public_list_feedback" in tools
|
||||
assert "admin_update_feedback_status" in tools
|
||||
@@ -774,7 +775,7 @@ class TestErrorHandlingAndEdgeCases:
|
||||
# Multiple underscores
|
||||
add_management_tools(mock_fastmcp_server, database_url=memory_db_url, prefix="admin___")
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# Should handle edge cases gracefully
|
||||
@@ -812,7 +813,7 @@ class TestIntegrationWithExistingTests:
|
||||
# Add all tools
|
||||
add_feedback_tools(mock_fastmcp_server, database_url=memory_db_url, prefix="complete")
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# Should have both individual and complete sets
|
||||
@@ -830,7 +831,7 @@ class TestIntegrationWithExistingTests:
|
||||
insights=test_insights
|
||||
)
|
||||
|
||||
tools = await mock_fastmcp_server.get_tools()
|
||||
tools = await tool_map(mock_fastmcp_server)
|
||||
submit_tool = tools["submit_feedback"]
|
||||
|
||||
# Should work with existing sample data
|
||||
@@ -848,9 +849,9 @@ class TestIntegrationWithExistingTests:
|
||||
add_retrieval_tools(feedback_server, prefix="readonly")
|
||||
|
||||
# Import with prefix
|
||||
await main_server.import_server(feedback_server, prefix="external")
|
||||
main_server.mount(feedback_server, "external")
|
||||
|
||||
tools = await main_server.get_tools()
|
||||
tools = await tool_map(main_server)
|
||||
tool_names = list(tools.keys())
|
||||
|
||||
# Should have properly composed tool names
|
||||
|
||||
Reference in New Issue
Block a user