Release 2026.10.01.4: statistics labels, event identity, quiet logging

README and CONTRIBUTING rewritten where they described the old behavior
or code that does not exist: the example test and mixin recipe now run,
MySQL is marked untested, the two Postgres driver paths are spelled out,
and the coverage figure is the measured 95%.
This commit is contained in:
2026-10-01 22:48:46 -06:00
parent c6c928d528
commit 7892a5d79f
8 changed files with 268 additions and 95 deletions
+57
View File
@@ -5,6 +5,63 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Calendar Versioning](https://calver.org/) (YYYY.MM.DD).
## [2026.10.01.4] - 2026-10-01
Statistics speak the same enum values as the rest of the API, events carry
the caller's identity, and importing the package no longer configures
logging.
### Changed
- `get_feedback_statistics` and `get_database_stats` key `by_type` and
`by_status` by enum value (`bug`, `in_progress`), the spelling
`list_feedback` returns and accepts as a filter. They grouped the raw
columns, which hold the enum NAMES on both SQLite (VARCHAR) and Postgres
(native enum types `feedbacktype`, `feedbackstatus`), so they returned
`BUG`, `IN_PROGRESS`. **Code reading the uppercase keys must switch to
lowercase.** Nothing stored changes. A row holding a value instead of a
name is counted under the same key; an unknown label is kept as-is.
- The identity resolver runs once per call **before** the tool, instead of
after it. The call's record reuses the result (still redacted, still
limited by `hook_timeout`, still never raising). An async resolver can
therefore delay the tool by up to `hook_timeout`, and it now also runs for
ok calls that sampling drops afterwards. It does not run in mode `off` or
for `exclude_tools`. The enricher still runs after the call.
- `PACKAGE_INFO` is read from the installed distribution's metadata. It
listed only the `postgresql` and `dev` extras and a classifier pyproject
did not have.
- psycopg2-binary is in the `dev` and `test` extras, so `make test-pg`
covers the feedback tools on Postgres too.
### Fixed
- `record_event` (and `record_llm_call`) inside a tool call never set
`user_sub`, `caller_kind` or `client_id`, because identity was resolved
only after the call. Inside a call they now come from the call's resolved
identity, matching its row; `client_id` falls back to the FastMCP context's
as for calls. Values passed explicitly win, and `record_event` takes
`caller_kind=` and `client_id=` as well as `user_sub=`. Outside a call,
including a task the call started that records after it returned, nothing
is filled in.
- Importing `fastmcp_feedback` called `logging.basicConfig` at INFO when the
root logger had no handlers, so every host process (the backfill CLI
included) printed an "initialized" line and had its root logger set up by
a library. The package now only adds a `NullHandler` to the
`fastmcp_feedback` logger and logs the version at DEBUG. **Servers that
relied on the package's INFO or WARNING lines must configure logging
themselves**; `fastmcp_feedback.setup_logging()` restores the old
behavior in one line.
- The backfill CLI logs a summary of its counts at INFO, so `-v` shows
something on a dry run.
### Documentation
- README: statistics keys, identity resolver timing, a Logging section, the
two Postgres driver paths (`[postgresql]` for the feedback tools,
`[instrumentation]` plus asyncpg for the sinks), MySQL marked untested,
and the measured coverage (95% lines and branches with the Postgres cases
enabled) in place of "98%".
- CONTRIBUTING: the example test, the mixin recipe and the commands now
match the package and were run; `make test`, `make test-pg` and
`make lint` replace commands that did not pass on this repository.
## [2026.10.01.3] - 2026-10-01
### Documentation
+119 -73
View File
@@ -1,6 +1,6 @@
# Contributing to FastMCP Feedback
Thank you for your interest in contributing! This guide will help you get started.
How to set up, test and submit changes.
## Development Setup
@@ -13,17 +13,16 @@ Thank you for your interest in contributing! This guide will help you get starte
### Getting Started
```bash
# 1. Fork the repository on git.supported.systems
# 1. Fork fastmcp-feedback/fastmcp-feedback on git.supported.systems
# 2. Clone your fork
git clone git@git.supported.systems:YOUR_USERNAME/fastmcp-feedback.git
cd fastmcp-feedback
# 3. Create virtual environment and install dependencies
uv venv
uv pip install -e ".[dev]"
# 3. Create .venv from uv.lock with the dev extra
uv sync --extra dev
# 4. Verify setup
uv run pytest
# 4. Verify setup (SQLite only; Postgres cases skip)
make test
```
### Development Workflow
@@ -35,17 +34,14 @@ git checkout -b feature/your-feature-name
# Make your changes...
# Run tests
uv run pytest
make test
# Run linting
uv run ruff check .
uv run ruff format .
# Run type checking (optional but appreciated)
uv run mypy src/fastmcp_feedback
make lint
uv run ruff format --check src/
# Commit with clear message
git commit -m "feat: Add support for custom validators"
git commit -m "Count feedback by submitter in get_feedback_statistics"
# Push and create PR
git push origin feature/your-feature-name
@@ -56,19 +52,21 @@ git push origin feature/your-feature-name
We use [Ruff](https://docs.astral.sh/ruff/) for linting and formatting:
```bash
# Check for issues
uv run ruff check .
# Lint src/, examples/ and the test files that are kept ruff-clean
make lint
# Auto-fix issues
uv run ruff check --fix .
# Format code
uv run ruff format .
# Format what you changed
uv run ruff format src/ path/to/your_test.py
```
Many of the older test files predate the ruff configuration and do not pass
it, so `ruff check .` over the whole repository reports errors that are not
yours. `src/` must stay clean, and new test files should be added to the
`lint` target in the Makefile.
### Style Guidelines
- **Line length**: 88 characters (Black default)
- **Line length**: 88 characters
- **Imports**: Sorted with isort (via Ruff)
- **Type hints**: Required for public APIs
- **Docstrings**: Google style for public functions
@@ -78,11 +76,11 @@ uv run ruff format .
### Running Tests
```bash
# Run all tests
# Run all tests (coverage is on by default, see pyproject.toml)
uv run pytest
# Run with coverage
uv run pytest --cov
# Without the coverage report, as the Makefile does
uv run pytest -q --no-cov
# Run specific test file
uv run pytest tests/unit/test_tools.py
@@ -90,11 +88,8 @@ uv run pytest tests/unit/test_tools.py
# Run specific test
uv run pytest -k "test_submit_feedback"
# Run only unit tests
uv run pytest -m unit
# Run only integration tests
uv run pytest -m integration
# Run one directory
uv run pytest tests/integration
```
### Running the tests against Postgres
@@ -118,7 +113,10 @@ shell on it. Every Postgres test gets its own freshly created database with
the `vector` extension, dropped when the test ends, so tests never see each
other's rows. To write a test that runs on both backends, take the `db_url`
fixture from `tests/conftest.py`: `db_url("name")` returns an async URL for
an empty database, the same one for the same name within a test.
an empty database, the same one for the same name within a test. The
feedback tools use a synchronous engine, so convert that URL's driver
(`sqlite`, `postgresql+psycopg2`) as the `feedback_db` fixture in
`tests/unit/test_consistency.py` does; psycopg2 comes with the dev extra.
To point the suite at another server, set the variable yourself; the user
needs permission to create and drop databases:
@@ -148,44 +146,60 @@ FFB_TEST_EMBED_URL=https://api.example.com/v1 FFB_TEST_EMBED_KEY=... \
- Use descriptive test names: `test_submit_feedback_with_invalid_type_raises_error`
- Use fixtures from `tests/conftest.py`
Example test:
Tests are async without a marker (`asyncio_mode = "auto"`). Call tools the
way a client does, through `fastmcp.Client`. `submit_feedback` takes one
argument, `request`, holding the fields of `SubmitFeedbackRequest`; `type`
is one of `bug`, `feature`, `improvement` or `question`. Tools report
failure in the result (`success: False` and an `error`) rather than raising.
```python
import pytest
from fastmcp import Client, FastMCP
from fastmcp_feedback import add_feedback_tools
@pytest.mark.unit
async def test_submit_feedback_creates_record(app_with_feedback, db_session):
"""Submitting feedback should create a database record."""
result = await app_with_feedback.call_tool(
"submit_feedback",
type="bug_report",
title="Test Bug",
description="Description"
)
assert result["status"] == "success"
assert result["id"] is not None
async def test_submit_feedback_creates_record(tmp_path):
app = FastMCP("test")
add_feedback_tools(app, database_url=f"sqlite:///{tmp_path}/feedback.db")
async with Client(app) as client:
result = await client.call_tool(
"submit_feedback",
{
"request": {
"type": "bug",
"title": "Export fails",
"description": "Exporting a scene with no objects raises",
"submitter": "test",
}
},
)
listed = await client.call_tool("list_feedback", {})
stats = await client.call_tool("get_feedback_statistics", {})
assert result.structured_content["success"] is True
[item] = listed.structured_content["feedback"]
assert item["id"] == result.structured_content["feedback_id"]
assert (item["type"], item["status"]) == ("bug", "open")
assert stats.structured_content["by_type"] == {"bug": 1}
```
## Pull Request Guidelines
### Before Submitting
1. **Tests pass**: `uv run pytest`
2. **Linting passes**: `uv run ruff check .`
1. **Tests pass**: `make test`, and `make test-pg` when you touched database code
2. **Linting passes**: `make lint`
3. **New features have tests**
4. **Documentation updated** (if applicable)
### PR Title Format
Use [Conventional Commits](https://www.conventionalcommits.org/):
Say what changes, in the imperative, as the commit log does:
- `feat: Add custom validation support`
- `fix: Handle empty feedback list correctly`
- `docs: Update installation instructions`
- `test: Add edge case tests for insights`
- `refactor: Simplify database connection logic`
- `Key feedback statistics by enum value`
- `Resolve caller identity before the tool runs`
- `Import instrumentation without the database extras`
### PR Description Template
@@ -215,16 +229,20 @@ fastmcp-feedback/
│ ├── models.py # Pydantic models and SQLAlchemy ORM
│ ├── database.py # Database connection management
│ ├── feedback.py # Standalone server factory
│ ├── insights.py # Privacy-compliant analytics
│ └── mixins/ # Modular tool mixins
│ ├── base.py
│ ├── submission.py
│ ├── retrieval.py
│ └── management.py
│ ├── insights.py # In-process usage metrics
│ ├── mixins/ # The feedback tools, grouped
│ │ ├── base.py # BaseFeedbackMixin
│ │ ├── submission.py
│ │ ├── retrieval.py
│ │ └── management.py
│ └── instrumentation/ # Middleware, sinks, events, embeddings, backfill
├── tests/
│ ├── conftest.py # Shared fixtures
│ ├── conftest.py # Shared fixtures, including db_backend and db_url
│ ├── _databases.py # Per-test Postgres databases
│ ├── unit/ # Unit tests
│ └── integration/ # Integration tests
├── docker-compose.yml # Postgres + pgvector for make test-pg
├── Makefile
├── pyproject.toml # Package configuration
├── README.md
├── CONTRIBUTING.md # This file
@@ -236,28 +254,56 @@ fastmcp-feedback/
### Adding a New Mixin
1. Create `src/fastmcp_feedback/mixins/your_mixin.py`
2. Inherit from `BaseMixin`
3. Implement `register_tools(app, prefix="")`
4. Export from `src/fastmcp_feedback/mixins/__init__.py`
5. Add to main `__init__.py` exports
6. Write tests in `tests/unit/test_your_mixin.py`
7. Update documentation
A mixin subclasses `BaseFeedbackMixin`, which holds the `FeedbackDatabase`
(`self.database`) and the `FeedbackInsights` (`self.insights`). Its tools are
methods decorated with `mcp_tool` from `fastmcp.contrib.mcp_mixin`;
`register_tools(app, prefix=...)`, inherited from FastMCP's `MCPMixin`, adds
them to a server.
```python
from fastmcp import FastMCP
from fastmcp_feedback import FeedbackDatabase
from fastmcp_feedback.mixins.base import BaseFeedbackMixin, Feedback, mcp_tool
class CountMixin(BaseFeedbackMixin):
@mcp_tool(description="Count feedback items")
async def count_feedback(self) -> dict:
async with self.database.session() as session:
return {"count": session.query(Feedback).count()}
app = FastMCP("demo")
CountMixin(FeedbackDatabase("sqlite:///:memory:")).register_tools(app, prefix="admin")
# The tool is registered as admin_count_feedback.
```
Then:
1. Put it in `src/fastmcp_feedback/mixins/your_mixin.py`
2. Export it from `src/fastmcp_feedback/mixins/__init__.py` and the package `__init__.py`
3. Write tests in `tests/unit/`, calling the tool through `fastmcp.Client`
4. Document it in README.md and on the docs site
### Adding a New Tool
1. Add to appropriate mixin or create new one
2. Use `@app.tool()` decorator with clear description
3. Add Pydantic models for request/response if complex
4. Write unit tests
5. Add integration test
6. Document in README
1. Add a method to the mixin it belongs to, decorated with `mcp_tool` and a
description written for the model that will call it
2. Return a dict; report failures as `{"success": False, "error": ...}`
instead of raising
3. Return enum fields as their values (`bug`, `in_progress`), as
`feedback_to_dict` does; the database stores the names
4. Use a Pydantic model for structured input
5. Test it on both backends where it touches the database (`make test-pg`)
## Versioning
We use **calendar versioning** (CalVer): `YYYY.MM.DD`
Version bumps happen automatically on release. Contributors don't need to update version numbers.
A release bumps the version in `pyproject.toml` and
`src/fastmcp_feedback/__init__.py` and adds a CHANGELOG entry. Contributors
don't need to update version numbers.
## Getting Help
+2 -1
View File
@@ -60,4 +60,5 @@ test-all: test test-pg
lint:
uv run --no-sync ruff check src/ examples/ tests/_compat.py tests/_databases.py \
tests/unit/test_test_databases.py tests/unit/test_instrumentation.py \
tests/unit/test_embeddings.py tests/unit/test_backfill.py
tests/unit/test_embeddings.py tests/unit/test_backfill.py \
tests/unit/test_consistency.py
+83 -17
View File
@@ -10,10 +10,10 @@ Record every tool call your server handles, and let people and models report pro
- **Secret Redaction** - Tokens and passwords removed from recorded arguments, results and errors by default
- **One-Line Integration** - Add feedback tools to any FastMCP server instantly
- **Modular Mixins** - Selective tool registration for fine-grained control
- **Multi-Database Support** - SQLite (dev), PostgreSQL, MySQL (production)
- **SQLite and PostgreSQL** - Both covered by the test suite. A `mysql` extra exists but MySQL is untested
- **Privacy-Compliant Analytics** - Optional usage insights without sensitive data
- **Type-Safe** - Full Pydantic validation and type hints
- **Well-Tested** - 98% coverage, run against FastMCP 2.12, 2.14, 3.x and 4.x
- **Tested** - 95% line and branch coverage with the Postgres cases enabled (measured for 2026.10.01.4), run against FastMCP 2.12, 2.14, 3.x and 4.x
## Installation
@@ -26,8 +26,11 @@ uv add fastmcp-feedback
# Using pip
pip install fastmcp-feedback
# With PostgreSQL support
# Feedback tools on PostgreSQL (psycopg2, synchronous)
uv add "fastmcp-feedback[postgresql]"
# Instrumentation's DatabaseSink on PostgreSQL (async)
uv add "fastmcp-feedback[instrumentation]" asyncpg
```
What each install covers:
@@ -38,7 +41,11 @@ What each install covers:
| `fastmcp-feedback[instrumentation]` | `sqlalchemy[asyncio]` (greenlet), `aiosqlite` | `DatabaseSink`, `build_metadata`, call history for linking feedback (`capture_recent_calls` across workers, `link_feedback`, `feedback_context`) |
| `fastmcp-feedback[instrumentation,embeddings]` | `pgvector` | `EmbeddingSink`, `OpenAIEmbedder`, `similar()`, the backfill helpers; pgvector is only needed on PostgreSQL |
Postgres hosts add `asyncpg` for the async sinks. Without the
The two database paths use different drivers. The feedback tools run a
synchronous engine, so on Postgres they need `[postgresql]` (psycopg2) and a
`postgresql://` URL. `DatabaseSink` and `EmbeddingSink` run an async engine,
so on Postgres they need `[instrumentation]` plus `asyncpg` and a
`postgresql+asyncpg://` URL. A server that does both installs both. Without the
`instrumentation` extra, `from fastmcp_feedback.instrumentation import
DatabaseSink` raises an `ImportError` that names the missing package and the
extra to install, and the middleware's database helpers return their empty
@@ -96,13 +103,37 @@ add_feedback_tools(app, database_url="sqlite:///feedback.db")
Pass a `database_url`. Without one the tools use an in-memory SQLite
database, which is handy for tests but loses everything when the process exits.
`get_feedback_statistics` keys its counts with the same lowercase values
`list_feedback` returns, so a key can be passed straight back as a filter:
```python
import asyncio
from fastmcp_feedback import FeedbackDatabase, RetrievalMixin, SubmissionMixin
from fastmcp_feedback import SubmitFeedbackRequest
async def main():
db = FeedbackDatabase("sqlite:///:memory:")
await db.initialize()
await SubmissionMixin(db).submit_feedback(SubmitFeedbackRequest(
type="bug", title="Export fails", description="...", submitter="cli"))
stats = await RetrievalMixin(db).get_feedback_statistics()
print(stats["by_type"], stats["by_status"]) # {'bug': 1} {'open': 1}
asyncio.run(main())
```
Before 2026.10.01.4 these keys were the uppercase names the database stores
(`BUG`, `OPEN`). Code that reads them needs the lowercase keys now.
### PostgreSQL
```python
add_feedback_tools(app, database_url="postgresql://user:pass@db.example.com/feedback")
```
Install the driver with `uv add "fastmcp-feedback[postgresql]"`.
Install the driver with `uv add "fastmcp-feedback[postgresql]"`. This is
psycopg2 for the feedback tools' synchronous engine; the instrumentation
sinks use asyncpg instead (see [Installation](#installation)).
### Prefixed Tool Names
@@ -351,6 +382,16 @@ go in `identity`. Enricher output goes in `extra`, except `server_version`,
which has its own column. Async hooks that take longer than `hook_timeout`
(0.25 s by default) are skipped for that call.
The identity resolver runs once per call, **before** the tool. Events the
tool records get the same `user_sub`, `caller_kind` and `client_id` as the
call's row, and the call's record reuses the result instead of asking again.
So keep it cheap: an async resolver can hold the tool back by up to
`hook_timeout`, and it also runs for calls that sampling later drops (sampling
cannot know beforehand whether a call will fail). It does not run in mode
`off` or for `exclude_tools`. The enricher still runs after the call, since it
receives the result. Before 2026.10.01.4 the resolver ran after the call and
events never carried identity.
### Linking feedback to the calls behind it
When a model reports "the render tool hung", the report is far more useful with
@@ -504,8 +545,8 @@ async def submit(case: str) -> dict:
mw.record_event("job.finished", key="job-wing", attrs={"duration_s": 3512, "converged": True})
```
`record_event(kind, *, key=None, attrs=None, user_sub=None, call_id=None)`
returns True when the event was queued and False when it was not. It never
`record_event(kind, *, key=None, attrs=None, user_sub=None, call_id=None,
caller_kind=None, client_id=None)` returns True when the event was queued and False when it was not. It never
raises and never blocks, and it takes the same path as call records: the same
background queue, the same drop-and-count policy when the queue is full.
@@ -522,10 +563,13 @@ background queue, the same drop-and-count policy when the queue is full.
JSON-safe: `NaN` and infinities become the strings `"nan"`/`"inf"` (Postgres
JSONB rejects them), datetimes become ISO strings, anything else unknown
becomes `str(value)`.
- **Inside a tool call**, the event gets that call's id as `call_id` and its
session, so it joins to the call's row in `ffb_tool_calls`. Outside a call,
`call_id` is None. A task started by a tool that records after the tool has
returned counts as outside. Pass `call_id=` to set it yourself.
- **Inside a tool call**, the event gets that call's id as `call_id`, its
session, and the `user_sub`, `caller_kind` and `client_id` the identity
resolver returned for the call, so it joins to the call's row in
`ffb_tool_calls`. Outside a call, those are None unless you pass them. A
task started by a tool that records after the tool has returned counts as
outside. Any of `call_id=`, `user_sub=`, `caller_kind=` and `client_id=`
passed explicitly wins over the call's value.
- **From any thread.** Called off the server's event loop (a watcher thread, a
sync tool running in a worker thread), the event is handed to the loop with
`call_soon_threadsafe`. The loop is known once the server has handled a tool
@@ -862,7 +906,8 @@ python -m fastmcp_feedback.instrumentation.backfill \
```
It prints the counts as JSON and exits 1 if any text failed to embed, 2 on
a setup error. `--create-tables` creates `ffb_embeddings` (and on PostgreSQL
a setup error. It logs warnings to stderr; `-v` adds INFO lines, including
a summary of the counts. `--create-tables` creates `ffb_embeddings` (and on PostgreSQL
the vector extension) when it is missing; without it a missing table is an
error. `--help` lists the rest (`--until`, `--limit`, `--batch-size`,
`--max-chars`, `--event-text-keys`).
@@ -874,6 +919,28 @@ before adding other middleware to include their time and record calls they
reject; call it last to time only the tool. On shutdown, `await mw.aclose()`
flushes queued records and closes the sinks.
## Logging
The package configures no logging. Its loggers sit under `fastmcp_feedback`
with a `NullHandler`, so importing it prints nothing. The middleware never
raises into your tools; when a sink, hook or redactor fails it logs a
WARNING instead, and those warnings only appear once your server configures
logging:
```python
import logging
logging.basicConfig(level=logging.WARNING)
logging.getLogger("fastmcp_feedback").setLevel(logging.INFO) # optional: more detail
```
At INFO the package also reports pruning, table-creation retries and the
servers and tools it sets up. `fastmcp_feedback.setup_logging()` is a
one-line alternative: `basicConfig` at INFO with a timestamped format, then
the package version. Before
2026.10.01.4, importing the package called it automatically whenever the root
logger had no handlers, which also changed the logging of the host process.
## Advanced Usage
### Mixin Architecture
@@ -993,7 +1060,7 @@ uv run python examples/simple_integration.py
## Contributing
We welcome contributions! FastMCP Feedback is designed to be contributor-friendly.
Contributions are welcome.
### Quick Contribution Setup
@@ -1003,15 +1070,14 @@ git clone git@git.supported.systems:YOUR_USERNAME/fastmcp-feedback.git
cd fastmcp-feedback
# Install with dev dependencies
uv venv
uv pip install -e ".[dev]"
uv sync --extra dev
# Create feature branch
git checkout -b feature/your-feature
# Make changes, run tests
uv run pytest
uv run ruff check .
make test
make lint
# Submit PR
```
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "fastmcp-feedback"
version = "2026.10.01.3" # YYYYMMDD.patch versioning - update for each release
version = "2026.10.01.4" # YYYYMMDD.patch versioning - update for each release
description = "Production-ready feedback collection system for FastMCP servers"
readme = "README.md"
license = "MIT"
+1 -1
View File
@@ -41,7 +41,7 @@ Quick Start:
import logging
# Version information - CalVer YYYY.MM.DD
__version__ = "2026.10.01.3"
__version__ = "2026.10.01.4"
__author__ = "Ryan Malloy"
__email__ = "ryan@supported.systems"
+4 -1
View File
@@ -1,4 +1,4 @@
"""Statistics labels, event identity and logging, as fixed in 2026.10.02.
"""Statistics labels, event identity and logging, as fixed in 2026.10.01.4.
Statistics run on SQLite and, with FFB_TEST_DATABASE_URL set, on Postgres
through the feedback tools' own synchronous engine (psycopg2).
@@ -407,6 +407,9 @@ def run_python(code, *args, env=None):
class TestLogging:
def test_import_configures_nothing_and_prints_nothing(self):
proc = run_python(
# FastMCP 2.x and 3.0 reset warning filters on import and authlib
# then warns; drop warnings so stderr holds only logging output.
"import warnings\nwarnings.showwarning = lambda *a, **k: None\n"
"import logging, fastmcp_feedback, fastmcp_feedback.instrumentation\n"
"root = logging.getLogger()\n"
"print(len(root.handlers), root.level)\n"
Generated
+1 -1
View File
@@ -601,7 +601,7 @@ wheels = [
[[package]]
name = "fastmcp-feedback"
version = "2026.10.1.3"
version = "2026.10.1.4"
source = { editable = "." }
dependencies = [
{ name = "fastmcp" },