From 7ffc1481121c2c3db92189b971f6751ed12c0720 Mon Sep 17 00:00:00 2001 From: Ryan Malloy Date: Wed, 2 Sep 2026 10:36:44 -0600 Subject: [PATCH] Docs: cover the 2026.09.02 review, and drop every em-dash Content. The types page now warns about the two framing bugs this release fixed, since both silently corrupt data on versions before it: a BLOB or CLOB in any position but last shifted every column after it, and a NULL collection did the same. The API page documents two things that are now user-visible: only one scrollable cursor may be open per connection (previously the second statement drew -285 and destroyed the cursor as collateral), and transaction control written as SQL is tracked (previously rollback() after a SQL BEGIN WORK sent nothing and reported success). The phase log gains a section on the review itself, including the pattern behind almost all eleven bugs. Test counts updated to 457/456. Style. 136 em-dashes across 22 content files, plus Hero.astro and two stylesheets, rewritten rather than substituted -- swapping the character for a comma leaves prose that reads like it lost an argument with a linter. Bullets that used a dash to gloss a term now use a colon; parenthetical asides became their own sentences or moved inside brackets. The rendered HTML is clean. Verified against the live site by content, not status code: these pages return 200 for every path and render the 404 body, so a stale deploy looks perfectly healthy. --- docs-site/src/components/Hero.astro | 2 +- .../src/content/docs/explain/architecture.md | 10 ++-- .../content/docs/explain/async-strategy.mdx | 8 ++-- .../content/docs/explain/buffered-reader.mdx | 26 +++++----- .../src/content/docs/explain/phase-log.md | 47 +++++++++++++++---- .../src/content/docs/explain/pure-python.md | 20 ++++---- .../content/docs/explain/sqli-protocol.mdx | 10 ++-- .../src/content/docs/how-to/async-fastapi.mdx | 4 +- .../content/docs/how-to/buffered-reader.md | 6 +-- .../src/content/docs/how-to/dev-container.mdx | 12 ++--- .../src/content/docs/how-to/executemany.mdx | 8 ++-- .../content/docs/how-to/migrate-from-ifxpy.md | 8 ++-- docs-site/src/content/docs/how-to/pool.mdx | 12 ++--- .../src/content/docs/how-to/smart-lobs.mdx | 2 +- docs-site/src/content/docs/how-to/tls.mdx | 6 +-- docs-site/src/content/docs/index.mdx | 12 ++--- docs-site/src/content/docs/reference/api.md | 33 +++++++++++-- .../src/content/docs/reference/benchmarks.mdx | 2 +- .../src/content/docs/reference/config.md | 4 +- docs-site/src/content/docs/reference/types.md | 32 +++++++------ .../src/content/docs/start/quickstart.mdx | 10 ++-- docs-site/src/content/docs/start/vs-ifxpy.mdx | 22 ++++----- docs-site/src/content/docs/start/wtf.md | 18 +++---- docs-site/src/styles/components.css | 4 +- docs-site/src/styles/theme.css | 8 ++-- 25 files changed, 190 insertions(+), 136 deletions(-) diff --git a/docs-site/src/components/Hero.astro b/docs-site/src/components/Hero.astro index dc2b1f3..5e11b29 100644 --- a/docs-site/src/components/Hero.astro +++ b/docs-site/src/components/Hero.astro @@ -13,7 +13,7 @@

Every other Informix driver wraps IBM's C SDK or the JDBC JAR. We weren't into that. - So we read the protocol and wrote it ourselves — PEP 249, sync + async, pooled, TLS. + So we read the protocol and wrote it ourselves: PEP 249, sync and async, pooled, TLS. Within 10% of IBM's own C driver on bulk fetches, 1.6× faster on bulk inserts. No compile step. No LD_LIBRARY_PATH ritual. No libcrypt.so.1 from 2018. diff --git a/docs-site/src/content/docs/explain/architecture.md b/docs-site/src/content/docs/explain/architecture.md index 67950d5..6db37b9 100644 --- a/docs-site/src/content/docs/explain/architecture.md +++ b/docs-site/src/content/docs/explain/architecture.md @@ -1,6 +1,6 @@ --- title: Architecture overview -description: How the layers stack — socket, framing, codec, resultset, cursor, connection, pool. +description: How the layers stack, from socket through framing, codec, resultset, cursor, connection, and pool. sidebar: order: 2 --- @@ -28,13 +28,13 @@ The driver is six layers, each with a single responsibility, each testable in is The lowest layer. Wraps `socket.socket` with a connection-scoped read buffer (Phase 39). One `recv(64K)` per ~64 KB of incoming data; parsers read into the buffer via `struct.unpack_from(buf, offset)` rather than slicing copies. -Everything above this layer is `bytes` and `bytearray` arithmetic — no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`. +Everything above this layer is `bytes` and `bytearray` arithmetic, with no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`. See [The buffered reader →](/explain/buffered-reader/) for why the buffer lives here and not on the parser. ## Protocol / PDU framing -`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean* — only how to read and write the byte shapes. +`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean*, only how to read and write the byte shapes. The PDU types and their fields were reverse-engineered from three sources: @@ -44,9 +44,9 @@ The PDU types and their fields were reverse-engineered from three sources: ## Codec / Per-column readers -`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types — see [SQL ↔ Python types](/reference/types/) for the full table. +`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types. See [SQL ↔ Python types](/reference/types/) for the full table. -Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]` — no per-column dispatch overhead. +Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]`, with no per-column dispatch overhead. Phase 38 went further with `exec()`-based codegen: for the hottest tables, the driver generates a flat decoder function with all readers inlined and dispatch decisions baked in. The generated function is the equivalent of unrolling the per-column dispatch into straight-line code. diff --git a/docs-site/src/content/docs/explain/async-strategy.mdx b/docs-site/src/content/docs/explain/async-strategy.mdx index cb6a75d..15036bc 100644 --- a/docs-site/src/content/docs/explain/async-strategy.mdx +++ b/docs-site/src/content/docs/explain/async-strategy.mdx @@ -1,6 +1,6 @@ --- title: Async strategy -description: Why informix-driver wraps a sync core in a thread pool instead of going fully async — and what that costs. +description: Why informix-driver wraps a sync core in a thread pool instead of going fully async, and what that costs. sidebar: order: 4 --- @@ -19,13 +19,13 @@ Three options for adding async support to a sync database driver: 2. **Thread-pool wrapping.** Keep the sync core. Wrap each public method with `loop.run_in_executor()`. ~250 lines of code, sync tests still apply, no protocol-layer changes. -3. **Dual implementations.** Maintain two parallel code paths — one sync, one async. Most code duplicated. Worst of both worlds. +3. **Dual implementations.** Maintain two parallel code paths, one sync and one async. Most code duplicated. Worst of both worlds. We picked option 2. ## Why option 2 was the right call -For typical database workloads — request-scoped connections, mostly waiting on I/O — the practical difference between option 1 and option 2 is small: +For typical database workloads, meaning request-scoped connections that mostly wait on I/O, the practical difference between option 1 and option 2 is small: - **Latency**: option 1 has a slight edge (no thread context switch), but the difference is dwarfed by the actual database round-trip (~80 µs LAN, ~ms WAN). For a single query, option 2 adds ~5–10 µs of executor overhead. - **Throughput under concurrency**: option 1 wins when you have N coroutines on M physical cores with M < N. The thread pool needs to context-switch between threads; the async loop just runs the next coroutine. For 10–100 concurrent FastAPI requests on a 4-core box, this difference is small. @@ -39,7 +39,7 @@ The honest costs: - **One worker thread per concurrent in-flight query.** With 100 concurrent queries, you have 100 threads. This is fine for I/O-bound work (Python releases the GIL during socket reads) but doesn't scale beyond a few hundred concurrent queries on a single process. - **Thread-pool sizing matters.** The default executor size (5 × CPU count) is fine for most workloads. For high-concurrency workloads, you may want a larger executor. -- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe — cancelled workers cannot leak onto recycled pool connections — but the underlying syscall does still complete. +- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe, in that cancelled workers cannot leak onto recycled pool connections, but the underlying syscall does still complete. ## What it doesn't cost diff --git a/docs-site/src/content/docs/explain/buffered-reader.mdx b/docs-site/src/content/docs/explain/buffered-reader.mdx index 0acc34a..727631b 100644 --- a/docs-site/src/content/docs/explain/buffered-reader.mdx +++ b/docs-site/src/content/docs/explain/buffered-reader.mdx @@ -7,7 +7,7 @@ sidebar: import { Aside } from '@astrojs/starlight/components'; -The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39 — a connection-scoped buffered reader — closed it from 2.4× to ~1.05–1.15× in about thirty minutes of code plus ten minutes of architectural debugging. +The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39, a connection-scoped buffered reader, closed it from 2.4× to ~1.05–1.15× in about thirty minutes of code plus ten minutes of architectural debugging. This page is about both the technical change and the failure mode that hid the win for two phases. @@ -26,7 +26,7 @@ The headline "I/O dominated" was true. The interesting half is the breakdown of - Actual `recv()` syscalls: ~153 ms - Python wrapper overhead: ~400 ms -That ~400 ms was our own buffer abstraction — a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause. +That ~400 ms was our own buffer abstraction: a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause. The kernel was doing maybe 25–30 ms of work. The other 130 ms of the gap-vs-IfxPy was friction we had introduced ourselves. @@ -63,21 +63,21 @@ Result: **one `recv()` per ~64 KB of incoming data**, not per field. The natural thing to call this is "BufferedSocketReader". The natural thing to do is put the bytearray on the reader. That's what I did first. -Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken` — process blocked in `recv()` waiting for bytes that weren't coming. +Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken`, meaning the process was blocked in `recv()` waiting for bytes that weren't coming. -The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes — 200 bytes for response 1, 400 bytes for response 2 — it consumed bytes for response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader. +The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes, 200 for response 1 and 400 for response 2, it consumed bytes belonging to response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader. -The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket` (the connection-scoped wrapper) — readers are short-lived parser-views, the buffer outlives them. +The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket`, the connection-scoped wrapper. Readers are short-lived parser-views, and the buffer outlives them. ```python -# WRONG (first pass) — buffer scoped to reader +# WRONG (first pass): buffer scoped to reader class BufferedSocketReader: def __init__(self, sock): self.sock = sock self.buf = bytearray() # ← dies with the reader self.offset = 0 -# RIGHT (Phase 39) — buffer scoped to connection +# RIGHT (Phase 39): buffer scoped to connection class IfxSocket: def __init__(self, sock): self.sock = sock @@ -122,7 +122,7 @@ The buffered reader ships **enabled by default** in version 2026.05.05.12. To op IFX_BUFFERED_READER=0 python my_app.py ``` -The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime — close and reopen the pool to flip behavior. +The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime, so close and reopen the pool to flip behavior.