Docs: cover the 2026.09.02 review, and drop every em-dash
Content. The types page now warns about the two framing bugs this release fixed, since both silently corrupt data on versions before it: a BLOB or CLOB in any position but last shifted every column after it, and a NULL collection did the same. The API page documents two things that are now user-visible: only one scrollable cursor may be open per connection (previously the second statement drew -285 and destroyed the cursor as collateral), and transaction control written as SQL is tracked (previously rollback() after a SQL BEGIN WORK sent nothing and reported success). The phase log gains a section on the review itself, including the pattern behind almost all eleven bugs. Test counts updated to 457/456. Style. 136 em-dashes across 22 content files, plus Hero.astro and two stylesheets, rewritten rather than substituted -- swapping the character for a comma leaves prose that reads like it lost an argument with a linter. Bullets that used a dash to gloss a term now use a colon; parenthetical asides became their own sentences or moved inside brackets. The rendered HTML is clean. Verified against the live site by content, not status code: these pages return 200 for every path and render the 404 body, so a stale deploy looks perfectly healthy.
This commit is contained in:
parent
8f341c4b2a
commit
7ffc148112
@ -13,7 +13,7 @@
|
||||
</h1>
|
||||
<p class="ifx-hero__lede">
|
||||
Every other Informix driver wraps IBM's C SDK or the JDBC JAR. We weren't into that.
|
||||
So we read the protocol and wrote it ourselves — PEP 249, sync + async, pooled, TLS.
|
||||
So we read the protocol and wrote it ourselves: PEP 249, sync and async, pooled, TLS.
|
||||
Within 10% of IBM's own C driver on bulk fetches, <strong>1.6× faster</strong> on
|
||||
bulk inserts. No compile step. No <code>LD_LIBRARY_PATH</code> ritual. No
|
||||
<code>libcrypt.so.1</code> from 2018.
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Architecture overview
|
||||
description: How the layers stack — socket, framing, codec, resultset, cursor, connection, pool.
|
||||
description: How the layers stack, from socket through framing, codec, resultset, cursor, connection, and pool.
|
||||
sidebar:
|
||||
order: 2
|
||||
---
|
||||
@ -28,13 +28,13 @@ The driver is six layers, each with a single responsibility, each testable in is
|
||||
|
||||
The lowest layer. Wraps `socket.socket` with a connection-scoped read buffer (Phase 39). One `recv(64K)` per ~64 KB of incoming data; parsers read into the buffer via `struct.unpack_from(buf, offset)` rather than slicing copies.
|
||||
|
||||
Everything above this layer is `bytes` and `bytearray` arithmetic — no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`.
|
||||
Everything above this layer is `bytes` and `bytearray` arithmetic, with no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`.
|
||||
|
||||
See [The buffered reader →](/explain/buffered-reader/) for why the buffer lives here and not on the parser.
|
||||
|
||||
## Protocol / PDU framing
|
||||
|
||||
`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean* — only how to read and write the byte shapes.
|
||||
`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean*, only how to read and write the byte shapes.
|
||||
|
||||
The PDU types and their fields were reverse-engineered from three sources:
|
||||
|
||||
@ -44,9 +44,9 @@ The PDU types and their fields were reverse-engineered from three sources:
|
||||
|
||||
## Codec / Per-column readers
|
||||
|
||||
`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types — see [SQL ↔ Python types](/reference/types/) for the full table.
|
||||
`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types. See [SQL ↔ Python types](/reference/types/) for the full table.
|
||||
|
||||
Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]` — no per-column dispatch overhead.
|
||||
Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]`, with no per-column dispatch overhead.
|
||||
|
||||
Phase 38 went further with `exec()`-based codegen: for the hottest tables, the driver generates a flat decoder function with all readers inlined and dispatch decisions baked in. The generated function is the equivalent of unrolling the per-column dispatch into straight-line code.
|
||||
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Async strategy
|
||||
description: Why informix-driver wraps a sync core in a thread pool instead of going fully async — and what that costs.
|
||||
description: Why informix-driver wraps a sync core in a thread pool instead of going fully async, and what that costs.
|
||||
sidebar:
|
||||
order: 4
|
||||
---
|
||||
@ -19,13 +19,13 @@ Three options for adding async support to a sync database driver:
|
||||
|
||||
2. **Thread-pool wrapping.** Keep the sync core. Wrap each public method with `loop.run_in_executor()`. ~250 lines of code, sync tests still apply, no protocol-layer changes.
|
||||
|
||||
3. **Dual implementations.** Maintain two parallel code paths — one sync, one async. Most code duplicated. Worst of both worlds.
|
||||
3. **Dual implementations.** Maintain two parallel code paths, one sync and one async. Most code duplicated. Worst of both worlds.
|
||||
|
||||
We picked option 2.
|
||||
|
||||
## Why option 2 was the right call
|
||||
|
||||
For typical database workloads — request-scoped connections, mostly waiting on I/O — the practical difference between option 1 and option 2 is small:
|
||||
For typical database workloads, meaning request-scoped connections that mostly wait on I/O, the practical difference between option 1 and option 2 is small:
|
||||
|
||||
- **Latency**: option 1 has a slight edge (no thread context switch), but the difference is dwarfed by the actual database round-trip (~80 µs LAN, ~ms WAN). For a single query, option 2 adds ~5–10 µs of executor overhead.
|
||||
- **Throughput under concurrency**: option 1 wins when you have N coroutines on M physical cores with M < N. The thread pool needs to context-switch between threads; the async loop just runs the next coroutine. For 10–100 concurrent FastAPI requests on a 4-core box, this difference is small.
|
||||
@ -39,7 +39,7 @@ The honest costs:
|
||||
|
||||
- **One worker thread per concurrent in-flight query.** With 100 concurrent queries, you have 100 threads. This is fine for I/O-bound work (Python releases the GIL during socket reads) but doesn't scale beyond a few hundred concurrent queries on a single process.
|
||||
- **Thread-pool sizing matters.** The default executor size (5 × CPU count) is fine for most workloads. For high-concurrency workloads, you may want a larger executor.
|
||||
- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe — cancelled workers cannot leak onto recycled pool connections — but the underlying syscall does still complete.
|
||||
- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe, in that cancelled workers cannot leak onto recycled pool connections, but the underlying syscall does still complete.
|
||||
|
||||
## What it doesn't cost
|
||||
|
||||
|
||||
@ -7,7 +7,7 @@ sidebar:
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39 — a connection-scoped buffered reader — closed it from 2.4× to ~1.05–1.15× in about thirty minutes of code plus ten minutes of architectural debugging.
|
||||
The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39, a connection-scoped buffered reader, closed it from 2.4× to ~1.05–1.15× in about thirty minutes of code plus ten minutes of architectural debugging.
|
||||
|
||||
This page is about both the technical change and the failure mode that hid the win for two phases.
|
||||
|
||||
@ -26,7 +26,7 @@ The headline "I/O dominated" was true. The interesting half is the breakdown of
|
||||
- Actual `recv()` syscalls: ~153 ms
|
||||
- Python wrapper overhead: ~400 ms
|
||||
|
||||
That ~400 ms was our own buffer abstraction — a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause.
|
||||
That ~400 ms was our own buffer abstraction: a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause.
|
||||
|
||||
The kernel was doing maybe 25–30 ms of work. The other 130 ms of the gap-vs-IfxPy was friction we had introduced ourselves.
|
||||
|
||||
@ -63,21 +63,21 @@ Result: **one `recv()` per ~64 KB of incoming data**, not per field.
|
||||
|
||||
The natural thing to call this is "BufferedSocketReader". The natural thing to do is put the bytearray on the reader. That's what I did first.
|
||||
|
||||
Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken` — process blocked in `recv()` waiting for bytes that weren't coming.
|
||||
Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken`, meaning the process was blocked in `recv()` waiting for bytes that weren't coming.
|
||||
|
||||
The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes — 200 bytes for response 1, 400 bytes for response 2 — it consumed bytes for response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader.
|
||||
The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes, 200 for response 1 and 400 for response 2, it consumed bytes belonging to response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader.
|
||||
|
||||
The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket` (the connection-scoped wrapper) — readers are short-lived parser-views, the buffer outlives them.
|
||||
The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket`, the connection-scoped wrapper. Readers are short-lived parser-views, and the buffer outlives them.
|
||||
|
||||
```python
|
||||
# WRONG (first pass) — buffer scoped to reader
|
||||
# WRONG (first pass): buffer scoped to reader
|
||||
class BufferedSocketReader:
|
||||
def __init__(self, sock):
|
||||
self.sock = sock
|
||||
self.buf = bytearray() # ← dies with the reader
|
||||
self.offset = 0
|
||||
|
||||
# RIGHT (Phase 39) — buffer scoped to connection
|
||||
# RIGHT (Phase 39): buffer scoped to connection
|
||||
class IfxSocket:
|
||||
def __init__(self, sock):
|
||||
self.sock = sock
|
||||
@ -122,7 +122,7 @@ The buffered reader ships **enabled by default** in version 2026.05.05.12. To op
|
||||
IFX_BUFFERED_READER=0 python my_app.py
|
||||
```
|
||||
|
||||
The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime — close and reopen the pool to flip behavior.
|
||||
The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime, so close and reopen the pool to flip behavior.
|
||||
|
||||
<Aside type="note">
|
||||
The flag exists to make A/B measurement easy. There's no expected reason to disable it in production. If you hit a workload where the buffered reader is slower, that's a bug and we'd like to know.
|
||||
@ -132,9 +132,9 @@ The flag is read once at connection construction. Existing connections in a pool
|
||||
|
||||
The general pattern: **what's visible gets optimization attention; what's invisible gets written off as irreducible**.
|
||||
|
||||
The codec is visible — there's a loop, a `_decode_varchar` function, a `struct.unpack` call. You can read the inner loop and reason about it. Phases 37 and 38 attacked it, both got modest wins.
|
||||
The codec is visible: there's a loop, a `_decode_varchar` function, a `struct.unpack` call. You can read the inner loop and reason about it. Phases 37 and 38 attacked it, both got modest wins.
|
||||
|
||||
The I/O machinery looked invisible. `_socket.read_exact` is eight lines. The cursor's `_SocketReader` wrapper is twelve. The framing reads — `read_short`, `read_int`, `read_exact(payload_size)` — are one-liners. What could be slow about that?
|
||||
The I/O machinery looked invisible. `_socket.read_exact` is eight lines. The cursor's `_SocketReader` wrapper is twelve. The framing reads (`read_short`, `read_int`, `read_exact(payload_size)`) are one-liners. What could be slow about that?
|
||||
|
||||
It was carrying ~30% of total wall time. Two phases of changelogs implicitly blamed "the protocol" for the remaining gap. The actual culprit was a few lines of `bytes.join` in a wrapper from Phase 1 that nobody had revisited.
|
||||
|
||||
@ -142,6 +142,6 @@ The lesson is small and easy to state: a profile turns vibes into an attack surf
|
||||
|
||||
## Read more
|
||||
|
||||
- **[Architecture overview →](/explain/architecture/)** — where the buffered reader sits in the layer stack.
|
||||
- **[Phase log →](/explain/phase-log/)** — the full progression from Phase 1 through Phase 39+.
|
||||
- **["The 156 Milliseconds I'd Been Hand-Waving About"](https://ryanmalloy.com/collaborations/the-156-milliseconds-i-d-been-hand-waving-about/)** — Claude's reflection on the session in which Phase 39 shipped, including the two pushbacks that triggered the work.
|
||||
- **[Architecture overview →](/explain/architecture/)**: where the buffered reader sits in the layer stack.
|
||||
- **[Phase log →](/explain/phase-log/)**: the full progression from Phase 1 through Phase 39+.
|
||||
- **["The 156 Milliseconds I'd Been Hand-Waving About"](https://ryanmalloy.com/collaborations/the-156-milliseconds-i-d-been-hand-waving-about/)**: Claude's reflection on the session in which Phase 39 shipped, including the two pushbacks that triggered the work.
|
||||
|
||||
@ -43,11 +43,11 @@ The driver was built across 39+ phases, each with a focused scope and a decision
|
||||
| 23 | Health checks | Pool validates idle connections before return |
|
||||
| 24 | Statement caching | Per-connection prepared-statement cache |
|
||||
| 25 | Fast-path call (`SQ_FPROUTINE`) | Direct UDF/SPL invocation, bypassing PREPARE |
|
||||
| 26 | **CRITICAL** | Pool returned connections with open transactions — fixed |
|
||||
| 26 | **CRITICAL** | Pool returned connections with open transactions (fixed) |
|
||||
| 27 | **CRITICAL** | Per-connection wire lock + async cancellation safety |
|
||||
| 28 | **HIGH** | `_raise_sq_err` bare-except masking wire desync — fixed |
|
||||
| 29 | Cursor finalizers | Server-side resource leak on mid-fetch raise — fixed |
|
||||
| 30 | Hardening pass | 5 medium-severity audit findings — all closed |
|
||||
| 28 | **HIGH** | `_raise_sq_err` bare-except masking wire desync (fixed) |
|
||||
| 29 | Cursor finalizers | Server-side resource leak on mid-fetch raise (fixed) |
|
||||
| 30 | Hardening pass | 5 medium-severity audit findings, all closed |
|
||||
|
||||
After Phase 30: **0 critical, 0 high, 0 medium audit findings remain.** Driver is production-ready.
|
||||
|
||||
@ -67,21 +67,50 @@ After Phase 30: **0 critical, 0 high, 0 medium audit findings remain.** Driver i
|
||||
|
||||
The Phase 37–39 trajectory is documented in detail at [The buffered reader →](/explain/buffered-reader/), including the architectural mistake the first pass got wrong.
|
||||
|
||||
## Field reports and systematic review (2026.08–2026.09)
|
||||
|
||||
Phase 30's audit found nothing left. Then real schemas arrived, and the audit's clean bill of health turned out to be a statement about the questions it had asked rather than about the driver.
|
||||
|
||||
A field report from a user running Informix 12 surfaced three type-framing bugs in a single afternoon: `INT8` / `SERIAL8` never decoded at all, `NCHAR` losing its first character, and `BOOLEAN` corrupting every column after it. Fuzzing the type matrix found more, and each one had the same shape as the last.
|
||||
|
||||
`2026.09.02` went back over the driver looking for that shape rather than for new symptoms, and found eleven bugs.
|
||||
|
||||
| Area | What was wrong |
|
||||
|---|---|
|
||||
| Row framing | Nothing ever checked that a row consumed its own payload. Fourteen framing bugs had reached users, every one detectable for free |
|
||||
| Smart LOBs | Read as a flat 72-byte field when they occupy 149, shifting every column after a non-final `BLOB` |
|
||||
| Composites | A NULL collection skipped its length field, the same bug as NULL `LVARCHAR` twelve lines away |
|
||||
| Transactions | `rollback()` after a SQL `BEGIN WORK` sent nothing and reported success |
|
||||
| Classification | Comments, CTEs and parenthesized selects were run as DML and failed with `-260` |
|
||||
| Scrollable cursors | A second statement on the connection got `-285` and destroyed the cursor too |
|
||||
| Statement release | Two of eight exits leaked a failed statement, bricking the connection |
|
||||
| Cursor finalizers | Cleanup could land inside another statement, and the lock probe was blind to its own thread |
|
||||
| Async | The whole layer shared the process-wide thread pool, so cancellations starved it |
|
||||
| Socket | Two readers on one stream agreed only by accident of how the server replies |
|
||||
|
||||
The pattern behind almost all of them: a hazard that was understood and guarded at the site where it was first observed, rather than at the abstraction that owned it. The guard then never travelled to its siblings. Four separate hand-written copies of the same UDT envelope, three of them wrong. Six hand-rolled copies of the same statement-release cleanup, two missing entirely.
|
||||
|
||||
Two of the fixes came from asking the server instead of guessing. `statement_type` and `statement_id` had both been parsed out of every DESCRIBE response into a metadata dict that nothing read, while the code that needed them inferred the answer from the first word of the SQL and got it wrong five ways.
|
||||
|
||||
The cheapest fix was also the most valuable. Asserting that a row decoder lands exactly on the end of its payload costs one integer comparison, and it would have caught all fourteen framing bugs at the byte where each happened rather than three releases later in somebody's result set.
|
||||
|
||||
Test count went from 241 to 457 across three server versions over this stretch.
|
||||
|
||||
## Notable architectural pivots
|
||||
|
||||
The decision log calls out four moments where the obvious choice would have been wrong:
|
||||
|
||||
1. **Phase 10/11** — abandoning `SQ_FPROUTINE` + `SQ_LODATA` for `SQ_FILE` intercept. Smaller, simpler, same correctness.
|
||||
2. **Phase 16** — thread-pool async instead of full async refactor. ~88% less code, same FastAPI surface.
|
||||
3. **Phase 27** — adding a per-connection wire lock instead of relying on PEP 249's "don't share connections" advice. Made accidental sharing safe rather than catastrophic.
|
||||
4. **Phase 39** — buffer on the connection, not on the reader. Got it wrong on the first pass; the bug surfaced as a hang on pipelined `executemany`. Fixed in ten minutes once the architectural mistake was named.
|
||||
1. **Phase 10/11**: abandoning `SQ_FPROUTINE` + `SQ_LODATA` for `SQ_FILE` intercept. Smaller, simpler, same correctness.
|
||||
2. **Phase 16**: thread-pool async instead of a full async refactor. ~88% less code, same FastAPI surface.
|
||||
3. **Phase 27**: adding a per-connection wire lock instead of relying on PEP 249's "don't share connections" advice. Made accidental sharing safe rather than catastrophic.
|
||||
4. **Phase 39**: buffer on the connection, not on the reader. Got it wrong on the first pass; the bug surfaced as a hang on pipelined `executemany`. Fixed in ten minutes once the architectural mistake was named.
|
||||
|
||||
## What's next
|
||||
|
||||
The roadmap (loose, not committed):
|
||||
|
||||
- **Phase 40+ (codec)**: Numpy-backed bulk decode for homogeneous columns. ~5× speedup target on analytical workloads.
|
||||
- **Phase 4x (protocol)**: Optional Cython acceleration for the codec hot loop. Would compromise "pure Python" — gated behind a build flag.
|
||||
- **Phase 4x (protocol)**: Optional Cython acceleration for the codec hot loop. Would compromise "pure Python", so it would sit behind a build flag.
|
||||
- **Phase 5x (API)**: Native `callproc` with named parameters, IBM-specific scrollable cursor extensions for full IfxPy parity.
|
||||
|
||||
The phase log is updated as work lands. The repo's [`CHANGELOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/CHANGELOG.md) is the source of truth for shipped changes.
|
||||
|
||||
@ -26,14 +26,14 @@ The order-of-magnitude intuition: pure-Python is ~2× slower than C-bound for **
|
||||
|
||||
The benefits are mostly deployment, not performance:
|
||||
|
||||
- **50 KB wheel** — installable in a slim Docker image without a build toolchain.
|
||||
- **No `libcrypt.so.1`** — works on Arch, Fedora 35+, RHEL 9, and any modern Linux.
|
||||
- **Python 3.10–3.14** — no minor-version-specific C extension breakage. We've shipped on the day each new Python released.
|
||||
- **Type annotations everywhere** — `py.typed` flag, full coverage in mypy / pyright.
|
||||
- **Auditable codepaths** — every byte that enters or leaves a socket goes through Python code you can read. No "the C extension does it" excuses.
|
||||
- **Async without `run_in_executor`** — native `async def` API, FastAPI-compatible.
|
||||
- **50 KB wheel**: installable in a slim Docker image without a build toolchain.
|
||||
- **No `libcrypt.so.1`**: works on Arch, Fedora 35+, RHEL 9, and any modern Linux.
|
||||
- **Python 3.10–3.14**: no minor-version-specific C extension breakage. We've shipped on the day each new Python released.
|
||||
- **Type annotations everywhere**: `py.typed` flag, full coverage in mypy / pyright.
|
||||
- **Auditable codepaths**: every byte that enters or leaves a socket goes through Python code you can read. No "the C extension does it" excuses.
|
||||
- **Async without `run_in_executor`**: native `async def` API, FastAPI-compatible.
|
||||
|
||||
For most real workloads, deployment friction matters more than 2 µs/row. The 92 MB OneDB tarball, the four `LD_LIBRARY_PATH` entries, the absent `libcrypt.so.1` — those costs are paid every time you deploy. The 2 µs/row codec gap is paid once per row, and only if your workload is read-heavy enough for it to dominate.
|
||||
For most real workloads, deployment friction matters more than 2 µs/row. The 92 MB OneDB tarball, the four `LD_LIBRARY_PATH` entries, the absent `libcrypt.so.1`: those costs are paid every time you deploy. The 2 µs/row codec gap is paid once per row, and only if your workload is read-heavy enough for it to dominate.
|
||||
|
||||
## Where the ceiling sits
|
||||
|
||||
@ -48,7 +48,7 @@ Five fields × ~250 ns/field + ~250 ns overhead = ~1.5 µs. We're at ~2.0 µs wh
|
||||
|
||||
Strategies for closing further:
|
||||
|
||||
- **Cython / mypyc compilation.** Could shave 30-50% off the codec hot loop. Would compromise the "pure Python" claim — there'd be a build step.
|
||||
- **Cython / mypyc compilation.** Could shave 30-50% off the codec hot loop. Would compromise the "pure Python" claim, because there'd be a build step.
|
||||
- **Bytecode optimization via `exec()`-codegen** (the Phase 38 approach). Marginal further wins; we've already extracted most of what's available.
|
||||
- **Numpy-backed bulk decode** for homogeneous columns. Promising for analytical workloads. ~5× speedup possible for `SELECT col FROM huge_table` over the current per-row approach. Probably Phase 41+.
|
||||
|
||||
@ -58,6 +58,6 @@ For I/O-bound workloads we're already at the ceiling. The buffered reader closed
|
||||
|
||||
Pure-Python costs us ~5–15% on bulk-fetch workloads and zero (or favorable) on everything else. The deployment, async, and modern-Python wins are large and don't depend on workload.
|
||||
|
||||
If the codec gap matters for your case — analytical reporting against a wide table, pulling millions of rows in a single SELECT — IfxPy is probably the right tool today. If you're doing transactional or bulk-load work, FastAPI services, or any deployment where IBM's C SDK is friction, `informix-driver` is the right tool.
|
||||
If the codec gap matters for your case, meaning analytical reporting against a wide table or pulling millions of rows in a single SELECT, IfxPy is probably the right tool today. If you're doing transactional or bulk-load work, FastAPI services, or any deployment where IBM's C SDK is friction, `informix-driver` is the right tool.
|
||||
|
||||
The driver chose the goal — *first pure-socket Informix driver in any language* — over the local optimum. Phase 37 onward is a sustained effort to make that choice cost as little as possible.
|
||||
The driver chose the goal, *first pure-socket Informix driver in any language*, over the local optimum. Phase 37 onward is a sustained effort to make that choice cost as little as possible.
|
||||
|
||||
@ -1,13 +1,13 @@
|
||||
---
|
||||
title: The SQLI wire protocol
|
||||
description: A short tour of Informix's SQLI protocol — PDU framing, the handshake, statement execution, fetch.
|
||||
description: A short tour of Informix's SQLI protocol, covering PDU framing, the handshake, statement execution, and fetch.
|
||||
sidebar:
|
||||
order: 1
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
SQLI is Informix's wire protocol — the same protocol IBM's CSDK and JDBC driver speak. It's a binary, length-prefixed PDU stream over a single TCP connection.
|
||||
SQLI is Informix's wire protocol, the same one IBM's CSDK and JDBC driver speak. It's a binary, length-prefixed PDU stream over a single TCP connection.
|
||||
|
||||
This page is a short tour. The byte-level reference (with hex annotations) lives in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md) in the repo.
|
||||
|
||||
@ -71,14 +71,14 @@ The full lifecycle for `SELECT id FROM users WHERE id = ?`:
|
||||
← SQ_ID
|
||||
```
|
||||
|
||||
For pipelined `executemany`, the driver sends `SQ_OPEN`+`SQ_FETCH` (or `SQ_BIND`+`SQ_EXEC`) for all N rows back-to-back without waiting for responses, then drains all responses at the end. This is what gives the 1.6× win over IfxPy on bulk inserts — see [Bulk inserts](/how-to/executemany/).
|
||||
For pipelined `executemany`, the driver sends `SQ_OPEN`+`SQ_FETCH` (or `SQ_BIND`+`SQ_EXEC`) for all N rows back-to-back without waiting for responses, then drains all responses at the end. This is what gives the 1.6× win over IfxPy on bulk inserts. See [Bulk inserts](/how-to/executemany/).
|
||||
|
||||
## Smart-LOB transfer
|
||||
|
||||
`SQ_FILE` (0x62) is a self-contained PDU type that carries chunks of BLOB/CLOB data. It's used by Informix's `lotofile` and `filetoblob` server functions. The driver intercepts these PDUs at the wire level and reassembles them client-side — no `SQ_FPROUTINE` / `SQ_LODATA` machinery needed.
|
||||
`SQ_FILE` (0x62) is a self-contained PDU type that carries chunks of BLOB/CLOB data. It's used by Informix's `lotofile` and `filetoblob` server functions. The driver intercepts these PDUs at the wire level and reassembles them client-side, with no `SQ_FPROUTINE` / `SQ_LODATA` machinery needed.
|
||||
|
||||
This was the architectural pivot in [Phase 10/11](/explain/phase-log/) that made smart-LOBs work end-to-end in pure Python. Reading and writing GB-sized BLOBs goes through the same socket as any other query.
|
||||
|
||||
<Aside type="note">
|
||||
The protocol has many more PDU types than this page covers — mostly variants for specific server features (PUT, GET-DESCRIPTOR, ROWDESC, DBINFO, COLLINFO, etc.). The complete list is in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md), with hex captures for each.
|
||||
The protocol has many more PDU types than this page covers, mostly variants for specific server features (PUT, GET-DESCRIPTOR, ROWDESC, DBINFO, COLLINFO, etc.). The complete list is in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md), with hex captures for each.
|
||||
</Aside>
|
||||
|
||||
@ -56,7 +56,7 @@ async def get_user(user_id: int, conn = Depends(get_conn)):
|
||||
|
||||
## Cancellation
|
||||
|
||||
If a client disconnects mid-request, FastAPI cancels the task. `informix-driver` is cancellation-safe — the cancellation propagates cleanly, the in-flight worker is reaped, and the connection returns to the pool clean (transactions rolled back). You don't need to wrap anything in `try/finally`.
|
||||
If a client disconnects mid-request, FastAPI cancels the task. `informix-driver` is cancellation-safe: the cancellation propagates cleanly, the in-flight worker is reaped, and the connection returns to the pool clean, with transactions rolled back. You don't need to wrap anything in `try/finally`.
|
||||
|
||||
<Aside type="note">
|
||||
This is a Phase 27 invariant: async cancellation cannot leak running workers onto recycled connections. The earlier behavior was a `High` audit finding; the fix is a CI tripwire test that's been green every commit since.
|
||||
@ -89,5 +89,5 @@ async def create_order(order: OrderIn, conn = Depends(get_conn)):
|
||||
`HTTPException` is an `Exception`, so raising one inside the block rolls back before FastAPI turns it into a response.
|
||||
|
||||
<Aside type="note">
|
||||
There is no `conn.transaction()` context manager on either the sync or async connection — an earlier version of this page showed one. Wrapping this in your own `@asynccontextmanager` helper is a few lines if you want the shorthand.
|
||||
There is no `conn.transaction()` context manager on either the sync or async connection, though an earlier version of this page showed one. Wrapping this in your own `@asynccontextmanager` helper is a few lines if you want the shorthand.
|
||||
</Aside>
|
||||
|
||||
@ -1,11 +1,11 @@
|
||||
---
|
||||
title: Optimize bulk SELECT
|
||||
description: How the buffered reader works in practice — when it's on, when it isn't, how to A/B-measure your workload.
|
||||
description: How the buffered reader works in practice, when it's on, when it isn't, and how to A/B-measure your workload.
|
||||
sidebar:
|
||||
order: 5
|
||||
---
|
||||
|
||||
The connection-scoped buffered reader (Phase 39) is **enabled by default** as of `2026.05.05.12`. For most workloads you don't need to touch anything — the bulk-fetch gap against IfxPy is now ~5–15% rather than ~140%.
|
||||
The connection-scoped buffered reader (Phase 39) is **enabled by default** as of `2026.05.05.12`. For most workloads you don't need to touch anything, since the bulk-fetch gap against IfxPy is now ~5–15% rather than ~140%.
|
||||
|
||||
For the architectural rationale, see [The buffered reader →](/explain/buffered-reader/).
|
||||
|
||||
@ -33,7 +33,7 @@ For typical bulk-SELECT workloads expect a 30–40% wall-time reduction. For wor
|
||||
|
||||
## When the speedup is largest
|
||||
|
||||
Workloads where every column read makes ~4–5 small `recv()` calls — i.e. tabular data, narrow rows, large row counts. The buffered reader replaces N small `recv()` calls with one `recv(64K)` per ~64 KB of incoming data.
|
||||
Workloads where every column read makes ~4–5 small `recv()` calls, meaning tabular data, narrow rows, and large row counts. The buffered reader replaces N small `recv()` calls with one `recv(64K)` per ~64 KB of incoming data.
|
||||
|
||||
| Workload shape | Speedup |
|
||||
|---|---:|
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Run the dev container
|
||||
description: IBM Informix Developer Edition in Docker — first-time setup, sbspace for smart-LOBs, common troubleshooting.
|
||||
description: IBM Informix Developer Edition in Docker, covering first-time setup, sbspace for smart-LOBs, and common troubleshooting.
|
||||
sidebar:
|
||||
order: 8
|
||||
---
|
||||
@ -20,9 +20,9 @@ docker run -d --name informix-dev \
|
||||
icr.io/informix/informix-developer-database:15.0.1.0.3DE
|
||||
```
|
||||
|
||||
- **`9088`** — clear-text SQLI listener
|
||||
- **`9089`** — TLS-enabled SQLI listener (server-side cert is self-signed; use `tls=True` in dev)
|
||||
- **`--privileged`** — required for the dev image's shared-memory tuning
|
||||
- **`9088`**: clear-text SQLI listener
|
||||
- **`9089`**: TLS-enabled SQLI listener (server-side cert is self-signed; use `tls=True` in dev)
|
||||
- **`--privileged`**: required for the dev image's shared-memory tuning
|
||||
|
||||
The image takes ~90 seconds to initialize. Watch for `oninit running`:
|
||||
|
||||
@ -84,10 +84,10 @@ pytest -m integration
|
||||
|
||||
**"Connection refused" on 9088**: the image is still initializing. Wait for `oninit running` in the logs.
|
||||
|
||||
**Login succeeds, queries fail with `-329` (database does not exist)**: you're connecting to a database that hasn't been created yet. Use `database="sysmaster"` for ad-hoc testing — it always exists.
|
||||
**Login succeeds, queries fail with `-329` (database does not exist)**: you're connecting to a database that hasn't been created yet. Use `database="sysmaster"` for ad-hoc testing, since it always exists.
|
||||
|
||||
**`-908` (system error / shared memory)**: the container needs `--privileged`. Restart with that flag.
|
||||
|
||||
<Aside type="tip">
|
||||
For a long-running dev environment, set `restart: unless-stopped` in a compose file. The image is well-behaved on restart — the database survives container shutdown.
|
||||
For a long-running dev environment, set `restart: unless-stopped` in a compose file. The image is well-behaved on restart, and the database survives container shutdown.
|
||||
</Aside>
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Bulk inserts (executemany)
|
||||
description: How to bulk-load with executemany — and the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
||||
description: How to bulk-load with executemany, including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
||||
sidebar:
|
||||
order: 4
|
||||
---
|
||||
@ -49,7 +49,7 @@ The default is `autocommit=False`, so this only catches you if you've explicitly
|
||||
|
||||
## Why it's faster than IfxPy
|
||||
|
||||
IfxPy's `executemany` calls `IfxPy.execute(stmt, tuple)` internally per row. That's one round-trip per row — for 10,000 rows on a 80 µs RTT, that's 800 ms of just waiting for ACKs.
|
||||
IfxPy's `executemany` calls `IfxPy.execute(stmt, tuple)` internally per row. That's one round-trip per row, so for 10,000 rows on an 80 µs RTT it adds up to 800 ms of just waiting for ACKs.
|
||||
|
||||
Phase 33 changed our `executemany` to **pipeline** the BIND+EXECUTE PDUs:
|
||||
|
||||
@ -78,7 +78,7 @@ with conn:
|
||||
cur = conn.cursor()
|
||||
for batch in chunks(huge_iterator, 10_000):
|
||||
cur.executemany("INSERT INTO logs ...", batch)
|
||||
# one transaction, many batched executemany calls — one commit at the end
|
||||
# one transaction, many batched executemany calls, one commit at the end
|
||||
```
|
||||
|
||||
10,000 rows per chunk is a reasonable default; the per-chunk Python memory cost is `~ N × bytes_per_row`. For 10k tuples of 5 small fields that's a few MB.
|
||||
@ -95,7 +95,7 @@ new_id = cur.fetchone()[0]
|
||||
|
||||
`sqlca.sqlerrd1` is where Informix records the serial value from the most recent INSERT on this connection, so read it before running any other statement.
|
||||
|
||||
There is no `cursor.lastrowid` — an earlier version of this page said there was.
|
||||
There is no `cursor.lastrowid`, though an earlier version of this page said there was.
|
||||
|
||||
For batch inserts that need the IDs, the idiomatic pattern is:
|
||||
|
||||
|
||||
@ -27,11 +27,11 @@ conn = informix_db.connect(
|
||||
)
|
||||
```
|
||||
|
||||
The `informix-driver` keyword-argument form is closer to `psycopg`/`asyncpg` shapes. Connection strings aren't supported (deliberately — they're a security and parsing footgun).
|
||||
The `informix-driver` keyword-argument form is closer to `psycopg`/`asyncpg` shapes. Connection strings aren't supported, deliberately: they're a security and parsing footgun.
|
||||
|
||||
## The DB-API surface is the same
|
||||
|
||||
`cursor()`, `execute()`, `fetchone()`, `fetchmany()`, `fetchall()`, `executemany()`, `description`, `rowcount`, `close()` — all behave per PEP 249.
|
||||
`cursor()`, `execute()`, `fetchone()`, `fetchmany()`, `fetchall()`, `executemany()`, `description`, `rowcount`, and `close()` all behave per PEP 249.
|
||||
|
||||
The exception hierarchy is identical: `Error`, `Warning`, `InterfaceError`, `DatabaseError`, `DataError`, `OperationalError`, `IntegrityError`, `InternalError`, `ProgrammingError`, `NotSupportedError`.
|
||||
|
||||
@ -45,9 +45,9 @@ The exception hierarchy is identical: `Error`, `Warning`, `InterfaceError`, `Dat
|
||||
|
||||
- **Async API** (`from informix_db import aio`)
|
||||
- **Connection pool** (`informix_db.create_pool` / `aio.create_pool`)
|
||||
- **Type-safe annotations** — `informix-driver` ships with `py.typed`
|
||||
- **Type-safe annotations**: `informix-driver` ships with `py.typed`
|
||||
- **Python 3.12+ support**
|
||||
- **Pipelined `executemany`** — 1.6× faster than IfxPy's per-row implementation
|
||||
- **Pipelined `executemany`**: 1.6× faster than IfxPy's per-row implementation
|
||||
|
||||
## Migrating incrementally
|
||||
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Use the connection pool
|
||||
description: Sync and async connection pools — sizing, timeouts, lifecycle, threading.
|
||||
description: Sync and async connection pools, covering sizing, timeouts, lifecycle, and threading.
|
||||
sidebar:
|
||||
order: 2
|
||||
---
|
||||
@ -32,7 +32,7 @@ with pool.connection() as conn:
|
||||
pool.close()
|
||||
```
|
||||
|
||||
The context manager guarantees the connection returns to the pool on normal exit *and* on exception. Connections returned to the pool get rolled back automatically — you never see a dirty connection from `pool.connection()`.
|
||||
The context manager guarantees the connection returns to the pool on normal exit *and* on exception. Connections returned to the pool get rolled back automatically, so you never see a dirty connection from `pool.connection()`.
|
||||
|
||||
## Async pool
|
||||
|
||||
@ -56,16 +56,16 @@ async def main():
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
Same semantics, `async`-aware. Cancellation is cancellation-safe — a cancelled task does not leak an in-flight worker onto a recycled connection.
|
||||
Same semantics, `async`-aware. Cancellation is safe here too: a cancelled task does not leak an in-flight worker onto a recycled connection.
|
||||
|
||||
## Sizing
|
||||
|
||||
A reasonable starting point: `min_size = 2`, `max_size = (CPU cores) × 2`. Most Informix workloads are I/O-bound, so the right size is "enough to saturate the network plus some headroom for spikes" — usually 8–16 for typical web/API services.
|
||||
A reasonable starting point: `min_size = 2`, `max_size = (CPU cores) × 2`. Most Informix workloads are I/O-bound, so the right size is "enough to saturate the network plus some headroom for spikes", usually 8–16 for typical web/API services.
|
||||
|
||||
`max_size` should be **smaller than the server's `MAX_CONCURRENT_CONNECTIONS`** — the server fails new logins past its limit, and the pool will surface that as `OperationalError` after waiting `acquire_timeout`.
|
||||
`max_size` should be **smaller than the server's `MAX_CONCURRENT_CONNECTIONS`**. The server fails new logins past its limit, and the pool will surface that as `OperationalError` after waiting `acquire_timeout`.
|
||||
|
||||
## Threading
|
||||
|
||||
PEP 249 says: connections should not be shared between threads. The pool gives each thread its own connection naturally — `pool.connection()` returns a different connection each time and each one stays held until the context manager exits.
|
||||
PEP 249 says: connections should not be shared between threads. The pool gives each thread its own connection naturally, since `pool.connection()` returns a different connection each time and each one stays held until the context manager exits.
|
||||
|
||||
Phase 27 added a per-connection wire lock that makes accidental sharing safe (interleaved PDUs serialize correctly), but you should still give each thread its own connection. The lock is a backstop, not a license.
|
||||
|
||||
@ -34,7 +34,7 @@ The `BLOB_PLACEHOLDER` token in the SQL marks where the BLOB data goes. Other pa
|
||||
|
||||
## Reading a CLOB
|
||||
|
||||
CLOBs use the same methods as BLOBs — there is no separate `read_clob_column`. `read_blob_column` returns `bytes` for both, so decode it with the encoding the column was written in:
|
||||
CLOBs use the same methods as BLOBs, and there is no separate `read_clob_column`. `read_blob_column` returns `bytes` for both, so decode it with the encoding the column was written in:
|
||||
|
||||
```python
|
||||
raw: bytes = cur.read_blob_column(
|
||||
|
||||
@ -30,7 +30,7 @@ conn = informix_db.connect(
|
||||
)
|
||||
```
|
||||
|
||||
Bring-your-own context is the recommended production pattern — you get full control of certificate verification, hostname checking, ciphers, and TLS version pinning.
|
||||
Bring-your-own context is the recommended production pattern, giving you full control of certificate verification, hostname checking, ciphers, and TLS version pinning.
|
||||
|
||||
## Dev / self-signed: tls=True
|
||||
|
||||
@ -38,7 +38,7 @@ Bring-your-own context is the recommended production pattern — you get full co
|
||||
informix_db.connect(host="127.0.0.1", port=9089, ..., tls=True)
|
||||
```
|
||||
|
||||
`tls=True` is a convenience for development — it builds a default context with `check_hostname=False` and `verify_mode=CERT_NONE`. **Do not use this in production.**
|
||||
`tls=True` is a convenience for development. It builds a default context with `check_hostname=False` and `verify_mode=CERT_NONE`. **Do not use this in production.**
|
||||
|
||||
## Server-side configuration
|
||||
|
||||
@ -48,7 +48,7 @@ The Informix server needs a TLS listener entry in `sqlhosts`:
|
||||
informix_tls onsoctcp myhost 9089
|
||||
```
|
||||
|
||||
Plus a server-side keystore. The IBM Developer Edition Docker image ships with a TLS listener already enabled on `9089` — no configuration needed.
|
||||
Plus a server-side keystore. The IBM Developer Edition Docker image ships with a TLS listener already enabled on `9089`, with no configuration needed.
|
||||
|
||||
<Aside type="tip">
|
||||
If your connection hangs at the handshake, you've probably pointed at the non-TLS port (`9088` instead of `9089`). The non-TLS listener will accept the TCP connection but won't speak TLS, so the handshake stalls.
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: informix-driver
|
||||
description: Pure-Python driver for IBM Informix IDS. Speaks the SQLI wire protocol over a raw socket — no CSDK, no JVM, no native libraries.
|
||||
description: Pure-Python driver for IBM Informix IDS. Speaks the SQLI wire protocol over a raw socket, with no CSDK, no JVM, and no native libraries.
|
||||
template: splash
|
||||
hero:
|
||||
tagline: ''
|
||||
@ -21,12 +21,12 @@ import { Card, CardGrid, Icon } from '@astrojs/starlight/components';
|
||||
<div class="ifx-feature">
|
||||
<Icon name="seti:python" class="ifx-feature__icon" />
|
||||
<h3>~10% behind IfxPy on bulk fetches</h3>
|
||||
<p>Phase 39's buffered reader closed the gap from 2.4× to within measurement noise of the C driver. The remaining ~10% is honest physics — for now.</p>
|
||||
<p>Phase 39's buffered reader closed the gap from 2.4× to within measurement noise of the C driver. The remaining ~10% is honest physics, for now.</p>
|
||||
</div>
|
||||
<div class="ifx-feature">
|
||||
<Icon name="puzzle" class="ifx-feature__icon" />
|
||||
<h3>50 KB wheel. Zero native deps.</h3>
|
||||
<p>No 92 MB OneDB tarball. No <code>libcrypt.so.1</code> from 2018. No <code>LD_LIBRARY_PATH</code> ritual. Works on Python 3.10–3.14 — including the versions IfxPy doesn't.</p>
|
||||
<p>No 92 MB OneDB tarball. No <code>libcrypt.so.1</code> from 2018. No <code>LD_LIBRARY_PATH</code> ritual. Works on Python 3.10–3.14, including the versions IfxPy doesn't.</p>
|
||||
</div>
|
||||
<div class="ifx-feature">
|
||||
<Icon name="sun" class="ifx-feature__icon" />
|
||||
@ -36,7 +36,7 @@ import { Card, CardGrid, Icon } from '@astrojs/starlight/components';
|
||||
<div class="ifx-feature">
|
||||
<Icon name="approve-check" class="ifx-feature__icon" />
|
||||
<h3>PEP 249, no surprises</h3>
|
||||
<p><code>connect()</code>, <code>Connection</code>, <code>Cursor</code>, <code>description</code>, <code>rowcount</code>, the full DB-API exception hierarchy — and threadsafe sharing through a per-connection wire lock.</p>
|
||||
<p><code>connect()</code>, <code>Connection</code>, <code>Cursor</code>, <code>description</code>, <code>rowcount</code>, the full DB-API exception hierarchy, plus threadsafe sharing through a per-connection wire lock.</p>
|
||||
</div>
|
||||
<div class="ifx-feature">
|
||||
<Icon name="document" class="ifx-feature__icon" />
|
||||
@ -66,7 +66,7 @@ That's it. No `IBM_DB_HOME`. No DSN file. No `libcrypt.so.1`.
|
||||
|
||||
The existing tools were not my style.
|
||||
|
||||
Every other Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix` — all of them. To our knowledge **`informix-driver` is the first pure-socket Informix driver in any language**.
|
||||
Every other Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That means all of them: `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge **`informix-driver` is the first pure-socket Informix driver in any language**.
|
||||
|
||||
The OneDB CSDK is a 92 MB tarball. It needs `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9). It needs four `LD_LIBRARY_PATH` entries. It needs `setuptools < 58`. And IfxPy itself is broken on Python 3.12+. For containerized deployments, ETL pipelines, FastAPI services, or anywhere a build toolchain on the runtime is friction, this driver is the alternative that didn't previously exist. Now it does.
|
||||
|
||||
@ -82,7 +82,7 @@ The OneDB CSDK is a 92 MB tarball. It needs `libcrypt.so.1` (deprecated 2018, mi
|
||||
[Read →](/start/vs-ifxpy/)
|
||||
</Card>
|
||||
<Card title="The buffered reader" icon="information">
|
||||
How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1× — and the architectural mistake the first pass got wrong.
|
||||
How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1×, and the architectural mistake the first pass got wrong.
|
||||
[Read →](/explain/buffered-reader/)
|
||||
</Card>
|
||||
<Card title="Architecture" icon="puzzle">
|
||||
|
||||
@ -51,7 +51,7 @@ informix_db.connect(
|
||||
|
||||
| Method / property | Description |
|
||||
|---|---|
|
||||
| `cursor(scrollable=False)` | Returns a new `Cursor`. |
|
||||
| `cursor(scrollable=False)` | Returns a new `Cursor`. Only one scrollable cursor may be open per connection; see below. |
|
||||
| `commit()` | Commits the current transaction. |
|
||||
| `rollback()` | Rolls back the current transaction. |
|
||||
| `close()` | Closes the connection. Idempotent. |
|
||||
@ -64,7 +64,7 @@ informix_db.connect(
|
||||
|
||||
`Connection` is also a context manager (`with informix_db.connect(...) as conn:`), which closes on exit.
|
||||
|
||||
There is no `conn.transaction()` helper and no `conn.autocommit` attribute — earlier versions of this page listed both, and neither has ever existed. Set autocommit at connect time with `connect(autocommit=True)`, and manage transactions with `commit()` / `rollback()`:
|
||||
There is no `conn.transaction()` helper and no `conn.autocommit` attribute. Earlier versions of this page listed both, and neither has ever existed. Set autocommit at connect time with `connect(autocommit=True)`, and manage transactions with `commit()` / `rollback()`:
|
||||
|
||||
```python
|
||||
conn = informix_db.connect(..., autocommit=False)
|
||||
@ -78,10 +78,33 @@ except Exception:
|
||||
raise
|
||||
```
|
||||
|
||||
:::caution[One statement per session]
|
||||
Informix gives a session a single statement slot. A non-scrollable cursor never trips over this, because it materializes its rows and releases the statement before `execute()` returns. A **scrollable** cursor holds the slot open on purpose, so while one is open, any other statement on that same connection raises `ProgrammingError`:
|
||||
|
||||
```python
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
scroll.execute("SELECT id, body FROM big_table ORDER BY id")
|
||||
scroll.fetch_first()
|
||||
|
||||
other = conn.cursor()
|
||||
other.execute("SELECT COUNT(*) FROM audit") # ProgrammingError
|
||||
```
|
||||
|
||||
Close the scrollable cursor first, or use a second connection for the other query. Re-running `execute()` on the scrollable cursor itself is fine; it closes its own server-side cursor first.
|
||||
|
||||
Before `2026.09.02` this was not refused. The server answered the second statement with `-285` and destroyed the scrollable cursor as well, whose next fetch then returned `-267`, "the transaction has been rolled back". Two errors, neither naming the cause.
|
||||
:::
|
||||
|
||||
:::note[Transaction control written as SQL]
|
||||
`commit()` and `rollback()` are the supported way to end a transaction, but `cursor.execute("BEGIN WORK")` and its `COMMIT` / `ROLLBACK` counterparts work too, and the connection tracks them from `2026.09.02` onward.
|
||||
|
||||
That matters because it used not to. Under `autocommit=True`, a SQL `BEGIN WORK` opened a real transaction the connection never learned about, and since `rollback()` is guarded by that state, **it returned successfully having sent nothing**. The rows it was asked to discard survived, and a pooled connection went back into circulation still holding the transaction and its locks.
|
||||
:::
|
||||
|
||||
:::note[Two version numbers, and why]
|
||||
The login response carries Informix's *internal* protocol version, not the release you installed: 12.10 announces itself as `9.56`, 14.10 as `9.59`, and 15 as `15.0.1.0.3`. At the protocol level the two older releases really are 9.x servers, which is why all three speak an identical SQLI dialect.
|
||||
|
||||
That string is available for free as `server_version_internal`. Because it reads as a wrong answer, `server_version` asks the server for its release with `DBINFO('version','full')` — one query on first access, cached for the life of the connection, and it falls back to the internal string rather than raising if no database is open.
|
||||
That string is available for free as `server_version_internal`. Because it reads as a wrong answer, `server_version` asks the server for its release with `DBINFO('version','full')`. That is one query on first access, cached for the life of the connection, and it falls back to the internal string rather than raising if no database is open.
|
||||
:::
|
||||
|
||||
## Cursor
|
||||
@ -105,7 +128,7 @@ That string is available for free as `server_version_internal`. Because it reads
|
||||
| `rownumber` | Current 0-indexed position, or `None` before the first row. |
|
||||
| `arraysize` | Default `fetchmany()` size. |
|
||||
|
||||
There are no `read_clob_column` / `write_clob_column` methods and no `lastrowid` attribute — earlier versions of this page listed all three and none have existed.
|
||||
There are no `read_clob_column` / `write_clob_column` methods and no `lastrowid` attribute. Earlier versions of this page listed all three, and none have existed.
|
||||
|
||||
CLOBs go through the BLOB methods. `read_blob_column` returns `bytes` either way, so decode it yourself with the column's encoding:
|
||||
|
||||
@ -120,7 +143,7 @@ raw = cur.read_blob_column("SELECT txt FROM docs WHERE id = ?", (1,))
|
||||
text = raw.decode("iso-8859-1")
|
||||
```
|
||||
|
||||
For a server-assigned `SERIAL` after an INSERT, ask the server — Informix exposes it through `DBINFO`:
|
||||
For a server-assigned `SERIAL` after an INSERT, ask the server. Informix exposes it through `DBINFO`:
|
||||
|
||||
```python
|
||||
cur.execute("INSERT INTO people (name) VALUES (?)", ("ada",))
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Performance baselines
|
||||
description: Single-connection benchmark results — codec, framing, end-to-end queries, vs IfxPy.
|
||||
description: Single-connection benchmark results for codec, framing, end-to-end queries, and IfxPy.
|
||||
sidebar:
|
||||
order: 5
|
||||
---
|
||||
|
||||
@ -13,7 +13,7 @@ sidebar:
|
||||
| `IFX_DEBUG_WIRE` | unset | When set to a truthy value, log wire-level PDU framing to stderr. Verbose; for debugging only. |
|
||||
| `IFX_PROTOCOL_TRACE` | unset | When set to a path, write annotated wire captures to the file. |
|
||||
| `IFX_DISABLE_PIPELINE` | unset | Disables pipelined `executemany` (Phase 33). Use only to A/B-measure. |
|
||||
| `INFORMIXSERVER` | — | Read by `connect()` if `server=` is not provided. |
|
||||
| `INFORMIXSERVER` | *(none)* | Read by `connect()` if `server=` is not provided. |
|
||||
|
||||
Environment variables are read at connection construction. Changing them at runtime doesn't affect existing connections.
|
||||
|
||||
@ -64,4 +64,4 @@ informix_db.connect(
|
||||
)
|
||||
```
|
||||
|
||||
`CLIENT_LOCALE` is set automatically from `client_locale=` — don't put it in `env=`.
|
||||
`CLIENT_LOCALE` is set automatically from `client_locale=`, so don't put it in `env=`.
|
||||
|
||||
@ -9,7 +9,7 @@ sidebar:
|
||||
|---|---|---|
|
||||
| `SMALLINT` / `INT` / `SERIAL` | `int` | Arbitrary precision on the Python side. |
|
||||
| `BIGINT` / `BIGSERIAL` | `int` | 8-byte two's complement on the wire. |
|
||||
| `INT8` / `SERIAL8` | `int` | The *legacy* 64-bit integer — a different wire format from `BIGINT`, not an alias. See below. |
|
||||
| `INT8` / `SERIAL8` | `int` | The *legacy* 64-bit integer, a different wire format from `BIGINT` rather than an alias. See below. |
|
||||
| `FLOAT` / `SMALLFLOAT` | `float` | IEEE 754 double / single. |
|
||||
| `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` | Exact precision preserved. |
|
||||
| `CHAR` / `NCHAR` | `str` | Fixed width, space-padded; trailing spaces stripped on decode. |
|
||||
@ -18,11 +18,11 @@ sidebar:
|
||||
| `DATE` | `datetime.date` | |
|
||||
| `DATETIME YEAR TO …` | `datetime.datetime` / `datetime.time` / `datetime.date` | The Python type depends on the field range. |
|
||||
| `INTERVAL DAY TO FRACTION` | `datetime.timedelta` | |
|
||||
| `INTERVAL YEAR TO MONTH` | `informix_db.IntervalYM` | Custom type — `datetime.timedelta` can't represent year-month intervals. |
|
||||
| `INTERVAL YEAR TO MONTH` | `informix_db.IntervalYM` | Custom type, because `datetime.timedelta` can't represent year-month intervals. |
|
||||
| `BYTE` / `TEXT` (legacy in-row blobs) | `bytes` / `str` | |
|
||||
| `BLOB` / `CLOB` (smart-LOBs) | `informix_db.BlobLocator` / `informix_db.ClobLocator` | Opaque server-side locators. Read via `cursor.read_blob_column`, write via `cursor.write_blob_column`. |
|
||||
| `ROW(…)` | `informix_db.RowValue` | Raw payload plus schema string — **not** decomposed into fields. See below. |
|
||||
| `SET(…)` / `MULTISET(…)` / `LIST(…)` | `informix_db.CollectionValue` | Raw payload plus element schema — **not** iterable. See below. |
|
||||
| `BLOB` / `CLOB` (smart-LOBs) | `informix_db.BlobLocator` / `informix_db.ClobLocator` | Opaque server-side locators, 72 bytes. Read via `cursor.read_blob_column`, write via `cursor.write_blob_column`. |
|
||||
| `ROW(…)` | `informix_db.RowValue` | Raw payload plus schema string, **not** decomposed into fields. See below. |
|
||||
| `SET(…)` / `MULTISET(…)` / `LIST(…)` | `informix_db.CollectionValue` | Raw payload plus element schema, **not** iterable. See below. |
|
||||
| `NULL` | `None` | |
|
||||
|
||||
## INT8 and SERIAL8 are not BIGINT
|
||||
@ -34,16 +34,18 @@ Informix has two unrelated 64-bit integer types and they share nothing on the wi
|
||||
| `BIGINT` / `BIGSERIAL` (type 52 / 53) | 8 bytes, two's complement, big-endian |
|
||||
| `INT8` / `SERIAL8` (type 17 / 18) | 10 bytes, sign-magnitude, halves stored high-last |
|
||||
|
||||
Both decode to plain Python `int`, so this only matters if you're reading the wire yourself. It's worth knowing that `INT8` stores `+n` and `−n` with *identical* magnitude bytes and the sign in a leading word — decoding it as a signed 64-bit integer looks correct for every positive value and is wrong for every negative one.
|
||||
Both decode to plain Python `int`, so this only matters if you're reading the wire yourself. It's worth knowing that `INT8` stores `+n` and `−n` with *identical* magnitude bytes and the sign in a leading word. Decoding it as a signed 64-bit integer looks correct for every positive value and is wrong for every negative one.
|
||||
|
||||
`INT8`/`SERIAL8` are common in schemas predating Informix 11.50, which is when `BIGINT` arrived.
|
||||
|
||||
:::caution[Upgrade if you use these types]
|
||||
`2026.08.27` fixed three decoding bugs: `INT8`/`SERIAL8` weren't decoded at all and returned raw `bytes`; `NCHAR` lost its first character; and `BOOLEAN` corrupted every column after it.
|
||||
|
||||
`2026.08.31` fixed two more in `LVARCHAR` framing — a phantom pad byte on odd-length values, and a missing length field on NULLs. Either shifted **every column selected after the LVARCHAR**, producing wrong integers, strings missing their first character, or an outright `IndexError` on wide rows. The same release stopped `DATETIME` binds silently discarding sub-second precision.
|
||||
`2026.08.31` fixed two more in `LVARCHAR` framing: a phantom pad byte on odd-length values, and a missing length field on NULLs. Either shifted **every column selected after the LVARCHAR**, producing wrong integers, strings missing their first character, or an outright `IndexError` on wide rows. The same release stopped `DATETIME` binds silently discarding sub-second precision.
|
||||
|
||||
If your schema has `LVARCHAR` columns, `2026.08.27` is not safe — upgrade to `2026.08.31`.
|
||||
`2026.09.02` fixed two more, both found by a new check that verifies each row consumed exactly its own payload. Smart LOBs were read as a flat 72-byte field when they actually occupy 149 bytes populated and 5 when NULL, so a `BLOB` or `CLOB` anywhere but the **last** column shifted every column after it. A NULL `SET` / `MULTISET` / `LIST` / `ROW` skipped its length field and did the same. That release also made a mismatch raise where it happens, naming the column and the byte delta, instead of handing back plausible-looking wrong values.
|
||||
|
||||
If your schema has `LVARCHAR` columns, `2026.08.27` is not safe. If you select a `BLOB` or `CLOB` in any position other than last, or a nullable collection column, nothing before `2026.09.02` is safe. Upgrade.
|
||||
:::
|
||||
|
||||
## LVARCHAR wire framing
|
||||
@ -54,7 +56,7 @@ Worth knowing if you're reading captures. Every Informix server we've tested des
|
||||
[1-byte null indicator][4-byte length][content]
|
||||
```
|
||||
|
||||
No padding, and the length field is present even when the indicator says NULL. NULL and empty string differ *only* in that indicator byte — both carry a length of zero.
|
||||
No padding, and the length field is present even when the indicator says NULL. NULL and empty string differ *only* in that indicator byte, since both carry a length of zero.
|
||||
|
||||
## DATETIME field ranges
|
||||
|
||||
@ -68,17 +70,17 @@ Informix's `DATETIME YEAR TO X` is field-range typed. The Python type returned d
|
||||
|
||||
## Decimal precision
|
||||
|
||||
`DECIMAL` columns preserve their declared precision and scale through the codec. A `DECIMAL(10,2)` column with value `123.45` decodes to `Decimal("123.45")` exactly — no float intermediate.
|
||||
`DECIMAL` columns preserve their declared precision and scale through the codec. A `DECIMAL(10,2)` column with value `123.45` decodes to `Decimal("123.45")` exactly, with no float intermediate.
|
||||
|
||||
For binding `Decimal` values into INSERT/UPDATE, the driver uses the column's declared scale. Pass `Decimal` for exact values; `float` works but may cause rounding at the column scale.
|
||||
|
||||
## NULL
|
||||
|
||||
`NULL` is `None` in both directions. Use `IS NULL` / `IS NOT NULL` in SQL — `WHERE x = ?` with `None` returns no rows even where `x IS NULL`.
|
||||
`NULL` is `None` in both directions. Use `IS NULL` / `IS NOT NULL` in SQL, because `WHERE x = ?` with `None` returns no rows even where `x IS NULL`.
|
||||
|
||||
## Type extensions
|
||||
|
||||
`informix_db.IntervalYM` represents `INTERVAL YEAR TO MONTH`. It takes a **single total month count** — mirroring the server's own representation — with `years` and `remainder_months` available as derived properties:
|
||||
`informix_db.IntervalYM` represents `INTERVAL YEAR TO MONTH`. It takes a **single total month count**, mirroring the server's own representation, with `years` and `remainder_months` available as derived properties:
|
||||
|
||||
```python
|
||||
from informix_db import IntervalYM
|
||||
@ -100,8 +102,8 @@ Negative intervals are supported; the sign lives on `months` and propagates to b
|
||||
`informix_db.RowValue` and `informix_db.CollectionValue` are returned for `ROW`, `SET`, `MULTISET`, and `LIST` columns. Both are **opaque wrappers, not decomposed values**:
|
||||
|
||||
```python
|
||||
row_val.raw # bytes — the server's textual representation, e.g. b"ROW('Alice',30)"
|
||||
row_val.schema # str — the column's declared schema
|
||||
row_val.raw # bytes: the server's textual representation, e.g. b"ROW('Alice',30)"
|
||||
row_val.schema # str: the column's declared schema
|
||||
|
||||
coll_val.raw # bytes, e.g. b'LIST{10,20,30}'
|
||||
coll_val.kind # 'set' | 'multiset' | 'list' | 'collection'
|
||||
@ -110,7 +112,7 @@ coll_val.element_schema # str
|
||||
|
||||
They are not iterable and do not expose fields by name. Fully parsing a composite type means reimplementing JDBC's `IfxComplexInput`, which we haven't done.
|
||||
|
||||
If you need individual fields today, project them in SQL — the server does the decomposition for you and you get ordinary typed columns back:
|
||||
If you need individual fields today, project them in SQL. The server does the decomposition for you and you get ordinary typed columns back:
|
||||
|
||||
```python
|
||||
cur.execute("SELECT person.name, person.age FROM staff") # str, int
|
||||
|
||||
@ -55,7 +55,7 @@ docker logs -f informix-dev
|
||||
```
|
||||
|
||||
<Aside type="tip">
|
||||
The `--privileged` flag is required by the dev image — it tries to manage shared memory limits. For production servers this isn't a thing.
|
||||
The `--privileged` flag is required by the dev image, which tries to manage shared memory limits. For production servers this isn't a thing.
|
||||
</Aside>
|
||||
|
||||
## 3. Run your first query
|
||||
@ -88,7 +88,7 @@ Then:
|
||||
python hello.py
|
||||
```
|
||||
|
||||
You should see five rows from Informix's system catalog. If you do, congratulations — you've spoken SQLI to an IBM database from pure Python with zero native code in the call stack.
|
||||
You should see five rows from Informix's system catalog. If you do, congratulations: you've spoken SQLI to an IBM database from pure Python with zero native code in the call stack.
|
||||
|
||||
## What just happened
|
||||
|
||||
@ -119,7 +119,7 @@ with informix_db.connect(host="127.0.0.1", port=9088, user="informix",
|
||||
print(cur.fetchone())
|
||||
```
|
||||
|
||||
`?` and `:1` both work — Informix's native paramstyle is `numeric`, but `?` is supported as a synonym.
|
||||
`?` and `:1` both work. Informix's native paramstyle is `numeric`, but `?` is supported as a synonym.
|
||||
|
||||
## 5. Use the connection pool
|
||||
|
||||
@ -144,7 +144,7 @@ with pool.connection() as conn:
|
||||
pool.close()
|
||||
```
|
||||
|
||||
The pool is thread-safe and has a per-connection wire lock — accidental sharing across threads doesn't corrupt the wire stream, though PEP 249 advice still holds (one connection per thread).
|
||||
The pool is thread-safe and has a per-connection wire lock, so accidental sharing across threads doesn't corrupt the wire stream, though PEP 249 advice still holds (one connection per thread).
|
||||
|
||||
## What's next
|
||||
|
||||
@ -154,7 +154,7 @@ The pool is thread-safe and has a per-connection wire lock — accidental sharin
|
||||
|
||||
2. **Connecting to production?** [Connect with TLS →](/how-to/tls/) covers TLS-listener configuration and bring-your-own-context patterns.
|
||||
|
||||
3. **Bulk-loading?** [Bulk inserts (executemany) →](/how-to/executemany/) — and the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
||||
3. **Bulk-loading?** [Bulk inserts (executemany) →](/how-to/executemany/), including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
||||
|
||||
4. **Migrating from IfxPy?** [Migrate from IfxPy →](/how-to/migrate-from-ifxpy/) covers the API differences and the things IfxPy does that we don't (yet).
|
||||
|
||||
|
||||
@ -7,7 +7,7 @@ sidebar:
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
[IfxPy](https://pypi.org/project/IfxPy/) is IBM's official Python driver — a C extension that wraps the OneDB Client SDK (CSDK), which itself wraps the same SQLI wire protocol `informix-driver` speaks directly. It's the reasonable comparison: same protocol, same server, same workload, different transport.
|
||||
[IfxPy](https://pypi.org/project/IfxPy/) is IBM's official Python driver, a C extension that wraps the OneDB Client SDK (CSDK), which itself wraps the same SQLI wire protocol `informix-driver` speaks directly. It's the reasonable comparison: same protocol, same server, same workload, different transport.
|
||||
|
||||
Numbers below are **median + IQR over 10+ rounds**, all against the same IBM Informix Developer Edition Docker container on the same host. Methodology and reproduction steps live in [`tests/benchmarks/compare/`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/compare) in the repo.
|
||||
|
||||
@ -33,9 +33,9 @@ Numbers below are **median + IQR over 10+ rounds**, all against the same IBM Inf
|
||||
|
||||
### Bulk inserts at scale
|
||||
|
||||
The clearest win is bulk insert throughput. `executemany(10_000_rows)` runs in **161 ms** vs IfxPy's **259 ms** — `informix-driver` is 1.6× faster.
|
||||
The clearest win is bulk insert throughput. `executemany(10_000_rows)` runs in **161 ms** vs IfxPy's **259 ms**, so `informix-driver` is 1.6× faster.
|
||||
|
||||
The mechanism is pipelining. Phase 33 changed `executemany` to send all N BIND+EXECUTE PDUs back-to-back **before** draining any response. IfxPy's C-level `IfxPy.execute(stmt, tuple)` makes one round-trip per row — N RTTs at ~80 µs each adds up to the 100 ms gap.
|
||||
The mechanism is pipelining. Phase 33 changed `executemany` to send all N BIND+EXECUTE PDUs back-to-back **before** draining any response. IfxPy's C-level `IfxPy.execute(stmt, tuple)` makes one round-trip per row, and N RTTs at ~80 µs each adds up to the 100 ms gap.
|
||||
|
||||
```python
|
||||
# Both drivers
|
||||
@ -44,8 +44,8 @@ cur.executemany(
|
||||
rows, # list of 10_000 tuples
|
||||
)
|
||||
|
||||
# informix-driver: 161 ms — 10k PDUs sent, then 10k responses drained
|
||||
# IfxPy: 259 ms — 10k round-trips, each blocking on response
|
||||
# informix-driver: 161 ms (10k PDUs sent, then 10k responses drained)
|
||||
# IfxPy: 259 ms (10k round-trips, each blocking on response)
|
||||
```
|
||||
|
||||
### Containerized deployment
|
||||
@ -62,7 +62,7 @@ IfxPy's deployment surface is dramatically larger:
|
||||
- 92 MB IBM OneDB Client tarball
|
||||
- `setuptools < 58` build pin
|
||||
- `LD_LIBRARY_PATH` configuration for four directories
|
||||
- `libcrypt.so.1` (deprecated 2018 — missing on Arch, Fedora 35+, RHEL 9)
|
||||
- `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9)
|
||||
- C compiler in the build image
|
||||
|
||||
For slim images, multi-stage builds, FaaS deployments, or anywhere build-toolchain-on-the-runtime is friction, `informix-driver` is the only reasonable option.
|
||||
@ -88,13 +88,13 @@ async def main():
|
||||
rows = await cur.fetchall()
|
||||
```
|
||||
|
||||
IfxPy has no async support — every call blocks the event loop. Using IfxPy from FastAPI requires `loop.run_in_executor()` boilerplate, and the thread pool isn't connection-aware so you give up the natural fairness of an async pool.
|
||||
IfxPy has no async support, so every call blocks the event loop. Using IfxPy from FastAPI requires `loop.run_in_executor()` boilerplate, and the thread pool isn't connection-aware so you give up the natural fairness of an async pool.
|
||||
|
||||
## When IfxPy wins
|
||||
|
||||
### Large analytical fetches
|
||||
|
||||
For queries pulling 10k+ rows where per-row decode cost dominates, IfxPy is currently 5–15% faster. The C-level `fetch_tuple` decoder is ~1.1 µs/row; our Python `parse_tuple_payload` is ~2.0 µs/row after Phase 39 (down from ~2.7 before). At 100k rows the gap is ~80 ms wall-clock — meaningful but not disqualifying.
|
||||
For queries pulling 10k+ rows where per-row decode cost dominates, IfxPy is currently 5–15% faster. The C-level `fetch_tuple` decoder is ~1.1 µs/row; our Python `parse_tuple_payload` is ~2.0 µs/row after Phase 39 (down from ~2.7 before). At 100k rows the gap is ~80 ms wall-clock, which is meaningful but not disqualifying.
|
||||
|
||||
The gap is closing phase by phase:
|
||||
|
||||
@ -109,7 +109,7 @@ If you're running analytical reports that pull millions of rows in a single SELE
|
||||
|
||||
### Workloads built around CSDK extensions
|
||||
|
||||
If your existing code uses IBM-specific cursor extensions (`cursor.callproc` with named parameters, IBM's specific scrollable cursor semantics around `last`/`prior`/`relative`, `cursor.set_chunk_size` for fetch tuning), the migration to `informix-driver` is straightforward but not zero-cost. We support the core PEP 249 surface plus our own scrollable cursor API — see [the migration guide](/how-to/migrate-from-ifxpy/).
|
||||
If your existing code uses IBM-specific cursor extensions (`cursor.callproc` with named parameters, IBM's specific scrollable cursor semantics around `last`/`prior`/`relative`, `cursor.set_chunk_size` for fetch tuning), the migration to `informix-driver` is straightforward but not zero-cost. We support the core PEP 249 surface plus our own scrollable cursor API. See [the migration guide](/how-to/migrate-from-ifxpy/).
|
||||
|
||||
## Methodology
|
||||
|
||||
@ -117,7 +117,7 @@ Benchmarks are pytest-benchmark fixtures in `tests/benchmarks/compare/` against
|
||||
|
||||
Reported numbers are **median over 10+ rounds**, with IQR included. Why median over mean: the first round of any run includes JIT warmup, page-cache miss, and a TCP slow-start round-trip. The mean is contaminated by these one-shot costs in a way that misrepresents steady-state behavior. Median + IQR is what we report.
|
||||
|
||||
IfxPy's IQR on the 100k-row SELECT is ~21% (Docker→host loopback noise, plus the C extension's allocation patterns). Our IQR is ~0.2%. The headline 1.15× ratio at 100k rows is partly that noise — a fair reading is "5–15% slower than IfxPy on large fetches", and the lower bound may already be within measurement noise.
|
||||
IfxPy's IQR on the 100k-row SELECT is ~21% (Docker→host loopback noise, plus the C extension's allocation patterns). Our IQR is ~0.2%. The headline 1.15× ratio at 100k rows is partly that noise. A fair reading is "5–15% slower than IfxPy on large fetches", and the lower bound may already be within measurement noise.
|
||||
|
||||
To reproduce:
|
||||
|
||||
@ -146,4 +146,4 @@ Use IfxPy when:
|
||||
- You're running large analytical SELECTs and the 5–15% decode-side gap matters
|
||||
- You're constrained to Python ≤ 3.11 anyway
|
||||
|
||||
For everything else — the cost-benefit favors `pip install informix-driver`.
|
||||
For everything else, the cost-benefit favors `pip install informix-driver`.
|
||||
|
||||
@ -8,7 +8,7 @@ sidebar:
|
||||
|
||||
The existing tools were not my style.
|
||||
|
||||
Every Informix driver in any language — `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix` — wraps either IBM's C Client SDK or the JDBC JAR. To our knowledge `informix-driver` is the **first pure-socket Informix driver in any language**.
|
||||
Every Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That covers `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge `informix-driver` is the **first pure-socket Informix driver in any language**.
|
||||
|
||||
## The problem with IBM's C SDK
|
||||
|
||||
@ -19,19 +19,19 @@ The IBM Informix Client SDK (CSDK), now packaged as part of OneDB Client, is a 9
|
||||
- Permissive `CFLAGS` for the C extension build
|
||||
- Manual download of the 92 MB ODBC tarball
|
||||
- Four `LD_LIBRARY_PATH` directories
|
||||
- `libcrypt.so.1` — deprecated in 2018, missing on Arch, Fedora 35+, RHEL 9
|
||||
- `libcrypt.so.1`, deprecated in 2018 and missing on Arch, Fedora 35+, RHEL 9
|
||||
|
||||
For containerized deployments, ETL pipelines, FastAPI services, or anywhere Python lives and IBM's C SDK is friction, the friction compounds. `informix-driver`'s install is `pip install informix-driver` (`import informix_db` — the distribution name dodges PyPI's 2008-vintage `informixdb` package, the import name is what you'd expect). The wheel is ~50 KB. There are zero runtime dependencies.
|
||||
For containerized deployments, ETL pipelines, FastAPI services, or anywhere Python lives and IBM's C SDK is friction, the friction compounds. `informix-driver`'s install is `pip install informix-driver` (`import informix_db`; the distribution name dodges PyPI's 2008-vintage `informixdb` package, while the import name is what you'd expect). The wheel is ~50 KB. There are zero runtime dependencies.
|
||||
|
||||
## What it does
|
||||
|
||||
`informix-driver` opens a TCP socket to an Informix server's SQLI listener and speaks the wire protocol directly — the same protocol IBM's JDBC driver uses, the same protocol the CSDK speaks under the hood. No native code is in the thread of execution.
|
||||
`informix-driver` opens a TCP socket to an Informix server's SQLI listener and speaks the wire protocol directly. It is the same protocol IBM's JDBC driver uses, and the same one the CSDK speaks under the hood. No native code is in the thread of execution.
|
||||
|
||||
The wire protocol was reverse-engineered through three sources:
|
||||
|
||||
1. **Decompiled IBM JDBC driver** (`com.informix.jdbc.IfxConnection` and friends), used as a clean-room reference for PDU shapes and protocol semantics.
|
||||
2. **Annotated `socat` captures** of real client/server traffic against the IBM Informix Developer Edition Docker image.
|
||||
3. **Differential testing** against `IfxPy` — every codec path is tested against the C driver's behavior on the same data.
|
||||
3. **Differential testing** against `IfxPy`, so every codec path is checked against the C driver's behavior on the same data.
|
||||
|
||||
The result is a PEP 249 compliant driver with a sync API, an async API (FastAPI / asyncio compatible), a connection pool, TLS support, smart-LOB read/write, scrollable cursors, fast-path stored procedure invocation, and bulk-insert / bulk-fetch performance within ~10–60% of the C driver depending on workload.
|
||||
|
||||
@ -76,12 +76,12 @@ Every finding from a system-wide failure-mode audit (data correctness, wire safe
|
||||
|
||||
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
|
||||
|
||||
400+ tests across unit / integration / benchmark suites. The integration suite runs against the official IBM Informix Developer Edition Docker images and passes 414/414 on **all three** of 12.10.FC12W1DE, 14.10.FC7W1DE, and 15.0.1.0.3DE — `make test-matrix` runs the lot.
|
||||
450+ tests across unit / integration / benchmark suites. The integration suite runs against the official IBM Informix Developer Edition Docker images and passes 457/457 on 15.0.1.0.3DE and 14.10.FC7W1DE, and 456/457 on 12.10.FC12W1DE (the single skip is a common table expression, which 12.10 predates). `make test-matrix` runs the lot.
|
||||
|
||||
That matrix exists because it turned out to be needed. A user reported corrupted result sets on Informix 12; the cause was three framing bugs that affected every version including the one we tested against, and they'd survived because no fixture used the affected types. Testing one server and inferring the rest is how that happens.
|
||||
|
||||
## Read next
|
||||
|
||||
- **[Install & first query →](/start/quickstart/)** — five minutes from `pip install` to a real SELECT against a Docker-hosted Informix.
|
||||
- **[Compared to IfxPy →](/start/vs-ifxpy/)** — full head-to-head benchmarks, methodology, and reproduction.
|
||||
- **[Architecture →](/explain/architecture/)** — how the layers stack: socket, framing, codec, resultset, cursor.
|
||||
- **[Install & first query →](/start/quickstart/)**: five minutes from `pip install` to a real SELECT against a Docker-hosted Informix.
|
||||
- **[Compared to IfxPy →](/start/vs-ifxpy/)**: full head-to-head benchmarks, methodology, and reproduction.
|
||||
- **[Architecture →](/explain/architecture/)**: how the layers stack, from socket through framing, codec, resultset, and cursor.
|
||||
|
||||
@ -243,7 +243,7 @@
|
||||
color: var(--sl-color-gray-2);
|
||||
}
|
||||
|
||||
/* Supported Systems "joint" badge — appears below every page's footer */
|
||||
/* Supported Systems "joint" badge, appears below every page's footer */
|
||||
.ifx-ss-badge {
|
||||
margin-top: 3.5rem;
|
||||
padding: 0;
|
||||
@ -337,7 +337,7 @@
|
||||
* ============================================================ */
|
||||
|
||||
@media (max-width: 640px) {
|
||||
/* Defensive guard against any descendant forcing horizontal scroll —
|
||||
/* Defensive guard against any descendant forcing horizontal scroll:
|
||||
the wire-dump's white-space: pre content was overflowing the
|
||||
hero column and propagating up to the page. overflow-x: hidden
|
||||
on .ifx-hero contains it without affecting page-level scroll. */
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
/*
|
||||
* informix-driver docs theme
|
||||
* - Charcoal base (no purple gradients, ever)
|
||||
* - Amber accent — CRT-monitor nod, distinct from sibling sites' cyan
|
||||
* - Amber accent, a CRT-monitor nod distinct from sibling sites' cyan
|
||||
* - Inter for body, IBM Plex Mono for technical bytes
|
||||
*/
|
||||
|
||||
@ -82,7 +82,7 @@
|
||||
--sl-color-hairline-shade: rgba(120, 80, 12, 0.28);
|
||||
}
|
||||
|
||||
/* Tighten heading rhythm — Starlight defaults are a touch loose for technical docs */
|
||||
/* Tighten heading rhythm; Starlight defaults are a touch loose for technical docs */
|
||||
.sl-markdown-content h2 {
|
||||
margin-top: 2.5rem;
|
||||
border-top: 1px solid var(--sl-color-hairline-light);
|
||||
@ -104,7 +104,7 @@
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
/* Tables: dense, technical, with amber column rules — for type-mapping & benchmark tables */
|
||||
/* Tables: dense, technical, amber column rules, for type-mapping & benchmark tables */
|
||||
.sl-markdown-content table {
|
||||
border-collapse: collapse;
|
||||
font-variant-numeric: tabular-nums;
|
||||
@ -124,7 +124,7 @@
|
||||
padding: 0.5rem 0.75rem;
|
||||
}
|
||||
|
||||
/* Anchor links — underline, no rainbow */
|
||||
/* Anchor links: underline, no rainbow */
|
||||
.sl-markdown-content a:not(.sl-anchor-link) {
|
||||
text-decoration: underline;
|
||||
text-decoration-color: var(--sl-color-accent);
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user