Ryan Malloy
a42dc5c5de
Opt-in via conn.cursor(scrollable=True). Opens the cursor with
SQ_SCROLL (24) before SQ_OPEN (6), keeps it open server-side, and
sends SQ_SFETCH (23) per scroll call instead of materializing the
result set up-front.
User-facing API is identical to Phase 17's in-memory scroll
(fetch_first/last/prior/absolute/relative, scroll, rownumber).
Only the internal mechanism differs:
| feature | default | scrollable=True
|-------------------|------------------|------------------
| memory | all rows | one row at a time
| round-trips/fetch | 0 (after NFETCH) | 1 per call
| cursor lifetime | closed after exec| open until close()
| best for | sequential iter | random access on
| huge result sets
Wire format (verified against JDBC ScrollProbe capture):
* SQ_SFETCH: [short SQ_ID=4][int 23][short scrolltype]
[int target][int bufSize=4096][short SQ_EOT]
scrolltype: 1=NEXT, 4=LAST, 6=ABSOLUTE
* SQ_SCROLL (24): emitted between CURNAME and SQ_OPEN
* SQ_TUPID (25): response tag with 1-indexed row position;
authoritative source for client-side position tracking
Position tracking uses the server's SQ_TUPID rather than client-
computed indexes. Total row count discovered lazily via SFETCH(LAST)
when negative absolute indexing requires it; cached in
_scroll_total_rows.
Trap on the way: initial SFETCH used SHORT for bufSize → server
hung silently. Same SHORT-vs-INT diagnostic pattern as Phase 4.x's
CURNAME+NFETCH. Captured JDBC trace, byte-diffed against ours,
found the mismatch (bufSize is INT in modern Informix per
isXPSVER8_40 / is2GBFetchBufferSupported).
Tests: 14 integration tests in test_scroll_cursor_server.py
covering lifecycle, sequential fetch, fetch_first/last/prior/
absolute/relative, negative indexing, scroll, empty result sets,
past-end, and random-access on a 100-row result set.
Total: 69 unit + 191 integration = 260 tests.
91 lines
6.7 KiB
Markdown
91 lines
6.7 KiB
Markdown
# Changelog
|
|
|
|
All notable changes to `informix-db`. Versioning is [CalVer](https://calver.org/) — `YYYY.MM.DD` for date-based releases, `YYYY.MM.DD.N` for same-day post-releases per PEP 440.
|
|
|
|
## 2026.05.04.2 — Server-side scrollable cursors
|
|
|
|
### Added
|
|
|
|
- **Server-side scrollable cursors** (Phase 18): opt in via `conn.cursor(scrollable=True)`. The cursor opens with `SQ_SCROLL` (24) before `SQ_OPEN` (6), the result set stays materialized server-side, and each scroll method sends `SQ_SFETCH` (23) to fetch one row at a time. Use this for huge result sets where in-memory materialization would be wasteful.
|
|
|
|
The user-facing API is identical to Phase 17's in-memory scroll (`fetch_first`, `fetch_last`, `fetch_prior`, `fetch_absolute`, `fetch_relative`, `scroll`, `rownumber`); only the internal mechanism differs:
|
|
|
|
| | Default cursor | `scrollable=True` |
|
|
|---|---|---|
|
|
| Memory | All rows materialized | One row at a time |
|
|
| Network round-trips per fetch | 0 (after initial NFETCH) | 1 (one SFETCH per call) |
|
|
| Cursor lifetime | Closed after `execute()` | Open until `close()` |
|
|
| Best for | Moderate result sets, sequential iteration | Huge result sets, random access |
|
|
|
|
Implementation discovers total row count lazily via SFETCH(LAST=4) when negative absolute indexing requires it; result is cached in `_scroll_total_rows`. Position tracking is authoritative from the server's `SQ_TUPID` (25) tag, not client-computed.
|
|
|
|
### Wire-protocol details
|
|
|
|
- `SQ_SFETCH` (23): `[short SQ_ID=4][int 23][short scrolltype][int target][int bufSize=4096][short SQ_EOT]`. scrolltype values: 1=NEXT, 4=LAST, 6=ABSOLUTE.
|
|
- `SQ_SCROLL` (24): emitted between CURNAME and SQ_OPEN to mark the cursor as scrollable.
|
|
- `SQ_TUPID` (25): server response carrying the 1-indexed row position the server just delivered. `[short 25][int rowID]`.
|
|
|
|
The trap on the way: I initially used SHORT for `bufSize` and the server hung silently — same SHORT-vs-INT diagnostic pattern as Phase 4.x's CURNAME+NFETCH. Captured a JDBC trace, byte-diffed against ours, found the mismatch.
|
|
|
|
### Tests
|
|
|
|
14 new integration tests in `test_scroll_cursor_server.py`. Total: **69 unit + 191 integration = 260 tests**.
|
|
|
|
## 2026.05.04.1 — Scroll cursors
|
|
|
|
### Added
|
|
|
|
- **Scroll cursor API** on `Cursor` (Phase 17):
|
|
- `cur.scroll(value, mode='relative'|'absolute')` — PEP 249 compatible
|
|
- `cur.fetch_first()` / `cur.fetch_last()` — jump to ends
|
|
- `cur.fetch_prior()` — backward step (SQL-standard semantics: from past-end yields the last row)
|
|
- `cur.fetch_absolute(n)` — 0-indexed jump; negative `n` indexes from the end
|
|
- `cur.fetch_relative(n)` — n-step from current position
|
|
- `cur.rownumber` — current 0-indexed position (None if before-first or no result set)
|
|
|
|
In-memory implementation — no new wire-protocol; the existing materialized result set in `cur._rows` is now indexed rather than iterated. For server-side scroll over huge result sets, `SQ_SFETCH` (tag 23) would be needed — Phase 18 if anyone hits the in-memory ceiling.
|
|
|
|
### Tests
|
|
|
|
14 new integration tests in `test_scroll_cursor.py`. Total: **69 unit + 177 integration = 246 tests**.
|
|
|
|
## 2026.05.04 — Library completion
|
|
|
|
The Phase 0 ambition — first pure-Python Informix SQLI driver — reaches feature completeness. Adds async, TLS, connection pool, smart-LOBs, fast-path RPC, composite UDTs.
|
|
|
|
### Added
|
|
|
|
- **Async API** (`informix_db.aio`) — `AsyncConnection`, `AsyncCursor`, `AsyncConnectionPool` for FastAPI / aiohttp / asyncio. Each blocking I/O call is offloaded to a worker thread via `asyncio.to_thread`; event loop never blocks.
|
|
- **Connection pool** (`informix_db.create_pool`) — thread-safe with min/max sizing, lazy growth, health-check on acquire, error-aware eviction.
|
|
- **TLS** — `tls=True` for self-signed dev servers, `tls=ssl.SSLContext` for production. Wrapping happens in `IfxSocket` so the rest of the protocol layer is unaware.
|
|
- **Smart-LOBs** (BLOB / CLOB) — full read/write end-to-end via `cursor.read_blob_column()` / `cursor.write_blob_column()` using the server's `lotofile` / `filetoblob` SQL functions intercepted at the `SQ_FILE` (98) protocol level.
|
|
- **Legacy in-row blobs** (BYTE / TEXT) — bind + read via the `SQ_BBIND` / `SQ_BLOB` / `SQ_FETCHBLOB` protocol family.
|
|
- **Fast-path RPC** (`Connection.fast_path_call`) — direct stored-procedure invocation bypassing PREPARE/EXECUTE; routine handles cached per-connection.
|
|
- **Composite UDT recognition** — `ROW`, `SET`, `MULTISET`, `LIST` columns return typed `RowValue` / `CollectionValue` wrappers exposing schema and raw bytes.
|
|
- **Type codecs** — `INTERVAL` (both DAY-TO-FRACTION and YEAR-TO-MONTH families), `DATETIME` (all qualifier ranges), `DECIMAL` / `MONEY` (BCD with sign+exp head byte and asymmetric base-100 complement for negatives), `DATE`, `BOOL`, all integer / float widths, `CHAR` / `VARCHAR` / `LVARCHAR`.
|
|
- **Transactions** — implicit `SQ_BEGIN` before each transaction in non-ANSI logged DBs; transparent no-ops on unlogged DBs.
|
|
- **PEP 249 exception hierarchy** — server `SQLCODE` mapped to the right exception class (`IntegrityError` for duplicate-key violations, `ProgrammingError` for syntax errors, etc.).
|
|
|
|
### Documentation
|
|
|
|
- [`README.md`](README.md) — overview and quick-start
|
|
- [`docs/USAGE.md`](docs/USAGE.md) — practical recipes and migration guide
|
|
- [`docs/PROTOCOL_NOTES.md`](docs/PROTOCOL_NOTES.md) — byte-level wire-format reference
|
|
- [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md) — phase-by-phase architectural decisions, with the *why* preserved
|
|
- [`docs/JDBC_NOTES.md`](docs/JDBC_NOTES.md) — index into the decompiled IBM JDBC reference
|
|
- [`docs/CAPTURES/`](docs/CAPTURES/) — annotated socat hex-dump captures
|
|
|
|
### Test coverage
|
|
|
|
232 tests total: **69 unit + 163 integration**. Unit tests run with no external dependencies; integration tests run against the IBM Informix Developer Edition Docker image.
|
|
|
|
### Known gaps (deferred)
|
|
|
|
- **Full ROW/COLLECTION recursive parsing**: Phase 12 ships type recognition + raw-bytes wrapper. Parsing the textual representation into typed Python tuples/sets/lists is deferred — most workloads can use SQL projections (`SELECT row_col.fieldname FROM tbl`) instead.
|
|
- **UDT parameter encoding for fast-path**: scalar params/returns work; passing a 72-byte BLOB locator as a UDT param requires extending the SQ_BIND encoder with the extended_owner/extended_name preamble for type > 18.
|
|
- **Native async I/O**: Phase 16 ships a thread-pool wrapper that's functionally equivalent for typical FastAPI workloads. Native async (asyncpg-style transport abstraction) would be Phase 17 if a real workload needs it.
|
|
|
|
## 2026.05.02 — Phase 1: connection lifecycle
|
|
|
|
Initial release. `connect()` / `close()` works end-to-end. Cursor / execute / fetch arrived in Phase 2 (subsequent commits within the same session).
|