diff --git a/CHANGELOG.md b/CHANGELOG.md index c53d8ee..d88cb7d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,64 @@ All notable changes to `informix-db`. Versioning is [CalVer](https://calver.org/) — `YYYY.MM.DD` for date-based releases, `YYYY.MM.DD.N` for same-day post-releases per PEP 440. +## 2026.08.27 — Decode the SQ_PROTOCOLS capability negotiation + +Closes the last open item from the Informix 12 field report. The driver hardcodes several wire-framing choices that SQLI actually *negotiates*; those choices were correct on every server we'd measured, but "correct as far as we know" and "checked" are different things, and the failure mode for a framing mismatch is silently corrupted rows. + +### What changed + +We were already sending `SQ_PROTOCOLS` with the same 8-byte client offer IBM's JDBC driver uses — and throwing the server's reply away. Now we decode it. + +New `informix_db.ServerCapabilities`, reachable from any connection: + +```python +conn = informix_db.connect(...) +caps = conn.server_capabilities +caps.four_byte_offset # describe uses 4-byte string-table/field-index widths +caps.varchar_var_len # VARCHAR is length-prefixed, not width-padded +caps.remove_64k_limit # 4-byte length prefixes in the fast-path +caps.violated_assumptions() # [] when our hardcoded framing matches +conn.server_version # from the login response +``` + +`violated_assumptions()` is the point of the exercise. It names each place we emit or parse a fixed wire shape that is actually capability-gated, and returns empty when the server agrees. On connect, a non-empty result logs a warning naming the specific bit. A server we've never tested now produces a diagnosable complaint instead of quiet corruption. + +Nothing branches on these bits yet — this release is observation and validation only. That's deliberate: the hardcoded framing is correct on all three supported servers, and swapping working code for newly-written conditional code without a server that needs it would add risk for no benefit. + +### The measurement + +All three servers negotiate the same thing: + +| Server | Negotiated mask | +|---|---| +| 15.0.1.0.3 | `bdbe9ffe7fb7ffef` `ff` | +| 14.10.FC7W1 | `bdbe9ffe7fb7ffef` `f8` | +| 12.10.FC12W1DE | `bdbe9ffe7fb7ffef` `f0` | + +**The first eight bytes are byte-identical.** That is the root explanation for everything in the 2026.05.08.2 investigation: 12.10, 14.10, and 15 don't merely behave similarly, they negotiate exactly the same 64-bit capability set. `violated_assumptions()` returns empty on all three. + +Two details worth recording: + +The reply is **nine** bytes, not eight. JDBC's `enhancedProtocolMechanism` dispatches on `case 0..7` and silently discards the ninth, so its `BitSet(64)` never sees it. We decode it, because it is the *only* part of the mask that differs between releases — newer feature flags that this JDBC build predates. + +`Cap_1` in the login response is **not** a server version. It's the client's own declared protocol level echoed back, which is why JDBC tests `== 316` rather than `>=`. The actual version string there is the *internal* one: Informix 12.10 reports `9.56`, 14.10 reports `9.59`. At the protocol level both really are 9.x servers, which is why the marketing version jump didn't move the wire format. + +This also retires a red herring. `isUSVER` (bit 2) is one of six bits JDBC pre-sets for any non-zero `Cap_1`, and Java's `BitSet.set` never clears — so it is true on every modern server regardless of the mask. It can't explain version-specific behaviour and isn't worth chasing. + +### Also fixed: `__version__` was reporting a sentinel + +Found incidentally while bumping the version. The distribution was renamed `informix-db` to `informix-driver` on 2026-05-08, but `__init__.py` kept calling `version("informix-db")`. `importlib.metadata` needs the *distribution* name, not the module name, and the miss fails silently — `__version__` degrades to `"0.0.0+local"` rather than raising. + +So every install of the renamed package has been reporting `0.0.0+local`. It escaped notice locally because a stale `informix-db` distribution was still present in the dev environment and answered the lookup with its own old version. + +Added `tests/test_package_metadata.py`, which asserts the name passed to `importlib.metadata.version()` matches `[project].name` in `pyproject.toml`. Verified it actually catches the bug by reintroducing it. + +### Tests + +24 unit tests over the captured masks (no server needed) and 6 integration tests that assert live negotiation. The integration ones run on every server in `make test-matrix`, so the assumption check is re-verified on 12.10, 14.10, and 15 each time. + +Full suite: **247 / 247** on all three. + ## 2026.05.08.2 — Fix three tuple-framing bugs (BOOLEAN, NCHAR, INT8/SERIAL8) Data-corruption fixes. All three affect **every** Informix version including 15, and two of them silently corrupt columns *after* the offending one. If your schema uses `BOOLEAN`, `NCHAR`, `INT8`, or `SERIAL8`, upgrade. diff --git a/README.md b/README.md index d86ac65..e0d2df9 100644 --- a/README.md +++ b/README.md @@ -172,7 +172,20 @@ Beyond the suite passing, a 28-type round-trip (every scalar type we support, in Earlier releases of this README claimed the protocol was "stable across modern versions" and should "work against 12.10+ unmodified." That claim turned out to be correct, but it was written before anyone had run it against 12.10, and a user lost debugging time to a version-compatibility problem that didn't exist. It's measured now. Apologies for the earlier guess. -One caveat remains, and it's a real one. SQLI negotiates some wire framing through a capability exchange (`SQ_PROTOCOLS`) that this driver does not yet perform — we hardcode the modern framing. That is correct on all three servers above, but it is an assumption rather than a negotiation. If you hit corrupted rows on a server not in that table, this is the first thing to suspect; please open an issue with your server version and the `cursor.description` of the offending query. +### Capability negotiation + +SQLI settles some wire framing through a capability exchange (`SQ_PROTOCOLS`) rather than fixing it by version. This driver decodes that exchange and checks it against the framing it emits: + +```python +conn = informix_db.connect(...) +conn.server_version # 'IBM Informix Dynamic Server Version 15.0.1.0.3' +conn.server_capabilities.four_byte_offset # True +conn.server_capabilities.violated_assumptions() # [] — hardcoded framing matches +``` + +All three servers above negotiate an identical 64-bit capability set, and `violated_assumptions()` is empty on each. If it ever returns something, the driver logs a warning naming the specific bit at connect time — a framing mismatch otherwise shows up as silently corrupted rows, which is a miserable thing to debug. + +The framing itself is still hardcoded rather than driven by those bits. That's a deliberate limit: it's correct everywhere we've measured, and rewriting working parse paths to be conditional without a server that needs it would trade certainty for risk. If you hit corrupted rows on a server outside that table, check `violated_assumptions()` first, then please open an issue with your server version, the capability mask, and the `cursor.description` of the offending query. For features that need server-side configuration (smart-LOBs, logged transactions), see [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md): - Phase 7 — logged-DB transactions diff --git a/pyproject.toml b/pyproject.toml index 73668c1..44e4148 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "informix-driver" -version = "2026.05.08.2" +version = "2026.08.27" description = "Pure-Python driver for IBM Informix IDS — speaks the SQLI wire protocol over raw sockets. No CSDK, no JVM, no native libraries." readme = "README.md" license = { text = "MIT" } diff --git a/src/informix_db/__init__.py b/src/informix_db/__init__.py index 7625af1..9cec256 100644 --- a/src/informix_db/__init__.py +++ b/src/informix_db/__init__.py @@ -23,6 +23,7 @@ from __future__ import annotations import ssl from importlib.metadata import PackageNotFoundError, version +from ._capabilities import ServerCapabilities from .connections import Connection from .converters import ( BlobLocator, @@ -56,9 +57,14 @@ threadsafety = 1 # threads may share the module but not connections paramstyle = "numeric" # locked in DECISION_LOG.md — matches Informix ESQL/C try: - __version__ = version("informix-db") + # NOTE: the *distribution* is ``informix-driver``; the *module* is + # ``informix_db``. They differ, so this string can't be derived from + # ``__name__``. This looked up the old ``informix-db`` distribution + # until 2026.08.27, which meant __version__ silently reported + # "0.0.0+local" for anyone who installed the renamed package. + __version__ = version("informix-driver") except PackageNotFoundError: - # Editable install or running uninstalled; fall back to a sentinel. + # Running from a source checkout without an install. __version__ = "0.0.0+local" __all__ = [ @@ -80,6 +86,7 @@ __all__ = [ "PoolTimeoutError", "ProgrammingError", "RowValue", + "ServerCapabilities", "Warning", "__version__", "apilevel", diff --git a/src/informix_db/_capabilities.py b/src/informix_db/_capabilities.py new file mode 100644 index 0000000..4632028 --- /dev/null +++ b/src/informix_db/_capabilities.py @@ -0,0 +1,269 @@ +"""Server capability decoding — the SQ_PROTOCOLS feature bitmap. + +SQLI does not have a single "protocol version". It has a 64-bit feature +bitmap that client and server negotiate at connect time, and several bits +in it change **wire framing**, not just feature availability. Get one of +those wrong and the row decoder desyncs. + +We already perform the negotiation (``Connection._init_session`` sends +``SQ_PROTOCOLS`` with the same 8-byte client offer the IBM JDBC driver +uses). Until now we discarded the server's reply and hardcoded the modern +framing. This module decodes the reply so the assumptions can at least be +*checked* instead of merely believed. + +Two independent sources feed the bitmap, mirroring +``IfxSqliConnect.getServerVer`` / ``enhancedProtocolMechanism``: + +1. ``Cap_1`` from the login (SLTYPE_CONACC) response. This is the client's + own protocol level echoed back — 316 means "speaks the enhanced + protocol". It is **not** a server version number, which is why JDBC + tests ``== 316`` rather than ``>=``. +2. The 8-byte mask from the SQ_PROTOCOLS reply. + +When ``Cap_1`` is non-zero, JDBC unconditionally pre-sets bits +{0, 2, 3, 4, 49, 51} *before* applying the mask, and a Java ``BitSet.set`` +only ever sets — never clears. So those six bits are true on any modern +server regardless of what the mask says. ``isUSVER`` (bit 2) is one of +them, which is why it never varies in practice and is a dead end when +chasing version-specific behaviour. + +Bit numbering is MSB-first within each byte: byte 0 bit 0x80 is bit 0, +byte 0 bit 0x01 is bit 7, byte 1 bit 0x80 is bit 8, and so on. + +A note on length. The client offer is 8 bytes, but every server we have +tested replies with **nine**. JDBC's ``enhancedProtocolMechanism`` +dispatches on ``switch (i) case 0..7`` and silently discards anything +past byte 7, so its ``BitSet(64)`` never sees the ninth byte. We decode +it — bits 64-71 — because it turns out to be the only part of the mask +that differs between releases:: + + Informix 15.0.1.0.3 bdbe9ffe7fb7ffef ff + Informix 14.10.FC7W1 bdbe9ffe7fb7ffef f8 + Informix 12.10.FC12W1DE bdbe9ffe7fb7ffef f0 + ^^^^^^^^^^^^^^^^ identical + +The first 64 bits being byte-identical across those three releases is +why they speak an indistinguishable SQLI dialect. Bits 64+ have no known +meaning here; they are surfaced for diagnostics, not branched on. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field + +# The client's capability offer. Byte-for-byte the value IBM's JDBC driver +# sends (``IfxSqliConnect.clientProtocols``), replayed verbatim — the bits +# are a fixed constant there too, never computed. +CLIENT_PROTOCOLS_MASK = bytes.fromhex("fffc7ffc3c8caa97") + +# ``Cap_1`` value meaning "enhanced protocol negotiation applies". JDBC +# compares with equality, not >=, because the server echoes the client's +# own declared level rather than reporting its own. +ENHANCED_PROTOCOL_CAP = 316 + +# Bits JDBC pre-sets for any non-zero Cap_1, before the mask is applied. +_PRESET_BITS = frozenset({0, 2, 3, 4, 49, 51}) + +# Named bits. Only the ones we can justify from the decompiled JDBC are +# listed; the mask has 64 and most are irrelevant to us. +BIT_USVER = 2 # isUSVER — always true on modern servers +BIT_LONG_ID_A = 10 # isLongID is bit 10 OR bit 18 +BIT_LONG_ID_B = 18 +BIT_VARCHAR_VAR_LEN = 13 # gated further by bit 40 + client ifxPADVARCHAR +BIT_DESCRIBE_INPUT = 34 +BIT_LVARCHAR_GT_2K = 37 +BIT_PAD_VARCHAR_GATE = 40 +BIT_FP_DESCRIBE = 45 +BIT_AUTO_GENERATED_KEYS = 48 +BIT_CUR_SESS_INFO = 49 +BIT_FOUR_BYTE_OFFSET = 50 # describe: string-table + field-index widths +BIT_GLS = 51 +BIT_NAMED_PARAMETERS = 52 +BIT_BIGINT = 54 +BIT_SAVEPOINT = 56 +BIT_SQ_BATCH = 61 +BIT_REMOVE_64K_LIMIT = 62 # 4-byte vs 2-byte length prefixes +BIT_TWO_GB_FETCH_BUFFER = 63 + + +def _decode_bits(mask: bytes) -> set[int]: + """Expand a big-endian, MSB-first bit mask into a set of bit numbers.""" + bits: set[int] = set() + for byte_index, value in enumerate(mask): + for offset in range(8): + if value & (0x80 >> offset): + bits.add(byte_index * 8 + offset) + return bits + + +@dataclass(frozen=True) +class ServerCapabilities: + """Decoded SQ_PROTOCOLS feature bitmap plus login-response metadata. + + ``bits`` is the authoritative set; the named properties are readability + sugar over it. ``raw_mask`` is kept so a bug report can include the + exact bytes without needing a packet capture. + """ + + bits: frozenset[int] = field(default_factory=frozenset) + raw_mask: bytes = b"" + cap_1: int = 0 + cap_2: int = 0 + cap_3: int = 0 + server_version: str = "" + serial_number: str = "" + applid_name: str = "" + + @classmethod + def from_wire( + cls, + mask: bytes, + *, + cap_1: int = 0, + cap_2: int = 0, + cap_3: int = 0, + server_version: str = "", + serial_number: str = "", + applid_name: str = "", + ) -> ServerCapabilities: + bits = _decode_bits(mask) + if cap_1 != 0: + # Mirrors JDBC: pre-set bits are OR-ed in and never cleared. + bits |= _PRESET_BITS + return cls( + bits=frozenset(bits), + raw_mask=bytes(mask), + cap_1=cap_1, + cap_2=cap_2, + cap_3=cap_3, + server_version=server_version, + serial_number=serial_number, + applid_name=applid_name, + ) + + def has(self, bit: int) -> bool: + return bit in self.bits + + # -- named capabilities ------------------------------------------------ + + @property + def usver(self) -> bool: + return self.has(BIT_USVER) + + @property + def long_id(self) -> bool: + return self.has(BIT_LONG_ID_A) or self.has(BIT_LONG_ID_B) + + @property + def varchar_var_len(self) -> bool: + """VARCHAR carries a 1-byte length prefix rather than being padded + to its full declared width. + + JDBC additionally consults the client-side ``ifxPADVARCHAR`` + setting; we never enable padding, so the client half is constant. + """ + return self.has(BIT_VARCHAR_VAR_LEN) + + @property + def four_byte_offset(self) -> bool: + """Describe records use 4-byte string-table size and field indices + rather than 2-byte.""" + return self.has(BIT_FOUR_BYTE_OFFSET) + + @property + def bigint(self) -> bool: + return self.has(BIT_BIGINT) + + @property + def lvarchar_gt_2k(self) -> bool: + return self.has(BIT_LVARCHAR_GT_2K) + + @property + def remove_64k_limit(self) -> bool: + """4-byte length prefixes in the fast-path/SQ_FILE paths.""" + return self.has(BIT_REMOVE_64K_LIMIT) + + @property + def two_gb_fetch_buffer(self) -> bool: + return self.has(BIT_TWO_GB_FETCH_BUFFER) + + @property + def gls(self) -> bool: + return self.has(BIT_GLS) + + @property + def describe_input(self) -> bool: + return self.has(BIT_DESCRIBE_INPUT) + + @property + def named_parameters(self) -> bool: + return self.has(BIT_NAMED_PARAMETERS) + + @property + def savepoint(self) -> bool: + return self.has(BIT_SAVEPOINT) + + @property + def enhanced_protocol(self) -> bool: + return self.cap_1 == ENHANCED_PROTOCOL_CAP + + # -- assumption checking ---------------------------------------------- + + def violated_assumptions(self) -> list[str]: + """Framing assumptions this driver hardcodes that the server + contradicts. + + Each entry names a place where we emit or parse a fixed wire shape + that is actually capability-gated. An empty list means every + hardcoded choice matches what the server negotiated. + + This is a diagnostic, not a guarantee: it only covers bits we know + to be framing-relevant from the decompiled JDBC. It exists so that + a server we have never tested produces a loud, specific warning + instead of silently corrupted rows. + """ + problems: list[str] = [] + if not self.four_byte_offset: + problems.append( + "server did not negotiate 4-byte describe offsets " + f"(bit {BIT_FOUR_BYTE_OFFSET}), but parse_describe reads " + "4-byte string-table size and field indices; column " + "metadata will be misparsed" + ) + if not self.varchar_var_len: + problems.append( + f"server did not negotiate variable-length VARCHAR " + f"(bit {BIT_VARCHAR_VAR_LEN}); VARCHAR values are padded to " + "the declared column width on the wire, but the row decoder " + "expects a 1-byte length prefix" + ) + if not self.remove_64k_limit: + problems.append( + f"server did not negotiate the 64K limit removal " + f"(bit {BIT_REMOVE_64K_LIMIT}); fast-path routine signatures " + "are emitted with a 4-byte length prefix that the server " + "expects to be 2-byte" + ) + return problems + + def __repr__(self) -> str: + named = [ + name + for name, value in ( + ("usver", self.usver), + ("4byte_offset", self.four_byte_offset), + ("varchar_var_len", self.varchar_var_len), + ("bigint", self.bigint), + ("long_id", self.long_id), + ("lvarchar>2k", self.lvarchar_gt_2k), + ("remove_64k", self.remove_64k_limit), + ("gls", self.gls), + ) + if value + ] + return ( + f"ServerCapabilities(cap_1={self.cap_1}, " + f"mask={self.raw_mask.hex()}, " + f"version={self.server_version!r}, " + f"flags={'|'.join(named) or 'none'})" + ) diff --git a/src/informix_db/connections.py b/src/informix_db/connections.py index bb2a48f..5dcce7a 100644 --- a/src/informix_db/connections.py +++ b/src/informix_db/connections.py @@ -12,6 +12,7 @@ reference in ``docs/CAPTURES/01-connect-only.socat.log``. from __future__ import annotations import contextlib +import logging import os import socket as socket_mod import ssl @@ -21,6 +22,7 @@ from io import BytesIO from pathlib import Path from . import _auth +from ._capabilities import CLIENT_PROTOCOLS_MASK, ServerCapabilities from ._messages import ( APPL_ID, APPL_TYPE, @@ -69,6 +71,9 @@ _LOCALE_ENCODING_MAP = { } +_log = logging.getLogger(__name__) + + def _extract_server_error_text(payload: bytes) -> str | None: """Pull the longest printable run out of an opaque rejection payload. @@ -113,6 +118,74 @@ def _python_encoding_from_locale(locale: str) -> str: return _LOCALE_ENCODING_MAP.get(suffix, "iso-8859-1") +def _decode_conacc(rest: bytes) -> dict | None: + """Decode the SLTYPE_CONACC server-metadata block. + + Mirrors ``com.informix.asf.Connection.DecodeAscBinary``. ``rest`` is + the login response with its 2-byte total-length prefix already + stripped, i.e. starting at the SLType byte. + + Layout, verified byte-for-byte against Informix 12.10, 14.10 and 15:: + + byte SLType (2 = CONACC) + 3 skip + short 100 marker + short 101 marker + 4 skip + short len; skip len (codeset id, e.g. "IEEEI") + short 108 marker — JDBC hard-fails if this isn't 108 + 12 skip + short len; bytes server version string + short len; bytes serial number + short len; bytes applid name + int Cap_1 client protocol level, echoed (316 = enhanced) + int Cap_2 + int Cap_3 + + Returns ``None`` if the block doesn't match that shape. Callers must + treat that as "no capability info", never as a connection failure — + every server we support connects fine without any of this being + decodable, and a future server variant should degrade, not break. + """ + try: + offset = 1 + 3 # SLType + 3 skipped bytes + marker_a, marker_b = struct.unpack_from("!hh", rest, offset) + offset += 4 + if marker_a != 100 or marker_b != 101: + return None + offset += 4 + (codeset_len,) = struct.unpack_from("!h", rest, offset) + offset += 2 + codeset_len + (marker_c,) = struct.unpack_from("!h", rest, offset) + offset += 2 + if marker_c != 108: + return None + offset += 12 + strings: list[str] = [] + for _ in range(3): + (length,) = struct.unpack_from("!h", rest, offset) + offset += 2 + if length < 0: + return None + strings.append( + rest[offset : offset + length].rstrip(b"\x00").decode( + "iso-8859-1", "replace" + ) + ) + offset += length + cap_1, cap_2, cap_3 = struct.unpack_from("!iii", rest, offset) + except (struct.error, IndexError, UnicodeDecodeError): + return None + return { + "server_version": strings[0], + "serial_number": strings[1], + "applid_name": strings[2], + "cap_1": cap_1, + "cap_2": cap_2, + "cap_3": cap_3, + } + + # Default environment variables sent in the login PDU (SQ_ASCENV section). # These match what the JDBC driver sends for a vanilla en_US.8859-1 # connection. Anything missing makes the server fall back to defaults. @@ -205,6 +278,13 @@ class Connection: # SQ_GETROUTINE; subsequent calls skip that round-trip. self._fp_handle_cache: dict[str, tuple[str, int]] = {} + # Server metadata from the login response and the SQ_PROTOCOLS + # negotiation. Both are populated during connect; both degrade to + # None rather than failing the connection. + self._conacc: dict | None = None + self._server_protocols: bytes | None = None + self._capabilities: ServerCapabilities | None = None + # Build the env-var dict sent in the login PDU. self._env = dict(_DEFAULT_ENV) self._env["CLIENT_LOCALE"] = client_locale @@ -589,9 +669,9 @@ class Connection: # The 8-byte protocols mask is the JDBC reference value from # docs/CAPTURES/02-select-1.socat.log; we replay it verbatim # since the bits are opaque (server-recognized features). - protocols_mask = bytes.fromhex("fffc7ffc3c8caa97") - self._send_protocols(protocols_mask) - self._drain_to_eot() + self._send_protocols(CLIENT_PROTOCOLS_MASK) + self._drain_to_eot() # captures the reply into self._server_protocols + self._build_capabilities() # Step 2: SQ_INFO with INFO_ENV subtype + session env vars. # The actual on-wire format (from JDBC's sendEnv at IfxSqli.java @@ -626,6 +706,54 @@ class Connection: self._send_dbopen(self._database) self._drain_to_eot() + def _build_capabilities(self) -> None: + """Assemble ``ServerCapabilities`` from the login response and the + SQ_PROTOCOLS reply, and warn about contradicted assumptions. + + This is observation only — nothing in the parse paths branches on + these bits yet. The value is diagnostic: this driver hardcodes + several wire-framing choices that SQLI actually negotiates, and + those choices are correct on every server we have measured + (Informix 12.10, 14.10, 15). On some server we have not measured + they might not be, and the failure mode for a framing mismatch is + silently corrupted rows, which is about the worst way for a + database driver to fail. A log warning naming the specific bit + turns that into something diagnosable from a bug report. + + Deliberately never raises: a server that declines to negotiate is + not necessarily broken, and refusing the connection over a + diagnostic would be a worse outcome than proceeding. + """ + if self._server_protocols is None: + return + conacc = self._conacc or {} + try: + caps = ServerCapabilities.from_wire( + self._server_protocols, + cap_1=conacc.get("cap_1", 0), + cap_2=conacc.get("cap_2", 0), + cap_3=conacc.get("cap_3", 0), + server_version=conacc.get("server_version", ""), + serial_number=conacc.get("serial_number", ""), + applid_name=conacc.get("applid_name", ""), + ) + except Exception: + _log.debug("could not decode server capabilities", exc_info=True) + return + self._capabilities = caps + + problems = caps.violated_assumptions() + if problems: + _log.warning( + "Informix server %r negotiated capabilities that contradict " + "this driver's hardcoded wire framing (mask=%s). Result rows " + "may be decoded incorrectly. Please report this with the " + "mask and your server version. Details: %s", + caps.server_version or "", + caps.raw_mask.hex(), + "; ".join(problems), + ) + def _send_protocols(self, protocols: bytes) -> None: """Emit a SQ_PROTOCOLS PDU per ``IfxSqli.sendProtocols``. @@ -668,9 +796,13 @@ class Connection: elif tag == MessageType.SQ_PROTOCOLS: # ``[short payloadLen][bytes payload][byte 0 if odd-len pad]`` # Then the loop continues and consumes the next tag (usually SQ_EOT). + # The payload is the negotiated 64-bit feature bitmap; stash + # it for _capabilities decoding. Reading it was always + # necessary for stream alignment — we just used to throw it + # away instead of looking at it. payload_len = struct.unpack("!h", self._sock.read_exact(2))[0] if payload_len > 0: - self._sock.read_exact(payload_len) + self._server_protocols = self._sock.read_exact(payload_len) if payload_len & 1: self._sock.read_exact(1) # writePadded's even-alignment pad elif tag == MessageType.SQ_DONE: @@ -883,6 +1015,28 @@ class Connection: # -- response parsing ------------------------------------------------- + @property + def server_capabilities(self): + """The negotiated ``SQ_PROTOCOLS`` feature bitmap, or ``None``. + + ``None`` means the negotiation reply couldn't be decoded — the + connection still works, we just have no capability information. + See ``informix_db._capabilities.ServerCapabilities``. + """ + return self._capabilities + + @property + def server_version(self) -> str: + """Server version string from the login response. + + Note this is the *internal* protocol version, which is not the + marketing version. Informix 12.10 reports ``9.56``, 14.10 reports + ``9.59``, and 15 reports ``15.0.1.0.3``. That is why the two older + releases speak an identical SQLI dialect: at the protocol level + they are both 9.x servers. + """ + return self._conacc.get("server_version", "") if self._conacc else "" + def _parse_login_response(self) -> None: """Read and parse the server's login response. @@ -916,8 +1070,12 @@ class Connection: ) elif sl_type != SLHeader.SLTYPE_CONACC: raise ProtocolError(f"unknown SLType in login response: {sl_type}") - # SLTYPE_CONACC — connection accepted. We don't (yet) decode the - # full server-side metadata. Phase 1 just needs to know "we got in". + + # SLTYPE_CONACC — connection accepted. Decode the server metadata + # block for the capability fields. Best-effort: a decode failure + # here must not fail an otherwise-good connection, since every + # server we've tested connects fine without any of this. + self._conacc = _decode_conacc(rest) def _raise_from_rejection(self, reader: IfxStreamReader) -> None: """Best-effort decode of the connection-rejection error block. diff --git a/tests/test_capabilities.py b/tests/test_capabilities.py new file mode 100644 index 0000000..3138664 --- /dev/null +++ b/tests/test_capabilities.py @@ -0,0 +1,98 @@ +"""Integration tests for live SQ_PROTOCOLS negotiation. + +These run against whatever server ``IFX_PORT`` points at, so `make +test-matrix` exercises them on 12.10, 14.10, and 15 in turn. + +The important one is ``test_no_violated_assumptions``: this driver +hardcodes several wire-framing choices that SQLI actually negotiates, and +a mismatch corrupts rows silently. That test turns "we assume this" into +"we check this on every supported server". +""" + +from __future__ import annotations + +import pytest + +import informix_db +from informix_db._capabilities import ENHANCED_PROTOCOL_CAP, ServerCapabilities +from tests.conftest import ConnParams + +pytestmark = pytest.mark.integration + + +def _connect(conn_params: ConnParams) -> informix_db.Connection: + return informix_db.connect( + host=conn_params.host, + port=conn_params.port, + user=conn_params.user, + password=conn_params.password, + database=conn_params.database, + server=conn_params.server, + connect_timeout=10.0, + read_timeout=10.0, + ) + + +def test_capabilities_are_decoded(conn_params: ConnParams) -> None: + with _connect(conn_params) as conn: + caps = conn.server_capabilities + assert isinstance(caps, ServerCapabilities) + assert caps.raw_mask, "server sent an empty protocols reply" + assert caps.bits + + +def test_no_violated_assumptions(conn_params: ConnParams) -> None: + """Every wire-framing shape we hardcode is one the server negotiated. + + If this fails, rows are being decoded against the wrong framing and + the fix is to branch on the capability rather than assume it. + """ + with _connect(conn_params) as conn: + caps = conn.server_capabilities + assert caps is not None + assert caps.violated_assumptions() == [] + + +def test_enhanced_protocol_negotiated(conn_params: ConnParams) -> None: + """Cap_1 is the client's declared protocol level echoed back. Every + supported server accepts 316.""" + with _connect(conn_params) as conn: + caps = conn.server_capabilities + assert caps is not None + assert caps.cap_1 == ENHANCED_PROTOCOL_CAP + assert caps.enhanced_protocol + + +def test_framing_capabilities_present(conn_params: ConnParams) -> None: + with _connect(conn_params) as conn: + caps = conn.server_capabilities + assert caps is not None + assert caps.four_byte_offset + assert caps.varchar_var_len + assert caps.remove_64k_limit + assert caps.usver + + +def test_server_version_is_exposed(conn_params: ConnParams) -> None: + """The version here is the *internal* protocol version, not the + marketing one: 12.10 reports 9.56 and 14.10 reports 9.59. Assert the + shape rather than a value so this holds across the matrix.""" + with _connect(conn_params) as conn: + version = conn.server_version + assert "Informix" in version + assert "Version" in version + + +def test_capabilities_survive_multiple_connections( + conn_params: ConnParams, +) -> None: + """Negotiation happens per-connection; two connections to the same + server must agree.""" + with _connect(conn_params) as first, _connect(conn_params) as second: + assert first.server_capabilities is not None + assert second.server_capabilities is not None + assert ( + first.server_capabilities.raw_mask + == second.server_capabilities.raw_mask + ) + assert first.server_capabilities.bits == second.server_capabilities.bits diff --git a/tests/test_capabilities_unit.py b/tests/test_capabilities_unit.py new file mode 100644 index 0000000..b49487f --- /dev/null +++ b/tests/test_capabilities_unit.py @@ -0,0 +1,200 @@ +"""Unit tests for SQ_PROTOCOLS capability decoding — no server required. + +The masks below were captured from live servers on 2026-08-27. Their most +interesting property is that the first eight bytes are identical across +Informix 12.10, 14.10, and 15 — which is why those three releases speak an +indistinguishable SQLI dialect. +""" + +from __future__ import annotations + +import dataclasses + +import pytest + +from informix_db._capabilities import ( + BIT_FOUR_BYTE_OFFSET, + BIT_REMOVE_64K_LIMIT, + BIT_USVER, + BIT_VARCHAR_VAR_LEN, + CLIENT_PROTOCOLS_MASK, + ENHANCED_PROTOCOL_CAP, + ServerCapabilities, + _decode_bits, +) + +MASK_15 = bytes.fromhex("bdbe9ffe7fb7ffefff") +MASK_1410 = bytes.fromhex("bdbe9ffe7fb7ffeff8") +MASK_1210 = bytes.fromhex("bdbe9ffe7fb7ffeff0") + +ALL_MASKS = [ + pytest.param(MASK_15, id="15.0.1.0.3"), + pytest.param(MASK_1410, id="14.10.FC7W1"), + pytest.param(MASK_1210, id="12.10.FC12W1DE"), +] + + +# -------------------------------------------------------------------------- +# Bit expansion +# -------------------------------------------------------------------------- + + +def test_bit_numbering_is_msb_first() -> None: + assert _decode_bits(b"\x80") == {0} + assert _decode_bits(b"\x01") == {7} + assert _decode_bits(b"\x00\x80") == {8} + assert _decode_bits(b"\x00\x01") == {15} + + +def test_bit_expansion_of_first_captured_byte() -> None: + """0xBD is 1011 1101 -> bits 0, 2, 3, 4, 5, 7.""" + assert _decode_bits(b"\xbd") == {0, 2, 3, 4, 5, 7} + + +def test_empty_mask_yields_no_bits() -> None: + assert _decode_bits(b"") == set() + + +# -------------------------------------------------------------------------- +# The captured masks +# -------------------------------------------------------------------------- + + +def test_first_eight_bytes_identical_across_versions() -> None: + """The finding this whole module exists to record: 12.10, 14.10, and 15 + negotiate exactly the same 64-bit capability set. Only the 9th byte — + which IBM's own JDBC driver discards — differs.""" + assert MASK_15[:8] == MASK_1410[:8] == MASK_1210[:8] + assert len({MASK_15[8], MASK_1410[8], MASK_1210[8]}) == 3 + + +@pytest.mark.parametrize("mask", ALL_MASKS) +def test_framing_bits_set_on_every_tested_server(mask: bytes) -> None: + """Every wire-framing choice this driver hardcodes is one the server + actually negotiated. If this ever fails, the row decoder is wrong.""" + caps = ServerCapabilities.from_wire(mask, cap_1=ENHANCED_PROTOCOL_CAP) + assert caps.four_byte_offset, "describe offsets would be misparsed" + assert caps.varchar_var_len, "VARCHAR framing would be misparsed" + assert caps.remove_64k_limit, "fast-path length prefix would be wrong" + assert caps.bigint + assert caps.long_id + assert caps.lvarchar_gt_2k + assert caps.gls + + +@pytest.mark.parametrize("mask", ALL_MASKS) +def test_no_violated_assumptions_on_tested_servers(mask: bytes) -> None: + caps = ServerCapabilities.from_wire(mask, cap_1=ENHANCED_PROTOCOL_CAP) + assert caps.violated_assumptions() == [] + + +@pytest.mark.parametrize("mask", ALL_MASKS) +def test_ninth_byte_is_decoded(mask: bytes) -> None: + """JDBC drops bits 64+; we keep them because they're the only part + that varies between releases.""" + caps = ServerCapabilities.from_wire(mask, cap_1=ENHANCED_PROTOCOL_CAP) + assert any(b >= 64 for b in caps.bits) + + +# -------------------------------------------------------------------------- +# Pre-set bits — the reason isUSVER is a dead end +# -------------------------------------------------------------------------- + + +def test_preset_bits_applied_when_cap1_nonzero() -> None: + """JDBC pre-sets {0,2,3,4,49,51} for any non-zero Cap_1 and BitSet.set + never clears, so these are true on any modern server regardless of the + mask. That's why isUSVER never varies and can't explain version-specific + behaviour.""" + caps = ServerCapabilities.from_wire(b"\x00" * 9, cap_1=ENHANCED_PROTOCOL_CAP) + assert {0, 2, 3, 4, 49, 51}.issubset(caps.bits) + assert caps.usver + + +def test_preset_bits_not_applied_when_cap1_zero() -> None: + caps = ServerCapabilities.from_wire(b"\x00" * 9, cap_1=0) + assert caps.bits == frozenset() + assert not caps.usver + + +def test_preset_cannot_clear_a_bit_the_mask_set() -> None: + caps = ServerCapabilities.from_wire(MASK_15, cap_1=ENHANCED_PROTOCOL_CAP) + assert caps.has(BIT_USVER) + assert caps.has(BIT_FOUR_BYTE_OFFSET) + + +# -------------------------------------------------------------------------- +# Assumption violations — the diagnostic path +# -------------------------------------------------------------------------- + + +def test_missing_four_byte_offset_is_reported() -> None: + # Everything set except bit 50. + bits = bytearray(b"\xff" * 9) + bits[BIT_FOUR_BYTE_OFFSET // 8] &= ~(0x80 >> (BIT_FOUR_BYTE_OFFSET % 8)) & 0xFF + caps = ServerCapabilities.from_wire(bytes(bits), cap_1=ENHANCED_PROTOCOL_CAP) + problems = caps.violated_assumptions() + assert len(problems) == 1 + assert "4-byte describe offsets" in problems[0] + + +def test_missing_varchar_var_len_is_reported() -> None: + bits = bytearray(b"\xff" * 9) + bits[BIT_VARCHAR_VAR_LEN // 8] &= ~(0x80 >> (BIT_VARCHAR_VAR_LEN % 8)) & 0xFF + caps = ServerCapabilities.from_wire(bytes(bits), cap_1=ENHANCED_PROTOCOL_CAP) + problems = caps.violated_assumptions() + assert len(problems) == 1 + assert "variable-length VARCHAR" in problems[0] + + +def test_missing_remove_64k_is_reported() -> None: + bits = bytearray(b"\xff" * 9) + bits[BIT_REMOVE_64K_LIMIT // 8] &= ~(0x80 >> (BIT_REMOVE_64K_LIMIT % 8)) & 0xFF + caps = ServerCapabilities.from_wire(bytes(bits), cap_1=ENHANCED_PROTOCOL_CAP) + problems = caps.violated_assumptions() + assert len(problems) == 1 + assert "64K limit" in problems[0] + + +def test_all_zero_mask_reports_every_assumption() -> None: + caps = ServerCapabilities.from_wire(b"\x00" * 9, cap_1=ENHANCED_PROTOCOL_CAP) + assert len(caps.violated_assumptions()) == 3 + + +# -------------------------------------------------------------------------- +# Misc +# -------------------------------------------------------------------------- + + +def test_client_offer_matches_jdbc_reference() -> None: + """IfxSqliConnect.clientProtocols, verbatim. Changing this changes what + the server negotiates, so it is pinned.""" + jdbc_reference = bytes([0xFF, 0xFC, 0x7F, 0xFC, 0x3C, 0x8C, 0xAA, 0x97]) + assert jdbc_reference == CLIENT_PROTOCOLS_MASK + assert len(CLIENT_PROTOCOLS_MASK) == 8 + + +def test_enhanced_protocol_requires_exact_cap1() -> None: + """JDBC tests == 316, not >=, because the server echoes the client's own + declared level rather than reporting its own version.""" + assert ServerCapabilities.from_wire(MASK_15, cap_1=316).enhanced_protocol + assert not ServerCapabilities.from_wire(MASK_15, cap_1=315).enhanced_protocol + assert not ServerCapabilities.from_wire(MASK_15, cap_1=317).enhanced_protocol + + +def test_repr_is_readable_and_includes_mask() -> None: + caps = ServerCapabilities.from_wire( + MASK_15, cap_1=316, server_version="IBM Informix Dynamic Server Version 15" + ) + text = repr(caps) + assert "bdbe9ffe7fb7ffefff" in text + assert "usver" in text + assert "cap_1=316" in text + + +def test_capabilities_are_frozen() -> None: + """Immutable so a connection's negotiated state can't be edited after + the fact and quietly disagree with what's on the wire.""" + caps = ServerCapabilities.from_wire(MASK_15, cap_1=316) + with pytest.raises(dataclasses.FrozenInstanceError): + caps.cap_1 = 1 # type: ignore[misc] diff --git a/tests/test_package_metadata.py b/tests/test_package_metadata.py new file mode 100644 index 0000000..61074d3 --- /dev/null +++ b/tests/test_package_metadata.py @@ -0,0 +1,76 @@ +"""Guards on the package's own metadata. + +The distribution is ``informix-driver``; the import module is +``informix_db``. That mismatch is a live trap: ``importlib.metadata`` +needs the *distribution* name, so the string can't be derived from +``__name__``, and getting it wrong fails silently — ``__version__`` +degrades to the ``0.0.0+local`` sentinel instead of raising. + +That is exactly what shipped between the 2026-05-08 rename and +2026.08.27: ``__init__`` still asked for ``informix-db``, so anyone who +installed the renamed package saw ``0.0.0+local``. It went unnoticed +because a stale ``informix-db`` distribution lingered in the dev +environment and answered the query. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +import tomllib + +import informix_db + +PYPROJECT = Path(__file__).resolve().parent.parent / "pyproject.toml" + + +@pytest.fixture(scope="module") +def pyproject() -> dict: + if not PYPROJECT.is_file(): + pytest.skip("pyproject.toml not present (installed-package test run)") + with PYPROJECT.open("rb") as handle: + return tomllib.load(handle) + + +def test_version_is_not_the_unresolved_sentinel() -> None: + """If this fails, the distribution name in ``__init__`` no longer + matches ``[project].name`` and every user sees a bogus version.""" + assert informix_db.__version__ != "0.0.0+local", ( + "__version__ fell back to its sentinel — the distribution name " + "looked up in informix_db/__init__.py does not match the installed " + "package. Check [project].name in pyproject.toml." + ) + + +def test_version_matches_pyproject(pyproject: dict) -> None: + declared = pyproject["project"]["version"] + # PEP 440 normalisation drops leading zeros: 2026.08.27 -> 2026.8.27. + normalised = ".".join( + str(int(part)) if part.isdigit() else part + for part in declared.split(".") + ) + assert informix_db.__version__ in {declared, normalised} + + +def test_init_looks_up_the_declared_distribution_name(pyproject: dict) -> None: + """Catch the rename trap directly: the name passed to + ``importlib.metadata.version()`` must equal ``[project].name``.""" + source = ( + Path(informix_db.__file__).resolve().parent / "__init__.py" + ).read_text(encoding="utf-8") + declared_name = pyproject["project"]["name"] + assert f'version("{declared_name}")' in source, ( + f"informix_db/__init__.py should call version({declared_name!r}) to " + "match [project].name in pyproject.toml" + ) + + +def test_version_is_importable_and_stringy() -> None: + assert isinstance(informix_db.__version__, str) + assert informix_db.__version__ + + +def test_public_surface_is_exported() -> None: + for name in informix_db.__all__: + assert hasattr(informix_db, name), f"{name} in __all__ but not defined"