diff --git a/CHANGELOG.md b/CHANGELOG.md index d88cb7d..ab35423 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,58 @@ All notable changes to `informix-db`. Versioning is [CalVer](https://calver.org/) — `YYYY.MM.DD` for date-based releases, `YYYY.MM.DD.N` for same-day post-releases per PEP 440. +## 2026.08.31 — Fix LVARCHAR tuple framing; DATETIME fractions on bind + +More data corruption, from the same field report that produced `2026.05.08.2`. **If your schema has `LVARCHAR` columns, upgrade** — anything selected after one could be wrong, and `2026.08.27` is not safe. + +### LVARCHAR shifted every column that followed it + +Two independent framing errors in the same envelope: + +**A phantom pad byte.** We appended an even-byte pad when the value length was odd. There is no pad. Wire capture on Informix 12.10 for `INT8 / LVARCHAR / INT8`: + +``` +00 01 00 00 07 d1 00 00 00 00 │ 00 │ 00 00 00 0b │ 50 61 63 6b 61 67 65 52 6f 6f 74 │ 00 01 00 00 00 0a … + a = INT8 2001 │ind │ len = 11 │ "PackageRoot" (11 bytes, odd) │ b = INT8 10 starts HERE +``` + +**A missing length field on NULL.** We returned as soon as the indicator said NULL, leaving its 4-byte length unread. The length is part of the envelope and is always present: + +``` +'odd' -> 00 | 00 00 00 03 | 6f 64 64 8 bytes +NULL -> 01 | 00 00 00 00 5 bytes +'' -> 00 | 00 00 00 00 5 bytes +``` + +NULL and empty string differ only in the indicator byte. + +Either error shifted everything downstream. The reported symptom: `INT8` `10` decoding as `2560` — the same value shifted one byte left — strings losing their first character, and wide rows raising `IndexError` or `ValueError: INT8 payload too short` once the drift ran off the end of the payload. + +The reporter isolated it by **reordering columns in the projection**: values were right when they came first and wrong when they came later. That's the fingerprint of positional drift, and it's a genuinely good diagnostic technique. + +### Why the tests missed it, again + +The only LVARCHAR fixture was `'lv value'` — 8 characters, even, never NULL. Neither faulty branch ever executed in 247 tests. Same shape of gap as `2026.05.08.2`: full coverage of the code paths, no coverage of the *data* that reaches them. + +New tests vary length parity deliberately (0, 1, 2, 3, 11, 255, 256), cover NULL and empty separately, and always put a column *after* the LVARCHAR — a trailing one can be mis-sized with no visible effect. `test_wide_row_survives_column_reordering` rotates the projection through every position, mirroring how this was found. + +### DATETIME lost sub-second precision on INSERT + +Found while verifying the above. `_encode_datetime` emitted `YEAR TO SECOND` unconditionally, so binding a `datetime` carrying microseconds into a `DATETIME YEAR TO FRACTION(n)` column stored zeros — silently. Reads were always correct, so the value only went missing on the way in. + +It now widens to `YEAR TO FRACTION(5)` when `microsecond` is non-zero and keeps the original encoding otherwise, so the long-exercised path is byte-identical for the common case. Binding into a narrower column still works; Informix converts qualifiers on assignment and truncates server-side. `FRACTION(5)` resolves to 10 µs, so Python's sixth digit is dropped — a limit of the type, now pinned by tests. + +### `server_version` reported the protocol version + +Also from the field report: `conn.server_version` returned `9.56.FC6` for a 12.10 server, which reads like a client-SDK version. The login response only carries Informix's *internal* protocol version — 12.10 announces itself as 9.56, 14.10 as 9.59 — and documenting that didn't make the name any less misleading. + +- `conn.server_version` now returns the release (`…Version 12.10.FC6`). It costs one `DBINFO` query on first access, cached thereafter, and falls back rather than raising if no database is open. +- `conn.server_version_internal` returns the raw login string, free as before. + +### Verified + +281/281 integration tests on 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE (`make test-matrix`), plus 123 unit tests. 37 of those tests are new. + ## 2026.08.27 — Decode the SQ_PROTOCOLS capability negotiation Closes the last open item from the Informix 12 field report. The driver hardcodes several wire-framing choices that SQLI actually *negotiates*; those choices were correct on every server we'd measured, but "correct as far as we know" and "checked" are different things, and the failure mode for a framing mismatch is silently corrupted rows. diff --git a/README.md b/README.md index e0d2df9..65bb816 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ Imports as `informix_db` (the distribution name is `informix-driver` because the **0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test. -**Test coverage:** 300+ tests across unit / integration / benchmark suites. Integration tests run against the official IBM Informix Developer Edition Docker image (15.0.1.0.3DE). +**Test coverage:** 400+ tests across unit / integration / benchmark suites. The integration suite passes 281/281 against **each** of Informix 12.10, 14.10, and 15 — `make test-matrix` runs all three. ## Quick start @@ -102,10 +102,11 @@ Informix uses dedicated TLS-enabled listener ports (configured server-side in `s | SQL type | Python type | |---|---| -| `SMALLINT` / `INT` / `BIGINT` / `SERIAL` | `int` | +| `SMALLINT` / `INT` / `SERIAL` / `BIGINT` / `BIGSERIAL` | `int` | +| `INT8` / `SERIAL8` | `int` (legacy 64-bit — a different wire format from `BIGINT`, not an alias) | | `FLOAT` / `SMALLFLOAT` | `float` | | `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` | -| `CHAR` / `VARCHAR` / `NCHAR` / `NVCHAR` / `LVARCHAR` | `str` | +| `CHAR` / `NCHAR` (fixed width) · `VARCHAR` / `NVCHAR` / `LVARCHAR` (variable) | `str` | | `BOOLEAN` | `bool` | | `DATE` | `datetime.date` | | `DATETIME YEAR TO ...` | `datetime.datetime` / `datetime.time` / `datetime.date` | @@ -153,9 +154,9 @@ All three tested against the official IBM developer-edition Docker images, full | Server | Image | Integration suite | |---|---|---| -| **15.0.1.0.3DE** | `icr.io/informix/informix-developer-database` | **241 / 241** | -| **14.10.FC7W1DE** | `ibmcom/informix-developer-database` | **241 / 241** | -| **12.10.FC12W1DE** | `ibmcom/informix-developer-database` | **241 / 241** | +| **15.0.1.0.3DE** | `icr.io/informix/informix-developer-database` | **281 / 281** | +| **14.10.FC7W1DE** | `ibmcom/informix-developer-database` | **281 / 281** | +| **12.10.FC12W1DE** | `ibmcom/informix-developer-database` | **281 / 281** | Reproduce the whole matrix: @@ -178,7 +179,8 @@ SQLI settles some wire framing through a capability exchange (`SQ_PROTOCOLS`) ra ```python conn = informix_db.connect(...) -conn.server_version # 'IBM Informix Dynamic Server Version 15.0.1.0.3' +conn.server_version # release, e.g. '…Version 12.10.FC6' (one cached query) +conn.server_version_internal # raw login string; 12.10 announces itself as 9.56 conn.server_capabilities.four_byte_offset # True conn.server_capabilities.violated_assumptions() # [] — hardcoded framing matches ``` diff --git a/docs-site/src/content/docs/reference/api.md b/docs-site/src/content/docs/reference/api.md index 5ac668d..e2e0167 100644 --- a/docs-site/src/content/docs/reference/api.md +++ b/docs-site/src/content/docs/reference/api.md @@ -58,7 +58,8 @@ informix_db.connect( | `fast_path_call(routine, *args)` | Direct UDF/SPL invocation, bypassing PREPARE/EXECUTE/FETCH. | | `encoding` | Resolved Python codec for `client_locale`. | | `closed` | `True` after `close()`. | -| `server_version` | Server version string from the login response. See note below. | +| `server_version` | Server release, e.g. `'…Version 12.10.FC6'`. Costs one query on first access, then cached. | +| `server_version_internal` | Raw login-response version string. Free. See note below. | | `server_capabilities` | Negotiated `ServerCapabilities`, or `None` if the reply couldn't be decoded. | `Connection` is also a context manager (`with informix_db.connect(...) as conn:`), which closes on exit. @@ -77,8 +78,10 @@ except Exception: raise ``` -:::note[server_version reports the *internal* version] -The login response carries Informix's internal protocol version, not its marketing version. Informix 12.10 reports `9.56`, 14.10 reports `9.59`, and 15 reports `15.0.1.0.3`. At the protocol level the two older releases really are 9.x servers — which is why all three speak an identical SQLI dialect. +:::note[Two version numbers, and why] +The login response carries Informix's *internal* protocol version, not the release you installed: 12.10 announces itself as `9.56`, 14.10 as `9.59`, and 15 as `15.0.1.0.3`. At the protocol level the two older releases really are 9.x servers, which is why all three speak an identical SQLI dialect. + +That string is available for free as `server_version_internal`. Because it reads as a wrong answer, `server_version` asks the server for its release with `DBINFO('version','full')` — one query on first access, cached for the life of the connection, and it falls back to the internal string rather than raising if no database is open. ::: ## Cursor diff --git a/docs-site/src/content/docs/reference/types.md b/docs-site/src/content/docs/reference/types.md index 9ee911d..113d34c 100644 --- a/docs-site/src/content/docs/reference/types.md +++ b/docs-site/src/content/docs/reference/types.md @@ -38,10 +38,24 @@ Both decode to plain Python `int`, so this only matters if you're reading the wi `INT8`/`SERIAL8` are common in schemas predating Informix 11.50, which is when `BIGINT` arrived. -:::caution[Fixed in 2026.08.27] -Releases before `2026.08.27` did not decode `INT8`/`SERIAL8` at all and returned raw `bytes`. The same release fixed `NCHAR` (which lost its first character and could desync the row) and `BOOLEAN` (which corrupted every column after it). If you use any of those three types, upgrade. +:::caution[Upgrade if you use these types] +`2026.08.27` fixed three decoding bugs: `INT8`/`SERIAL8` weren't decoded at all and returned raw `bytes`; `NCHAR` lost its first character; and `BOOLEAN` corrupted every column after it. + +`2026.08.31` fixed two more in `LVARCHAR` framing — a phantom pad byte on odd-length values, and a missing length field on NULLs. Either shifted **every column selected after the LVARCHAR**, producing wrong integers, strings missing their first character, or an outright `IndexError` on wide rows. The same release stopped `DATETIME` binds silently discarding sub-second precision. + +If your schema has `LVARCHAR` columns, `2026.08.27` is not safe — upgrade to `2026.08.31`. ::: +## LVARCHAR wire framing + +Worth knowing if you're reading captures. Every Informix server we've tested describes `LVARCHAR` columns as UDTVAR (type 40, `extended_name='lvarchar'`), and the value arrives wrapped in a UDT envelope: + +``` +[1-byte null indicator][4-byte length][content] +``` + +No padding, and the length field is present even when the indicator says NULL. NULL and empty string differ *only* in that indicator byte — both carry a length of zero. + ## DATETIME field ranges Informix's `DATETIME YEAR TO X` is field-range typed. The Python type returned depends on which fields are present: diff --git a/pyproject.toml b/pyproject.toml index 44e4148..c256e96 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "informix-driver" -version = "2026.08.27" +version = "2026.08.31" description = "Pure-Python driver for IBM Informix IDS — speaks the SQLI wire protocol over raw sockets. No CSDK, no JVM, no native libraries." readme = "README.md" license = { text = "MIT" } diff --git a/src/informix_db/_resultset.py b/src/informix_db/_resultset.py index 4c261a5..303fa47 100644 --- a/src/informix_db/_resultset.py +++ b/src/informix_db/_resultset.py @@ -480,8 +480,7 @@ def compile_row_decoder( lines.append(" offset += 4") lines.append(" raw = payload[offset:offset + length]") lines.append(" offset += length") - lines.append(" if length & 1:") - lines.append(" offset += 1") + # No even-byte pad — see _TC_LVARCHAR in the legacy chain. lines.append(f" {v} = _D{i}(raw, encoding)") elif kind == _RK_DECIMAL: @@ -638,14 +637,35 @@ def _legacy_dispatch_one_column( if tc == _TC_UDTVAR and col.extended_name == "lvarchar": indicator = payload[offset] offset += 1 - if indicator == 1: - return offset, None + # The 4-byte length is present even when the indicator says NULL — + # it is part of the envelope, not of the value. Returning early on + # the indicator left those 4 bytes on the wire and desynced every + # following column. Wire evidence (Informix 12.10), INT8 / LVARCHAR + # / INT8 with the middle column varying: + # 'odd' -> 00 | 00 00 00 03 | 6f 64 64 (8 bytes) + # NULL -> 01 | 00 00 00 00 (5 bytes) + # '' -> 00 | 00 00 00 00 (5 bytes) + # Note NULL and empty-string differ only in the indicator byte. length = int.from_bytes(payload[offset:offset + 4], "big", signed=True) offset += 4 + if indicator == 1: + return offset, None raw = payload[offset:offset + length] offset += length - if length & 1: - offset += 1 + # NO even-byte pad. The UDT envelope is [indicator][int len][bytes] + # and the next column starts immediately after the last content + # byte. Verified on the wire (Informix 12.10) for an odd-length + # value — INT8(2001), LVARCHAR('PackageRoot'), INT8(10): + # 00 01 00 00 07 d1 00 00 00 00 a = INT8 2001 + # 00 null indicator + # 00 00 00 0b length 11 + # 50 61 63 6b 61 67 65 52 6f 6f 74 "PackageRoot" (odd) + # 00 01 00 00 00 0a 00 00 00 00 b = INT8 10, starts immediately + # A pad here consumed one byte too many and shifted every + # subsequent column: INT8 10 decoded as 2560 (0x0A00), strings + # lost their first character, and wide rows ran off the end of + # the payload. Only fired for odd-length values, which is why the + # test fixture ("lv value", 8 chars) never caught it. return offset, raw.decode(encoding) # Unknown — surface ``encoded_length`` bytes raw. @@ -747,8 +767,7 @@ def parse_tuple_payload( offset += 4 raw = payload[offset:offset + length] offset += length - if length & 1: - offset += 1 + # No even-byte pad — see _TC_LVARCHAR in the legacy chain. values.append(decoder(raw, encoding)) continue @@ -832,14 +851,22 @@ def parse_tuple_payload( values.append(_decode_base(tc, raw, encoding)) continue + # LVARCHAR as a bare type code (43), i.e. without the UDT envelope: + # ``[int length][bytes]``, no even-byte pad. + # + # Caveat worth stating plainly: every Informix server we have tested + # (12.10, 14.10, 15) describes *all* LVARCHAR columns as UDTVAR (40) + # with extended_name='lvarchar' — including casts like + # ``'abc'::LVARCHAR`` — so this branch is unreachable in practice and + # its framing is inferred from the UDTVAR evidence rather than + # observed directly. It is kept consistent with that branch on the + # reasoning that the content encoding shouldn't depend on how the + # column happens to be described. if tc == _TC_LVARCHAR: - # [int length][bytes][pad if odd] length = int.from_bytes(payload[offset:offset + 4], "big", signed=True) offset += 4 raw = payload[offset:offset + length] offset += length - if length & 1: - offset += 1 values.append(_decode_base(tc, raw, encoding)) continue @@ -960,17 +987,18 @@ def parse_tuple_payload( if tc == _TC_UDTVAR and col.extended_name == "lvarchar": indicator = payload[offset] offset += 1 - if indicator == 1: - values.append(None) - continue + # Length is present even when NULL, and there is no even-byte + # pad — see the matching branch in _legacy_dispatch_one_column + # for the wire evidence. length = int.from_bytes( payload[offset:offset + 4], "big", signed=True ) offset += 4 + if indicator == 1: + values.append(None) + continue raw = payload[offset:offset + length] offset += length - if length & 1: - offset += 1 values.append(raw.decode(encoding)) continue diff --git a/src/informix_db/connections.py b/src/informix_db/connections.py index 5dcce7a..93ba434 100644 --- a/src/informix_db/connections.py +++ b/src/informix_db/connections.py @@ -284,6 +284,9 @@ class Connection: self._conacc: dict | None = None self._server_protocols: bytes | None = None self._capabilities: ServerCapabilities | None = None + # Lazily filled by the server_version property. None = not asked + # yet; "" = asked and failed (don't retry on every access). + self._server_version_full: str | None = None # Build the env-var dict sent in the login PDU. self._env = dict(_DEFAULT_ENV) @@ -1026,17 +1029,61 @@ class Connection: return self._capabilities @property - def server_version(self) -> str: - """Server version string from the login response. + def server_version_internal(self) -> str: + """The version string carried in the login response. - Note this is the *internal* protocol version, which is not the - marketing version. Informix 12.10 reports ``9.56``, 14.10 reports - ``9.59``, and 15 reports ``15.0.1.0.3``. That is why the two older - releases speak an identical SQLI dialect: at the protocol level - they are both 9.x servers. + This is Informix's *internal* protocol version, not the release + you installed: 12.10 reports ``9.56``, 14.10 reports ``9.59``, + and 15 reports ``15.0.1.0.3``. At the protocol level the two + older releases really are 9.x servers, which is why all three + speak an identical SQLI dialect. + + Free — it arrives with the login response. For the release + number, use :attr:`server_version`. """ return self._conacc.get("server_version", "") if self._conacc else "" + @property + def server_version(self) -> str: + """The server's release version, e.g. + ``'IBM Informix Dynamic Server Version 12.10.FC6'``. + + **Performs one query on first access**, then caches the result + for the life of the connection. The login response only carries + the internal protocol version (see + :attr:`server_version_internal`), which reads as a wrong answer + — a 12.10 server announces itself as 9.56 — so the release + number has to be asked for with ``DBINFO('version','full')``. + + Falls back to the internal string if the query fails, which it + can when no database is open. Never raises: a version lookup + should not be able to break a working connection. + """ + if self._server_version_full is None: + self._server_version_full = self._query_server_version() + return self._server_version_full or self.server_version_internal + + def _query_server_version(self) -> str: + """One-shot ``DBINFO`` lookup for the release version. + + Deliberately not done during connect: it costs a round-trip that + most callers never need, and connect latency is on the hot path + for pooled workloads. + """ + try: + cur = self.cursor() + try: + cur.execute( + "SELECT FIRST 1 DBINFO('version','full') FROM systables" + ) + row = cur.fetchone() + finally: + cur.close() + except Exception: + _log.debug("could not query server version", exc_info=True) + return "" + return str(row[0]).strip() if row and row[0] else "" + def _parse_login_response(self) -> None: """Read and parse the server's login response. diff --git a/src/informix_db/converters.py b/src/informix_db/converters.py index 61e52d8..e62a4c9 100644 --- a/src/informix_db/converters.py +++ b/src/informix_db/converters.py @@ -761,9 +761,22 @@ def _encode_date(value: datetime.date) -> EncodedParam: def _encode_datetime(value: datetime.datetime) -> EncodedParam: """Encode a Python ``datetime.datetime`` as Informix DATETIME (type=10). - Emit YEAR TO SECOND form — covers the common case of stored - timestamps without microseconds. (Phase 6.x can add YEAR TO - FRACTION(N) variants if microseconds are needed.) + Emits YEAR TO SECOND when ``microsecond`` is zero, and YEAR TO + FRACTION(5) when it isn't. + + The conditional matters. Emitting YEAR TO SECOND unconditionally + silently discarded sub-second precision on every bind: a + ``DATETIME YEAR TO FRACTION(5)`` column handed + ``datetime(..., microsecond=120000)`` stored ``.00000``, with no + error. Only widening when there is a fraction to carry keeps the + long-exercised YEAR TO SECOND path byte-identical for the common + case, and Informix converts between qualifiers on assignment, so a + FRACTION(5) bind into a narrower column truncates server-side + rather than failing. + + FRACTION(5) is the widest Informix supports and holds 10 µs + resolution; Python's microsecond field is finer, so the last digit + is dropped. Same trade-off ``_encode_timedelta`` already makes. Format (per ``Decimal.javaToIfx`` line 457): byte[0..1] = short total length of data following (= digit_count/2 + 1) @@ -787,10 +800,20 @@ def _encode_datetime(value: datetime.datetime) -> EncodedParam: (value.second, 2), ] digit_str = "".join(f"{v:0{w}d}" for v, w in fields) # 14 digits + if value.microsecond: + # 6 fraction digits = exactly 3 more BCD pairs, so the digit + # string stays even and the exponent byte is unchanged (the + # integer part is still 7 base-100 pairs). FRACTION(5) carries + # 5 significant digits; the 6th is padding the wire format + # requires. Qualifier: digit_count=19, start=YEAR(0), end= + # FRACTION(5)=15 — matching what the decoder reads back. + digit_str += f"{value.microsecond:06d}" # -> 20 digits + prec = (19 << 8) | (0 << 4) | 15 + else: + prec = (14 << 8) | (0 << 4) | 10 digit_bytes = bytes(int(digit_str[i : i + 2]) for i in range(0, len(digit_str), 2)) - inner = bytes([0xC7]) + digit_bytes # 8 bytes (1 exp + 7 BCD pairs) - raw = len(inner).to_bytes(2, "big") + inner # +2 byte length prefix = 10 bytes - prec = (14 << 8) | (0 << 4) | 10 + inner = bytes([0xC7]) + digit_bytes # 1 exp byte + 7 or 10 BCD pairs + raw = len(inner).to_bytes(2, "big") + inner # +2 byte length prefix return (10, prec, raw) diff --git a/tests/test_capabilities.py b/tests/test_capabilities.py index 3138664..5319bc3 100644 --- a/tests/test_capabilities.py +++ b/tests/test_capabilities.py @@ -74,15 +74,61 @@ def test_framing_capabilities_present(conn_params: ConnParams) -> None: def test_server_version_is_exposed(conn_params: ConnParams) -> None: - """The version here is the *internal* protocol version, not the - marketing one: 12.10 reports 9.56 and 14.10 reports 9.59. Assert the - shape rather than a value so this holds across the matrix.""" + """Assert the shape rather than a value so this holds across the matrix.""" with _connect(conn_params) as conn: version = conn.server_version assert "Informix" in version assert "Version" in version +def test_server_version_reports_release_not_protocol_version( + conn_params: ConnParams, +) -> None: + """A field report flagged ``server_version`` as looking like a + client-SDK version: the login response announces 12.10 servers as 9.56 + and 14.10 as 9.59. ``server_version`` now answers with the release; the + raw login string moved to ``server_version_internal``.""" + with _connect(conn_params) as conn: + release = conn.server_version + internal = conn.server_version_internal + assert "Informix" in internal + # On 12.10/14.10 these genuinely differ; on 15 they agree. + if "9.5" in internal: + assert release != internal + assert "9.5" not in release, ( + f"server_version still reports the protocol version: {release!r}" + ) + + +def test_server_version_is_cached(conn_params: ConnParams) -> None: + """It costs a round-trip, so repeated access must not repeat it.""" + with _connect(conn_params) as conn: + assert conn.server_version == conn.server_version + assert conn._server_version_full is not None + + +def test_server_version_degrades_without_a_database( + conn_params: ConnParams, +) -> None: + """The DBINFO lookup needs an open database. With none, the property + must fall back rather than raise — a version lookup should never be + able to break a working connection.""" + conn = informix_db.connect( + host=conn_params.host, + port=conn_params.port, + user=conn_params.user, + password=conn_params.password, + database=None, + server=conn_params.server, + connect_timeout=10.0, + read_timeout=10.0, + ) + try: + assert isinstance(conn.server_version, str) + finally: + conn.close() + + def test_capabilities_survive_multiple_connections( conn_params: ConnParams, ) -> None: diff --git a/tests/test_datetime_fraction.py b/tests/test_datetime_fraction.py new file mode 100644 index 0000000..3cad924 --- /dev/null +++ b/tests/test_datetime_fraction.py @@ -0,0 +1,151 @@ +"""Regression tests for DATETIME sub-second precision on bind. + +``_encode_datetime`` used to emit YEAR TO SECOND unconditionally, so +binding a ``datetime`` carrying microseconds into a +``DATETIME YEAR TO FRACTION(n)`` column stored zeros — silently, with no +error and no warning. Reads were always fine, which is what made it hard +to notice: the value only went missing on the way in. + +It now emits YEAR TO FRACTION(5) when ``microsecond`` is non-zero and +keeps the original YEAR TO SECOND encoding otherwise. + +FRACTION(5) is Informix's widest and resolves to 10 µs, so Python's +sixth microsecond digit is dropped. That's a real limit of the type, not +a driver choice, and the tests below pin the truncation so it can't drift +into something worse. +""" + +from __future__ import annotations + +import datetime + +import pytest + +import informix_db +from informix_db.converters import _encode_datetime +from tests.conftest import ConnParams + + +def _connect(conn_params: ConnParams) -> informix_db.Connection: + return informix_db.connect( + host=conn_params.host, + port=conn_params.port, + user=conn_params.user, + password=conn_params.password, + database=conn_params.database, + server=conn_params.server, + connect_timeout=10.0, + read_timeout=10.0, + ) + + +# -------------------------------------------------------------------------- +# Encoder unit tests — no server needed +# -------------------------------------------------------------------------- + + +def test_encoder_uses_year_to_second_without_microseconds() -> None: + """The long-exercised path must stay byte-identical.""" + type_code, prec, raw = _encode_datetime( + datetime.datetime(2026, 8, 31, 12, 30, 15) + ) + assert type_code == 10 + assert prec == (14 << 8) | 10 # digit_count 14, YEAR..SECOND + assert raw == b"\x00\x08\xc7\x14\x1a\x08\x1f\x0c\x1e\x0f" + assert len(raw) == 10 # 2 len + 1 exp + 7 BCD pairs + + +def test_encoder_widens_to_fraction_when_microseconds_present() -> None: + type_code, prec, raw = _encode_datetime( + datetime.datetime(2026, 8, 31, 12, 30, 15, 120000) + ) + assert type_code == 10 + assert prec == (19 << 8) | 15 # digit_count 19, YEAR..FRACTION(5) + assert len(raw) == 13 # 2 len + 1 exp + 10 BCD pairs + # Exponent byte is unchanged: the integer part is still 7 base-100 + # pairs, the fraction just adds three more after the point. + assert raw[2] == 0xC7 + # Trailing pairs carry 120000 as BCD 12/00/00. + assert raw[-3:] == b"\x0c\x00\x00" + + +def test_encoder_pads_fraction_to_six_digits() -> None: + """One microsecond must not shift the BCD pairs.""" + _, prec, raw = _encode_datetime( + datetime.datetime(2026, 8, 31, 12, 30, 15, 1) + ) + assert prec == (19 << 8) | 15 + assert len(raw) == 13 + assert raw[-3:] == b"\x00\x00\x01" # 000001 + + +# -------------------------------------------------------------------------- +# Round-trip against a real server +# -------------------------------------------------------------------------- + + +@pytest.mark.integration +@pytest.mark.parametrize( + ("microsecond", "expected"), + [ + (0, 0), + (120000, 120000), + (500000, 500000), + (1, 0), # below FRACTION(5) resolution + (999999, 999990), # truncated to 5 significant digits + ], +) +def test_fraction_round_trip( + conn_params: ConnParams, microsecond: int, expected: int +) -> None: + value = datetime.datetime(2026, 8, 31, 12, 30, 15, microsecond) + with _connect(conn_params) as conn: + cur = conn.cursor() + cur.execute( + "CREATE TEMP TABLE t_dt_frac " + "(k INT, t DATETIME YEAR TO FRACTION(5))" + ) + cur.execute("INSERT INTO t_dt_frac VALUES (?, ?)", (1, value)) + cur.execute("SELECT t FROM t_dt_frac") + (got,) = cur.fetchone() + assert got == value.replace(microsecond=expected) + + +@pytest.mark.integration +def test_fraction_bind_into_year_to_second_column( + conn_params: ConnParams, +) -> None: + """Widening the bind must not break narrower columns — Informix + converts between qualifiers on assignment, truncating server-side.""" + with _connect(conn_params) as conn: + cur = conn.cursor() + cur.execute( + "CREATE TEMP TABLE t_dt_sec (k INT, t DATETIME YEAR TO SECOND)" + ) + cur.execute( + "INSERT INTO t_dt_sec VALUES (?, ?)", + (1, datetime.datetime(2026, 8, 31, 12, 30, 15, 987654)), + ) + cur.execute("SELECT t FROM t_dt_sec") + assert cur.fetchone() == ( + datetime.datetime(2026, 8, 31, 12, 30, 15), + ) + + +@pytest.mark.integration +def test_fraction_survives_alongside_lvarchar( + conn_params: ConnParams, +) -> None: + """The reported schema pairs FRACTION(5) columns with LVARCHARs.""" + ts = datetime.datetime(2026, 8, 31, 12, 30, 15, 120000) + with _connect(conn_params) as conn: + cur = conn.cursor() + cur.execute( + "CREATE TEMP TABLE t_dt_lv " + "(s LVARCHAR(512), t DATETIME YEAR TO FRACTION(5), n INT8 NOT NULL)" + ) + cur.execute( + "INSERT INTO t_dt_lv VALUES (?, ?, ?)", ("PackageRoot", ts, 10) + ) + cur.execute("SELECT s, t, n FROM t_dt_lv") + assert cur.fetchone() == ("PackageRoot", ts, 10) diff --git a/tests/test_lvarchar_framing.py b/tests/test_lvarchar_framing.py new file mode 100644 index 0000000..a2cf70c --- /dev/null +++ b/tests/test_lvarchar_framing.py @@ -0,0 +1,249 @@ +"""Regression tests for LVARCHAR tuple framing, reported 2026-08-31. + +Two independent framing errors, both of which shifted every column that +followed an LVARCHAR: + +1. **A phantom pad byte.** We appended an even-byte pad when the value + length was odd. There is no pad — the next column begins immediately + after the last content byte. +2. **A missing length on NULL.** We returned as soon as the null + indicator said NULL, leaving the 4-byte length field unread. The + length is part of the envelope and is always present. + +Both survived a 247-test suite because the only LVARCHAR fixture used +``'lv value'`` — 8 characters, even, and never NULL. Neither faulty +branch ever executed. + +The tests below therefore vary length parity and nullness deliberately, +and always place a column *after* the LVARCHAR, because the damage lands +downstream: a trailing LVARCHAR can be mis-sized with no visible effect. + +Wire evidence (Informix 12.10) for INT8 / LVARCHAR / INT8: + + 'odd' -> 00 | 00 00 00 03 | 6f 64 64 8 bytes + NULL -> 01 | 00 00 00 00 5 bytes + '' -> 00 | 00 00 00 00 5 bytes + +NULL and empty string differ only in the indicator byte. +""" + +from __future__ import annotations + +import datetime + +import pytest + +import informix_db +from tests.conftest import ConnParams + +pytestmark = pytest.mark.integration + + +def _connect(conn_params: ConnParams) -> informix_db.Connection: + return informix_db.connect( + host=conn_params.host, + port=conn_params.port, + user=conn_params.user, + password=conn_params.password, + database=conn_params.database, + server=conn_params.server, + connect_timeout=10.0, + read_timeout=10.0, + ) + + +# -------------------------------------------------------------------------- +# Length parity — the phantom pad byte +# -------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "text", + [ + "", # 0 — even, empty + "a", # 1 — ODD + "ab", # 2 + "abc", # 3 — ODD + "PackageRoot", # 11 — ODD, the reported value + "lv value", # 8 — the old fixture that hid the bug + "x" * 255, # 255 — ODD, spans a length byte boundary + "y" * 256, # 256 + ], +) +def test_lvarchar_length_parity_does_not_shift_next_column( + conn_params: ConnParams, text: str +) -> None: + """A trailing sentinel column catches any over- or under-read.""" + with _connect(conn_params) as conn: + cur = conn.cursor() + cur.execute( + "CREATE TEMP TABLE t_lv_parity " + "(a INT8 NOT NULL, s LVARCHAR(512), b INT8 NOT NULL, c VARCHAR(8))" + ) + cur.execute( + "INSERT INTO t_lv_parity VALUES (?, ?, ?, ?)", + (2001, text, 10, "tail"), + ) + cur.execute("SELECT a, s, b, c FROM t_lv_parity") + assert cur.fetchone() == (2001, text, 10, "tail") + + +def test_odd_length_lvarchar_reproduces_the_report( + conn_params: ConnParams, +) -> None: + """The exact failure shape: INT8 decoded as its own value shifted one + byte left (10 -> 2560), and the following string losing a character.""" + with _connect(conn_params) as conn: + cur = conn.cursor() + cur.execute( + "CREATE TEMP TABLE t_lv_report " + "(a INT8 NOT NULL, k LVARCHAR(512) NOT NULL," + " b INT8 NOT NULL, v LVARCHAR(1024))" + ) + cur.execute( + "INSERT INTO t_lv_report VALUES (?, ?, ?, ?)", + (2001, "PackageRoot", 10, "/content/package"), + ) + cur.execute("SELECT a, k, b, v FROM t_lv_report") + row = cur.fetchone() + assert row == (2001, "PackageRoot", 10, "/content/package") + assert row[2] != 2560, "INT8 shifted one byte left" + assert row[3].startswith("/"), "leading character lost" + + +# -------------------------------------------------------------------------- +# NULL — the missing length field +# -------------------------------------------------------------------------- + + +def test_null_lvarchar_does_not_shift_next_column( + conn_params: ConnParams, +) -> None: + with _connect(conn_params) as conn: + cur = conn.cursor() + cur.execute( + "CREATE TEMP TABLE t_lv_null " + "(a INT8 NOT NULL, s LVARCHAR(512), b INT8 NOT NULL)" + ) + cur.execute("INSERT INTO t_lv_null VALUES (?, ?, ?)", (3001, None, 20)) + cur.execute("SELECT a, s, b FROM t_lv_null") + assert cur.fetchone() == (3001, None, 20) + + +def test_null_and_empty_lvarchar_are_distinguished( + conn_params: ConnParams, +) -> None: + """They differ only in the indicator byte, so it's easy to conflate.""" + with _connect(conn_params) as conn: + cur = conn.cursor() + cur.execute( + "CREATE TEMP TABLE t_lv_ne (k INT, s LVARCHAR(64), tail INT)" + ) + cur.execute("INSERT INTO t_lv_ne VALUES (?, ?, ?)", (1, None, 111)) + cur.execute("INSERT INTO t_lv_ne VALUES (?, ?, ?)", (2, "", 222)) + cur.execute("SELECT k, s, tail FROM t_lv_ne ORDER BY k") + assert cur.fetchall() == [(1, None, 111), (2, "", 222)] + + +def test_consecutive_null_lvarchars(conn_params: ConnParams) -> None: + """Each NULL under-read by 4 bytes, so several in a row compound.""" + with _connect(conn_params) as conn: + cur = conn.cursor() + cur.execute( + "CREATE TEMP TABLE t_lv_many_null " + "(a INT8 NOT NULL, s1 LVARCHAR(256), s2 LVARCHAR(256)," + " s3 LVARCHAR(256), b INT8 NOT NULL)" + ) + cur.execute( + "INSERT INTO t_lv_many_null VALUES (?, ?, ?, ?, ?)", + (4001, None, None, None, 40), + ) + cur.execute("SELECT a, s1, s2, s3, b FROM t_lv_many_null") + assert cur.fetchone() == (4001, None, None, None, 40) + + +# -------------------------------------------------------------------------- +# Wide, mixed shape — several LVARCHARs interleaved with other types +# -------------------------------------------------------------------------- + + +_WIDE_COLS = [ + "tag", "key_txt", "def_txt", "cur_txt", "note_txt", + "ver", "made_by", "ident", "made_on", +] +_WIDE_ROW = ( + "Gadget", + "PackageRoot", # 11 — ODD + None, # NULL + "/content/package", # 16 + "z", # 1 — ODD + 77, + "maker", + 3001, + datetime.datetime(2026, 8, 31, 12, 30, 15, 120000), +) + + +def _make_wide(cur) -> None: + cur.execute( + "CREATE TEMP TABLE t_lv_wide (" + " tag VARCHAR(32) NOT NULL," + " key_txt LVARCHAR(512) NOT NULL," + " def_txt LVARCHAR(1024)," + " cur_txt LVARCHAR(1024)," + " note_txt LVARCHAR(1024)," + " ver INT8 DEFAULT 0 NOT NULL," + " made_by VARCHAR(100)," + " ident INT8 NOT NULL," + " made_on DATETIME YEAR TO FRACTION(5))" + ) + cur.execute( + f"INSERT INTO t_lv_wide VALUES ({', '.join(['?'] * len(_WIDE_COLS))})", + _WIDE_ROW, + ) + + +def test_wide_mixed_row(conn_params: ConnParams) -> None: + with _connect(conn_params) as conn: + cur = conn.cursor() + _make_wide(cur) + cur.execute(f"SELECT {', '.join(_WIDE_COLS)} FROM t_lv_wide") + assert cur.fetchone() == _WIDE_ROW + + +@pytest.mark.parametrize("shift", range(len(_WIDE_COLS))) +def test_wide_row_survives_column_reordering( + conn_params: ConnParams, shift: int +) -> None: + """The reporter isolated this by reordering columns — values were + correct first in the list and wrong later on. Rotating the projection + exercises every position for every type.""" + with _connect(conn_params) as conn: + cur = conn.cursor() + _make_wide(cur) + order = _WIDE_COLS[shift:] + _WIDE_COLS[:shift] + cur.execute(f"SELECT {', '.join(order)} FROM t_lv_wide") + expected = tuple(_WIDE_ROW[_WIDE_COLS.index(c)] for c in order) + assert cur.fetchone() == expected + + +def test_select_star_wide_row(conn_params: ConnParams) -> None: + """``SELECT *`` raised IndexError once enough LVARCHARs accumulated.""" + with _connect(conn_params) as conn: + cur = conn.cursor() + _make_wide(cur) + cur.execute("SELECT * FROM t_lv_wide") + row = cur.fetchone() + names = [d[0] for d in cur.description] + assert row == tuple(_WIDE_ROW[_WIDE_COLS.index(n)] for n in names) + + +def test_repeated_lvarchar_columns(conn_params: ConnParams) -> None: + """Selecting the same LVARCHAR twice doubles any per-column drift.""" + with _connect(conn_params) as conn: + cur = conn.cursor() + _make_wide(cur) + cur.execute( + "SELECT ident, key_txt, ident, key_txt, ver FROM t_lv_wide" + ) + assert cur.fetchone() == (3001, "PackageRoot", 3001, "PackageRoot", 77)