diff --git a/docs/DECISION_LOG.md b/docs/DECISION_LOG.md index 05d4e9a..813ab12 100644 --- a/docs/DECISION_LOG.md +++ b/docs/DECISION_LOG.md @@ -45,8 +45,20 @@ Format: every decision has a date, a status (`active` / `superseded` / `revisite ## 2026-05-02 — Test target **Status**: active -**Decision**: `icr.io/informix/informix-developer-database` (the IBM Informix Developer Edition image), port 9088 (native SQLI). -**Why**: Free, official, no license click-through, supports plain-password auth out of the box. Pinning the digest (not `:latest`) is a Phase 1 requirement. +**Decision**: `icr.io/informix/informix-developer-database` (the Developer Edition image, now maintained by HCL Software since the 2017 IBM→HCL transfer of Informix), port 9088 (native SQLI). +**Pinned digest** (captured 2026-05-02 from `docker pull`): +`sha256:8202d69ba5674df4b13140d5121dd11b7b26b28dc60119b7e8f87e533e538ba1` +**On-disk footprint**: 2.23 GB unpacked / 665 MB compressed. +**Default credentials** (from container startup logs, accept-license run): +- OS/DB user: `informix` +- Password: `in4mix` +- HQ admin password: `Passw0rd` (don't need this) +- DBA user/password: empty +- DBSERVERNAME: defaults to `informix` (same as the user) +- TLS_CONNECTIONS: OFF (plain auth on port 9088) +- Always-present databases: `sysmaster`, `sysuser` (built during init) +**Container startup**: `docker run -d --name ifx --privileged -p 9088:9088 -e LICENSE=accept -e SIZE=small icr.io/informix/informix-developer-database@sha256:8202d69b...` +**Why**: Free, official, no license click-through, supports plain-password auth out of the box. The digest is locked from Phase 0 onward — `:latest` is the canonical source of flaky integration suites in DB-driver projects, so all `docker-compose.yml` files reference the digest, never the tag. --- @@ -96,6 +108,56 @@ DATETIME / INTERVAL / DECIMAL / NUMERIC / MONEY remain in Phase 6+ — their enc --- +## 2026-05-02 — JDBC reference: `ifxjdbc.jar` 4.50.JC10 + +**Status**: active +**Decision**: Use the user-provided `ifxjdbc.jar` from `/home/rpm/bingham/rtmt/lib/` as the JDBC reference, working copy at `build/ifxjdbc.jar`. +**JAR identity**: `Implementation-Version: 4.50.10-SNAPSHOT`, build 146, dated 2023-03-07. Printable version string: `4.50.JC10`. SHA256 `dc5622cb4e95678d15836b684b6ef1783d37bc0cdd2725208577fc300df4e5f1`. +**Discarded**: Maven Central `com.ibm.informix:jdbc:4.50.4.1` (not downloaded — the local copy is newer). +**Why**: A newer reference is strictly better — the wire protocol is backwards-compatible, so anything `4.50.JC10` knows how to send/receive will be accepted by older servers. Avoids the Maven download. + +--- + +## 2026-05-02 — Decompiler: CFR 0.152 + +**Status**: active +**Decision**: Use CFR 0.152 (https://github.com/leibnitz27/cfr) as the JDBC decompiler. Cached at `build/tools/cfr.jar`. +**Discarded**: Procyon, Fernflower, Ghidra (Ghidra MCP port pool was exhausted; CFR alone proved sufficient). +**Why**: CFR produces the most readable Java for modern bytecode, ships as a single fat JAR, has no install step. Decompiles 478 .java files in seconds. + +--- + +## 2026-05-02 — Confirmed: CSM is dead in modern Informix + +**Status**: active +**Decision**: Do NOT plan for CSM (Communications Support Module) support. Ever. +**Evidence**: `com.informix.asf.Connection.getOptProperties()` (decompiled) literally throws: `"CSM Encryption is no longer supported"` if `SECURITY` or `CSM` opt-prop is set. +**Why**: This used to be the supplied-encryption-plugin layer. IBM removed it; modern Informix uses TLS/SSL exclusively. Removes CSM from every phase plan. + +--- + +## 2026-05-02 — Wire framing primitives confirmed (from JDBC) + +**Status**: active (pending PCAP corroboration) +**Decision**: Adopt these wire-framing primitives in `_protocol.py` from day one: +- All multi-byte integers are **big-endian** (network byte order) +- SmallInt = 2 bytes, Int = 4 bytes, BigInt = 8 bytes, Real = 4 bytes IEEE 754, Double = 8 bytes IEEE 754 +- Variable-length payloads (string, decimal, datetime, interval, BLOB): `[short length][bytes][optional 0x00 pad if length is odd]` — **the 16-bit alignment requirement is mandatory; missing it desynchronizes the parser** +- Strings emitted as `[short len+1][bytes][0x00 nul terminator]` (the +1 is the trailing nul) +- Post-login messages have NO header: each is `[short messageType][payload]` and the next message begins immediately after the previous one's payload ends +- Login PDU has its own SLheader (6 bytes) + PFheader structure +**Source**: `com.informix.lang.JavaToIfxType` (encoders), `com.informix.asf.IfxDataInputStream`/`IfxDataOutputStream` (framing), `com.informix.asf.Connection` (login PDU). Documented byte-by-byte in `PROTOCOL_NOTES.md`. + +--- + +## 2026-05-02 — Plain-password auth: no challenge-response round trip + +**Status**: active +**Decision**: For MVP, treat plain-password auth as a single round trip: client sends one binary login PDU containing the password inline; server replies with one PDU containing version + capabilities or an error block. +**Why**: `Connection.encodeAscBinary()` writes the password as a length-prefixed string within the login PDU body. There is no separate auth phase, no salt, no hashing, no `SQ_CHALLENGE`/`SQ_RESPONSE` exchange. Those constants (129/130) are reserved for PAM and other interactive auth methods, used AFTER the binary login PDU when the server initiates them. + +--- + ## (template — copy below this line for new entries) ``` diff --git a/docs/JDBC_NOTES.md b/docs/JDBC_NOTES.md index 682978e..615551e 100644 --- a/docs/JDBC_NOTES.md +++ b/docs/JDBC_NOTES.md @@ -1,66 +1,169 @@ # IBM JDBC Driver — Wire Protocol Class Index -> **Phase 0 spike artifact.** Reverse-lookup index into the decompiled `com.ibm.informix:jdbc:4.50.4.1` JAR. This document tells us which Java class to read when we want to understand how the JDBC driver implements a given wire-protocol concern. +> **Phase 0 spike artifact.** Reverse-lookup index into the decompiled IBM Informix JDBC driver. This document tells us which Java class to read when we want to understand how the JDBC driver implements a given wire-protocol concern. > -> **Legal note**: the decompiled source lives in `build/jdbc-src/` and is **not committed to this repository**. It is consulted as a clean-room understanding reference only. The Python implementation in `src/informix_db/` is written from `PROTOCOL_NOTES.md` (which cites observed packet bytes), not from the Java source. +> **Legal note**: the decompiled source lives in `build/jdbc-src/` and is **not committed to this repository**. It is consulted as a clean-room understanding reference only. The Python implementation in `src/informix_db/` is written from `PROTOCOL_NOTES.md` (which cites both the observed packet bytes and the JDBC class+method that emits them), never copied from the Java source. + +--- + +## JAR identity + +| Field | Value | +|-------|-------| +| Source | `/home/rpm/bingham/rtmt/lib/ifxjdbc.jar` (user-provided; an IBM tooling/RTMT install) | +| Working copy | `build/ifxjdbc.jar` | +| SHA256 | `dc5622cb4e95678d15836b684b6ef1783d37bc0cdd2725208577fc300df4e5f1` | +| `Implementation-Title` | `Informix JDBC Driver` | +| `Implementation-Version` | `4.50.10-SNAPSHOT` | +| `Build-Date` | `2023-03-07T11:30:44-0600` | +| `Build-Number` | `146` | +| `Main-Class` | `com.informix.jdbc.Version` (printable: "IBM Informix JDBC Driver Version 4.50.JC10") | +| Total entries | 685 (.class files) | +| Decompiled .java files | 478 | ## Decompilation ```bash -# Get the JAR -curl -O https://repo1.maven.org/maven2/com/ibm/informix/jdbc/4.50.4.1/jdbc-4.50.4.1.jar - -# Decompile (CFR — https://www.benf.org/other/cfr/) -java -jar cfr.jar jdbc-4.50.4.1.jar --outputdir build/jdbc-src/ +java -jar build/tools/cfr.jar build/ifxjdbc.jar --outputdir build/jdbc-src/ --silent ``` -Driver version: `4.50.4.1` (latest as of 2026-05-02 on Maven Central). +CFR version: 0.152 (May 2024). Downloaded from https://github.com/leibnitz27/cfr/releases/download/0.152/cfr-0.152.jar. + +--- ## Top-level package layout -TBD — populate after decompilation. +``` +com.informix.asf # wire-transport layer (sockets, framing primitives, SSL, HTTP-tunnel) +com.informix.jdbc # JDBC API surface: Driver, Connection, Statement, ResultSet, type wrappers +com.informix.jdbc.types # JDBC type-wrapper sub-types +com.informix.jdbc.udt # User-defined type support +com.informix.jdbcx # XA / DataSource / ConnectionPool extensions +com.informix.jns # Java Naming Service (sqlhosts lookup) +com.informix.judr # Java User-Defined Routines +com.informix.lang # Encoding/decoding bridge between Java types and Informix wire types +com.informix.smartTrigger # Smart-trigger event subscription +com.informix.types # Informix-side type system (Interval, Decimal, etc.) +com.informix.util # Tracing, error messages, properties +org.bson.* # MongoDB BSON support — ignore for SQLI work +``` -Expected (from research): -- `com.informix.jdbc` — driver entry, connection, statement, result-set -- Likely subpackages for protocol I/O, type system, error mapping +**For SQLI wire-protocol implementation, only `com.informix.asf`, `com.informix.jdbc` (selected files), `com.informix.lang`, and `com.informix.util.IfxErrMsg` matter.** Everything else is JDBC API surface, ORM helpers, or unrelated bundles. -## Class index (responsibility → class) +--- -| Concern | Class | File path under `build/jdbc-src/` | Notes | -|---------|-------|------------------------------------|-------| -| Driver entry point | `com.informix.jdbc.IfxDriver` | TBD | implements `java.sql.Driver` | -| Connection | `com.informix.jdbc.IfxConnection` | TBD | extends `java.sql.Connection` | -| Wire socket I/O | TBD | TBD | look for `DataOutputStream` / `DataInputStream` users | -| Message framing | TBD | TBD | length-prefix + type-tag handlers | -| Login handshake | TBD | TBD | username/password/database selection | -| Auth method dispatch | TBD | TBD | plain / obfuscated / GSSAPI | -| Statement execute | TBD | TBD | EXECUTE / EXECUTE IMMEDIATE entry points | -| Prepared statement | TBD | TBD | parameter descriptors | -| Result-set parsing | TBD | TBD | column descriptors + row decoding | -| Type codecs (encoders) | TBD | TBD | `IfxTypeId` likely; per-type encoder methods | -| Type codecs (decoders) | TBD | TBD | per-type decoder methods | -| Error decoding (SQLSTATE) | TBD | TBD | error-message → SQLException mapping | -| Disconnection | TBD | TBD | logout / socket close | -| Protocol trace | `com.informix.jdbc.*.getProtoTrace` | TBD | Useful debug hook; understand what it logs | +## Class index — wire-protocol concern → class -## Method-level pointers +### Wire transport (the layer we're rebuilding in `_socket.py` + `_protocol.py`) -> As we identify specific methods that map to specific wire bytes, record them here. Format: `Class#method() → wire effect`. +| Concern | Class | Path | Notes | +|---------|-------|------|-------| +| Raw socket lifecycle, SSL, login orchestration | `com.informix.asf.Connection` | `build/jdbc-src/com/informix/asf/Connection.java` (1228 LOC) | The wire-level connection. Owns `Socket asfSocket`, the buffered streams, `IfxDataInputStream/OutputStream`. This is what our `connections.py` layer maps to (just the socket+framing parts; SQL state lives a layer up). | +| Wire write primitives | `com.informix.asf.IfxDataOutputStream` | `…/asf/IfxDataOutputStream.java` (205 LOC) | `writeShort`, `writeInt`, `writeChar`, `writeDate`, `writeDateTime`, `writeDecimal`, `writeReal`, `writeDouble`, `writeSmallInt`, `writeLongInt`, `writeLongBigint`, `writePadded` (16-bit alignment). | +| Wire read primitives | `com.informix.asf.IfxDataInputStream` | `…/asf/IfxDataInputStream.java` (235 LOC) | Mirror of the above: `readShort`, `readInt`, `readChar` (length-prefixed string), `readDate`, `readDateTime`, `readDecimal`, `readReal`, `readDouble`, `readSmallInt`, `readLongInt`, `readLongBigint`, `readPadded`. | +| Protocol tracing (logging hook) | `com.informix.asf.SqliDbg` | `…/asf/SqliDbg.java` (135 LOC) | Records C2S / S2C transitions. Worth reading to learn which fields IBM thinks are interesting enough to log. | +| Debug-instrumented streams | `com.informix.asf.IfxDebugDataInputStream` / `OutputStream` | `…/asf/IfxDebug*.java` | Subclasses of the regular streams that emit trace records. Used when `SQLITRACEFILE` opt-prop is set. | +| HTTP-tunneled SQLI | `com.informix.asf.HttpConnection`, `HttpBufferedInputStream`, `HttpBufferedOutputStream` | `…/asf/Http*.java` | SQLI-over-HTTP for proxy/firewall traversal. **Phase 6+ at the earliest. Skip for MVP.** | -- _(none yet)_ +### Type encoding/decoding (JavaToIfx ↔ IfxToJava) -## Things to grep for +| Concern | Class | Path | Notes | +|---------|-------|------|-------| +| Java→wire encoders | `com.informix.lang.JavaToIfxType` | `…/lang/JavaToIfxType.java` | `JavaToIfxInt(int)` → 4-byte BE; `JavaToIfxSmallInt(short)` → 2-byte BE; `JavaToIfxLongBigInt(long)` → 8-byte BE; `JavaToIfxLongInt(long)` → 10-byte legacy variable-numeric (skip MVP); `JavaToIfxChar(String)` length-prefixed; `JavaToIfxDate`, `JavaToIfxDateTime`, `JavaToIfxDecimal`, `JavaToIfxReal`, `JavaToIfxDouble`. | +| Wire→Java decoders | `com.informix.lang.IfxToJavaType` | `…/lang/IfxToJavaType.java` | Mirror of the above. | +| IDS type metadata / type codes | `com.informix.lang.IfxTypes` | `…/lang/IfxTypes.java` | Constants for the IDS internal type IDs. The other half of the answer to "what's in column descriptors". | +| INTERVAL day-fraction / year-month structures | `com.informix.lang.IntervalDF`, `IntervalYM`, `Interval` | `…/lang/Interval*.java` | Phase 6+. | + +### SQLI protocol (the layer we're rebuilding in `connections.py` + `cursors.py`) + +| Concern | Class | Path | Notes | +|---------|-------|------|-------| +| Protocol contract | `com.informix.jdbc.IfxProtocol` (interface) | `…/jdbc/IfxProtocol.java` (146 LOC) | 50-method interface — read this FIRST to understand the protocol's "vocabulary" without drowning in implementation. Methods include `executeBegin/Commit/Rollback/Savepoint`, `executeVersion`, `executeProtocols`, `handlePAMAuthentication`, `executeOpenDatabase`, `executeCommand`, `executePrepare`, `executeExecute`, `executeFetchBlob`, `executeReadSmBlob`, `executeFastPath`, `handlePrivateServerExchange`. | +| SQLI protocol implementation | `com.informix.jdbc.IfxSqli` | `…/jdbc/IfxSqli.java` (6501 LOC) | The concrete `IfxProtocol` impl. Owns `IfxDataInputStream is` / `IfxDataOutputStream os` (inherited from `IfxSqliConnect`). Search for `os.writeShort(...)` to find every message-emit site. | +| Connection state machine + login orchestration | `com.informix.jdbc.IfxSqliConnect` | `…/jdbc/IfxSqliConnect.java` (5029 LOC) | Holds the `com.informix.asf.Connection`, owns the high-level connection state. Exposes `IfmxConnection`. | +| Message-type tag enumeration | `com.informix.jdbc.IfxMessageTypes` | `…/jdbc/IfxMessageTypes.java` (163 LOC) | All 142+ SQLI message type constants. **Goldmine.** See "Message type tag table" in `PROTOCOL_NOTES.md`. | +| JDBC Driver entry point | `com.informix.jdbc.IfxDriver` | `…/jdbc/IfxDriver.java` (584 LOC) | URL parsing, property handling, `Driver.connect()` impl. Useful for understanding what opt-props exist and which ones affect protocol behavior. | +| Connection extension interface | `com.informix.jdbc.IfmxConnection` (interface) | `…/jdbc/IfmxConnection.java` | Adds Informix-specific methods to `java.sql.Connection`: UDT info, HDR, `scrubConnection()` for pool reuse, `IFX_USEPUT`, prepared-statement cache. | + +### Auth handlers + +| Concern | Class | Notes | +|---------|-------|-------| +| Plain-password auth | inline in `Connection.encodeAscBinary()` | Username + password are written into the binary login PDU directly. No separate auth round-trip. **MVP target.** | +| Legacy text-mode auth (`-pPASSWORD` argv) | `Connection.EncodeAscString()` | Triggered when `capabilities == 0`. Emits literal `sqlexec USER -pPASSWORD VERSION SERIAL -dDB -fIEEEM ` argv string. Skip MVP. | +| PAM challenge/response | `IfmxPAM`, `IfxPAMChallenge`, `IfxPAMResponse`; `IfxSqli.handlePAMAuthentication(String)` | Triggered by server-initiated `SQ_CHALLENGE=129` → client replies with `SQ_RESPONSE=130`. **Phase 6+.** | +| Trusted Context | flag in `encodeAscBinary` (option bit `ASF_TRUSTCTXT = 0x4000000` in stmtoptions int) | Phase 6+. | +| GSSAPI / Kerberos | not directly visible in `Connection.java`; likely handled via JCE provider plumbing | Phase 6+ if ever. | +| **CSM is removed** | `Connection.getOptProperties()` throws `"CSM Encryption is no longer supported"` if `SECURITY` or `CSM` is set | Confirmed dead. Cross off forever. | +| SSL / TLS | inline in `Connection.openSocket()` | Wraps `Socket` with `SSLSocket`. Default protocols: TLSv1.3, TLSv1.2, TLSv1.1, TLSv1 (Java 11+). **Phase 6+.** | + +### Errors + +| Concern | Class | Notes | +|---------|-------|-------| +| Error code → message text | `com.informix.util.IfxErrMsg` | Maps SQLCODE to localized text. Used to construct exceptions and warnings. The `Message` CLI tool (`com.informix.jdbc.Message`) is just a thin wrapper. | +| Warning code → message text | `com.informix.util.IfxWarnMsg` | Same, for warnings. | +| ASF-level (transport) exceptions | `com.informix.asf.IfxASFException`, `IfxASFRemoteException` | Network / login-rejection errors. Distinct from SQL-level exceptions. | +| Tracing | `com.informix.util.Trace`, `TraceFlag` | Used throughout. | + +### Stuff to ignore + +- `com.informix.jdbc.types.*`, `com.informix.jdbc.udt.*` — JDBC API helpers, not protocol +- `com.informix.jdbcx.*` — DataSource/XA/Pool wrappers; sit ABOVE the protocol layer +- `com.informix.jns.*` — sqlhosts lookup; not relevant when we just connect to `host:port` directly +- `com.informix.judr.*` — Java UDR support; not protocol +- `com.informix.smartTrigger.*` — event-subscription feature; not MVP +- `com.informix.jdbc.IfxBlob`, `IfxClob`, `IfxLob*`, `IfxSmBlob`, `IfxSmartLobOutputStream` — large object support; Phase 6+ +- `org.bson.*` — bundled MongoDB BSON; for the JSON/BSON column type. Phase 6+ at best +- `com.informix.jdbc.IfxBSONObject`, `IfxJSON`, `IfxImpExp` — BSON/JSON wrappers and import/export utilities + +--- + +## Method-level pointers (specific Java methods → wire effect) + +> Format: `Class#method() → wire bytes emitted/consumed`. Filled in as we read deeper. + +### Connection / login +- `com.informix.asf.Connection#openSocket()` → opens the TCP (or SSL) socket to `ipAddr:PortNumber`, sets TCP_NODELAY, optional KEEPALIVE. +- `com.informix.asf.Connection#sendConnectionRequest(...)` → emits the binary login PDU (`encodeAscBinary` path) or legacy text login (`EncodeAscString` path). +- `com.informix.asf.Connection#encodeAscBinary(...)` → emits the binary login PDU. **See `PROTOCOL_NOTES.md §2 Login Sequence` for the byte-by-byte breakdown.** Markers used: 100, 101, 104, 106, 107, 108, 116, 127. +- `com.informix.asf.Connection#EncodeSLheader(o, pfPDUsize, slType, slAttribute, slOptions)` → 6-byte SL header preceding the binary login: `[short totalSize][byte slType][byte slAttribute][short slOptions]`. +- `com.informix.asf.Connection#EncodeAscString(...)` → emits the legacy text login: `"sqlexec USER -pPASSWORD VERSION SERIAL -dDB -fIEEEM "`. +- `com.informix.asf.Connection#recvConnectionResponse()` → reads `[short length][byte SLType][skipBytes 3][markers 100,101][...]` → dispatches to `DecodeAscBinary` / `DecodeAscString`. +- `com.informix.asf.Connection#DecodeAscBinary(in)` → reads server response: VersionNumber, SerialNumber, ApplidName, Cap_1, Cap_2, Cap_3, then a switch on a marker (102=SQ_ASCINITRESP, 103=redirect, 127=SQ_ASCEOT clean). +- `com.informix.asf.Connection#disconnectOrderly()` → sends bare `[short SQ_EXIT=56]`, expects server to echo back `[short 56]` or `[short 12=SQ_EOT]`. May receive interleaved `[short 99=SQ_XACTSTAT]` mid-stream. + +### Type encoding +- `com.informix.lang.JavaToIfxType#JavaToIfxInt(int)` → 4-byte big-endian. +- `com.informix.lang.JavaToIfxType#JavaToIfxSmallInt(short)` → 2-byte big-endian. +- `com.informix.lang.JavaToIfxType#JavaToIfxLongBigInt(long)` → 8-byte big-endian (BIGINT type). +- `com.informix.lang.JavaToIfxType#JavaToIfxChar(String)` → returns `[short length+1][bytes][0x00 nul terminator]`. The `+1` accounts for the trailing nul. +- `com.informix.asf.IfxDataOutputStream#writePadded(byte[])` → write bytes, then write a 0x00 byte if length is odd. **The 16-bit-alignment requirement.** + +### Disconnect +- `com.informix.asf.Connection#disconnectOrderly()` (above) +- Bare `[short 56]` is the only message the client emits; bare `[short 56]` or `[short 12]` is the response. No header. + +--- + +## Useful greps ```bash -# Wire I/O entry points -grep -rln "DataOutputStream\|DataInputStream" build/jdbc-src/ +# Find every wire-emit site in IfxSqli.java (writeShort calls reveal message tags being sent) +grep -nE 'os\.writeShort\(' build/jdbc-src/com/informix/jdbc/IfxSqli.java -# Type code constants -grep -rln "TYPEID\|IfxTypeId\|TypeId" build/jdbc-src/ +# Find every wire-read site +grep -nE '\.readShort\(\)' build/jdbc-src/com/informix/jdbc/IfxSqli.java -# Auth method strings -grep -rln "OBFUSCATE\|PWDOBFUSCATION\|GSS\|KERBEROS" build/jdbc-src/ +# Find login state-machine logic +grep -nE 'executeOpenDatabase|executeVersion|executeProtocols|handlePAM|handlePrivate' build/jdbc-src/com/informix/jdbc/IfxSqli.java + +# Find type-codec dispatch +grep -nE 'switch.*[Cc]olType|case.*IfxTypes\.' build/jdbc-src/com/informix/jdbc/IfxSqli.java + +# Authentication strings +grep -rln 'OBFUSCATE\|PWDOBFUSCATION\|GSS\|KERBEROS' build/jdbc-src/ # SQLSTATE / error mapping -grep -rln "SQLSTATE\|SQLException" build/jdbc-src/ +grep -rln 'SQLSTATE' build/jdbc-src/ ``` diff --git a/docs/PROTOCOL_NOTES.md b/docs/PROTOCOL_NOTES.md index 9e4f36b..db91896 100644 Binary files a/docs/PROTOCOL_NOTES.md and b/docs/PROTOCOL_NOTES.md differ