Ryan Malloy
e2c48f855e
Cursor class scaffolded with full PEP 249 surface:
src/informix_db/cursors.py — Cursor with execute, fetchone, fetchmany,
fetchall, description, rowcount, arraysize, close, iterator,
context manager. Sends SQ_COMMAND chains for parameterless SQL
(Phase 4 adds SQ_BIND/SQ_EXECUTE for params).
src/informix_db/_resultset.py — ColumnInfo, parse_describe,
parse_tuple_payload. Best-effort SQ_DESCRIBE parser; refines in
Phase 2.1.
src/informix_db/connections.py — Connection.cursor() now returns a
real Cursor; new _send_pdu() lets Cursor share the connection's
socket without violating encapsulation.
Protocol findings landed in PROTOCOL_NOTES.md §6:
§6a — SQ_PREPARE format with named tags (the "trailing 22, 49"
are SQ_NDESCRIBE and SQ_WANTDONE chained into the same PDU).
Confirmed against IfxSqli.sendPrepare line 1062.
§6c — Server requires post-login init sequence (SQ_PROTOCOLS →
SQ_INFO → SQ_ID(env vars) → SQ_DBOPEN) BEFORE any PREPARE works.
Discovered the hard way: PREPARE without this sequence gets no
response; SQ_DBOPEN without SQ_PROTOCOLS gets sqlcode=-759
("Database not available"). The login PDU's database field is
a hint, not an open.
§6e — SQ_TUPLE corrected: [short warn][int size][bytes payload]
(not [int 0][short payloadLen] as earlier draft claimed).
Two more constants added to _messages.MessageType:
SQ_NDESCRIBE = 22, SQ_WANTDONE = 49
Tests: 40 unit + 7 integration (added 2 new — cursor() returns a
Cursor, parameter binding raises NotSupportedError). All green, ruff
clean. Removed obsolete "cursor() raises NotImplementedError" test.
What works end-to-end now: connect, cursor(), close, parameter-attempt
gating. What doesn't yet: cursor.execute("SELECT 1") — server requires
the post-login init sequence we don't yet send.
Discovered captures (kept for next session's analysis):
docs/CAPTURES/06-py-select1-attempt.socat.log
docs/CAPTURES/07-py-replay-jdbc-prepare.socat.log
docs/CAPTURES/08-py-with-dbopen.socat.log
docs/CAPTURES/09-py-full-replay.socat.log
Three new tasks created tracking the remaining Phase 2 blockers:
post-login init sequence, proper SQ_DESCRIBE parser, SQ_ID action
vocabulary helpers.
672 lines
32 KiB
Markdown
672 lines
32 KiB
Markdown
# SQLI Wire Protocol Notes
|
||
|
||
> **Phase 0 spike artifact.** Byte-level reference for the Informix SQLI wire protocol, derived from clean-room study of the decompiled IBM Informix JDBC driver (`ifxjdbc.jar`, `Implementation-Version: 4.50.10-SNAPSHOT` / printable `4.50.JC10`, build 146 from 2023-03-07; SHA256 in `JDBC_NOTES.md`) cross-checked against packet captures of the reference exchange (pending).
|
||
|
||
---
|
||
|
||
## Source attribution conventions
|
||
|
||
Each documented byte sequence cites both sources of evidence:
|
||
|
||
- 🟡 **JDBC**: cross-referenced against `<class>#<method>()` in the decompiled tree (see `JDBC_NOTES.md`)
|
||
- 🔵 **PCAP**: observed in `docs/CAPTURES/<file>.pcap` at offset `<n>`
|
||
- ✅ **CONFIRMED**: corroborated by both 🟡 and 🔵
|
||
- 🟠 **UNVERIFIED**: only one of the two sources
|
||
|
||
**Current state** (2026-05-02): all 🟡 findings are present from JDBC reading. PCAP capture is pending (Phase 0 task #7) and required for ✅. Treat everything below as 🟠 pending PCAP cross-check.
|
||
|
||
---
|
||
|
||
## 1. Wire framing primitives 🟡
|
||
|
||
### Endianness
|
||
|
||
**Big-endian (network byte order) for all multi-byte integers.**
|
||
|
||
Source: `com.informix.lang.JavaToIfxType.JavaToIfxInt(int)` line 51-54:
|
||
```java
|
||
public static byte[] JavaToIfxInt(int i) {
|
||
byte[] b = new byte[]{(byte)(i >> 24), (byte)(i >> 16), (byte)(i >> 8), (byte)i};
|
||
return b;
|
||
}
|
||
```
|
||
Same pattern for `JavaToIfxSmallInt` (2 bytes) and `JavaToIfxLongBigInt` (8 bytes). Java's `DataOutputStream` defaults to big-endian; Informix matches that.
|
||
|
||
### Width table
|
||
|
||
| Wire type | Width (bytes) | Java method | Notes |
|
||
|-----------|---------------|-------------|-------|
|
||
| SmallInt | 2 | `writeSmallInt(short)` / `writeShort` | SMALLINT, message-type tags, length prefixes |
|
||
| Int | 4 | `writeInt(int)` | INTEGER, capability flags, sizes |
|
||
| LongBigInt| 8 | `writeLongBigint(long)` | BIGINT (use this for 64-bit ints) |
|
||
| LongInt | 10 | `writeLongInt(long)` | **Legacy variable-numeric LONG INT — skip MVP**, predates 64-bit ints |
|
||
| Real | 4 | `writeReal(float)` | IEEE 754 single |
|
||
| Double | 8 | `writeDouble(double)` | IEEE 754 double |
|
||
| Date | 4 | `writeDate(Date)` | day-count from Informix epoch (1899-12-31, to confirm) |
|
||
|
||
### Variable-length encoding (string, decimal, datetime, interval, BYTE, TEXT)
|
||
|
||
```
|
||
[short length][bytes payload][optional 0x00 pad if length is odd]
|
||
```
|
||
|
||
**The 16-bit alignment is a hard requirement.** Source: `IfxDataOutputStream.writePadded(byte[])`:
|
||
```java
|
||
public void writePadded(byte[] b) throws IOException {
|
||
this.write(b, 0, b.length);
|
||
if ((b.length & 1) >= 1) {
|
||
this.write(0);
|
||
}
|
||
}
|
||
```
|
||
Mirror in `IfxDataInputStream.readPadded`. Every variable-payload message needs this padding or the next short read will misalign and the entire parser desynchronizes.
|
||
|
||
### String encoding
|
||
|
||
`com.informix.lang.JavaToIfxType.JavaToIfxChar(String)` returns:
|
||
```
|
||
[short length+1][bytes][0x00 nul terminator]
|
||
```
|
||
The `+1` is the trailing nul. So an N-character string takes `2 + N + 1` bytes, then padded to even.
|
||
|
||
Note: there is also a `JavaToIfx4BytesChar` variant for wide-char strings. Probably for GLS multibyte locales. Phase 6+.
|
||
|
||
---
|
||
|
||
## 2. Connection establishment 🟡
|
||
|
||
### TCP setup
|
||
|
||
- Default port: **9088** (native SQLI). Port 9089 is SSL.
|
||
- `Socket.connect(InetSocketAddress(host, port), loginTimeout)`
|
||
- `setTcpNoDelay(true)` — Nagle off
|
||
- `setKeepAlive(socKeepAlive)` — opt-in via `IFX_SOC_KEEPALIVE` property
|
||
- Optional SSL wrap via `SSLSocketFactory`. **Phase 6+; skip for MVP.**
|
||
- Buffered streams: `BufferedInputStream(in, 4096)` / `BufferedOutputStream(out, 4096)` over the raw socket streams. The `IfxDataInputStream`/`IfxDataOutputStream` wrap the buffered streams.
|
||
|
||
### Capability bits (driving login path selection)
|
||
|
||
- `capabilities == 0` → legacy text-mode login (`EncodeAscString`)
|
||
- `capabilities > 0` → modern binary login (`encodeAscBinary`) **— this is what we implement**
|
||
|
||
The capabilities bitfield itself is opaque from the client side; the JDBC driver computes it based on opt-props. For MVP we will hardcode a sensible value matching what JDBC sends in a vanilla connection (will derive from PCAP).
|
||
|
||
### Other constants seen in `Connection.java`
|
||
|
||
| Constant | Value | Notes |
|
||
|----------|-------|-------|
|
||
| `MAX_BUFF_SIZE` | 32768 | upper bound for individual PDUs |
|
||
| `MIN_BUFF_SIZE` | 140 | lower bound |
|
||
| `STREAM_BUF_SIZE` | 4096 | socket buffer size |
|
||
| `PFCONREQ_BUF_SIZE` | 2048 | login request buffer |
|
||
| `SL_HEADER_SIZE` | 6 | the SLheader is 6 bytes |
|
||
| `applType` | `"sqlexec "` | 12-char nul-padded application type |
|
||
| `applID` | `"sqli"` | application ID string |
|
||
| `PROT_SQLIOL` | `"ol "` | 8-char nul-padded protocol identifier |
|
||
| `NET_TLITCP` | `"tlitcp "` | 8-char nul-padded network type |
|
||
| `floatType` | `"IEEEM"` | float encoding type identifier (5 chars) |
|
||
|
||
---
|
||
|
||
## 3. Login sequence (binary path) 🟡
|
||
|
||
### 3a. Client→server: SLheader (6 bytes)
|
||
|
||
```
|
||
[short totalPduSize+6] # 2 bytes — PDU payload length + 6 (header overhead)
|
||
[byte slType] # 1 byte — 1=SLTYPE_CONREQ (connection request)
|
||
[byte slAttribute] # 1 byte — 60=PF_PROT_SQLI_0600 (protocol version)
|
||
[short slOptions] # 2 bytes — 0 in vanilla connect
|
||
```
|
||
|
||
Source: `Connection.EncodeSLheader(o, pfPDUsize, 1, 60, 0)`.
|
||
|
||
### 3b. Client→server: PFheader (the binary login PDU)
|
||
|
||
This is the body that the SLheader announces. Total size depends on credentials and env vars. The byte layout, in emission order from `Connection.encodeAscBinary()`:
|
||
|
||
```
|
||
[short 100] # SQ_ASSOC marker — start of "association" record
|
||
[short 101] # SQ_ASCBINARY marker — binary-format indicator
|
||
[int 61] # ?? known constant; meaning TBD
|
||
[short floatType.len+1] # = 6
|
||
[bytes "IEEEM"][byte 0] # float-format identifier, nul-terminated
|
||
|
||
[short 108] # SQ_ASCBPARMS marker — binary parameters block
|
||
[bytes "sqlexec\0\0\0\0\0"] # applType, fixed 12 bytes
|
||
[short version.len+1] # = 6
|
||
[bytes "9.280"][byte 0] # client version (hardcoded "9.280" — note: NOT JDBC version!)
|
||
[short serial.len+1] # = 12
|
||
[bytes "RDS#R000000"][byte 0] # client serial (hardcoded constant)
|
||
[short applID.len+1] # = 5
|
||
[bytes "sqli"][byte 0] # application ID
|
||
|
||
[int capabilities] # client capability flags (opaque bitfield)
|
||
[int 0] # reserved
|
||
[int 0] # reserved
|
||
|
||
[short 1] # ?? section marker (NOT in IfxMessageTypes)
|
||
[short username.len+1]
|
||
[bytes username][byte 0]
|
||
|
||
# password (optional)
|
||
if password is null:
|
||
[short 0] # zero-length password
|
||
else:
|
||
[short password.len+1]
|
||
[bytes password][byte 0]
|
||
|
||
[bytes "ol\0\0\0\0\0\0"] # PROT_SQLIOL — 8 bytes
|
||
[int 61] # ?? same magic as before
|
||
[bytes "tlitcp\0\0"] # NET_TLITCP — 8 bytes
|
||
[int 1] # UTYPE_INTERNET (= 1)
|
||
|
||
[short 104] # SQ_ASCINITREQ marker
|
||
[short 11] # ASF_XCONNECT (= 11)
|
||
[int stmtoptions] # = 3 | (sqlhGroup ? 0x2000000 : 0) | (trustedCtx ? 0x4000000 : 0)
|
||
# bit 0+1 = ASF_AMBIG_SEOL (=3)
|
||
# bit 25 = ASF_GRPREF (sqlh group)
|
||
# bit 26 = ASF_TRUSTCTXT (trusted context)
|
||
[short servername.len+1]
|
||
[bytes servername][byte 0]
|
||
|
||
# database (optional)
|
||
if dbname is null:
|
||
[short 0]
|
||
else:
|
||
[short dbname.len+1]
|
||
[bytes dbname][byte 0]
|
||
|
||
[short 0][short 0][short 0][short 0] # 4 reserved/empty option slots (8 bytes total)
|
||
|
||
# Environment vars (encodeEnv)
|
||
[short 106] # SQ_ASCENV marker
|
||
[short num_envvars]
|
||
for each (name, value) in envlist:
|
||
[short name.len+1]
|
||
[bytes name][byte 0]
|
||
[short value.len+1]
|
||
[bytes value][byte 0]
|
||
|
||
# Process info (encodeEnvPInfo)
|
||
[short 107] # SQ_ASCPINFO marker
|
||
[int 0] # reserved
|
||
[int procId] # OS process ID
|
||
[int threadId] # thread ID
|
||
[short hostname.len+1]
|
||
[bytes hostname][byte 0]
|
||
[short 0] # reserved
|
||
[short cwd.len+1]
|
||
[bytes cwd][byte 0]
|
||
|
||
# AppName section (SQ_ASCMISC_60)
|
||
[short 116] # SQ_ASCMISC_60
|
||
[short total_section_length] # = 10 + appname.len + 1
|
||
[int 0] # reserved
|
||
[int 0] # reserved
|
||
[short appname.len+1]
|
||
[bytes appname][byte 0]
|
||
|
||
# End-of-PDU
|
||
[short 127] # SQ_ASCEOT — end-of-transmission for login
|
||
```
|
||
|
||
### 3c. Server→client: response
|
||
|
||
`Connection.recvConnectionResponse()` reads:
|
||
|
||
```
|
||
[short totalLength] # length of the response PDU
|
||
[bytes (totalLength - 2)] # payload, parsed below
|
||
```
|
||
|
||
Then the payload starts with:
|
||
|
||
```
|
||
[byte SLType] # 2=CONACC (accepted), 3=CONREJ (rejected), 13=REDIRECT
|
||
[skipBytes 3] # slAttribute + slOptions — usually unused on response
|
||
[short 100] # SQ_ASSOC marker
|
||
[short 101] # SQ_ASCBINARY marker
|
||
... DecodeAscBinary payload ...
|
||
```
|
||
|
||
`DecodeAscBinary(in)` reads:
|
||
|
||
```
|
||
[skipBytes 4]
|
||
[short subLength]
|
||
[skipBytes subLength]
|
||
[short 108] # SQ_ASCBPARMS marker
|
||
[skipBytes 12]
|
||
[short verLen][bytes serverVersion] # e.g. "15.00.UC1"
|
||
[short serLen][bytes serverSerial]
|
||
[short appLen][bytes appID]
|
||
[int Cap_1][int Cap_2][int Cap_3] # server capability flags (3 × 4 bytes)
|
||
[skipBytes 2]
|
||
[short subLen1][skipBytes subLen1] # optional reserved
|
||
[short subLen2][skipBytes subLen2] # optional reserved
|
||
[skipBytes 24]
|
||
[short marker] # final disposition:
|
||
# 102 = SQ_ASCINITRESP → error/warning section follows
|
||
# 103 = redirect
|
||
# 127 = SQ_ASCEOT (clean)
|
||
```
|
||
|
||
Error block (when marker=102):
|
||
```
|
||
[skipBytes 6]
|
||
[short svcError]
|
||
[short osError]
|
||
[short Warnings]
|
||
[short numErrors]
|
||
for i in 0..numErrors-1:
|
||
[short offset] # if -1, stop
|
||
[readChar errMsg] # length-prefixed string
|
||
```
|
||
|
||
### 3d. Login PDU summary
|
||
|
||
- One client-emitted PDU for the entire login (username + password inline)
|
||
- One server-emitted PDU for the response (success → server version + capabilities; failure → error block)
|
||
- **No challenge-response round trip for plain auth.** PAM auth uses `SQ_CHALLENGE=129`/`SQ_RESPONSE=130` AFTER the binary login PDU, but only if the server requests it. Plain-auth happy path is two messages total.
|
||
|
||
---
|
||
|
||
## 4. Message framing (post-login) 🟡
|
||
|
||
After successful login, messages are **bare** — no SLheader, no PFheader. Each message is:
|
||
|
||
```
|
||
[short messageType] # 2-byte big-endian tag from IfxMessageTypes
|
||
[payload] # variable, depends on message type
|
||
```
|
||
|
||
Evidence: `Connection.disconnectOrderly()` writes only:
|
||
```java
|
||
byte[] b = JavaToIfxType.JavaToIfxSmallInt((short)56); // SQ_EXIT = 56
|
||
this.asfIfxOs.write(b);
|
||
this.asfIfxOs.flush();
|
||
```
|
||
And reads back exactly the same shape. No length prefix, no framing overhead. The receiving side keys off the message type tag to know how many bytes follow.
|
||
|
||
This is a **stream-oriented** protocol, not a packet-oriented one. The wire is a continuous sequence of `[tag][payload][tag][payload]…` and the parser must know each tag's payload structure to know where the next tag begins.
|
||
|
||
---
|
||
|
||
## 5. Message type tags 🟡
|
||
|
||
The full enumeration is in `com.informix.jdbc.IfxMessageTypes` (163 lines, ~140 constants). Key categories below.
|
||
|
||
### Connection lifecycle (post-login)
|
||
|
||
| Tag | Decimal | Hex | Purpose |
|
||
|-----|---------|-----|---------|
|
||
| `SQ_VERSION` | 53 | 0x35 | client/server version negotiation |
|
||
| `SQ_INFO` | 81 | 0x51 | info request (sub-types: INFO_VERSION=2, INFO_TYPE=3, INFO_CAPABILITY=4, INFO_DB=5, INFO_ENV=6) |
|
||
| `SQ_PROTOCOLS` | 126 | 0x7E | protocol negotiation |
|
||
| `SQ_CONNECT` | 112 | 0x70 | connection setup |
|
||
| `SQ_DBOPEN` | 36 | 0x24 | open database |
|
||
| `SQ_DBCLOSE` | 37 | 0x25 | close database |
|
||
| `SQ_DBLIST` | 26 | 0x1A | list databases |
|
||
| `SQ_DISCONNECT` | 114 | 0x72 | logical disconnect |
|
||
| `SQ_EXIT` | 56 | 0x38 | terminate session (on the wire for disconnect) |
|
||
| `SQ_EOT` | 12 | 0x0C | end of transmission (returned by server on disconnect ack) |
|
||
|
||
### Auth (server-initiated challenge-response, PAM only)
|
||
|
||
| Tag | Decimal | Purpose |
|
||
|-----|---------|---------|
|
||
| `SQ_CHALLENGE` | 129 | server challenges client (PAM) |
|
||
| `SQ_RESPONSE` | 130 | client response to challenge |
|
||
| `SQ_ACCEPT` | 127 | server accepts |
|
||
| `SQ_ACK` | 128 | acknowledgement |
|
||
|
||
**Plain-password auth does NOT use these.** They are only for PAM and other challenge-response auth schemes. Skip MVP.
|
||
|
||
### Statement execution
|
||
|
||
| Tag | Decimal | Purpose |
|
||
|-----|---------|---------|
|
||
| `SQ_COMMAND` | 1 | execute SQL directly (immediate execution, no preparation) |
|
||
| `SQ_PREPARE` | 2 | prepare a statement (returns a handle) |
|
||
| `SQ_ID` | 4 | reference a prepared statement by id |
|
||
| `SQ_BIND` | 5 | bind a parameter to a prepared statement |
|
||
| `SQ_OPEN` | 6 | open cursor on a prepared statement |
|
||
| `SQ_EXECUTE` | 7 | execute a prepared statement |
|
||
| `SQ_DESCRIBE` | 8 | describe columns of a prepared statement |
|
||
| `SQ_NFETCH` | 9 | fetch next row(s) |
|
||
| `SQ_CLOSE` | 10 | close cursor |
|
||
| `SQ_RELEASE` | 11 | release prepared statement |
|
||
| `SQ_CURNAME` | 3 | name a cursor |
|
||
|
||
### Result responses (server→client)
|
||
|
||
| Tag | Decimal | Purpose |
|
||
|-----|---------|---------|
|
||
| `SQ_TUPLE` | 14 | a row of data |
|
||
| `SQ_DONE` | 15 | end of result set / completion of DML |
|
||
| `SQ_EOT` | 12 | end of transmission |
|
||
| `SQ_ERR` | 13 | error |
|
||
|
||
### Transactions
|
||
|
||
| Tag | Decimal | Purpose |
|
||
|-----|---------|---------|
|
||
| `SQ_BEGIN` | 35 | begin transaction |
|
||
| `SQ_BEGIN_NOREPL` | 115 | begin (no replication) |
|
||
| `SQ_CMMTWORK` | 19 | commit work |
|
||
| `SQ_RBWORK` | 20 | rollback work |
|
||
| `SQ_SVPOINT` | 21 | declare savepoint |
|
||
| `SQ_SQLISETSVPT` | 137 | set savepoint (newer protocol) |
|
||
| `SQ_SQLIRELSVPT` | 138 | release savepoint |
|
||
| `SQ_SQLIRBACKSVPT` | 139 | rollback to savepoint |
|
||
|
||
### Distributed transactions (XA) — Phase 6+
|
||
|
||
`SQ_X*` family: 65-74, 82. `SQ_XBEGIN`, `SQ_XCOMMIT`, `SQ_XROLLBACK`, `SQ_XPREPARE`, etc.
|
||
|
||
### LOB / Blob — Phase 6+
|
||
|
||
`SQ_FETCHBLOB` (38), `SQ_BLOB` (39), `SQ_BBIND` (41), `SQ_SBBIND` (52), `SQ_FILE_READ` (106), `SQ_FILE_WRITE` (107).
|
||
|
||
### RPC layer (sub-protocol, tag range 200-205) — Phase 6+
|
||
|
||
| Tag | Decimal | Purpose |
|
||
|-----|---------|---------|
|
||
| `SQ_INVOKE` | 200 | RPC invocation |
|
||
| `SQ_REPLY` | 201 | RPC reply |
|
||
| `SQ_EXCEPTION` | 202 | RPC exception |
|
||
| `SQ_VERSION_REQ` | 203 | version request |
|
||
| `SQ_VERSION_REPLY` | 204 | version reply |
|
||
| `SQ_AMFPARAM` | 205 | AMF parameter (used by stored procedures?) |
|
||
|
||
---
|
||
|
||
## 6. Statement execution: `SELECT 1` (validated 2026-05-02)
|
||
|
||
### 6a. SQ_PREPARE format ✅ (named tags resolved)
|
||
|
||
Confirmed against `02-select-1.socat.log` and verified against `IfxSqli.sendPrepare`. The "trailing shorts" 22 and 49 are NOT options — they're chained message tags:
|
||
|
||
```
|
||
[short SQ_PREPARE=2] # 2 bytes
|
||
[short numQmarks] # 2 bytes — count of `?` parameters (= 0 for no params)
|
||
[int sqlLen] # 4 bytes — SQL byte count, NOT including nul (modern server with isRemove64KLimitSupported)
|
||
[bytes sql] # sqlLen bytes
|
||
[byte 0 if (4+sqlLen) is odd] # writeChar pad to even byte alignment
|
||
[short SQ_NDESCRIBE=22] # request column metadata
|
||
[short SQ_WANTDONE=49] # request completion notification
|
||
[short SQ_EOT=12] # flush
|
||
```
|
||
|
||
This is what `IfxSqli.sendPrepare` writes (line 1062 of the decompiled source: `os.writeSmallInt(2)` then `os.writeSmallInt(numqmarks)` then `os.writeChar(b)` then `os.writeSmallInt(22)` then `os.writeSmallInt(49)`).
|
||
|
||
The SQL bytes use `JavaToIfx4BytesChar` on modern servers — 4-byte big-endian length prefix + raw SQL bytes (no nul terminator from this routine; the caller's writeChar adds an even-alignment pad if needed).
|
||
|
||
### 6b. SQ_DESCRIBE response (in progress)
|
||
|
||
Server replies with an 88-byte response containing column metadata. Partial decode:
|
||
|
||
```
|
||
[short SQ_DESCRIBE=8]
|
||
[short 2] # statement id (echo from PREPARE)
|
||
[int 0] # ??
|
||
[int 4] # appears to be java.sql.Types.INTEGER (=4)
|
||
[short 1] # column count (= 1 for SELECT 1)
|
||
... per-column descriptor block ...
|
||
... ends with [short SQ_COST=55][int 1][int 1][short SQ_EOT=12] ...
|
||
```
|
||
|
||
Each column descriptor includes: name (length-prefixed nul-terminated string), IDS type code (the SQ_DESCRIBE response observed `[short 2]` = IDS INTEGER = 2 in IfxTypes), precision, scale, nullability flags. The exact byte layout of one column descriptor needs further study — defer to Phase 2's `_resultset.py` implementation, which can iterate against the existing capture.
|
||
|
||
### 6c. ⚠ Server requires post-login init sequence BEFORE any PREPARE works
|
||
|
||
Discovered during Phase 2 implementation: even though the login PDU includes a `database` field, the server doesn't fully accept it. Sending `SQ_PREPARE` directly after login produces no response (server silently drops). Sending `SQ_DBOPEN` directly after login returns `SQ_ERR sqlcode=-759` ("Database not available").
|
||
|
||
JDBC's actual post-login init sequence (from `02-select-1.socat.log`):
|
||
|
||
```
|
||
1. C → S : SQ_PROTOCOLS (14 bytes) protocol-feature negotiation
|
||
S → C : SQ_PROTOCOLS (16 bytes) server's supported protocol bitmap
|
||
2. C → S : SQ_INFO (8 bytes) info request — observed [short SQ_INFO=81][short 6=INFO_ENV][short 38=?][short SQ_EOT]
|
||
S → C : SQ_EOT (2 bytes) (followed by SQ_DONE in a later capture frame)
|
||
3. C → S : SQ_ID env vars (40 bytes) DBTEMP=/tmp, SUBQCACHESZ=10
|
||
S → C : SQ_DONE (28 bytes)
|
||
4. C → S : SQ_DBOPEN sysmaster (18 bytes)
|
||
S → C : (varies — sometimes SQ_DESCRIBE-like, sometimes SQ_DONE)
|
||
```
|
||
|
||
**Implication for our driver**: the Connection class needs to do this dance after `_parse_login_response`. Each step's response needs proper parsing (SQ_PROTOCOLS in particular has a `[short payloadLen][bytes payload][short 0?][short SQ_EOT]` structure that a naive 2-byte-tag-only reader won't handle).
|
||
|
||
This is the next session's first task. Without it, no SELECTs work.
|
||
|
||
### 6d. JDBC's full prepare/fetch/release sequence (for reference)
|
||
|
||
```
|
||
C → S : SQ_PREPARE (88 / 54 bytes — see 6a)
|
||
S → C : SQ_DESCRIBE (88 bytes — column metadata)
|
||
C → S : SQ_ID action=3 (associate cursor name, e.g. "_ifxc00000000000000")
|
||
C → S : SQ_ID action=9 (= SQ_NFETCH; fetch up to 4096 bytes)
|
||
S → C : SQ_TUPLE (the row data — see 6d)
|
||
S → C : SQ_DONE (completion + SQ_COST)
|
||
C → S : SQ_ID action=10 (close cursor)
|
||
C → S : SQ_ID action=11 (release statement)
|
||
```
|
||
|
||
The action codes inside `SQ_ID` map roughly to other `SQ_*` tag values (9=NFETCH, 10=CLOSE, 11=RELEASE, 3=CURNAME) — same vocabulary, different framing.
|
||
|
||
**For Python MVP:** the `SQ_COMMAND=1` (execute-immediate) path likely lets us skip the cursor lifecycle entirely for parameterless SELECTs. Worth trying first; fall back to the full lifecycle if SQ_COMMAND has limitations we discover.
|
||
|
||
### 6e. SQ_TUPLE format ✅ (corrected — see 6c above for the real layout)
|
||
|
||
```
|
||
[short SQ_TUPLE=14] # 2 bytes
|
||
[int 0] # 4 bytes — flags / reserved
|
||
[short payloadLen] # 2 bytes — total bytes of column-value payload
|
||
[bytes payload] # payloadLen bytes — column values back-to-back,
|
||
# each encoded per its type's wire format
|
||
```
|
||
|
||
For `SELECT 1`: payloadLen=4, payload=`00 00 00 01` (the INTEGER value 1, 4 bytes BE).
|
||
|
||
For multi-column rows, columns appear in order with no per-column delimiter — the parser uses the column descriptors from SQ_DESCRIBE to know each field's length.
|
||
|
||
Per `IfxSqli.receiveTuple` (line 2519): the actual fields are `[short warn][int size when isUSVER, else short][bytes size]` — corrected from earlier "[int 0][short payloadLen]" reading which was a misalignment. Updated `_resultset.py` accordingly.
|
||
|
||
### 6f. SQ_DONE format (partial)
|
||
|
||
```
|
||
[short SQ_DONE=15]
|
||
[int 0] # ??
|
||
[short rowcount] # number of rows affected/returned
|
||
[int sqlcode] # observed 0x301 = 769 (lookup table?)
|
||
[int 0] # ??
|
||
[short SQ_COST=55] # cost-info sub-message
|
||
[int cost1] # observed 1
|
||
[int cost2] # observed 1
|
||
[short SQ_EOT=12]
|
||
```
|
||
|
||
---
|
||
|
||
## 7. Result-set framing 🟠
|
||
|
||
Pending PCAP capture. Hypothesized structure:
|
||
|
||
- Each row is `[short SQ_TUPLE=14][row payload]`
|
||
- Row payload is the column values in order, encoded per their type's wire format (with the 16-bit alignment rule)
|
||
- NULL representation: TBD (probably a per-column flag bit or sentinel value)
|
||
- End-of-result: `[short SQ_DONE=15]` followed by completion metadata (estimated rows, actual rows, etc.)
|
||
|
||
`com.informix.jdbc.IfxColumnInfo` likely holds the column-descriptor structure. Read this once we have a PCAP to align against.
|
||
|
||
---
|
||
|
||
## 8. Disconnection ✅ (well-corroborated by JDBC alone)
|
||
|
||
**Client→server:**
|
||
```
|
||
[short SQ_EXIT=56]
|
||
```
|
||
|
||
**Server→client:**
|
||
```
|
||
[short SQ_EXIT=56] # echo
|
||
or
|
||
[short SQ_EOT=12] # end of transmission
|
||
```
|
||
|
||
Server may interleave one or more `[short SQ_XACTSTAT=99]` records before the final ack (transaction status info during shutdown). Client reads and discards these until it sees 56 or 12.
|
||
|
||
Source: `Connection.disconnectOrderly()` (Connection.java:227-265).
|
||
|
||
---
|
||
|
||
## 9. Error responses 🟠
|
||
|
||
Two error paths observed:
|
||
|
||
### 9a. Login-time errors (during login PDU exchange)
|
||
|
||
Error block embedded in the response, marked by `[short 102]` (SQ_ASCINITRESP):
|
||
```
|
||
[skipBytes 6]
|
||
[short svcError] # Informix service error code
|
||
[short osError] # OS-level error code
|
||
[short numWarnings]
|
||
[short numErrors]
|
||
for i in 0..numErrors:
|
||
[short offset] # if -1, stop
|
||
[readChar errMsg] # length-prefixed string
|
||
```
|
||
|
||
Throws `IfxASFRemoteException(svcError, osError, errMsg)`.
|
||
|
||
### 9b. Statement-time errors (post-login)
|
||
|
||
Hypothesized: `[short SQ_ERR=13][error payload]`. Structure of the payload TBD; needs PCAP. Likely contains SQLCODE, ISAM error code, error message text, possibly SQLSTATE. Look at `IfxSqli` error-decode methods once we have a capture.
|
||
|
||
---
|
||
|
||
## 10. Type codecs
|
||
|
||
### 10a. IDS type codes (column descriptors)
|
||
|
||
`com.informix.lang.IfxTypes` is the constants class. Pending — read this file and tabulate the type codes we care about for MVP (SMALLINT, INT, BIGINT, FLOAT, REAL, CHAR, VARCHAR, NCHAR, NVARCHAR, BOOLEAN, DATE, NULL marker).
|
||
|
||
### 10b. Wire encodings (Phase-2 MVP types)
|
||
|
||
| IDS Type | Type Code | Wire format | Reader | Writer |
|
||
|----------|-----------|-------------|--------|--------|
|
||
| SMALLINT | TBD | 2 bytes BE | `readSmallInt()` | `writeSmallInt(short)` |
|
||
| INTEGER | TBD | 4 bytes BE | `readInt()` | `writeInt(int)` |
|
||
| BIGINT | TBD | 8 bytes BE | `readLongBigint()` | `writeLongBigint(long)` |
|
||
| FLOAT (DOUBLE) | TBD | 8 bytes IEEE 754 | `readDouble(prec)` | `writeDouble(double)` |
|
||
| REAL (SMALLFLOAT) | TBD | 4 bytes IEEE 754 | `readReal(prec)` | `writeReal(float)` |
|
||
| CHAR(N) | TBD | length-prefixed, padded to even | `readChar()` | `writeChar(byte[])` |
|
||
| VARCHAR(N) | TBD | length-prefixed, padded to even | `readChar()` | (similar) |
|
||
| BOOLEAN | TBD | 1 byte 't'/'f' (per Informix convention; verify) | TBD | TBD |
|
||
| DATE | TBD | 4 bytes — int day count from Informix epoch (1899-12-31, verify) | `readDate()` | `writeDate(Date)` |
|
||
|
||
### 10c. Phase 6+ types
|
||
|
||
DATETIME (qualifier-byte precision), INTERVAL (qualifier-encoded), DECIMAL/NUMERIC (Informix BCD), MONEY, LVARCHAR, BYTE/TEXT (length-prefixed BLOBs), CLOB/BLOB (smart blobs with handles), ROW, COLLECTION (LIST/SET/MULTISET).
|
||
|
||
---
|
||
|
||
## 11. Open questions
|
||
|
||
- Exact byte layout of `SQ_COMMAND` payload — does it include statement options after the SQL text?
|
||
- NULL representation in row data — null bitmap, sentinel value, or per-column flag?
|
||
- Result-set column-descriptor structure (the bytes that come back before the first `SQ_TUPLE`).
|
||
- How `Cap_1`/`Cap_2`/`Cap_3` (server capability flags) are bit-allocated.
|
||
- The leading `[int 61]` magic in the login PDU — what does 61 mean? (Possibly version-related: PF_PROT_SQLI_0600 = 60, PF_PROT_SQLI_WITH_CSS = 61?)
|
||
- Why the embedded `versionNumber = "9.280"` string (not the JDBC version, not the server version).
|
||
- The `[short 1]` marker between `Cap_3` and the username — what's it for?
|
||
- Whether the server emits any banner at TCP-accept time before the client's login PDU.
|
||
|
||
All of these resolve from packet capture in Phase 0 task #7.
|
||
|
||
---
|
||
|
||
## 12. Wire-validation findings (2026-05-02 — captures vs JDBC source)
|
||
|
||
Captures collected via `socat` MITM relay (no sudo needed; listens on 9090, forwards to 9088):
|
||
|
||
```bash
|
||
socat -d -d -x TCP-LISTEN:9090,reuseaddr TCP:127.0.0.1:9088 2>docs/CAPTURES/<scenario>.socat.log &
|
||
IFX_PORT=9090 java -cp build/ifxjdbc.jar:build/ tests.reference.RefClient <scenario>
|
||
```
|
||
|
||
Three scenarios captured: `01-connect-only` (1.7 KB), `02-select-1` (6.7 KB), `02-dml-cycle` (9.9 KB). All in `docs/CAPTURES/`. The `>` lines are client→server, `<` are server→client.
|
||
|
||
### Validated against the wire ✅
|
||
|
||
- **Endianness is big-endian.** Confirmed: SLheader `01 c3 01 3c 00 00` decodes as length=0x01c3=451, slType=1, slAttribute=0x3c=60, slOptions=0.
|
||
- **Login PDU structure matches `Connection.encodeAscBinary` field-by-field**: SLheader 6 bytes, then SQ_ASSOC=100, SQ_ASCBINARY=101, int 61, IEEEM, SQ_ASCBPARMS=108, sqlexec padded, "9.280", "RDS#R000000", "sqli", capabilities, ..., username "informix", password "in4mix", PROT_SQLIOL "ol", int 61, NET_TLITCP "tlitcp", int 1, SQ_ASCINITREQ=104, ASF_XCONNECT=11, stmtoptions=3, servername, env vars (DBPATH, CLIENT_LOCALE=en_US.8859-1, CLNT_PAM_CAPABLE, DBDATE, IFX_UPDDESC, NODEFDAC), process info, AppName, SQ_ASCEOT=127.
|
||
- **Server response structure matches `DecodeAscBinary`**: SLheader (slType=2 CONACC, slAttribute=60), SQ_ASSOC, SQ_ASCBINARY, int 61, server's float type "IEEEI", "srvinfx" padded, the version string `"IBM Informix Dynamic Server Version 15.0.1.0.3"`, "serial", "informix" instance name, the three capability ints, "on" + "soctcp" + secondary protocol identifiers, container hostname `2327c4354ea8`, `/home/informix`, `/opt/ibm/informix/v15.0.1.0.3/bin/oninit`, SQ_ASCEOT=127.
|
||
- **Post-login messages are bare** (no SLheader, no length prefix). Each message is `[short tag][payload]`. Confirmed across all post-login traffic in the captures.
|
||
- **`[short 0x000c]` (SQ_EOT=12) is a per-PDU flush/submit marker**, not just a disconnect ack. Every client→server "logical request" in the post-login captures ends with `00 0c`. Server replies the same way. So the on-wire pattern is:
|
||
```
|
||
[short tag1][payload1][short tag2][payload2]...[short SQ_EOT=12] # one logical request
|
||
```
|
||
- **Disconnection is bare `[short SQ_EXIT=56]` both directions**, sometimes with interleaved `SQ_XACTSTAT=99` records before the final ack. Confirmed.
|
||
|
||
### Wire findings that AMENDED the JDBC-derived hypothesis 🔵
|
||
|
||
- **The capability section is three 4-byte ints** (Cap_1, Cap_2, Cap_3) — confirming the JDBC source's three `writeInt` calls. The values are NOT all zero as my earlier text claimed; the wire shows:
|
||
- `Cap_1 = 0x0000013c` (= 316). Looks like `(capability_class << 8) | protocol_version`, where `protocol_version = 0x3c = PF_PROT_SQLI_0600` (= 60). The high byte `0x01` appears to be a "supported feature class" indicator.
|
||
- `Cap_2 = 0`
|
||
- `Cap_3 = 0`
|
||
|
||
An earlier draft of this section misread the byte alignment and claimed `Cap_1=1, Cap_2=0x3c000000`. That was wrong — the `0x3c` byte that appeared to be in `Cap_2` was actually the *last byte of Cap_1*. Phase 1 caught this when we wrote a byte-for-byte PDU diff against the JDBC reference (`tests/test_pdu_match.py`); the dev-image server is permissive enough to accept the wrong values, so the connection succeeded but the bytes weren't structurally identical to JDBC's reference PDU. **Lesson**: server-accepts ≠ structurally-correct. Always diff against IBM's reference for protocol-bearing bytes.
|
||
|
||
Our `connections.py` now sends the correct values. The PDU diff test asserts byte-for-byte match for offsets 2..280 (everything except the SLheader length and process-specific tail).
|
||
|
||
### Validated post-login execution shape 🔵
|
||
|
||
The first SELECT the JDBC driver issues after login is internal — looking up locale info from `informix.systables`. The wire bytes for that 88-byte client→server message:
|
||
|
||
```
|
||
00 02 short SQ_PREPARE = 2
|
||
00 00 short 0 (flags / reserved)
|
||
00 00 00 49 int 73 (length of SQL bytes including trailing nul)
|
||
73 65 6c 65 63 74 20 46 49 52 53 54 20 31 20 73 69 74 65 20 66 72 6f 6d 20 69 6e 66 6f 72 6d 69 78 2e 73 79 73 74 61 62 6c 65 73 20 77 68 65 72 65 20 74 61 62 6e 61 6d 65 20 3d 20 27 20 47 4c 5f 43 4f 4c 4c 41 54 45 27 00
|
||
73 bytes of SQL text (incl. nul):
|
||
"select FIRST 1 site from informix.systables where tabname = ' GL_COLLATE'\0"
|
||
00 16 short 22 (= ?)
|
||
00 31 short 49 (= ?)
|
||
00 0c short SQ_EOT = 12 — flush
|
||
```
|
||
|
||
So the SQ_PREPARE message body is approximately:
|
||
```
|
||
[short SQ_PREPARE=2]
|
||
[short flags] # observed 0
|
||
[int sqlLen] # length INCLUDING trailing nul terminator
|
||
[bytes sql][nul]
|
||
[short ?] # observed 0x16 — purpose TBD
|
||
[short ?] # observed 0x31 — purpose TBD
|
||
[short SQ_EOT=12] # flush
|
||
```
|
||
|
||
The corresponding response (82 bytes) starts `00 08` (= SQ_DESCRIBE=8), with column metadata including `00 04 28 63 6f 6e 73 74 61 6e 74 29 00 00` = the column name `"(constant)\0"` (length-prefixed at length=4 — but content is "(constant)" 10 chars + nul; so the length=4 here is NOT the column name length). The exact column-descriptor layout needs more analysis, deferred to Phase 2.
|
||
|
||
### Phase 0 exit verdict
|
||
|
||
| Phase 0 exit criterion | 🟡 JDBC | 🔵 PCAP | Status |
|
||
|-------------------|--------|--------|--------|
|
||
| Wire framing primitives (endianness, widths, padding) | ✅ | ✅ | ✅ confirmed |
|
||
| Login byte layout (binary path) | ✅ | ✅ | ✅ confirmed |
|
||
| Disconnection | ✅ | ✅ | ✅ confirmed |
|
||
| Message-type tag enumeration | ✅ | ✅ (sample) | ✅ confirmed |
|
||
| `SQ_PREPARE`/`SELECT` request layout | partial | ✅ | ✅ confirmed (high-level) |
|
||
| Result-set column descriptor layout | ⬜ | partial | 🟠 needs Phase 2 work |
|
||
| Error response (login) | ✅ | ⬜ (no failure case captured) | 🟠 ok for Phase 0 |
|
||
| Error response (statement-time) | ⬜ | ⬜ | 🟠 deferred to Phase 2 |
|
||
| Capability bitfield semantics | ⬜ | observed | 🟠 ok for Phase 0 |
|
||
|
||
**Phase 0 exit recommendation**: 🟢 **GO**. The four hard exit criteria (login, message tags, SELECT round-trip, JDBC↔PCAP corroboration) are confirmed. The remaining gaps (result-set descriptor exact layout, error responses, capability semantics) are bounded and tractable — they'll be characterized in Phase 2 against the existing captures + targeted new captures, and don't change the project's architecture.
|
||
|
||
The "narrow scope" off-ramp is **not needed**. The wire protocol is well-formed, plain-password auth works, and the mismatch between JDBC source and wire on capability ints is small and explainable (decompiler artifact). Proceed to Phase 1.
|