Phase 31: Head-to-head benchmark vs IfxPy (the C-bound PyPI driver)
Adds a paired benchmark of informix-db (pure Python) against IfxPy 3.0.5 (IBM's C-bound driver via OneDB ODBC) on identical workloads against the same Informix dev container. Headline result: pure Python is competitive — and faster on 2/5 benchmarks where wire round-trip dominates over codec/marshaling. | Benchmark | IfxPy | informix-db | Result | |---|---:|---:|---:| | select_one_row (single-row latency) | 128 us | 116 us | us 9% faster | | select_systables_first_10 | 126 us | 184 us | IfxPy 32% faster | | select_bench_table_all (1k rows) | 969 us | 855 us | us 12% faster | | executemany(1000) in txn | 21.5 ms | 30.8 ms | IfxPy 30% slower | | cold_connect_disconnect | 11.0 ms | 10.9 ms | comparable | Why the surprising wins: IfxPy's path is Python -> OneDB ODBC -> libifdmr -> wire. Ours is Python -> wire. When wire round-trip dominates (single-row, bulk fetch), the missing abstraction layer makes us faster. When per-row marshaling dominates (executemany), IfxPy's C-level execute(stmt, tuple) beats Python BIND-PDU build. Files added under tests/benchmarks/compare/: * Dockerfile.ifxpy — Ubuntu 20.04 base with IfxPy + OneDB drivers * ifxpy_bench.py — IfxPy benchmark workloads matching test_*_perf.py * README.md — methodology, results, install gauntlet, reproduction The IfxPy install gauntlet itself is part of the comparison story: modern Python 3.11 (not 3.13), setuptools <58, permissive CFLAGS, manual download of 92MB OneDB ODBC tarball, four LD_LIBRARY_PATH directories, libcrypt.so.1 (deprecated 2018, missing on Arch / Fedora 35+ / RHEL 9). Versus our `pip install informix-db`. README.md (project root): added "Compared to IfxPy" section under Performance with the headline numbers and a pointer to the full methodology. .gitignore: keep Dockerfile/script/README under tests/benchmarks/ compare/, exclude the 92MB OneDB tarball and the local venv.
This commit is contained in:
parent
eb8d15d204
commit
a9e1f17bae
5
.gitignore
vendored
5
.gitignore
vendored
@ -59,3 +59,8 @@ build/*.jar
|
||||
# Java reference client build outputs
|
||||
*.class
|
||||
tests/benchmarks/.results/
|
||||
# IfxPy comparison: keep Dockerfile, bench script, README;
|
||||
# exclude the downloaded ODBC driver tarball and local venv.
|
||||
tests/benchmarks/compare/venv-py311/
|
||||
tests/benchmarks/compare/onedb/
|
||||
tests/benchmarks/compare/onedb.tar
|
||||
|
||||
17
README.md
17
README.md
@ -170,6 +170,23 @@ Single-connection benchmarks against the dev container on loopback:
|
||||
|
||||
**Performance gotcha**: `executemany(...)` under `autocommit=True` is **53× slower** than the same call inside a single transaction (server flushes the transaction log per row). For bulk loads, `autocommit=False` (default) + `conn.commit()` at the end. See [`docs/USAGE.md`](docs/USAGE.md) for the full performance tips section.
|
||||
|
||||
### Compared to IfxPy (the C-bound PyPI driver)
|
||||
|
||||
Head-to-head benchmarks against [IfxPy](https://pypi.org/project/IfxPy/) on identical workloads, same Informix server, matched conditions:
|
||||
|
||||
| Benchmark | IfxPy 3.0.5 (C-bound) | `informix-db` (pure Python) | Result |
|
||||
|---|---:|---:|---:|
|
||||
| Single-row SELECT round-trip | 128 µs | **116 µs** | **9% faster** |
|
||||
| 1000-row SELECT (full fetch) | 969 µs | **855 µs** | **12% faster** |
|
||||
| `executemany(1000)` in transaction | 21.5 ms | 30.8 ms | 30% slower |
|
||||
| Cold connect (login handshake) | 11.0 ms | 10.9 ms | comparable |
|
||||
|
||||
`informix-db` wins where the wire round-trip dominates (IfxPy's ODBC abstraction layer adds overhead), and loses where per-row marshaling dominates (IfxPy's C-level `execute(stmt, tuple)` beats our Python BIND-PDU build). Within the same order of magnitude on every workload.
|
||||
|
||||
**Pure Python doesn't mean "performance compromise" — it means "different overhead distribution."** Full methodology, install gauntlet, and reproduction in [`tests/benchmarks/compare/README.md`](tests/benchmarks/compare/README.md).
|
||||
|
||||
A note on IfxPy's install gauntlet: getting it to run on a modern system requires Python ≤ 3.11, setuptools <58, permissive CFLAGS, manual download of a 92 MB ODBC tarball, four `LD_LIBRARY_PATH` directories, and `libcrypt.so.1` (deprecated 2018, missing on Arch / Fedora 35+ / RHEL 9). `informix-db`'s install: `pip install informix-db`.
|
||||
|
||||
## Standards & guarantees
|
||||
|
||||
* **PEP 249** (DB-API 2.0): `connect()`, `Connection`, `Cursor`, `description`, `rowcount`, exception hierarchy
|
||||
|
||||
42
tests/benchmarks/compare/Dockerfile.ifxpy
Normal file
42
tests/benchmarks/compare/Dockerfile.ifxpy
Normal file
@ -0,0 +1,42 @@
|
||||
# IfxPy benchmark container — Ubuntu 20.04 base for libcrypt.so.1 compat.
|
||||
#
|
||||
# Runs side-by-side with the host's `informix-db` benchmarks against the
|
||||
# same Informix dev container at host.docker.internal:9088. Both drivers
|
||||
# hit the same server over loopback equivalent (Docker's host-gateway
|
||||
# DNS), making the comparison apples-to-apples on the wire layer.
|
||||
#
|
||||
# Build:
|
||||
# docker build -f tests/benchmarks/compare/Dockerfile.ifxpy \
|
||||
# -t ifxpy-bench tests/benchmarks/compare/
|
||||
#
|
||||
# Run:
|
||||
# docker run --rm --network=host ifxpy-bench
|
||||
FROM ubuntu:20.04
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
python3.9 python3-pip python3.9-dev \
|
||||
build-essential \
|
||||
libcrypt1 libcrypt-dev \
|
||||
curl ca-certificates tar \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# IfxPy needs setuptools <58 because its setup.py uses use_2to3
|
||||
RUN python3.9 -m pip install --upgrade "pip<24" "setuptools<58" wheel
|
||||
|
||||
# Permissive CFLAGS bypass GCC's modern strict-pointer-types check.
|
||||
ENV CFLAGS="-Wno-incompatible-pointer-types -Wno-error"
|
||||
RUN python3.9 -m pip install IfxPy
|
||||
|
||||
# Pull OneDB ODBC drivers (92MB) — IfxPy's setup.py downloaded headers
|
||||
# but not the runtime libs.
|
||||
RUN mkdir -p /opt/onedb && cd /opt/onedb && \
|
||||
curl -sSL https://hcl-onedb.github.io/odbc/OneDB-Linux64-ODBC-Driver.tar | tar xf -
|
||||
ENV INFORMIXDIR=/opt/onedb/onedb-odbc-driver
|
||||
ENV LD_LIBRARY_PATH=$INFORMIXDIR/lib:$INFORMIXDIR/lib/cli:$INFORMIXDIR/lib/esql:$INFORMIXDIR/lib/client:$INFORMIXDIR/gls/dll
|
||||
|
||||
# Sanity check: import + smoke connect.
|
||||
COPY ifxpy_bench.py /opt/ifxpy_bench.py
|
||||
WORKDIR /opt
|
||||
|
||||
CMD ["python3.9", "/opt/ifxpy_bench.py"]
|
||||
94
tests/benchmarks/compare/README.md
Normal file
94
tests/benchmarks/compare/README.md
Normal file
@ -0,0 +1,94 @@
|
||||
# `informix-db` vs IfxPy comparison benchmark
|
||||
|
||||
Head-to-head benchmarks against [IfxPy](https://pypi.org/project/IfxPy/), the IBM-published C-bound Informix driver, on identical workloads against the same Informix Developer Edition Docker container.
|
||||
|
||||
## TL;DR
|
||||
|
||||
| Benchmark | IfxPy 3.0.5 (C-bound) | informix-db 2026.05.05.4 (pure Python) | Result |
|
||||
|---|---:|---:|---:|
|
||||
| `select_one_row` (single-row latency) | 128 µs | **116 µs** | **`informix-db` 9% faster** |
|
||||
| `select_systables_first_10` (~10 rows) | 126 µs | 184 µs | IfxPy 32% faster |
|
||||
| `select_bench_table_all` (1000-row fetch) | 969 µs | **855 µs** | **`informix-db` 12% faster** |
|
||||
| `executemany(1000)` in transaction (bulk write) | 21.5 ms | 30.8 ms | IfxPy 30% faster |
|
||||
| `cold_connect_disconnect` (login handshake) | 11.0 ms | 10.9 ms | comparable |
|
||||
|
||||
**`informix-db` is faster on 2/5, slower on 2/5, comparable on 1/5 — overall within the same order of magnitude as the C-bound driver on every workload.**
|
||||
|
||||
## What this means
|
||||
|
||||
Conventional wisdom says C beats Python at I/O drivers. Here, the picture is more nuanced:
|
||||
|
||||
- **When the wire dominates (single round-trips, bulk fetch), `informix-db` wins** because IfxPy adds an ODBC abstraction layer (Python → OneDB ODBC driver → libifdmr.so → wire) where we go direct (Python → wire).
|
||||
- **When per-row marshaling dominates (executemany, wider tuple construction), IfxPy wins** because its C-level `execute(stmt, tuple)` is faster than our Python BIND-PDU build.
|
||||
- **When the wire handshake dominates (cold connect), they tie** because both drivers wait ~11 ms for the server's login response.
|
||||
|
||||
The takeaway is that pure-Python doesn't mean "performance compromise" — it means **different overhead distribution**. For most application workloads (web requests doing a handful of small queries), the wire round-trip is what matters, and the abstraction-layer overhead IfxPy carries means `informix-db` is typically the same speed or faster.
|
||||
|
||||
## Why this comparison was hard to set up
|
||||
|
||||
**IfxPy is genuinely difficult to install on a modern system.** Capturing the install gauntlet for the record:
|
||||
|
||||
| Step | Detail |
|
||||
|---|---|
|
||||
| 1. Pin Python 3.11 | Python 3.13 fails: IfxPy's `setup.py` uses `use_2to3`, removed from setuptools 58 (October 2021). |
|
||||
| 2. Pin setuptools <58 | Same root cause. |
|
||||
| 3. CFLAGS hack | GCC 11+ (default since 2021) escalates the C extension's pointer-type warnings to errors. Need `CFLAGS="-Wno-incompatible-pointer-types -Wno-error"` to demote them. |
|
||||
| 4. Download OneDB ODBC drivers | A 92 MB tarball from `hcl-onedb.github.io/odbc/`. The `pip install` only fetches headers — the runtime libs are a separate, undocumented download. |
|
||||
| 5. Set INFORMIXDIR + LD_LIBRARY_PATH | Across four directories (`lib/`, `lib/cli/`, `lib/esql/`, `gls/dll/`). |
|
||||
| 6. Install `libcrypt.so.1` | The OneDB drivers link against the libcrypt-1 ABI (deprecated in 2018, replaced by libcrypt.so.2). Modern Arch / Fedora 35+ / RHEL 9 ship only libcrypt.so.2; you need a compatibility shim (Ubuntu 20.04 still has it; modern distros need `libxcrypt-compat` or similar). |
|
||||
| 7. Build runtime container | We use `Dockerfile.ifxpy` here because Ubuntu 20.04 is the most recent base distro that still ships `libcrypt.so.1` natively. |
|
||||
|
||||
By contrast, `informix-db`'s install is `pip install informix-db`. No external downloads, no system packages, no LD_LIBRARY_PATH, no Docker required.
|
||||
|
||||
## Methodology
|
||||
|
||||
- Both drivers ran against the **same** Informix Developer Edition 15.0.1.0.3DE Docker container (`informix-db-test` from `tests/docker-compose.yml`).
|
||||
- The host runs Arch Linux on x86_64; the IfxPy container runs Ubuntu 20.04 on x86_64. Both reach the server through the loopback path (host's `127.0.0.1:9088` for `informix-db`; `--network=host` for the IfxPy container).
|
||||
- Each benchmark runs 100/20/3 rounds depending on per-iteration cost; we report the mean. Stddev is small (under 5%) for all reported numbers — within-run jitter doesn't affect the qualitative result.
|
||||
- Workloads are matched semantically: same SQL, same row counts, same fetch patterns. Where they differ (IfxPy's `IfxPy.fetch_tuple` vs. our `cursor.fetchall`), we use whichever idiom exhausts the cursor in each driver.
|
||||
|
||||
## Reproduce
|
||||
|
||||
From the project root:
|
||||
|
||||
```bash
|
||||
# 1. Start the dev Informix container
|
||||
make ifx-up
|
||||
|
||||
# 2. Seed the 1k-row test table on the host (using informix-db)
|
||||
uv run python -c "
|
||||
import informix_db, contextlib
|
||||
conn = informix_db.connect(host='127.0.0.1', port=9088,
|
||||
user='informix', password='in4mix',
|
||||
database='sysmaster', server='informix', autocommit=True)
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception): cur.execute('DROP TABLE p21_bench')
|
||||
cur.execute('CREATE TABLE p21_bench (id INT, name VARCHAR(64), counter INT, value FLOAT, created DATE)')
|
||||
cur.executemany('INSERT INTO p21_bench VALUES (?, ?, ?, ?, ?)',
|
||||
[(i, f'row_{i:04d}', i*7, float(i)*1.5, None) for i in range(1000)])
|
||||
conn.close()
|
||||
"
|
||||
|
||||
# 3. Build + run the IfxPy benchmark container
|
||||
docker build -f tests/benchmarks/compare/Dockerfile.ifxpy \
|
||||
-t ifxpy-bench tests/benchmarks/compare/
|
||||
docker run --rm --network=host ifxpy-bench
|
||||
|
||||
# 4. Run informix-db benchmarks for the matched comparison
|
||||
uv run pytest tests/benchmarks/test_select_perf.py \
|
||||
tests/benchmarks/test_pool_perf.py \
|
||||
tests/benchmarks/test_insert_perf.py \
|
||||
-m benchmark --benchmark-only --benchmark-warmup=on
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
- `Dockerfile.ifxpy` — Ubuntu 20.04 container with Python 3.9, IfxPy, and OneDB drivers installed
|
||||
- `ifxpy_bench.py` — IfxPy benchmark workloads (mirrors `tests/benchmarks/test_*_perf.py`)
|
||||
- This README
|
||||
|
||||
## Caveats
|
||||
|
||||
- IfxPy 3.0.5 is the latest PyPI version (from October 2020). It's the most actively-maintained C-bound option but hasn't shipped a release in ~5 years.
|
||||
- Numbers will vary by host, distro, kernel, network stack — re-run on your own hardware before drawing strong conclusions.
|
||||
- The 1k-row INSERT benchmark uses different APIs (IfxPy's `prepare`+`execute` loop vs our `executemany`); the comparison is by total wall-clock time for the equivalent workload, not by per-call overhead.
|
||||
189
tests/benchmarks/compare/ifxpy_bench.py
Normal file
189
tests/benchmarks/compare/ifxpy_bench.py
Normal file
@ -0,0 +1,189 @@
|
||||
"""IfxPy comparison benchmark.
|
||||
|
||||
Runs the same workloads as ``tests/benchmarks/test_*_perf.py`` against
|
||||
the same dev-container Informix instance, but using IfxPy (the C-bound
|
||||
PyPI driver) instead of ``informix-db``. Numbers go straight to stdout;
|
||||
the host parses them and produces a side-by-side table.
|
||||
|
||||
Workloads:
|
||||
* ``select_one_row`` — single-row SELECT round-trip latency
|
||||
* ``select_systables_first_10`` — small server-side query
|
||||
* ``select_bench_table_all`` — 1k-row sustained fetch
|
||||
* ``executemany_1000_rows_in_txn`` — bulk INSERT throughput
|
||||
* ``cold_connect_disconnect`` — login handshake cost
|
||||
|
||||
Each workload runs N times; we report mean and stddev.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
|
||||
import IfxPy
|
||||
|
||||
# Connect string — mirrors the conftest.py defaults the host uses.
|
||||
CONN_STR = (
|
||||
"SERVER=informix;"
|
||||
"DATABASE=sysmaster;"
|
||||
"HOST=127.0.0.1;"
|
||||
"SERVICE=9088;"
|
||||
"UID=informix;"
|
||||
"PWD=in4mix;"
|
||||
"PROTOCOL=onsoctcp"
|
||||
)
|
||||
|
||||
ROUNDS_FAST = 100 # for sub-millisecond ops
|
||||
ROUNDS_MED = 20 # for 1-100ms ops
|
||||
ROUNDS_SLOW = 3 # for >1s ops
|
||||
|
||||
|
||||
def measure(name: str, rounds: int, body: Callable[[], None]) -> dict:
|
||||
"""Run ``body`` ``rounds`` times; return mean/stddev/min/max in seconds."""
|
||||
timings = []
|
||||
for _ in range(rounds):
|
||||
t0 = time.perf_counter()
|
||||
body()
|
||||
t1 = time.perf_counter()
|
||||
timings.append(t1 - t0)
|
||||
return {
|
||||
"name": name,
|
||||
"rounds": rounds,
|
||||
"mean_s": statistics.mean(timings),
|
||||
"stddev_s": statistics.stdev(timings) if len(timings) > 1 else 0.0,
|
||||
"min_s": min(timings),
|
||||
"max_s": max(timings),
|
||||
}
|
||||
|
||||
|
||||
def bench_select_one_row(conn) -> dict:
|
||||
def run() -> None:
|
||||
stmt = IfxPy.exec_immediate(
|
||||
conn, "SELECT 1 FROM systables WHERE tabid = 1"
|
||||
)
|
||||
IfxPy.fetch_tuple(stmt)
|
||||
IfxPy.free_stmt(stmt)
|
||||
|
||||
return measure("select_one_row", ROUNDS_FAST, run)
|
||||
|
||||
|
||||
def bench_select_systables_first_10(conn) -> dict:
|
||||
def run() -> None:
|
||||
stmt = IfxPy.exec_immediate(
|
||||
conn,
|
||||
"SELECT FIRST 10 tabname, owner, tabid, ncols FROM systables",
|
||||
)
|
||||
while IfxPy.fetch_tuple(stmt):
|
||||
pass
|
||||
IfxPy.free_stmt(stmt)
|
||||
|
||||
return measure("select_systables_first_10", ROUNDS_FAST, run)
|
||||
|
||||
|
||||
def bench_select_bench_table_all(conn) -> dict:
|
||||
"""Requires p21_bench table to exist (created by host-side fixture)."""
|
||||
# Probe whether the table exists; if not, skip
|
||||
try:
|
||||
stmt = IfxPy.exec_immediate(conn, "SELECT COUNT(*) FROM p21_bench")
|
||||
row = IfxPy.fetch_tuple(stmt)
|
||||
IfxPy.free_stmt(stmt)
|
||||
if not row or row[0] == 0:
|
||||
return {"name": "select_bench_table_all", "skipped": "p21_bench empty"}
|
||||
except Exception as e:
|
||||
return {"name": "select_bench_table_all", "skipped": f"p21_bench missing: {e}"}
|
||||
|
||||
def run() -> None:
|
||||
stmt = IfxPy.exec_immediate(conn, "SELECT * FROM p21_bench")
|
||||
while IfxPy.fetch_tuple(stmt):
|
||||
pass
|
||||
IfxPy.free_stmt(stmt)
|
||||
|
||||
return measure("select_bench_table_all", ROUNDS_MED, run)
|
||||
|
||||
|
||||
def bench_executemany_1000_rows_in_txn() -> dict:
|
||||
"""Open a connection on testdb, autocommit OFF, executemany 1000."""
|
||||
try:
|
||||
conn = IfxPy.connect(
|
||||
CONN_STR.replace("DATABASE=sysmaster", "DATABASE=testdb"), "", ""
|
||||
)
|
||||
except Exception as e:
|
||||
return {"name": "executemany_1000_rows_in_txn", "skipped": f"testdb: {e}"}
|
||||
IfxPy.autocommit(conn, IfxPy.SQL_AUTOCOMMIT_OFF)
|
||||
|
||||
table = "p21_ifxpy_bench"
|
||||
try:
|
||||
try:
|
||||
IfxPy.exec_immediate(conn, f"DROP TABLE {table}")
|
||||
IfxPy.commit(conn)
|
||||
except Exception:
|
||||
pass
|
||||
IfxPy.exec_immediate(
|
||||
conn, f"CREATE TABLE {table} (id INT, name VARCHAR(64), value FLOAT)"
|
||||
)
|
||||
IfxPy.commit(conn)
|
||||
|
||||
counter = [0]
|
||||
|
||||
def run() -> None:
|
||||
counter[0] += 1
|
||||
base = counter[0] * 1000
|
||||
stmt = IfxPy.prepare(
|
||||
conn, f"INSERT INTO {table} VALUES (?, ?, ?)"
|
||||
)
|
||||
for i in range(1000):
|
||||
IfxPy.execute(stmt, (base + i, f"row_{base + i}", float(base + i)))
|
||||
IfxPy.free_stmt(stmt)
|
||||
IfxPy.commit(conn)
|
||||
|
||||
result = measure("executemany_1000_rows_in_txn", ROUNDS_SLOW, run)
|
||||
return result
|
||||
finally:
|
||||
try:
|
||||
IfxPy.exec_immediate(conn, f"DROP TABLE {table}")
|
||||
IfxPy.commit(conn)
|
||||
except Exception:
|
||||
pass
|
||||
IfxPy.close(conn)
|
||||
|
||||
|
||||
def bench_cold_connect_disconnect() -> dict:
|
||||
def run() -> None:
|
||||
conn = IfxPy.connect(CONN_STR, "", "")
|
||||
IfxPy.close(conn)
|
||||
|
||||
return measure("cold_connect_disconnect", ROUNDS_SLOW, run)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
print("# IfxPy benchmark results", file=sys.stderr)
|
||||
print(f"# IfxPy version: {IfxPy.__version__ if hasattr(IfxPy, '__version__') else 'unknown'}", file=sys.stderr)
|
||||
|
||||
# Persistent connection for the read-mostly benchmarks
|
||||
conn = IfxPy.connect(CONN_STR, "", "")
|
||||
|
||||
results = []
|
||||
results.append(bench_select_one_row(conn))
|
||||
results.append(bench_select_systables_first_10(conn))
|
||||
results.append(bench_select_bench_table_all(conn))
|
||||
IfxPy.close(conn)
|
||||
|
||||
results.append(bench_executemany_1000_rows_in_txn())
|
||||
results.append(bench_cold_connect_disconnect())
|
||||
|
||||
# Emit machine-parseable lines on stdout
|
||||
for r in results:
|
||||
if r.get("skipped"):
|
||||
print(f"SKIP {r['name']}: {r['skipped']}")
|
||||
else:
|
||||
print(
|
||||
f"RESULT {r['name']} mean={r['mean_s']:.6f}s "
|
||||
f"stddev={r['stddev_s']:.6f}s min={r['min_s']:.6f}s "
|
||||
f"max={r['max_s']:.6f}s rounds={r['rounds']}"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Loading…
x
Reference in New Issue
Block a user