Add Building TLE Catalogs guide, cross-reference pg-orrery-catalog

New guide: guides/catalog-management.mdx covering the full
download/merge/load pipeline with pg-orrery-catalog CLI.

Updated tracking-satellites.mdx to reference the companion tool
instead of "No TLE fetching". Added cross-reference in benchmarks
setup instructions.
This commit is contained in:
Ryan Malloy 2026-02-18 10:19:13 -07:00
parent 747b7ae60a
commit 26ae80d340
4 changed files with 308 additions and 4 deletions

View File

@ -70,6 +70,7 @@ export default defineConfig({
{ label: "JPL DE Ephemeris", slug: "guides/de-ephemeris" }, { label: "JPL DE Ephemeris", slug: "guides/de-ephemeris" },
{ label: "Orbit Determination", slug: "guides/orbit-determination" }, { label: "Orbit Determination", slug: "guides/orbit-determination" },
{ label: "Satellite Pass Prediction", slug: "guides/pass-prediction" }, { label: "Satellite Pass Prediction", slug: "guides/pass-prediction" },
{ label: "Building TLE Catalogs", slug: "guides/catalog-management" },
], ],
}, },
{ {

View File

@ -0,0 +1,302 @@
---
title: Building TLE Catalogs
sidebar:
order: 12
---
import { Steps, Aside, Tabs, TabItem, Code } from "@astrojs/starlight/components";
Every pg_orrery workflow starts with TLEs in a table. The [Tracking Satellites](/guides/tracking-satellites/) guide shows how to insert a few satellites by hand --- but a real catalog has tens of thousands of objects from multiple sources, each with different freshness and coverage. `pg-orrery-catalog` handles the download, merge, and load pipeline.
## The problem with multiple TLE sources
Three major sources provide TLE data, each with trade-offs:
| Source | Auth | Coverage | Freshness |
|--------|------|----------|-----------|
| [Space-Track](https://www.space-track.org) | Login required | Full catalog (~30k+ on-orbit) | Hours to days |
| [CelesTrak](https://celestrak.org) | None | Active sats + operator supplemental GP | Minutes to hours |
| [SatNOGS](https://db.satnogs.org) | None | Community-tracked objects | Varies |
The same satellite often appears in all three. CelesTrak's supplemental GP (SupGP) data is particularly valuable --- operators like SpaceX submit Starlink ephemerides that are often hours fresher than Space-Track's own catalog.
The question is which entry to keep. `pg-orrery-catalog` answers with epoch-based deduplication: when the same NORAD ID appears in multiple sources, the entry with the newest epoch wins. This means SupGP data automatically overrides stale Space-Track entries where available.
## Install
```bash
# Run directly (no install needed)
uvx pg-orrery-catalog --help
# Or install permanently
uv pip install pg-orrery-catalog
# For direct database loading (adds psycopg)
uv pip install "pg-orrery-catalog[pg]"
```
## Download, build, load
The typical workflow is three steps. Each can run independently.
<Steps>
1. **Download** TLE data from remote sources into the local cache:
```bash
pg-orrery-catalog download
```
This fetches from all configured sources (CelesTrak by default, Space-Track if credentials are set). Files are cached in `~/.cache/pg-orrery-catalog/` and reused unless stale (>24h) or `--force` is passed.
To download from a specific source:
```bash
pg-orrery-catalog download --source celestrak
pg-orrery-catalog download --source spacetrack --force
```
2. **Build** a merged catalog and output it:
<Tabs>
<TabItem label="Pipe to psql">
```bash
pg-orrery-catalog build | psql -d mydb
```
</TabItem>
<TabItem label="Save SQL file">
```bash
pg-orrery-catalog build --table satellites -o catalog.sql
```
</TabItem>
<TabItem label="Export 3LE">
```bash
pg-orrery-catalog build --format 3le -o merged.tle
```
</TabItem>
<TabItem label="JSON output">
```bash
pg-orrery-catalog build --format json -o catalog.json
```
</TabItem>
</Tabs>
With no arguments, `build` merges all cached files. You can also pass specific TLE files:
```bash
pg-orrery-catalog build /path/to/spacetrack.tle /path/to/celestrak.tle
```
The merge reports what happened:
```
spacetrack_everything: 33053 objects (33053 new, 0 updated)
celestrak_active: 14376 objects (2 new, 0 updated)
satnogs_full: 1488 objects (121 new, 5 updated)
supgp_starlink: 9703 objects (77 new, 7398 updated)
Total: 33253 unique objects
Regimes: LEO: 31542, GEO: 1203, MEO: 385, HEO: 123
```
Notice how SupGP updated 7,398 Starlink entries --- those are fresher epochs from SpaceX overriding stale Space-Track data.
3. **Load** directly into PostgreSQL (requires `[pg]` extra):
```bash
pg-orrery-catalog load \
--database-url postgresql:///mydb \
--table satellites \
--create-index
```
The `--create-index` flag creates both GiST and SP-GiST indexes on the `tle` column, ready for spatial queries and KNN ordering.
</Steps>
## Configuration
Three layers, highest precedence first:
1. **CLI flags** --- `--table`, `--source`, `--database-url`
2. **Environment variables** --- `SPACETRACK_USER`, `SOCKS_PROXY`, `DATABASE_URL`
3. **Config file** --- `~/.config/pg-orrery-catalog/config.toml`
### Space-Track credentials
Space-Track requires a free account. Set credentials via environment variables:
```bash
export SPACETRACK_USER="you@example.com"
export SPACETRACK_PASSWORD="secret"
pg-orrery-catalog download --source spacetrack
```
Or in the config file:
```toml
[spacetrack]
user = "you@example.com"
password = "secret"
```
### SOCKS proxy
CelesTrak is sometimes unreachable from certain networks. Route through a SOCKS5 proxy:
```bash
export SOCKS_PROXY="localhost:1080"
pg-orrery-catalog download
```
### Full config reference
```toml
[spacetrack]
user = "you@example.com"
password = "secret"
[celestrak]
proxy = "localhost:1080"
supgp_groups = ["starlink", "oneweb", "planet", "orbcomm"]
[output]
table = "satellites"
[database]
url = "postgresql://localhost/mydb"
```
## Working with the generated SQL
The SQL output creates a table with three columns:
```sql
CREATE TABLE satellites (
id serial,
name text,
tle tle
);
```
Once loaded, the full pg_orrery function set is available:
```sql
-- Where is every LEO satellite right now?
SELECT name, observe('40.0N 105.3W 1655m'::observer, tle, now()) AS topo
FROM satellites
WHERE tle_mean_motion(tle) > 11.25;
-- Which satellites are overhead right now?
SELECT name, topo_elevation(observe('40.0N 105.3W 1655m'::observer, tle, now())) AS el
FROM satellites
WHERE topo_elevation(observe('40.0N 105.3W 1655m'::observer, tle, now())) > 10
ORDER BY el DESC;
-- Predict ISS passes for the next 24 hours
SELECT pass_aos(p), pass_max_el(p), pass_los(p)
FROM satellites,
predict_passes(tle, '40.0N 105.3W 1655m'::observer,
now(), now() + interval '24 hours', 10.0) p
WHERE norad_id = 25544;
```
## NORAD ID encoding
TLE files use a 5-character field for the NORAD catalog number. With more than 100,000 tracked objects, the original 5-digit numeric format ran out of space. The encoding has evolved through four cases:
| Case | Format | Range | Example |
|------|--------|-------|---------|
| Traditional | `ddddd` | 0 -- 99,999 | `25544` (ISS) |
| Alpha-5 | `Ldddd` | 100,000 -- 339,999 | `T0002` = 270,002 |
| Super-5 case 3 | `xxxxX` | 340,000 -- 906,309,663 | `0000A` = 340,000 |
| Super-5 case 4 | `xxxXd` | 906,309,664+ | `000A0` = 906,309,664 |
Alpha-5 skips the letters I and O (they look like 1 and 0). Super-5 uses a base-64 alphabet: digits 0--9, uppercase A--Z, lowercase a--z, plus `+` and `-`.
`pg-orrery-catalog` decodes all four cases, matching the `get_norad_number()` implementation in pg_orrery's vendored SGP4 library. This means Alpha-5 objects like Starlink satellites (NORAD IDs above 100,000) load correctly.
<Aside type="note" title="Alpha-5 verification">
You can verify the decoding independently:
```bash
python3 -c "
from pg_orrery_catalog.tle import decode_norad
print(f'T0002 = {decode_norad(\"T0002\")}') # 270002
print(f'A0001 = {decode_norad(\"A0001\")}') # 100001
print(f'Z9999 = {decode_norad(\"Z9999\")}') # 339999
"
```
</Aside>
## Cache management
Downloaded TLE files are stored under `~/.cache/pg-orrery-catalog/`, organized by source:
```
~/.cache/pg-orrery-catalog/
celestrak/
celestrak_active.tle
supgp_starlink.tle
supgp_oneweb.tle
...
satnogs/
satnogs_full.tle
spacetrack/
spacetrack_everything.tle
```
Check what's cached:
```bash
pg-orrery-catalog info --cache
```
Files older than 24 hours are considered stale and re-downloaded automatically. Use `--force` to override fresh cache entries.
## Automating catalog updates
For a regularly-updated catalog, a cron job or systemd timer works well:
```bash
# Update catalog daily at 03:00
0 3 * * * pg-orrery-catalog download && pg-orrery-catalog build --table satellites | psql -d mydb
```
Or with the direct load command:
```bash
0 3 * * * pg-orrery-catalog download && pg-orrery-catalog load --database-url postgresql:///mydb --table satellites --create-index
```
<Aside type="caution" title="Table replacement">
The default SQL output includes `DROP TABLE IF EXISTS` before `CREATE TABLE`. This replaces the entire table on each load. If you need to preserve the table and upsert, export as JSON and handle the merge in your application logic.
</Aside>
## Using as a library
`pg-orrery-catalog` can also be imported as a Python library:
```python
from pg_orrery_catalog.tle import decode_norad, parse_3le_file
from pg_orrery_catalog.catalog import merge_sources
from pg_orrery_catalog.regime import regime_summary
from pg_orrery_catalog.output.sql import generate_sql
# Parse and merge
merged, stats = merge_sources(["spacetrack.tle", "celestrak.tle"])
print(f"{stats.total_unique} unique objects")
# Classify
regimes = regime_summary(merged)
print(regimes) # {'LEO': 31542, 'MEO': 385, 'GEO': 1203, 'HEO': 123}
# Generate SQL
sql = generate_sql(merged, table="my_catalog")
```
## What's next
With a catalog loaded, see:
- [Tracking Satellites](/guides/tracking-satellites/) --- observe, predict passes, screen conjunctions
- [Satellite Pass Prediction](/guides/pass-prediction/) --- detailed pass prediction workflows
- [Conjunction Screening](/guides/conjunction-screening/) --- find close approaches using GiST indexes
- [Benchmarks](/performance/benchmarks/) --- performance data with catalogs of 33k--66k objects

View File

@ -43,7 +43,7 @@ pg_orrery propagates TLEs and computes look angles. It does not replace the full
- **No real-time GUI.** GPredict and STK provide map displays, polar plots, and Doppler displays. pg_orrery returns numbers. Use any visualization tool to render its output. - **No real-time GUI.** GPredict and STK provide map displays, polar plots, and Doppler displays. pg_orrery returns numbers. Use any visualization tool to render its output.
- **No rotator control.** Hamlib drives antenna rotators. pg_orrery computes the azimuth and elevation values Hamlib would consume, but it has no hardware interface. - **No rotator control.** Hamlib drives antenna rotators. pg_orrery computes the azimuth and elevation values Hamlib would consume, but it has no hardware interface.
- **No TLE fetching.** Bring your own TLEs from Space-Track, CelesTrak, or any provider. pg_orrery parses and propagates them. - **TLE fetching via companion tool.** pg_orrery itself doesn't download TLEs, but [`pg-orrery-catalog`](/guides/catalog-management/) handles the full pipeline: download from Space-Track, CelesTrak, and SatNOGS, merge with epoch-based dedup, and load into PostgreSQL.
- **Orbit determination available.** Since v0.4.0, pg_orrery can fit TLEs from ECI, topocentric, or angles-only observations via differential correction. See the [Orbit Determination guide](/guides/orbit-determination/). - **Orbit determination available.** Since v0.4.0, pg_orrery can fit TLEs from ECI, topocentric, or angles-only observations via differential correction. See the [Orbit Determination guide](/guides/orbit-determination/).
- **No high-precision propagation.** SGP4/SDP4 accuracy degrades with TLE age. For operational conjunction assessment, use SP ephemerides or owner/operator-provided state vectors. pg_orrery's GiST screening finds candidates; you verify with better data. - **No high-precision propagation.** SGP4/SDP4 accuracy degrades with TLE age. For operational conjunction assessment, use SP ephemerides or owner/operator-provided state vectors. pg_orrery's GiST screening finds candidates; you verify with better data.

View File

@ -284,7 +284,7 @@ For a 65,886-object catalog and a 2-hour window from Eagle, Idaho:
<Tabs> <Tabs>
<TabItem label="Requirements"> <TabItem label="Requirements">
- PostgreSQL 17 with pg_orrery installed - PostgreSQL 17 with pg_orrery installed
- A satellite catalog table with ~12,000 TLEs (available from CelesTrak) - A satellite catalog table with ~12,000 TLEs (see [Building TLE Catalogs](/guides/catalog-management/) or download directly from CelesTrak)
- A star catalog table (any subset of Hipparcos or Yale BSC) - A star catalog table (any subset of Hipparcos or Yale BSC)
- No concurrent queries during measurement - No concurrent queries during measurement
- `shared_buffers` and `work_mem` at default or higher - `shared_buffers` and `work_mem` at default or higher
@ -293,9 +293,10 @@ For a 65,886-object catalog and a 2-hour window from Eagle, Idaho:
```sql ```sql
CREATE EXTENSION pg_orrery; CREATE EXTENSION pg_orrery;
-- Load a TLE catalog -- Load a TLE catalog (pg-orrery-catalog handles this)
-- pg-orrery-catalog build --table satellite_catalog | psql -d mydb
CREATE TABLE satellite_catalog (tle tle); CREATE TABLE satellite_catalog (tle tle);
-- (COPY from CelesTrak bulk TLE file) -- (or COPY from CelesTrak bulk TLE file)
-- Verify catalog size -- Verify catalog size
SELECT count(*) FROM satellite_catalog; SELECT count(*) FROM satellite_catalog;