diff --git a/docs/astro.config.mjs b/docs/astro.config.mjs index 7768437..256e409 100644 --- a/docs/astro.config.mjs +++ b/docs/astro.config.mjs @@ -70,6 +70,7 @@ export default defineConfig({ { label: "JPL DE Ephemeris", slug: "guides/de-ephemeris" }, { label: "Orbit Determination", slug: "guides/orbit-determination" }, { label: "Satellite Pass Prediction", slug: "guides/pass-prediction" }, + { label: "Building TLE Catalogs", slug: "guides/catalog-management" }, ], }, { diff --git a/docs/src/content/docs/guides/catalog-management.mdx b/docs/src/content/docs/guides/catalog-management.mdx new file mode 100644 index 0000000..9968948 --- /dev/null +++ b/docs/src/content/docs/guides/catalog-management.mdx @@ -0,0 +1,302 @@ +--- +title: Building TLE Catalogs +sidebar: + order: 12 +--- + +import { Steps, Aside, Tabs, TabItem, Code } from "@astrojs/starlight/components"; + +Every pg_orrery workflow starts with TLEs in a table. The [Tracking Satellites](/guides/tracking-satellites/) guide shows how to insert a few satellites by hand --- but a real catalog has tens of thousands of objects from multiple sources, each with different freshness and coverage. `pg-orrery-catalog` handles the download, merge, and load pipeline. + +## The problem with multiple TLE sources + +Three major sources provide TLE data, each with trade-offs: + +| Source | Auth | Coverage | Freshness | +|--------|------|----------|-----------| +| [Space-Track](https://www.space-track.org) | Login required | Full catalog (~30k+ on-orbit) | Hours to days | +| [CelesTrak](https://celestrak.org) | None | Active sats + operator supplemental GP | Minutes to hours | +| [SatNOGS](https://db.satnogs.org) | None | Community-tracked objects | Varies | + +The same satellite often appears in all three. CelesTrak's supplemental GP (SupGP) data is particularly valuable --- operators like SpaceX submit Starlink ephemerides that are often hours fresher than Space-Track's own catalog. + +The question is which entry to keep. `pg-orrery-catalog` answers with epoch-based deduplication: when the same NORAD ID appears in multiple sources, the entry with the newest epoch wins. This means SupGP data automatically overrides stale Space-Track entries where available. + +## Install + +```bash +# Run directly (no install needed) +uvx pg-orrery-catalog --help + +# Or install permanently +uv pip install pg-orrery-catalog + +# For direct database loading (adds psycopg) +uv pip install "pg-orrery-catalog[pg]" +``` + +## Download, build, load + +The typical workflow is three steps. Each can run independently. + + +1. **Download** TLE data from remote sources into the local cache: + + ```bash + pg-orrery-catalog download + ``` + + This fetches from all configured sources (CelesTrak by default, Space-Track if credentials are set). Files are cached in `~/.cache/pg-orrery-catalog/` and reused unless stale (>24h) or `--force` is passed. + + To download from a specific source: + + ```bash + pg-orrery-catalog download --source celestrak + pg-orrery-catalog download --source spacetrack --force + ``` + +2. **Build** a merged catalog and output it: + + + + ```bash + pg-orrery-catalog build | psql -d mydb + ``` + + + ```bash + pg-orrery-catalog build --table satellites -o catalog.sql + ``` + + + ```bash + pg-orrery-catalog build --format 3le -o merged.tle + ``` + + + ```bash + pg-orrery-catalog build --format json -o catalog.json + ``` + + + + With no arguments, `build` merges all cached files. You can also pass specific TLE files: + + ```bash + pg-orrery-catalog build /path/to/spacetrack.tle /path/to/celestrak.tle + ``` + + The merge reports what happened: + + ``` + spacetrack_everything: 33053 objects (33053 new, 0 updated) + celestrak_active: 14376 objects (2 new, 0 updated) + satnogs_full: 1488 objects (121 new, 5 updated) + supgp_starlink: 9703 objects (77 new, 7398 updated) + Total: 33253 unique objects + Regimes: LEO: 31542, GEO: 1203, MEO: 385, HEO: 123 + ``` + + Notice how SupGP updated 7,398 Starlink entries --- those are fresher epochs from SpaceX overriding stale Space-Track data. + +3. **Load** directly into PostgreSQL (requires `[pg]` extra): + + ```bash + pg-orrery-catalog load \ + --database-url postgresql:///mydb \ + --table satellites \ + --create-index + ``` + + The `--create-index` flag creates both GiST and SP-GiST indexes on the `tle` column, ready for spatial queries and KNN ordering. + + +## Configuration + +Three layers, highest precedence first: + +1. **CLI flags** --- `--table`, `--source`, `--database-url` +2. **Environment variables** --- `SPACETRACK_USER`, `SOCKS_PROXY`, `DATABASE_URL` +3. **Config file** --- `~/.config/pg-orrery-catalog/config.toml` + +### Space-Track credentials + +Space-Track requires a free account. Set credentials via environment variables: + +```bash +export SPACETRACK_USER="you@example.com" +export SPACETRACK_PASSWORD="secret" +pg-orrery-catalog download --source spacetrack +``` + +Or in the config file: + +```toml +[spacetrack] +user = "you@example.com" +password = "secret" +``` + +### SOCKS proxy + +CelesTrak is sometimes unreachable from certain networks. Route through a SOCKS5 proxy: + +```bash +export SOCKS_PROXY="localhost:1080" +pg-orrery-catalog download +``` + +### Full config reference + +```toml +[spacetrack] +user = "you@example.com" +password = "secret" + +[celestrak] +proxy = "localhost:1080" +supgp_groups = ["starlink", "oneweb", "planet", "orbcomm"] + +[output] +table = "satellites" + +[database] +url = "postgresql://localhost/mydb" +``` + +## Working with the generated SQL + +The SQL output creates a table with three columns: + +```sql +CREATE TABLE satellites ( + id serial, + name text, + tle tle +); +``` + +Once loaded, the full pg_orrery function set is available: + +```sql +-- Where is every LEO satellite right now? +SELECT name, observe('40.0N 105.3W 1655m'::observer, tle, now()) AS topo +FROM satellites +WHERE tle_mean_motion(tle) > 11.25; + +-- Which satellites are overhead right now? +SELECT name, topo_elevation(observe('40.0N 105.3W 1655m'::observer, tle, now())) AS el +FROM satellites +WHERE topo_elevation(observe('40.0N 105.3W 1655m'::observer, tle, now())) > 10 +ORDER BY el DESC; + +-- Predict ISS passes for the next 24 hours +SELECT pass_aos(p), pass_max_el(p), pass_los(p) +FROM satellites, + predict_passes(tle, '40.0N 105.3W 1655m'::observer, + now(), now() + interval '24 hours', 10.0) p +WHERE norad_id = 25544; +``` + +## NORAD ID encoding + +TLE files use a 5-character field for the NORAD catalog number. With more than 100,000 tracked objects, the original 5-digit numeric format ran out of space. The encoding has evolved through four cases: + +| Case | Format | Range | Example | +|------|--------|-------|---------| +| Traditional | `ddddd` | 0 -- 99,999 | `25544` (ISS) | +| Alpha-5 | `Ldddd` | 100,000 -- 339,999 | `T0002` = 270,002 | +| Super-5 case 3 | `xxxxX` | 340,000 -- 906,309,663 | `0000A` = 340,000 | +| Super-5 case 4 | `xxxXd` | 906,309,664+ | `000A0` = 906,309,664 | + +Alpha-5 skips the letters I and O (they look like 1 and 0). Super-5 uses a base-64 alphabet: digits 0--9, uppercase A--Z, lowercase a--z, plus `+` and `-`. + +`pg-orrery-catalog` decodes all four cases, matching the `get_norad_number()` implementation in pg_orrery's vendored SGP4 library. This means Alpha-5 objects like Starlink satellites (NORAD IDs above 100,000) load correctly. + + + +## Cache management + +Downloaded TLE files are stored under `~/.cache/pg-orrery-catalog/`, organized by source: + +``` +~/.cache/pg-orrery-catalog/ + celestrak/ + celestrak_active.tle + supgp_starlink.tle + supgp_oneweb.tle + ... + satnogs/ + satnogs_full.tle + spacetrack/ + spacetrack_everything.tle +``` + +Check what's cached: + +```bash +pg-orrery-catalog info --cache +``` + +Files older than 24 hours are considered stale and re-downloaded automatically. Use `--force` to override fresh cache entries. + +## Automating catalog updates + +For a regularly-updated catalog, a cron job or systemd timer works well: + +```bash +# Update catalog daily at 03:00 +0 3 * * * pg-orrery-catalog download && pg-orrery-catalog build --table satellites | psql -d mydb +``` + +Or with the direct load command: + +```bash +0 3 * * * pg-orrery-catalog download && pg-orrery-catalog load --database-url postgresql:///mydb --table satellites --create-index +``` + + + +## Using as a library + +`pg-orrery-catalog` can also be imported as a Python library: + +```python +from pg_orrery_catalog.tle import decode_norad, parse_3le_file +from pg_orrery_catalog.catalog import merge_sources +from pg_orrery_catalog.regime import regime_summary +from pg_orrery_catalog.output.sql import generate_sql + +# Parse and merge +merged, stats = merge_sources(["spacetrack.tle", "celestrak.tle"]) +print(f"{stats.total_unique} unique objects") + +# Classify +regimes = regime_summary(merged) +print(regimes) # {'LEO': 31542, 'MEO': 385, 'GEO': 1203, 'HEO': 123} + +# Generate SQL +sql = generate_sql(merged, table="my_catalog") +``` + +## What's next + +With a catalog loaded, see: + +- [Tracking Satellites](/guides/tracking-satellites/) --- observe, predict passes, screen conjunctions +- [Satellite Pass Prediction](/guides/pass-prediction/) --- detailed pass prediction workflows +- [Conjunction Screening](/guides/conjunction-screening/) --- find close approaches using GiST indexes +- [Benchmarks](/performance/benchmarks/) --- performance data with catalogs of 33k--66k objects diff --git a/docs/src/content/docs/guides/tracking-satellites.mdx b/docs/src/content/docs/guides/tracking-satellites.mdx index 0c4f20e..e9adc24 100644 --- a/docs/src/content/docs/guides/tracking-satellites.mdx +++ b/docs/src/content/docs/guides/tracking-satellites.mdx @@ -43,7 +43,7 @@ pg_orrery propagates TLEs and computes look angles. It does not replace the full - **No real-time GUI.** GPredict and STK provide map displays, polar plots, and Doppler displays. pg_orrery returns numbers. Use any visualization tool to render its output. - **No rotator control.** Hamlib drives antenna rotators. pg_orrery computes the azimuth and elevation values Hamlib would consume, but it has no hardware interface. -- **No TLE fetching.** Bring your own TLEs from Space-Track, CelesTrak, or any provider. pg_orrery parses and propagates them. +- **TLE fetching via companion tool.** pg_orrery itself doesn't download TLEs, but [`pg-orrery-catalog`](/guides/catalog-management/) handles the full pipeline: download from Space-Track, CelesTrak, and SatNOGS, merge with epoch-based dedup, and load into PostgreSQL. - **Orbit determination available.** Since v0.4.0, pg_orrery can fit TLEs from ECI, topocentric, or angles-only observations via differential correction. See the [Orbit Determination guide](/guides/orbit-determination/). - **No high-precision propagation.** SGP4/SDP4 accuracy degrades with TLE age. For operational conjunction assessment, use SP ephemerides or owner/operator-provided state vectors. pg_orrery's GiST screening finds candidates; you verify with better data. diff --git a/docs/src/content/docs/performance/benchmarks.mdx b/docs/src/content/docs/performance/benchmarks.mdx index 628d5d6..4f38d2a 100644 --- a/docs/src/content/docs/performance/benchmarks.mdx +++ b/docs/src/content/docs/performance/benchmarks.mdx @@ -284,7 +284,7 @@ For a 65,886-object catalog and a 2-hour window from Eagle, Idaho: - PostgreSQL 17 with pg_orrery installed - - A satellite catalog table with ~12,000 TLEs (available from CelesTrak) + - A satellite catalog table with ~12,000 TLEs (see [Building TLE Catalogs](/guides/catalog-management/) or download directly from CelesTrak) - A star catalog table (any subset of Hipparcos or Yale BSC) - No concurrent queries during measurement - `shared_buffers` and `work_mem` at default or higher @@ -293,9 +293,10 @@ For a 65,886-object catalog and a 2-hour window from Eagle, Idaho: ```sql CREATE EXTENSION pg_orrery; - -- Load a TLE catalog + -- Load a TLE catalog (pg-orrery-catalog handles this) + -- pg-orrery-catalog build --table satellite_catalog | psql -d mydb CREATE TABLE satellite_catalog (tle tle); - -- (COPY from CelesTrak bulk TLE file) + -- (or COPY from CelesTrak bulk TLE file) -- Verify catalog size SELECT count(*) FROM satellite_catalog;