Add Building TLE Catalogs guide, cross-reference pg-orrery-catalog
New guide: guides/catalog-management.mdx covering the full download/merge/load pipeline with pg-orrery-catalog CLI. Updated tracking-satellites.mdx to reference the companion tool instead of "No TLE fetching". Added cross-reference in benchmarks setup instructions.
This commit is contained in:
parent
747b7ae60a
commit
26ae80d340
@ -70,6 +70,7 @@ export default defineConfig({
|
||||
{ label: "JPL DE Ephemeris", slug: "guides/de-ephemeris" },
|
||||
{ label: "Orbit Determination", slug: "guides/orbit-determination" },
|
||||
{ label: "Satellite Pass Prediction", slug: "guides/pass-prediction" },
|
||||
{ label: "Building TLE Catalogs", slug: "guides/catalog-management" },
|
||||
],
|
||||
},
|
||||
{
|
||||
|
||||
302
docs/src/content/docs/guides/catalog-management.mdx
Normal file
302
docs/src/content/docs/guides/catalog-management.mdx
Normal file
@ -0,0 +1,302 @@
|
||||
---
|
||||
title: Building TLE Catalogs
|
||||
sidebar:
|
||||
order: 12
|
||||
---
|
||||
|
||||
import { Steps, Aside, Tabs, TabItem, Code } from "@astrojs/starlight/components";
|
||||
|
||||
Every pg_orrery workflow starts with TLEs in a table. The [Tracking Satellites](/guides/tracking-satellites/) guide shows how to insert a few satellites by hand --- but a real catalog has tens of thousands of objects from multiple sources, each with different freshness and coverage. `pg-orrery-catalog` handles the download, merge, and load pipeline.
|
||||
|
||||
## The problem with multiple TLE sources
|
||||
|
||||
Three major sources provide TLE data, each with trade-offs:
|
||||
|
||||
| Source | Auth | Coverage | Freshness |
|
||||
|--------|------|----------|-----------|
|
||||
| [Space-Track](https://www.space-track.org) | Login required | Full catalog (~30k+ on-orbit) | Hours to days |
|
||||
| [CelesTrak](https://celestrak.org) | None | Active sats + operator supplemental GP | Minutes to hours |
|
||||
| [SatNOGS](https://db.satnogs.org) | None | Community-tracked objects | Varies |
|
||||
|
||||
The same satellite often appears in all three. CelesTrak's supplemental GP (SupGP) data is particularly valuable --- operators like SpaceX submit Starlink ephemerides that are often hours fresher than Space-Track's own catalog.
|
||||
|
||||
The question is which entry to keep. `pg-orrery-catalog` answers with epoch-based deduplication: when the same NORAD ID appears in multiple sources, the entry with the newest epoch wins. This means SupGP data automatically overrides stale Space-Track entries where available.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# Run directly (no install needed)
|
||||
uvx pg-orrery-catalog --help
|
||||
|
||||
# Or install permanently
|
||||
uv pip install pg-orrery-catalog
|
||||
|
||||
# For direct database loading (adds psycopg)
|
||||
uv pip install "pg-orrery-catalog[pg]"
|
||||
```
|
||||
|
||||
## Download, build, load
|
||||
|
||||
The typical workflow is three steps. Each can run independently.
|
||||
|
||||
<Steps>
|
||||
1. **Download** TLE data from remote sources into the local cache:
|
||||
|
||||
```bash
|
||||
pg-orrery-catalog download
|
||||
```
|
||||
|
||||
This fetches from all configured sources (CelesTrak by default, Space-Track if credentials are set). Files are cached in `~/.cache/pg-orrery-catalog/` and reused unless stale (>24h) or `--force` is passed.
|
||||
|
||||
To download from a specific source:
|
||||
|
||||
```bash
|
||||
pg-orrery-catalog download --source celestrak
|
||||
pg-orrery-catalog download --source spacetrack --force
|
||||
```
|
||||
|
||||
2. **Build** a merged catalog and output it:
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="Pipe to psql">
|
||||
```bash
|
||||
pg-orrery-catalog build | psql -d mydb
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem label="Save SQL file">
|
||||
```bash
|
||||
pg-orrery-catalog build --table satellites -o catalog.sql
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem label="Export 3LE">
|
||||
```bash
|
||||
pg-orrery-catalog build --format 3le -o merged.tle
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem label="JSON output">
|
||||
```bash
|
||||
pg-orrery-catalog build --format json -o catalog.json
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
With no arguments, `build` merges all cached files. You can also pass specific TLE files:
|
||||
|
||||
```bash
|
||||
pg-orrery-catalog build /path/to/spacetrack.tle /path/to/celestrak.tle
|
||||
```
|
||||
|
||||
The merge reports what happened:
|
||||
|
||||
```
|
||||
spacetrack_everything: 33053 objects (33053 new, 0 updated)
|
||||
celestrak_active: 14376 objects (2 new, 0 updated)
|
||||
satnogs_full: 1488 objects (121 new, 5 updated)
|
||||
supgp_starlink: 9703 objects (77 new, 7398 updated)
|
||||
Total: 33253 unique objects
|
||||
Regimes: LEO: 31542, GEO: 1203, MEO: 385, HEO: 123
|
||||
```
|
||||
|
||||
Notice how SupGP updated 7,398 Starlink entries --- those are fresher epochs from SpaceX overriding stale Space-Track data.
|
||||
|
||||
3. **Load** directly into PostgreSQL (requires `[pg]` extra):
|
||||
|
||||
```bash
|
||||
pg-orrery-catalog load \
|
||||
--database-url postgresql:///mydb \
|
||||
--table satellites \
|
||||
--create-index
|
||||
```
|
||||
|
||||
The `--create-index` flag creates both GiST and SP-GiST indexes on the `tle` column, ready for spatial queries and KNN ordering.
|
||||
</Steps>
|
||||
|
||||
## Configuration
|
||||
|
||||
Three layers, highest precedence first:
|
||||
|
||||
1. **CLI flags** --- `--table`, `--source`, `--database-url`
|
||||
2. **Environment variables** --- `SPACETRACK_USER`, `SOCKS_PROXY`, `DATABASE_URL`
|
||||
3. **Config file** --- `~/.config/pg-orrery-catalog/config.toml`
|
||||
|
||||
### Space-Track credentials
|
||||
|
||||
Space-Track requires a free account. Set credentials via environment variables:
|
||||
|
||||
```bash
|
||||
export SPACETRACK_USER="you@example.com"
|
||||
export SPACETRACK_PASSWORD="secret"
|
||||
pg-orrery-catalog download --source spacetrack
|
||||
```
|
||||
|
||||
Or in the config file:
|
||||
|
||||
```toml
|
||||
[spacetrack]
|
||||
user = "you@example.com"
|
||||
password = "secret"
|
||||
```
|
||||
|
||||
### SOCKS proxy
|
||||
|
||||
CelesTrak is sometimes unreachable from certain networks. Route through a SOCKS5 proxy:
|
||||
|
||||
```bash
|
||||
export SOCKS_PROXY="localhost:1080"
|
||||
pg-orrery-catalog download
|
||||
```
|
||||
|
||||
### Full config reference
|
||||
|
||||
```toml
|
||||
[spacetrack]
|
||||
user = "you@example.com"
|
||||
password = "secret"
|
||||
|
||||
[celestrak]
|
||||
proxy = "localhost:1080"
|
||||
supgp_groups = ["starlink", "oneweb", "planet", "orbcomm"]
|
||||
|
||||
[output]
|
||||
table = "satellites"
|
||||
|
||||
[database]
|
||||
url = "postgresql://localhost/mydb"
|
||||
```
|
||||
|
||||
## Working with the generated SQL
|
||||
|
||||
The SQL output creates a table with three columns:
|
||||
|
||||
```sql
|
||||
CREATE TABLE satellites (
|
||||
id serial,
|
||||
name text,
|
||||
tle tle
|
||||
);
|
||||
```
|
||||
|
||||
Once loaded, the full pg_orrery function set is available:
|
||||
|
||||
```sql
|
||||
-- Where is every LEO satellite right now?
|
||||
SELECT name, observe('40.0N 105.3W 1655m'::observer, tle, now()) AS topo
|
||||
FROM satellites
|
||||
WHERE tle_mean_motion(tle) > 11.25;
|
||||
|
||||
-- Which satellites are overhead right now?
|
||||
SELECT name, topo_elevation(observe('40.0N 105.3W 1655m'::observer, tle, now())) AS el
|
||||
FROM satellites
|
||||
WHERE topo_elevation(observe('40.0N 105.3W 1655m'::observer, tle, now())) > 10
|
||||
ORDER BY el DESC;
|
||||
|
||||
-- Predict ISS passes for the next 24 hours
|
||||
SELECT pass_aos(p), pass_max_el(p), pass_los(p)
|
||||
FROM satellites,
|
||||
predict_passes(tle, '40.0N 105.3W 1655m'::observer,
|
||||
now(), now() + interval '24 hours', 10.0) p
|
||||
WHERE norad_id = 25544;
|
||||
```
|
||||
|
||||
## NORAD ID encoding
|
||||
|
||||
TLE files use a 5-character field for the NORAD catalog number. With more than 100,000 tracked objects, the original 5-digit numeric format ran out of space. The encoding has evolved through four cases:
|
||||
|
||||
| Case | Format | Range | Example |
|
||||
|------|--------|-------|---------|
|
||||
| Traditional | `ddddd` | 0 -- 99,999 | `25544` (ISS) |
|
||||
| Alpha-5 | `Ldddd` | 100,000 -- 339,999 | `T0002` = 270,002 |
|
||||
| Super-5 case 3 | `xxxxX` | 340,000 -- 906,309,663 | `0000A` = 340,000 |
|
||||
| Super-5 case 4 | `xxxXd` | 906,309,664+ | `000A0` = 906,309,664 |
|
||||
|
||||
Alpha-5 skips the letters I and O (they look like 1 and 0). Super-5 uses a base-64 alphabet: digits 0--9, uppercase A--Z, lowercase a--z, plus `+` and `-`.
|
||||
|
||||
`pg-orrery-catalog` decodes all four cases, matching the `get_norad_number()` implementation in pg_orrery's vendored SGP4 library. This means Alpha-5 objects like Starlink satellites (NORAD IDs above 100,000) load correctly.
|
||||
|
||||
<Aside type="note" title="Alpha-5 verification">
|
||||
You can verify the decoding independently:
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
from pg_orrery_catalog.tle import decode_norad
|
||||
print(f'T0002 = {decode_norad(\"T0002\")}') # 270002
|
||||
print(f'A0001 = {decode_norad(\"A0001\")}') # 100001
|
||||
print(f'Z9999 = {decode_norad(\"Z9999\")}') # 339999
|
||||
"
|
||||
```
|
||||
</Aside>
|
||||
|
||||
## Cache management
|
||||
|
||||
Downloaded TLE files are stored under `~/.cache/pg-orrery-catalog/`, organized by source:
|
||||
|
||||
```
|
||||
~/.cache/pg-orrery-catalog/
|
||||
celestrak/
|
||||
celestrak_active.tle
|
||||
supgp_starlink.tle
|
||||
supgp_oneweb.tle
|
||||
...
|
||||
satnogs/
|
||||
satnogs_full.tle
|
||||
spacetrack/
|
||||
spacetrack_everything.tle
|
||||
```
|
||||
|
||||
Check what's cached:
|
||||
|
||||
```bash
|
||||
pg-orrery-catalog info --cache
|
||||
```
|
||||
|
||||
Files older than 24 hours are considered stale and re-downloaded automatically. Use `--force` to override fresh cache entries.
|
||||
|
||||
## Automating catalog updates
|
||||
|
||||
For a regularly-updated catalog, a cron job or systemd timer works well:
|
||||
|
||||
```bash
|
||||
# Update catalog daily at 03:00
|
||||
0 3 * * * pg-orrery-catalog download && pg-orrery-catalog build --table satellites | psql -d mydb
|
||||
```
|
||||
|
||||
Or with the direct load command:
|
||||
|
||||
```bash
|
||||
0 3 * * * pg-orrery-catalog download && pg-orrery-catalog load --database-url postgresql:///mydb --table satellites --create-index
|
||||
```
|
||||
|
||||
<Aside type="caution" title="Table replacement">
|
||||
The default SQL output includes `DROP TABLE IF EXISTS` before `CREATE TABLE`. This replaces the entire table on each load. If you need to preserve the table and upsert, export as JSON and handle the merge in your application logic.
|
||||
</Aside>
|
||||
|
||||
## Using as a library
|
||||
|
||||
`pg-orrery-catalog` can also be imported as a Python library:
|
||||
|
||||
```python
|
||||
from pg_orrery_catalog.tle import decode_norad, parse_3le_file
|
||||
from pg_orrery_catalog.catalog import merge_sources
|
||||
from pg_orrery_catalog.regime import regime_summary
|
||||
from pg_orrery_catalog.output.sql import generate_sql
|
||||
|
||||
# Parse and merge
|
||||
merged, stats = merge_sources(["spacetrack.tle", "celestrak.tle"])
|
||||
print(f"{stats.total_unique} unique objects")
|
||||
|
||||
# Classify
|
||||
regimes = regime_summary(merged)
|
||||
print(regimes) # {'LEO': 31542, 'MEO': 385, 'GEO': 1203, 'HEO': 123}
|
||||
|
||||
# Generate SQL
|
||||
sql = generate_sql(merged, table="my_catalog")
|
||||
```
|
||||
|
||||
## What's next
|
||||
|
||||
With a catalog loaded, see:
|
||||
|
||||
- [Tracking Satellites](/guides/tracking-satellites/) --- observe, predict passes, screen conjunctions
|
||||
- [Satellite Pass Prediction](/guides/pass-prediction/) --- detailed pass prediction workflows
|
||||
- [Conjunction Screening](/guides/conjunction-screening/) --- find close approaches using GiST indexes
|
||||
- [Benchmarks](/performance/benchmarks/) --- performance data with catalogs of 33k--66k objects
|
||||
@ -43,7 +43,7 @@ pg_orrery propagates TLEs and computes look angles. It does not replace the full
|
||||
|
||||
- **No real-time GUI.** GPredict and STK provide map displays, polar plots, and Doppler displays. pg_orrery returns numbers. Use any visualization tool to render its output.
|
||||
- **No rotator control.** Hamlib drives antenna rotators. pg_orrery computes the azimuth and elevation values Hamlib would consume, but it has no hardware interface.
|
||||
- **No TLE fetching.** Bring your own TLEs from Space-Track, CelesTrak, or any provider. pg_orrery parses and propagates them.
|
||||
- **TLE fetching via companion tool.** pg_orrery itself doesn't download TLEs, but [`pg-orrery-catalog`](/guides/catalog-management/) handles the full pipeline: download from Space-Track, CelesTrak, and SatNOGS, merge with epoch-based dedup, and load into PostgreSQL.
|
||||
- **Orbit determination available.** Since v0.4.0, pg_orrery can fit TLEs from ECI, topocentric, or angles-only observations via differential correction. See the [Orbit Determination guide](/guides/orbit-determination/).
|
||||
- **No high-precision propagation.** SGP4/SDP4 accuracy degrades with TLE age. For operational conjunction assessment, use SP ephemerides or owner/operator-provided state vectors. pg_orrery's GiST screening finds candidates; you verify with better data.
|
||||
|
||||
|
||||
@ -284,7 +284,7 @@ For a 65,886-object catalog and a 2-hour window from Eagle, Idaho:
|
||||
<Tabs>
|
||||
<TabItem label="Requirements">
|
||||
- PostgreSQL 17 with pg_orrery installed
|
||||
- A satellite catalog table with ~12,000 TLEs (available from CelesTrak)
|
||||
- A satellite catalog table with ~12,000 TLEs (see [Building TLE Catalogs](/guides/catalog-management/) or download directly from CelesTrak)
|
||||
- A star catalog table (any subset of Hipparcos or Yale BSC)
|
||||
- No concurrent queries during measurement
|
||||
- `shared_buffers` and `work_mem` at default or higher
|
||||
@ -293,9 +293,10 @@ For a 65,886-object catalog and a 2-hour window from Eagle, Idaho:
|
||||
```sql
|
||||
CREATE EXTENSION pg_orrery;
|
||||
|
||||
-- Load a TLE catalog
|
||||
-- Load a TLE catalog (pg-orrery-catalog handles this)
|
||||
-- pg-orrery-catalog build --table satellite_catalog | psql -d mydb
|
||||
CREATE TABLE satellite_catalog (tle tle);
|
||||
-- (COPY from CelesTrak bulk TLE file)
|
||||
-- (or COPY from CelesTrak bulk TLE file)
|
||||
|
||||
-- Verify catalog size
|
||||
SELECT count(*) FROM satellite_catalog;
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user