Add Building TLE Catalogs guide, cross-reference pg-orrery-catalog
New guide: guides/catalog-management.mdx covering the full download/merge/load pipeline with pg-orrery-catalog CLI. Updated tracking-satellites.mdx to reference the companion tool instead of "No TLE fetching". Added cross-reference in benchmarks setup instructions.
This commit is contained in:
parent
747b7ae60a
commit
26ae80d340
@ -70,6 +70,7 @@ export default defineConfig({
|
|||||||
{ label: "JPL DE Ephemeris", slug: "guides/de-ephemeris" },
|
{ label: "JPL DE Ephemeris", slug: "guides/de-ephemeris" },
|
||||||
{ label: "Orbit Determination", slug: "guides/orbit-determination" },
|
{ label: "Orbit Determination", slug: "guides/orbit-determination" },
|
||||||
{ label: "Satellite Pass Prediction", slug: "guides/pass-prediction" },
|
{ label: "Satellite Pass Prediction", slug: "guides/pass-prediction" },
|
||||||
|
{ label: "Building TLE Catalogs", slug: "guides/catalog-management" },
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|||||||
302
docs/src/content/docs/guides/catalog-management.mdx
Normal file
302
docs/src/content/docs/guides/catalog-management.mdx
Normal file
@ -0,0 +1,302 @@
|
|||||||
|
---
|
||||||
|
title: Building TLE Catalogs
|
||||||
|
sidebar:
|
||||||
|
order: 12
|
||||||
|
---
|
||||||
|
|
||||||
|
import { Steps, Aside, Tabs, TabItem, Code } from "@astrojs/starlight/components";
|
||||||
|
|
||||||
|
Every pg_orrery workflow starts with TLEs in a table. The [Tracking Satellites](/guides/tracking-satellites/) guide shows how to insert a few satellites by hand --- but a real catalog has tens of thousands of objects from multiple sources, each with different freshness and coverage. `pg-orrery-catalog` handles the download, merge, and load pipeline.
|
||||||
|
|
||||||
|
## The problem with multiple TLE sources
|
||||||
|
|
||||||
|
Three major sources provide TLE data, each with trade-offs:
|
||||||
|
|
||||||
|
| Source | Auth | Coverage | Freshness |
|
||||||
|
|--------|------|----------|-----------|
|
||||||
|
| [Space-Track](https://www.space-track.org) | Login required | Full catalog (~30k+ on-orbit) | Hours to days |
|
||||||
|
| [CelesTrak](https://celestrak.org) | None | Active sats + operator supplemental GP | Minutes to hours |
|
||||||
|
| [SatNOGS](https://db.satnogs.org) | None | Community-tracked objects | Varies |
|
||||||
|
|
||||||
|
The same satellite often appears in all three. CelesTrak's supplemental GP (SupGP) data is particularly valuable --- operators like SpaceX submit Starlink ephemerides that are often hours fresher than Space-Track's own catalog.
|
||||||
|
|
||||||
|
The question is which entry to keep. `pg-orrery-catalog` answers with epoch-based deduplication: when the same NORAD ID appears in multiple sources, the entry with the newest epoch wins. This means SupGP data automatically overrides stale Space-Track entries where available.
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run directly (no install needed)
|
||||||
|
uvx pg-orrery-catalog --help
|
||||||
|
|
||||||
|
# Or install permanently
|
||||||
|
uv pip install pg-orrery-catalog
|
||||||
|
|
||||||
|
# For direct database loading (adds psycopg)
|
||||||
|
uv pip install "pg-orrery-catalog[pg]"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Download, build, load
|
||||||
|
|
||||||
|
The typical workflow is three steps. Each can run independently.
|
||||||
|
|
||||||
|
<Steps>
|
||||||
|
1. **Download** TLE data from remote sources into the local cache:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pg-orrery-catalog download
|
||||||
|
```
|
||||||
|
|
||||||
|
This fetches from all configured sources (CelesTrak by default, Space-Track if credentials are set). Files are cached in `~/.cache/pg-orrery-catalog/` and reused unless stale (>24h) or `--force` is passed.
|
||||||
|
|
||||||
|
To download from a specific source:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pg-orrery-catalog download --source celestrak
|
||||||
|
pg-orrery-catalog download --source spacetrack --force
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Build** a merged catalog and output it:
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
<TabItem label="Pipe to psql">
|
||||||
|
```bash
|
||||||
|
pg-orrery-catalog build | psql -d mydb
|
||||||
|
```
|
||||||
|
</TabItem>
|
||||||
|
<TabItem label="Save SQL file">
|
||||||
|
```bash
|
||||||
|
pg-orrery-catalog build --table satellites -o catalog.sql
|
||||||
|
```
|
||||||
|
</TabItem>
|
||||||
|
<TabItem label="Export 3LE">
|
||||||
|
```bash
|
||||||
|
pg-orrery-catalog build --format 3le -o merged.tle
|
||||||
|
```
|
||||||
|
</TabItem>
|
||||||
|
<TabItem label="JSON output">
|
||||||
|
```bash
|
||||||
|
pg-orrery-catalog build --format json -o catalog.json
|
||||||
|
```
|
||||||
|
</TabItem>
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
With no arguments, `build` merges all cached files. You can also pass specific TLE files:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pg-orrery-catalog build /path/to/spacetrack.tle /path/to/celestrak.tle
|
||||||
|
```
|
||||||
|
|
||||||
|
The merge reports what happened:
|
||||||
|
|
||||||
|
```
|
||||||
|
spacetrack_everything: 33053 objects (33053 new, 0 updated)
|
||||||
|
celestrak_active: 14376 objects (2 new, 0 updated)
|
||||||
|
satnogs_full: 1488 objects (121 new, 5 updated)
|
||||||
|
supgp_starlink: 9703 objects (77 new, 7398 updated)
|
||||||
|
Total: 33253 unique objects
|
||||||
|
Regimes: LEO: 31542, GEO: 1203, MEO: 385, HEO: 123
|
||||||
|
```
|
||||||
|
|
||||||
|
Notice how SupGP updated 7,398 Starlink entries --- those are fresher epochs from SpaceX overriding stale Space-Track data.
|
||||||
|
|
||||||
|
3. **Load** directly into PostgreSQL (requires `[pg]` extra):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pg-orrery-catalog load \
|
||||||
|
--database-url postgresql:///mydb \
|
||||||
|
--table satellites \
|
||||||
|
--create-index
|
||||||
|
```
|
||||||
|
|
||||||
|
The `--create-index` flag creates both GiST and SP-GiST indexes on the `tle` column, ready for spatial queries and KNN ordering.
|
||||||
|
</Steps>
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Three layers, highest precedence first:
|
||||||
|
|
||||||
|
1. **CLI flags** --- `--table`, `--source`, `--database-url`
|
||||||
|
2. **Environment variables** --- `SPACETRACK_USER`, `SOCKS_PROXY`, `DATABASE_URL`
|
||||||
|
3. **Config file** --- `~/.config/pg-orrery-catalog/config.toml`
|
||||||
|
|
||||||
|
### Space-Track credentials
|
||||||
|
|
||||||
|
Space-Track requires a free account. Set credentials via environment variables:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export SPACETRACK_USER="you@example.com"
|
||||||
|
export SPACETRACK_PASSWORD="secret"
|
||||||
|
pg-orrery-catalog download --source spacetrack
|
||||||
|
```
|
||||||
|
|
||||||
|
Or in the config file:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[spacetrack]
|
||||||
|
user = "you@example.com"
|
||||||
|
password = "secret"
|
||||||
|
```
|
||||||
|
|
||||||
|
### SOCKS proxy
|
||||||
|
|
||||||
|
CelesTrak is sometimes unreachable from certain networks. Route through a SOCKS5 proxy:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export SOCKS_PROXY="localhost:1080"
|
||||||
|
pg-orrery-catalog download
|
||||||
|
```
|
||||||
|
|
||||||
|
### Full config reference
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[spacetrack]
|
||||||
|
user = "you@example.com"
|
||||||
|
password = "secret"
|
||||||
|
|
||||||
|
[celestrak]
|
||||||
|
proxy = "localhost:1080"
|
||||||
|
supgp_groups = ["starlink", "oneweb", "planet", "orbcomm"]
|
||||||
|
|
||||||
|
[output]
|
||||||
|
table = "satellites"
|
||||||
|
|
||||||
|
[database]
|
||||||
|
url = "postgresql://localhost/mydb"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Working with the generated SQL
|
||||||
|
|
||||||
|
The SQL output creates a table with three columns:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE satellites (
|
||||||
|
id serial,
|
||||||
|
name text,
|
||||||
|
tle tle
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
Once loaded, the full pg_orrery function set is available:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Where is every LEO satellite right now?
|
||||||
|
SELECT name, observe('40.0N 105.3W 1655m'::observer, tle, now()) AS topo
|
||||||
|
FROM satellites
|
||||||
|
WHERE tle_mean_motion(tle) > 11.25;
|
||||||
|
|
||||||
|
-- Which satellites are overhead right now?
|
||||||
|
SELECT name, topo_elevation(observe('40.0N 105.3W 1655m'::observer, tle, now())) AS el
|
||||||
|
FROM satellites
|
||||||
|
WHERE topo_elevation(observe('40.0N 105.3W 1655m'::observer, tle, now())) > 10
|
||||||
|
ORDER BY el DESC;
|
||||||
|
|
||||||
|
-- Predict ISS passes for the next 24 hours
|
||||||
|
SELECT pass_aos(p), pass_max_el(p), pass_los(p)
|
||||||
|
FROM satellites,
|
||||||
|
predict_passes(tle, '40.0N 105.3W 1655m'::observer,
|
||||||
|
now(), now() + interval '24 hours', 10.0) p
|
||||||
|
WHERE norad_id = 25544;
|
||||||
|
```
|
||||||
|
|
||||||
|
## NORAD ID encoding
|
||||||
|
|
||||||
|
TLE files use a 5-character field for the NORAD catalog number. With more than 100,000 tracked objects, the original 5-digit numeric format ran out of space. The encoding has evolved through four cases:
|
||||||
|
|
||||||
|
| Case | Format | Range | Example |
|
||||||
|
|------|--------|-------|---------|
|
||||||
|
| Traditional | `ddddd` | 0 -- 99,999 | `25544` (ISS) |
|
||||||
|
| Alpha-5 | `Ldddd` | 100,000 -- 339,999 | `T0002` = 270,002 |
|
||||||
|
| Super-5 case 3 | `xxxxX` | 340,000 -- 906,309,663 | `0000A` = 340,000 |
|
||||||
|
| Super-5 case 4 | `xxxXd` | 906,309,664+ | `000A0` = 906,309,664 |
|
||||||
|
|
||||||
|
Alpha-5 skips the letters I and O (they look like 1 and 0). Super-5 uses a base-64 alphabet: digits 0--9, uppercase A--Z, lowercase a--z, plus `+` and `-`.
|
||||||
|
|
||||||
|
`pg-orrery-catalog` decodes all four cases, matching the `get_norad_number()` implementation in pg_orrery's vendored SGP4 library. This means Alpha-5 objects like Starlink satellites (NORAD IDs above 100,000) load correctly.
|
||||||
|
|
||||||
|
<Aside type="note" title="Alpha-5 verification">
|
||||||
|
You can verify the decoding independently:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -c "
|
||||||
|
from pg_orrery_catalog.tle import decode_norad
|
||||||
|
print(f'T0002 = {decode_norad(\"T0002\")}') # 270002
|
||||||
|
print(f'A0001 = {decode_norad(\"A0001\")}') # 100001
|
||||||
|
print(f'Z9999 = {decode_norad(\"Z9999\")}') # 339999
|
||||||
|
"
|
||||||
|
```
|
||||||
|
</Aside>
|
||||||
|
|
||||||
|
## Cache management
|
||||||
|
|
||||||
|
Downloaded TLE files are stored under `~/.cache/pg-orrery-catalog/`, organized by source:
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.cache/pg-orrery-catalog/
|
||||||
|
celestrak/
|
||||||
|
celestrak_active.tle
|
||||||
|
supgp_starlink.tle
|
||||||
|
supgp_oneweb.tle
|
||||||
|
...
|
||||||
|
satnogs/
|
||||||
|
satnogs_full.tle
|
||||||
|
spacetrack/
|
||||||
|
spacetrack_everything.tle
|
||||||
|
```
|
||||||
|
|
||||||
|
Check what's cached:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pg-orrery-catalog info --cache
|
||||||
|
```
|
||||||
|
|
||||||
|
Files older than 24 hours are considered stale and re-downloaded automatically. Use `--force` to override fresh cache entries.
|
||||||
|
|
||||||
|
## Automating catalog updates
|
||||||
|
|
||||||
|
For a regularly-updated catalog, a cron job or systemd timer works well:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Update catalog daily at 03:00
|
||||||
|
0 3 * * * pg-orrery-catalog download && pg-orrery-catalog build --table satellites | psql -d mydb
|
||||||
|
```
|
||||||
|
|
||||||
|
Or with the direct load command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
0 3 * * * pg-orrery-catalog download && pg-orrery-catalog load --database-url postgresql:///mydb --table satellites --create-index
|
||||||
|
```
|
||||||
|
|
||||||
|
<Aside type="caution" title="Table replacement">
|
||||||
|
The default SQL output includes `DROP TABLE IF EXISTS` before `CREATE TABLE`. This replaces the entire table on each load. If you need to preserve the table and upsert, export as JSON and handle the merge in your application logic.
|
||||||
|
</Aside>
|
||||||
|
|
||||||
|
## Using as a library
|
||||||
|
|
||||||
|
`pg-orrery-catalog` can also be imported as a Python library:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pg_orrery_catalog.tle import decode_norad, parse_3le_file
|
||||||
|
from pg_orrery_catalog.catalog import merge_sources
|
||||||
|
from pg_orrery_catalog.regime import regime_summary
|
||||||
|
from pg_orrery_catalog.output.sql import generate_sql
|
||||||
|
|
||||||
|
# Parse and merge
|
||||||
|
merged, stats = merge_sources(["spacetrack.tle", "celestrak.tle"])
|
||||||
|
print(f"{stats.total_unique} unique objects")
|
||||||
|
|
||||||
|
# Classify
|
||||||
|
regimes = regime_summary(merged)
|
||||||
|
print(regimes) # {'LEO': 31542, 'MEO': 385, 'GEO': 1203, 'HEO': 123}
|
||||||
|
|
||||||
|
# Generate SQL
|
||||||
|
sql = generate_sql(merged, table="my_catalog")
|
||||||
|
```
|
||||||
|
|
||||||
|
## What's next
|
||||||
|
|
||||||
|
With a catalog loaded, see:
|
||||||
|
|
||||||
|
- [Tracking Satellites](/guides/tracking-satellites/) --- observe, predict passes, screen conjunctions
|
||||||
|
- [Satellite Pass Prediction](/guides/pass-prediction/) --- detailed pass prediction workflows
|
||||||
|
- [Conjunction Screening](/guides/conjunction-screening/) --- find close approaches using GiST indexes
|
||||||
|
- [Benchmarks](/performance/benchmarks/) --- performance data with catalogs of 33k--66k objects
|
||||||
@ -43,7 +43,7 @@ pg_orrery propagates TLEs and computes look angles. It does not replace the full
|
|||||||
|
|
||||||
- **No real-time GUI.** GPredict and STK provide map displays, polar plots, and Doppler displays. pg_orrery returns numbers. Use any visualization tool to render its output.
|
- **No real-time GUI.** GPredict and STK provide map displays, polar plots, and Doppler displays. pg_orrery returns numbers. Use any visualization tool to render its output.
|
||||||
- **No rotator control.** Hamlib drives antenna rotators. pg_orrery computes the azimuth and elevation values Hamlib would consume, but it has no hardware interface.
|
- **No rotator control.** Hamlib drives antenna rotators. pg_orrery computes the azimuth and elevation values Hamlib would consume, but it has no hardware interface.
|
||||||
- **No TLE fetching.** Bring your own TLEs from Space-Track, CelesTrak, or any provider. pg_orrery parses and propagates them.
|
- **TLE fetching via companion tool.** pg_orrery itself doesn't download TLEs, but [`pg-orrery-catalog`](/guides/catalog-management/) handles the full pipeline: download from Space-Track, CelesTrak, and SatNOGS, merge with epoch-based dedup, and load into PostgreSQL.
|
||||||
- **Orbit determination available.** Since v0.4.0, pg_orrery can fit TLEs from ECI, topocentric, or angles-only observations via differential correction. See the [Orbit Determination guide](/guides/orbit-determination/).
|
- **Orbit determination available.** Since v0.4.0, pg_orrery can fit TLEs from ECI, topocentric, or angles-only observations via differential correction. See the [Orbit Determination guide](/guides/orbit-determination/).
|
||||||
- **No high-precision propagation.** SGP4/SDP4 accuracy degrades with TLE age. For operational conjunction assessment, use SP ephemerides or owner/operator-provided state vectors. pg_orrery's GiST screening finds candidates; you verify with better data.
|
- **No high-precision propagation.** SGP4/SDP4 accuracy degrades with TLE age. For operational conjunction assessment, use SP ephemerides or owner/operator-provided state vectors. pg_orrery's GiST screening finds candidates; you verify with better data.
|
||||||
|
|
||||||
|
|||||||
@ -284,7 +284,7 @@ For a 65,886-object catalog and a 2-hour window from Eagle, Idaho:
|
|||||||
<Tabs>
|
<Tabs>
|
||||||
<TabItem label="Requirements">
|
<TabItem label="Requirements">
|
||||||
- PostgreSQL 17 with pg_orrery installed
|
- PostgreSQL 17 with pg_orrery installed
|
||||||
- A satellite catalog table with ~12,000 TLEs (available from CelesTrak)
|
- A satellite catalog table with ~12,000 TLEs (see [Building TLE Catalogs](/guides/catalog-management/) or download directly from CelesTrak)
|
||||||
- A star catalog table (any subset of Hipparcos or Yale BSC)
|
- A star catalog table (any subset of Hipparcos or Yale BSC)
|
||||||
- No concurrent queries during measurement
|
- No concurrent queries during measurement
|
||||||
- `shared_buffers` and `work_mem` at default or higher
|
- `shared_buffers` and `work_mem` at default or higher
|
||||||
@ -293,9 +293,10 @@ For a 65,886-object catalog and a 2-hour window from Eagle, Idaho:
|
|||||||
```sql
|
```sql
|
||||||
CREATE EXTENSION pg_orrery;
|
CREATE EXTENSION pg_orrery;
|
||||||
|
|
||||||
-- Load a TLE catalog
|
-- Load a TLE catalog (pg-orrery-catalog handles this)
|
||||||
|
-- pg-orrery-catalog build --table satellite_catalog | psql -d mydb
|
||||||
CREATE TABLE satellite_catalog (tle tle);
|
CREATE TABLE satellite_catalog (tle tle);
|
||||||
-- (COPY from CelesTrak bulk TLE file)
|
-- (or COPY from CelesTrak bulk TLE file)
|
||||||
|
|
||||||
-- Verify catalog size
|
-- Verify catalog size
|
||||||
SELECT count(*) FROM satellite_catalog;
|
SELECT count(*) FROM satellite_catalog;
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user