CSS MPC Toolkit

Value-added data products derived from the MPC/SBN PostgreSQL database, developed by the Catalina Sky Survey at the University of Arizona.

Overview

The Minor Planet Center (MPC) publishes asteroid observations and orbital elements through the Planetary Data System (PDS) Small Bodies Node (SBN) as a PostgreSQL database. Catalina Sky Survey maintains a local replica of this database. This project provides SQL scripts, Python libraries, and derived data products useful to the NEO community.

Interactive Application

NEO Discovery Statistics Explorer

App: app/discovery_stats.py

Interactive Dash web application for exploring NEO discovery statistics, survey performance, and follow-up timing. Six tabbed pages:

Tab	Content
MPEC Browser	Live feed of recent Minor Planet Electronic Circulars with parsed observations, orbital elements, residuals, and observer credits. Auto-selects latest discovery MPEC. Enrichment from JPL SBDB/Sentry, NEOfixer (Find_Orb), and ESA NEOCC with per-object risk assessment. Site-specific short-term observability chart via NEOfixer ephemeris API.
Discoveries by Year	Stacked bar chart by year/survey/size class with annual or cumulative views; size distribution histogram; top-15 stations table
Size Distribution vs. NEOMOD3	Half-magnitude bin chart comparing MPC discoveries to the NEOMOD3 population model (Nesvorny et al. 2024, Icarus 411); completeness curve with 1-sigma errors; reference table
Multi-survey Comparison	Venn diagrams (1–3 surveys) showing co-detection during discovery apparitions (±200 days); pairwise co-detection heatmap; survey reach
Follow-up Timing	Response curves showing how quickly other surveys observe newly-discovered NEOs; per-survey response time distributions; follow-up network heatmap; trend by year
Discovery Circumstances	Sky map (RA/Dec) of discovery positions with ecliptic and galactic plane overlays; apparent V magnitude histogram; rate of motion vs. H scatter; position angle rose diagram

Each tab has a Download CSV button that exports the currently filtered data. Shared banner controls: survey grouping, plot height, light/dark theme, service health indicators, reset buttons. Data sourced from two SQL queries cached to CSV with 1-day auto-invalidation.

Survey groupings: Catalina Survey (703, G96, E12), Catalina Follow-up (I52, V06, G84), Pan-STARRS (F51, F52), ATLAS (T05, T07, T08, T03, M22, W68, R17), Bok NEO Survey (V00), Rubin/LSST (X05), LINEAR, NEAT, Spacewatch, LONEOS, NEOWISE, and Others.

source venv/bin/activate
python app/discovery_stats.py          # http://127.0.0.1:8050/
python app/discovery_stats.py --refresh  # force DB re-query

Requires dash, plotly, pandas, numpy, psycopg2 (all in requirements.txt).

Available Data Products

NEO Discovery Tracklets

Script: sql/discovery_tracklets.sql

Computes discovery circumstances for all Near-Earth Objects (NEOs) identified from the MPC's orbital element database. NEO selection uses orbital criteria (q < 1.32 AU or orbit_type_int IN (0, 1, 2, 3, 20)), which includes near-Earth comets.

Output columns (12):

Column	Description
primary_designation	Asteroid number (if numbered) or provisional designation
packed_primary_provisional_designation	MPC packed format designation
avg_mjd_discovery_tracklet	Mean MJD epoch of the discovery tracklet
avg_ra_deg	Mean Right Ascension (decimal degrees)
avg_dec_deg	Mean Declination (decimal degrees)
median_v_magnitude	Median V-band magnitude (with band corrections)
nobs	Number of observations in discovery tracklet
span_hours	Time span of discovery tracklet (hours)
rate_deg_per_day	Great-circle rate of motion (deg/day)
position_angle_deg	Position angle of motion (0=N, 90=E)
discovery_site_code	MPC observatory code of the discovery site
discovery_site_name	Observatory name

Completeness: 43,629 NEO discovery tracklets (as of 2026-02-08). Known missing: 2020 GZ1 (no orbit in mpc_orbits), 2009 US19 and 2024 TZ7 (no discovery observation flag in obs_sbn).

The output CSV is consumed by NEOlyzer and is available as a release asset for download.

ADES Export (NEOCP Observations)

Scripts: lib/ades_export.py, sql/ades_export.sql

Exports NEOCP (NEO Confirmation Page) archived observations in ADES format (XML or PSV), conforming to general.xsd (version 2022). The NEOCP tables carry ADES-native uncertainty fields (rmsRA, rmsDec, rmsCorr, rmsTime) not available in the main obs_sbn table.

Usage:

# All current NEOCP observations
python3 -m lib.ades_export --host $PGHOST --format xml --all -o neocp_live.xml

# Single designation from live NEOCP
python3 -m lib.ades_export --host $PGHOST --format psv --desig CE5W292 -o output.psv

# Historical lookup from archive
python3 -m lib.ades_export --host $PGHOST --archive --desig "2024 YR4" -o yr4.xml

Requires psycopg2: pip install psycopg2-binary

Conversion Library

Module: lib/mpc_convert.py

Reusable conversion functions for MPC 80-column observation format:

Function	Input	Output
mpc_date_to_iso8601()	"2024 12 27.238073"	"2024-12-27T05:42:49.5Z"
ra_hms_to_deg()	"08 56 40.968"	134.1707
dec_dms_to_deg()	"-00 16 11.93"	-0.2700
mpc_cat_to_ades()	"V"	"Gaia2"
mpc_mode_to_ades()	"C"	"CCD"
unpack_designation()	"K24Y04R"	"2024 YR4"
pack_designation()	"2024 YR4"	"K24Y04R"
parse_obs80()	80-col line	ADES field dict

PostgreSQL equivalents are in sql/css_utilities_functions.sql, designed for a css_utilities schema on the local replica.

Database Operations

Health Check

Script: scripts/db_health_check.sh

Diagnostic toolkit covering replication status, table health (dead tuples, vacuum/analyze staleness), index usage, configuration review, and active connections.

bash scripts/db_health_check.sh --host $PGHOST
bash scripts/db_health_check.sh --host $PGHOST --output health_$(date +%Y%m%d).txt

Tuning Recommendations

File: scripts/db_tune_recommendations.sql

PostgreSQL configuration recommendations for a large MPC/SBN replica (251 GB RAM, HDD). Covers shared_buffers, work_mem, maintenance_work_mem, autovacuum thresholds, and per-table overrides for obs_sbn.

Huge Pages

File: scripts/enable_huge_pages.md

Step-by-step guide for enabling huge pages on RHEL 8 for PostgreSQL with 64 GB shared_buffers. Deferred to next maintenance reboot due to memory fragmentation.

Project Structure

CSS_MPC_toolkit/
├── README.md                           # This file
├── CLAUDE.md                           # Claude Code project guide
├── app/                                # Interactive Dash web application
│   ├── discovery_stats.py              #   NEO discovery explorer (6 tabs)
│   └── assets/                         #   CSS, logo, static files
├── lib/                                # Python library layer
│   ├── db.py                           #   DB connections, timed queries
│   ├── orbits.py                       #   Query builders for mpc_orbits
│   ├── orbit_classes.py                #   Orbit classification, Tisserand
│   ├── mpc_convert.py                  #   MPC format conversion functions
│   ├── mpec_parser.py                  #   MPEC fetch, parse, classify
│   ├── api_clients.py                  #   JPL/NEOfixer/NEOCC API wrappers
│   ├── ades_export.py                  #   ADES XML/PSV export
│   └── ades_validate.py                #   XSD validation
├── sql/
│   ├── discovery_tracklets.sql         #   NEO discovery tracklet statistics
│   ├── css_utilities_functions.sql     #   PostgreSQL conversion functions
│   ├── ades_export.sql                 #   ADES-ready columns from NEOCP
│   ├── viz/                            #   Reference queries for visualization
│   ├── common/
│   │   └── indexes.sql                 #   Shared index definitions
│   └── debug/
│       └── discovery_tracklets_diag.sql
├── scripts/
│   ├── run_pipeline.sh                 #   Run SQL, validate, upload
│   ├── validate_output.sh              #   Output validation checks
│   ├── upload_release.sh               #   Upload CSV to GitHub Releases
│   ├── db_health_check.sh              #   Database diagnostic toolkit
│   ├── daily_refresh.sh                #   Cron script for daily cache refresh
│   ├── db_tune_recommendations.sql     #   PostgreSQL tuning guide
│   └── enable_huge_pages.md            #   Huge pages setup guide (RHEL 8)
├── notebooks/                          #   Jupyter Lab exploration
│   ├── 01_query_profiling.ipynb
│   ├── 02_orbital_elements.ipynb
│   └── 03_data_quality.ipynb
├── schema/                             #   ADES format XSD schemas
│   ├── general.xsd
│   └── submit.xsd
├── sandbox/                            #   Analysis notes, exploratory outputs
└── docs/
    ├── deployment.md                   #   Server provisioning and operations guide
    ├── source_tables.md                #   Required MPC/SBN tables and columns
    └── band_corrections.md             #   Photometric band-to-V corrections

Quick Start

Prerequisites

• PostgreSQL client (psql)
• Access to an MPC/SBN database replica (set $PGHOST or pass --host)
• Python 3.10+ with venv/ (pinned in requirements.txt)
• Key packages: dash, plotly, pandas, numpy, psycopg2
• gh CLI for uploading release assets (optional)

Run the Discovery Tracklets Pipeline

# Run the query directly
psql -h $PGHOST -d mpc_sbn -f sql/discovery_tracklets.sql \
    --csv -o NEO_discovery_tracklets.csv

# Or use the automated pipeline
./scripts/run_pipeline.sh

Install SQL Functions on the Replica

# Requires CREATE SCHEMA / CREATE FUNCTION privileges
psql -h $PGHOST -U <owner> mpc_sbn -f sql/css_utilities_functions.sql

Run the Health Check

bash scripts/db_health_check.sh --host $PGHOST

Distribution

Data products are distributed in two ways:

1. CSV files via GitHub Releases -- download the latest release asset for flat-file access
2. SQL scripts in this repository -- run directly against your own MPC/SBN database replica to generate the data locally

Database Requirements

These scripts are designed to run against a PostgreSQL replica of the MPC/SBN database. See docs/source_tables.md for the required tables and columns. The database is a PostgreSQL 15.2 replica receiving logical replication from MPC.

License

MIT License -- Copyright (c) 2026 University of Arizona, Catalina Sky Survey

Contact

Rob Seaman Catalina Sky Survey, Lunar and Planetary Laboratory, University of Arizona rseaman@arizona.edu