Add Standard Core Stack GEE Ingestion Workflow

utilities/scripts/gee/core_stack_gee_assets.yaml

@@ -0,0 +1,122 @@
+schema_version: 1
+description: >
+  Config-driven Core Stack Earth Engine ingestion for standard GeoPackage and
+  GeoJSON vector assets. Local exports are tab-delimited tables with one
+  GeoJSON geometry column, then staged to GCS with a .csv suffix for Earth
+  Engine table ingestion.
+
+service_account_json: data/gee_confs/core-stack-learn-818963fa8f26.json
+
+outputs:
+  directory: data/gee/core_stack
+  local_extension: .tsv
+  build_summary_yaml: data/gee/core_stack/core_stack_gee_build_summary.yaml
+  upload_summary_yaml: data/gee/core_stack/core_stack_gee_upload_summary.yaml
+  status_summary_yaml: data/gee/core_stack/core_stack_gee_status.yaml
+  public_summary_yaml: data/gee/core_stack/core_stack_gee_public_summary.yaml
+  verify_summary_yaml: data/gee/core_stack/core_stack_gee_verify_summary.yaml
+  manifest_dir: data/gee/core_stack/manifests
+  log_file: logs/core_stack_gee_ingest.log
+
+upload:
+  gcs_bucket: core_stack
+  gcs_prefix: gee/core_stack_assets
+  gcs_object_extension: .csv
+  csv_delimiter: "\t"
+  csv_qualifier: '"'
+  crs: EPSG:4326
+  geodesic: false
+  max_vertices: 0
+  max_error_meters: 1.0
+  chunk_size_mb: 64
+
+assets:
+  cs_admin_standard:
+    asset_id: projects/corestack-datasets/assets/datasets/cs_admin_standard
+    kind: feature_layer
+    format: gpkg
+    source: data/admin-boundary/cs_admin_standard.gpkg
+    layer: cs_admin_standard
+    geometry_column: geometry
+    properties:
+      include: "*"
+
+  cs_village_facility_proximity:
+    asset_id: projects/corestack-datasets/assets/datasets/cs_village_facility_proximity
+    kind: village_proximity_wide
+    source: data/facilities/outputs/cs_village_facility_proximity.gpkg
+    geometry_column: geometry
+    admin_columns:
+      cs_feature_id: cs_feature_id
+      state_name: state_name
+      district_name: district_name
+      TEHSIL: tehsil_name
+      pc11_village_id: village_id
+      NAME: village_name
+    include_l3:
+      distance_template: l3_{class_name}_distance_km
+      facility_uid_template: l3_{class_name}_facility_uid
+    include_l2:
+      distance_template: l2_{group_name}_distance_km
+      facility_uid_template: l2_{group_name}_facility_uid
+      selected_l3_template: l2_{group_name}_selected_l3
+    include_l1_summary:
+      enabled: true
+      nearest_distance_template: l1_{domain_name}_nearest_l2_distance_km
+      nearest_l2_template: l1_{domain_name}_nearest_l2_group
+
+  cs_pan_india_facilities:
+    asset_id: projects/corestack-datasets/assets/datasets/cs_pan_india_facilities
+    kind: facility_membership_points
+    source: data/facilities/outputs/pan_india_facilities.gpkg
+    geometry_column: geometry
+    columns:
+      facility:
+        - facility_uid
+        - facility_name
+        - facility_code
+        - source: latitude
+          target: facility_latitude
+        - source: longitude
+          target: facility_longitude
+        - urban_rural
+        - pincode
+        - establishment_year
+        - district_lgd
+        - village_census11
+        - village_name
+      membership:
+        - class_l1_domain
+        - class_l2_filter_group
+        - class_l3_facility_class
+        - class_l4_facility_subtype
+        - class_k1
+        - class_k2
+        - class_k3
+        - class_k4
+        - class_k5
+        - class_k6
+        - class_k7
+        - class_k8
+      generated:
+        - facility_membership_uid
+
+  cs_village_livestock_census_20:
+    asset_id: projects/corestack-datasets/assets/datasets/cs_village_livestock_census_20
+    kind: feature_layer
+    format: gpkg
+    source: data/livestock/cs_village_livestock_census_20.gpkg
+    layer: cs_village_livestock_census_20
+    geometry_column: geometry
+    properties:
+      include: "*"
+
+  cs_antyodaya_2020:
+    asset_id: projects/corestack-datasets/assets/datasets/cs_antyodaya_2020
+    kind: feature_layer
+    format: gpkg
+    source: data/antyodaya/output/cs_antyodaya_2020.gpkg
+    layer: cs_antyodaya_2020
+    geometry_column: geometry
+    properties:
+      include: "*"

utilities/scripts/gee/core_stack_gee_ingest.md

@@ -0,0 +1,114 @@
+# Core Stack GEE Ingestion
+
+This is the standard ingestion path for large Core Stack GeoPackage and
+GeoJSON vector assets.
+
+The script writes a local tab-delimited table with one GeoJSON geometry column,
+stages the same bytes to GCS with a `.csv` object suffix, then starts an Earth
+Engine table-ingestion task from a manifest.
+
+## Files
+
+- Script: `utilities/scripts/gee/core_stack_gee_ingest.py`
+- Config: `utilities/scripts/gee/core_stack_gee_assets.yaml`
+- Service account: `data/gee_confs/core-stack-learn-818963fa8f26.json`
+- Local outputs: `data/gee/core_stack/`
+- Manifests: `data/gee/core_stack/manifests/`
+- Log: `logs/core_stack_gee_ingest.log`
+
+## Configured Assets
+
+- `cs_admin_standard`
+- `cs_village_facility_proximity`
+- `cs_pan_india_facilities`
+- `cs_village_livestock_census_20`
+- `cs_antyodaya_2020`
+
+`cs_pan_india_facilities` currently uses the existing source file
+`data/facilities/outputs/pan_india_facilities.gpkg`; it is named with the `cs_`
+asset key in the GEE config without making a duplicate 2 GB GeoPackage.
+
+## Commands
+
+Inspect configured sources:
+
+```bash
+uv run --with pyyaml --with shapely \
+  python utilities/scripts/gee/core_stack_gee_ingest.py inspect --asset all
+```
+
+Smoke-build all configured assets:
+
+```bash
+uv run --with pyyaml --with shapely \
+  python utilities/scripts/gee/core_stack_gee_ingest.py build \
+  --asset all --limit 10 --jobs 2 --output-suffix .smoke \
+  --overwrite --max-rss-mb 5000
+```
+
+Dry-run manifests for the smoke outputs:
+
+```bash
+uv run --with pyyaml --with shapely \
+  python utilities/scripts/gee/core_stack_gee_ingest.py upload \
+  --asset all --output-suffix .smoke --dry-run --jobs 2
+```
+
+Build full local ingestion tables:
+
+```bash
+nohup uv run --with pyyaml --with shapely \
+  python utilities/scripts/gee/core_stack_gee_ingest.py build \
+  --asset all --jobs 2 --overwrite --max-rss-mb 5000 \
+  > logs/core_stack_gee_build_full_$(date -u +%Y%m%dT%H%M%SZ).log 2>&1 &
+```
+
+Start GEE ingestion after full tables are built:
+
+```bash
+nohup uv run --with earthengine-api --with google-cloud-storage --with pyyaml --with shapely \
+  python utilities/scripts/gee/core_stack_gee_ingest.py upload \
+  --asset all --jobs 2 --replace-existing \
+  > logs/core_stack_gee_upload_$(date -u +%Y%m%dT%H%M%SZ).log 2>&1 &
+```
+
+Check task status later:
+
+```bash
+uv run --with earthengine-api --with google-cloud-storage --with pyyaml \
+  python utilities/scripts/gee/core_stack_gee_ingest.py status
+```
+
+The status command also writes `data/gee/core_stack/core_stack_gee_status.yaml`
+by default.
+
+Make existing ingested assets public after tasks succeed:
+
+```bash
+uv run --with earthengine-api --with google-cloud-storage --with pyyaml \
+  python utilities/scripts/gee/core_stack_gee_ingest.py make-public --asset all
+```
+
+Verify completed GEE assets and sample schemas:
+
+```bash
+uv run --with earthengine-api --with google-cloud-storage --with pyyaml \
+  python utilities/scripts/gee/core_stack_gee_ingest.py verify --asset all
+```
+
+## Notes
+
+- The runtime memory guard defaults to `5000` MB and is checked at chunk
+  boundaries.
+- Special exporters accept either plain column names or `{source, target}`
+  column mappings in YAML. The facilities point asset aliases raw
+  `latitude`/`longitude` to `facility_latitude`/`facility_longitude` so Earth
+  Engine keeps them as ordinary properties while still using the GeoJSON
+  geometry column.
+- For large GeoJSON FeatureCollections, install `ijson` or convert to GeoJSONL:
+  `uv run --with ijson --with pyyaml --with shapely ...`.
+- `upload --make-public` is best-effort and waits for ingestion to finish
+  before applying ACLs. For long overnight uploads, submit tasks first, monitor
+  them, then use `make-public` or make assets public manually in Earth Engine if
+  IAM blocks the script.
+- `verify` writes `data/gee/core_stack/core_stack_gee_verify_summary.yaml`.