Mirrored from Forgejo — originally by @vxfemboy. https://virus.ad/femboy/kibble/issues/84
kibble fetch "builder error" on datasets with image siblings or parquet-only layouts
Some HF datasets fail with download failed for <id>: builder error while others (single .jsonl) succeed. Reproduced building the kitby-v1 corpus:
ianncity/GLM-5.2-Conversation — a single dataset.jsonl plus a .webp sibling → builder error (had to direct-download the jsonl).r0b0tlab/qwen3.8-max-distillation-50k — parquet shards under data/ (no top-level jsonl) → builder error.SupraLabs/reasoning-corpus-4K-5M-v1 (single jsonl) and greghavens/kimi-k3-... (jsonl) → succeed.
The message text (download failed for {url}: {e} with {url} = the bare dataset id, {e} = builder error) points at a bare id reaching reqwest as a URL on the parquet-convert / fallback path — a RequestBuilder build failure, not an HTTP error. is_data_file already accepts parquet, so the parquet case should resolve via data/ or the parquet-convert branch; something in that path constructs the request from the id instead of a resolve URL.
Impact: common dataset layouts (image sibling; parquet-only) need a manual direct-download workaround.
Fix direction: ensure every fetch path builds a full resolve/main/... (or /parquet) URL, never the bare id; and skip non-data siblings (.webp/.png/...) before the builder step.
Found during kitby-v1 data foundry (P1).
kibble fetch"builder error" on datasets with image siblings or parquet-only layoutsSome HF datasets fail with
download failed for <id>: builder errorwhile others (single.jsonl) succeed. Reproduced building the kitby-v1 corpus:ianncity/GLM-5.2-Conversation— a singledataset.jsonlplus a.webpsibling → builder error (had to direct-download the jsonl).r0b0tlab/qwen3.8-max-distillation-50k— parquet shards underdata/(no top-level jsonl) → builder error.SupraLabs/reasoning-corpus-4K-5M-v1(single jsonl) andgreghavens/kimi-k3-...(jsonl) → succeed.The message text (
download failed for {url}: {e}with{url}= the bare dataset id,{e}=builder error) points at a bare id reachingreqwestas a URL on the parquet-convert / fallback path — aRequestBuilderbuild failure, not an HTTP error.is_data_filealready accepts parquet, so the parquet case should resolve viadata/or the parquet-convert branch; something in that path constructs the request from the id instead of a resolve URL.Impact: common dataset layouts (image sibling; parquet-only) need a manual direct-download workaround.
Fix direction: ensure every fetch path builds a full
resolve/main/...(or/parquet) URL, never the bare id; and skip non-data siblings (.webp/.png/...) before the builder step.Found during kitby-v1 data foundry (P1).