Doc-transform: Implement document transformation

[kartpop]

Is your feature request related to a problem? Please describe.
Partners often deal with scanned PDFs or document formats which are not very amenable with AI services like OpenAI’s vector stores. This increases the chances of poor RAG performance and may limit the platform’s utility.

Describe the solution you'd like
Build the foundational document transformation pipeline for the AI Platform. The primary goal is to enhance the /documents/upload endpoint to support on-demand, pluggable document conversion. This will allow users to prepare documents for optimal RAG performance directly within the platform by simplifying the conversion of documents to LLM-friendly formats, eg. pdf (i.e. OCR images encoded in pdf)→ markdown.

Reference doc - Document management and transformation

[onestardao]

Yeah this is definitely one of the upstream choke points for most RAG systems —
not just from format incompatibility, but from semantic distortion during flattening.

We’ve been working on something similar — basically injecting “structure-aware anchors” before OCR → MD.
Otherwise the logical flow of a scanned report (say, a nested table inside a multi-column layout) just collapses into noise.

If you’re curious, we even tested a method where we “suspend” the doc’s semantic blocks into a latent graph before flattening.
That way, the final markdown can reattach meaning during retrieval time.
Kinda like having a soft map of what’s lost in translation.

Happy to swap notes if you’re building the transformer module from scratch.