feat(tables): add export, import column creation, infinite row pagination (#4373)

* feat(tables): add export, import column creation, infinite row pagination

- Add `/api/table/[tableId]/export` route streaming CSV/JSON downloads
- Rename `/import-csv` route to `/import` and extend to auto-create new
  columns from unmapped CSV headers via `createColumns` form field
- Switch table view to `useInfiniteQuery` so tables larger than 1000
  rows fully load; reconcile created rows into the paginated cache so
  "New row" past 1000 no longer reverts on invalidate
- Wire scroll-driven prefetch (600px from bottom) and pre-drain pages
  before append to keep new-row position consistent
- Polish import-csv dialog flow and add Export action to header

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(table): route boundary validation through Zod contracts

Switch the table query hooks and the import/export routes to the
codebase's contract-based request pattern so the API validation audit
and boundary policy ratchet pass.

- `hooks/queries/tables.ts` now calls every endpoint via
  `requestJson(contract, ...)`; the only remaining raw `fetch` calls are
  the streaming export download and the multipart CSV upload, both
  annotated `boundary-raw-fetch:`.
- The import and export routes parse `params`, the `format` query, and
  the multipart form fields with shared schemas from
  `@/lib/api/contracts/tables`. Two new contract schemas
  (`csvImportCreateColumnsSchema`, `tableExportFormatSchema`) cover the
  fields specific to these routes.
- Bumps the audit baseline by one route to account for the new export
  endpoint.

* fix(table): address PR bot review

- Move the `isAppendingRowRef` reset into the create-row mutation's
  `onSettled` callback. The previous `try/finally` cleared the guard
  immediately after `mutate()` returned, before the request completed,
  so a rapid second click on "New row" could fire a duplicate create.
- Drop the unused `addTableColumns` wrapper from `lib/table/service`.
  The CSV import flow only ever uses the transaction-bound
  `addTableColumnsWithTx`; the standalone wrapper was dead code.

* fix(table): re-throw on infinite-query fetch error in append-row drain

`useInfiniteQuery.fetchNextPage()` resolves (rather than rejects) when a
page request fails — the resolved value carries `status: 'error'` while
`hasNextPage` still reflects the last successful page. The drain loop in
`handleAppendRow` relied on a thrown error to bail, so a failed mid-drain
fetch could spin indefinitely and leave the append guard stuck on.

Re-throw inside `fetchNextPageWrapped` when the result is an error so the
caller's `try/catch` runs as intended.

* fix(table): import TABLE_LIMITS from constants to keep server code out of client bundle

The client hook `use-table-data.ts` was importing `TABLE_LIMITS` as a
value from the `@/lib/table` barrel, which transitively pulls in
`service.ts` and the `postgres` driver. Turbopack then tried to bundle
`fs`, `net`, `tls`, and `perf_hooks` into the client component graph and
the production build failed.

Import `TABLE_LIMITS` directly from `@/lib/table/constants` (a pure
constants module) and keep the type imports against the barrel.

* fix(table): run batch unique check inside import transaction

`checkBatchUniqueConstraintsDb` queried the global `db` connection, so
inside a single import transaction (one tx wrapping all batches) the
constraint lookup couldn't see uncommitted rows from prior batches —
duplicates that crossed `CSV_MAX_BATCH_SIZE` boundaries slipped through.

Accept an optional executor and pass `trx` from `batchInsertRowsWithTx`
so the lookup observes the in-flight transaction state.

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

Author: waleedlatif1

apps/sim/app/api/table/[tableId]/export/route.ts

@@ -0,0 +1,131 @@
+import { createLogger } from '@sim/logger'
+import { type NextRequest, NextResponse } from 'next/server'
+import { tableExportFormatSchema, tableIdParamsSchema } from '@/lib/api/contracts/tables'
+import { getValidationErrorMessage } from '@/lib/api/server'
+import { checkSessionOrInternalAuth } from '@/lib/auth/hybrid'
+import { generateRequestId } from '@/lib/core/utils/request'
+import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
+import { queryRows } from '@/lib/table/service'
+import { accessError, checkAccess } from '@/app/api/table/utils'
+
+const logger = createLogger('TableExport')
+
+const EXPORT_BATCH_SIZE = 1000
+
+type ExportFormat = 'csv' | 'json'
+
+interface RouteParams {
+  params: Promise<{ tableId: string }>
+}
+
+/** GET /api/table/[tableId]/export - Streams the full table contents as CSV or JSON. */
+export const GET = withRouteHandler(async (request: NextRequest, { params }: RouteParams) => {
+  const requestId = generateRequestId()
+  const { tableId } = tableIdParamsSchema.parse(await params)
+
+  const auth = await checkSessionOrInternalAuth(request, { requireWorkflowId: false })
+  if (!auth.success || !auth.userId) {
+    return NextResponse.json({ error: 'Authentication required' }, { status: 401 })
+  }
+
+  const { searchParams } = new URL(request.url)
+  const formatValidation = tableExportFormatSchema.safeParse(
+    searchParams.get('format') ?? undefined
+  )
+  if (!formatValidation.success) {
+    return NextResponse.json(
+      { error: getValidationErrorMessage(formatValidation.error) },
+      { status: 400 }
+    )
+  }
+  const format: ExportFormat = formatValidation.data
+
+  const access = await checkAccess(tableId, auth.userId, 'read')
+  if (!access.ok) return accessError(access, requestId, tableId)
+  const { table } = access
+
+  const columns = table.schema.columns
+  const safeName = sanitizeFilename(table.name)
+  const filename = `${safeName}.${format}`
+
+  const stream = new ReadableStream<Uint8Array>({
+    async start(controller) {
+      const encoder = new TextEncoder()
+      try {
+        if (format === 'csv') {
+          controller.enqueue(encoder.encode(`${toCsvRow(columns.map((c) => c.name))}\n`))
+        } else {
+          controller.enqueue(encoder.encode('['))
+        }
+
+        let offset = 0
+        let firstJsonRow = true
+        while (true) {
+          const result = await queryRows(
+            tableId,
+            table.workspaceId,
+            { limit: EXPORT_BATCH_SIZE, offset, includeTotal: false },
+            requestId
+          )
+
+          for (const row of result.rows) {
+            if (format === 'csv') {
+              const values = columns.map((c) => formatCsvValue(row.data[c.name]))
+              controller.enqueue(encoder.encode(`${toCsvRow(values)}\n`))
+            } else {
+              const prefix = firstJsonRow ? '' : ','
+              firstJsonRow = false
+              controller.enqueue(encoder.encode(prefix + JSON.stringify({ ...row.data })))
+            }
+          }
+
+          if (result.rows.length < EXPORT_BATCH_SIZE) break
+          offset += result.rows.length
+        }
+
+        if (format === 'json') controller.enqueue(encoder.encode(']'))
+        controller.close()
+
+        logger.info(`[${requestId}] Exported table ${tableId}`, {
+          format,
+          rowCount: table.rowCount,
+        })
+      } catch (err) {
+        logger.error(`[${requestId}] Export failed for table ${tableId}`, err)
+        controller.error(err)
+      }
+    },
+  })
+
+  return new NextResponse(stream, {
+    status: 200,
+    headers: {
+      'Content-Type': format === 'csv' ? 'text/csv; charset=utf-8' : 'application/json',
+      'Content-Disposition': `attachment; filename="${filename}"`,
+      'Cache-Control': 'no-store',
+    },
+  })
+})
+
+function sanitizeFilename(name: string): string {
+  const cleaned = name.replace(/[^a-zA-Z0-9_-]+/g, '_').replace(/^_+|_+$/g, '')
+  return cleaned || 'table'
+}
+
+function formatCsvValue(value: unknown): string {
+  if (value === null || value === undefined) return ''
+  if (value instanceof Date) return value.toISOString()
+  if (typeof value === 'object') return JSON.stringify(value)
+  return String(value)
+}
+
+function toCsvRow(values: string[]): string {
+  return values.map(escapeCsvField).join(',')
+}
+
+function escapeCsvField(field: string): string {
+  if (/[",\n\r]/.test(field)) {
+    return `"${field.replace(/"/g, '""')}"`
+  }
+  return field
+}