progressive-json

Stream JSON data as it arrives, not when it's complete

The Problem

Waiting for complete JSON responses creates slow, unresponsive experiences. Users see blank screens while large datasets load.

progressive-json streams JSON data chunk by chunk, updating your client as each piece arrives.

How It Works

The server sends JSON in progressive chunks, and your app processes each piece immediately:

Features

• ⚡ Instant Updates - See data as it streams in
• 🎯 Smart References - Handle complex nested data seamlessly
• ⚛️ React Ready - Simple useProgressiveJson hook
• 🛡️ TypeScript - Full type safety included
• 🔄 Universal - Works with any streaming endpoint
• 🔌 Plugin System - Extend functionality with custom message types

Core Concepts

References and Placeholders

Progressive JSON works by creating "placeholders" in your initial JSON structure, then filling them in as data becomes available.

1. Create references using generateRefKey()
2. Use references as placeholders in your initial JSON structure
3. Fill placeholders using API functions as data becomes available

// 1. Create reference
const userNameRef = generateRefKey();

// 2. Use as placeholder
writer(init({ user: { name: userNameRef } }));

// 3. Fill with actual data
writer(value(userNameRef, "Alice"));

Mixing API Calls

You can call different API functions on the same reference:

// This is perfectly valid:
writer(text(logRef, "Starting... "));
writer(text(logRef, "progress... "));
writer(value(logRef, "Complete!")); // Replaces entire content

API Reference

Core Functions

init(data)

Initialize the JSON structure with placeholders for dynamic content.

const userNameRef = generateRefKey();
const postsRef = generateRefKey();

init({
  user: { name: userNameRef, age: 30 },
  posts: postsRef,
  staticData: "Loaded!"
})

value(ref, value)

Set or replace a placeholder with its final value.

value(userNameRef, "Alice")
value(postsRef, [{ id: 1, title: "First Post" }])

text(ref, value)

Append text to a placeholder progressively.

text(logRef, "Processing... ")
text(logRef, "almost done... ")
text(logRef, "complete!")
// Result: "Processing... almost done... complete!"

push(ref, value)

Add a single item to an array placeholder.

push(notificationsRef, { id: 1, message: "New notification" })
push(notificationsRef, { id: 2, message: "Another notification" })

concat(ref, values)

Add multiple items to an array placeholder at once.

concat(usersRef, [
  { id: 1, name: "Alice" }, 
  { id: 2, name: "Bob" }
])

When to Use Each Function

• value() - Set or replace entire value (final data)
• text() - Build up text progressively (streaming text)
• push() - Add single items to arrays one by one
• concat() - Add multiple items to arrays efficiently

Utility Functions

generateRefKey()

Creates a unique reference key for placeholders.

const myRef = generateRefKey();

writeln(res) & writeChunkHeaders(res)

Server utilities for streaming responses.

// Source code - no black magic!
export function writeln(res: { write: (chunk: string) => void }) {
  return (placeholder: Placeholder) => {
    res.write(JSON.stringify(placeholder) + "\n");
  };
}

export function writeChunkHeaders(res: { setHeader: (name: string, value: string) => void }) {
  res.setHeader("Content-Type", "application/json; charset=utf-8");
  res.setHeader("Transfer-Encoding", "chunked");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");
}

Usage:

writeChunkHeaders(res);
const writer = writeln(res);

Plugin System

Extend progressive-json with custom message types and behaviors.

Creating a Plugin

import type { Plugin } from "@yoyo-org/progressive-json";

const myPlugin: Plugin = {
  type: "my-type",
  handleMessage: (message, store, context) => {
    // Handle your custom message type
    return context.updateAtPath(store, message.key, (obj, lastKey) => {
      obj[lastKey] = message.value;
    });
  },
};

Using Plugins

import { useProgressiveJson } from "@yoyo-org/progressive-json";

const { store } = useProgressiveJson({
  url: "...",
  plugins: [myPlugin],
});

Type-Safe Plugins

For full type safety, use the two-generic interface:

type CustomMessage = { 
  type: "custom"; 
  key: string; 
  value: string; 
};

const typedPlugin: Plugin<CustomMessage, MyStoreType> = {
  type: "custom",
  handleMessage: (message, store, context) => {
    // message is fully typed as CustomMessage
    // store is fully typed as MyStoreType
    return context.updateAtPath(store, message.key, (obj, lastKey) => {
      obj[lastKey] = message.value;
    });
  },
};

Server-Side Plugin Messages

To send custom plugin messages from the server, create helper functions and a custom writer:

// 1. Create helper function for your custom message type
function custom(key: string, value: string, metadata?: { timestamp: string }) {
  return { type: "custom", key, value, metadata };
}

// 2. Use in your server endpoint
app.get("/api/custom-plugin", async (req, res) => {
  writeChunkHeaders(res);
  const writer = writeln(res);
  
  const customDataRef = generateRefKey();
  
  // Send initial structure
  writer(init({ customData: customDataRef }));
  
  // Send custom plugin message
  writer(res, custom(customDataRef, "Hello from custom plugin!", {
    timestamp: new Date().toISOString(),
  }));
  
  res.end();
});

Complete Plugin Example

Client-side plugin:

type CustomMessage = { 
  type: "custom"; 
  key: string; 
  value: string; 
  metadata?: { timestamp: string };
};

const customPlugin: Plugin<CustomMessage> = {
  type: "custom",
  handleMessage: (message, store, context) => {
    console.log("Received custom message:", message.value);
    console.log("Timestamp:", message.metadata?.timestamp);
    
    return context.updateAtPath(store, message.key, (obj, lastKey) => {
      obj[lastKey] = message.value;
    });
  },
};

Server-side helper:

function custom(key: string, value: string, metadata?: { timestamp: string }) {
  return { type: "custom", key, value, metadata };
}

// Usage in server
writer(custom(myRef, "Custom data", { 
  timestamp: new Date().toISOString() 
}));

Quick Start

Install

npm install @yoyo-org/progressive-json

1. Server Example (Express)

import express from "express";
import cors from "cors";
import {
  writeln,
  writeChunkHeaders,
  init,
  value,
  text,
  generateRefKey,
} from "@yoyo-org/progressive-json";

const app = express();
app.use(cors());

app.get("/api/progressive", async (req, res) => {
  writeChunkHeaders(res);
  const writer = writeln(res);

  // Create references
  const userNameRef = generateRefKey();
  const postsRef = generateRefKey();
  const logRef = generateRefKey();

  // Send initial structure
  writer(init({
    user: { name: userNameRef, age: 30 },
    posts: postsRef,
    staticData: "Loaded!",
    log: logRef,
  }));
  
  await new Promise(r => setTimeout(r, 100));

  // Send data progressively
  writer(value(userNameRef, "Alice"));
  await new Promise(r => setTimeout(r, 100));

  writer(value(postsRef, [
    { id: 1, title: "First Post" },
    { id: 2, title: "Second Post" }
  ]));
  await new Promise(r => setTimeout(r, 100));

  // Stream text progressively
  const words = "Streaming text word by word".split(" ");
  for (const word of words) {
    writer(text(logRef, word + " "));
    await new Promise(r => setTimeout(r, 200));
  }

  res.end();
});

app.listen(3001, () => {
  console.log("Server running at http://localhost:3001");
});

2. Client Example (React)

import { useProgressiveJson } from "@yoyo-org/progressive-json";

export function ProgressiveDemo() {
  const { store } = useProgressiveJson({
    url: "http://localhost:3001/api/progressive",
  });

  if (!store) return <div>Loading...</div>;

  return (
    <div>
      <h2>Progressive JSON Demo</h2>
      <pre>{JSON.stringify(store, null, 2)}</pre>
      
      {/* Live streaming text */}
      <div>
        <strong>Live log:</strong> {store.log}
      </div>
    </div>
  );
}

3. TypeScript Support

Define your data structure for full type safety:

interface MyData {
  user: { name: string; age: number };
  posts: Array<{ id: number; title: string }>;
  log: string;
  staticData: string;
}

const { store } = useProgressiveJson<MyData>({
  url: "http://localhost:3001/api/progressive",
});

// store is now fully typed as MyData | null

Advanced Examples

Array Operations

// Server side
const itemsRef = generateRefKey();

writer(init({ items: itemsRef }));

// Add items one by one
writer(push(itemsRef, { id: 1, name: "Item 1" }));
writer(push(itemsRef, { id: 2, name: "Item 2" }));

// Add multiple items at once
writer(concat(itemsRef, [
  { id: 3, name: "Item 3" },
  { id: 4, name: "Item 4" }
]));

Nested References

// Server side
const userRef = generateRefKey();
const profileRef = generateRefKey();

writer(init({
  user: userRef,
  metadata: { timestamp: Date.now() }
}));

writer(value(userRef, {
  name: "Alice",
  profile: profileRef
}));

writer(value(profileRef, {
  bio: "Software developer",
  avatar: "https://example.com/avatar.png"
}));

Progressive Text Streaming

// Server side - perfect for AI responses
const responseRef = generateRefKey();
writer(init({ aiResponse: responseRef }));

const response = "This is a progressive response from AI";
for (const char of response) {
  writer(text(responseRef, char));
  await new Promise(r => setTimeout(r, 50));
}

License

MIT License - see LICENSE for details.

---

Made with ❤️ by @yoyo-67