Collaborative Documents

Real-time collaborative code editing with Monaco, Yjs CRDTs, and room-based permissions.

Overview

collab_docs provides persistent, multi-user document editing. Documents are attached to chat rooms or standalone, edited with the Monaco code editor in the browser, and synchronized in real-time using Yjs CRDTs over Socket.IO. The server maintains in-memory Yjs documents with periodic persistence to SQLite. A full permission system controls access at owner/editor/viewer/room-member levels.

Architecture

collab_docs.doh.yaml
├── collab_docs_server    (nodejs)
│   ├── Yjs document lifecycle (in-memory + periodic persistence)
│   ├── Socket.IO sync protocol
│   ├── Permission system (4 groups)
│   ├── REST API (CRUD + page routes)
│   └── Programmatic API (createDoc, readDocContent, etc.)
└── collab_docs_client    (browser)
    ├── CollabDocManager         (full-page doc list + editor)
    ├── CollabDocManagerToolbar  (compact toolbar for chat rooms)
    ├── CollabDocManagerEditor   (Monaco + Yjs binding)
    └── collab_shared            (transport provider)

Depends on: express_router, user_host, collab_shared (browser) Loaded by: chat.doh.yaml Depended on by: collab_richtext

File Structure

Configuration

These are compile-time constants, not pod-configurable.

Database Schema

Document metadata shape:

{
  id: string,              // UUID
  title: string,
  room_id: string | null,  // Chat room ID or null for standalone
  owner: string,           // Username
  editors: string[],       // Usernames with write access
  viewers: string[],       // Usernames with read-only access
  type: string,            // 'plaintext' | 'richtext' | 'surface' | 'orchestrator' | 'agent_workflow' | 'tracker'
  language: string,        // Monaco language ID ('markdown', 'javascript', etc.) or 'json' for graph types
  created_at: number,
  updated_at: number
}

Document state shape:

{
  id: string,      // Document ID (matches documents table)
  state: string    // Base64-encoded Yjs state (Y.encodeStateAsUpdate output)
}

Permissions

Contexts

Groups

Permission Flow

1. Build context: { documentId, owner, editors, viewers, roomMembers }
2. Check Doh.permit(user, action, context) against groups
3. Cascade: owner → editor → room member → viewer

Room Member Permission Inheritance

The collab_doc_room_member group grants read and write to any authenticated user who is a member of the room to which the document is attached (doc.room_id). Room membership is resolved at permit-check time by querying chat.room_members for the room.

The condition checks ctx.roomMembers?.includes(user.username). For the lobby room, roomMembers is the special wildcard array ['*'], and the condition treats this as matching all authenticated users — so all authenticated users can read and write lobby documents.

// collab_doc_room_member condition (simplified)
condition: (user, ctx) => {
  if (ctx?.roomMembers?.includes('*')) return true;       // lobby wildcard
  if (ctx?.roomMembers?.includes(user?.username)) return true;
  return false;
}

This means:

• Standalone documents (no room_id) have no room member group — only explicit editors/viewers and the owner.
• Room documents are automatically accessible to all room members without explicit sharing.
• Lobby documents are accessible to all authenticated users.

REST API Reference

POST /api/collab/docs

// Request
{ title: string, room_id?: string, editors?: string[], viewers?: string[], type?: string, language?: string }
// Response
{ doc: { id, title, room_id, owner, editors, viewers, type, language, created_at, updated_at } }

Page Routes

Socket Events Reference

Document Operations

collab:doc:open response detail:

{
  doc_meta: { id, title, room_id, owner, editors, viewers, type, language, created_at, updated_at },
  readOnly: boolean,          // User has read-only access
  syncStep1: number[],        // Yjs state vector (Uint8Array as array)
  syncStep2: number[],        // Full document state
  awarenessStates: number[],  // Remote cursor/presence states
  clientCount: number         // Active editors
}

Sync Protocol

Server Broadcasts

Lobby Events

For real-time document list updates on the /docs manager page.

IMPORTANT: Always use socket.emit('collab:doc:list', ...) for listing documents, not Doh.ajaxPromise(). The socket handler routes correctly to the list operation; ajaxPromise may trigger creation instead.

Document Lifecycle: In-Memory vs Persistent State

Loading

When a client opens a document (collab:doc:open) and the document is not already in the activeDocs map:

1. A new Y.Doc and Awareness are created in memory.
2. The server loads the base64-encoded Yjs state from collab_docs.doc_state and applies it to the Yjs document.
3. A ydoc.on('update', ...) listener is registered to trigger persistence debouncing on every edit.
4. The document is stored in activeDocs as { ydoc, awareness, clients: Map, persistTimer, gcTimer }.

Persistence Debounce

Every Yjs update triggers schedulePersist(docId). This debounces by PERSIST_DEBOUNCE_MS (5 seconds): if an update arrives within 5 seconds of the last scheduled save, the timer resets. When the timer fires, persistDocState runs:

1. Y.encodeStateAsUpdate(ydoc) serializes the full document.
2. The result is base64-encoded and written to collab_docs.doc_state via ReplaceIntoDB.
3. The document's updated_at timestamp in collab_docs.documents is updated.

Awareness state (cursors/presence) is never persisted. It is in-memory only and cleared when the document is garbage collected.

Garbage Collection

When a client closes a document (collab:doc:close) or disconnects, the client is removed from the document's clients map and scheduleGC(docId) is called. The GC timer fires after GC_TIMEOUT_MS (5 minutes). If the document still has no clients at that point:

1. persistDocState is called immediately to save any unsaved edits.
2. ydoc.destroy() releases the Yjs document from memory.
3. The entry is removed from activeDocs.

If any new client opens the document before the GC timer fires, the timer is cancelled.

The GC cycle in summary:

client opens doc    → cancel GC timer, add to clients
client closes doc   → remove from clients, start 5-min GC timer
5 min passes        → persist final state, destroy ydoc, remove from activeDocs
next client opens   → load from DB, create new ydoc

Lobby Welcome Document

On server startup, ensureLobbyDocExists() checks for a document with ID lobby-welcome-doc. If it does not exist, it is created automatically with:

The initial Yjs content is pre-populated with a welcome markdown message. Because the document's room_id is 'lobby', the collab_doc_room_member group applies with the wildcard ['*'] room members array, granting all authenticated users read and write access. No explicit editor assignment is needed.

The system user (owner: 'system') is not a real account and cannot authenticate. Only room membership grants access to this document.

Programmatic API

Exported via Doh.Globals.CollabDocs for use by other modules (e.g., bot integrations):

Doh.Module('my_module', ['collab_docs_server'], function() {
  const docs = Doh.Globals.CollabDocs;

  // Create a document
  const doc = await docs.createDoc({
    title: 'Meeting Notes',
    content: '# Notes\n',
    room_id: 'room-123',
    owner: 'alice'
  });

  // Read/write content
  const content = docs.readDocContent(doc.id);
  const previous = docs.writeDocContent(doc.id, '# Updated\n');

  // Undo/redo
  docs.undoDocEdit(doc.id, 'system');
  docs.redoDocEdit(doc.id, 'system');

  // Query
  const roomDocs = docs.listRoomDocs('room-123');
  const single = docs.getDocById(doc.id);
  const canEdit = docs.canAccess(user, doc, 'write');
});

Exported Functions

Document Types

Graph types (surface, orchestrator, agent_workflow) share the same Yjs layout — arrays of node/item Y.Maps and edge Y.Maps plus a metadata map. The items key differs: surfaces use 'items', orchestrator and agent_workflow use 'nodes'.

Bot Tool Integration (graph_edit)

The collab_docs bot tool group includes a graph_edit tool for graph document types. Instead of freehand content replacement (which edit_document provides for text types), graph documents are edited via discrete operations applied in a single Yjs transaction:

• add_node — type, name, config (dimensions and position are automatic)
• update_node — node_id, name, partial config update
• remove_node — node_id (also removes connected edges)
• add_edge — source_node_id, source_port, target_node_id, target_port (for workflow nodes); source_anchor, target_anchor (for canvas widgets)
• remove_edge — edge_id

Node naming — Every node should be given a descriptive name (e.g. "Fetch User Data", "Validate Response"). Names appear in node headers and distinguish nodes of the same type.

Canvas widgets — In addition to workflow node types (orch_*, agent_*), graph_edit supports base Surface widget types: sticky_note, text_label, shape, frame, media, collab_embed, url_browser. These use anchor-based connections (top/right/bottom/left) instead of ports. Text fields (text, title) are automatically converted to Yjs collaborative types (Y.XmlFragment / Y.Text) on the server.

Auto-layout — After every graph_edit batch, the server runs a layout engine (graph_layout.js in collab_shared) that positions nodes using topological layering with actual type-specific dimensions. Large fan-out layers split into sub-columns. Agents do not need to calculate positions.

Operations can reference nodes created in the same batch using $N syntax (e.g., $0 for the first add_node result). All operations execute atomically — connected clients see the changes in real-time via Yjs sync.

Client Patterns

Usage in Chat

When a user enters a room:

1. chat_client.js calls docToolbar.loadDocs()
2. Client emits collab:doc:list with room_id
3. Server filters by room and permissions
4. Client renders doc tiles in toolbar
5. Clicking a tile opens the Monaco editor

Testing

1. Open /docs — verify document list loads
2. Create a new document — verify it appears in the list
3. Open document in two browsers — verify real-time sync (typing appears in both)
4. Verify cursor awareness (colored cursors from other users)
5. Test permissions: share doc with viewer, verify read-only mode
6. Open /collab/<doc-id> — verify fullscreen editor loads
7. In chat room, verify doc toolbar shows room documents
8. Verify GC: open a document, close it, wait 5 minutes, reopen — verify content is preserved (loaded from DB)
9. Verify lobby welcome doc: authenticated user should be able to read and edit lobby-welcome-doc without being explicitly listed as editor