Annotation Format Schemav1.1
A portable format for structured UI feedback
Overview
The Annotation Format Schema (AFS) is an open format created and used by Agentation for capturing UI feedback in a way that AI coding agents can reliably parse and act on. Think of it like smart Figma comments for your running app: persistent annotations attached to specific elements, with threads, status tracking, resolution workflows, and structured metadata that agents can actually understand.
This spec defines the annotation object shape. Tools can emit annotations in this format, and agents can consume them regardless of how they were created.
What This Unlocks
A structured schema isn't just about clean data — it enables entirely new workflows:
- Two-way communication: Agents can reply to annotations, asking “Should this be 24px or 16px?” and get responses in the same thread
- Status tracking: See what's pending, acknowledged, resolved, or dismissed at a glance
- Cross-page queries: “What annotations do I have?” works across your entire site
- Bulk operations: “Clear all annotations” or “Show me blocking issues only”
- Persistent history: Feedback survives page refreshes and browser sessions
Without a schema, feedback is fire-and-forget. With one, it becomes a conversation.
Design Goals
- Agent-readable: Structured data that LLMs can parse without guessing
- Framework-agnostic: Works with any UI, though React gets extra context
- Tool-agnostic: Any tool can emit, any agent can consume
- Human-authored: Designed for feedback from humans (or automated reviewers)
- Minimal core: Few required fields, many optional for richer context
Annotation Object
An annotation represents a single piece of feedback attached to a UI element.
Note: The server may add metadata fields (sessionId, createdAt, updatedAt) when syncing annotations.
Required Fields
{
id: string; // Unique identifier (e.g. "ann_abc123")
comment: string; // Human feedback ("Button is misaligned")
elementPath: string; // CSS selector path ("body > main > button.cta")
timestamp: number; // Unix timestamp (ms)
x: number; // % of viewport width (0-100)
y: number; // px from document top (or viewport if isFixed)
element: string; // Tag name ("button", "div", "input")
}Recommended Fields
{
url: string; // Page URL where annotation was created
boundingBox: { // Element position at annotation time
x: number;
y: number;
width: number;
height: number;
};
}Optional Context Fields
{
// React-specific (when available)
reactComponents: string; // Component tree ("App > Dashboard > Button")
// Element details
cssClasses: string; // Class list ("btn btn-primary disabled")
computedStyles: string; // Key CSS properties
accessibility: string; // ARIA attributes, role
nearbyText: string; // Visible text in/around element
selectedText: string; // Text highlighted by user
// Feedback classification
intent: "fix" | "change" | "question" | "approve";
severity: "blocking" | "important" | "suggestion";
// Annotation kind (defaults to "feedback")
kind: "feedback" | "placement" | "rearrange";
// Layout mode: placement data
placement: {
componentType: string; // e.g. "Hero", "Card", "Navigation"
width: number; // px
height: number; // px
scrollY: number; // scroll position when placed
text?: string; // optional label text
};
// Layout mode: rearrange data
rearrange: {
selector: string; // CSS selector of the section
label: string; // human-readable label
tagName: string; // HTML tag name
originalRect: { x: number; y: number; width: number; height: number };
currentRect: { x: number; y: number; width: number; height: number };
};
}Lifecycle Fields
{
status: "pending" | "acknowledged" | "resolved" | "dismissed";
resolvedAt: string; // ISO timestamp
resolvedBy: "human" | "agent";
thread: ThreadMessage[]; // Back-and-forth conversation
}Browser Component Fields
These optional fields are set by the Agentation browser component for UI rendering:
{
isFixed: boolean; // Element has fixed/sticky positioning
isMultiSelect: boolean; // Created via drag selection
fullPath: string; // Full DOM path (vs shorter elementPath)
nearbyElements: string; // Info about nearby DOM elements
}Full TypeScript Definition
type Annotation = {
// Required
id: string;
comment: string;
elementPath: string;
timestamp: number;
x: number; // % of viewport width (0-100)
y: number; // px from document top (or viewport if isFixed)
element: string; // Tag name ("button", "div")
// Recommended
url?: string;
boundingBox?: {
x: number;
y: number;
width: number;
height: number;
};
// Optional context
reactComponents?: string;
cssClasses?: string;
sourceFile?: string;
attributes?: Record<string, string>;
computedStyles?: string;
accessibility?: string;
nearbyText?: string;
selectedText?: string;
// Browser component fields
isFixed?: boolean; // Element has fixed/sticky positioning
isMultiSelect?: boolean; // Created via drag selection
fullPath?: string; // Full DOM path
nearbyElements?: string; // Info about nearby elements
// Feedback classification
intent?: "fix" | "change" | "question" | "approve";
severity?: "blocking" | "important" | "suggestion";
// Annotation kind
kind?: "feedback" | "placement" | "rearrange";
// Layout mode data
placement?: {
componentType: string;
width: number;
height: number;
scrollY: number;
text?: string;
};
rearrange?: {
selector: string;
label: string;
tagName: string;
originalRect: { x: number; y: number; width: number; height: number };
currentRect: { x: number; y: number; width: number; height: number };
};
// Lifecycle
status?: "pending" | "acknowledged" | "resolved" | "dismissed";
resolvedAt?: string;
resolvedBy?: "human" | "agent";
thread?: ThreadMessage[];
};
type ThreadMessage = {
id: string;
role: "human" | "agent";
content: string;
timestamp: number;
};Browser metadata
The browser component can add optional identifying attributes, a source-file reference and same-origin frame context. These fields travel with annotations through callbacks and MCP storage; they do not change the required AFS fields.
// Optional fields emitted by the React component
sourceFile?: string; // path:line:column when available
attributes?: Record<string, string>;
frame?: {
path: Array<{ index: number; id?: string; url: string }>;
x: number; // % of child viewport width
y: number; // child document px, or child viewport px if fixed
fixed: boolean;
boundingBox: { x: number; y: number; width: number; height: number };
};Frame paths describe the nesting from the top document to the selected element’s document. Frame geometry is kept in child coordinates so a saved single-element marker can follow scrolling without changing its identity.
Event Envelope
For real-time streaming, annotations are wrapped in an event envelope:
type AgentationEvent = {
type: "annotation.created" | "annotation.updated" | "annotation.deleted"
| "session.created" | "session.updated" | "session.closed"
| "thread.message" | "action.requested";
timestamp: string; // ISO 8601
sessionId: string;
sequence: number; // Monotonic for ordering/replay
payload: Annotation | Session | ThreadMessage | ActionRequest;
};The sequence number enables clients to detect missed events and request replay. See MCP for SSE streaming details.
JSON Schema
For validation in any language:
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://agentation.com/schema/annotation.v1.1.json",
"title": "Annotation",
"type": "object",
"required": [
"id",
"comment",
"elementPath",
"timestamp",
"x",
"y",
"element"
],
"properties": {
"id": {
"type": "string"
},
"comment": {
"type": "string"
},
"elementPath": {
"type": "string"
},
"timestamp": {
"type": "number"
},
"x": {
"type": "number",
"description": "% of viewport width (0-100)"
},
"y": {
"type": "number",
"description": "px from document top"
},
"element": {
"type": "string"
},
"url": {
"type": "string",
"format": "uri"
},
"boundingBox": {
"type": "object",
"properties": {
"x": {
"type": "number"
},
"y": {
"type": "number"
},
"width": {
"type": "number"
},
"height": {
"type": "number"
}
},
"required": [
"x",
"y",
"width",
"height"
]
},
"reactComponents": {
"type": "string"
},
"isFixed": {
"type": "boolean"
},
"isMultiSelect": {
"type": "boolean"
},
"fullPath": {
"type": "string"
},
"nearbyElements": {
"type": "string"
},
"intent": {
"enum": [
"fix",
"change",
"question",
"approve"
]
},
"severity": {
"enum": [
"blocking",
"important",
"suggestion"
]
},
"kind": {
"enum": [
"feedback",
"placement",
"rearrange"
],
"default": "feedback"
},
"placement": {
"type": "object",
"properties": {
"componentType": {
"type": "string"
},
"width": {
"type": "number"
},
"height": {
"type": "number"
},
"scrollY": {
"type": "number"
},
"text": {
"type": "string"
}
},
"required": [
"componentType",
"width",
"height",
"scrollY"
]
},
"rearrange": {
"type": "object",
"properties": {
"selector": {
"type": "string"
},
"label": {
"type": "string"
},
"tagName": {
"type": "string"
},
"originalRect": {
"$ref": "#/properties/boundingBox"
},
"currentRect": {
"$ref": "#/properties/boundingBox"
}
},
"required": [
"selector",
"label",
"tagName",
"originalRect",
"currentRect"
]
},
"status": {
"enum": [
"pending",
"acknowledged",
"resolved",
"dismissed"
]
}
}
}Example Annotation
{
"id": "ann_k8x2m",
"comment": "Button is cut off on mobile viewport",
"elementPath": "body > main > .hero-section > button.cta",
"timestamp": 1705694400000,
"x": 45.5,
"y": 480,
"element": "button",
"url": "http://localhost:3000/landing",
"boundingBox": { "x": 120, "y": 480, "width": 200, "height": 48 },
"reactComponents": "App > LandingPage > HeroSection > CTAButton",
"cssClasses": "cta btn-primary",
"nearbyText": "Get Started Free",
"intent": "fix",
"severity": "blocking",
"status": "pending"
}Layout Mode Example
Layout mode annotations use the kind field to distinguish from regular feedback:
{
"id": "ann_d3s1gn",
"comment": "Place a Hero component here",
"elementPath": "body",
"timestamp": 1709510400000,
"x": 50,
"y": 200,
"element": "body",
"kind": "placement",
"placement": {
"componentType": "Hero",
"width": 800,
"height": 400,
"scrollY": 0,
"text": "Hero"
},
"status": "pending"
}Markdown Output Format
For pasting into chat-based agents, annotations can be serialized as markdown:
## Annotation #1
**Element:** button.cta
**Path:** body > main > .hero-section > button.cta
**React:** App > LandingPage > HeroSection > CTAButton
**Position:** 120px, 480px (200×48px)
**Feedback:** Button is cut off on mobile viewport
**Severity:** blockingSee Output Formats for detail level options (Compact → Forensic).
Implementations
Tools that emit or consume this format:
| Project | Description |
|---|---|
| Agentation (React) | Click-to-annotate toolbar for React apps |
| Agentation MCP Server | Exposes annotations to Claude Code and other MCP clients |
Building an Implementation
To emit Agentation Format annotations from your tool:
- Capture the required fields:
id,comment,elementPath,timestamp,x,y,element - Add recommended fields for better agent accuracy:
url,boundingBox - For React apps, traverse the fiber tree to get
reactComponents - Output as JSON for MCP/API consumption, or markdown for chat pasting
See the Agentation source for reference implementations of element detection and React component traversal.
Why This Format?
Existing agent protocols (MCP, A2A, ACP) standardize tools and messaging, but they don't define a UI feedback grammar. They rely on whatever structured context you feed them.
This format fills that gap: a portable wire format specifically for "human points at UI, agent needs to find and fix the code." We hope it's useful to others building similar tools.
Versioning
Current version: v1.1
Schema URL: annotation.v1.1.json