Direct GPU Screen Perception for Autonomous Desktop Agents
VACT eliminates screenshot-based Vision-Language Model latency. It captures display framebuffers directly in GPU VRAM, runs bilateral edge filtering and connected component labeling via compute shaders, and streams sub-2KB differential vector DAGs over local Win32 named pipes in under 20ms.
git clone https://github.com/fy2ne/VACT.git
Why Screenshot VLMs Fail at Real Desktop Control
Vision-Language Models take screenshots, encode them to JPEG, upload them across the internet, and guess 2D coordinates. That pipeline is fundamentally unsuited for responsive desktop tasks.
| Perception Attribute | Screenshot VLMs (Claude Computer Use, etc.) | VACT Direct GPU Kernel |
|---|---|---|
| Frame Extraction | CPU GDI/PrintWindow encode to JPEG/PNG (40–120ms) | Zero-copy VRAM surface via DXGI Desktop Duplication (1.2ms) |
| Per-Turn Latency | 1,800ms – 4,500ms network round-trip | 18ms – 22ms local compute cycle (up to 60 FPS capable) |
| Wire Payload | 1.5 MB – 2.5 MB image data per step | < 2 KB structured differential vector mutations |
| Token Consumption | ~2,200 input tokens per step (~$0.05 / action) | 0 image tokens; standard text tool calls only ($0.00 vision cost) |
| Target Precision | Estimated visual coordinates (frequent hallucination & miss clicks) | Mathematically exact centroid moments [cx, cy] computed via CCL |
| Video & Canvas Blindspots | HTML5 videos, WebGL 3D, DirectX viewports blur or lag behind | Direct photon extraction: analyzes whatever the GPU outputs to the display |
| Network & Privacy | Desktop screen content sent to third-party cloud servers | 100% air-gapped on-device Win32 Named Pipe (\\.\pipe\VACT) |
The Zero-Copy GPU Perception Pipeline
VACT executes six deterministic stages entirely on your local machine to translate raw hardware display photons into structured vector ASTs.
DXGI Output Duplication
Directly binds to the Windows DWM display output via IDXGIOutputDuplication. Frames are acquired directly into a Direct3D 11 texture without CPU memory copying.
Bilateral Edge Filtering
An HLSL compute shader applies spatial and range Gaussian weighting in parallel across 64-thread workgroups. Denoises video playback artifacts while keeping pixel-sharp UI borders.
Connected Component Labeling
Parallel CCL kernel with stride-2 subsampling segments contiguous interactive UI controls, buttons, text fields, and icons into normalized bounding boxes.
Single-Pass WinRT OCR
Executes full-frame WinRT hardware OCR with automatic downsampling for 4K displays. Replaces legacy multi-crop COM calls, dropping text detection from 320ms to 10ms.
Vector Scene Graph Assembly
Correlates bounding boxes with OCR glyphs, resolves active window focus, associates taskbar applications, and computes exact sub-pixel centroid click targets [cx, cy].
Named Pipe IPC Streaming
Diff engine calculates monotonic mutations (INSERT / UPDATE / DELETE) and emits a sub-2KB framed frame over local pipe \\.\pipe\VACT directly to client SDKs.
Interactive Vector Scene Graph Inspector
Hover over any detected desktop node below to inspect the exact JSON scene graph emitted by the VACT kernel to client agents. No fake SVGs — this is the real protocol schema.
{
"id": 6,
"type": "TEXT_INPUT",
"label": "Text Area",
"bounds": [130, 150, 860, 470],
"centroid": [560, 385],
"interactable": true,
"focused": true,
"depth": 2,
"parent_id": 1,
"surface": "IDXGIOutputDuplication",
"dt_ms": 18.2
}1-Bit Ordered Bayer Dither Matrix
Move your cursor across the canvas below to reveal the pixelated retro face beneath the ordered dither threshold.
Verified Native SDKs
Official bindings connecting your agent decision loops directly to the local Win32 named pipe \\.\pipe\VACT.
import { VactClient } from 'vact-sdk';
const client = new VactClient();
await client.connect();
// Hydrated AST state available synchronously at any time
const scene = client.getState();
console.log(`Active Window: ${scene.active_window}`);
// Reactive State Barrier: waits for window focus transition before dispatch
await client.waitForStateChange({
predicate: (g) => g.active_window.includes('Notepad'),
timeoutMs: 3000,
});
// Dispatch deterministic hardware click via sub-pixel centroid
await client.agent().click(6);
await client.agent().type(6, 'VACT Protocol Active');Zero-Friction Local Execution
Run the daemon, check environment health, and launch an autonomous mission with a single command.
Clone Repository & Verify Environment
Run the built-in diagnostic doctor to verify DXGI output duplication and WinRT dependencies.
git clone https://github.com/fy2ne/VACT.git && cd VACT
.\vact.cmd doctor
Run Autonomous Agent Mission
The root launcher automatically boots vactd in the background if not running, then dispatches the mission.
.\vact.cmd --mission "Open Notepad and type VACT Protocol Active"
Inspect Real-Time HUD Overlay (Optional)
Launch the transparent Minecraft F3+G sci-fi HUD matrix over your desktop to observe real-time bounding box segmentation.
cargo run -p vactd -- start --overlay
Why Desktop Agents Fail: The Architectural Limits of Vision-Language Models
fy2ne dissects why 2,000-token screenshot models bleed inference costs and latency, how accessibility tree scrapers get blinded by hardware-accelerated canvases, and how deterministic direct GPU compute pipelines eliminate coordinate hallucination.
Read fy2ne's Technical Manifesto ↗