VACT
Vector Agent Context Telemetry
Visual Action & Control Telemetry by fy2ne
Direct3D 11 • Rust Kernel • Sub-5ms Vector Telemetry

The Perception Protocol for Autonomous Desktop Agents

Engineered by fy2ne to replace slow screenshot Vision-Language Models with deterministic GPU compute shaders, Connected Component Labeling, and sub-2KB differential DAG streams at native 60 FPS.

View on GitHub Machine Spec (JSON)
Direct3D 11 GPU Compute Shaders • Native Rust Kernel • Win32 Named Pipes
Direct3D 11 Compute
DXGI Desktop Duplication
HLSL CS 5.0
Rust 2024 (vactd)
Bilateral Edge Filter
Sobel Gradient Tensor
Connected Component Labeling
Win32 Named Pipes

Integrate

1// crates/vactd/src/main.rs — VACT Direct3D 11 Hardware Perception Daemon
2"syn-k">use vactd::compute::{DxgiDuplicator, ShaderPipeline};
3"syn-k">use vactd::ipc::PipeServer;
4"syn-k">use std::sync::Arc;
5
6#[tokio::main]
7"syn-k">async "syn-k">fn main() -> Result<(), Box<dyn std::error::Error>> {
8    println!("⚡ Initializing VACT 1.0 Kernel (Direct3D 11 Hardware Acceleration)");
9    
10    // Acquire zero-copy desktop duplication context (display 0)
11    "syn-k">let duplicator = Arc::new(DxgiDuplicator::new(0)?);
12    "syn-k">let pipeline = ShaderPipeline::new(&duplicator.device)?;
13
14    // Bind local Win32 named pipe for sub-5ms agent telemetry
15    "syn-k">let server = PipeServer::bind(r"\\.\pipe\VACT")?;
16    println!("✓ Telemetry pipe active at \\.\pipe\VACT (60 FPS stream)");
17
18    // Run 60 FPS perception loop with Bilateral Filter + CCL reduction
19    server.run_event_loop(duplicator, pipeline)."syn-k">await
20}
View repository on GitHub (fy2ne/VACT) Latency: <4.8ms • Payload: ~120 tokens
Autonomous Perception Engine

Engineered for Autonomous Model Ingestion

VACT structures screen perception directly into clean, deterministic vector JSON deltas. AI models receive exact bounding coordinates without pixel hallucinations or multi-megabyte payloads.

LIVE WIN32 NAMED PIPE STREAM: \\.\pipe\VACT ● Monotonic 60 FPS Stream 
{
  "protocol": "VACT/1.0",
  "sequence": 84920,
  "ack_sequence": 84919,
  "timestamp_ns": 1757089421092834,
  "tree_state": "DIFF",
  "frame_metrics": {
    "dt_ms": 4.82,
    "fps": 60.1,
    "surface": "IDXGIOutputDuplication"
  },
  "mutations": [
    {
      "op": "UPDATE",
      "id": "btn_deploy_worker",
      "role": "button",
      "bbox": [880.0, 420.0, 1060.0, 462.0],
      "center": [970.0, 441.0],
      "label": "Deploy Production Worker",
      "confidence": 0.9987,
      "hash": "0x7a89b01c"
    }
  ],
  "author": "fy2ne (https://fy2ne.me)"
}

Architected for speed, not screenshots

Zero-copy DXGI capture

Acquires full desktop display framebuffers directly into GPU VRAM using IDXGIOutputDuplication in under 0.8ms without CPU copying.

Bilateral range filtering

Executes photometric and spatial Gaussian kernels in compute shaders to eliminate video noise while preserving sharp font and vector UI boundaries.

Sobel gradient tensors

Convolves 3×3 spatial derivative matrices across 64-thread workgroups in Direct3D 11 to extract directional edge magnitudes at 1.1ms.

Connected component labeling

Performs two-pass contiguous pixel reduction on GPU compute cores to cluster elements into discrete, normalized interactive bounding boxes.

Monotonic differential DAG

Streams strictly incremental mutations over the IPC wire instead of re-transmitting static screen real estate, dropping network bandwidth by 99.8%.

Sub-pixel target precision

Provides mathematically grounded centroids [cx, cy] so agents execute mouse actions without visual coordinate drift or hallucinations.

99.8% token cost reduction

Replaces high-resolution bitmap token consumption (2,200 tokens per step) with ultra-lightweight differential vectors (80–180 tokens per step).

Sub-5ms end-to-end latency

Delivers perceptual telemetry at 60 FPS with an average 4.8ms frame time, unlocking genuine real-time reactive desktop interaction.

Air-gapped Win32 IPC pipe

Operates entirely over local OS named pipes \\.\pipe\VACT with zero network exposure, zero telemetry leaks, and zero external dependencies.

Mathematical Formulation

Deterministic equations executed directly on Direct3D 11 GPU compute hardware for edge-preserving bilateral smoothing, Sobel gradient tensor magnitude, and Connected Component Labeling geometry.

Bilateral Range Filter HLSL CS 5.0
If(x)=1Wp∑xi∈ΩI(xi)exp⁡(−∥xi−x∥22σs2)exp⁡(−∥I(xi)−I(x)∥22σr2)I_f(\mathbf{x}) = \frac{1}{W_p} \sum_{\mathbf{x}_i \in \Omega} I(\mathbf{x}_i) \exp\left(-\frac{\|\mathbf{x}_i - \mathbf{x}\|^2}{2\sigma_s^2}\right) \exp\left(-\frac{\|I(\mathbf{x}_i) - I(\mathbf{x})\|^2}{2\sigma_r^2}\right)

Preserves high-contrast text and UI borders while filtering photometric noise over spatial neighborhood $\Omega$ with spatial variance $\sigma_s$ and radiometric variance $\sigma_r$.

Sobel Gradient Tensor ||∇I|| Magnitude
∥∇I(x,y)∥=(Gx∗I)2+(Gy∗I)2,Gx=[−101−202−101],Gy=[−1−2−1000121]\|\nabla I(x, y)\| = \sqrt{(G_x * I)^2 + (G_y * I)^2}, \quad G_x = \begin{bmatrix}-1 & 0 & 1 \\ -2 & 0 & 2 \\ -1 & 0 & 1\end{bmatrix}, \quad G_y = \begin{bmatrix}-1 & -2 & -1 \\ 0 & 0 & 0 \\ 1 & 2 & 1\end{bmatrix}

Evaluates directional derivative convolution matrices $G_x$ and $G_y$ across 16×16 workgroups to isolate high-energy UI element boundaries.

Bounding Box Reduction CCL Geometry
Bk=[min⁡p∈Ckxp,  min⁡p∈Ckyp,  max⁡p∈Ckxp,  max⁡p∈Ckyp],ck=(xmin⁡+xmax⁡2,  ymin⁡+ymax⁡2)\mathcal{B}_k = \left[ \min_{p \in \mathcal{C}_k} x_p, \; \min_{p \in \mathcal{C}_k} y_p, \; \max_{p \in \mathcal{C}_k} x_p, \; \max_{p \in \mathcal{C}_k} y_p \right], \quad \mathbf{c}_k = \left( \frac{x_{\min} + x_{\max}}{2}, \; \frac{y_{\min} + y_{\max}}{2} \right)

Parallel reduction step gathering minimum and maximum extents for connected component label $\mathcal{C}_k$, calculating sub-pixel centroids $\mathbf{c}_k$.

Repository Structure

The official codebase is maintained on GitHub under fy2ne/VACT.

fy2ne / VACT MIT License
crates/vactd Windows DXGI Duplication & Direct3D 11 Compute Shader Daemon
crates/vact-protocol Zero-copy binary and JSON wire serialization primitives
shaders/ Direct3D 11 HLSL compute shaders (bilateral, sobel, ccl)
website/ Machine specification portal and documentation (vact.fy2ne.me)