OSWorkerBench
100 long-horizon office workflows for computer-use agents, released with Tencent's UI-Mate
GUI agents are usually measured on short, self-contained tasks. Real office work is neither. OSWorkerBench is a benchmark of 100 long-horizon office workflows spanning 41 applications and 10 job families, released alongside UI-Mate, the foundation GUI agent from Tencent’s Hy Team (Tencent Hy Team, 2026).
Built during my research internship in Tencent’s Qingyun Program, where I was a core contributor to the technical report and owned task construction for the benchmark.
My contributions
A capability-grounded generation pipeline. Rather than writing tasks by hand and hoping for coverage, I designed a pipeline that generates tasks from an explicit map of application capabilities — so the benchmark’s spread across applications and job families is a property of the construction, not an accident.
Every task automatically scoreable. Each task is scored by weighted checkpoints on final application state, so grading needs no human in the loop and gives partial credit for partial progress on long workflows. Getting this right took iterative human-in-the-loop verification: pilot-agent rollouts to find tasks that were ambiguous or unreachable, and synthetic test cases to confirm each checkpoint fired when — and only when — it should.
Multimodal demonstrations as one-shot guidance. I recorded human demonstrations for the hard tasks. This makes OSWorkerBench the first computer-use benchmark to supply multimodal demonstrations as one-shot guidance, and the first to test procedural transfer: can an agent given a demonstration of a related but non-identical task carry the procedure across?
What the benchmark looks like
Panel (d) is the part I care about most. Grading a long office workflow is where benchmarks usually give up and fall back on a human reading transcripts. Scoring against weighted checkpoints on final application state keeps it automatic, and gives partial credit for partial progress instead of collapsing a twelve-step workflow into one pass/fail bit.
Links
Project page · Code · Technical report
