2026-10-30

AI for Software Development

Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

Relevance: Evaluates coding agents on real user coding sessions, turning multimodal requests (image, audio, video, documents) into working application revisions. It covers localized edits and coordinated repository changes, directly addressing AI-assisted code modification in realistic development workflows.

💡 Summary 📄 Full paper

VibeGame: Prompt-to-Game Development with AI-Native Engine and Self-Evolving Adversarial Agent Team

Relevance: Presents an end-to-end LLM-driven software development framework for building playable 2D games. It uses an AI-native engine with schema-validated project state, multi-agent implementation and review loops, and reusable skeletons distilled from past projects.

💡 Summary 📄 Full paper

Scaling Lean Theorem Proving for Real-World Code Verification

Relevance: Applies LLM agents to verification of real Rust implementations by translating code into Lean and proving or refuting generated specifications. This supports code correctness assurance and bug discovery for software engineering.

💡 Summary 📄 Full paper

Harness Engineering

Matched Capabilities, Different Behaviors: A Controlled Study of MCP and CLI Interfaces for Language Agents

Relevance: Shows that the interface layer (MCP vs. CLI) shapes agent behavior, failures, and recovery, and proposes DART to repair interface-specific realization. This is a direct study of how harness components affect agent behavior and how they can be inspected and improved.

💡 Summary 📄 Full paper

Evaluating DeepSeek V4.1 Flash-1017 for loss of control and harmful manipulation risks

Relevance: Argues that final outputs are insufficient for detecting loss-of-control behavior and calls for process-level observability across reasoning traces and action logs. It also recommends incremental safety evaluation of harnessed deployments.

📄 Full paper

Human-in-the-Loop Evaluation

No paper recommendations for this topic.

Human-AI Collaboration

RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents

Relevance: Evaluates CLI agents under a scripted user who gives progressive information, corrections, and authorization decisions. It measures steerability, trust calibration, and safety boundary management, which are core aspects of human-agent teaming.

💡 Summary 📄 Full paper

Simulated Users

PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas

Relevance: A persona agent plays 44 user personas that rewrite instructions and answer the agent’s questions, simulating users in multi-turn settings. It studies how simulated user interaction changes computer-use agent outcomes and satisfaction.

💡 Summary 📄 Full paper

RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents

Relevance: Its REALCLI-COLLAB mode relies on a scripted user that delivers progressive corrections and constraint revisions. This simulated user supports evaluation of collaboration quality in multi-turn agent interactions.

💡 Summary 📄 Full paper