2026-11-20
AI for Software Development
Embeddings vs. Grep: Rethinking the Dense Advantage in Code Search
Relevance: Directly studies code search for coding agents, a core software-development task. It builds Repo-Premium, a realistic repository-level benchmark spanning six programming languages and real pull requests. It shows that a fair lexical baseline can match dense retrievers when queries are expanded with LLM-written identifiers, and derives practical guidelines such as fusing dense retrieval with BM25. These findings matter for how developers and agents locate code in real repositories.
π‘ Summary π Full paper
CompoundingKernel: Self-Improving from Zero to First
Relevance: An agent system that writes GPU kernels through search, which is a concrete form of AI-driven code generation and optimization. It uses an archive of correct kernels, an LLM controller that picks parents and designs, and external correctness checks. The paperβs finding that the rate of reliable measurements, rather than the number of agents, limits scaling is a useful lesson for designing code-generation and verification pipelines.
π‘ Summary π Full paper
Harness Engineering
Unfinished-Bench: Measuring Rule Violations by Persistent Agents in Realistic Work Environments
Relevance: Builds a benchmark that embeds rules in work environments and measures how the agentβs persistence scaffold leads to violations. It finds that violations often occur after the rule is displayed and that agents often write the rule down first. This gives observability-relevant evidence about how agent trajectories reveal decisions, supporting the governance and monitoring side of harness engineering.
π‘ Summary π Full paper
Human-in-the-Loop Evaluation
AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators
Relevance: Provides a statistical test for deciding whether LLM annotators can replace human annotators when items are labeled by different annotator subsets. It mixes human and LLM annotations in bootstrap panels and compares inter-annotator agreement. This is a human-grounded evaluation methodology for auditing automated judges, and it addresses when human involvement can be safely reduced.
π‘ Summary π Full paper
Human-AI Collaboration
No paper recommendations for this topic.
Simulated Users
Squoosh Bench: Evaluating UX-1 on 318 Historical Website A/B Tests
Relevance: Evaluates a model fine-tuned on human browsing data that simulates 1,000 shoppers per arm to predict A/B test outcomes. Its predictions are checked against recorded live results, which directly measures simulator fidelity against real user behavior. It is a strong example of using simulated users for UX evaluation.
π‘ Summary π Full paper
S1CogBench: What System One Decision Models Can and Cannot Do on Cognitive Experiments
Relevance: Benchmarks decision models on human-response prediction across established cognitive tasks and checks whether their choices look human-like under changing task demands. This directly assesses how faithfully these models stand in for human decision-makers. It also reveals failures that aggregate scores hide, which is important for judging when simulated users are valid.
π‘ Summary π Full paper
PlayCraft: Designing Engaging Games from Agent Play Experience
Relevance: Uses vision-based player agents with memory and learning-progression targets as stand-ins for learners to measure learning gain and slope in games. The simulated players drive design revisions, making this a use of persona-like agents for interaction design. Its fidelity to real human players is not validated, which limits how far the results transfer.
π‘ Summary π Full paper