2025-06-20
AI for Software Development
TestCase-Eval: Can LLMs Generate High-Quality Test Cases for Algorithm Problems? A Systematic Evaluation of Fault Coverage and Exposure
Relevance: This paper introduces TestCase-Eval, a benchmark to systematically evaluate Large Language Models (LLMs) in generating high-quality test cases for algorithm problems. It assesses two crucial aspects: fault coverage (how well test sets probe diverse scenarios) and fault exposure (whether LLMs can craft inputs to reveal specific code bugs). This directly impacts AI’s role in software quality assurance, offering insights into LLMs’ utility in automating critical development tasks like testing. From an HCI perspective, understanding LLMs’ capabilities in this domain helps shape the design of developer tools that effectively integrate AI-generated tests, improving trust and efficiency.
💡 Summary 📄 Full paper
Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers
Relevance: This paper proposes a method to optimize LLM training protocols to improve performance and controllability on ‘long-tail’ (rare or underrepresented) use cases. By using ‘training-time markers,’ it allows for explicit control over generation attributes at inference time. In software development, developers often encounter unique or specific scenarios that general models struggle with. Providing ‘control levers’ to target these edge cases, as this paper suggests, greatly enhances the utility and customizability of AI tools for developers, allowing them to better adapt AI assistance to their particular needs and improve output quality in challenging situations.
💡 Summary 📄 Full paper
AI Agents
Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligence
Relevance: This paper introduces a novel paradigm for AI agents that integrate physical world interaction (embodiment) with web-scale reasoning. It develops a unified simulation platform and a benchmark for tasks requiring coordinated intelligence across physical and digital realms (e.g., cooking from online recipes). This work directly impacts HCI by defining a new frontier for how users might interact with highly capable, integrated AI agents that operate fluidly between digital and physical spaces. It highlights the challenges in designing interfaces and interaction modalities for such advanced, general-purpose agents.
💡 Summary 📄 Full paper
OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents
Relevance: OS-Harm is a new benchmark designed to measure the safety of LLM-based computer-use agents that interact directly with graphical user interfaces. It covers various harm categories like deliberate misuse, prompt injection, and model misbehavior. From an HCI perspective, evaluating and ensuring the safety of such autonomous agents is paramount for their widespread adoption and user trust. This research provides a crucial tool for assessing how reliably these agents operate in real-world computing environments, directly informing the design of safer, more robust human-AI interaction patterns and mitigating potential risks.
💡 Summary 📄 Full paper
LLM Evaluation Methods
Alignment Quality Index (AQI) : Beyond Refusals: AQI as an Intrinsic Alignment Diagnostic via Latent Geometry, Cluster Divergence, and Layer wise Pooled Representations
Relevance: This paper introduces the Alignment Quality Index (AQI), a novel geometric and prompt-invariant metric to intrinsically assess LLM alignment by analyzing latent space activations. Unlike behavioral proxies (e.g., refusal rates), AQI can detect hidden misalignments and jailbreak risks, even when outputs appear compliant, and serves as an early warning for ‘alignment faking.’ From an HCI standpoint, this is critical for building user trust and ensuring ethical AI deployment, especially in high-stakes domains. Robust, intrinsic alignment evaluation is fundamental to developing AI systems that reliably reflect human values.
💡 Summary 📄 Full paper
MultiFinBen: A Multilingual, Multimodal, and Difficulty-Aware Benchmark for Financial LLM Evaluation
Relevance: This work introduces MultiFinBen, the first comprehensive benchmark for financial LLMs that is multilingual, multimodal, and difficulty-aware. It includes novel tasks requiring complex reasoning over mixed-language inputs and visual-text financial documents. From an HCI perspective, evaluating LLMs in such complex, real-world, and domain-specific scenarios is essential for understanding their practical utility and limitations. The findings expose significant performance gaps, highlighting the need for better models and potentially new interaction paradigms to enable effective human-AI collaboration in specialized, multimodal financial tasks.
💡 Summary 📄 Full paper
AssertBench: A Benchmark for Evaluating Self-Assertion in Large Language Models
Relevance: AssertBench is a benchmark designed to evaluate LLMs’ self-assertion capability – their ability to maintain consistent truth evaluation even when presented with contradictory user assertions. It explores how directional framing by users influences model agreement. This directly addresses an HCI concern regarding user trust and model reliability. If LLMs are easily swayed by user framing, it undermines their perceived objectivity and trustworthiness. AssertBench provides a valuable tool for understanding and improving LLMs’ robustness to human input, which is crucial for designing reliable and consistent human-AI conversational interfaces.
💡 Summary 📄 Full paper
Reinforcement Learning
Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
Relevance: This paper investigates Reinforcement Learning with Verifiable Rewards (RLVR) for improving LLM reasoning, introducing CoT-Pass@K as a more precise evaluation metric. It demonstrates that RLVR can genuinely incentivize the generalization of correct reasoning paths. From an HCI viewpoint, understanding how RL training leads to ‘correct reasoning’ is vital for interpreting agent behaviors and building trust. If RL ensures logical integrity and generalizable reasoning, it makes the outputs of RL-trained LLMs more transparent and reliable for human users, facilitating better human-agent collaboration and problem-solving.
💡 Summary 📄 Full paper
Efficient Medical VIE via Reinforcement Learning
Relevance: This research applies Reinforcement Learning with Verifiable Rewards (RLVR) to Visual Information Extraction (VIE) in medical contexts, achieving state-of-the-art performance with limited data. It uses a balanced precision-recall reward mechanism to reduce hallucinations and ensure field coverage. For HCI, this paper showcases how RL can be applied in sensitive, high-stakes domains like healthcare, where accuracy and reliability are paramount. The emphasis on ‘verifiable rewards’ and ‘reasoning during training and inference’ directly relates to the human need to trust and understand AI decisions in critical applications, improving human-agent collaboration in medical tasks.
💡 Summary 📄 Full paper
Explainable AI
No paper recommendations for this topic.