LLMs Can Now Measure Human and AI Values

📄 Full Paper

💬 Ask

Large language models (LLMs) have emerged as both subjects and tools for measuring values. However, existing methods for measuring values in LLMs have limitations, such as relying on static questionnaires or dictionary-based approaches. A new paper, “Measuring Human and AI Values based on Generative Psychometrics with Large Language Models”, introduces a novel, data-driven approach called Generative Psychometrics for Values (GPV) that leverages the capabilities of LLMs for more comprehensive and accurate value measurement.

Text-Revealed Perceptions and GPV

GPV is rooted in the theory of text-revealed selective perceptions, which posits that individuals’ values influence their perceptions of the world. GPV uses LLMs to parse texts into value-laden perceptions and then measures the relative importance of each value based on these perceptions. This approach avoids the limitations of prior methods by enabling the measurement of values in open-ended, context-specific ways.

GPV’s Application to Human Values

The paper demonstrates the stability and validity of GPV in measuring human values using a large corpus of human-authored blog posts. The authors found that GPV:

• Outperforms prior psychological tools in measuring human values
• Demonstrates high stability, meaning that the same individuals exhibit consistent value tendencies across different scenarios.
• Shows strong construct validity, indicating that the measured values align with the theoretical structure of the value system.

GPV’s Application to AI Values

The paper also extends GPV to the measurement of values in LLMs. The authors find that:

• GPV effectively measures LLM values based on their free-form outputs, mitigating response biases seen in prior methods.
• Different value systems have different predictive power for LLM safety, with VSM proving more predictive than Schwartz’s value theory, suggesting the importance of considering alternative value systems in AI safety research.
• Specific values within VSM influence LLM safety, with values like Long Term Orientation positively contributing to safety while values like Masculinity negatively contributing, highlighting the importance of considering the specific impacts of values on safety.

Key Contributions of the GPV Paradigm

GPV represents a significant advance in value measurement, offering a flexible and powerful approach for both human and AI value assessment. Key contributions include:

• LLM-based value measurement paradigm grounded in text-revealed selective perceptions.
• Fine-tuned LLM model (ValueLlama) for accurate perception-level value measurement.
• Demonstration of GPV’s stability, validity, and superiority over prior psychological tools in measuring human values.
• Novel approach to LLM value measurement that leverages scalable and free-form outputs.
• Comparative analysis of measurement paradigms highlighting the predictive power of VSM for LLM safety.

Future Directions and Potential Impact

The development of GPV opens up exciting new possibilities for value measurement in both human and AI contexts. Future research could explore:

• Multilingual value measurement, as the language used can influence the values displayed by LLMs.
• The spectrum of LLM values, investigating how different prompts and training data impact LLM value systems.
• Developing value-aligned LLMs that can better understand and represent human values.

Overall, GPV represents a powerful tool for understanding and aligning values in both humans and AI. Its potential for advancing our understanding of value systems and for building more ethical and responsible AI systems is significant.

Chat about this paper

To chat about this paper, you'll need a free Gemini API key from Google AI Studio.

Your API key will be stored securely in your browser's local storage.