In our experiments, adding CoT-derived groups reduces Brier error by 4.5–15% compared to topic-only multicalibration, while preserving similar topic-level calibration. 5/n
I've been thinking a lot about AI slop lately, and there are a few thoughts that I wanted to document, not because they are finalized and won’t change, but just in case others find them useful. 1/n
4. I don’t have a prescribed path ahead. But my path is: implement a “standards and expectations” guide for AI use for my lab, encourage frequent discussions with my students about the “proper” (in my view, at least) use of AI in research, ... 9/n
... pointing out (hopefully kindly) to authors when they have violated the norms and standards that I hold authors to, and / or even refusing to review papers that do so. 10/n
The discourse that informs our views is just as important as how we vote, but harder to architect well.
We headed to Boston on our summer Moonshots tour to meet the team building a platform for deliberation at national scale. It's designed to assemble better groups, nudge quiet voices, and surface topics that spark quality debate. More soon! laude.org/moonshots/one/deli…@ArielProcaccia@bakkermichiel@baileyflan@Arfung@lessig@ChrisRytting
Paper 2: Prediction-Powered Inference Across Many Tasks for AI Evaluation and Social Science Research
We develop methods for statistically valid inference from many related tasks, with many cheap proxy labels (via, e.g., an LM) and only a few high-quality labels per task. 🧵
Paper 1: Conformal Language Modeling via Posterior Sampling
How can we control hallucination risk in LLM generations without sacrificing the usefulness and coherence of the response?
In our new paper, we use conformal methods to calibrate the generation process itself. 🧵