Speaker
Description
Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing tasks, but their predictions may be accompanied by substantial uncertainty, particularly when applied to unfamiliar or ambiguous inputs. This paper develops a distribution-free framework for uncertainty quantification in black-box LLMs using conformal prediction. The proposed approach constructs prediction sets with finite-sample coverage guarantees without requiring knowledge of the underlying data distribution or access to the internal architecture and parameters of the LLM. By using calibration data and suitable nonconformity scores derived from model outputs, the framework provides statistically valid measures of predictive uncertainty while treating the LLM as a black-box system. We investigate the behavior of the proposed method under different levels of model confidence, sample sizes, and distributional shifts. Experimental results demonstrate that conformal prediction can provide reliable coverage while maintaining informative and practically useful prediction sets. The proposed framework offers a general and model-agnostic approach for improving the reliability, interpretability, and uncertainty awareness of black-box LLM applications.