Add StructEval structured-output benchmark paper

README.md

@@ -125,6 +125,7 @@ For more video content, check out the [official JSON Schema YouTube channel](htt
 - (2025) [Blaze: Compiling JSON Schema for 10x Faster Validation](https://arxiv.org/abs/2503.02770?utm_source=awesome-jsonschema) - This paper introduces Blaze, a JSON Schema validator compiles complex schemas to an efficient representation in seconds to minutes, adding minimal overhead at build time. Blaze incorporates several unique optimizations to reduce the validation time by an average of approximately 10x compared existing validators on a variety of datasets. In some cases, Blaze achieves a reduction in validation time of multiple orders of magnitude compared to the next fastest validator. We also demonstrate that several popular validators produce incorrect results in some cases, while Blaze maintains strict adherence to the JSON Schema specification..
 - (2025) [Elimination of annotation dependencies in validation for Modern JSON Schema](https://arxiv.org/abs/2503.11288?utm_source=awesome-jsonschema) - This paper proves that the elimination of annotation dependent keywords cannot, in general, avoid an exponential increase of the schema dimension. We provide an algorithm to eliminate these keywords that, despite the theoretical lower bound, behaves quite well in practice, as we verify with an extensive set of experiments..
 - (2025) [JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models](https://arxiv.org/abs/2501.10868?utm_source=awesome-jsonschema) - This paper introduces JSONSchemaBench, a benchmark for constrained decoding comprising 10K real-world JSON schemas that encompass a wide range of constraints with varying complexity. We pair the benchmark with the existing official JSON Schema Test Suite and evaluate six state-of-the-art constrained decoding frameworks, including Guidance, Outlines, Llamacpp, XGrammar, OpenAI, and Gemini. Through extensive experiments, we gain insights into the capabilities and limitations of constrained decoding on structured generation with real-world JSON schemas..
+- (2025) [StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs](https://arxiv.org/abs/2505.20139?utm_source=awesome-jsonschema) - A comprehensive benchmark for structured-output generation and conversion across 18 non-renderable and renderable formats, with syntax, structural-fidelity, and visual-fidelity evaluation; the official implementation is available at https://github.com/TIGER-AI-Lab/StructEval.
 - (2024) [Validation of Modern JSON Schema: Formalization and Complexity](https://arxiv.org/abs/2307.10034?utm_source=awesome-jsonschema) - In this paper, we give the first formal description of Modern JSON Schema, which we consider a central contribution of the work that we present here. We then prove that its data validation problem is PSPACE-complete. We prove that the origin of the problem lies in dynamic references, and not in annotation-dependent validation. We study the schema and data complexities, showing that the problem is PSPACE-complete with respect to the schema size even with a fixed instance, but is in PTIME when the schema is fixed and only the instance size is allowed to vary. Finally, we run experiments that show that there are families of schemas where the difference in asymptotic complexity between dynamic and static references is extremely visible, even with small schemas..
 - (2023) [An Analysis of Defects in Public JSON Schemas](https://www.cri.minesparis.psl.eu/classement/doc/A-794.pdf?utm_source=awesome-jsonschema) - Analysis of common defects found in publicly available schemas leading to recommend changes to the spec.
 - (2023) [Comprehending Semantic Types in JSON Data with Graph Neural Networks](https://arxiv.org/abs/2307.12807?utm_source=awesome-jsonschema) - Graph neural networks for semantic type detection in JSON.

data.yaml

@@ -501,6 +501,12 @@
   type: paper
   summary: This paper introduces JSONSchemaBench, a benchmark for constrained decoding comprising 10K real-world JSON schemas that encompass a wide range of constraints with varying complexity. We pair the benchmark with the existing official JSON Schema Test Suite and evaluate six state-of-the-art constrained decoding frameworks, including Guidance, Outlines, Llamacpp, XGrammar, OpenAI, and Gemini. Through extensive experiments, we gain insights into the capabilities and limitations of constrained decoding on structured generation with real-world JSON schemas.
 
+- title: "StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs"
+  url: https://arxiv.org/abs/2505.20139
+  year: 2025
+  type: paper
+  summary: A comprehensive benchmark for structured-output generation and conversion across 18 non-renderable and renderable formats, with syntax, structural-fidelity, and visual-fidelity evaluation; the official implementation is available at https://github.com/TIGER-AI-Lab/StructEval
+
 - title: "Elimination of annotation dependencies in validation for Modern JSON Schema"
   url: https://arxiv.org/abs/2503.11288
   year: 2025