Ensuring Consistent JSON Output from LLMs on Amazon Bedrock
Learn practical techniques to obtain structured outputs in JSON using Amazon Bedrock
Arya Subramanyam
Amazon Employee
Published May 2, 2025
As organizations increasingly leverage large language models (LLMs) for automation and data extraction, ensuring the output is in a predictable, structured format-like JSON-is critical for downstream processing. In this post, I’ll walk through the main strategies for enforcing output formats from LLMs, with a focus on how to implement these techniques using AWS services such as Amazon Bedrock, Step Functions, and supporting open-source libraries. I’ll also share practical tips and resources to help you get started.
LLMs are powerful, but by default, their outputs are unstructured text. For most enterprise use cases-data pipelines, customer feedback analysis, or workflow automation you need reliable, machine-readable formats (typically JSON). Structured output enables you to:
- Automate data ingestion and transformation
- Validate and enforce data schemas
- Build robust, error-tolerant workflows
Crafting precise prompts is the foundational and most accessible method for guiding LLMs toward structured outputs, but optimal results come from combining clear instructions with thoughtful inference parameter tuning. Strategic prompting acts as a "soft constraint," steering the model toward your desired format by providing explicit structure, examples, and delimiters, while careful adjustment of inference parameters-such as lowering temperature to reduce randomness-further increases output consistency and reliability. For AWS users, this blend of prompt engineering and inference tuning forms the basis for more advanced techniques and remains compatible across all Bedrock-supported models, ensuring robust and predictable results for a wide range of applications
- Specify the Format: Directly instruct the model to respond in JSON and outline the required keys and structure. Example:
- Provide Examples: Include a sample response in your prompt to train the model on the desired schema via One-Shot or Few-Shot Prompting. Example:
- Use Delimiters: Structure prompts with clear separators (e.g., XML tags for Claude) to isolate instructions from input data. Example:
- Control Inference Parameters: Fine-tune settings like
temperature,top_p, andmax_tokensto reduce randomness and ensure the model consistently outputs well-structured JSON.
- Iterate and Validate: Refine prompts based on output quality and add post-processing validation steps.
For mission-critical applications requiring strict formatting, some LLMs offer specialized JSON generation modes. Anthropic Claude on Amazon Bedrock exemplifies this capability, using a technique called response prefill to bypass conversational templates and force JSON compliance. Unlike basic prompting, JSON mode treats the model as a structured data generator rather than a chat assistant, significantly reducing malformed outputs. While Claude doesn’t have a formal “JSON Mode” toggle like some other models, you can reliably enforce structured output using tool schemas and careful prompting. Read more about Claude’s JSON mode here.
Example:
Output:
Limitations:
- JSON mode doesn’t guarantee schema compliance, always validate outputs.
- Token limits may truncate complex JSON structures.
Amazon Bedrock's Converse API elevates structured output generation through tool use-a paradigm where the LLM acts as an API caller bound by predefined schemas. By defining JSON templates as virtual "tools," you transform the model into a deterministic function that returns validated objects. This AWS-native approach is particularly valuable for enterprises needing to integrate LLMs with existing APIs and data pipelines while maintaining strict type safety.
- Define a tool with a JSON Schema for your desired output.
- Pass the tool definition and your prompt to the Converse API.
- Extract the JSON from the model’s tool use response.
This approach is ideal for use cases where you want the LLM to act as a function caller or to ensure strict adherence to a schema. Read more about this on this page, Call a tool with the Converse API.
Define a JSON schema and let the LLM act as a function caller:
Read more about how to use this feature here: Generating JSON with the Amazon Bedrock Converse API.
Even with the best prompting and libraries, LLMs are non-deterministic-occasional format errors will occur. For production-grade reliability on AWS, combine automated validation and retries using Step Functions and Amazon Bedrock Flows:
- AWS Step Functions: Orchestrate multi-step validation pipelines, such as invoking a Claude model, running a Lambda for JSON schema validation, and automatically retrying or escalating on failure. This ensures each step-generation, validation, and error handling-is managed and auditable.
- Bedrock Prompt Flow: Use Prompt Flow to design, test, and deploy complex, multi-step generative workflows directly within Bedrock. Prompt Flow allows you to chain together model invocations, conditional logic, and integrations with other AWS services. For example, you can build a flow that takes structured input, invokes a Claude prompt, validates the output, and routes the result or error for further action-all visually or programmatically.
- Automated Retries for Malformed Outputs: Both Step Functions and Prompt Flow support automated retries and conditional branching, so if the output fails validation, the workflow can re-invoke the model with adjusted parameters or escalate to a human review.
Here is a sample workflow you can take inspiration from:
When working beyond Bedrock's managed services-or when dealing with highly custom models-open-source libraries bridge the gap between LLM flexibility and production reliability. These tools offer model-agnostic validation, schema enforcement, workflow orchestration, and evaluation capabilities that complement AWS services and are especially useful in hybrid or self-hosted architectures:
- LangChain: Integrates directly with Amazon Bedrock, offering output parsers (like the JSON Output parser and StructuredOutputParser) to extract and validate JSON from model responses.
- Jsonformer: Enables structured output enforcement by constraining LLM generations to match a specified JSON schema. Jsonformer is also available through LangChain.
- Pydantic: Pydantic JSON Schema is frequently used in Python workflows to define, validate, and parse JSON schemas from LLM outputs. Pydantic allows automatic creation and customization of JSON schemas from models.
- lm-format-enforcer: This project enforces JSON by filtering the tokens that the language model is allowed to generate at every timestep, thus ensuring that the output format is respected, while minimizing the limitations on the language model.
- Start with strong prompt engineering: Clearly instruct the model to output JSON, provide examples, and specify the schema.
- Leverage Bedrock tool use: Where possible, use the Converse API’s tool use capability for schema enforcement.
- Add post-processing validation: Use libraries or AWS services to validate and retry on errors.
- Combine techniques: For maximum reliability, combine prompt engineering, tool use, and workflow validation.
“Using a combination of techniques together is key for 100% reliability. Keep in mind, LLMs are non-deterministic by nature.”
Structured LLM outputs are achievable through a layered approach: precise prompting, model-specific JSON modes, AWS-native tooling, and validation workflows. By leveraging Amazon Bedrock’s Converse API for schema enforcement and Step Functions for error handling, teams can build production-grade pipelines that integrate seamlessly with existing AWS infrastructure.
For complex use cases, combine AWS services with open-source libraries like LangChain to balance flexibility and reliability. Remember-LLMs are non-deterministic, so always validate and retry!
Need help implementing this? Ask questions in the comments.
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.