Open AI Models Explained

Models

Flagship models

Models overview

Context window

• Input tokens (inputs you include in the messages array with chat completions)

• Output tokens (tokens generated in response to your prompt)

• Reasoning tokens (used by the model to plan a response)

Model ID aliases and snapshots

Current model aliases

GPT-4o

GPT-4o mini

• o1: reasoning model designed to solve hard problems across domains

• o1-mini: fast and affordable reasoning model for specialized tasks

GPT-4 Turbo and GPT-4

GPT-3.5 Turbo

DALL·E

TTS

Whisper

Embeddings

Moderation

GPT base

How we use your data

Default usage policies by endpoint

• Image inputs via the o1, gpt-4o, gpt-4o-mini, chatgpt-4o-latest, or gpt-4-turbo models (or previously gpt-4-vision-preview) are not eligible for zero retention.

• Audio outputs are stored for 1 hour to enable multi-turn conversations, and are not currently eligible for zero retention.

• When Structured Outputs is enabled, schemas provided (either as the response_format or in the function definition) are not eligible for zero retention, though the completions themselves are.

• When using Stored Completions via the store: true option in the API, those completions are stored for 30 days. Completions are stored in an unfiltered form after an API response, so please avoid storing completions that contain sensitive data.

• Objects related to the Assistants API are deleted from our servers 30 days after you delete them via the API or the dashboard. Objects that are not deleted via the API or dashboard are retained indefinitely.

• Evaluation data: When you create an evaluation, the data related to that evaluation is deleted from our servers 30 days after you delete it via the dashboard. Evaluation data that is not deleted via the dashboard is retained indefinitely.

Model endpoint compatibility

Text generation

Quickstart

Choosing a model

• A large model like gpt-4o offers a very high level of intelligence and strong performance, with higher cost per token.

• A small model like gpt-4o-mini offers intelligence not quite on the level of the larger model, but it's faster and less expensive per token.

• A reasoning model like the o1 family of models is slower to return a result, and uses more tokens to "think," but is capable of advanced reasoning, coding, and multi-step planning.

Building prompts

Messages and roles

Giving the model additional data to use for generation

Conversations and context

Managing context for text generation

• Output tokens are the tokens generated by a model in response to a prompt. Each model has different limits for output tokens. For example, gpt-4o-2024-08-06 can generate a maximum of 16,384 output tokens.

• A context window describes the total tokens that can be used for both input and output tokens (and for some models, reasoning tokens). Compare the context window limits of our models. For example, gpt-4o-2024-08-06 has a total context window of 128k tokens.

Optimizing model outputs

Vision

Quickstart

Uploading Base64 encoded images

Multiple image inputs

Low or high fidelity image understanding

• low enables the "low res" mode. The model receives a low-resolution 512px x 512px version of the image. It represents the image with a budget of 85 tokens. This allows the API to return faster responses and consume fewer input tokens for use cases that do not require high detail.

• high enables "high res" mode, which first lets the model see the low-resolution image (using 85 tokens) and then creates detailed crops using 170 tokens for each 512px x 512px tile.

Managing images

Image size guidelines

Limitations

• Medical images: The model is not suitable for interpreting specialized medical images like CT scans and shouldn't be used for medical advice.

• Non-English: The model may not perform optimally when handling images with text of non-Latin alphabets, such as Japanese or Korean.

• Small text: Enlarge text within the image to improve readability, but avoid cropping important details.

• Rotation: The model may misinterpret rotated or upside-down text and images.

• Visual elements: The model may struggle to understand graphs or text where colors or styles—like solid, dashed, or dotted lines—vary.

• Spatial reasoning: The model struggles with tasks requiring precise spatial localization, such as identifying chess positions.

• Accuracy: The model may generate incorrect descriptions or captions in certain scenarios.

• Image shape: The model struggles with panoramic and fisheye images.

• Metadata and resizing: The model doesn't process original file names or metadata, and images are resized before analysis, affecting their original dimensions.

• Counting: The model may give approximate counts for objects in images.

• CAPTCHAS: For safety reasons, our system blocks the submission of CAPTCHAs.

Calculating costs

Low res cost

High res cost

• Scale to fit within a 2048px x 2048px square, maintaining original aspect ratio

• Scale so that the image's shortest side is 768px long

• Count the number of 512px squares in the image—each square costs 170 tokens

• Add 85 tokens to the total

Cost calculation examples

• A 1024 x 1024 square image in detail: high mode costs 765 tokens
• 1024 is less than 2048, so there is no initial resize.
• The shortest side is 1024, so we scale the image down to 768 x 768.
• 4 512px square tiles are needed to represent the image, so the final token cost is 170 * 4 + 85 = 765.

• A 2048 x 4096 image in detail: high mode costs 1105 tokens
• We scale down the image to 1024 x 2048 to fit within the 2048 square.
• The shortest side is 1024, so we further scale down to 768 x 1536.
• 6 512px tiles are needed, so the final token cost is 170 * 6 + 85 = 1105.

• A 4096 x 8192 image in detail: low most costs 85 tokens
• Regardless of input size, low detail images are a fixed cost.

FAQ

Can I fine-tune the image capabilities in gpt-4?

Can I use gpt-4 to generate images?

What type of files can I upload?

Is there a limit to the size of the image I can upload?

Can I delete an image I uploaded?

Where can I learn more about the considerations of GPT-4 with Vision?

How do rate limits for GPT-4 with Vision work?

Can GPT-4 with Vision understand image metadata?

What happens if my image is unclear?