"OpenAI-compatible" is useful shorthand, but it should not be treated as a guarantee that two APIs behave identically.

An endpoint may accept an OpenAI SDK request while differing in supported endpoints, parameters, streaming behavior, tool calling, structured outputs, model capabilities or error handling.

For production integrations, compatibility is better treated as something to test in layers.

Here's a practical checklist.

1. Define exactly what you're testing

Before running tests, record:

Model:
Exact model ID/version:
Provider:
Base URL / endpoint:
Region:
API surface: Chat Completions / Responses / other
SDK and version:
Test date:

This matters because:

Model ≠ Provider ≠ Endpoint.

The same underlying model can behave differently depending on who serves it, which API surface is used and which version is deployed.

Likewise, support for one OpenAI-compatible endpoint does not automatically imply support for every OpenAI API.

2. Connectivity

Can the client actually reach the service?

Check:

• DNS resolution
• TLS handshake and certificate validation
• HTTP connectivity
• Base URL and path handling
• Redirect behavior
• Endpoint reachability

A TCP or HTTP connection only proves that the service is reachable. It doesn't prove authentication, protocol compatibility or model capability.

Be particularly careful with redirects when credentials are present. Clients should not blindly forward authorization credentials to a different origin.

3. Authentication

Does the endpoint accept the authentication mechanism your application expects?

Test at least:

• Valid credentials
• Missing credentials
• Invalid credentials
• Permission-restricted credentials, where applicable
• HTTP status and error response structure

Don't assume that every provider uses 401, 403 or provider-specific error codes in exactly the same way.

Record what was actually observed rather than inferring an internal provider reason from the status code alone.

And never log or publish complete API keys or authorization headers.

4. Model discovery

If the provider exposes a model-list endpoint:

• Does /v1/models or the documented equivalent work?
• Is the response structurally usable by your client?
• What exact model IDs are returned?
• Is the model you intend to use present?
• Can that model actually complete a request?

A successful /models response doesn't prove the listed model supports every feature you need.

And some providers may document models separately rather than expose model discovery exactly as OpenAI does.

5. Basic chat compatibility

Test the exact request shape your application uses.

Check:

• messages
• Supported roles
• Content structure
• Model ID
• Sampling parameters
• Maximum output controls
• Stop behavior, if required
• Response structure
• finish_reason
• Usage fields
• Provider-specific fields

Also test unsupported parameters.

There is an important difference between an endpoint that:

rejects an unsupported parameter

and one that:

accepts it but silently ignores it.

Both can appear "compatible" in a simple integration test, but they have very different implications for production behavior.

6. Streaming

A successful non-streaming request does not prove streaming compatibility.

Test the full stream lifecycle:

• Does the request actually use a streaming transport?
• Can your client parse the event framing correctly?
• Are content deltas valid?
• Are tool-call arguments streamed correctly, if applicable?
• Is usage available?
• How is successful completion signaled?
• What happens when the stream is interrupted?
• Can partial output be detected and handled safely?

Don't assume every OpenAI-compatible streaming endpoint uses exactly the same terminal event or chunk semantics.

"Some text arrived incrementally" is not enough to establish protocol compatibility.

7. Tool calling

For agent workloads, don't stop after confirming that the API accepts a tools field.

Test an actual tool-call round trip:

User request
→ model selects a tool
→ tool name and arguments are returned
→ your application executes the tool
→ the result is sent back to the model
→ the model continues correctly

Also test:

• Argument validity
• Required fields
• tool_choice, if your application depends on it
• Multiple or parallel tool calls, if required
• Tool-call IDs
• Tool-result message format
• Streaming tool arguments
• Behavior when no tool should be called
• Behavior when a tool fails

Receiving a tool_calls field is evidence of only one part of the workflow.

It does not prove that the full agent loop is compatible.

8. Structured output

Separate at least three cases:

Prompted JSON

The prompt asks the model to produce JSON.

JSON Mode

The API provides a mode such as:

response_format: {"type": "json_object"}

This should produce valid JSON, but it does not necessarily guarantee your application schema.

JSON Schema / Structured Outputs

The API accepts a schema and is expected to produce output conforming to that schema.

Test:

• Whether the requested mode is actually supported by the specific model
• Whether the response parses as JSON
• Whether it validates against the required schema
• Required and optional fields
• Types
• Nested structures
• Enums and constraints you depend on
• Truncated outputs
• Refusals or unsatisfiable constraints

Don't classify all of these as simply:

"Structured Output: supported."

A provider may support JSON Mode while not supporting strict schema-constrained output.

9. Context, parameters and limits

Compatibility problems often appear near limits rather than in hello-world requests.

Check:

• Context-window limits
• Maximum output length
• Long-input behavior
• Truncation behavior
• Unsupported parameter behavior
• Rate limits
• Concurrency limits
• Request-size limits
• Timeout behavior

Also test the parameters your application actually depends on.

A parameter being accepted by the server does not necessarily prove that it affects generation.

10. Errors and failure behavior

Deliberately exercise failure paths.

Useful cases include:

• 401
• 403
• 404
• 429
• 5xx
• Network timeout
• Invalid model
• Invalid parameter
• Context overflow
• Invalid JSON response
• Malformed stream
• Interrupted stream
• Invalid tool arguments
• Schema validation failure

Record both the HTTP status and the structured provider error when available.

Avoid hiding failures behind automatic retries during compatibility testing.

If the first request returns 503, a retry succeeds and the test reports only "PASS", you've lost important evidence about the endpoint's actual behavior.

11. Separate compatibility, capability and reliability

These are different questions.

Compatibility

Does the endpoint follow the protocol behavior your application expects?

Capability

Can this particular model perform the feature?

For example:

• Tool Calling
• Structured Output
• Vision
• Long context

Reliability

How consistently does the endpoint behave across repeated requests and over time?

An endpoint can therefore be:

reachable + chat-compatible + streaming-compatible + tool-calling unsupported

without being "broken."

Likewise, one successful tool call doesn't establish production reliability.

12. Make the result reproducible

For meaningful comparisons, record:

Model/version:
Provider:
Base URL / region:
API surface:
SDK/version:
Test date and window:
Parameters:
Workload/test case:
Number of trials:
Concurrency:
Success criteria:
Observed failures:
Latency / TTFT, if measured:
Usage, if reported:
Relevant sanitized evidence:

Avoid declaring a feature universally supported after one successful request.

A better claim is:

"This model/provider combination passed this test under these conditions."

That is narrower, but much more useful.

Discussion

For developers integrating Qwen, DeepSeek, GLM, Kimi, MiniMax, Doubao/Seed, Hunyuan/Hy, MiMo or other Chinese-developed models:

What OpenAI-compatibility issue has actually caused you trouble?

If possible, include:

Model:
Provider:
API surface:
SDK/version:
Observed behavior:
Expected behavior:

Please redact API keys, authorization headers, customer data and other sensitive information before posting.