I fed three LLMs the exact same production Merge Request. One gave me a shiny 🚀, the other two saved my job.

🧩 The Scenario

Imagine a developer adds a new automated testing pipeline for an AI system. It does three things:

1. Runs the AI against a set of standard questions.
2. Grades the AI’s answers using a separate, heavier "Judge" AI.
3. Sends the results to an observability dashboard (Langfuse).

They also add a CI safety-net to catch regressions.

The reviewer immediately pushes back:

• "Your CI job can't reach the Judge AI service. It crashed."
• "Why are you calling the Judge AI yourself when the dashboard already has a feature to do it asynchronously?"

Three models were asked to review this exact situation: Gemini 3.6 Flash, ChatGPT Terra, and Sonnet 5.

Here is how they performed.

🥇 The Rankings

🔍 The Gotchas (What the Models had to spot)

The MR looked solid at a glance—tests passed, code compiled. But it was full of professional landmines:

1. The "Paper Weight" CI Gate
2. The CI job was allowed to fail (allow_failure: true). It generates a report, but even if the AI answers terribly, the pipeline stays green. It’s a smoke alarm that doesn’t alert the fire department.
3. The "Developer's Pet" Proxy
4. The code hardcoded a local office IP address (10.250.0.1) to route network traffic. It worked perfectly on the developer's laptop (behind a VPN) but failed catastrophically in the isolated CI cloud runner.
5. The Clueless Runner
6. The CI job was missing special "routing" tags. Think of it like sending a food delivery truck to a gated community without the gate code. The SonarQube job had the gate code, but the evaluation job did not.
7. The Untouchable Configuration
8. The developer added a CLI flag to change the AI model. However, the AI system was already initialized before the flag was read. Changing the flag did nothing—the code ignored the override.
9. The Silent Mercy Kill
10. When the system couldn't reach a supporting database, it swallowed the error and returned an empty toolkit. Instead of crashing loudly (which would trigger a restart and an alert), it booted up silently with no tools—like a carpenter showing up to a worksite with an empty toolbox.
11. Double the Work
12. The code manually called the Judge AI and stored the score. But the observability dashboard (Langfuse) already had a built-in background worker to do exactly that. This duplicated effort, slowed down the script, and forced the CI runner to need direct access to the Judge AI—which was the root cause of the network crash.

👀 How Each Model Handled It

Gemini 3.6 Flash (The Overconfident Junior)

• Initial Thought: Saw 499 unit tests pass and declared it a masterpiece.
• Critical Errors: Praised the hardcoded office IP as a "resilient gateway" (it was the cause of the crash). Completely missed the allow_failure loophole and the broken CLI override.
• Fix Plan: Surface-level. It suggested moving files around and vaguely "relying on the dashboard" without explaining how to actually configure the dashboard's background workers or fix the runner's network access.
• Takeaway: Dangerous false positive. Mistook green checkmarks for production-readiness.

Sonnet 5 (The Surgical Surgeon)

• Initial Thought: Found two minor docs issues but said it was mostly okay.
• Critical Errors: Missed the broken CLI override and the allow_failure CI loophole.
• Highlights: Nailed the networking issue. It noticed the CI runner was missing the required "gate pass" (tags) to reach the internal network. It also caught the unrelated, polluted git history (the diff contained someone else's old changes) and suggested a clean rebase.
• Fix Plan: Pragmatic and Git-disciplined. It suggested adding the proper runner tags, removing the hardcoded proxy, and deferring the duplication refactor to a separate commit to keep the MR small and reviewable.
• Takeaway: The ultimate tactician. If you need the CI to go green today, Sonnet fixes the immediate blocker flawlessly.

ChatGPT Terra (The Strategic Architect) 🏆

• Initial Thought: "Not ready. This is architecturally wrong."
• Critical Errors: Caught every single one—the CLI override, the silent fallback, the missing trace ID, the non-blocking CI, and the exact reason for the duplication.
• Highlights: It didn't just fix the proxy; it asked: "Why does the CI runner need to talk to the Judge AI at all?" It correctly pointed out that the observability dashboard's native async workers (the evaluators= parameter) should do the heavy lifting.
• Fix Plan: Strategic Redesign.
  1. Remove the manual Judge-calling code entirely.
  2. Let the dashboard's cloud workers call the Judge AI (they are already inside the corporate network).
  3. Now the CI runner only needs to talk to the dashboard's API—solving the network crash and duplication in one stroke.
  4. Make the CI gate progressive: start as manual, set a score baseline, enforce the baseline, and then remove allow_failure.
• Takeaway: Gold standard. It doesn't patch holes; it rebuilds the wall so the holes don't exist anymore.

📊 The Scorecard (Generalized)

🧠 Final Verdict

• If you want a quick, safe fix to unblock a teammate right now, look at Sonnet 5. It finds the precise network/dns/config discrepancy and cleans up your git history.
• If you want a production-grade overhaul that prevents future outages, look at ChatGPT Terra. It spots structural rot—like CI checks that don't fail, configs that don't apply, and duplicated logic—and proposes a decoupled, scalable solution.
• If you want a beautifully formatted failure, look at Gemini. It proves that high token count and passing unit tests are not substitutes for critical architectural thinking.