The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft
Summary
This workshop provides a deep dive into comparing the real-world performance of top-tier LLMs (GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, Mistral Large 3) across complex, multi-step applications. The core message is that standard benchmarks are insufficient; model behavior must be tested in specific scenarios. Key techniques demonstrated include robust Retrieval-Augmented Generation (RAG) grounding, efficient tool calling (including parallel calls), and advanced evaluation frameworks like Microsoft's ASSERT and DSPy, which help automate test set generation and prompt optimization for model portability.
Key takeaways
-
Model Differences are Scenario-Specific
8:58
Models exhibit significant differences in core capabilities (e.g., counting, groundedness, tool calling efficiency). For instance, GPT-5.5 often demonstrated superior performance in parallel tool calling and adherence to constraints compared to competitors.
-
RAG Grounding Requires Strict Prompting
13:58
Models, especially smaller ones, are prone to hallucinating information from their internal weights rather than strictly adhering to provided context. Prompt engineering is necessary to force models to admit when information is not source-provided.
-
Evaluation Must Move Beyond Simple Checks
While string checks are basic, robust evaluation requires advanced methods like LLM-as-a-Judge (which is slow/expensive) or using frameworks like ASSERT to generate comprehensive test sets from a high-level specification (YAML).
-
Prompt Optimization for Model Independence
DSPy uses genetic algorithms (Jeppa) to automatically write and optimize system prompts based on defined constraints, allowing developers to achieve high performance using cheaper, smaller models (e.g., Mistral) while maintaining model independence.
Technical details
-
Model Comparison & Portability
223s
The workshop compares GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3. The goal is to ensure AI applications can function reliably even if access to one model is cut off, requiring the ability to swap models with minimal code changes.
-
Tool Calling and Agent Design
3027s
Tool calling is the basis of all agents. The speaker demonstrated moving from basic tool calls to 'tool calling in a loop' to test efficiency. GPT-5.5 was noted for its ability to perform parallel tool calling, which is highly efficient for server-side operations.
-
Advanced Evaluation Frameworks
The speaker introduced two key tools: 1) **ASSERT** (Microsoft Research), which generates test cases and judging rubrics from a YAML specification; and 2) **DSPy**, which uses genetic algorithms to automatically optimize the system prompt to meet defined constraints.
-
Multimodal and Multilingual Pitfalls
1303s
Image input requires the model to inherently understand the image; prompt engineering cannot fix poor image understanding. Furthermore, model performance can degrade significantly when switching languages (e.g., failing when switching from English to Spanish).
Mentioned resources
- ASSERT (GitHub)
- LangChain v1
- Microsoft Foundry
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.