# The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft

## Executive summary

This workshop provides a deep dive into comparing the real-world performance of top-tier LLMs (GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, Mistral Large 3) across complex, multi-step applications. The core message is that standard benchmarks are insufficient; model behavior must be tested in specific scenarios. Key techniques demonstrated include robust Retrieval-Augmented Generation (RAG) grounding, efficient tool calling (including parallel calls), and advanced evaluation frameworks like Microsoft's ASSERT and DSPy, which help automate test set generation and prompt optimization for model portability.

## Key takeaways

- Model Differences are Scenario-Specific: Models exhibit significant differences in core capabilities (e.g., counting, groundedness, tool calling efficiency). For instance, GPT-5.5 often demonstrated superior performance in parallel tool calling and adherence to constraints compared to competitors.
- RAG Grounding Requires Strict Prompting: Models, especially smaller ones, are prone to hallucinating information from their internal weights rather than strictly adhering to provided context. Prompt engineering is necessary to force models to admit when information is not source-provided.
- Evaluation Must Move Beyond Simple Checks: While string checks are basic, robust evaluation requires advanced methods like LLM-as-a-Judge (which is slow/expensive) or using frameworks like ASSERT to generate comprehensive test sets from a high-level specification (YAML).
- Prompt Optimization for Model Independence: DSPy uses genetic algorithms (Jeppa) to automatically write and optimize system prompts based on defined constraints, allowing developers to achieve high performance using cheaper, smaller models (e.g., Mistral) while maintaining model independence.

## Technical details

- Model Comparison & Portability: The workshop compares GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3. The goal is to ensure AI applications can function reliably even if access to one model is cut off, requiring the ability to swap models with minimal code changes.
- Tool Calling and Agent Design: Tool calling is the basis of all agents. The speaker demonstrated moving from basic tool calls to 'tool calling in a loop' to test efficiency. GPT-5.5 was noted for its ability to perform parallel tool calling, which is highly efficient for server-side operations.
- Advanced Evaluation Frameworks: The speaker introduced two key tools: 1) **ASSERT** (Microsoft Research), which generates test cases and judging rubrics from a YAML specification; and 2) **DSPy**, which uses genetic algorithms to automatically optimize the system prompt to meet defined constraints.
- Multimodal and Multilingual Pitfalls: Image input requires the model to inherently understand the image; prompt engineering cannot fix poor image understanding. Furthermore, model performance can degrade significantly when switching languages (e.g., failing when switching from English to Spanish).

## Practical implications

- Implement automated, structured evaluation pipelines (using ASSERT) to bootstrap test sets before production.
- Use DSPy to abstract prompt engineering away from manual effort, ensuring model independence and cost optimization.
- When designing agents, prioritize models that support parallel tool calling for maximum efficiency.
- Always test RAG systems with specific prompts designed to force the model to acknowledge when context is missing.

## Topics

Large Language Models (LLMs), Prompt Engineering, Retrieval-Augmented Generation (RAG), Tool Calling/Function Calling, Agent Frameworks, Model Evaluation, Multimodal AI, ASSERT (GitHub), LangChain v1, Microsoft Foundry

Source: https://www.youtube.com/watch?v=UGjk3Uk0MxM