The Video Signal

Topic

Prompt Engineering

All digests tagged Prompt Engineering

· 59:49

From LLM Theory to Practical Agentic Implementations - Seth Juarez

The talk demystifies Large Language Models (LLMs), arguing that their power does not come from inherent intelligence but from engineered 'harnesses' and fundamental primitives. The speaker outlines a progression of techniques—from simple completions to complex agentic loops—that allow developers to build robust, reliable AI systems by controlling the model's input context, output structure, and execution flow. Key concepts include using structured JSON for control flow, implementing tool calling via runtimes (not the LLM itself), establishing guardrails, and orchestrating multi-step workflows through shared 'threads.'

Key takeaways

  1. LLMs are Next Token Predictors 1:42

    At its core, an LLM is a giant mathematical function that predicts the next token based on input tokens. The perceived intelligence comes from the surrounding runtime logic (the harness), not the model itself.

  2. The Agentic Loop is the Core Primitive 21:20

    True agency is achieved by implementing an 'agentic loop'—a while loop in the runtime that continuously checks for and executes tool calls, feeding the results back into the context until no more tools are needed. This is the fundamental unit of agency.

  3. Guardrails and Bindings Enhance Reliability 35:05

    To prevent misuse or incorrect execution (e.g., unauthorized memory access), developers must implement guardrail functions in their runtime logic. Furthermore, 'bindings' allow injecting fixed parameters (like `user ID`) into function calls, removing the burden of context-dependent reasoning from the LLM.

  4. Composition Workflow for Multi-Step Agents 50:50

    Complex tasks are solved by chaining multiple specialized agents or prompts (e.g., Research $ ightarrow$ Draft $ ightarrow$ Review). All these components share a single, persistent 'thread' context to maintain state and pass control flow sequentially.

Watch on YouTube Full article

· 20:17

Paste This Into Claude, Never Hit a Token Limit Again

This guide details advanced strategies for minimizing token consumption when interacting with Large Language Models (LLMs) like Claude and Codex. The core problem is 'reused input,' where every subsequent message resends the entire conversation history, leading to massive waste (up to 96% of tokens). Solutions are tiered: Level 1 involves user habits (e.g., editing mistakes instead of re-asking); Level 2 introduces automated tools like the 'Token Saver skill'; and Level 3 utilizes advanced frameworks such as Ringer for local, pre-model request constraint.

Key takeaways

  1. Edit Mistakes Instead of Re-Asking 4:40

    When correcting an AI error or typo, use the edit function rather than starting a new chat thread and stating 'that was wrong.' This prevents unnecessary token accumulation. (04:40)

  2. Start Clean Tasks for Job Changes 15:35

    When switching to a new, specific task, start a fresh chat thread. Continuing long conversations is good for focus but highly token-intensive and increases the volume of 'reused input.' (15:35)

  3. Carry Forward Artifacts, Not Arguments 15:35

    In multi-step processes (e.g., research followed by writing), only pass the final artifact or result from Stage A to Stage B. Do not include all previous drafts, criticisms, or rejected sources. (15:35)

  4. Use Local Retrieval and Caching

    Whenever possible, perform searches locally rather than relying on the model to search files, as this is a massive token burner. Also, use external databases (like OpenBrain) for frequently needed data points to avoid recalculation. (18:51)

  5. Enforce Hard Token Limits

    For critical workflows, use an intermediary tool like Ringer to enforce hard limits on the input/output packet size, ensuring predictable resource usage. (18:51)

Watch on YouTube Full article

· 53:25

Doors of (AI)pportunity: The Front and Backdoors of LLMs - Kasimir Schulz & Kenneth Yeung

The talk provides an advanced deep dive into LLM security vulnerabilities, moving beyond simple prompt injection attacks. Speakers demonstrate sophisticated methods including Knowledge Return Oriented Prompting (KROP), Policy Puppetry, and direct manipulation of the model's computational graph to implant backdoors. The discussion highlights that even seemingly safe formats (like ONNX) can be exploited if the underlying architecture is compromised, posing significant supply chain risks for AI-driven applications.

Key takeaways

  1. Multiple Layers of Defense are Required 35:35

    System prompts alone are insufficient protection. Developers must implement multiple layers of defense to mitigate attacks, as demonstrated by the failure of simple system prompt rules against advanced exploits.

  2. Indirect Prompt Injection is a Major Threat 43:25

    Attacks do not require direct user interaction. Compromise can occur through injected text in external sources like customer reviews, emails (especially those processed by Retrieval Augmented Generation/RAG systems), or calendar invites.

  3. Model Backdoors are Difficult to Detect 50:15

    Backdoors can be implanted into the computational graph of a model (e.g., changing an animal classification logic based on a specific color trigger). These backdoors can remain dormant until a precise trigger condition is met, making them highly effective and difficult to detect.

Watch on YouTube Full article

· 17:11

The Unreasonable Effectiveness of Separating the Task from the Model — Maxime Rivest, DSPy

The talk introduces DSPy, an open-source Python framework designed to bring software engineering principles—reusability, composability, and testability—to AI programs. The core philosophy is the 'unreasonable effectiveness of separating the task from the model' by defining tasks strictly through a programmatic interface called the Signature. This approach allows developers to focus on the business logic (the contract) while keeping the underlying implementation flexible for experimentation with different models, weights, and techniques.

Key takeaways

  1. The Power of the Signature

    By defining a task's inputs and outputs (the Signature) first, developers create an abstraction layer that allows them to swap out underlying implementations (models, prompts, etc.) without changing the core workflow. This provides significant agility for AI engineering.

  2. Three Pillars of Task Specification 8:00

    To fully specify an AI task, DSPy advocates using three components: 1) Instructions (what should happen), 2) Constraints/Code (what must happen, enforced by code like self-recheck or chaining), and 3) Evaluation Metrics (what good looks like).

  3. Future Direction: Automation and Learning 10:35

    DSPy is evolving toward automating implementation details. Future versions, including DSPy 4.0, aim to allow models to write code beneath a signature and enable programs to learn directly from user interactions while respecting the defined inputs/outputs.

Watch on YouTube Full article

· 26:22

Engineers... STOP Picking GPT-5.6 Sol OR Claude Fable 5… FUSE THEM

The video argues that in agentic engineering, the optimal approach is not to choose a single 'winner' model (e.g., GPT 5.6 Sol vs. Claude Fable 5), but rather to implement Model Fusion. This involves building custom agent harnesses that coordinate multiple state-of-the-art models working together. The process utilizes specialized commands—`/opinion` for diverse perspectives, `/fusion` for consolidating results, and `/auto validate` for intelligent on-the-fly review—to significantly outperform single-agent workflows.

Key takeaways

  1. Model Fusion: AND, Not OR

    The most powerful approach is combining the compute and intelligence of multiple models rather than selecting a single winner. This pattern combines concepts previously known as architect editor, prompt chaining, and agent chaining.

  2. Three Core Commands for Orchestration 2:00

    A custom fusion harness uses three commands: `/opinion` (to gather multiple perspectives), `/fusion` (to combine and consolidate results), and `/auto validate` (for intelligent on-the-fly validation, addressing the review constraint of agentic engineering).

  3. Value of Fusion vs. Single Agent 5:45

    Fusion allows agents to identify consensus, divergence, and discarded information, providing a comprehensive view that is critical for high-stakes strategic decisions.

Watch on YouTube Full article

· 25:58

Build Your Own App In Just 30 Minutes! Full Course with Andrew Ng

This course teaches build-engineering principles for modern software development by leveraging AI tools (like ChatGPT or Gemini). Instead of writing code manually, users learn 'prompting'—the art of giving precise instructions to an AI system to generate functional web applications (HTML files). The process emphasizes iterative refinement and mastering five key building blocks: Goal, Input, Layout, Special Features, and Output. By following this framework, users can build complex tools, such as a birthday card generator or a ping pong game, with minimal coding experience.

Key takeaways

  1. The Power of Prompting 4:30

    Creating software in the AI era involves telling the AI what to do (prompting) rather than typing out code. The more specific and precise the prompt, the more predictable the resulting application will be.

  2. The Five Building Blocks of Prompts 6:10

    To build effective prompts, consider these five components: 1) The Goal (what to create), 2) User Input (data the user provides), 3) Layout (arrangement of parts), 4) Special Features (additional functionality), and 5) Output (the desired result format).

  3. Iterative Development and Troubleshooting 11:20

    Software development is an iterative process. If the initial AI-generated app has bugs or needs improvement, users must continue the conversation with specific instructions (e.g., 'Nothing happens when I click on the generate card button. Can you fix it for me?').

Watch on YouTube Full article