Serverless Toronto
Beyond Prompt Engineering
Evals-first LLM applications with DSPy
Serj Smorodinsky & Brett Kennedy
Co-authors, Building LLM Applications with DSPy (Manning)
Contents
Section
Time
Part 1 — Why
5 min
2A — DSPy basics
8 min
2B — Methodology
8 min
2C — ATIS: baseline and evaluation
12 min
2D — Optimizers
13 min
2E — Deployment
8 min
2F — Agent vs DSPy vs DSPy with an Agent
5 min
Close
1 min
PART 1
Why
What is the number one bottleneck in development in 2026?
Answers in the chat.
How we verify software
Three checks, built over decades.
Automated tests
Run on every commit
Code review
A person reads the diff
QA
Someone exercises the change
Each one answers the same question: is this change correct?
None of them is the act of writing the code.
Why
1
How we verify a data science pipeline
The same idea. Different checks.
A representative dataset
Covers the inputs the system will actually see
An objective metric
Annotators agree on what the correct answer is
Methodology
Train, validation and test splits. No leakage. The test set used once.
These are the equivalent of tests, code review and QA.
Why
2
Reviewing an agent-written prompt optimizer
experiments/agentic/train.py · 388 lines · agent-only track of agentic_atis
line 1
line 388
Data and splits
Is the evaluation data representative, and is test kept out? Lines 29–35, 49–50, 67–75, 156–169.
Metric
Is it the right metric? Lines 187–199 turn raw text into a label; lines 327–329 compute macro F1.
Prompt
What does the model actually see? Lines 76–155 build it from rules and selected examples.
The other 260 lines are arguments, API calls, cost tracking and logging.
I wish we had a standard way to express this.
Why
3
Why DSPy, when a coding agent writes the code?
DSPy-based optimization follows best practice and produces very short code to review, whether a coding agent or a person wrote it.
Lines of code in the ATIS example later in this talk:
7
Signature
2
Metric
3
Baseline and evaluation
2
Optimizer
1
Export
About 15 lines in total, not counting data loading. Reviewing them means reviewing the methodology: a metric, a baseline, an optimizer, and a saved program with its score.
Why
5
What you work on when you stop looking at the prompt
In DSPy the prompt text is generated, so the work moves to modeling the task:
  • What the task is, and whether it is one call or several
  • The label set, and which inputs you decide not to handle
  • The dataset, and whether it represents production traffic
  • The metric, and what counts as correct
  • The splits
  • Which model, and which module
  • How much the score varies between runs
This is the list that decides whether the application works.
Why
6
Writing code is not the bottleneck
An agent writes the code in seconds.
The six checks are where the time goes.
For LLM applications, three of them do not exist yet.
2A
DSPy basics
What DSPy is
A prompt is a string you write and edit.
A DSPy program is code that declares inputs and outputs. DSPy generates the prompt text from that declaration.
You also declare the inference method, separately from the prompt text:
Inference method
What it does
Predict
a single call
ChainOfThought
a call that reasons before answering
BestOfN, Refine, dspy.majority
several calls reduced to one answer
A custom module
a composition of any of these
An optimizer changes the generated prompt, using a dataset and a metric.
DSPy basics
7
Four concepts
LM
The language model you call
Signature
The declared inputs and outputs, plus an instruction
Module
The inference method: Predict, ChainOfThought, and others
Prediction
The returned object, with one attribute per declared output
DSPy basics
8
A complete program
import dspy
 
lm = dspy.LM("openai/gpt-4o-mini", api_key=OPENAI_API_KEY)
dspy.settings.configure(lm=lm)
predictor = dspy.Predict("question, context -> answer, confidence")
prediction = predictor(question="What is the capital of France?",
                       context="")
print(prediction.answer, prediction.confidence)
chapter_1/Listing_1_1.py — no prompt was written.
DSPy basics
9
Signature as a class
class IntentSignature(dspy.Signature):
    """
    Classify the message into one of the possible labels.
    """
    message: str = dspy.InputField()
    labels: List[str] = dspy.InputField()
    intent_label: str = dspy.OutputField()
  • The docstring becomes the instruction in the prompt
  • The fields become the inputs and outputs
  • It is a class in the repository, so a change to it shows up in a diff
DSPy basics
10
Reading the generated prompt
lm.inspect_history(n=1)
print(intent_classifier.signature.instructions)
The first prints the rendered messages that were sent, and the completion that came back.
The second prints just the instruction text, which for an unoptimized program is the docstring.
Generated prompts are only acceptable if you can read them.
DSPy basics
11
Switching models
lm_mini = dspy.LM(model='gpt-4o-mini', cache=False, temperature=0.0)
lm_41 = dspy.LM(model='gpt-4.1-turbo', cache=False, temperature=0.0)
 
dspy.settings.configure(lm=lm_41) # global default
with dspy.context(lm=lm_mini): # for a block
    prediction = prog(question='...')
prog.set_lm(lm_41) # for one module
chapter_2/Listing_2.11.py
DSPy basics
12
2B
Methodology
The loop
Baseline
The simplest program that does the task
Evaluate
Score it on data you did not optimize on
Optimize
Search for a better prompt, then evaluate again
The order matters. Without a baseline score you cannot tell whether optimization helped.
Chapter 1.6. The full version in the book is 11 steps, from preparing the dataset to comparing the optimized program against the baseline.
Methodology
13
CHECK 6
Baseline
classifier = dspy.Predict(IntentSignature)
  • No examples in the prompt, no tuned instruction
  • Score it before changing anything
  • Every later number is compared against it
Most teams start at optimization, which means they have nothing to compare to.
Methodology
14
CHECK 4 · A REPRESENTATIVE DATASET
Datasets and splits
train
Given to the optimizer
validation
Used by the optimizer to score its candidates
dev
What you look at while working
test
Used once, before release
If you optimize and report on the same set, the number is not an estimate of production behavior.
Chapter 4.1. In the repository: common/utils.py:create_examples_from_set
Methodology
15
CHECK 5 · AN OBJECTIVE METRIC
The metric
def validate_answer(example, prediction, trace=None):
    return (example.intent_label.lower()
            == prediction.intent_label.strip().lower())
  • Takes an example and a prediction, returns a score
  • The same function is used by Evaluate and by the optimizers
  • The normalization inside it is part of the definition of correct
chapter_4/evaluate_dspy.py
Methodology
16
When the metric is not a string comparison
Classification
Exact match. No ambiguity, and annotators agree by construction.
Summaries, open answers, agent traces
There is no single correct output.
Options, in order of cost: lexical overlap (precision, recall, F1), embedding similarity, LLM as a judge.
A judge is a program as well. Before using it as a metric, measure how often it agrees with human labels. Chapter 8 covers groundedness, completeness, and calibrating a judge.
Methodology
17
Running an evaluation
By hand
for example in tqdm(dev_set):
    prediction = classifier(
        **example.inputs())
    scores.append(
        validate_answer(
            example, prediction))
With DSPy
evaluator = dspy.Evaluate(
    devset=dev_set,
    num_threads=10,
    display_table=5)
results = evaluator(
    classifier, metric=validate_answer)
  • num_threads is the difference between minutes and seconds
  • display_table prints failing examples
  • Set cache=False on the LM when you run the same inputs more than once
Methodology
18
CHECK 6 · METHODOLOGY
Four rules
  • Do not optimize on the set you report on
  • Do not report a single run
  • Keep the metric fixed while you compare prompts
  • Record which model, which data, and which score
Methodology
19
2C
ATIS: baseline and evaluation
The task
Utterance
Intent label
Train rows
show me the flights from dallas to san francisco
Flight Booking Request
3,666
cheapest airfare from tacoma to orlando
Airfare and Fees Questions
423
show me the ground transportation in denver
Ground Transportation Inquiry
255
which airlines serve atlanta
Airline Information Request
157
what does the fare code f mean
Abbreviation and Fare Code Meaning Inquiry
147
what types of aircraft does delta fly
Aircraft Type Inquiry
81
how many delta flights leave from washington
Flight Quantity Inquiry
51
how far is downtown from the airport in dallas
Airport Distance Inquiry
20
how many passengers fit on a d9s
Aircraft Seating Capacity Inquiry
16
what are all the available meals
Inquiry about In-flight Meals
6
ATIS: Airline Travel Information System. Utterances are verbatim from the train split (4,978 rows). 17 intent classes after mapping, one label per utterance. Chapter 3.
ATIS
20
The data
ds = load_dataset("tuetschek/atis")
df['intent'] = df['intent'].map(ATIS_INTENT_MAPPING)
df = df.dropna(subset='intent')
  • Raw intent names are mapped to readable labels: flight → Flight Booking Request
  • Rows with more than one intent (flight+airfare, airline+flight_no) are dropped
Dropping them is a decision, not a detail. Chapter 3.6 covers handling multiple intents if you need them.
ATIS
21
Examples
dspy.Example(message=row['text'],
             labels=unique_intents,
             intent_label=row['intent'])
    .with_inputs('message', 'labels')
with_inputs declares which fields the program receives.
Everything else on the example is a label, used only by the metric.
ATIS
22
The splits used here
100
train examples, sampled from the ATIS train split
876–1,000
held-out evaluation examples the optimizer never sees
Sampled with random_state=42, so the runs are repeatable.
Chapter 5 reports on 1,000 dev examples. Chapter 6 reports on the 876 usable examples of the ATIS test split.
ATIS
23
Baseline using the OpenAI API directly
system_prompt = f'''You are an expert of intent classification.
Your task is to classify the intent of customer messages of an
airline company into one of the provided labels.
    Input: customer message
    Output: One of the following classes: {",".join(unique_intents)}'''
This is the version most applications have today: a hand-written prompt, called directly.
chapter_4/evaluate_openai.py
ATIS
24
Baseline using DSPy
classifier = dspy.Predict(IntentSignature)
evaluator = dspy.Evaluate(devset=dev_set, num_threads=10)
results = evaluator(classifier, metric=validate_answer)
Same data, same metric function. Run both and compare.
ATIS
25
Baseline results
Module
LM
Accuracy, 3 runs
Duration (s)
Predict
gpt-4o-mini
92.1 / 93.1 / 93.1
82.9 / 102.1 / 77.1
Predict
gpt-4.1-nano
85.9 / 86.2 / 86.8
83.2 / 121.9 / 86.2
ChainOfThought
gpt-4o-mini
90.5 / 93.8 / 92.8
167.6 / 205.4 / 196.5
ChainOfThought
gpt-4.1-nano
88.9 / 89.5 / 89.6
78.5 / 108.6 / 84.6
  • ChainOfThought is not better here, and on 4o-mini it takes about twice as long
  • 4o-mini is more accurate than 4.1-nano
  • Zero exceptions for both models. Worth tracking, some models produce many.
Table 4.1.
ATIS
26
Variation between runs
92.1 / 93.1 / 93.1
Same code, same data, three runs.
A one-point difference between two prompts may be noise. To reduce it:
  • Evaluate on more examples
  • Run each configuration several times
  • Set cache=False, otherwise repeated runs return the same cached answers
ATIS
27
Per-class results
print(classification_report(groundtruths, predictions))
Overall accuracy hides classes with few examples and low recall.
This is where you find out that the class one customer depends on is the one performing worst. Chapter 4.5.
ATIS
28
2D
Optimizers
Few-shot optimizers and instruction optimizers
Field descriptions
message (str) → intent_label (Literal[…15 labels])
Instructions
Classify the message into one of the possible labels.
Demonstrations
airports in new york → Airport Information and Queries …
Current input
What meals are served on the 3pm flight to Madrid?
Instruction optimizers · chapter 6
COPRO, GEPA
Few-shot optimizers · chapter 5
LabeledFewShot, BootstrapFewShot, KNN
MIPROv2, SIMBA and InferRules change both. Examples are the rows in your dataset; demonstrations are the pairs placed in the prompt.
Optimizers
29
CHAPTER 5
Few-shot optimizers: they change the demonstrations
The instruction stays exactly as you wrote it.
Optimizer
What it does
LabeledFewShot
Takes k examples from the train set and puts them in the prompt with their labels
BootstrapFewShot
Runs the program on train examples, keeps the ones the metric accepts, uses those as demos. With ChainOfThought the demos include the generated reasoning.
BootstrapFewShotWithRandomSearch
Builds several candidate demo sets, evaluates each, keeps the best
KNN
Selects different demos for every input, by embedding similarity to the incoming message
BootstrapFewShot can use a separate, stronger model as the teacher that generates the demos. Chapter 5.3.2.
Optimizers
30
What that produces
optimizer = LabeledFewShot(k=50)
few_shot_model = optimizer.compile(student=intent_classifier,
                                   trainset=train_examples)
The prompt now carries 50 labeled messages ahead of the real one.
The cost is paid on every request after that, in input tokens. KNN keeps the prompt smaller but adds an embedding lookup per request.
Optimizers
31
CHAPTER 6
Instruction optimizers: they change the instruction text
print(intent_classifier.signature.instructions)
# "Classify the message into one of the possible labels."
  • The instruction is the signature docstring
  • It appears in the system message after the field descriptions and before the demos
  • Without a docstring, DSPy writes one for you: “Given the fields message, produce the fields intent_label.”
  • An instruction optimizer replaces that text with one it searched for
Write a docstring even if you plan to optimize. It is the starting point for most of these optimizers.
Optimizers
32
The instruction optimizers
Optimizer
How it works
COPRO
Hill climbing over candidate instructions. Also rewrites output field prefixes.
MIPROv2
Proposes instructions and selects demos, searched with Bayesian optimization.
InferRules
Generates rules from the training data, appended to the instruction.
SIMBA
Reflective; each step adds one demo or one rule, and may remove demos.
GEPA
Genetic algorithm keeping a Pareto front; uses feedback from the metric.
MIPROv2, SIMBA and GEPA ground their candidates in the module source and a summary of the dataset. Ensemble is separate: it combines several finished programs into a majority vote.
Optimizers
33
What optimization costs
Optimizer
Parameters
Time
Tokens
LM calls
COPRO
breadth 2, depth 2
~4 min
601,112
1,212
MIPROv2
auto="light"
~2 min
~424,000
~675
InferRules
5 candidates, 10 rules
~9 min
~1,900,000
~1,500
SIMBA
3 candidates, 20 steps
~12 min
~1,500,000
~4,900
GEPA
minimal settings
~1 min
~42,000
108
This is not a ranking. The parameters were set low so the examples run quickly, and each optimizer has different parameters. Time and tokens depend on the parameters and on the size of the validation set.
Measured on Google Colab, chapter 6.
Optimizers
35
Results, gpt-4o-mini
Optimizer
Module
Parameters
Accuracy
Baseline
Predict
91.1
Baseline
ChainOfThought
89.16
COPRO
ChainOfThought
breadth 6, depth 6
90.11
LabeledFewShot
ChainOfThought
50 examples
94.18
MIPROv2
ChainOfThought
medium, 5 bootstrapped, 10 labeled
94.04
About 3 points over the baseline. Classification is an easy task for these models, so there is not much room above 91. The simplest few-shot optimizer did as well as the most elaborate instruction optimizer here.
Tables 5.1 and 6.6.
Optimizers
36
Results, gpt-4.1-nano
Optimizer
Module
Parameters
Accuracy
Baseline
Predict
81.74
Baseline
ChainOfThought
80.82
LabeledFewShot
Predict
25 examples
93.38
LabeledFewShot
ChainOfThought
50 examples
94.06
MIPROv2
ChainOfThought
heavy, 15 bootstrapped, 25 labeled
93.15
The unoptimized nano program is about 9 points behind 4o-mini. The optimized one reaches the same range. nano is roughly two thirds the price per token.
Tables 5.2 and 6.7.
Optimizers
37
2E
Deployment
The prompt nobody wrote
SYSTEM     Your input fields are: 1. `message` (str):
           Your output fields are: 1. `intent_label` (Literal[…15 labels]):
           [[ ## message ## ]] {message}
           [[ ## intent_label ## ]] {intent_label} # must exactly match one of: …
Field descriptions
from the signature
           In adhering to this structure, your objective is:
               Classify the message into one of the possible labels.
Instruction
the docstring, unchanged
USER       [[ ## message ## ]] airports in new york
ASSISTANT  [[ ## intent_label ## ]] Airport Information and Queries
           … 3 more message / label pairs
Demonstrations
chosen by the optimizer
USER       [[ ## message ## ]] What meals are served on the 3pm flight to Madrid?
           Respond with the corresponding output fields, starting with the field
           `[[ ## intent_label ## ]]` …, and then ending with `[[ ## completed ## ]]`.
Current input
Deployment
38
Adding DSPy to the backend
  • The backend is not Python
  • We do not want another runtime dependency in the service
  • We do not want an optimizer running in production
All three are reasonable.
Deployment
39
Training and inference are separate
Training
offline
Inference
online, in the service
Labeled data
Train
Evaluate
Model artifact
Request
Load model
Predict
Response
Ship the model artifact
With DSPy, the optimizer is the training step and the saved program is the model artifact. The service only runs the inference side.
Deployment
40
Can we export the best prompt?
# full program, as a pickle
simba_prog.save("prog_v1", save_program=True)
simba_prog2 = dspy.load("prog_v1", allow_pickle=True)
 
# metadata only, as JSON
simba_prog.save("prog_v1.json", save_program=False)
simba_prog2 = dspy.Predict(ClosedIntentSignature)
simba_prog2.load("prog_v1.json")
Yes. With save_program=False the program is written as JSON: the instructions, the demonstrations, and the field definitions. Nothing in it requires DSPy to read it.
chapter_6/Listing_6.16.py, Listing_6.17.py
Deployment
41
Yes, ask your coding agent. But how do we test it?
  1. Give the JSON and the signature to a coding agent
  2. Ask for the equivalent call in your stack
Is the ported version the same program?
Reading the two side by side does not answer that. The instruction text can match while the request still differs: how the fields are formatted, and how the answer is parsed.
Deployment
42
Rerun the evaluation
  1. Keep the dataset and the metric function
  2. Point them at the ported implementation
  3. Compare the score to the DSPy score
If the numbers do not match, the port is wrong.
The same procedure applies when a model is deprecated, when a provider changes the model behind a name, or when someone edits the prompt by hand.
Deployment
43
What to keep for each release
  • The prompt
  • The dataset it was measured on
  • The metric function
  • The score
  • The model and its version
  • The date
This is what separates a pilot from something that was approved for production.
Deployment
44
2F
Agent vs DSPy vs DSPy with an Agent
Agent, DSPy, and DSPy with an agent
agentic
A coding agent, no DSPy
Edits prompt wording, disambiguation rules, label guidance and few-shot selection. Run, read the predictions, find the confusion, write a rule, rerun. Keep the change if macro F1 improved.
dspy
DSPy, no redesign
Exactly five MIPROv2 runs from a fixed run plan. Runs differ only in hyperparameters and seed. No error analysis feeds back.
agentic_on_dspy
A coding agent on top of DSPy
May rewrite the signature docstring, redesign label guidance, change the objective, switch optimizer family, and write the feedback text that GEPA reads.
Each track is a folder with its own AGENTS.md that says what the agent may change.
Agent vs DSPy
46
Who is allowed to edit your prompt?
Harness A: DSPy
Harness B: a coding agent
Propose candidate prompts
Score each one on the metric
Keep the best, search again
Read the code, errors and predictions
Edit the prompt
Rerun, keep or discard
The prompt
instructions and demonstrations
wording, rules, label guidance
Both are called automated prompt engineering. They fail in different places. github.com/SerjSmor/agentic_atis
Agent vs DSPy
45
What each track is allowed to touch
Change
agentic
dspy
agentic_on_dspy
Prompt wording
agent
optimizer
both
Few-shot demo selection
agent
optimizer
both
Instruction search over candidates
no
yes
yes
Label guidance redesign
yes
no
yes
Change the metric or objective
no
no
yes
Switch optimizer family
no
no
yes
Error analysis leads to a program change
yes
no
yes
Everything else is pinned: the split, the model, the evaluation code, and the metric definitions.
Agent vs DSPy
47
Same everything, except the harness
Frozen split
50 train, 50 validation, 30 test. Generated once by prepare.py.
Model pinned
gpt-4.1-nano on every track.
Primary metric
Macro F1. Weighted and micro F1 are recorded, not optimized.
Test is locked
Iterate on validation. Test is for one final run.
Everything logged
Tokens, cost, compile time and evaluation time, on every run.
Cost capped
A cost ceiling on each experiment.
50 validation examples is a small set. Every number from this experiment inherits that.
Agent vs DSPy
48
Results on validation
Track
Optimizer
Macro F1
Weighted F1
Micro F1 (accuracy)
Cost per run
agentic
none
0.574
0.910
0.88
$0.0050
dspy
MIPROv2
0.533
0.928
0.88
$0.0081
agentic_on_dspy
GEPA
0.772
0.943
0.94
$0.0401
  • DSPy with a coding agent has the highest macro F1: 0.772, against 0.574 for the agent alone and 0.533 for DSPy alone.
  • Accuracy does not separate the agent and DSPy tracks. Both are 0.88. Macro F1 does.
  • The best track also costs the most: $0.040 per run, about 8 times the agent alone.
gpt-4.1-nano, 50 validation examples, best run per track. DSPy costs include compilation and evaluation.
Agent vs DSPy
49
Conclusions
1
DSPy is the best harness for prompt optimization.
2
A coding agent working with DSPy writes shorter code, and code that follows best practices.
Where to start: label 50 examples, write the metric function, run the baseline and write down the number.
Summary
1
DSPy standardizes prompt evaluation and optimization, for people and for coding agents.
A signature, a metric, an evaluation and an optimizer: about 15 lines to review. In agentic_atis, a coding agent working on top of DSPy had the highest macro F1: 0.772.
2
The prompt itself becomes irrelevant. Composition is much more interesting.
The inference method is a declaration: Predict, ChainOfThought, BestOfN, Refine, or your own module. DSPy writes the prompt.
3
You usually need optimization for large-scale tasks, where a small model keeps the cost down.
On ATIS, optimized gpt-4.1-nano went from 81.7% to about 94%, the range of gpt-4o-mini, at about two thirds of the price per token.
Thank you to the DSPy team
DSPy 3.4 is out. It supports Jev, TypeSafe AI’s System One model.
Jev returns typed decisions with probabilities and confidence, instead of generating text. The integration is experimental.
pip install "dspy[typesafe]==3.4.0"
dspy.configure(lm=dspy.experimental.TypeSafe("jev-latest"))