Open Bench · Unit 6 of 8
Teaching a computer to guessCan a model predict which molecule will work, and should you believe it?
Students train a small model on real chemistry data, then deliberately break it, because a student who has only ever seen a model succeed has learned nothing useful about models.
Objectives
By the end of the hour, a student can do each of these without being prompted.
- 01
Explain what training a model means, using no mathematics.
- 02
Run a notebook that trains a classifier on a public dataset.
- 03
Explain why a model that scores ninety-nine percent can be worthless.
- 04
Break a working model on purpose and diagnose what you broke.
Run of show
The hour, minute by minute.
Timings are a plan, not a rule. The segment most worth protecting when you run late is noted in the teacher notes below.
- 10 min
Prediction without magic
A model finds patterns in examples it has seen and applies them to ones it has not. That is the entire concept. Everything else is engineering.
- 15 min
Run the notebook
Students open a prepared notebook and train a classifier on a public benchmark. Nobody writes code from scratch; they change values and watch what happens.
- 15 min
The ninety-nine percent trap
Hand them a dataset where ninety-nine percent of examples are negative. A model that always answers no scores ninety-nine percent. Let them discover this rather than telling them.
- 15 min
Break it on purpose
Students sabotage their own model: test on the training data, shuffle the labels, remove a feature. Each sabotage produces a specific symptom they must name.
- 5 min
What this means for real discovery
Connect back to the funnel from Unit 2. A model that cuts the number of molecules you have to test is worth a fortune. A model you cannot audit is worth less than nothing.
The activity
Train it, then break it
- 1
Open the prepared notebook and train the baseline model as written.
- 2
Record its score on data it has never seen.
- 3
Now test it on the data it trained on and record the inflated score.
- 4
Shuffle the labels randomly, retrain, and record what happens.
- 5
Write one paragraph per sabotage explaining the symptom and the cause.
What you hand in
A notebook with three deliberate failures, each diagnosed in writing. This is worth more than a notebook where everything worked.
Tools, all free
Free notebooks with a free GPU. No install, runs in a browser.
Free curated datasets and benchmarks for drug discovery, built so methods can be compared fairly.
Free open-source library. Used here only through the prepared notebook.
The misconception to expect
“A higher accuracy number means a better model.”
Accuracy on an unbalanced dataset is nearly meaningless, and accuracy on data the model has already seen is entirely meaningless. What matters is performance on examples genuinely held out, measured in a way that suits the question.
Teaching it
Notes from the room.
How you know it landed
A student has it when they can look at a reported accuracy figure and ask the right first question, which is what the model was tested on.
- 01
This unit is Builder track only, not because middle schoolers cannot grasp it, but because the notebook debugging consumes the hour. Explorers get the same ideas without code in a shortened Unit 7.
- 02
Something in the notebook will break for someone. Budget for it, and treat the breakage as content rather than as lost time.
- 03
A student will ask whether this is how ChatGPT works. Say that the principle is the same and the scale is not, and move on; the digression will otherwise eat the lesson.
Reading for this unit
The papers behind the hour.
Optional for Explorers, expected for Builders. Full notes on each one are in the reading room.
- R-13
A foundation model for clinician-centered drug repurposing
Nature Medicine · 2024 · Free full text
A graph neural network trained on a medical knowledge graph ranks existing drugs as candidate treatments for 17,080 diseases, including diseases with no approved therapy, and explains each prediction as a path through the graph.
- R-17
Artificial intelligence foundation for therapeutic science (Therapeutics Data Commons)
Nature Chemical Biology · 2022 · Free full text
A collection of curated datasets, standardized tasks and leaderboards spanning the stages of drug discovery, built so that competing machine-learning methods can be compared on identical footing.
- R-18
Artificial intelligence in pediatric allergy research
European Journal of Pediatrics · 2024 · Free full text
A practical guide to running machine-learning studies in pediatric allergy, covering core concepts, how to structure an analysis pipeline from variable selection to interpretation, and the common pitfalls. It notes that most work so far has used simple single-source data and reported its methods poorly.