{ "flow": "judged", "engine": "mlx", "device": "Device(gpu, 0)", "allowed_models": null, "started": "2026-10-13T17:00:39-0700", "steps": [ { "command": "spill example dolly-qa --tiny", "exit": 0, "seconds": 0.1, "stdout": "created dolly-qa-tiny/ with NOTICE.txt, README.md, dolly-qa.csv\n\nnext: spill init dolly-qa-tiny/dolly-qa.csv --input prompt --output answer --project dolly-qa-tiny-project\n", "stderr_tail": "" }, { "command": "spill init dolly-qa-tiny/dolly-qa.csv --input prompt --output answer --project dolly --teacher qwen2.5:1.5b --judge qwen2.5:1.5b", "exit": 0, "seconds": 0.7, "stdout": "scoring: judge (the outputs look like free text or are not a small repeated label set (84 distinct in 84 rows))\nspill init dolly-qa.csv: scoring judge (the outputs look like free text or are not a small repeated label set (84 distinct in 84 rows)). Train 58, validation 13, final test 13 rows. Cost: $0. Project -> dolly\n train, validation and test were generated with seed 0; duplicate inputs and shared groups stay together\n\nnext: spill plan dolly\n", "stderr_tail": "" }, { "command": "spill plan dolly --engine mlx", "exit": 0, "seconds": 0.8, "stdout": "spill plan dolly: scoring judge, models qwen2.5:0.5b (student), qwen2.5:1.5b (teacher), qwen2.5:1.5b (judge), bf16 weights, no quantization, on mlx (--engine mlx). Est. 2 min. Cost: $0. Run -> dolly/runs/\nmodels (each is a choice: edit streamweights.toml, or pass --student, --teacher, --judge):\n student qwen2.5:0.5b Qwen/Qwen2.5-0.5B-Instruct@7ae557604adf safetensors bf16 0.92 GB, on disk (qwen2.5:0.5b on mlx: the smallest curated model, so a first result takes minutes on any machine; this machine's 39 GB could also train qwen2.5:1.5b or qwen2.5:7b (--student))\n teacher qwen2.5:1.5b Qwen/Qwen2.5-1.5B-Instruct@989aa7980e4c safetensors bf16 2.88 GB, on disk (named by you; answers the training prompts only, never a validation or test row)\n training the teacher's answers train the student with mix (log-probs kept: same tokenizer); your labels are repeated x2\n judge qwen2.5:1.5b Qwen/Qwen2.5-1.5B-Instruct@989aa7980e4c safetensors bf16 2.88 GB, on disk (the teacher, because a named teacher judges unless you name a judge; its scores are one model's assessment, not accuracy)\n baseline nearest training answer by TF-IDF (no model, nothing downloaded)\nengine: mlx: --engine mlx\nscoring: judge: reference overlap against your outputs, plus judged scores from qwen2.5:1.5b (one model's assessment)\ndownloads:\n qwen2.5:0.5b: 0.93 GB already on disk\n qwen2.5:1.5b: 2.88 GB already on disk\n nothing to download\ndisk:\n checkpoints 0.88 GB (9 x 106 MB, every published checkpoint is kept; deleting them is a later feature), staging 0.14 GB, adapter 35 MB\n export (only if you run it): merged 0.92 GB, GGUF q8_0 about 0.49 GB (assumption: 0.53 x bf16); free here 46.02 GB\nstages:\n 1. teacher qwen2.5:1.5b answers the 58 training prompts, keeping its top-20 log-probs: 4 s [assumption] 58 rows, batch 512, resident: prefill 4 s (5,663 tokens, shared prefix once) + decode 1 s (1 x 20 passes x 0.03 s); no matching measurement yet (rate measured by spill tune)\n 2. nearest training answer baseline (TF-IDF, no model): 1 s [assumption] word counting over the training inputs, no model; under a second at this size\n 3. tune qwen2.5:0.5b on 174 training rows (58 yours x2 plus 58 from the teacher, loss mix): 72 s [measurement] 44,970 token-passes x 6 x 0.49B params / 1.86 TFLOP/s = 72 s; rate measured by spill tune\n 4. qwen2.5:0.5b prompted, untrained, on 13 validation rows: <1 s [assumption] 13 rows, batch 512, resident: prefill 0 s (1,184 tokens, shared prefix once) + decode 0 s (1 x 20 passes x 0.01 s); no matching measurement yet (rate measured by spill tune)\n 5. qwen2.5:0.5b trained, on 13 validation rows: <1 s [assumption] 13 rows, batch 512, resident: prefill 0 s (1,182 tokens, shared prefix once) + decode 0 s (1 x 20 passes x 0.01 s); no matching measurement yet (rate measured by spill tune)\n 6. teacher qwen2.5:1.5b on 13 validation rows: 1 s [assumption] 13 rows, batch 512, resident: prefill 1 s (1,184 tokens, shared prefix once) + decode 1 s (1 x 20 passes x 0.03 s); no matching measurement yet (rate measured by spill tune)\n 7. judge qwen2.5:1.5b scores 3 comparators and 2 pairs (both orders) on 13 validation rows: 91 requests (one model's assessment): 12 s [assumption] 91 rows, batch 512, resident: prefill 10 s (15,830 tokens, shared prefix once) + decode 1 s (1 x 46 passes x 0.03 s); no matching measurement yet (rate measured by spill tune)\ntotal: 2 min; model loading is not included\nnote: the judge (qwen2.5:1.5b) is the same model as the teacher (qwen2.5:1.5b); a judge may not judge its own outputs, so this row is reported as reference only and excluded from rankings\n\nnext: spill build dolly\n", "stderr_tail": "" }, { "command": "spill build dolly --engine mlx", "exit": 0, "seconds": 90.7, "stdout": "updated streamweights.toml: evaluation.metric = 'judge_mean' (was 'token_f1'; the judge's assessment is the primary metric); the build is a new run\nspill build dolly: scoring judge, models qwen2.5:0.5b (student), qwen2.5:1.5b (teacher), qwen2.5:1.5b (judge), bf16 weights, no quantization, on mlx (--engine mlx). Est. 2 min. Cost: $0. Run -> dolly/runs/\nstage 1/7: teacher qwen2.5:1.5b answers the 58 training prompts, keeping its top-20 log-probs; estimate 4 s (assumption)\nstage 1 done in 14 s\nstage 2/7: nearest training answer baseline (TF-IDF, no model); estimate 1 s (assumption)\nstage 2 done in <1 s\nstage 3/7: tune qwen2.5:0.5b on 174 training rows (58 yours x2 plus 58 from the teacher, loss mix); estimate 72 s (measurement)\nstage 3 done in 35 s; 88 steps, final loss 0.2399120032787323\nstage 4/7: qwen2.5:0.5b prompted, untrained, on 13 validation rows; estimate <1 s (assumption)\nstage 4 done in 4 s\nstage 5/7: qwen2.5:0.5b trained, on 13 validation rows; estimate <1 s (assumption)\nstage 5 done in 2 s\nstage 6/7: teacher qwen2.5:1.5b on 13 validation rows; estimate 1 s (assumption)\nstage 6 done in 4 s\nstage 7/7: judge qwen2.5:1.5b scores 3 comparators and 2 pairs (both orders) on 13 validation rows: 91 requests (one model's assessment); estimate 12 s (assumption)\nstage 7 done in 30 s\n\nnearest training answer (TF-IDF, no model) judge score 1-5 (one model's assessment, not accuracy) 2.462 (13 rows)\nstudent, prompted, untrained judge score 1-5 (one model's assessment, not accuracy) 4.846 (13 rows)\nstudent, trained judge score 1-5 (one model's assessment, not accuracy) 4.538 (13 rows)\nteacher judge score 1-5 (one model's assessment, not accuracy) reference only, judge is the same model (13 rows)\n The judge qwen2.5:1.5b is one model's assessment of each output against the rubric; its scores are not accuracy.\n teacher: the judge (qwen2.5:1.5b) is the same model as the teacher (qwen2.5:1.5b); a judge may not judge its own outputs, so this row is reported as reference only and excluded from rankings.\n trained vs untrained, both orders: trained won 0, tied 11, lost 2 of 13 rows; the two orders disagreed on 8 (62%); a disagreement counts as a tie.\n trained vs baseline, both orders: trained won 8, tied 4, lost 0 of 12 rows; the two orders disagreed on 4 (33%); a disagreement counts as a tie.\n Calibration: the judge agreed with the exact-match check on 80% of the 15 rows that have one (judge score of 4 or more against the reference appearing word for word in the output, on the 15 rows whose reference has at most 5 words).\nrun run-20261013-170041-12b2-2c7311: dolly/runs/run-20261013-170041-12b2-2c7311\n\nnext: spill report dolly\n", "stderr_tail": "2 \u00b7 ~0.0 s per token per prompt \u00b7 7.6 GB peak\u001b[K\n\u001b[1Frows 90/91 \u00b7 pass 118 (0.0 s) \u00b7 213.2 tok/s \u00b7 ETA 0m 00s \u00b7 bf16 \u00b7 batch 1 \u00b7 ~0.0 s per token per prompt \u00b7 7.6 GB peak\u001b[K\n\u001b[1Frows 90/91 \u00b7 pass 119 (0.0 s) \u00b7 213.1 tok/s \u00b7 ETA 0m 00s \u00b7 bf16 \u00b7 batch 1 \u00b7 ~0.0 s per token per prompt \u00b7 7.6 GB peak\u001b[K\n\u001b[1Frows 90/91 \u00b7 pass 120 (0.0 s) \u00b7 213.0 tok/s \u00b7 ETA 0m 00s \u00b7 bf16 \u00b7 batch 1 \u00b7 ~0.0 s per token per prompt \u00b7 7.6 GB peak\u001b[K\n\u001b[1Frows 90/91 \u00b7 pass 121 (0.0 s) \u00b7 212.9 tok/s \u00b7 ETA 0m 00s \u00b7 bf16 \u00b7 batch 1 \u00b7 ~0.0 s per token per prompt \u00b7 7.6 GB peak\u001b[K\n\u001b[1Frows 90/91 \u00b7 pass 122 (0.0 s) \u00b7 212.8 tok/s \u00b7 ETA 0m 00s \u00b7 bf16 \u00b7 batch 1 \u00b7 ~0.0 s per token per prompt \u00b7 7.6 GB peak\u001b[K\n\u001b[1Frows 91/91 \u00b7 pass 123 (0.0 s) \u00b7 212.0 tok/s \u00b7 ETA 0m 00s \u00b7 bf16 \u00b7 batch 0 \u00b7 ~0.0 s per token per prompt \u00b7 7.6 GB peak\u001b[K\n\n" }, { "command": "spill report dolly", "exit": 0, "seconds": 0.2, "stdout": "# Run run-20261013-170041-12b2-2c7311: dolly\n\nScoring: judge: reference overlap against your outputs, plus judged scores from qwen2.5:1.5b (one model's assessment). Metric: judge score 1-5 (one model's assessment, not accuracy) (metric version 1). Validation rows: 13. Status: completed.\n\n## Results\n\nReference overlap compares each output with the reference you provided. It measures similarity to that one answer, not whether an answer is good, and is not accuracy.\n\n| comparator | token F1 | ROUGE-L | exact match | rows | failures | truncated |\n|---|---|---|---|---|---|---|\n| nearest training answer (TF-IDF, no model) | 0.108 | 0.098 | 0.000 | 13 | 0 | n/a |\n| student, prompted, untrained | 0.441 | 0.419 | 0.077 | 13 | 0 | 1 |\n| student, trained | 0.590 | 0.573 | 0.154 | 13 | 0 | 0 |\n| teacher | 0.459 | 0.426 | 0.077 | 13 | 0 | 0 |\n\n## Judged evaluation (one model's assessment, not accuracy)\n\nJudge: qwen2.5:1.5b, 13 validation rows. The rubric, judge model identity, prompt templates and decoding are part of the protocol fingerprint; comparing runs with any of them changed is refused.\n\n| comparator | judge score (1-5) | rows scored | unparseable | score counts 1/2/3/4/5 |\n|---|---|---|---|---|\n| nearest training answer (TF-IDF, no model) | 2.462 | 13 | 0 | 7/1/1/0/4 |\n| student, trained | 4.538 | 13 | 0 | 1/0/1/0/11 |\n| student, prompted, untrained | 4.846 | 13 | 0 | 0/0/1/0/12 |\n| teacher | reference only | 0 | 0 | excluded: the judge (qwen2.5:1.5b) is the same model as the teacher (qwen2.5:1.5b); a judge may not judge its own outputs, so this row is reported as reference only and excluded from rankings |\n\nPairwise, each pair judged in both orders to cancel position bias:\n\n| pair | win | tie | loss | rows scored | orders disagreed | picked first position |\n|---|---|---|---|---|---|---|\n| student, trained vs student, prompted, untrained | 0% | 85% | 15% | 13 | 8 (62%) | 62% |\n| student, trained vs nearest training answer (TF-IDF, no model) | 67% | 33% | 0% | 12 | 4 (33%) | 64% |\n\nCalibration: 80% agreement with exact match on 15 rows (judge score of 4 or more against the reference appearing word for word in the output, on the 15 rows whose reference has at most 5 words). Judge-only positives: 3; match-only: 0.\n\n\nInvalid, unparseable and failed outputs count as wrong and stay in every denominator. `unavailable` means the metric does not exist for that row; it is never shown as 0.\n\n## What the numbers say\n\nOn 13 validation rows, judge score 1-5 (one model's assessment, not accuracy): nearest training answer (TF-IDF, no model): 2.462; student, prompted, untrained: 4.846; student, trained: 4.538.\nThe trained student scored 2.077 higher than the nearest training answer (TF-IDF, no model).\nThe student, prompted, untrained scored 0.308 higher than the trained student.\nThe judge qwen2.5:1.5b is one model's assessment of each output against the rubric; its scores are not accuracy.\nteacher: the judge (qwen2.5:1.5b) is the same model as the teacher (qwen2.5:1.5b); a judge may not judge its own outputs, so this row is reported as reference only and excluded from rankings.\ntrained vs untrained, both orders: trained won 0, tied 11, lost 2 of 13 rows; the two orders disagreed on 8 (62%); a disagreement counts as a tie.\ntrained vs baseline, both orders: trained won 8, tied 4, lost 0 of 12 rows; the two orders disagreed on 4 (33%); a disagreement counts as a tie.\nCalibration: the judge agreed with the exact-match check on 80% of the 15 rows that have one (judge score of 4 or more against the reference appearing word for word in the output, on the 15 rows whose reference has at most 5 words).\n\nNo significance test was run and none is implied. These are the observed differences on this validation set; a few rows can move a score on a set this size. Changing the batch shape can also move a score through numerical differences; that is a diagnostic, not evidence either way.\n\n## Data\n\nTrain 58, validation 13, final test 13 rows (split generated, seed 0). The final test split was not used by this run and is scored only by `spill test`.\n\n\n## Models\n\n- student: qwen2.5:0.5b (Qwen/Qwen2.5-0.5B-Instruct@7ae557604adf, safetensors bf16)\n- teacher: qwen2.5:1.5b (Qwen/Qwen2.5-1.5B-Instruct@989aa7980e4c, safetensors bf16)\n- judge: qwen2.5:1.5b (Qwen/Qwen2.5-1.5B-Instruct@989aa7980e4c, safetensors bf16)\n\n## How it ran\n\n- distill: done, 14.2 s, mlx on macOS arm64\n- baseline: done, 0.1 s, torch-cpu on macOS arm64\n- train: done, 35.5 s, mlx on macOS arm64\n- eval:untrained: done, 3.7 s, mlx on macOS arm64\n- eval:trained: done, 1.8 s, mlx on macOS arm64\n- eval:teacher: done, 4.4 s, mlx on macOS arm64\n- judge: done, 29.6 s, mlx on macOS arm64\n\n## Where things are\n\n- predictions: predictions/.jsonl; disagreements with the trained student: disagreements.jsonl\n- adapter: artifacts/adapter/\n- manifest (fingerprints, model hashes, dependency versions): manifest.json\n\n## Next\n\n- `spill export dolly`\n- `spill test dolly`\n- `spill compare dolly`\n\nreport: dolly/runs/run-20261013-170041-12b2-2c7311/report.md\nindex: dolly/REPORT.md\n\nnext: spill export dolly\n", "stderr_tail": "" }, { "command": "spill export dolly", "exit": 0, "seconds": 14.1, "stdout": "spill export dolly: merge the run's LoRA adapter into its bf16 base, then load the artifact in an independent runtime on 8 validation rows. Cost: $0. Record -> dolly/exports/\n merging the adapter into the bf16 base (float32 arithmetic, rounded back to bf16)\n merged shard 1/1: model.safetensors\n verifying safetensors with Transformers on 8 validation rows\nverified safetensors: transformers 5.19.0, torch 2.14.1, float32 on CPU (12 cores); 5 of 8 predictions differ from the training engine's. Quality on these 8 rows (token F1 against the reference (overlap, not accuracy)): source engine 0.596, artifact 0.487, change -0.110. Verified means it loaded and ran, not that quality was kept\n time to first token cold 0.1416 s, warm 0.0952 s; 47.99 tokens/s; peak memory 3.41 GB\nrecord: dolly/exports/export-20261013-170212-93cec1\n\nnext: python dolly/exports/export-20261013-170212-93cec1/run_safetensors.py \"\"\n", "stderr_tail": "" }, { "command": "python dolly/exports/*/run_safetensors.py \"...\"", "exit": 0, "seconds": 3.0, "stdout": "Hironobu Sakaguchi created Final Fantasy.\n", "stderr_tail": "\nLoading weights: 0%| | 0/290 [00:00