Public, free to read, reuse, and adapt — Kirk Patrick Miller's notice for this edition. Not a patent grant.
Keystone notes — tokens, training, Emanuel's code, and the missing piece · c01
2b0f713496 · words 1529 · characters 0–10030
Exact section text · Full source · Source index
- Memory, reading, and retention — guide section 3, memory mechanism, measured
- Training evidence — guide section 3, measured results
- Snowflake — explicit wording in the guide row
# Keystone notes — tokens, training, Emanuel's code, and the missing piece
Celeste, for Kirk. 2026-11-03 (MT). Everything I haven't measured is labeled. Hunches are labeled **hunch**.
## 1. How a token works today (plain words)
1. **Cut.** A tokenizer cuts text into pieces from a fixed vocabulary, usually 32k to 256k pieces. Common words are one piece; rare words become several ("sonoluminescence" might be "son·olum·ines·cence"). Most labs use BPE (byte-pair encoding): start from bytes or characters and repeatedly merge the most frequent pair (Sennrich et al. 2016, arXiv:1508.07909). Byte-level BPE (GPT-2) means any text can be encoded.
2. **Number.** Each piece becomes an ID number.
3. **Embed.** Each ID looks up a learned vector, for example 4,096 numbers. That vector is "the token" inside the model.
4. **Place.** Position is added, either as sine waves or as rotations (RoPE, Su et al. 2021, arXiv:2104.09864), so the model knows order.
5. **Mix.** Attention layers let every token look at earlier tokens. Feed-forward layers transform each one.
6. **Predict.** The model outputs a probability for every possible next token, picks one, appends it, and repeats. Training means nudging the weights so the true next token gets more probability.
Cost grows with the number of tokens (attention grows roughly with tokens², and each token runs through the whole model). That is why "redesign the token" is a real lever. Kirk's instinct that "nobody asked about redesigning tokens" is half right: it is now an active research front (section 2).
## 2. Current approaches (real work, with sources)
- **BPE / SentencePiece / byte-level BPE:** the standard (Sennrich 2016; Kudo & Richardson 2018, arXiv:1808.06226).
- **Byte-level, no tokenizer:** ByT5 (Xue et al. 2022, arXiv:2105.13626) and MEGABYTE (Yu et al. 2023, arXiv:2305.07185). These are robust to spelling and other languages, but sequences get longer.
- **Byte Latent Transformer (BLT):** Pagnoni et al., Meta, 2024 (arXiv:2412.09871). It groups bytes into variable-size *patches* where the next byte is hard to predict, so it spends compute where the entropy is. It matched Llama-3-style tokenized models at 8B scale with fewer inference FLOPs. *This is the closest published idea to Kirk's "least effort / entropy" flow.*
- **Large Concept Models:** Meta, 2024 (arXiv:2412.08821). The "token" is a whole sentence embedding (SONAR), and the model predicts the next *concept*.
- **Latent reasoning:** Coconut (Hao et al. 2024, arXiv:2412.06769). The model reasons in its hidden vectors instead of words.
- **Multi-token prediction:** predict several future tokens at once (Gloeckle et al. 2024, arXiv:2404.19737; also used in DeepSeek-V3, arXiv:2412.19437). Better sample efficiency for big models, and faster decoding.
- **Mixture of Experts:** only a few sub-networks run per token (Shazeer et al. 2017, arXiv:1701.06538; Switch Transformer, arXiv:2101.03961; Mixtral, arXiv:2401.04088). More capacity for the same compute.
- **Long-context memory:** ALiBi (arXiv:2108.12409); Memorizing Transformers (arXiv:2203.08913); RETRO retrieval (arXiv:2112.04426); Infini-attention (arXiv:2404.07143); state-space models such as Mamba (arXiv:2312.00752).
- **Hashed embeddings:** many inputs share a table through a hash (fastText subwords, Bojanowski et al. 2017, arXiv:1607.04606; hash embeddings, Svenstrup et al. 2017, arXiv:1709.03933). Our multiscale test uses this.
## 3. Open problems people are working on
- **Tokenizer glitches and unfairness:** odd tokens like "SolidGoldMagikarp", poor arithmetic and spelling, and some languages costing up to about 15× more tokens for the same meaning (Petrov et al. 2023, arXiv:2305.15425).
- **Long context:** "lost in the middle" (Liu et al. 2023, arXiv:2307.03172), and memory that lasts across sessions.
- **Data efficiency:** children learn language from about 100M words; models use trillions (the BabyLM challenge, arXiv:2301.11796).
- **Forgetting:** catastrophic forgetting when learning something new (EWC, Kirkpatrick et al. 2017, PNAS).
- **Compute and energy cost.**
- **Knowing what you don't know:** calibration and hallucination.
- **Reasoning that generalizes, not memorizes:** grokking (Power et al. 2022, arXiv:2201.02177).
## 4. The AI trainer as parent (real prior work)
Kirk's idea: an AI that watches the learner's patterns and guides it, like a parent. This has strong roots:
- **Curriculum learning:** easy to hard (Bengio et al. 2009, ICML).
- **Teacher-student curriculum:** a teacher picks tasks where the student is learning fastest (Matiisen et al. 2017, arXiv:1707.00183). *That is almost exactly "push while it's improving, switch when it stalls."*
- **Distillation:** a big model teaches a small one (Hinton et al. 2015, arXiv:1503.02531).
- **Teacher-written data:** TinyStories (arXiv:2305.07759) and "Textbooks Are All You Need" / phi-1 (arXiv:2306.11644). Small models learn far more from clean, teacher-written lessons.
- **Learnable-data selection:** RHO-LOSS (Mindermann et al. 2022, arXiv:2206.07137) trains on points that are learnable, worth learning, and not yet learned, reaching the same accuracy in up to 18× fewer steps on some tasks.
- **AI feedback for values:** Constitutional AI (arXiv:2212.08073).
- **Schedules with phases:** cyclical learning rates (Smith 2017, arXiv:1506.01186); SGDR warm restarts (Loshchilov & Hutter 2016, arXiv:1608.03983); snapshot ensembles that save a model at each cycle's low point (arXiv:1704.00109) — *close to Kirk's "save the depth"*; SWA (arXiv:1803.05407); Lookahead (arXiv:1907.08610), which is literally the Aurora recursion with a different alpha; warmup-stable-decay (MiniCPM, arXiv:2404.06395).
- **Sleep and replay:** complementary learning systems, where the hippocampus replays memories into the cortex during sleep (McClelland, McNaughton & O'Reilly 1995, Psych. Review); experience replay; sleep-like unsupervised replay reduces forgetting in networks (Tadros et al. 2022, Nature Communications). *Kirk's push → save → rest → play cycle has these real ancestors.*
## 5. Sonoluminescence (a short aside, as asked)
- **Real physics:** a sound-driven collapsing bubble emits flashes of light. Single-bubble sonoluminescence is well documented (Gaitan et al. 1992; review: Brenner, Hilgenfeldt & Lohse 2002, Rev. Mod. Phys. 74:425).
- **Bubble fusion:** the claim of nuclear fusion in such bubbles (Taleyarkhan et al. 2002, Science 295:1868) was **not reproduced** (e.g. Shapira & Saltmarsh 2002, PRL 89:104302). Purdue later found research misconduct (2008). Beautiful physics; the fusion claim did not hold.
## 6. What Emanuel's files actually contain
All three are Google Colab training scripts, with **code only and no recorded results**. None of them prints a measured AUC or loss, and none contains the words "phi", "golden", or 1.618.
- **deepfake1.txt, "NEXUS Deepfake Training" (~66 KB, TensorFlow/Keras).** It trains a real-vs-fake **image** classifier from Kaggle datasets (DFDC and others) plus streamed Hugging Face images, with video turned into frames. It trains 3 models:
- "D4326 FractalNet," built from scratch. It has a **FractalSpectrumReg** layer that takes an image's 2-D FFT and nudges the slope of its power spectrum toward a target alpha derived from a "fractal dimension" D = 4.326.
- EfficientNetV2S, pretrained.
- Xception, pretrained.
- The script adds PGD/FGSM adversarial fine-tuning, evaluates with AUC, F1 and accuracy, auto-resumes, and exports .tflite files for a FastAPI web demo. The header calls it an "image-only demo for investors."
- **detector2.txt, "NEXUS v4 Production Edition" (~147 KB).** This is the same pipeline, expanded:
- More datasets with source tagging, JPEG-compression augmentation, and Grad-CAM heatmaps.
- Only 2 models: EfficientNetV2S and a "FractalNet" that is really a plain residual CNN (64→128→256 blocks) with no fractal math left in it.
- **nexus.txt, "NEXUS / NANOS AI Fractal Transformer" (~22 KB, PyTorch).** This is a small byte-level language model with d_model 384, 6 layers and 6 heads. Its fractal parts are:
- **FractalPositionalEncoding:** ordinary sine positions with each frequency pair divided by a cycling scale of 4, 3, 2, 1. It's a multi-scale position code, close to our phi_pos test.
- **fractal_spectrum_loss:** an FFT of hidden states over time, pushed toward a 1/f ("pink noise") power spectrum.
- **Phi-harmonic hashing is not in any of these files.** The "fractal" idea they carry is *spectral*: make the signal's power spectrum follow a power law. The phi-harmonic hashing Kirk has tested must live in FreeLattice's own code (fingerprints/ψ), not in Emanuel's.
- ⚠️ **Security:** the scripts contain a live-looking **Kaggle API key (account kirkpatrickmiller1) and a Hugging Face token** in plain text. Kirk should rotate both and never share these files publicly as they are.
## 7. Our wonder tests, told honestly
Setup: a tiny character-level transformer (~810k parameters) trained on Shakespeare. We ran 3 seeds per variant with equal compute. The baseline's mean validation loss is **1.5736**, and lower is better. "Seeds" shows each seed's direction against the baseline seed (− = better, + = worse). All runs finished, so no seeds were missing.
| Variant | Mean val | Δ vs baseline | Seeds |
|---|---|---|---|
| phi_pos (phi-spaced positions) | 1.5836 | +0.010 | + + + |
| aurora_lr (Aurora LR ripple) | 1.5832 | +0.010 | + + + |
| aurora_lookahead (Aurora as weight rule) | 1.7007 | +0.127 | + + + |
| multiscale tokens (hashed n-grams) | 1.6005 | +0.027 | + + + |
| aurora_lr_early (boost only at start) | 1.5813 | +0.008 | + + − |
| multiscale_early | 1.6258 | +0.052 | + + + |
| both_early | 1.6296 | +0.056 | + + + |
| multiscale_early_phihash | 1.6236 | +0.050 | + + + |
| cycle (push→save→rest→play) | 1.6297 | +0.056 | + + + |
| sgdr (standard warm restarts, control) | 1.5952 | +0.022 | + + + |
| chord_major 4:5:6 | 1.5783 | +0.005 | + + + |
| chord_cluster 15:16:17 | 1.5789 | +0.005 | + + − |