Nihonga 日本画

Phase 3 · measure the missing update

Similarity is a clue; images decide what it misses

We measured all 32 transformer blocks before choosing which ones to replace. On 64 prompts, at three denoising stages, we compared the hidden states entering and leaving every block. Blocks 2, 3, 4 and 5 had the smallest consistent changes, so we chose them as bridge candidates. The choice came from the measurements.

To make that choice, we took each block’s average relative L2 change at each of the three stages, kept the largest of those three averages, and ranked all 32 blocks from smallest to largest. The first four were blocks 2, 4, 5 and 3. After selecting them, we temporarily skipped each of those four blocks separately to measure how much that changed the model’s velocity prediction. The all-block activation comparison and the four-candidate bypass tests are two different experiments.

The initial candidate ranking

This rule favors blocks with consistently small updates across the sampled denoising stages, rather than a block that appears unimportant at only one stage. It is a ranking of local hidden-state changes, not yet bypass velocity errors or image quality. “Largest stage average” means the largest of three means over 64 prompts; it is not the worst individual prompt. The initial rank order is 2, 4, 5, 3; written in model order, the selected set is 2–5.

The activation measurements

We measured all 32 blocks on 64 fixed internal prompts at denoising steps 0, 19 and 39 of the 40-step teacher: 6,144 block/stage observations. Hooks compare image-token states immediately before and after each block, excluding the text prefix. Each cosine is averaged over image tokens, then across prompts. The table shows the four tested candidates; the graph and downloadable record cover every block.

• early · dashed
• middle · dotted
• final · solid

These views zoom into blocks 1–6, including a neighbor on each side of the tested 2–5 batch. The cosine axis is deliberately narrowed to 0.994–1.000; its larger-looking differences are still small absolute changes. Relative L2 is ‖h_out − h_in‖₂ / ‖h_in‖₂, reported as a percentage on a separate linear axis. Cosine compares direction; L2 also responds to changes in magnitude. The green wash identifies the tested batch, not an acceptance region.

Show all 32 blocks · full-range cosine and logarithmic relative L2

The L2 overview uses a log scale: equal vertical distances represent equal ratios rather than equal percentage-point changes. It preserves block 0’s update, which exceeds 800% of its input norm at the early stage, while keeping other blocks visible. All plotted observations fit the stated axes; none are clipped. The stage legend above applies to both overviews.

For example, block 2’s early cosine is 0.998321, but the update still has a norm equal to 8.681% of its input’s norm. High cosine does not imply a negligible update or 99.8% image quality. Block 1 and some later blocks also have high cosine; the shortlist was determined by the maximum stage-average relative L2 change, rather than cosine alone. These four earn a first test, not automatic approval for removal. Prompt-level distributions and per-capability aggregates are in the measurements.

What happens when each update is removed?

An identity bypass returns h_out = h_in, so the missing block contributes no update. We tested each candidate separately on the same five pilot prompts × three stages, using saved teacher-trajectory inputs and rebuilt student prefix caches. Each block has 15 velocity comparisons, with intact-teacher replay controls matching exactly within the diagnostic. These five-prompt Phase 3 probes are distinct from the later ten-prompt bridge-validation set.

Block 2 has the strongest activation similarity of the tested candidates early and in the middle, yet its omission causes the largest mean velocity errors at all three stages. Block 4 has the lowest mean bypass error early and in the middle; block 5 is lowest at the final stage. Similarity helps nominate experiments, but bypass sensitivity and finished images change the ranking.

The worst-case column is the largest error among that block’s 15 prompt/stage cases. Velocity error is a prediction difference, not a percentage of image-quality loss. These individual interventions do not establish that several blocks can be removed together, and no bridge is fitted in this diagnostic.

What the first image comparisons showed

Five prompts, one per capability, were generated with block 4 or block 5 skipped: ten bypass images. In the astronaut example, skipping block 4 loses the spacesuit. Skipping block 5 retains it. That initially favors block 5 for closer study, while all four candidates remain eligible for learned bridges.

Each sheet reads teacher / skip block 4 / skip block 5, left to right. These are identity-bypass images from Phase 3, before bridge training or healing. Teacher images came from an earlier run; unresolved cross-run variation limits causal pixel comparisons. Observations are manual, not a blinded quality study. Open any sheet at full resolution.

Pixel MAE and PSNR record changes in composition and appearance, but do not directly measure prompt adherence or aesthetic quality. None of these identity bypasses is accepted as the finished model.

Phases 4–5 · recover the missing update cheaply

A small residual bridge in place of a full block

The bridge projects 4,096 features down to 256, applies GELU, projects back up, and adds the input: h′ = h + W_up GELU(W_down h). Each bridge has 2,097,152 parameters. We fit four independent candidates, replacing one original block at a time; the four bridges are not inserted together.

The student keeps the same 4,096-feature input and output interface. Only the correction travels through the 256-feature bottleneck; the residual path keeps the original information. GELU is a nonlinear transformation of those 256 features. The bridge processes each token separately without attention and learns to approximate the removed block’s update. The diagrams show shapes and operations, not identical teacher and student values or measured speedups. Open either diagram for the standalone SVG.

What this bridge sees, and what it ignores

The rank-256 bridge receives exactly one thing: each token’s hidden state. It never looks at other tokens, does not know which denoising step it is in, and has no notion of where a token sits in the image. That is enough for block 5, whose update is small, but not for a block like 13. The attention bridge designed in Phase 7 adds each missing input.

Bridge-only pretraining keeps the rest of the model frozen. Each candidate gets 500 updates using cached hidden states from 20 training prompts, with 10 validation prompts selecting checkpoints. The fitting objective balances prompt tokens and early, middle and late image tokens, combining normalized direction matching with a smaller relative squared-error term. Block 5’s selected checkpoint is update 300.

Does a better hidden-state fit improve the whole prediction?

We replay each identity and trained student against the verified teacher, using the same saved inputs at three stages. Each candidate has 30 comparisons; there are 240 student forwards across four candidates and two variants.

• identity bypass
• pretrained bridge

How to read the percentages. Velocity error is ‖student − teacher‖₂ / ‖teacher‖₂, averaged equally across prompts and stages. For block 2, 12.8788036% becomes 12.0234947%: a drop of 0.855309 percentage points. Dividing that drop by the original 12.8788036% gives a 6.64% relative reduction. This is a fraction of prediction error removed, not an image-quality improvement. “27 / 30” counts the cases with lower error.

Block 5 has the lowest mean trained error, 5.518%, and a worst case of 15.095%, so it is selected for the healing pilot. Block 4 removes a larger fraction of its identity error but ends at a slightly higher mean, 5.873%. Block 3 slightly regresses at the early and middle stages despite improving its overall mean. The small reused validation set does not establish statistical superiority.

Phase 6 · first make the supervision repeatable

A replay discrepancy paused training

Fresh teacher predictions initially disagreed with historical saved velocities beyond our 0.1% relative-L2 tolerance. Repeating a forward in the same process was stable, and full tensor-state fingerprints matched across fresh processes. The exact cause of the historical discrepancy remains unresolved.

The numerical contract we verified

We pinned the model revision and transformer source, H100 hardware and software versions, BF16 weights, deterministic algorithms, math attention, TF32 and reduction settings, exact input order and cache lifecycle. Each prompt gets a fresh cache; early-stage conditioning is extracted before cached middle and late forwards. Reference files are loaded after predictions are made. The audit covers 301 state tensors, including otherwise unregistered positional tables.

Under this contract, all 90 teacher predictions match the new saved references exactly. Historical targets and weights remain preserved. This establishes a working repeatable setup; it does not isolate which individual setting caused the old mismatch or prove that every setting is necessary.

Did the bridges need to be trained again?

We captured 360 fresh block/stage hidden-state pairs for the same 30 prompts and token selections. Frozen bridge validation objectives differ from the historical objectives by less than 0.05%. The replay change alone therefore does not justify replacing the Phase 5 training results.

The hidden-fit objective combines direction and relative squared errors; it has no direct percentage interpretation. Its 38–61% reductions cannot be compared as if they were the same metric as the 3.74–17.97% velocity-error reductions.

Phase 6 | architecture healing

Full-student healing and generated-image QA

The original teacher has 32 transformer blocks. The current student has 31 surviving blocks and one pretrained bridge at slot 5. Phase 5 trained that bridge to replace the missing block's hidden-state update. Phase 6 then tested whether further training helps the student match the teacher's denoising velocity at the same prompt, noisy latent and timestep. Both models still use 40 denoising steps.

The intended healing pass updates the entire student: all surviving transformer weights and the single bridge, 6,899,117,824 parameters in total. The teacher stays frozen. We attached no LoRA in the full-student runs. A separate LoRA for few-step speed distillation belongs to a later phase.

All five pilots began at 5.518% with the pretrained bridge. Every saved training run selected update 0 on the original 10 prompts. The bridge + LoRA and bridge-only trials are diagnostic ablations; the two full-student trials answer the intended healing-scope question. A passed run means its execution controls passed, not that its trained checkpoint improved quality.

How the full-student trials worked

Each run made 60 shuffled updates from 20 training prompts at three saved noise stages. The loss combined teacher-student velocity error, hidden-state alignment at surviving slots 2-4, and bridge output alignment at slot 5. Ten different validation prompts chose among updates 0, 30 and 60. Exact teacher replay, baseline reproduction, gradient flow to the bridge and surviving weights, and selected-checkpoint reload were checked. Checkpoints, optimizer state, predictions, reports and hashes were saved separately.

At update 60 the 0.0001 run reached 54.387%, a severe regression. Reducing the rate 100-fold kept the run near baseline, but its 5.654% still exceeded 5.518%. These short runs therefore did not produce a better healed student. They do not establish whether broader data, different loss weights or another optimizer could help.

A closer look at the lower-rate run shows why its mean can rise despite local gains: 21 of 30 validation prompt/stage comparisons improved, but the early-noise average rose from 9.947% to 10.371%. The largest regressions were early-noise cases. This is a small, reused validation set; the pattern guides diagnosis but does not identify a cause.

A broader check of the selected bridge

We tested the unchanged pretrained-bridge student on 50 additional validation prompts, 10 from each prompt group, at early, middle and late noise stages. The original 10-prompt calibration measured 5.531% in this replay, close to the earlier 5.518%. The new 50-prompt mean was 5.393%. No student weight was updated.

This extends the selected bridge's fixed-input check to 150 new prompt/stage comparisons. It is still a validation set, not an independent image-quality benchmark. Early noise remains the largest source of teacher-student disagreement.

The trained checkpoint on the same new inputs

The low-rate update-60 checkpoint still regresses on the original 10 prompts: its control mean was 5.655%. On the 50 new prompts, however, it improved mean teacher agreement from 5.393% to 5.251%; 121 of 150 paired prompt/stage comparisons improved. The new prompt selection was fixed before this checkpoint comparison. The disagreement between samples makes the original 10-prompt selection uncertain.

This is teacher-prediction agreement on saved noisy inputs, not an image-quality result. The separate matched-image pilot below examines what these numerical gains look like after full generation.

Longer full-student healing

We resumed the low-rate full-student checkpoint at update 60 with its saved optimizer state. Forty previously unused training prompts provided 120 new prompt/noise-stage updates; no validation prompt entered training. All 6,899,117,824 student weights were trainable, with one bridge and no LoRA. On the 50 monitored validation prompts, mean velocity relative L2 fell from 5.249% at update 60 to 5.139% at update 120 and 5.110% at update 180.

On 20 separate prompts untouched by training and checkpoint monitoring, the means were 5.439% pretrained, 5.427% at update 60, 5.398% at update 120, and 5.406% at update 180. Update 120 was best on this small test; the extra updates to 180 did not improve it further there.

These are teacher-prediction differences on saved noisy inputs. Early-noise agreement worsened with more updates while late-noise agreement improved; the overall test gain is small. The image comparison below tests full generations separately.

What the generated images show

The first pilot generated 10 fixed-seed teacher/pretrained-bridge image pairs. A later pilot generated 11 matched triples with teacher, pretrained bridge, and low-rate update 60. The two student versions usually retain similar subjects and styles; their composition and detail differences are mixed. In the poster below, the teacher renders the requested CRIMSON HARBOR title while both students replace HARBOR with unrelated letters. The trained checkpoint does not fix this known text failure.

This visual review is small and unblinded. It does not establish a quality ranking: a lower teacher-velocity error on saved inputs has not shown a clear generated-image advantage here. Full-resolution PNGs are available by opening each poster.

Five-way image comparison after more updates

Eleven fixed-prompt, fixed-seed cases now compare the teacher, pretrained bridge, and cumulative updates 60, 120, and 180. Only two prompts request prominent text. The Northern Parade title is legible in every student version. In the other poster, update 120 renders CRIMSON HARBOR and Visit Kyoto legibly; the pretrained and update-60 versions miss HARBOR, and update 180 reads SIMSON and omits HARBOR. That is not a steady text improvement or deterioration as healing proceeds. Replacing block 5 with the bridge harms this particular poster relative to the teacher, but two posters cannot establish general bridge text fidelity. Most other image differences are subtle or mixed.

The viewer below shows all 11 prompts: four on the first sheet, four on the second, and three on the third. Each row holds the same prompt and seed across the five columns: teacher, pretrained, update 60, update 120, update 180. Open a sheet for full resolution.

This inspection was unblinded and includes one previously tracked poster. Only two of these 11 prompts request prominent text. The fresh text-focused test below checks whether that poster result repeats.

A fresh test of the update-120 text hypothesis

We rendered 16 previously unused explicit-text prompts: eight posters and eight other text scenes. Each uses the same fixed prompt and seed for teacher, pretrained bridge, update 120, and update 180, producing 64 saved images. The rubric was written before image review: a prompt passes only when every requested phrase is readable; ambiguous lettering remains uncertain. No weights were updated.

Update 120 clearly restores the Northern Orchard title that the pretrained bridge and update 180 omit, although its small Visit Kyoto tagline is borderline. On a fresh ENDLESS HARBOR / Visit Kyoto poster, all three students render both phrases correctly, so the CRIMSON HARBOR failure did not repeat on that close template. Update 180 breaks Midnight River, which pretrained and update 120 render correctly. The strict full-prompt score is tied between pretrained and update 120 at 13 clear passes; update 120 has one additional uncertain case. This supports a local recovery and an update-180 regression, not a general claim that update 120 fixes text.

The viewer below shows all 16 prompts, four per sheet, with the four models in each row. The ENDLESS HARBOR, Northern Orchard, and Midnight River update-120 PNGs can also be opened at full resolution; each sheet links to its full-size image. The review is unblinded and uses one seed per prompt.

The observed 40-step generation averages on these 10 ordered pairs were 7.03 seconds for the teacher and 5.95 seconds for the student, a 1.18x ratio. This ordered pilot does not control for run order or establish a production speed benchmark. Velocity error measures denoising predictions, not image quality.

All 45 remaining validation text prompts

To check whether update 120 consistently improves text, we rendered all 45 other unused validation prompts with explicit text. The teacher, pretrained bridge, update 120, and update 180 produced 180 matched 40-step images at one fixed seed per prompt. A written rubric required every requested phrase to be readable. One AI visual reviewer scored anonymous A/B/C/D columns before revealing model identities; uncertain text was recorded separately. This step trained no weights.

The pretrained bridge and update 120 received the same status on every prompt: 41 clear passes, three failures, and one uncertain. Update 120 clearly beat update 180 on two posters, PAPER STATION and NORTHERN PARADE. The teacher had 43 clear passes. This broader qualitative check gives no general text-rendering advantage to update 120 over pretrained, despite its CRIMSON HARBOR recovery in the earlier image pilot.

All four models garbled tiny Moonlight Motel print in one storefront and missed the Live in Berlin tagline on GOLDEN RIVER. Three student images spelled Fresh Bread Daily with an extra letter. The Electric Frontier flag lettering was too small to judge confidently in three student images. Literal text scoring also misses layout mistakes: one image has Live Music Tonight at the top but garbled lettering where the prompt requested it in sand. The viewer below contains all 15 labeled sheets, three prompts per sheet. One seed and one AI reviewer do not establish a population-level quality ranking; the separate internal test prompts remain unused.

These 45 prompts mainly ask for exact words on signs, posters, menus, or other surfaces. Turning each into a visual question would mostly repeat the text score. The broader 1,002-prompt validation split also includes 260 alignment prompts with counts, colors, objects, and spatial relations that support more specific question-based checks. Aesthetics and fine-detail prompts need different evidence; one question score cannot rank overall image quality.

Deciding test: 129 blind prompt-adherence questions

Velocity agreement and text tests had not clearly separated update 120 from the pretrained bridge, so one last test asked whether update 120 follows concrete prompt requirements more often. We used every unused validation prompt in four checkable categories: counting, spatial relations, colors, and multiple objects. That is 129 prompts in English and Chinese. Each was rendered by both models with the same seed, size, and 40-step sampler: 258 matched images and no training. Each category had one fixed question, such as "Are the requested number(s) of each named object visible?" Images were scored pass, fail, or uncertain as anonymous A/B columns before the model key was opened. The promotion rule was saved before any image existed.

• 83 vs 86clear passes out of 129: pretrained bridge vs update 120
• 126 / 129prompts where both models got the same result
• 3 – 0clear paired wins for update 120 vs pretrained
• +2.3 ptsgain; 95% CI [0.0, 5.4]; bar was +5 with CI above 0

1. Pretrained bridge (selected)83
2. Update 120 (healed)86

What the differences look like

Below are all five prompts where the two models scored differently, followed by two typical failures shared by both and one typical tie. The pretrained bridge is always on the left and update 120 on the right; tags show the blind scores. Two of the three wins hold up: the right count of octopuses and a missing candle restored. The giraffe was scored the wrong way round; on human review the pretrained bridge is the one that hides the neck behind the mug (corrected below). Against 129 prompts, however, they are too few to clear the predeclared bar. Most prompts look like the last example: almost the same image, same verdict.

Both checkpoints still fail about half of the counting prompts and about a third of the spatial-relation prompts in the same way. Those errors come from the base model and the bridge, not from healing. The viewer below shows all 129 prompts unblinded, three per sheet, with the score for every image.

Research decision: all Phase 6 evidence side by side

Every comparison points the same way. Update 120 is numerically a little closer to the teacher on saved inputs: lower velocity error on 46 of 60 untouched comparisons, by 0.04 points. It ties on text and is not reliably better on generated images. Phase 6 therefore closes with the pretrained single-bridge student as the selected model. The full-student checkpoints remain saved as evidence, but none is promoted.

Limits: one seed per prompt and one AI visual reviewer. A real effect of a few points cannot be ruled out, but this test was sized to detect the 5-point gain needed to justify the extra healing training. Per-prompt scores, the rubric, the blind key, and hashes are in the downloadable record.

Then: remove all four blocks at once

Phase 6 removed one block, and the pretrained bridge alone was enough. Phase 7 tested the four original candidates together, then the smaller cut of blocks 4 and 5. Importance is now recalculated inside the smaller student before choosing another removal. Healing remains conditional on a measured gap.

The LoRA in this plan is for speed. After architecture evaluation, a speed-distillation LoRA will teach the evaluated student to follow the teacher's 40-step trajectories in about 8 larger steps. That is roughly 5x fewer transformer passes; actual end-to-end speed must be measured.

Phase 7 | adaptive architecture pruning

A two-block cut that fails blinded review. Reassess from one block

Removing one block (Phase 6) is too small a cut to be worth releasing. Phase 7 tried bigger cuts. We first removed all four candidate blocks, 2, 3, 4 and 5, going from 32 blocks to 28, then tried a smaller cut of only blocks 4 and 5. Each student is compared with the teacher on the same noisy inputs. Its velocity error is how different its denoising answer is from the teacher's, where 0% means identical. The reference is the student with only block 5 removed. The predeclared gate allows at most 1 point more than that student on 10 check prompts, which is 6.52%.

Four blocks, bridges only: not enough

We tried the cheap options first: the four Phase 5 bridges stacked, and one bridge for the whole removed region at two sizes. All 90 teacher predictions were matched exactly first.

1. One-block bridge (Phase 6 choice) · 31 blocks5.518%
2. Blocks 2-5 skipped · 28 blocks22.155%
3. Four per-block bridges stacked · 28 blocks19.641%
4. One region bridge, 2.1M parameters · 28 blocks18.930%
5. One region bridge, 8.4M parameters · 28 blocks17.731%

In the intact teacher, replacing blocks 2 and 3 individually caused more error than replacing blocks 4 and 5. These measurements describe that original architecture; they do not rank the surviving blocks after other removals:

The region bridge learns the text-prompt tokens almost perfectly, but leaves about 12% of the image hidden state unexplained. A bridge transforms each token on its own, while the removed blocks mix all tokens through attention, so quadrupling its size barely helps:

Four blocks, healed: better, but not close

Healing trains all 6.25 billion surviving weights and the bridge against the teacher. The first attempt gave each training step one prompt at one noise moment. The early moment, the first denoising step where the layout is decided, has about ten times larger errors and gradients, and it broke. Giving every step all three moments with equal weight fixed that at small learning rates, and 3e-6 was selected from a four-rate grid. Further passes then leveled off. On the 60 training prompts the early moment improved by about 60%, but on new prompts by only 4%: with just 60 saved early inputs, the student memorized them. We captured 600 extra early-moment examples, each verified by two identical deterministic teacher passes. They stopped the memorizing, but the early moment stayed unstable.

1. Training step 016.63%
2. Training step 6015.16%
3. Training step 12014.59%
4. Training step 18014.26%
5. Training step 24014.21%
6. Training step 30013.78%
7. Training step 36013.47%
8. Training step 42013.24%
9. Training step 48013.61%
10. One-block student (target)5.39%

Two blocks: much closer

So we tested the smaller cut, removing only blocks 4 and 5, which were the easy ones on their own. One bridge maps the teacher's state before block 4 to its state after block 5. It was fitted with the same recipe, then measured in the full 30-block student:

The best bridge, with 4.2M parameters, reaches 7.97%, 2.45 points above the one-block student. Healing it with the four-block recipe improves the middle and late moments steadily, but the early moment gets worse after step 60:

What the images show

The gate measures answer agreement, not pictures. So we generated the same 30 validation prompts with the teacher and four candidates: 25 prompts never used before, plus 5 text prompts. Every model got the same prompt, seed, size and 40 steps. A, B and C are the 30-block students, healed at steps 60 and 240 and bridge only. D is the healed 28-block student.

Failures: D on "GOLDEN RIVER" / "Live in Berlin": "GOLDLEN RAVER"; D on "Welcome Home": "Wolome Home"; D on "Happy Birthday Emma": "Happy B'rday Emma"; D on 营业中: wrong first character; B on "The Rusty Anchor" + small "Moonlight Motel" sign: "The Rusty Anchar", sign has no text; C on "The Rusty Anchor" + small "Moonlight Motel" sign: sign has no text; D on "The Rusty Anchor" + small "Moonlight Motel" sign: "The Rusty Coit", sign has no text.

The image review is a first screen by one unblinded reviewer (Claude), on 30 prompts with one seed each. It is not a blinded benchmark. Removing two of 32 blocks is a modest cut, and its actual speed gain has not been measured. Candidate A still fails the historical velocity gate; a larger blinded comparison remains required.

Candidate A under closer test: rejected

The plan was to use candidate A (blocks 4-5 replaced, 30 blocks) as the base for further pruning. Before building on it we measured three things: its image quality in a larger blinded test, its real speed, and which block could go next inside it. The blinded image test decided the question.

Blinded image test: candidate A loses to the teacher

Before viewing images we registered 159 fresh validation prompts (35 counting, 34 spatial, 30 color, 30 multiple-object, 30 text), one seed each, rendered by the teacher and A at 40 steps. Each image was scored pass, fail or uncertain against one fixed question, with left/right positions randomized and the identity key committed by hash and revealed only after all 318 scores were committed.

One model-blind AI reviewer and one seed per prompt. The 10 and 50 validation prompts used for velocity are development data, not an untouched final test.

Measured speed of candidate A

30 balanced, randomized teacher/A pairs on one H100, 1024×1024, 40 steps, BF16, same prompts and seeds, warm models. Median end-to-end time: 21.91 s for the teacher and 20.58 s for A. Paired speedup 1.064× (bootstrap 95% CI 1.063-1.065). Peak allocated memory: 36.93 GiB vs 36.14 GiB.

Which block next inside A? The importance scan

A alone measures 7.49% on all 180 comparisons (7.40% on the 50 wider prompts and 7.97% on the 10 check prompts, reproducing the earlier 7.41% and 7.97%).

Which block next? Importance measured inside the accepted student

The original ranking measured blocks in the intact teacher. After a cut, surviving blocks may compensate or become more redundant, so we measure again inside the smaller student. Each trial bypasses one surviving block, compares the prediction with the teacher on the same 60 development prompts at early, middle and late moments (180 comparisons), then restores the block. Bypasses never accumulate and no weight is updated. Before any trial, all 90 intact-teacher controls reproduced exactly and the student reproduced its saved predictions.

Because A is rejected, its scan describes a model we will not build on. The same scan is repeated inside candidate B: the accepted one-block student, 31 blocks plus the pretrained block-5 bridge. Block 4 is eligible again in B. Everything after this point starts from B.

Candidate B alone measures 5.40% against the teacher on these 180 comparisons (A: 7.49%). Each row adds one bypass to B; the last columns show the same block bypassed inside A.

Learned bridge for block 13: the velocity gates fail

Block 13 ranked first inside the accepted student, so we replaced it with a rank-256 bridge while keeping the block-5 bridge and every other weight frozen. Pass rules were committed before the run. The bridge was fitted to block 13’s own inputs and outputs captured inside the student: 60 training prompts at three moments plus 600 early training examples. The 10 check prompts were used only to choose the checkpoint. The fit worked on its own terms: the hidden-state objective fell from 0.0294 (plain skip) to 0.0182, a 38% reduction.

Qualitative image check. Because the teacher is not a perfect reference, we also rendered the teacher, B and C on the 30 earlier Phase 7 prompts with matched seeds (the regenerated teacher images matched the earlier renders byte for byte) and judged each image against its prompt. C was worse than B on 15 prompts, similar on 13 and better on 2. The failures appear at unchanged composition: hair smearing into the jaw, circuit-board traces dissolving, an incoherent house cross-section, misspelled signs (“Happy Birtliday Emma”, “The Rusty Achor”) where B is exact, and a missing alarm clock. B stays close to the teacher.

Unblinded review by an AI visual reviewer, one seed per prompt; a screen, not a benchmark.

Better bridges for block 13: conditioning helps, attention helps more

Every real block receives the model’s shared timestep modulation (scale and gate values that tell it how noisy the image is); the first bridge ignored it, and its errors were worst at early, high-noise steps. We fitted three variants on identical data: a 4× wider plain bridge, and a bridge that mirrors a block’s MLP branch, h + tanh(gate) · MLP(LN(h) · (1 + scale)), with scale and gate learned mixes of the model’s own modulation, at rank 256 and 1024.

Inside the attention bridge

The best bridge for block 13 is a miniature block: modulated attention followed by a modulated MLP, the same structure as the original, with a quarter of the attention heads and a much narrower MLP. Its attention output starts at zero, so a fresh attention branch leaves the conditioned MLP bridge unchanged.

Timestep modulation: scale and gate. At every step the model turns the timestep into one vector shared by all blocks: [scale1, gate1, scale2, gate2], 4 × 4,096 numbers. A real block multiplies its normalised input by (1 + scale), which turns each channel up or down for this noise level, and its output by tanh(gate), which decides how much of the update to add. The bridge learns per-channel mixes of these same values (about 25,000 weights) and starts exactly like a real block: scale_attn = scale1, gate_attn = gate1, scale_mlp = scale2, gate_mlp = gate2. Prompt tokens use the row for a clean image (t = 0), as in the model.

Heads: 4,096 → 1,024 = 8 × 128. Each token’s query, key and value are projected to 1,024 numbers and split into 8 heads of 128. Each head compares tokens in its own 128-number space, so different heads can attend to different things; their outputs are joined and projected back to 4,096. The original block has 32 such heads. The bridge keeps 128 per head, because the model’s position code is built for that size, and saves cost with fewer heads.

RMSNorm on Q and K. Before comparing tokens, each 128-number query and key is rescaled to a standard length, x / √mean(x²), times learned weights. An attention score is a dot product; without this, a few large vectors can make scores explode and the softmax lock onto one token. The original blocks do the same.

RoPE: position as a rotation. On its own, attention does not know where a token is: shuffle the tokens and the scores do not change. RoPE pairs up each head’s 128 numbers into 64 two-dimensional pairs and rotates every pair by angle = position × frequency. Two patches showing the same letter at different places get the same arrows pointing in different directions. When a rotated query meets a rotated key, their score depends only on the difference of the angles, that is, on how far apart the two tokens are, so a head can learn “look at the patch just to my right” once and use it anywhere in the image. Nothing about the rotation is learned: frequency = 1 / 10000^(2i/d) per pair, fast for some pairs (neighbours) and slow for others (coarse layout), like the hands of a clock. In Qwen-Image 2.1 the 64 pairs are split by axis: 8 follow the image or text segment, 28 the row and 28 the column; image tokens get a (row, column) on their grid, centred at zero, and prompt tokens advance one step per word. The angles are precomputed once as a table. RoPE is applied to Q and K only, after RMSNorm; V, the content being passed along, is not rotated. This is why the attention bridge can help spelling: to write “Anchor”, the patch for “c” must know that “n” is just left of it and “h” just right.

Why rotations rather than classic position embeddings? The classic approach adds a learned (or fixed sinusoidal) vector for each absolute position to every token once, at the input. RoPE differs in four ways that matter here. (1) Relative by construction: the score depends exactly on the offset between two tokens, so a pattern learned in one corner works everywhere; with absolute vectors the model has to infer offsets indirectly. (2) Applied in every attention layer, directly to queries and keys, so position information cannot fade as it travels through 32 blocks, and it never mixes into the content passed in V. (3) Any grid size: the formula works for any number of rows and columns, which suits a model that renders 1024×1024, 1376×768 or 896×1184 images; a learned table only covers the sizes it was trained on. (4) No extra weights, and two-dimensional by simply splitting the pairs between rows and columns. RoPE is not perfect: positions far beyond the trained range still behave worse, but it has become the standard choice in modern diffusion transformers.

Images. On the same 30 prompts and seeds, the conditioned rank-1024 bridge was worse than B on 8 prompts, similar on 20 and better on 2, against 15 worse for the first bridge. It fixes the hair smearing, spells “Happy Birthday Emma” correctly, keeps the alarm clock and one clean pair of glasses, and stops duplicating the pot. It does not recover fine local detail (lace and circuit board, the peacock eye, the star), the bent spoon or the melting clocks, and “The Rusty Anchor” is still misspelled.

Unblinded AI visual review, two reviewers, one seed per prompt; reviewers were not fully consistent, so column-by-column comparisons within a sheet are the reliable signal.

Is the misspelling systematic? Four seeds per text prompt

One seed per prompt cannot separate a real loss from bad luck, so we rendered every text prompt with three more seeds for the teacher, B and the best bridge (conditioned rank 1024, 3,000 updates) and counted exact spellings.

The attention bridge spells “The Rusty Anchor” on one seed

On the original seed, every earlier bridge wrote “The Rusty Achor”. The attention bridge is the first to write the sign correctly, like the teacher and B. Across four seeds, however, it is right once (B: three times), so attention moves spelling in the right direction without restoring it.

Healing the whole student does not bring spelling back

If one bridge cannot carry block 13, perhaps the rest of the network can adapt around it. We healed the two-bridge student (blocks 5 and 13 removed) with LoRA adapters (rank 16) on the attention of all 30 remaining blocks, the bridges trainable too, on the saved recordings (450 text prompts). A first run with a higher learning rate made the velocity worse at every evaluation and kept only its best-velocity checkpoint, so its spelling could not be checked; it was stopped. The second run saved every checkpoint and judged each by both tests: velocity on 180 comparisons (quantitative) and exact spelling of three signs over four seeds (qualitative).

Is it block 13, or block 13 on top of a weak block-5 bridge?

B already spells a little worse than the teacher, and its block-5 bridge is the simplest one. Errors could compound. To separate the two, block 13 was removed from the intact teacher (block 5 kept): once as a plain skip and once with the best block-13 bridge.

Choosing the next block by spelling

A scan skipped each block of B and measured velocity on 80 saved text prompts. It filters overall damage but cannot see spelling: block 13, which breaks it, sits mid-table, because letters are a tiny fraction of the pixels. So the five blocks with the lowest velocity cost and not next to block 5 (17, 25, 20, 18, 16) were each skipped inside B and rendered on the text prompts with four seeds. Block 25 ranked second only after the first images had been launched; it was added afterwards, as recorded in a dated addendum.

Removing the chosen blocks together

Blocks 20 and 17 each keep spelling when skipped alone. Removals can compound, so combinations were skipped together inside B, with no new bridges, and judged by velocity on 180 comparisons and by the four-seed text grid.

General images with 5 and 20 removed. On the 30 standard prompts it is worse than B on 9, similar on 20 and better on 1. Fine detail and texture are intact (lace, circuit board, skin, hair). The losses are mid-scale structure: a checkerboard breaks apart, simple solids lose their geometry, an alarm clock disappears, crowded scenes and figures lose coherence, and one letter is malformed (so about 10/12 exact signs). The first block-13 bridge was worse on 15 of these prompts and the best per-token one on 8: removing 20 with no bridge already matches the best block-13 bridge on general quality while keeping spelling. A bridge for block 20 is the next step.

A bridge for block 20

Block 20 got the recipe that worked best for block 13 (timestep-conditioned MLP plus an 8-head attention branch, trained end to end on 450 text and general recordings), starting from “skip block 20”. Every 100 updates it was saved and measured; two checkpoints were rendered on the 30 standard prompts and the four-seed text grid.

A new bridge for block 5

Block 5 had kept its first bridge, a plain rank-256 bottleneck fitted per token in Phase 5. It was rebuilt with exactly the architecture and training of the block-20 bridge (timestep-conditioned rank-1024 MLP plus an 8-head attention branch with its own prefix cache, about 25.2 million weights, trained end to end against the teacher velocity for 600 updates), with the block-20 bridge (step 600) frozen. Every 100 updates the bridge was saved, measured on 180 velocity comparisons, and rendered on the four-seed text grid; the best spelling checkpoint was also rendered on the 25 general prompts.

The 30-block student is B plus both attention bridges (block 5 at step 200, block 20 at step 600).

Healing the 30-block student

A rank-16 LoRA on q, k, v and out of the 30 original blocks (15.7 million weights, learning rate 2e-5) was trained together with both bridges (learning rate 1e-5) against the teacher velocity, on 450 text and 50 general recordings plus the original training data, for 200 updates. Every 50 updates the student was saved, measured on the 180 velocity comparisons, and rendered on the four-seed text grid.

Version 1: two blocks removed

The plan changes. The first release removes two blocks (5 and 20) instead of four. Its base is the unhealed 30-block student, the best speller; the healed checkpoint at step 200 is kept as a backup. Next, a speed-distillation (lightning) LoRA takes sampling from 40 steps to 8, about five times faster, since the model already runs without classifier-free guidance. Every checkpoint is judged by velocity, by the text grids and by general images, and the chosen one is fused into the weights. Quantization follows, judged the same way. Removing blocks three and four may come in a later version.

Few steps without distillation, and a published lightning LoRA

Before training our own speed LoRA, the intact teacher was sampled with the same prompts and seeds as every earlier grid in 8 and 4 steps without any LoRA, and in 4 steps with the LightX2V Lightning LoRA for Qwen-Image 2.1 (a published distribution-matching distillation, rank 16 on attention, used here for comparison only; it is not reused).

What removing two blocks buys

Batch 1 at 1024 by 1024 on one H100, each component timed separately. The transformer is almost all the time: text encoding takes 0.03 s and VAE decoding 0.13 s per image. The transformer shrinks from 7.12 to 6.73 billion weights (14.2 to 13.6 GB), but the untouched text encoder holds 17.5 GB, so memory falls by only 0.7 GB.

Our own lightning LoRAs

Method: trajectory distillation. The intact teacher (32 blocks, 40 steps) was run on 500 training prompts (450 with text) and 45 held-out prompts, saving the latent after every step: 545 trajectories, 11.9 GB. The student’s 8 (or 4) steps are exact subsets of the teacher’s 40-step schedule. Starting from a teacher latent, one student step is trained to land where 5 (or 10) teacher steps land. A rank-32 LoRA on the attention of the 30 blocks of version 1 learns it (31.5 million weights; the BF16 bridges stay frozen). Every 500 updates the LoRA is saved and judged by the distance between its final latent and the teacher’s 40-step final latent for the same noise, and by the same 45 images.

The 8-step LoRA learned the teacher’s layouts and most letters within 500 updates and then stopped improving. The 4-step LoRA learned the layouts but stayed soft and grey: copying the teacher’s path averages at 4 steps, which distribution matching (the published LightX2V LoRA) avoids. The rest of this section is about 8 steps.

Making the loss see letters: OCR text regions

A misspelled sign hardly changes the loss: letters cover a few percent of the latent, and every token counts the same. So the loss was told where the letters are. Each teacher training image (its trajectory’s final latent, decoded by the VAE exactly as the pipeline does) was read by an off-the-shelf OCR detector (EasyOCR, Chinese and English). Each detected word box was mapped onto the latent token grid, where one token covers 16 by 16 pixels, and widened by one token. Those tokens then count 10 times in the loss: an error on a letter costs ten times an error on a wall. Only the boxes are used, not what the OCR reads, so a misread word still marks the right place. Text was found in 370 of the 450 text images, covering a median 12% of their tokens (and in 3 of the 50 general images).

The best 8-step checkpoint was continued for 2000 updates with these weights, a learning rate decaying to zero and 2 examples per update.

The main issue: structure

Averages hide it. On the 25 general prompts the 8-step LoRA is as close to the teacher as version 1 at 40 steps (median PSNR 17.7 against 17.8 dB; 8 steps without the LoRA, 15.0 dB), and weighting letters did not change its general images. But side by side with version 1 at 40 steps, 5 of the 25 images are clearly worse, and textures are slightly softer:

All 25 general prompts side by side: teacher | version 1 at 40 steps | ours at 8 steps.

Learning from version 1’s own images

Version 1 recorded its own 40-step trajectories (BF16 bridges, as it ships) on 1000 training prompts, now balanced: 500 with text and 500 general, drawn from the Phase 1 training split across 38 subjects and checked against the held-out prompts (1045 trajectories with the 45 held-out ones). OCR found text in 406 of the 500 text images. A fresh 8-step LoRA was trained on them from zero, with letters weighted 10 times and a decaying learning rate. Its targets are now images version 1 can actually draw, so its ceiling is version 1 at 40 steps.

Structure. Against version 1 at 40 steps, 20 of the 25 general images match closely; 5 are clearly worse. The four friends are now right, but a new case appeared (the laptop lighting a face disappears), so the count stays at 5: the checkerboard pattern, the star’s outline, the semi-transparent cyclist, the missing alarm clock and the missing laptop. They are not random: precise geometry, a small object the image depends on, or a thin subject. These five did not change between updates 500 and 2000.

Where the failures are made: a drift test

If the student’s first jump lands slightly off, the later steps may never recover. On the five failure prompts the 8-step LoRA was started not from noise but from version 1’s own latent after 5, 10 or 20 of its 40 steps.

A gentler start

Version 1 takes the start from noise to step 5 of 40 in a few small Euler steps of its own schedule, then the LoRA does the seven big jumps. Two variants: with the LoRA switched off for the start (which needs the LoRA kept separate), and with it on throughout (what a single model with the LoRA fused into its weights does).

With the LoRA off for the start, 12 steps reproduce version 1 almost exactly (7.7%). Fused, the distance looks worse, but the images are not broken: the start drifts to a different composition and every step stays clean, so the star, cyclist, alarm clock and checkerboard come out sharp. Distance cannot tell a different good image from a broken one. A separate LoRA was ruled out, so the fused route was scored in full.

How version 1 is run

The LoRA is fused into the weights (W + B·A in 120 attention layers), so the model is a single checkpoint. The 10-step schedule is points 0, 1, 3, 5, 10, 15, 20, 25, 30 and 35 of the stock 40-step schedule for the image size, then the end. The stock scheduler cannot produce it on its own, and passing those values naively would shift and stretch them a second time. Two ways to run it:

A · scheduler class

A small scheduler that builds the stock 40-step schedule for the image size and keeps those ten points; the reference implementation, shipped with the model.

B · plain sigmas

A helper returns the ten values for a width and height; they are passed as sigmas to the stock pipeline with a neutral scheduler (no shift, no terminal stretch). Works in diffusers as is, and in ComfyUI and other interfaces through a custom-sigmas input.

shipping

Both routes ship as files next to the weights: nihonga.py (the bridge modules, the loader, the scheduler class and the sigma helper) and sigmas.json (the ten values for each standard image size, for interfaces without code).

Nihonga v1

The release checkpoint uses the adapter of update 2500 (15 of 16 spelling items at 10 steps, against 14 for update 2000). Built in the shipping format, the transformer was reloaded through the shipped code with every tensor identical; the scheduler class and the plain sigmas used exactly the intended schedule and give pixel-identical images.

The research record

What is complete, what remains open

Phase 0

Architecture inspected: 32 blocks and 4,096-wide hidden states.

Phase 1

20k prompt pilot serialized, with a 100k template pool and fixed split assignments.

Phase 2

Original-model images, judged baseline, repeatability analysis and H100 efficiency measurements saved.

Phase 3

Layer diagnostics and identity-bypass image pilots; blocks 2–5 retained as learned-replacement candidates.

Phase 4

Residual bottleneck bridges implemented and insertion checked.

Phase 5

Four independent bridge-pretraining pilots completed; checkpoints and optimizer states saved.

Phase 6

Repeatable teacher contract verified; bridge and full-student healing pilots saved. The 120-update continuation was compared on an untouched 20-prompt fixed-input test, an 11-prompt five-way comparison, 16- and 45-prompt text reviews, and a blinded 129-prompt adherence test. Complete: the pretrained bridge is selected; update 120 (86 vs 83 clear passes, +2.3 points, 95% CI [0.0, 5.4]) missed the predeclared promotion bar; after correcting one reversed blind score (the hidden giraffe) it is 85 vs 84, +0.8 points.

Phase 7

In progress. Removing blocks 2-5 (28 blocks) failed even after healing: 13.2% velocity error against 5.4% for the one-block student, with broken images. Removing only blocks 4-5 (30 blocks) reaches 7.41% after healing. It is still above the numerical gate, but in a first 30-prompt image screen it looks comparable to the teacher in this sample, and the healed step-60 student rendered all 5 text prompts correctly. Candidate A is not accepted: in a blinded 159-prompt test it fell behind the teacher on adherence and text. The one-block student remains the selected model.

Later phases

Importance has been recalculated inside candidate A and is being recalculated inside the accepted one-block student; a learned replacement follows, with reranking after each accepted removal. A speed-distillation LoRA from 40 to about 8 steps and optional quantization remain planned.

Reproducibility and updates

Each executable notebook step has a method description and a plain-language explanation of its result. Targets, raw outputs, metrics, selected weights, optimizer state and manifests are serialized locally, with expensive experiment artifacts backed up remotely. Matching saved steps reuse their results; changed inputs stop rather than silently overwrite the record.

• Download the measurementsJSON · exact values and source SHA-256 hashes
• Research repository ↗Phase notebooks and original plan
• Page source ↗Public research record
• Earlier bridge + LoRA graphStandalone SVG · diagnostic ablation

This page is a dated snapshot of saved experiments, not a live training dashboard. Its refresh script reads selected local summaries and copies existing comparison images; it runs no models and publishes no weights, caches or credentials. Every plotted value is available in the downloadable record.

Current result · development pilot

One block removed is still the accepted model. The next cut is ranked inside it

Phase 6 showed that one removed block, 5, needs only its pretrained bridge: 5.518% velocity error, with healing adding nothing measurable. That cut is too small to release. In Phase 7, removing four blocks (2-5) stayed far from the teacher even after 480 healing steps: 13.2% against 5.4%, with wrong subjects and broken text in the images. Removing only blocks 4 and 5, with one 4.2M-parameter bridge and 60 healing steps, reaches 7.41%. That is still above the predeclared gate, but in a first 30-prompt image screen it looked like the teacher and rendered all five text prompts correctly. The larger blinded 159-prompt test then showed A falling behind the teacher, so A is rejected as a base.

• 30 + 1surviving transformer blocks plus one bridge in the strongest candidate, from 32 original blocks
• 7.41%velocity error on 50 validation prompts, against 5.39% for the one-block student
• 5 / 5text prompts rendered correctly, like the teacher
• 0 / 5text prompts correct for the healed 28-block student

Research direction: the accepted model is still the one-block student. Importance scans inside both candidate A and the accepted student are complete; block 13 is the gentlest bypass in the accepted student. A rank-256 learned bridge for block 13 then failed the preregistered velocity gates, so the accepted model is unchanged. A's measured speedup is 1.064×.

Velocity errors are measured on saved teacher inputs at three denoising moments, not on finished images. Image results come from one model-blind AI reviewer with one seed per prompt.