Fish 3D / 4D reconstruction from a static BRUV video

Experiment: sharper 3DGS models (frame selection + image formation)

Started 2026-11-11. Goal: find out why the final gs_final models (4D GS, fins preset, 30 fps, 4000 iters; see docs/method_choice.md, docs/RESULTS.md) look blurred, and fix it.

Hypotheses

• No quality-aware frame selection. 04d_hifps.py takes the longest contiguous visits (cap_segments, capped at 24 s) regardless of sharpness / size / pose quality, although most fish have many more clean 10 fps frames over other visits (e.g. w40_fish1: 3259 kept frames). Within-fish sharpness (Laplacian variance in the eroded mask, native-resolution) varies 3-8x (p90/p10): fish2 7.7x, w58_fish8 3.4x, w40_fish7 4.6x, w12_fish3 4.2x.
• Pose error — misaligned views average into blur exactly like motion blur.
• Low native resolution — fish are 100-250 px long; crops are upsampled.
• Water — distance-dependent attenuation/backscatter changes colour/contrast per frame; the per-frame colour gain only partly absorbs it -> washed-out average.
• Under-training — fish2/fish3 (the best) had 10000 iters, everyone else 4000.

Fish set (6)

fish2 (triggerfish, good), w12_fish3 (butterflyfish, good), w02_fish6 (cleaner wrasse, small), w58_fish8 (redmouth grouper), w32_fish1 (hind, fair, belly artifact), w40_fish7 (hind, poor).

Fixed evaluation (identical for all arms)

• Held-out set per fish: sharp, well-posed frames drawn from ALL visits (not only the 24 s training segments), ~40-60 frames, spread in time and viewing direction. Frames within +-0.3 s of a held-out frame are excluded from training in EVERY arm.
• Same test-time re-fit (pose + colour, model frozen) and same loss for every arm.
• Metrics: PSNR full (composited over plate) and masked, LPIPS (penalises blur, unlike PSNR), sharpness ratio = gradient energy render / real inside the mask, sharpness of canonical turntable views, plus side-by-side contact sheets (real | B0 | arm) for visual check.

Phases

• A — diagnosis. Per-frame quality score (scale-normalised sharpness, motion-blur proxy from image speed, native size, pose confidence, occlusion/identity from 04b). Correlate with per-frame residual of the current gs_final models -> which factor dominates.
• B — one change at a time (6 fish, 4000 iters unless stated): | arm | change | |---|---| | B0 | current gs_final, re-evaluated on the new held-out set | | B1 | 12000 iterations (iteration confound) | | S1 | picky within the current 24 s: best ~50% by quality, view-coverage constraint | | S2 | picky from ALL visits, same number of training frames as B0 (quality vs quantity) | | S3 | S2 with soft per-frame weights instead of hard drops | | C1 | global white balance from the background plates (sand gray-world), applied to crops+plates | | C2 | water image-formation model in the renderer: I = J e^{-bD z} + Binf (1 - e^{-bB z}), z from pose depth | | M1 | motion-blur-aware rendering: average of 3-5 renders along the per-frame motion, learnable exposure | | L1 | lossless crops (decode video directly, no JPEG intermediate) |
• C — combine winners on the 6 fish; if the gain holds, re-run all 44 fish into gs_v2 (gs_final is kept), regenerate videos and the website.

Decision rule: adopt a change if LPIPS and sharpness ratio improve on >= 4/6 fish with PSNR not dropping > 0.3 dB, and it looks better. Risks: coverage holes when too picky (hence the coverage constraint); water model trading off with per-frame colour gain (regularise); motion-blur model costs 3-5x render time. Not adopted (probe at most): per-frame deblur / super-resolution networks — hallucinated detail is inconsistent across views.

Results go to docs/quality_experiment.md.

Quality experiment: sharper 3DGS models — results

Plan: quality_experiment_plan.md. This file collects the results phase by phase. Fish set (6): fish2, w12_fish3, w02_fish6, w58_fish8, w32_fish1, w40_fish7.

Shared evaluation infrastructure

None of these needs FISH_WINDOW: the window follows from the fish name.

Per-frame quality (12_frame_quality.py)

All image measures at native camera resolution: the crop square is cut from the full video frame (frames/f_*.jpg, hf_frames/<name>/h_*.jpg; integer offset, no resampling) and the crop's mask is resized onto it. Checked: the patch registers exactly with the crops (MAD 0 at zero offset, rising at +-0.5 px), and the 30 fps decode is pixel-identical to the 10 fps decode at k = 0.

• sharp_lap: trimmed Laplacian energy (mean squared 3x3 Laplacian, clipped at the 98th percentile of |Laplacian|) inside SAM mask AND soft alpha > 0.95, eroded by 20 % of the fish thickness. Why not the plain Laplacian variance in the eroded SAM mask: the 10 fps SAM masks leak 1-2 px beyond the fish edge in places, and with a 4 %-of-length erosion the fish/background edge dominated the measure. On identical video frames the 10 fps and the 30 fps crops (their own SAM masks) gave 2-4x different values (w58_fish8 frame 4619: 310 vs 73); median |log ratio| over ~60 coincident frames 0.45 (w32_fish1) / 0.21 (w58_fish8) for the plain variance, 0.09 / 0.02 with soft-alpha intersection + trimming. Residual disagreements come from frames where the two SAM masks include different objects.
• sharp_rank = within-fish percentile of sharp_lap (10 + 30 fps pooled), because texture differs by species.
• speed_px_s (centroid speed, least squares over +-0.1 s), dir_ratio (gradient energy along / across the motion), length_px, fit_iou (projected pose ellipsoid vs mask), pose flags, view direction (view_az, view_el, view_side), dist (pose depth in half-lengths), blue_frac, frame-filter fields (keep, weight, occlusion, reasons), touches, bg_ok, and the gs_final repair / weight / error of the frame where present.
• score = gate (0.40 sharp_rank + 0.20 (1 - rank speed) + 0.15 rank length + 0.25 q_pose), gate = keep pose_valid ff_weight (1 - occlusion) (0.5 if touches) (0.8 if slow), q_pose = clip((fit_iou - 0.4) / 0.4, 0, 1), halved for shape / depth outliers.

Held-out set (13_heldout.py)

Candidates: 10 fps base frames (they have plates and soft alpha) that are kept, pose-valid, not slow, not outliers, bg_ok, not touching the border, unoccluded (no occlusion / overlap reason), ellipsoid fit >= the fish median and in the sharper half (thresholds relaxed in steps when too few pass). Greedy choice by score + view novelty + visit novelty - exclusion cost, >= 0.5 s apart.

Exclusion rule: every training frame (10 or 30 fps) within +-0.3 s of a held-out frame is excluded in every arm; heldout.json lists the ids (exclude.base, exclude.hf; 30 fps ids matched by time, id = 3*(i10-1)+1+k, t = start + (id-1)/30). Use quality_lib.training_exclusion(name, 'hf') in a trainer.

Deviation from the plan (40-60 frames): one held-out frame removes 7 frames at 10 fps and 19 at 30 fps. 40-60 held-out frames would remove most of the training data of the smaller fish (w02_fish6 has 135 usable 10 fps frames). The selection therefore stops (above 12 frames) before more than 35 % of the usable 10 fps frames or 30 % of the gs_final 30 fps training frames would be excluded, and penalises frames whose exclusion window is full of usable frames.

128 held-out frames in total (plan: 40-60 per fish; see above). The arms must retrain with the exclusion, so they lose 3-36 % of the 30 fps training frames of gs_final. The fair baseline for the arms is therefore not B0 but "B0x" = the gs_final recipe retrained with the exclusion; B0 is kept as an optimistic reference.

quality_eval.py protocol

Same for every model, gaussians frozen, evaluated at the base crop resolution (256 px): initial pose = raw 05 pose (independent of what the model saw), discrete hypothesis search (current / twin / flip / twin+flip / upright twins / +-25 deg yaw, each translation-aligned and refined 20 steps at half resolution, lowest loss wins), then 200 Adam steps on pose, colour gain/bias and (deforming models) bending + caudal angle, coarse-to-fine. Loss = the 06_train_gs.py --bg_composite loss with fixed weights (composite over the plate, 0.8 L1 + 0.2 D-SSIM, soft-alpha / inside / far alpha priors, L2 on the deltas). Metrics: PSNR full (observed crop) and on the SAM mask, LPIPS-alex on the fish box (224 px) and on the full crop, sharpness ratio (trimmed gradient energy render / real, same mask and clip level), Laplacian ratio, IoU; six canonical views (gradient energy of the unbent model). contamination = held-out frames within 0.3 s of a training frame of the evaluated model.

A. Diagnosis

Date: 2026-11-11. Inputs: 12_frame_quality.py on the 6 fish (10 fps + 30 fps), 13_heldout.py, tests/quality_eval.py on gs_final (B0), tests/quality_diag.py (re-render of gs_final on its own training frames with the refined poses / bending / caudal / colour of training, composited over the plates at the training resolution). Raw outputs: work/quality_exp/diag/<fish>.json, <fish>_diag.png, diag_summary.json, work/quality_exp/<fish>/eval/B0/.

A1. B0 = gs_final on the held-out set

• Caveat: B0 is optimistic. gs_final was trained without the exclusion; "contaminated" = held-out frames within 0.3 s of a gs_final training frame (fish2: all, its 30 fps data cover 304 of 310 usable 10 fps frames). Measured effect: contaminated vs clean held-out frames LPIPS 0.164 vs 0.229 (w12_fish3, 9 vs 15 frames), 0.092 vs 0.122 (w58_fish8, 5 vs 18).
• Reproducibility: a second full run of B0 gave PSNR-mask differences <= 0.0024 dB and LPIPS differences <= 0.0001 (metrics_run1.json vs metrics.json). Runtime 5-11 s per held-out frame with two evaluations sharing the GPU (2-7 min per fish). --reuse_tto recomputes the metrics from the stored fit in seconds.
• The evaluator handles older models: the 10 fps fish2/gs_robust_4d (no fins, no plate training) scores below gs_final on every metric (LPIPS 0.174 vs 0.165, PSNR mask 20.21 vs 21.31).
• "raw pose switched": the hypothesis search replaced the raw 05 pose by a discrete alternative (twin / flip / upright twin / yaw) in 2-27 frames per fish; part of these are near-identical twins of side-on fish, but w32_fish1 (27/40) and w12_fish3 (13/24) show how often the raw poses sit in the wrong discrete mode.
• Gradient energies of the canonical views are only comparable within a fish (texture).

A2. gs_final on its own training frames: residual vs quality features

Per-frame residuals of gs_final (median over the training frames):

(Sharpness = trimmed gradient energy inside the fish, same mask and clip level for render and crop. Equivalent blur = Gaussian sigma for which the blurred real crop has the render's gradient energy, converted to native pixels.)

Spearman rho of the per-frame residual with the frame's features (training frames; PSNR mask: + = feature goes with a better fit; LPIPS: - = better):

Sharpness ratio (render / real) vs the input's sharpness rank: -0.58 / -0.26 / -0.59 / -0.79 / -0.63 / -0.78: the sharper the photo, the larger the gap, because the render's sharpness is nearly the same in every frame (scatter in <fish>_diag.png, panel 2). By decile of the input sharpness, fish2's LPIPS rises from 0.115 (deciles 4-6) to 0.20 (deciles 9-10), PSNR mask falls from 24.4 to 18.7 dB. By decile of the ellipsoid fit, LPIPS falls from the worst to the best decile in every fish (w12_fish3 0.265 -> 0.161, w02_fish6 0.183 -> 0.066, fish2 0.222 -> 0.145, w32_fish1 0.131 -> 0.114 with a peak of 0.194 in decile 3; w40_fish7 nearly flat 0.201 -> 0.191).

Measurement checks:

• The view-direction correlation is not only an LPIPS-box artefact (a side view puts more well-matched background into the fish box): PSNR on the mask shows it too (6/6 positive).
• The sharpness ratio is partly noise: inside dark, low-texture fish (w58_fish8, w40_fish7) much of the real gradient energy is sensor / JPEG grain, which a render cannot (and should not) reproduce. With both images pre-blurred (sigma 1.5 crop px) the held-out medians are still 0.32 / 0.49 / 0.73 / 0.38 / 0.25 / 0.09 (fish order as above): the renders lack mid-frequency detail, not only noise. LPIPS stays the primary metric, as planned.

A3. Data selection of 04d (hypothesis 1)

The 30 fps decode is pixel-identical to the 10 fps decode, so any sharpness difference between the 30 fps training set and the other 10 fps frames is a selection effect of cap_segments (longest visits, 24 s cap). Native sharpness (trimmed Laplacian energy, usable frames):

Within-fish sharpness varies 6-35x (p90/p10, robust measure; usable 10 fps frames: fish2 18x, w12_fish3 6.4x, w02_fish6 35x, w58_fish8 11x, w32_fish1 14x, w40_fish7 13x). The sub-frame phase k of the 30 fps grid makes no difference (paired k1/k0, k2/k0 sharpness ratios 0.98-1.05).

A4. Water / colour (hypothesis 4)

The per-frame colour gains of gs_final are small (p10-p90 within 0.94-1.05 per channel for w12_fish3, within 0.97-1.03 for the other five), and the distance correlations above are weak and of mixed sign. The colour cast lives in the canonical colours instead (the models are cyan; e.g. w12_fish3's yellow body is pale green in views.png), i.e. it is a colour-fidelity problem for the turntables (arm C1), not a cause of blur. Red gain vs distance is clearly negative only for w12_fish3 (rho -0.62).

A5. Conclusion: what dominates

• The blur is a property of the model, not of the individual input frames. Render sharpness is nearly constant over a 10x range of input sharpness; renders are below the median training frame in every fish (0.06-0.72x) and below the 10th percentile in three (w58_fish8, w32_fish1, w40_fish7: 96-98 % of frames). Equivalent blur 0.7-2.8 native px. Sharp frames are matched worse, blurry frames are not matched better.
• The per-frame residual is driven by geometry: pose / silhouette agreement, viewing direction and native size, consistently in all six fish. Input sharpness, image speed (motion blur), distance and colour drift are not consistent predictors. A uniformly blurred model whose error grows where the pose is least constrained (head- / tail-on views, poor ellipsoid fit, small fish) is the signature of pose misalignment averaged into the texture (hypothesis 2) together with limited resolution (3). Under-training (5) would also give a uniform blur and is not excluded by these data (fish2 with 10000 iterations is not sharper relative to its inputs: ratio 0.20).
• Frame selection (1) is wasteful in 3 of 6 fish: the 30 fps segments of w58_fish8 (0.22x), w32_fish1 (0.81x) and w40_fish7 (0.74x) are blurrier than the fish's other visits, and 174 / 1209 / 31 sharper 10 fps frames are unused. For w12_fish3 and w02_fish6 the segments are already the sharper part; fish2's segments cover nearly everything.
• Water colour (4) and input motion blur are minor for sharpness.

Steering for phase B:

• Run B0x (gs_final recipe retrained with the exclusion) as the fair baseline, and B1 (12000 iterations) early: a uniform model blur could still be under-training.
• S1 / S2 / S3 should select by pose quality as much as by sharpness: the 12 score in its current weighting (0.40 sharpness) does not predict the residual; fit_iou, view_side and length_px do. Suggested arm score: drop or down-weight frames with low ellipsoid fit and keep the view coverage constraint; sharpness as a secondary criterion. S2 has the most room on w58_fish8, w32_fish1, w40_fish7 (and w12_fish3).
• Add a pose arm (P1) not in the plan: stronger per-frame pose refinement / alignment (e.g. a pose-only refinement pass at full resolution after densification, or a higher pose learning rate late in training), since misalignment is the leading candidate for the uniform blur.
• M1 (motion-blur rendering): low priority (speed does not predict the residual). C1: for colour fidelity of the canonical model, not for sharpness. C2, L1: low priority.

B. Datasets

Date: 2026-11-11. The data side of the arms S1, S2, S3, L1, C1 for the 6 fish. Every arm is a NEW folder next to <name>_hf (<root>/fish/<name>_<arm>, root = work/ or work/w<MM>/); _hf, gs_final, the base dirs and their frame_quality.json are only read. Train an arm with the usual command and --fish <name>_<arm> (FISH_WINDOW=<MM> for window fish), e.g. 06_train_gs.py --fish w32_fish1_s2 --fins --deform --iters 4000 --hyp_max_frames 300 --out <tag>.

Held-out exclusion is already applied in every arm folder: the frames within +-0.3 s of a held-out frame (by time, arm_lib.excluded_by_time, same rule as quality_lib.training_exclusion) are not in meta.json / poses.json / frame_filter.json at all, so no trainer flag is needed (checked by 14_arm_data.py --summary: heldout_frames_present = 0 for every arm, see work/quality_exp/datasets.json). Note for trainers: quality_lib.training_exclusion() takes the BASE name (w32_fish1), not the arm folder name; on an arm folder the exclusion is a no-op.

Code

Arm score (arm_lib.q_arm, used by S1, S2, S3): q_arm = gate (0.35 q_pose + 0.35 q_sharp + 0.15 q_size + 0.15 q_motion) with gate and q_ of 12_frame_quality.py. Phase A asked for selection by pose quality as much as by sharpness, so pose fit gets the same weight as sharpness (12's score: 0.40 sharpness / 0.25 pose). A first version with 0.40 pose / 0.25 sharpness chose the same S2 windows for w12_fish3, so the choice is not sensitive there. The view enters as a constraint, not as a score term: view bins = 8 azimuth bins of 45 deg (centred on head-on) x 3 elevation bands (< -30, -30..30, > 30 deg) of the camera direction in the fish frame (view_az, view_el of 12, from the poses).

frame_select.json (format agreed with the trainer side): <dataset dir>/frame_select.json = {"note": ..., "frames": {"<id>": weight}}, weight in [0, 1], 0 = drop, a frame missing from the file is dropped when the file is used. Present in _s1 (weight 1 for every kept frame; the folder holds only those frames, so it is redundant there) and _s3 (the soft weights). Not present in _s2, _l1, _c1.

The arms

• S1 <name>_s1 (14_arm_data.py --arms s1): the _hf training frames (valid pose, keep, weight > 0) minus the exclusion; keep 50 % by q_arm; every view bin keeps at least max(1, ceil(25 %)) of its frames (its best ones), so no bin that had frames becomes empty. Only the kept frames are in the folder (copied crops / plates / soft alpha, filtered jsons).
• S2 <name>_s2 (04d_hifps.py --fish <name> --out-suffix _s2 --select quality --exclude-heldout --pose-source like_hf, FISH_WINDOW as usual): anchors are 10 fps base frames over ALL visits with q_arm > 0 (frames near a held-out frame count 0); window = anchor +- 5 frames at 10 fps (+-0.5 s, same pose segment, frames not yet taken), value = sum of q_arm in the window / 11. Phase 1: one best window per view bin that holds >= max(3, 2 % of the usable frames) and is not yet covered; phase 2: best windows until the PREDICTED number of training frames (30 fps ids whose interpolated pose is valid, keep of both bracketing 10 fps frames, not excluded; same rules as the poses stage) reaches 1.03 x the budget. Budget = training frames of <name>_hf after the exclusion (B0x has the same number). The windows are merged into runs, decoded, segmented (SAM 2.1, QC, re-prompting), cropped at the _hf crop resolution, poses interpolated from the same pose source as _hf (like_hf: fish2 gs_robust refined poses, the others raw 05 poses), frame filter mapped, then trimmed to exactly the budget (frames removed from the ends of the lowest-value runs, coverage runs last) before plates + soft alpha (04c via the 30 fps adapter) and diag. Decoded frames in hf_frames/<name>_s2/ (the _hf frames / masks / SAM cache are not touched). Selection details in <name>_s2/hifps_report.json ('select', 'trim').
• S3 <name>_s3 (12_frame_quality.py --fish <name> --dataset _s2, then 14_arm_data.py --arms s3): a copy of _s2 (same frames: no hard drops beyond S2) with soft weights w = 0.25 + 0.75 x percentile rank of q_arm among the S2 training frames (mean 0.63) in frame_select.json. frame_filter_s3.json = frame_filter.json with its weights multiplied by w, for the existing trainer (--filter_file frame_filter_s3.json, its per-frame loss weight). Use one of the two, not both, or the weights are applied twice.
• L1 <name>_l1 (04d_hifps.py --fish <name> --out-suffix _l1 --lossless --like _hf --exclude-heldout): the same frames, crop squares and SAM masks as _hf (minus the exclusion); the RGB is re-cut from PNG frames decoded straight from the video (hf_frames/<name>_l1/, 2.5 GB per fish). Poses, frame filter, plates, soft alpha and bg_alpha.json are copied from _hf (the plates are medians of the 10 fps JPEG frames; their JPEG noise is averaged out). Colour finding: the stream is tagged BT.709 (tv range), but the JPEG path of the whole pipeline ends up with BT.601 colours (ffmpeg converts yuv420p -> yuvj420p by range only, the JPEG decoder applies BT.601). A default rgb24 decode would use BT.709 and differ by -10 (G) / -7 (R) levels from every other image, so --lossless forces in_color_matrix=bt601:in_range=tv. Checks (fish box, 16 frames per fish, all 6 fish): the same id matches best in 16/16 frames, MAD 2.0-2.2 levels vs the JPEG frame (= the JPEG loss), mean difference B +0.8..+0.9, G -0.5, R +0.6..+0.7 levels; crop alpha bit-identical to _hf (max difference 0); RGB MAD inside the mask 1.3-1.6 levels.
• C1 <name>_c1 (14_arm_data.py --arms c1): _hf frames minus the exclusion, ONE diagonal colour transform per fish applied to the RGB of crops/ AND crops_bg/ (alpha and plate confidence unchanged), recorded in <name>_c1/colour_transform.json ({"type": "diagonal", "gains_rgb": [r, g, b], "space": "8-bit sRGB values as stored", ...}, out = clip(round(in x gain))). Estimate (arm_lib.plate_wb_gains): up to 300 plates, pixels with plate confidence >= 128 and soft alpha <= 13 (not the fish), luminance between its 60th and 99th percentile (sand / pale rock); m = their mean colour; gains = (1 / m)^0.8, normalised to keep the luminance of m; then all gains scaled down if the 99.5th percentile of the corrected brightest channel of fish pixels would exceed 250 (clip guard). Why strength 0.8: the full grey-world correction (exponent 1, also tried with the median and the mode of the chromaticity and with the mean over all plate pixels) turned the sand pink in all 4 fish checked, i.e. it over-corrects the red channel, which is also the noisiest. Previews (crop | C1 crop | plate | C1 plate): work/quality_exp/<name>/c1_preview.jpg. Evaluation note: the held-out crops of the base dirs are not corrected; the per-channel colour gain of the test-time fit can absorb a diagonal transform, or apply gains_rgb to the held-out crops (for colour-fidelity comparisons of the canonical views the correction is the point of the arm).

Numbers

Training frames = valid pose, keep, weight > 0, after the exclusion (what the trainer trains on, before its own every-10th validation split).

S1 vs all _hf training frames (medians; sharpness = native trimmed Laplacian energy, only comparable within a fish):

S2 vs _hf (medians over the usable frames of each 30 fps data set, frame_quality.json; for _hf this includes its frames near held-out frames; the 10 fps columns count the usable 10 fps frames each data set draws on):

Reading: S2 has real room only where the fish has many usable frames outside the production segments: w32_fish1 (263 of 1904 usable 10 fps frames, 12 visits instead of 1, 1.72x sharper), w58_fish8 (6 visits instead of 2, 1.36x sharper) and to a small degree w40_fish7 (1.06x). For fish2 the _hf segments already cover all usable frames, so S2 is the same footage cut into windows (0.86x sharpness at 30 fps; 629 instead of 631 training frames: 2 short because every usable 10 fps frame is already used). w02_fish6 uses 78 of its 80 usable 10 fps frames either way and ends up blurrier at 30 fps (0.45x: different in-between frames around the same anchors), w12_fish3 trades sharpness (0.79x) for more visits (7 vs 3), view bins (13 vs 10) and size: its production segments are its sharpest part (Phase A, A3). S2's runs are short (1-2 s windows, 12-24 runs), so temporal smoothness and the pose-hypothesis chains see shorter chains than in _hf.

C1 gains (R, G, B) and L1:

Smoke test

All 36 runs passed (6 fish x S1, S2, S3, S3 with --filter_file frame_filter_s3.json, L1, C1): 06_train_gs.py --fish <name>_<arm> --fins --deform --iters 50 --out _smoke (FISH_WINDOW for window fish) with the working-tree trainer of 2026-11-11 ~12:00-14:00 (which the trainer agent was editing at the time). Deviation: all runs except the first two (w58_fish8 S1, S2) added --tto_steps 10, because with the GPU shared by other jobs the default 150-step final evaluation made a 50-iteration run take 5-10 min; it only shortens the final evaluation. Every run loaded the expected training frames (train + val = the "training frames" above; the trainer's valid-frame count additionally includes keep = false frames), all plates and soft alphas, and wrote a model; the _smoke / _smoke_ff output folders were deleted afterwards. frame_filter_s3.json was picked up (mean weight 0.59-0.63 vs 0.91-1.0 with frame_filter.json). Logs: work/quality_exp/logs/smoke_A.log, smoke_B.log (+ .json).

Plates usable for the composite loss (bg_ok of 04c, frames with a valid pose):

S2 of w32_fish1 (and less so w40_fish7) has clearly fewer frames with a usable plate: its 1 s windows are spread over 12 short visits where 04c's local plate is worse (frames without bg_ok are trained with the masked loss, see 06's docstring). This is a confound of the S2 arm to keep in mind.

Reproduce

# S2 (GPU: SAM + ViTMatte, 6-35 min per fish), per fish with its FISH_WINDOW
python 04d_hifps.py --fish w32_fish1 --out-suffix _s2 --select quality --exclude-heldout --pose-source like_hf
# L1 (CPU, < 1 min per fish)
python 04d_hifps.py --fish w32_fish1 --out-suffix _l1 --lossless --like _hf --exclude-heldout
# S3 needs the S2 frame quality; S1 / C1 only need _hf (no FISH_WINDOW needed)
python 12_frame_quality.py --fish w32_fish1 --dataset _s2
python 14_arm_data.py --fish fish2 w12_fish3 w02_fish6 w58_fish8 w32_fish1 w40_fish7 --arms s1,c1,s3 --summary

B. Trainer options

Date: 2026-11-11. New options of scripts/06_train_gs.py (with gs_model.py, gs_render.py) for the arms. All are off by default; the default run is unchanged (checked below). Tests: scripts/tests/quality_opts_test.py (unit, eval), synthetic data from scripts/tests/make_synth_fish.py (new flags --motion_blur N --exposure e, --water; defaults unchanged, no extra RNG draws).

Rendering a water model elsewhere: gs_model.water_observe(rgb, alpha, depth, bD, bB, Binf, scale) with depth from gs_render.render_ex(..., depth=True) (info["depth"]) or render_batch(..., depth=True). 07_render.py: turntables show the restored colours, sequence.mp4 the observed ones (--seq_water restored to switch). 10_fish_videos.py: FishModel.render(..., observed=True) applies the water; the tracking overlay (render at a real frame's pose) uses it, turntables / poster / swim stay restored. tests/quality_eval.py does not apply the water model yet (not my file): for a fair held-out comparison of a --water model it must render as observed (water from model.pt / poses_refined.json, water_observe with the render depth) before the colour fit and the metrics; otherwise its per-frame colour fit has to absorb the whole water effect and the restored colours are scored against the water-cast frames. Canonical views (views.png) of a water model are restored colours by construction.

Default behaviour unchanged

Same command at HEAD (copies of the three HEAD files) and with the new code (2000 iterations; gsplat training is not bit-deterministic, so HEAD was run twice to get the noise level):

The new code is within the HEAD-to-HEAD spread everywhere, with the same loss at iteration 500 and the same pose-repair counts. By construction the default path makes the same calls (the only changed expressions, fw != 1 instead of fw < 1 and filter weight x 1.0, are identical while all weights are <= 1).

Synthetic tests

quality_opts_test.py unit: SO(3) log / exp round trip (max error 1.5e-11) and a finite gradient at 0, pose interpolation end points / half way / extrapolation, water limits (z = 0 -> J, z -> inf -> alpha Binf), WaterModel round trip, render depth (plane at z = 3 exact in gsplat and torch, rgb / alpha unchanged with depth=True, render_batch = render_ex, backends agree), frame_select / exclusion parsing: all pass.

Smoke runs: --frame_select (weights 2 / 1 / 0) + --exclude_frames (6 ids) on _synth: 6 excluded, 19 unselected, 128 trained, effective weights odd / even frames 1.28 / 0.64 (ratio 2, mean 1); --eval_only --water reloads the water model; --fins --deform --water --motion_blur 3 on _synth_fins_4d (30 fps, caudal) runs; on this blur-free, water-free data e falls to 0.17 and bD x scale to ~0.002-0.003, as it should. Real-data smoke on w12_fish3_hf (--fins --deform --exclude_frames auto --water --motion_blur 3, 1500 iterations): the exclusion resolves to 134 of the data set's frames (train 417 -> 298, internal val 46 -> 31), 277 / 20 / 1 training frames with two / one / no temporal neighbour, peak GPU memory 0.61 GB, 788 s (GPU shared); learned e = 0.27, bD x scale = (.017, .009, .010), bB x scale = (.004, .006, .007), Binf = (.09, .44, .34) (red attenuated most, as expected). A smoke test only, not a result (1500 iterations, no comparison run).

Evaluation (quality_opts_test.py eval): on every 2nd training frame (77 frames) the model is rendered SHARP and RESTORED (no water, gain 1, bias 0) at its refined pose and compared with the GT model at the GT pose (no blur / water / gain), inside the GT mask: PSNR, mean colour error (levels), gradient-energy ratio render / GT (quality_lib.sharpness), LPIPS on the fish box over grey. 3000 iterations, default trainer otherwise. Outputs: work/quality_exp/trainer_opts_tests/eval_*.json and contact sheets eval_*.jpg (crop | GT | models).

Motion blur (make_synth_fish.py --motion_blur 9 --exposure 0.5 [--plates]: each frame = mean of 9 renders of the analytic trajectory over half the frame interval; at 10 fps the fish moves roughly 9-13 px during the exposure in the 256 px crops, estimated from the trajectory speed):

With the plate-composite loss (the production --fins setting) --motion_blur recovers nearly the sharpness of a model trained on unblurred frames (PSNR 26.1 vs 26.4, LPIPS 0.048 vs 0.031, from 0.148) and learns e = 0.42 for a true 0.5. With the masked loss (no plate) it does not work: the target is crop x hard mask on black, so every blur streak that leaves the SAM mask is penalised and the exposure collapses to ~0 (also with a late start); with e fixed at the true value the model is sharper but grows dark rims at the silhouette (PSNR 21.7). Frames with bg_ok false use the masked loss in fins mode and may pull e down. Use --motion_blur only with --bg_composite / --fins. The nearest interpolation (literal plan) failed even before that: one noisy neighbour pose per side is not a usable velocity at the pose noise of this data (refined rotation error ~2-3 deg vs ~3 deg of motion per frame); the windowed fit is the default.

Water (make_synth_fish.py --water --plates: per fish unit bD = (0.08, 0.03, 0.02), bB = (0.06, 0.05, 0.05), Binf = (0.10, 0.42, 0.50), distance 7-18 fish units; the fish turns cyan as in the real models), --bg_composite:

The attenuation is recovered within ~10 %; backscatter coefficient and veiling light trade off against each other (at these distances Binf (1 - exp(-bB z)) ~ Binf bB z: the products .0047/.016/.019 vs GT .006/.021/.025), so the restored colours are still a little pale (mean restored RGB .54 .57 .46 vs GT .57 .51 .42). The baseline's canonical colours are the observed (water-cast) ones. The water model also fits the observed frames better (+1.1 dB) and its gradient-energy ratio is higher mostly because the restored colours have more contrast. On real data the distance range of a fish is smaller and the gain shrink matters more (identifiability); treat the learned values with care.

Notes for the arm runs

• Fair baseline B0x = the gs_final command + --exclude_frames auto on <name>_hf.
• M1 = B0x + --motion_blur 5 (--fins is required in practice, see above). The held-out frames are real (blurred) frames rendered sharp, so PSNR on held-out frames can drop for a sharper model (on _synth_blur_pl the internal val TTO PSNR fell from 31.4 to 30.1 while the GT comparison improved by 3.9 dB): judge M1 by LPIPS, the sharpness ratio and the contact sheets.
• C2 = B0x + --water; quality_eval.py must render it as observed (see above).
• S3: either --frame_select (normalised to mean 1 here) or --filter_file frame_filter_s3.json, not both. Use --frame_select (see "B. Review" below: the filter-file route lowers the mean loss weight to ~0.6 and with it the densification gradients).

B. Review before the arm runs (adversarial check, 2026-11-11)

Scope: fairness of the evaluation, the arm data sets, the trainer options. Scripts used for the checks are one-off (not committed); the fixes are in tests/quality_eval.py, quality_lib.py, 04d_hifps.py, 14_arm_data.py.

Checked, no defect found

• Held-out exclusion in every data set. Independent check (times recomputed from the ids: 10 fps start + (i-1)/10, 30 fps start + (id-1)/30, start = window x 60 s; meta time_s agrees for every frame of every data set): in _s1, _s2, _s3, _l1, _c1 of all 6 fish no frame of meta / poses / frame_filter / frame_select lies within 0.3 s of a held-out frame. For _hf the frames within 0.3 s are exactly heldout.json exclude.hf (fish2 287, w12_fish3 140, w02_fish6 166, w58_fish8 91, w32_fish1 19, w40_fish7 182 ids), which 06_train_gs.py --exclude_frames auto removes (base name parsed correctly for fish2_hf and w32_fish1_hf). Training-eligible frames (valid pose, keep, weight > 0) after the exclusion: _hf 631 / 329 / 218 / 552 / 700 / 429 = S2 (fish2 629) = S3 = L1 = C1; S1 half. The trainer's initialisation, pose-hypothesis repair, robust weights and Binf estimate draw only on train_idx.
• S2 processing = _hf processing on the frame ids both contain (93-700 per fish): identical crop squares, SAM masks (IoU 1.000), soft alpha and plates (MAD 0, fish2 ~1 level), poses within 0.12 deg.
• Default trainer unchanged: by inspection the default path makes the same calls (_render_one = old render_frame, MB = 1, no global optimiser, lam_color_train = lambda_color); the one changed condition (fw != 1 instead of fw < 1) is equivalent because no frame_filter*.json weight in work/ exceeds 1 (135 files checked) and robust weights are <= 1. Together with the HEAD-vs-new runs above (identical loss at iteration 500) no rerun was needed. Unit tests: quality_opts_test.py unit, gs_test_units.py, hifps_unit.py all pass.
• Water model: premultiplied formation J e^{-bD z} + alpha Binf (1 - e^{-bB z}) then + (1 - alpha) plate is right for a plate that already contains the water; depth = gsplat RGB+ED expected camera z (= accumulated z / alpha, bounded by the gaussians' z range even where alpha -> 0; unit test: plane at z = 3 exact in both backends), detached in training (colour cannot move geometry or poses); parameters log / logit (positive, Binf in (0, 1)); per-frame gain applied after the water as in the TTO. Works with --fins (composite unchanged, sheet gaussians attenuated like the body) and --deform (depth of the deformed gaussians, also in the batched hypothesis renders).
• Motion blur: sub-frame offsets tau = s e dt / 2 (total exposure e x frame interval, e = sigmoid, so 0 < e < 1 without a clamp); R(tau) = Exp(tau omega) R with omega fitted from Log(R_j R_i^T) in camera axes = rotation about the fish origin, consistent with p_cam = R p + t; nearest mode sign of f = s e dt / (2 (t_j - t_i)) correct for both neighbours and for extrapolation; motion detached, gradients reach the frame's pose / bending and e; densification sums the sub-render gradients (each carries 1/N of the loss, so the sum has the scale of a sharp render); neighbours limited to training frames of the same segment and < 30 deg (flips). Evaluation renders sharp.

Defects found and fixed

• LPIPS region depended on the model (quality_eval.py): the fish box was the bbox of SAM mask OR rendered alpha > 0.5, so a model with a larger / bloated alpha got a larger box with more (easy) background. Now the box is the SAM mask bbox (+10 %), identical for every model; the old value is kept as lpips_fish_union. B0 recomputed from its stored fit (--reuse_tto; the union value reproduces the old numbers exactly; the previous file is kept as eval/B0/metrics_pre_review.json).
• The sharpness ratio can be gamed by noise. Adding Gaussian noise to the stored B0 composites:

(medians over the held-out frames; the photo itself + 3-level noise scores sharp_match 0.997-0.999.) A 3DGS model with speckle / floaters inside the fish could therefore "improve" the sharpness ratio without any real detail. New metrics in quality_eval.py: sharp_match = sum(grad render . grad photo) / sum |grad photo|^2 inside the eroded mask after a sigma 1.5 px pre-blur of both images (only detail in the right place counts; noise cannot raise it) and sharp_ratio_pb (energy ratio after the same pre-blur, less noise-sensitive). Decision rule: a sharp_ratio gain counts only if sharp_match (and LPIPS) do not get worse. For the dark, low-texture fish (w58_fish8, w32_fish1, w40_fish7) sharp_match is near 0 already for B0 (the photo's in-mask gradients are mostly grain and the render's detail is not in register), so there it is a weak signal; LPIPS stays primary.

• Water (C2) and colour-corrected (C1) models were not evaluated with their image formation. quality_eval.py now renders a --water model as observed (water_observe with the render depth, parameters from model.pt; checked on w12_fish3_hf/qt_smoke_opts), and maps a model trained on a _c1 data set back with 1 / gains_rgb (found via colour_transform.json next to the model dir or in the data set named in model.pt), because the held-out crops / plates are uncorrected. Without it the TTO colour gain would have to undo a 1.4-1.9x red gain against its prior (0.1 x log(gain)^2: ~0.04 for w12_fish3, comparable to the whole loss) and C1 would lose by construction. Checked: with identity gains the metrics are bit-identical to B0. Note: canonical-view gradient energies (views_*) of C1 / C2 models are in corrected / restored colours (more contrast) and are not comparable with the other arms.
• Contamination fallback for model.pt files without frame times assumed 10 fps for every data set not ending in _hf (arm folders are 30 fps); it now reads the data set's meta fps. (New models store their times, so this only matters for old models.)
• S2 lost the background plate on many frames of w32_fish1 and w40_fish7: frames with bg_ok false are trained with the masked loss instead of the plate composite (the production --fins recipe), and the S2 selection ignored plates. Whether a 30 fps frame gets a plate is almost fully determined by its 10 fps frame (P(bg_ok | base bg_ok) 0.96-1.00, P(bg_ok | base not) 0.02-0.18), so it is known before the selection. Fix: 04d_hifps.py --sel-plate-factor F multiplies the arm score of base frames without a plate by F (default 1 = the original S2). New data sets <name>_s2p / <name>_s3p (F = 0.2) for the two fish where S2 had clearly fewer plates than _hf; for the other four S2 already has as many or more plate frames than _hf (fish2 498 vs 503, w12_fish3 188 vs 110, w02_fish6 186 vs 185, w58_fish8 437 vs 436) and _s2 / _s3 stay the arm. Training frames after the exclusion (medians; sharpness = native trimmed Laplacian energy of frame_quality.json):

S2p keeps most of S2's sharpness gain for w32_fish1 with more plates than _hf; for w40_fish7 the sharpness advantage over _hf mostly disappears once plates are required (its sharper visits are the ones without a usable plate). Exclusion re-checked (0 frames near a held-out frame in _s2p / _s3p). _s3p = _s2p with the S3 weights (mean 0.626). Build: FISH_WINDOW=32 04d_hifps.py --fish w32_fish1 --out-suffix _s2p --select quality --exclude-heldout --pose-source like_hf --sel-plate-factor 0.2, 12_frame_quality.py --fish w32_fish1 --dataset _s2p, 14_arm_data.py --fish w32_fish1 --arms s3 --s3_from s2p --s3_to s3p (same for w40_fish7 with window 40). Recommendation: run S2 / S3 on _s2p / _s3p for these two fish (or both variants: the difference isolates the plate effect).

• S3 via --filter_file frame_filter_s3.json is not equivalent to --frame_select: the filter weights are not normalised (mean 0.59-0.63 vs ~0.95 for the other arms), so every loss gradient, including the screen-space gradients that drive densification (--grad_thresh on their mean), shrinks by ~0.65 relative to the other arms (fewer gaussians, relatively stronger regularisers). Use 06_train_gs.py --fish <name>_s3 --frame_select (weights normalised to mean 1 over the training frames).

Remaining caveats (not fixed)

• M1 is scored sharp against real (blurred) held-out frames; PSNR can drop for a sharper model (see above). L1 is scored against the JPEG-derived base crops (bias < 1 level).
• For _hf-based runs with --exclude_frames the excluded frames stay in the data set (untrained); their mask areas still enter the occlusion / area-drop flags of neighbouring frames. Arm folders do not contain them. Negligible information, but not byte-for-byte the same setting.
• The arm score weights (q_arm) and the C1 strength (0.8) are untuned judgement calls.

C. Results: arm matrix (2026-11-11/12)

Setup

• Runner scripts/run_quality_matrix.py: 4 GPU jobs at a time, evaluations before new trainings, 20-minute stale-log watchdog with retry, resumable state in work/quality_exp/matrix/state.json, control.json to skip or add jobs while it runs, and a restarted driver adopts the still-running children of the previous one (it was restarted twice, to add arms; no job was lost or run twice). Every PID is in work/pids_qexp.txt. Report scripts/tests/quality_matrix_report.py writes work/quality_exp/matrix/results.{md,json}, work/quality_exp/<fish>/compare.png (real | B0 | B0x | the 3 arms with the lowest LPIPS, 8 held-out frames on the fish box, plus the canonical flank / top view) and compare_views.png (canonical flank / top / head views of the same models at 384 px).
• Every trained arm: 06_train_gs.py --fish <data set> --fins --deform --iters 4000 --hyp_max_frames 300 --exclude_frames auto [--res 384 for fish2 / w02_fish6] --out q_<arm> (the gs_final recipe plus the exclusion), then tests/quality_eval.py --model <dir> --fish <name> --tag <arm> (evaluator after the review fixes of edf1054). The evaluator reports 0 contaminated held-out frames for every trained model.
• Arms (data sets from section B): B0x (_hf), B1 (_hf, 12000 iterations), S1 (_s1), S2 (_s2; _s2p for w32_fish1 / w40_fish7, review fix 5), S3 (_s3 / _s3p, --frame_select), C1 (_c1), C2 (_hf --water), M1 (_hf --motion_blur 3, windowed motion fit), L1 (_l1). B0 is the existing gs_final evaluation (post-review numbers, not retrained).
• Added to the plan: B0x2 = the B0x command run again (training noise floor; w12_fish3, w58_fish8, w32_fish1) and S1r = a uniform random subset of the _hf frames with S1's size via --frame_select (scripts/tests/make_s1r_select.py, seed 12345; fish2, w02_fish6, w58_fish8, w32_fish1). S1r tells whether S1's effect comes from choosing good frames or just from having fewer frames, i.e. twice the updates per frame at the same iteration count.
• Dropped / shortened to stay near 10 h: L1 only on fish2, w12_fish3, w02_fish6 (the textured fish; the dark, low-texture fish were the least likely to show a JPEG effect); S2 / S3 on the original _s2 / _s3 of w32_fish1 / w40_fish7 (without the plate fix) not run; M1 with 3 sub-renders, not 5.
• Cost: 58 trainings + 58 evaluations, 0 failures, 0 stale restarts, 14:22 -> 00:24 (10.0 h wall). With 4 jobs sharing the GPU a 4000-iteration job took 17-37 min (optimisation 16-31 min), B1 66-90 min, M1 43-61 min (about 1.7x B0x), an evaluation 3-9 min.

Noise floor first

gsplat training is not deterministic. Running the identical B0x command again (B0x2) changes the held-out LPIPS by up to 0.0034, the sharpness ratio by up to x1.14 and masked PSNR by up to 0.15 dB (3 fish). w32_fish1's B0 is a fourth near-rerun: it shares 97 % of B0x's training frames and differs by +0.0018 LPIPS. By the plan's rule a rerun "wins" on 2 of 3 fish, and w32_fish1's rerun is "significantly" worse by the paired Wilcoxon test (p < 0.001, 20 % of frames better). The paired test only covers the sampling of held-out frames, not the training randomness, so per-fish p-values overstate significance. Effects below about 0.0035 LPIPS or x1.15 sharpness on one fish are noise. The table "Beyond the training noise floor" below counts only changes above the largest rerun difference. With 3 reruns that is still a lenient threshold.

Contamination (B0 vs B0x)

On the three fish whose held-out frames were mostly inside gs_final's training window (fish2 16/16, w02_fish6 10/12, w12_fish3 9/24), B0 beats its fair retrain B0x by 0.011-0.020 LPIPS and 0.86-1.35 dB masked PSNR. On the others the gap is 0.004 / 0.2 dB or less. fish2's B0 also had 10000 iterations. Phase A's B0 numbers were optimistic by more than any arm gains, so all comparisons below are against B0x.

Per arm

• B1 (12000 iterations, 3x training time): the only change that sharpens the model consistently.
• Sharpness ratio higher on 6/6 fish (x1.01-4.3, geometric mean x1.66), beyond noise on 4.
• LPIPS better on 5/6 (beyond noise on 4: fish2 -0.011, w12_fish3 -0.005, w32_fish1 -0.006, w58_fish8 -0.004). Worse on w02_fish6 (+0.005 mean), although 67 % of its frames improved.
• PSNR unchanged (mean +0.02 dB masked). Gaussians -3 % (w12_fish3, already at 46.6k near the 50000 cap) to +67 % (fish2) (corrected in review D; first version said 30-50 %).
• Plan rule: 4/6 -> adopt.
• S1 (best half of the _hf frames):
• The sharpest arm: sharpness ratio x1.97 geometric mean, beyond noise on 5/6.
• LPIPS better on 5/6, but only fish2 by much (-0.018).
• Loses masked PSNR (mean -0.18 dB; w12_fish3 -0.63, w02_fish6 -0.22). Halving the frames opens view holes: w12_fish3's oblique / head-on held-out frames 2955 and 5275 turn to mush (compare.png).
• Plan rule: 4/6 -> adopt on paper.
• S1r (random half, same size): LPIPS fish2 -0.008 (S1 -0.018), w58_fish8 -0.005 (S1 -0.002), w32_fish1 -0.003 (S1 -0.004), w02_fish6 +0.003 (S1 -0.002). Sharpness only x0.97-1.34 (S1 x1.11-2.69).
• So about half of S1's LPIPS gain is "fewer frames = more updates per frame" (mean over the 4 fish: S1 -0.0066, S1r -0.0033), the mechanism of B1, which B1 gets without dropping views. The quality selection adds the other half on the fish with a large sharpness spread: S1 beats S1r by more than noise on fish2 (0.010) and w02_fish6 (0.005), not on w58_fish8 / w32_fish1. Its higher sharpness ratio comes from the quality selection: the training frames are sharper. (Corrected in review D; the first version said "mostly".)
• S2 (best frames from all visits, same count as B0x):
• Not sharper (x1.00).
• LPIPS mean -0.0028, beyond noise only on w12_fish3 (-0.011) and w58_fish8 (-0.004).
• Masked PSNR +0.22 dB mean (w58_fish8 +0.64, w32_fish1 +0.30). 2/6 -> no by the rule.
• S3 (S2 + soft quality weights): the best data arm.
• Lowest mean LPIPS of all arms (-0.0045), beyond noise on 3 fish (w12_fish3 -0.017, w58_fish8 -0.006, w32_fish1 -0.006). It is not worse by more than noise on any fish.
• Best PSNR: +0.26 dB masked, +0.17 dB full (w58_fish8 +0.78, w32_fish1 +0.32).
• Not sharper by the ratio (x1.06; x0.64 / x0.82 on w32_fish1 / w58_fish8, dark fish where the ratio mostly measures grain, see the review). The rule therefore gives 2/6 -> no.
• It removes w32_fish1's belly artifact: S2 and S3 (_s2p / _s3p) have no bright patch below the belly and a proper caudal peduncle (compare_views.png). B0, B0x, B1, S1r and C2 all keep it; S1 shrinks it; B1 renders it sharper and more prominent (bright fin-like streaks below the belly).
• Caveat (review D): most of S2 / S3's held-out gain comes from held-out frames whose visit _hf does not cover, and on w58_fish8 all of it does; see D2.
• C1 (plate white balance): no effect (LPIPS +0.0006 mean, PSNR -0.01 dB). The renders are redder by construction. Drop.
• C2 (water model): LPIPS worse on 6/6 (mean +0.0036; fish2 +0.009, beyond noise on 2), PSNR -0.05 dB. The learned attenuation is tiny (bD x scale 0.001-0.02 per half-length). The per-frame colour gain and the plate absorb the water and its parameters are not identifiable on these fish (distance range too small). Drop.
• M1 (motion blur, 3 sub-renders): sharper by the ratio (x1.18), but sharp_match is lower on 3/6 fish and LPIPS is worse on 3/6 (w02_fish6 +0.010, 8 % of frames better). This is the noise-not-detail pattern from the review. The learned exposure is 0.24-0.41, but 0.997 (saturated) for w32_fish1: there the blur absorbs pose error instead of motion. Costs 1.7x. Drop.
• L1 (lossless crops): no consistent effect (w12_fish3 -0.008, w02_fish6 +0.008, fish2 +0.003). Drop.

Visual check (contact sheets and canonical / turntable views)

I looked at compare.png and compare_views.png of all six fish (plus extra sheets with B0x2 / S1r / C2):

• fish2: B1 and S1 show the flank's orange-line stripes slightly crisper than B0x (held-out frames 1065 and 3006, canonical flank). B0's 10000-iteration model looks like B1. S2 / C1 / C2 / M1 / L1 cannot be told apart from B0x.
• w12_fish3: S3 is visibly better on a few frames: the eye bar and stripes in 1512, the thin profile 2755, and 5255. S1 is visibly worse on 2955 and 5275. B0x vs B0x2 already differ as much as most arms do.
• w32_fish1: the only clearly visible model change. S2 / S3 lose the belly artifact and get a cleaner outline; in the held-out composites the difference is small because the artifact sits on the side the camera rarely sees.
• w02_fish6, w58_fish8, w40_fish7: no visible difference between any arms. On the dark fish every model lacks the speckle and the pale fin margins of the photos. w40_fish7 has no usable tail in any arm.
• Honest bottom line: no arm makes the models look sharp. Every render stays far blurrier than the photo. The gap between real and model is much larger than between any two arms. The B1 / S1 / S3 gains are visible only side by side and only on the textured fish. The one change you would notice without a side-by-side is the cleaner w32_fish1 shape from S2 / S3.

Tables (work/quality_exp/matrix/results.md, generated by tests/quality_matrix_report.py)

Held-out set and protocol of section A (identical for every arm). "win" = the plan's rule vs B0x: LPIPS (fish box, mean) lower AND sharpness ratio (median) higher AND sharp_match (median) not lower AND masked PSNR not more than 0.3 dB lower; adopt at >= 4/6 fish.

Per fish x arm (held-out set; PSNR / LPIPS / IoU = means, sharpness = medians)

Deltas vs B0x (arm - B0x); LPIPS paired: share of held-out frames better, Wilcoxon p

Decision rule per arm (wins = LPIPS lower AND sharp ratio higher AND sharp match not lower AND PSNR mask >= -0.3 dB, vs B0x)

Beyond the training noise floor (B0x2 = B0x rerun on 3 fish: max |dLPIPS| 0.0034, max sharp-ratio factor 1.14, max |dPSNR mask| 0.15 dB)

Learned image-formation parameters (M1 exposure fraction; C2 water per fish half-length)

Recommendation (phase C)

Review D (below) re-checked these conclusions and gives the go / no-go per change for the rollout (D5).

• Combine B1 + S3: 12000 iterations on the S3 data, i.e. quality-weighted frames from all visits, plate-aware for fish whose good visits lack plates. B1 is the only consistent sharpening; S3 gives the best LPIPS / PSNR and the better w32_fish1 shape, and keeps view coverage. They act on different things (optimisation per frame vs which frames), so the gains should add up, but this combination has not been run. Cost: 3x training time (66-90 min per fish with 4 jobs sharing the GPU). A cheaper variant worth testing alongside it: S3 at 8000 iterations.
• Do not adopt C1, C2, M1, L1 (no gain or worse; C2 / M1 add cost and parameters that the data do not determine). Do not adopt S1's hard halving: its LPIPS gain is mostly the per-frame-update effect, which B1 gives without the view holes.
• Decide phase C with at least 2 seeds per arm (or pool over more fish). Single runs differ by up to 0.0034 LPIPS / x1.14 sharpness, which is the size of most effects here.
• The dominant blur is still untouched. No arm lowers a fish's LPIPS by more than 10 % (best: S1 on fish2, 0.189 -> 0.171). Phase A pointed at pose / geometry (ellipsoid fit and view direction predict the error), and the experiment agrees: the data arms help mainly through which views and poses go in. The next arm should be pose refinement (P1, section A5). Iterations or image-formation models cannot fix it.

Not done / caveats

• L1 on 3 fish only, B0x2 on 3 fish, S1r on 4 fish. S2 / S3 for w32_fish1 / w40_fish7 only in the plate-fixed variant.
• M1 with N = 3 (the synthetic test used 5).
• The noise floor rests on 3 reruns (+ the near-rerun w32_fish1 B0) and is therefore a lower bound.
• The canonical-view gradient energy (flank view grad E) is unstable on the dark fish: w32_fish1 jumps x22-43 between models with the belly artifact's edge. C1 / C2 views are in corrected / restored colours. Do not use it for decisions.
• Sharpness ratios of the dark fish (w58_fish8, w32_fish1, w40_fish7) are partly grain (review); their sharp_match stays at 0.02-0.17 in every arm.
• No phase C run, no gs_v2, no video or website changes.

Reproduce (from scripts/, venv python): python -u run_quality_matrix.py --slots 4 (B0x2 / S1r via work/quality_exp/matrix/control.json {"add": ["w12_fish3:B0x2", ..., "fish2:S1r", ...]} after python tests/make_s1r_select.py --fish fish2 w02_fish6 w58_fish8 w32_fish1), then python tests/quality_matrix_report.py.

D. Adversarial review of the results (2026-11-12)

A second, independent check of section C before the rollout decision. Script: scripts/tests/quality_review_d.py (reads only metrics.json, train_log.json, meta.json and heldout.json; no GPU).

D1. Re-derived and checked: no defect

• Tables. All the per fish x arm numbers, the deltas vs B0x, the Wilcoxon p-values, the plan-rule wins and the "beyond noise" counts of section C reproduce exactly from the per-frame values of the 68 metrics.json files.
• Same held-out set. Every arm of a fish was evaluated on the same held-out ids, in the same order, equal to heldout.json. Every evaluation used the same settings: 256 px, 200 + 20 steps, LPIPS alex. Every trained arm has 0 contaminated frames.
• Same training settings except the arm's change. Within a fish, crop_res is the same in every data set (512 px for fish2 / w02_fish6, 256 px for the others), and so are --res (384 for the two 512 px fish, native otherwise), seed 0, --hyp_max_frames 300 and frame_filter.json. Trained frames after the trainer's validation split: the _hf, S2, S3, C1 and L1 arms are within 2 frames of each other; S1 / S1r have half (by design).
• Two evaluations re-run from scratch (fresh test-time fit, new tags B1_rev2 / S3_rev2):
• fish2 B1: LPIPS 0.17806 vs 0.17806, masked PSNR 20.451 vs 20.451, sharp_match 0.1634 vs 0.1634.
• w12_fish3 S3: LPIPS 0.2134 vs 0.2135, masked PSNR 16.31 vs 16.30, sharp ratio 0.521 vs 0.521.
• The evaluator is reproducible to about 1e-4 LPIPS, far below the training noise.

D2. Confound: S2 / S3 train next to held-out frames that _hf never came near

The held-out set is drawn from all visits (section A). _hf covers 1-3 visits; S2 / S3 draw from all of them. So the S2 data put training frames close in time (beyond the 0.3 s exclusion) to held-out frames whose visit _hf does not contain at all:

These are exactly the three fish where S2 / S3 beat the noise floor. Per held-out frame, dLPIPS vs B0x split by proximity (3 s radius, pooled over fish):

How to read this:

• Where _hf already had nearby frames, S2 / S3 gain nothing (+0.0007).
• The whole gain is on frames from visits that _hf does not cover.
• Part of it is real generalisation. Frames far from all training data of both models improve as well: w12_fish3 5255-5275 by -0.02 to -0.04 (S3 is even 5 s farther from them than B0x), and w32_fish1 (23 frames) by -0.007.
• On w58_fish8 all of the S2 / S3 gain sits on the 17 newly covered frames. Its single far frame is worse, and compare.png shows no visible improvement (S3 is slightly worse on 219, 4571 and 5327).
• The held-out design favours the all-visits arm by construction. S2 / S3's mean LPIPS gain is therefore partly "has seen this visit" (its lighting, distance and fin state), not a sharper model. B1's gain does not have this problem: it is the same or larger on frames far from training.

D3. Corrections and additions to section C

• S1 vs S1r (corrected in C): about half of S1's LPIPS gain is the fewer-frames effect (S1 -0.0066 vs S1r -0.0033, mean over the 4 fish), not "most". S1 beats S1r by more than noise on fish2 and w02_fish6, the fish with the largest sharpness spread among the selected frames (1.73x, 2.27x).
• B1 changes more than the iteration count. Several schedule settings scale with --iters or run at a fixed interval: densification runs until 60 %, hypothesis rounds every 1500 iterations until 70 %, a robust round every 500 iterations. So B1 gets 5 hypothesis rounds instead of 2 and 23 robust rounds instead of 8.
• The measured effect of this is small: 0-2 extra pose switches per fish, and 0 robust drops in either run.
• This is what --iters 12000 does in production, so it is part of the change, not a defect.
• Gaussians: -3 % to +67 % (corrected in C). w12_fish3 is at 46.6k, close to the 50000 cap, so B1 cannot densify there. That may be why B1 does nothing for its sharpness.
• Sharpness ratio is partly grain. After the 1.5 px pre-blur (sharp_ratio_pb), B1's sharpening is x1.25 as a geometric mean (x1.66 raw).
• It is beyond the rerun noise (pre-blurred, up to x1.12) on fish2, w58_fish8 and w32_fish1 (x1.23-1.28).
• w40_fish7's x4.32 raw / x2.00 pre-blurred is not trustworthy. That fish has no rerun, and even C1 / C2 / M1 reach x1.57-1.71 there.
• S1 is x1.52 pre-blurred and S3 x1.03.
• Visual check. I opened compare.png of fish2, w12_fish3 and w58_fish8 and compare_views.png of w32_fish1. The q_ models have no turntable videos; the canonical views are their turntable frames.
• I agree with section C on fish2 (B1 / S1 crisper stripes than B0x) and on w12_fish3 (S2 / S3 better on 1512, 2755 and 5255; S3 worse on 3012).
• New finding: on w32_fish1, B1 sharpens the belly artifact into more prominent bright streaks. More iterations sharpen errors as well as detail.
• A rollout side effect of S2 / S3 that section C missed. tracking.mp4 and turntable_swim.mp4 (10_fish_videos.py) replay the longest contiguous runs of the data set. S2 replaces those runs with many short ones, so the replayed clips get shorter:

S2 also needs the SAM + matting stage per fish (6-35 min of GPU each) and, for some fish, the plate-aware variant.

D4. What stands

• B1 is the only change with a consistent gain that does not depend on which visits the training data cover:
• LPIPS beyond noise on 4/6 fish; worse beyond noise on w02_fish6 (+0.0048).
• 78 % of the held-out frames far from any training frame improve.
• Grain-free sharpening beyond noise on 3 fish; PSNR flat.
• S3's gain is real on w12_fish3 and w32_fish1 (including far frames) and fixes w32_fish1's belly artifact. Its mean LPIPS advantage is inflated by visit coverage (D2), and on w58_fish8 it is not supported at all.
• Section C's bottom line is unchanged: no arm closes more than about 10 % of any fish's LPIPS, and every render stays far blurrier than the photos.

D5. Go / no-go per change for the gs_v2 rollout

E. Phase C and rollout: gs_v2 on all 44 fish (2026-11-12)

This is the "rollout" step of the plan (phase C). Section D was already taken by the review, so it is E here.

E1. Recipe

Only B1 got a GO in D5, so there was nothing to combine. The v2 recipe is B1 plus the watch-out from D5, a higher gaussian cap:

06_train_gs.py --fish <name>_hf --fins --deform --iters 12000 --hyp_max_frames 300 [--res 384 for 512 px crops] --max_gauss 100000 --out gs_v2

Everything else is as in gs_final: the same data set (<name>_hf), seed and trainer. S3 was not combined (NO-GO in D5), and C1 / C2 / M1 / L1 were not used. There is no --water, so the turntables show the normal gaussian colours.

E2. Verification on the 6 fish (arm v2 = q_v2, with the held-out exclusion)

run_quality_matrix.py --arms v2 --state work/quality_exp/matrix/state_v2.json (same protocol as section C: --exclude_frames auto, 0 contaminated held-out frames). Report: tests/quality_matrix_report.py --arms B0,B0x,B0x2,B1,v2 --suffix _v2 -> work/quality_exp/matrix/results_v2.{md,json}, work/quality_exp/<fish>/compare_v2.png / compare_views_v2.png (real | B0 | B0x | B1 | v2).

• Decision rule: v2 wins on 4/6 fish vs B0x (fish2, w12_fish3, w58_fish8, w32_fish1), the same count as B1. It is never worse than B0x by more than the noise floor (B1 was worse beyond noise on w02_fish6). So v2 is adopted.
• v2 vs the best single arm (B1): the same within noise. dLPIPS v2 - B1 is -0.0003 / -0.0002 / -0.0035 / +0.0008 / -0.0009 / +0.0019 (mean -0.0004). For the five fish below the old cap, v2 is in effect a second independent run of B1. The two runs agree on the sign of dLPIPS on all 6 fish: better on 5 in both runs, and worse on w02_fish6 in both (+0.0048, +0.0013). That makes the 12000-iteration effect more solid than one run could.
• The cap matters only for w12_fish3. It goes from 45k (capped) to 86k gaussians. LPIPS is unchanged vs B1; the grain-free sharpness is x1.19 vs x1.03 (beyond the x1.12 noise) and masked PSNR is +0.11 dB vs B1. In compare_views_v2.png its flank stripes look about as crisp as B1, perhaps slightly crisper. Nothing got worse, so the cap is kept for all fish. It only binds on fish that reach 50k.
• Visual check (compare_v2.png of fish2 and compare_views_v2.png of w12_fish3): v2 looks like B1. The fish2 stripes are slightly crisper than B0x (held-out frames 254, 286 and 3033). Every render remains far softer than the photo.

<!-- E3+ (rollout) filled in after the 44-fish run -->

Results: every fish of the video, reconstructed

Final production run on 2026-11-10 (13:15 to 20:15, one RTX 4090, at most 4 GPU jobs at once). Method: see docs/method_choice.md. Each fish gets 30 fps data from 04d_hifps.py, capped at 24 s of footage. It is then trained with 4D Gaussian splatting and the fin preset (06_train_gs.py --fins --deform --iters 4000 --hyp_max_frames 300 --out gs_final), and its videos come from 10_fish_videos.py.

• 44 fish: fish1-3 of the original 48:00-58:00 window plus 41 fish from seven more windows (02, 12, 22, 32, 40, 58, 68 min). All 44 have a model, species.json, turntable.mp4, turntable_swim.mp4, tracking.mp4 and poster.png.
• No fallbacks were needed. Every 04d run produced at least 245 valid 30 fps poses (the fallback threshold was 200). Every training finished on its first attempt, and no job stalled. fish2 and fish3 reuse their existing 10000-iteration 30 fps fin models (gs_fins_4d, copied to gs_final). All other fish were trained here with 4000 iterations.
• Data: 29,326 crops at 30 fps (24,198 with a valid pose) from 25,529 kept 10 fps frames.
• Species (BioCLIP 2.5 zero-shot over 332 Red Sea species, 64 crops per fish): 30 high, 8 medium and 6 low confidence. 11 yellowfin hind, 8 chocolate-dip chromis, 6 checkerboard wrasse, 5 orange-lined triggerfish, 5 "cheeklined wrasse" (all low or medium confidence, see below), 3 Eritrean butterflyfish, 2 "blacktip grouper" (low), 2 redmouth grouper, 1 bluestreak cleaner wrasse, 1 coral hind.
• Held-out quality: every 10th valid frame is held out, and its pose and colour are re-fitted with the model frozen.
• Full crop composited over the background plate: median 26.4 dB (19.7 to 32.5).
• Masked PSNR: median 21.2 dB.
• IoU against the SAM mask: median 0.881 (0.775 to 0.936).
• Runtime per fish (median, with 4 jobs sharing the GPU): 04d 12 min (w02 fish up to 50 min, see below), training 25 min including the final evaluation, videos 75 s.
• Visual review (see "Review" at the end and the review column): 24 models good, 15 fair, 5 poor. The 5 poor ones are all large dark hinds (w22_fish1, w32_fish5, w40_fish7, w58_fish1, w58_fish5); they stay in the table but are moved to a separate "weak models" section of the gallery. 7 species labels are marked doubtful.

Where to look:

• work/gallery.html: one card per fish, with the poster, species label and links to the videos.
• work/site/ (built by scripts/11_site.py, served by scripts/serve_site.py): the full results website, with per-fish pages and an interactive 3D viewer.
• Per fish: work/[w<MM>/]fish/<name>_hf/gs_final/, which holds point_cloud.ply, poses_refined.json, train_log.json, val_grid_comp.png and the videos/ folder. The species file is work/[w<MM>/]fish/<name>/species.json.
• Machine-readable summary: work/production/results.json. Driver state and logs are in work/production/ and work/logs_produce/.

Reproduce: cd scripts; ..\.venv\Scripts\python -u run_production.py --slots 4 (resumable: it skips steps whose outputs exist), then ..\.venv\Scripts\python make_results.py to rebuild the table below and work/gallery.html.

All fish

Column notes:

• seen: the 10 fps visits of the window (all of them, not only the 30 fps segments).
• frames: kept 10 fps frames / 30 fps crops used for the model.
• full / masked: held-out PSNR in dB, after the per-frame re-fit.
• IoU: against the SAM mask on held-out frames.
• review: the visual grade of the 3D model (good / fair / poor) and whether the species label is doubtful. It comes from docs/review_notes.json; see "Review" below.

<!-- table -->

<!-- /table -->

Discussion

What works

• The whole chain runs unattended over the whole video. Detection, tracking, identity filtering, segmentation, pose estimation, 30 fps extraction, background plates, 4D GS and videos all ran without a fallback or a crash for 44 fish of 10 species. The data cleaning of round 4a (identity filter, dropped mixed or occluded fish) is what makes this possible.
• The models align with the real crops. Tracking videos and posters show the model rendered at the refined pose on top of the real crop, and the outlines match (median IoU 0.88).
• Distinctive fish come out recognisably:
• Eritrean butterflyfish: stripes, the dark rear patch and the deep disc-shaped body (w12_fish3).
• Bluestreak cleaner wrasse: black stripe on blue (w02_fish6).
• Orange-lined triggerfish: yellow caudal fan, translucent dorsal and anal fins (fish2, w58_fish6).
• Chocolate-dip chromis: two-tone body with a forked tail (fish3, w22_fish3).
• Checkerboard wrasse: the pattern.
• The fin preset generalises to wrasses and butterflyfish. These were untested before this run. On them it gives smooth bodies with fin outlines and no floaters, and the turntable_swim videos replay the body bending. On the large dark hinds it works less well (see below). One model has a baked-in artifact: w32_fish1 has a bright, sand-coloured patch on the belly that shows in the side views of its turntable.
• fish1 (the resident grouper of the default window) looks better than before. With the 30 fps preset its held-out IoU is 0.926 and masked PSNR 24.3, against IoU 0.86 for the old 10 fps gs_robust. This is not a like-for-like comparison: the held-out frames, the frame rate and the PSNR definition all differ. The model is still a plain dark body (graded fair).
• Species overlay: the top-1 name, probability and confidence label appear on every turntable, tracking video and poster. The high-confidence calls match what the contact sheets show.

Failure modes and caveats (honest list)

• Groupers are the weakest group. They are large and dark, and they hover near coral that cuts their silhouette. Their per-frame colours are dark against a bright background. The lowest scores are all hinds:
• w40_fish7: 19.7 dB full; its turntable is a blob with an odd outline. It was 35% coral-cut.
• w58_fish5: 21.0 dB.
• w58_fish1: 22.2 dB.
• w32_fish5: 22.4 dB.
• w22_fish1: IoU 0.775.
• w12_fish1 and w40_fish1: IoU 0.82-0.83.

Their 3D bodies look plausible from the side but are flat and featureless. The model at the pose looks slightly thinner than the real fish, and the spots are lost.

• Full-crop PSNR measures the background plate as much as the fish. w12_fish3 (the butterflyfish, one of the visually best models) scores only 21.7 dB full but 17.1 masked: it swims in front of waving soft coral, which the static plate cannot reproduce. Compare fish by masked PSNR and IoU, and use the full-crop number only as a check for gross errors.
• Canonical colour is tinted cyan or green in the turntables. The per-frame colour gain absorbs the water colour, so the canonical colours are not true colours. Spots and fine patterns are smeared. Fin sheets carry some background colour.
• Tail articulation is small. The caudal joint barely moves on real data; body bending does most of the swimming motion in turntable_swim.
• Only up to 24 s of 30 fps footage per fish. The tracking video covers those segments, not every visit (the "seen" column lists all visits).
• The "cheeklined wrasse" groups are not trustworthy.
• w02_fish8, w02_fish9, w02_fish10, w12_fish9 and w58_fish10 are drab brown wrasses merged from many short tracklets. Species confidence is low (36-46%), except w02_fish8 (medium, 67%, possibly a parrotfish).
• They may mix individuals of similar species.
• Their models are plain brown bodies.
• Treat the species as "unknown wrasse".
• The "blacktip grouper" calls are low confidence. w32_fish3 (60%) and w40_fish5 (62%) are barred brown fish; the label is a best guess.
• The same individual is counted in several windows. The resident yellowfin hinds, triggerfish and chromis are surely the same animals across windows. Each window's fish is a separate entry; individuals were not matched across windows.
• Coral occluders. Besides w40_fish7, these fish still have many coral-cut silhouettes: w22_fish3 (39%), w32_fish4 (28%), w12_fish5 (32%), and w32_fish5 (behind coral for its first ~9 s). Their models are plausible, but parts hidden in most frames are guessed from symmetry.
• The w02 window is slow in 04d (background stage ~4 s per crop instead of ~1 s; w02_fish1 took 51 min). Only the runtime suffers.
• The iteration count differs. fish2 and fish3 use 10000 iterations, the rest 4000. The decision step measured the cost of 4000 iterations at 0.2-0.35 dB.
• Gaps in the tracking videos for butterflyfish. The deep-bodied butterflyfish have no 3D pose in 33-50% of their tracking frames (w12_fish3 36%, w22_fish4 33%, w40_fish6 50%). Those frames show the crop and a "no 3D pose for this frame" panel.
• Visit lists differ slightly between the poster and this table. The poster merges detections into visits with its own gap rule (e.g. fish2: 6 visits on the poster, 5 here). Both come from the same detections.

Review

The review step (2026-11-10) checked every row of the table against the files on disk and looked at the outputs. The grades and notes are in docs/review_notes.json; make_results.py reads that file for the review column and the gallery.

Checks against the files (all passed):

• All 44 fish have point_cloud.ply, model.pt, train_log.json, poses_refined.json, species.json, turntable.mp4, turntable_swim.mp4, tracking.mp4, poster.png and videos.json.
• Every video decodes with ffprobe: turntable 300 frames, turntable_swim 154-450 frames, tracking 487-1276 frames at 1920x1080. Every video and poster is newer than its model.
• The full and masked PSNR and the IoU in the table match final.val_tto of each train_log.json to the printed precision.
• For every fish, the species and probability in the table and in the video overlay (videos.json) are the top-1 of its species.json.
• The fish2 and fish3 gs_final models are byte-identical to their gs_fins_4d. All other logs show --fins --deform, 4000 iterations and --out gs_final.
• The summary totals are correct: species counts, 30 / 8 / 6 confidence labels, 29,326 crops at 30 fps with 24,198 valid poses, 25,529 kept 10 fps frames, and the medians.

What I looked at: all 44 posters, frames from the tracking, turntable and swim videos of 8 fish (fish2, w02_fish8, w12_fish3, w22_fish1, w32_fish1, w40_fish5, w40_fish7, w58_fish3), the 30 fps crop sheets of w40_fish7 and w32_fish1, and turntable tiles of the weakest hinds.

• Overlays: the species label, probability and confidence colour are present and legible on every poster, tracking frame and turntable.
• Alignment: the render at the refined pose lines up with the crop, including the head-on frames of fish2. The worst alignment is on the head-on frames of the hinds (w22_fish1).
• No mixed-species fish or junk fish were found among the 44. w40_fish7 looked suspicious (merged-looking silhouettes). Its crop sheet shows one yellowfin hind, cut by soft coral and often seen head-on. It is kept, but graded poor.

Grades:

• Good (24): fish2, fish3, w02_fish1, w02_fish2, w02_fish4, w02_fish6, w02_fish7, w12_fish2, w12_fish3, w12_fish5, w22_fish2, w22_fish3, w22_fish4, w32_fish4, w40_fish2, w40_fish4, w40_fish6, w58_fish6, w58_fish7, w58_fish8, w58_fish11, w68_fish1, w68_fish3, w68_fish5. The shape and pattern of the species are recognisable.
• Fair (15): fish1, w02_fish8, w02_fish9, w02_fish10, w12_fish1, w12_fish9, w32_fish1, w32_fish3, w40_fish1, w40_fish3, w40_fish5, w58_fish2, w58_fish3, w58_fish9, w58_fish10. The body is plausible and aligned, but plain, very dark, or (w32_fish1) has an artifact.
• Poor (5): w22_fish1, w32_fish5, w40_fish7, w58_fish1, w58_fish5. A featureless oval or blob with no readable tail. The hind problem described above is worst here.

Doubtful species labels (7):

• w02_fish8: its blunt head and lunate tail look more like a parrotfish.
• w02_fish9, w02_fish10, w12_fish9 and w58_fish10: brown wrasses of unknown species.
• w32_fish3: low-confidence "blacktip grouper".
• w40_fish5: low-confidence "blacktip grouper"; a slender fish with a pointed snout whose runners-up are wrasses.

The species pages of work/site show the classifier output without these flags.

Best results to look at first:

• fish2 (triggerfish, translucent fins)
• w12_fish3 and w22_fish4 (butterflyfish)
• w02_fish6 (cleaner wrasse)
• fish3 and w68_fish1 (chromis)
• w58_fish11 and w02_fish1 (checkerboard wrasse)
• w58_fish8 (redmouth grouper)

Method choice for the final production

Decision (2026-11-10): 4D Gaussian splatting with the fin preset on 30 fps data (06_train_gs.py --fins --deform on the <name>_hf crops of 04d_hifps.py), shortened to 4000 iterations with --hyp_max_frames 300, and 04d capped at 24 s of 30 fps footage per fish (--max-total-hard-s 24, new option). The output folder is gs_final. fish1 is kept.

Candidates

Held-out metrics

Every 10th valid frame is held out; per-frame pose / colour are re-fitted on it (TTO) and the model is frozen. Full = PSNR of the render composited over the plate on the whole crop, masked = PSNR on the SAM mask, IoU against the SAM mask, near-fish = PSNR in the band just outside the mask (where fins live). Rows of one block use the same held-out frames. Sources: work/fish/<f>/eval_cmp/compare.json (round 3b), the review re-evaluation work/fish/<f>_hf/eval_rev/ and the train logs of the runs made for this decision (dec_*, logs in work/logs_decide/).

10 fps data (held-out 10 fps frames)

30 fps data (held-out 30 fps frames of fish2_hf / fish3_hf)

Notes on fairness: the round-3b table re-fitted each model with its own loss (composite for the fins models, masked for the others), which flatters the fins models. The "fair" rows re-fit the no-fins 30 fps ablation with the composite loss too: the fins advantage is then +0.8 dB full / +0.95 dB masked on fish3_hf and +1.5 / +2.1 dB on fish2_hf. The 10 fps baselines inside the 30 fps blocks were not re-fitted that way (their gap is larger than the fair one). Near-fish values depend on the far_r0 / far_r1 training values; all rows here use the defaults.

Visual judgement (grey turntables, composited validation grids)

• fish3 (chromis): only the 30 fps fins models show the forked caudal fin (10000 and 6000 iterations alike). The 10 fps models (gs_robust_4d, gs_fins, and the new 10 fps fins + deform run) all end in a paddle / smeared tail, despite the better 10 fps numbers of the last one.
• fish2 (triggerfish): the 30 fps fins models show the translucent dorsal and anal fin sheets and the yellow caudal fan; the 4000-iteration production preset keeps all three. The 10 fps models have no fin sheets and a ball-shaped tail.
• fish1 (grouper, default window): gs_robust is a plausible but blob-like body (no fins, flat ventral side). The composited validation renders align with the crops, the metrics are above the baseline (24.63 vs 24.26 PSNR, IoU 0.86) and the species call is medium confidence (Cephalopholis hemistiktos 68%). The frames are hard (coral in front, cleaner wrasse on the body, other fish crossing), but this is the main resident grouper of the default window, so it is kept and retrained with the final preset like every other fish.
• w58_fish3 (coral hind, first window fish through the whole chain): 04d -> production preset -> 07 turntable -> 10_fish_videos poster all ran with FISH_WINDOW=58. The model has the grouper outline with a rounded caudal fin and pectoral / pelvic hints; composites align with the crops (comp 17-29 dB per frame). The canonical colour is cyan-tinted and the spots are smeared (the per-frame colour gain restores the brown in the composites). The poster shows species, confidence and visit times correctly.
• No floaters in any turntable; none of the models is junk.

Runtime (RTX 4090, two jobs sharing the GPU)

Expected production cost per new fish: 04d with the 24 s cap (~720 frames) ~14 min, training ~20 min. For the 42 fish to train (fish1 + 41 window fish; fish2 / fish3 reuse their existing 30 fps models; w58_fish3 is already done) that is about 24 GPU-job-hours; with 2 concurrent jobs about 12 h, with 3-4 jobs ~8 h if the GPU throughput scales (it is ~90 % busy with two jobs, so expect less than linear scaling). The data preparation can run in one slot while the other slots train. If this is too slow, use the knobs in that order: drop to --iters 3000, lower the cap to --max-total-hard-s 18, then train the low-priority fish (below) with the 10 fps fallback (no 04d).

Why this choice

• It is the only method whose models show fins and a forked tail on real data, and it is also the best on full-crop, masked and near-fish PSNR on both test fish, after the fair re-fit.
• 4000 iterations lose 0.2-0.35 dB against 10000 (fish2_hf 25.05 vs 25.40, fish3_hf 27.80 at 6000 vs 27.98) and keep the visible fin structure; the third repair round of a 6000-iteration run switched no frames, so stopping repair at 2 rounds costs nothing.
• The robust 3D / 4D presets at 10 fps have a slightly higher IoU against the SAM mask, but only because the SAM mask itself excludes the translucent fins. That is not a quality criterion for fin reconstruction.
• 10 fps fins + deform (no 04d needed) is the best 10 fps option by the numbers, so it is the fallback whenever 04d gives too little 30 fps data.

Exact command lines

Run from scripts/ with the venv python; window fish need FISH_WINDOW=<window minute> (the fish's window in work/fish_index.json; default-window fish fish1-3 run without it).

cd scripts
# 1) 30 fps data (once per fish; several fish of one window may share one call)
$env:FISH_WINDOW='58'
..\.venv\Scripts\python -u 04d_hifps.py --fish w58_fish5 w58_fish6 --max-total-hard-s 24
# 2) training (add --res 384 when work/w58/fish/<name>_hf/meta.json has crop_res 512)
..\.venv\Scripts\python -u 06_train_gs.py --fish w58_fish5_hf --fins --deform --iters 4000 --hyp_max_frames 300 --out gs_final
# 3) videos (species.json is taken from the base fish dir)
..\.venv\Scripts\python -u 10_fish_videos.py --fish w58_fish5_hf --model gs_final --window 58
Remove-Item Env:FISH_WINDOW

Template: 06_train_gs.py --fish <name>_hf --fins --deform --iters 4000 --hyp_max_frames 300 [--res 384] --out gs_final

Special cases:

• fish2, fish3 (default window): the _hf data and the 10000-iteration gs_fins_4d models exist and are the best models available. Copy work/fish/fish2_hf/gs_fins_4d to work/fish/fish2_hf/gs_final (same for fish3) instead of retraining.
• fish1 (default window): 04d_hifps.py --fish fish1 --max-total-hard-s 24 (04d's fish1-specific defaults allow at most 2 segments), then the template on fish1_hf (crop_res decides --res).
• w58_fish3: done during this decision step with the exact template (data without the 24 s cap: 1192 frames / 39.6 s), output work/w58/fish/w58_fish3_hf/gs_final.
• Fallback (10 fps, no 04d), when 04d gives fewer than ~200 valid 30 fps poses (hifps_report.json -> poses.valid), no segment of at least 2 s, or fails: 06_train_gs.py --fish <name> --fins --deform --iters 4000 --hyp_max_frames 300 --out gs_final (videos with --fish <name>).
• If a training fails twice: robust 3D preset, 06_train_gs.py --fish <name> --iters 6000 --out gs_final.
• Do not run 04b / 05 without --only: they iterate over every fish dir with a meta.json and would also touch the *_hf dirs.
• 04d's pose source is auto: newest gs*/poses_refined.json of the base fish, else the raw 05 poses (the case for all window fish). The robust hypothesis repair of 06 fixes the remaining flips (w58_fish3_hf: 86 % valid interpolated poses, median centre error 0.9 px).

Priority order (train the fish with the best material first, so a time-out hits the weakest): species-confident, large fish first; last the drab-wrasse groups that may mix individuals (w02_fish8, w02_fish9, w02_fish10, w12_fish9, w58_fish10) and the heavily coral-occluded fish (w22_fish3, w40_fish7, w12_fish5, w32_fish4, w32_fish5).

Known limits of the chosen method

• The 30 fps model sees only the 04d segments (here at most 24 s per fish), so its tracking video covers only those segments, not every visit of the fish.
• Caudal articulation is barely used on real data (1.5-3.7 deg of per-frame tail angle); body bending does most of the work.
• Fin colour includes some background (fish2's fin sheets are pale green).
• About 2.5x slower per iteration than the robust 3D preset; hypothesis repair dominates for fish with many frames.
• Synthetic plates are easier than the real ones; the real-data gain was shown on two fish only (one triggerfish, one chromis), plus the window grouper w58_fish3 with the production preset (no baseline comparison there); wrasses and butterflyfish are untested.
• Training alone is not much faster than with two jobs (w58_fish3: 1015 s alone vs 1030 s for fish2_hf next to a 04d job), so 2-3 parallel jobs give close to linear throughput only while the GPU is not saturated; check before going to 4.