Nathan fm - #79
Open
nathanchenseanwalter wants to merge 165 commits into
Open
Conversation
Changed default hyperparameters in the models. Added demo for profile reconstruction. Added script for dataset standardization (has to be run once before model training to store normalization coefficients).
…, the wrong configuration was used to find the correct signal name. Also, removed warning for duplicated tensor conversion.
…for debugging purposes.
…for debugging purposes.
…the dataset class.
…s and opening an H5 file prior to distributing the dataset across all workers. Significant updates in the Fast time series baseline and actuator reconstruction classes.
The basic encoders are now all working. Examples are in scripts.
- Model - Optimizer state - Scheduler state - Current loss - Current epoch For the sake of continual training.
- Model - Optimizer state - Scheduler state - Current loss - Current epoch For the sake of continual training.
…got to remove unused modalities. This follows the standard getitem function now.
Quick fix for the data standardization. Invalid values have to be ignored. Fix in the function to create H5 files. bolo data does not have to be flipped anymore as the data is now stored in the correct format.
* Nathan fm (#53) * chore: Update `pyproject.toml` to reorder authors, enhance README with environment setup instructions, and add validation notes in `validation.txt`. Refactor `dummy_model_2.py` for improved modality configuration and introduce `TextEncoder` enhancements in `text_baseline.py`. * Refactor demo scripts to utilize new `Prediction4FusionModel` and `DictMSELoss`. Update `run_demo_2.py` and `run_demo_3.py` for improved model initialization and data handling. Enhance `TokamakH5Dataset` to handle degenerate signals and improve data extraction logic. Remove unused `latent_space.py` and integrate new modality fusion models in `modality_fusion.py`. * Remove unused shot list configuration files and refactor trainer class to introduce MultimodalTrainer and UnimodalTrainer for improved training structure. * Refactor modality models and trainer classes for improved structure and functionality. Removed unused TimeSeriesEncoder and Decoder, introduced FastTimeSeriesEncoder and SpectrogramAutoEncoder. Updated UnimodalTrainer to support logging and checkpoint management. Enhanced TokamakH5Dataset for better data handling and added checkpoint loading functionality in spectrogram reconstruction script. * Add padding collate function and update training script for unimodal autoencoder - Introduced `collate_fn_pad` to handle variable-length tensors in batches. - Updated `train_unimodal_autoencoder.py` to use the new collate function. - Modified `train_unimodal.sh` to include additional signal modalities for training. - Added new autoencoder classes for fast time series and spatial profile modalities, ensuring output shape consistency with adaptive pooling. - Enhanced video autoencoder implementation for better reconstruction quality. * Remove spectrogram reconstruction script and refactor modality models - Deleted `spectrogram_reconstruction.py` as part of the restructuring. - Refactored modality models to introduce baseline versions for actuator, slow time series, fast time series, spatial profile, spectrogram, and video. - Updated model registry and signal-to-model mappings to reflect new baseline architecture. - Enhanced `TokamakH5Dataset` to support additional parameters for FFT and hop length. - Improved training script for unimodal autoencoders to utilize new baseline models and added support for variable-length tensors. * Update .gitignore to include pixi environments and add link to HSI-compression-benchmark in SpectrogramBaselineAutoEncoder docstring * Remove unused shot list files and delete deprecated scripts for training and data handling * Remove deprecated training scripts for CO2, ECE, MHR, and unimodal training * Dev peter (#48) * Removed the argument "batch_size" from the trainers. Changed default hyperparameters in the models. Added demo for profile reconstruction. Added script for dataset standardization (has to be run once before model training to store normalization coefficients). * Bugfix in the dataset class. When iterating over movie configurations, the wrong configuration was used to find the correct signal name. Also, removed warning for duplicated tensor conversion. * Added base script for video reconstruction. Copied from Aza's branch for debugging purposes. * Added base script for video reconstruction. Copied from Aza's branch for debugging purposes. * Minor changes in the example scripts. More preprocessing options for the dataset class. * Fixed a bug where the dataset class failed when using multiple workers and opening an H5 file prior to distributing the dataset across all workers. Significant updates in the Fast time series baseline and actuator reconstruction classes. * Lots of bugfixes in the dataset, trainer, and models. The basic encoders are now all working. Examples are in scripts. * Dev peter (#50) * Removed the argument "batch_size" from the trainers. Changed default hyperparameters in the models. Added demo for profile reconstruction. Added script for dataset standardization (has to be run once before model training to store normalization coefficients). * Bugfix in the dataset class. When iterating over movie configurations, the wrong configuration was used to find the correct signal name. Also, removed warning for duplicated tensor conversion. * Added base script for video reconstruction. Copied from Aza's branch for debugging purposes. * Added base script for video reconstruction. Copied from Aza's branch for debugging purposes. * Minor changes in the example scripts. More preprocessing options for the dataset class. * Fixed a bug where the dataset class failed when using multiple workers and opening an H5 file prior to distributing the dataset across all workers. Significant updates in the Fast time series baseline and actuator reconstruction classes. * Lots of bugfixes in the dataset, trainer, and models. The basic encoders are now all working. Examples are in scripts. * Extended checkpointing - the trainer stores now: - Model - Optimizer state - Scheduler state - Current loss - Current epoch For the sake of continual training. * Extended checkpointing - the trainer stores now: - Model - Optimizer state - Scheduler state - Current loss - Current epoch For the sake of continual training. * Adapted the other reconstruction scripts to match the new API. * Bugfix in the dataset class. When splitting inputs and targets, I forgot to remove unused modalities. This follows the standard getitem function now. * Prepared an option to preprocess movies. This has to be fully integrated!!! --------- Co-authored-by: Peter Steiner <61472983+renierts@users.noreply.github.com> * Dev peter (#55) * Removed the argument "batch_size" from the trainers. Changed default hyperparameters in the models. Added demo for profile reconstruction. Added script for dataset standardization (has to be run once before model training to store normalization coefficients). * Bugfix in the dataset class. When iterating over movie configurations, the wrong configuration was used to find the correct signal name. Also, removed warning for duplicated tensor conversion. * Added base script for video reconstruction. Copied from Aza's branch for debugging purposes. * Added base script for video reconstruction. Copied from Aza's branch for debugging purposes. * Minor changes in the example scripts. More preprocessing options for the dataset class. * Fixed a bug where the dataset class failed when using multiple workers and opening an H5 file prior to distributing the dataset across all workers. Significant updates in the Fast time series baseline and actuator reconstruction classes. * Lots of bugfixes in the dataset, trainer, and models. The basic encoders are now all working. Examples are in scripts. * Extended checkpointing - the trainer stores now: - Model - Optimizer state - Scheduler state - Current loss - Current epoch For the sake of continual training. * Extended checkpointing - the trainer stores now: - Model - Optimizer state - Scheduler state - Current loss - Current epoch For the sake of continual training. * Adapted the other reconstruction scripts to match the new API. * Bugfix in the dataset class. When splitting inputs and targets, I forgot to remove unused modalities. This follows the standard getitem function now. * Prepared an option to preprocess movies. This has to be fully integrated!!! * Added a baseline fusion transformer for latent space prediction. Quick fix for the data standardization. Invalid values have to be ignored. Fix in the function to create H5 files. bolo data does not have to be flipped anymore as the data is now stored in the correct format. --------- Co-authored-by: Nathaniel Chen <nathanchen1101@gmail.com>
…eparation to moving to Stellar.
…5 files. Implemented calculating incremental statistics. Corrected values in the modality configuration. Removed redundant script standardize_dataset.py
TODO: Write a documentation.
…simple file transfer.
- Added information on how to use all the scripts for data fetching. Updated read_mds.sh - Added a switch for globus file transfer. This simply stores the H5 files on Omega and we can add more data later.
Moved prepare_data.py to scripts, added a batch script to do this on compute nodes. Added more point names to the data fetching scripts for Omega. Added docstring to the WelfordTensor class. Updated modalities.yaml with the new point names added.
…_preprocessing_stats more transparent. Bugfix in modalities.yaml - Channels were missing in ECE.
…tats. This is still not efficient enough and causes memory issues.
Bugfixes in the trainer. Cosmetic changes in tracking.py
- PEP-8 corrections - Support plots of time signals and videos Train-val-test split in fast_time_series_reconstruction.py
- Channels was not handled properly (if selecting slices of a signal). - Drawing: Restrict plotting to valid signals (not the padded sections after the actual signal). - Introduced masked loss for fast time series reconstruction.
…_series_baseline.py to filterscope_baseline.py). Updates in the dataset class: Clipping for log transform can go down to -.99 (sufficient because we subtract 1.0). Updates in drawing.py: We can now draw all kinds of different plots (except for profiles for now). Added functionality to draw correlation plots, which is important for finding feature distributions. Added masked loss functions to not consider out-of-range time slices for training.
added 30 commits
August 18, 2026 04:07
…fp16-safe confidence floor Review findings on the R1 global pool, all opt-in-path or docs only: - Scope the vocab-normalization rationale in norm_log_confidence and _global_reveal: the log-vocab term is load-bearing only for MIXED-vocab layouts (the cache-derived production set, 64k spectro beside 1k slow-TS). Under a uniform vocab -- the static FROZEN_MODALITIES table and the bp pilot line -- log V is a common divisor, so it is a shared monotone transform and the pool degenerates to raw-probability top-K. Still cross-modal and confidence-driven there; just nothing for the normalization to fix. - stable=True on the frame-wide argsort in _global_reveal, so exact score ties do not resolve by sort-backend luck and pooled reveal order is device-reproducible. - norm_log_confidence computes in float32: 1e-12 underflows to 0 in float16, so the clamp could not prevent log() from returning -inf. Scores only feed argsort. - test_global_pool_defers_low_confidence_modality: docstring now states what it really checks (a validity smoke test; deferral is covered by the direct _global_reveal tests), and its redundant function-local import torch is gone.
…f/rollout_context hardenings
…lp renders argparse %-formats help strings, so '~38% of loss' was read as a '%o' conversion and build_arg_parser().format_help() raised TypeError. Pre-existing (reproduced at HEAD~1), but it made the new rollout-quality flags undiscoverable via --help.
…rollout x checkpointing poisoned it)
Token-accuracy skill was MEASURED blind to the pathology this harness exists to
gate: on bp128_d512L8 / shot 199597, steps 10500 -> 20000, token skill is flat
(mhr +0.249 -> +0.259) while the decoded mhr TM 1-20 kHz band collapses. Spec
section 5 mandates BOTH metrics; only the token half was implemented.
Adds the decoded band-restricted nRMSE skill (1 - nrmse/nrmse_persistence) as the
PRIMARY metric, reported per (modality, band) and never averaged across bands --
over those same two steps mhr AE 50-250 kHz goes the other way (+0.175 -> +0.767),
so a band-aggregated number cancels the failure out. Also emits the skill over the
k in {10, 40, 80} horizon prefixes, flagging any k truncated by the end of the
diagnostic's digitiser record.
The band-power inverse (quantile dequantization via the trained bin_edges_128.npz,
equal-width band reduction) and the data-valid-frame derivation are LIFTED from
scripts/evaluation/ignite_bp_cases.py, and decoded_space_skill from
ignite_case_panels.py -- copied, not imported, because those are untracked user
scripts and a committed harness importing them would be broken for everyone else.
Semantics verified identical: --from_traces recomputes the archived 2026-08-15
rollouts through this code path and reproduces mhr TM +0.152962 @11k /
-0.862308 @20k, delta 0.00e+00 against the traces the origin script stored.
Decoded metrics are guarded off unless every modality is a vocab-8 band-power
modality whose trained edges match its token count, so the quantile edges can
never be silently applied to a 64k-vocab FSQ modality.
Default --cache_dir moves to a durable proj-shared copy of the extended bp128
cache: 199597 (the shot the documented inversion was measured on) exists in no
canonical cache, and the only copy was in purge-by-atime scratch.
…rappers Commit 2f152c2 renamed scripts/slurm_frontier/_frontier_common.sh to _frontier_settings.sh without updating its callers, so line 25 of both dynamics training wrappers sources a file that does not exist. Under `set -euo pipefail` that kills the job immediately -- before any module load, before srun -- so the Phase-1 CTF training launches cannot start at all. Scoped deliberately to the two wrappers this plan executes (train_dynamics.sh and train_dynamics_ctf.sh). 54 other wrappers under scripts/slurm_frontier/ carry the same stale reference and are left alone rather than swept up in a plan commit.
summarize() zipped the full --shots list against only the records that survived the insufficient-valid-frames skip, so a skipped shot shifted every later shot's decoded per_shot/k_valid labels onto the wrong id. Stamp the shot into each record at append time (both the live-rollout and --from_traces paths) and key the maps from the record.
…t off) The trainer only ever wrote rolling dynamics_latest.pt + best-on-val dynamics_best.pt, so a skill-vs-step curve could never have more than two points per arm. Snapshots fire on ckpt steps (use a multiple of --ckpt_every); default 0 keeps behaviour identical. The ctf wrapper defaults SNAPSHOT_EVERY=2000 — the Phase-1 arms are measured by their curves, not their endpoints. Golden compat: 3 passed.
…verify infinite loop
Controller ruling: with a bias, a zeros-text input (what the dataset feeds for undocumented shots) produced t=bias, differing from the trained null (t=0 exactly, via drop_text/dropout_p=1.0). bias=False makes all three null paths coincide exactly at t=0. Strengthened test_phaseb_text_embed.py to assert the exact equalities this now guarantees.
…gated, default off) FrameCodeDataset grows a 4th tuple element (per-shot text embedding, zero-width when unused); train() gains --text_embed_path/--text_embed_dim/--text_dropout_p/--text_key and writes the corresponding cfg_text_* keys into every checkpoint payload; eval_dynamics reconstructs text_embed_dim/text_dropout_p from a checkpoint and feeds the shot's embedding into rollout_shot only when the loaded model actually needs it. With no text flags, dataset/collate/loop/checkpoint behaviour is unchanged and no model call receives a text kwarg.
FrameCodeDataset.__getitem__/_collate_frames return (codes, act, presence, text) since this task's text-embedding wiring landed (presence already made it a 3-tuple earlier); test_phaseb_dataset.py still unpacked the old 2-tuple shape. Also fixes two unrelated stale mocks the frontier env's fuller collection surfaced: the _single_shot_dataset / actuator_frames monkeypatches didn't accept t0_start, and the placeholder-code dtype assertion still expected int16 when precompute has written int32 placeholders since 3b26f39.
…processed.h5
Additive text_embed group (input/total f2 + provenance attrs) copied from the
consolidated text_embeddings.h5 into each shot's {shot}_processed.h5, so the shot
file carries everything about that shot. Consolidated H5 remains the canonical
source IGNITE training/eval reads directly; this is co-location only.
…e preservation Guard against a process dying mid-injection: write attrs["complete"]=True as the final attribute so a partial text_embed group is detected and repaired (deleted and re-injected) on the next run instead of being trusted as skipped_existing forever. --verify now fails a sampled group missing the sentinel. Also stop forcing input/total to float16 and copy the source dataset's dtype verbatim.
…egression test train()'s (text_embed_path is None) != (text_embed_dim > 0) was backwards in all four cases: a plain no-flag run (the production default) died at startup with the TOGETHER SystemExit, the valid both-given case also died, and either flag given alone silently passed instead of failing. Fixed to (text_embed_path is not None) != (text_embed_dim > 0). Added a 4-case truth-table regression test that calls the real train() (not a reimplementation of the boolean) on a tiny FROZEN_MODALITIES-compatible cache: path-only and dim-only must raise SystemExit mentioning TOGETHER; neither and both (with a real tiny embeddings H5) must not raise that guard and must run train() to completion for one optimizer step -- this is also the first test that exercises train() end-to-end.
…her half-flag guard, provenance minors Whole-branch review follow-ups for per-shot text embeddings (all in one wave): - eval_dynamics.py: --text_key now defaults to the checkpoint's stamped value (falls back to "input" for old ckpts); an explicit CLI value that differs from the stamp prints a loud warning but is honored; a differing --text_embed_path gets a one-line info note. - text_embed.split_bundle: optional `counters` dict tallies which anchor-fallback branch fired (missing_general/summaries/planned/shot_anchor), no behavior change. - embed_shot_text.py: passes one shared counters dict through run_embed, logs non-zero counts, stamps them as shard attrs (anchor_*), and --merge sums them into root attrs; also stamps model_id/max_length provenance per shard and asserts cross-shard consistency at merge, fixes the truncated-flag off-by-one (max_length - 1, the real EOS-reserved cutoff), stamps embed_dim from the merged array's width instead of the module constant, clarifies the resume-duplicate-shot error message, and drops the unused --batch_size flag. - train_dynamics.sh / train_dynamics_ctf.sh: reject TEXT_EMBED_PATH/TEXT_EMBED_DIM set separately (was silently launching an unconditioned run or passing an empty path into the trainer's TOGETHER guard). - train_dynamics.py FrameCodeDataset: an empty (non-None) text_embeds dict now raises a clear SystemExit instead of a bare StopIteration. - Stale test-file header comment fixed; new tests cover the counters, the empty-dict guard, and the anchor-fallback branches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
flashattention on frontier