change all codes to avx2 as now all machines at ITP have this compile option

[GeorgGrassler]

[krystophny]

This should be -march=core-avx2 -mtune=generic.

[krystophny]

Automated triage pass (Claude), grounded in the repository at 07d67ba. Verify before acting.

Still relevant. The April 2025 flag sweep that followed this issue landed on -march=x86-64-v2, which is SSE4.2/POPCNT — it contains no AVX at all, let alone AVX2. So the request in the title was never actually implemented; what happened is that -march=sandybridge was replaced by an equivalent-generation vendor-neutral name. Below is the spec to finish it.

Purpose

Every Release build of the group's Fortran codes currently targets a 2008-era instruction set. The hot kernels here — spline evaluation in libneo, the symplectic guiding-centre integrators in SIMPLE, the resonance integrals in NEO-RT — are dense floating-point loops that gfortran will happily vectorise, but at x86-64-v2 it can emit at most 128-bit SSE2 packed doubles and no fused multiply-add. Moving the baseline to AVX2 doubles the vector width to 256 bits and gives the compiler FMA, which is exactly what -ftree-vectorize and -funroll-loops (already on, libneo/CMakeLists.txt:159-162) need to pay off. The counter-pressure is portability: the whole point of a fixed baseline rather than -march=native is that one build must run on every ITP workstation, on VSC5 and on Viper. This issue asserts that assumption now holds for AVX2; the job is to raise the baseline and add a build-time guard so the assumption is enforced rather than believed.

Current state

The baseline is duplicated verbatim across seven repositories, with three distinct spellings — that duplication is half the problem.

libneo/CMakeLists.txt:168-180 (GNU branch):

        # Architecture-specific flags
        if(CMAKE_BUILD_TYPE MATCHES Fast OR CMAKE_HOST_SYSTEM_PROCESSOR MATCHES "aarch64|arm64|ARM64")
            # Fast mode or ARM: use native architecture
            # gfortran on ARM needs -mcpu=native, not -march=native
            if(CMAKE_HOST_SYSTEM_PROCESSOR MATCHES "aarch64|arm64|ARM64")
                add_compile_options(-mcpu=native)
            else()
                add_compile_options(-march=native)
            endif()
        else()
            # Release on x86-64: use v2 baseline for broad compatibility
            add_compile_options(-march=x86-64-v2 -mtune=generic)
        endif()

Note that the Intel branch of the same file is already what this issue asks for — libneo/CMakeLists.txt:209-211 uses -xHost for Fast and -xCORE-AVX2 for Release. The GNU path is simply out of step with the Intel path in the same CMakeLists.txt.

The other occurrences:

Repo / file	Line	Current flags
libneo/CMakeLists.txt	179	-march=x86-64-v2 -mtune=generic
NEO-2/CMakeLists.txt	29-31	-march=native (APPLE) / -march=x86-64-v2 -mtune=generic
NEO-RT/CMakeLists.txt	29-31	-march=x86-64-v2 -mtune=generic (Release); 37-39 -march=native -mtune=native (Fast)
SIMPLE/CMakeLists.txt	366-370	-march=x86-64-v2 -mtune=native ← mtune=native, inconsistent with everyone else
KAMEL/KiLCA/CMakeLists.txt	12-13	-O3 -march=x86-64-v2 -mtune=generic (CXX and Fortran)
KAMEL/CMakeLists.txt	85	-mtune=generic with no -march at all → implicit x86-64 (SSE2)
MEPHIT/CMakeLists.txt	24, 27	-march=native unconditionally — not redistributable
KiLCA-FLR2/CMakeLists.txt	13	-march=native
NEO-RT/POTATO/CMakeLists.txt	13	-march=native -mtune=native
code/scripts/setup/stellopt/make_scluster.inc	20	-O2 -fexternal-blas -fbacktrace -march=native
code/scripts/setup/stellopt/make_vsc5.inc	20	same
code/scripts/setup/kamel.sh	10, 33	gfortran -c -Wall -Wtabs -mtune=generic $CFLAGS — no -march

History confirming the request was answered only halfway: NEO-RT@9f7d8a1 (2025-04-24, "Fix CPU flag") changed -march=sandybridge -mtune=generic → -march=x86-64-v2 -mtune=generic; KAMEL@e5ebb2fb (2025-04-28, "Build flags"); NEO-2@3a2f550 (2025-04-23); libneo@040a163 (2025-08-20); SIMPLE@0016bff (2025-08-22). None of these reach AVX2.

On the naming in @krystophny's comment: -march=core-avx2 is accepted by gcc (Haswell) but it is an Intel-family name and it also implies -mtune=core-avx2, which is why the comment pairs it with -mtune=generic. Since VSC5 and Viper are AMD, prefer the vendor-neutral -march=x86-64-v3 (gcc ≥ 11; VSC5 loads gcc 12.2.0 per code/scripts/setup/vsc5.sh:20). x86-64-v3 = AVX, AVX2, FMA, BMI1/2, MOVBE, F16C — the same capability set, without AMD machines being tuned as Haswell.

Reproduction / evidence

Show that the current Release build emits no AVX2, then that the proposed flags do:

cd $CODE/libneo
cmake -B build-v2 -DCMAKE_BUILD_TYPE=Release
cmake --build build-v2 -j
objdump -d build-v2/libneo.a | grep -cE 'vfmadd|ymm'      # expected from the current v2 flags: 0; run to record

cmake -B build-v3 -DCMAKE_BUILD_TYPE=Release \
      -DLIBNEO_ARCH_BASELINE=x86-64-v3
cmake --build build-v3 -j
objdump -d build-v3/libneo.a | grep -cE 'vfmadd|ymm'      # expected: > 0

Capability check on the target machines (run on an ITP workstation, a VSC5 login node, a VSC5 compute node via srun, and Viper — the compute nodes are what matters, login nodes can differ):

grep -o -m1 -E 'avx2|fma|bmi2' /proc/cpuinfo | sort -u
# expected on every ITP/VSC5/Viper node: avx2, bmi2, fma

Timing evidence for the change being worth it — SIMPLE has a ready-made workload:

cd $CODE/SIMPLE
cmake -B build-v2 -DCMAKE_BUILD_TYPE=Release && cmake --build build-v2 -j
cmake -B build-v3 -DCMAKE_BUILD_TYPE=Release -DSIMPLE_ARCH_BASELINE=x86-64-v3 && cmake --build build-v3 -j
# run the same golden-record case under test/golden_record/ with both binaries, compare wall time

Record the numbers in the PR. If AVX2 buys less than a few percent on these kernels, that is itself a result worth writing down before churning every repo.

Behaviour (BDD)

Scenario: Release build targets AVX2 on x86-64
  Given libneo configured with CMAKE_BUILD_TYPE=Release on an x86-64 host
  When cmake -B build && cmake --build build runs
  Then the compile line for src/interpolate.f90 in build/compile_commands.json
       contains "-march=x86-64-v3" and "-mtune=generic"
   And "objdump -d build/libneo.a | grep -c vfmadd" is greater than zero

Scenario: Baseline is overridable for older hardware
  Given libneo configured with -DLIBNEO_ARCH_BASELINE=x86-64-v2
  When the build runs
  Then the compile line contains "-march=x86-64-v2" and not "x86-64-v3"

Scenario: Unsupported baseline fails at configure time, not at runtime
  Given a compiler that rejects -march=x86-64-v3 (e.g. gcc 9)
  When cmake -B build -DCMAKE_BUILD_TYPE=Release runs
  Then configure fails with a message naming LIBNEO_ARCH_BASELINE
   And no object files are produced

Scenario: ARM and Fast builds are unaffected
  Given CMAKE_HOST_SYSTEM_PROCESSOR is arm64, or CMAKE_BUILD_TYPE=Fast on x86-64
  When the build runs
  Then the compile line contains "-mcpu=native" or "-march=native" respectively
   And it contains no "-march=x86-64-v3"

Scenario: Golden records survive the flag change
  Given NEO-RT built Release with -march=x86-64-v3
  When pytest test/golden_record/test_golden_record.py runs
  Then every field matches test/golden_record/golden.h5 within RTOL = 5e-3
       of that field's own peak magnitude

Implementation plan

Do libneo first and merge it before the consumers; the rest are mechanical.

1. libneo/cmake/Modules/ArchBaseline.cmake (new). Define libneo_set_arch_baseline(<PREFIX>): reads cache variable ${PREFIX}_ARCH_BASELINE (default x86-64-v3), skips entirely on aarch64|arm64|ARM64 and on CMAKE_BUILD_TYPE MATCHES Fast, and otherwise uses include(CheckCCompilerFlag) / check_fortran_compiler_flag to verify -march=${baseline} before add_compile_options(-march=${baseline} -mtune=generic). message(FATAL_ERROR ...) naming the cache variable if the check fails. Emit message(STATUS "Architecture baseline: ${baseline}") so it is visible in build logs.
2. libneo/CMakeLists.txt — add option(LIBNEO_ARCH_BASELINE ...) near the other options (next to LIBNEO_DETERMINISTIC_FP, line 50), include(ArchBaseline) after line 58 (set(CMAKE_MODULE_PATH ...)), and replace lines 168-180 with a call to libneo_set_arch_baseline(LIBNEO). Leave the Intel branch (line 209-211) alone; -xCORE-AVX2 is already correct.
3. libneo/test/ — add test_arch_flags.cmake or a small CTest that greps ${CMAKE_BINARY_DIR}/compile_commands.json for -march=; register it with add_test. CMAKE_EXPORT_COMPILE_COMMANDS is only forced on under LIBNEO_DETERMINISTIC_FP (libneo/CMakeLists.txt:53-55), so set it unconditionally or have the test configure a throwaway build dir.
4. SIMPLE/CMakeLists.txt:366-370 — replace with the same logic, cache variable SIMPLE_ARCH_BASELINE. Fix the -mtune=native → -mtune=generic inconsistency at line 370 while you are there; mtune=native in a build that is meant to be portable is a latent surprise.
5. NEO-RT/CMakeLists.txt:29-31 — same, NEORT_ARCH_BASELINE. Do not touch the Fast branch at 37-39. NEO-RT/POTATO/CMakeLists.txt:13 is a separate legacy subproject; either leave it on -march=native or bring it in line — say which in the PR description.
6. NEO-2/CMakeLists.txt:29-31 — same, NEO2_ARCH_BASELINE.
7. KAMEL/CMakeLists.txt:85 — add an explicit -march=x86-64-v3; today the missing -march means plain SSE2. KAMEL/KiLCA/CMakeLists.txt:12-13 — bump both CMAKE_CXX_OPT_FLAGS and CMAKE_F_OPT_FLAGS from x86-64-v2 to x86-64-v3.
8. MEPHIT/CMakeLists.txt:24,27 — -march=native here means MEPHIT builds are not portable across the group at all. Replace with the baseline; this is the one repo where the change is a behaviour fix and not just a bump, so benchmark it separately.
9. code/scripts/setup/kamel.sh:10,33 — add -march=x86-64-v3 to the two gfortran -c lines (they currently have -mtune=generic and no -march).
10. code/scripts/setup/stellopt/make_scluster.inc:20 and make_vsc5.inc:20 — replace -march=native in FLAGS_R with -march=x86-64-v3 -mtune=generic, so the STELLOPT build on the shared cluster filesystem is usable from any node.
11. code/README.md (or docs/development-model.md on chore/relocate-infra) — one paragraph stating the group-wide baseline, the <PROJECT>_ARCH_BASELINE escape hatch, and the rule that Release must be redistributable while Fast may use -march=native.

Since libneo is the hub and the others consume it, the ordering is: libneo PR → consumers. If the CMake helper cannot be shared cleanly across repos before libneo is released, copy the ~15 lines into each repo and open a follow-up to deduplicate; do not block the whole thing on the sharing mechanism.

Acceptance criteria

• objdump -d build/libneo.a | grep -c vfmadd returns a nonzero count for a default -DCMAKE_BUILD_TYPE=Release build on x86-64; the same command returns 0 on 07d67ba-era builds.
• cmake -B build -DCMAKE_BUILD_TYPE=Release -DLIBNEO_ARCH_BASELINE=x86-64-v2 still configures and builds.
• Every repo listed above has exactly one -march= for the x86-64 Release path, and rg -n 'march=x86-64-v2' /mnt/storage/triage/repos returns nothing outside vendored third-party trees.
• pytest test/golden_record/test_golden_record.py in NEO-RT passes at the existing RTOL = 5e-3 (NEO-RT/test/golden_record/test_golden_record.py:37) with no widening of the tolerance. If it fails, that is a finding to report, not a tolerance to relax.
• SIMPLE's test/golden_record/ suite passes unchanged.
• SIMPLE_DETERMINISTIC_FP=ON / LIBNEO_DETERMINISTIC_FP=ON builds still produce the deterministic results they produce today — check that -ffp-contract=off (libneo/CMakeLists.txt:134) is still applied after the arch flags, so enabling FMA hardware does not silently reintroduce contraction.
• Verified avx2 fma bmi2 present in /proc/cpuinfo on: an ITP workstation, a VSC5 compute node, a Viper compute node, and the GitHub Actions ubuntu-24.04 runner used by code/.github/workflows/main.yml:23. Paste the output in the PR.
• Benchmark table in the PR: wall time for one SIMPLE golden-record case at v2 vs v3, same machine, same input.

References

• Intel, Intel 64 and IA-32 Architectures Optimization Reference Manual — AVX2 256-bit packed-double throughput and FMA latency, the mechanism by which this change is supposed to help.
• The x86-64 psABI microarchitecture levels (x86-64-v1…v4) were standardised jointly by Red Hat, SUSE, Intel and AMD in 2020 and are documented in the GCC manual under -march (GCC 11 release notes); v3 is defined as AVX2 + FMA + BMI1/2 + MOVBE + F16C.
• On the numerical consequence: enabling FMA changes results because a*b+c is computed with a single rounding. Muller et al., Handbook of Floating-Point Arithmetic, 2nd ed. (Birkhäuser, 2018), ch. 5, covers why this is normally an accuracy improvement but never a bit-identical one — which is precisely what the scale-relative golden-record tolerance in NEO-RT/test/golden_record/test_golden_record.py:27-37 already anticipates.
• Comparable practice in sibling codes: ASCOT5's build uses an explicitly selectable target architecture rather than -march=native for the same reason (one binary shared across a cluster). Worth a look before settling the Fast vs Release split.

Scope and risk

Single PR per repository, small diffs, but seven repositories with libneo strictly first — treat it as one tracked change with libneo as the blocking PR. Two real risks: (1) any machine in the group without AVX2 gets SIGILL at runtime instead of a build error, which is why step 1 adds a configure-time compiler check and the acceptance criteria demand /proc/cpuinfo evidence from compute nodes and not just login nodes; (2) FMA contraction shifts every floating-point result at the 1e-15 relative level per operation, which can accumulate through the guiding-centre integrators — if a golden record fails, investigate rather than widen RTOL.