diff --git a/CLAUDE.md b/CLAUDE.md index 5f71c0e..0a29e22 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -343,6 +343,9 @@ Adding a filter touches many files. Missing any step causes silent failures (fil platform that ships it. The `package-deps-*` scripts assert this list is present before zipping and fail the build if any are missing, so a dead download URL becomes a red build instead of a silently-incomplete bundle. + A bare filename is looked up in the plugin directory; an entry containing a + `/` is resolved from the **bundle root** instead, which is how zsmooth's + per-CPU builds outside the autoload directory stay covered. - For local testing, run the download script to populate `deps/` (e.g. `deps/windows-x64/vapoursynth/vs-plugins/`, `deps/macos-arm64/vapoursynth/plugins/`). - **Credit it**: add an entry to `licenses/NOTICES.txt` and a `_ComponentTile` @@ -868,6 +871,166 @@ which any documentation would have shown: > is silently lost at 16-bit. Good news: bm3d is never reached, so it needs no > deps addition. Implementable, but effort 3 and blocked on the licence. +### FFmpeg is pinned to one series, and the pin is asserted (2026-08-31) + +The bundled FFmpeg is what actually interprets everything in +`pipeline_executor.rs` — the accumulated `-vf` chain, the colour metadata +flags, `-ss`/`-frames:v` trimming, the hardware-encoder options. So a version +skew between platforms means the same job encodes differently depending on +where it ran, silently. It is the same class of hazard as the fmtconv and +zsmooth version pins, and it had been true for months without anyone noticing: + +| | FFmpeg, measured 2026-08-31 | how | +|---|---|---| +| Windows | **master N-125978** (post-9.0) | BtbN `master-latest`, **unpinned** | +| macOS x64 | 9.0.1 | evermeet `getrelease`, **unpinned** | +| macOS arm64 | 9.0.1 | martin-riedl `latest`, **unpinned** | +| Linux | **7.1**, two majors behind | BtbN `n7.1`, pinned | + +Three floated on "latest" and the fourth was pinned to a series BtbN then +garbage-collected, which 404'd the Linux deps build outright. All four now pin +**9.0**. + +> **A version pin against a rolling tag is not a pin.** BtbN publish every +> series to one `latest` tag and drop old ones as they age, so `n7.1` was always +> going to become a 404 — it was a matter of when. Pin the **series** +> (`n9.0-latest`, newest build of the 9.0 branch), which is how all three +> upstreams actually publish, and treat a series bump as a deliberate, +> all-platforms-together change. + +> **`curl` without `-f` writes the 404 body to the output file.** That is why +> the failure surfaced as `tar: Error is not recoverable` half a step later +> rather than as a download error naming the URL — the "tarball" was nine bytes +> reading `Not Found`. Every FFmpeg fetch now uses `-f`. + +Pinning differs per host because their retention does, and that is deliberate: + +- **BtbN** (Windows, Linux) — series URL, rolls patches within 9.0. +- **evermeet** (macOS x64) — an exact version, `FFMPEG_MACOS_X64_VERSION`. It + keeps old versions reachable (verified: 7.1, 8.0 and 9.0.1 all still resolve), + so a full-version pin is durable here. +- **martin-riedl** (macOS arm64) — only `latest` plus opaque build-id paths of + unknown retention, so it takes latest and is **checked afterwards**. + +Two guards, and both are needed. `assert_ffmpeg_series` in each script runs +`ffmpeg -version` on what was actually installed and fails the build if it is +not the pinned series — that catches an upstream silently moving a URL, which +no amount of pinning can prevent. And `app/test/ffmpeg_version_pin_test.dart` +(push gate) reads all three scripts and fails if their pins disagree, if a pin +is not a bare `major.minor`, or if a script stops asserting. Without the second, +the pins are just comments. + +Note the version parser accepts `n9.0.1` and `9.0.1` and deliberately **rejects +a `master` build** (`N-125978-...`), so reverting any platform to an unpinned +master URL is a red build rather than a silent regression. + +### zsmooth ships once per CPU baseline, and is loaded by path (issue #82, 2026-08-28) + +A plugin can also have **no** dispatch at all. zsmooth is compiled for a whole +CPU baseline — upstream publishes only `haswell` (AVX2) and `znver4` for x86, +per the targeting essay in vapoursynth#1185 — so on a pre-2013 CPU the library +**loads fine** and then dies the instant a filter runs. + +Reported on a Celeron J4105 and a Core i7 870, neither of which has AVX at all. +The symptom is a bare `vspipe exited with exit code -1073741795`, which is +`0xC000001D` **STATUS_ILLEGAL_INSTRUCTION** — a different fault from CTMF's +`0xC0000005`, and worth knowing apart: illegal instruction means the binary +needs a CPU feature this machine lacks, access violation means a genuine bug in +the kernel that ran. Both print nothing else, so the encode surfaces as ffmpeg +reading an empty pipe. `format_exit_status` now decodes both. + +Everything reaching `core.zsmooth.*` was affected: CCD, Cnr4, SpotLess → +RemoveDirt, Noise Reduction → mClean and TemporalDegrain2, and hybrid_mv. QTGMC +was not — havsfunc uses `rgvs`. + +> **Shipping one portable build for everyone is the obvious fix and the wrong +> one.** Measured (Zig 0.15.2, same source, 720x576, fps, best of 3), as a +> fraction of haswell speed: +> +> | | CCD r0 | CCD r1 | Cnr4 | CCD 16-bit | RemoveGrain | Repair | Median | +> |---|---|---|---|---|---|---|---| +> | `x86_64` | 0.50 | 0.33 | 0.33 | 0.39 | 0.68 | 0.66 | 0.73 | +> | `x86_64_v2` | 0.72 | 0.63 | 0.71 | 0.69 | 0.69 | 0.65 | 0.73 | +> +> That is 2-3x on the two filters the Chroma Denoise pass is *made of*, charged +> to every modern machine to serve the rare old one. Ratios hold at 1080p and +> multithreaded. A locally built `-Dcpu=haswell` matched the shipped binary +> within ±3%, so these are like-for-like and not a toolchain artefact. + +So x86 bundles ship **both** builds and the worker picks at load time. Three +things about the mechanism: + +- **The builds cannot share a directory.** Each registers the namespace + `zsmooth`, so whichever autoloads second is rejected — and on macOS/Linux + `vapoursynth/plugins` is autoloaded implicitly by R78, so "just don't set the + env var" is not available either. They live in `vapoursynth/zsmooth/`, outside + any autoload path, and the generated script carries an explicit + `core.std.LoadPlugin`. `VAPOURSYNTH_EXTRA_PLUGIN_PATH` takes **one** directory + — verified, a `;`-separated pair silently loads only the first — so the second + plugin directory idea does not work. +- **The namespace stays `zsmooth`, so no call site changed.** That is the whole + reason for loading by path rather than under a `forcens` alias: the vendored + `mclean.py`, `removedirt.py`, `temporaldegrain2.py` and `hybrid_mv.py` all say + `core.zsmooth.X`, and an alias would have needed a `_zs()` indirection through + every one of them (and left them on the slow build for AVX2 users). +- **No selected build means no `LoadPlugin`.** Bundles up to 1.9.0 autoload a + single zsmooth, and the app can be upgraded before the deps download finishes, + so a worker that always emitted the line would fail every job in that window. + `DependencyLocator::zsmooth_plugin()` returns `None` there and the script is + byte-identical to the pre-split one. Verified both ways end to end. + +`x86_64_v2` (SSE4.2/POPCNT, Nehalem 2009 on) is the fallback rather than plain +`x86_64`: it is 1.4-2.1x faster on the filters that matter, 0.6 MB smaller, and +covers both CPUs in the report. Note v2 buys **nothing** over v1 on the +RemoveGrain/Repair/Median kernels — the gain is specific to CCD and Cnr4. + +macOS x64 was already affected in the other direction: it builds from source for +the issue #39 minos floor and had always used Zig's *default* baseline, i.e. the +0.50/0.33 column. It now builds both, so Intel Macs get the fast path for the +first time. + +> **The macOS haswell build needs an fftw patch, and the bug is one line +> upstream.** zsmooth's Zig fftw port sets `HAVE_MEMALIGN` on every non-Windows +> target, but macOS has no `memalign()` — it is declared in ``, which +> **the same file already knows macOS lacks** (`HAVE_MALLOC_H` is gated on +> `!is_mac`). fftw's `kalloc.c` only reaches that branch when `MIN_ALIGNMENT` is +> 32, i.e. when AVX is on, so it is invisible at every SSE-level baseline and +> kills **only** the haswell build — with a clang implicit-declaration error +> inside a dependency, which reads like a toolchain problem rather than a +> one-line config bug. `HAVE_POSIX_MEMALIGN` is already true, so clearing it +> falls through to `posix_memalign`. +> +> The patch clones the fftw fork at the ref `build.zig.zon` names, edits that +> line, and repoints the dependency as a **path** dependency — path deps take no +> hash, so this is deterministic and survives a cache wipe, unlike editing Zig's +> global package cache. Both a pre-check and a post-check hard-fail, so an +> upstream fix surfaces as a build error telling you to remove the patch rather +> than silently doing nothing. +> +> Verified by **cross-compiling from Windows** (`-Dtarget=x86_64-macos.12.0 +> -Dcpu=haswell`), which reproduces the failure exactly and confirms the fix in +> about four minutes — far cheaper than a macOS CI round trip, and worth +> remembering for any Zig-built plugin: the target does not have to be the host. + +> **The fallback is not a different picture, only a slower one.** The two builds +> produced identical chroma means (U=123.884, V=131.096) on the same clip, so +> falling back costs throughput and nothing else. Do not treat the choice as +> output-affecting. + +Guards: `test_154`/`test_155` (both scripts, both bundle layouts) and +`zsmooth_never_offers_a_build_this_cpu_cannot_run` in `dependency_locator.rs` — +which is the durable one, since it runs on every platform whatever hardware CI +draws. The Dart side loads the chosen build and **renders a frame** (the fault is +in the kernel, so constructing the node proves nothing), asking the worker's +`--probe-cpu` for the CPU rather than deriving it. `deps-expected-plugins.json` +entries for zsmooth are bundle-relative **paths**, and all three packaging guards +understand that form now. + +**This is not verifiable in CI.** Every hosted runner has AVX2, so no CI job can +exercise the fallback; the local check is to hide the haswell build and re-run. +Intel SDE (`sde -nhm --`) is the only way to prove a build runs on a CPU you do +not have. + ### A plugin's own CPU auto-detect is not trustworthy (CTMF, 2026-08-25) `ctmf.CTMF`'s AVX-512 kernel for **8-bit** input @@ -953,10 +1116,14 @@ What an effort-2 addition actually costs, beyond the usual filter wiring: 5. **A deps release actually built and published**, which is CI work and cannot be done or verified locally — see the rc flow in "Testing a deps change". -> **Windows has no from-source build path, and that decides the version.** -> `download-deps-windows.ps1` only fetches published release archives, so a -> plugin is only addable if upstream ships a Windows binary — and every platform -> must then pin the version Windows can get. FillBorders and Bwdif were the first +> **Windows has essentially no from-source build path, and that decides the +> version.** `download-deps-windows.ps1` fetches published release archives, so +> a plugin is only addable if upstream ships a Windows binary — and every +> platform must then pin the version Windows can get. (The single exception, +> added 2026-08-28: zsmooth's portable build compiles there with a pinned Zig +> toolchain, because Zig brings its own libc and needs no MSVC. Do not read that +> as a general from-source path — it exists because upstream ships no binary that +> runs without AVX2, see the zsmooth section above.) FillBorders and Bwdif were the first > two candidates and were **rejected on this basis**: their newest Windows > binaries are several releases behind their source (FillBorders v2 vs v4, Bwdif > r4.1 vs r5.1), and pinning everything back that far would have cost features @@ -2034,7 +2201,9 @@ Depth Sources". > `vapoursynth_integration_test`'s "all required plugins load" list is the > runtime contract for a **complete deps install**, and it must name every -> namespace a filter can reach. `zsmooth` (Chroma Denoise / CCD) was added to the +> namespace a filter can reach — except `zsmooth`, which is deliberately not +> autoloaded any more and has its own load-and-render test beside that list. +> `zsmooth` (Chroma Denoise / CCD) was added to the > bundle after that list was written and went uncovered for a while, so a bundle > missing it passed the suite while the filter failed at job time with "No > attribute with the name zsmooth exists". Add the namespace whenever you add a @@ -2660,8 +2829,11 @@ version skew between platforms would change chroma per-OS. generating the script (`--config`, keep the `.vpy`) and running it under vspipe with passes commented out; on Windows confirm with `$LASTEXITCODE` (`0xC0000005` = access violation, `0xC000001D` = illegal instruction, which - means the binary needs a CPU feature this machine lacks). See "A plugin's own - CPU auto-detect is not trustworthy" for the CTMF case. + means the binary needs a CPU feature this machine lacks). Both codes are now + decoded into the reported error by `format_exit_status`, so a fresh report + should say which it was rather than just a negative number. See "A plugin's + own CPU auto-detect is not trustworthy" for the CTMF case (`0xC0000005`) and + "zsmooth ships once per CPU baseline" for the AVX2 one (`0xC000001D`). ## Platform-Specific Notes @@ -2713,7 +2885,7 @@ version skew between platforms would change chroma per-OS. - Fully self-contained deps (no Homebrew at runtime): Python 3.12 (python-build-standalone), VS built from source - Worker sets: `PYTHONHOME`, `PYTHONPATH`, `VAPOURSYNTH_CONF_PATH`, `DYLD_LIBRARY_PATH` - `vspipe` is a wrapper script that generates config dynamically (needed because `VAPOURSYNTH_PLUGIN_PATH` is additive, not a replacement) -- **FFmpeg** is sourced pre-built as a static binary that links only system frameworks: **x64** from evermeet.cx, **arm64** from martin-riedl.de (Homebrew's arm64 ffmpeg is dynamically linked to ~17 Homebrew dylibs and is NOT self-contained, so it can't be bundled). **x64 plugins** build from source under Rosetta, except `tmedian` which comes pre-built from Stefan-Olt/vs-plugin-build. **`zsmooth` is the one plugin built from source on x64 but taken pre-built on arm64**: the author's x86_64 binary is `minos 13.0` and this bundle targets 12.0, so the minos guard rejects it (it would fail to load on Monterey — exactly issue #39). It is written in Zig, so the x64 branch fetches a pinned Zig toolchain and builds with `-Dtarget=x86_64-macos.12.0`; `ZIG_VERSION` must satisfy zsmooth's `minimum_zig_version`, and the build needs network access for zsmooth's own Zig dependencies. arm64 keeps the pre-built binary, which is under its 15.0 target. +- **FFmpeg** is sourced pre-built as a static binary that links only system frameworks: **x64** from evermeet.cx, **arm64** from martin-riedl.de (Homebrew's arm64 ffmpeg is dynamically linked to ~17 Homebrew dylibs and is NOT self-contained, so it can't be bundled). **x64 plugins** build from source under Rosetta, except `tmedian` which comes pre-built from Stefan-Olt/vs-plugin-build. **`zsmooth` is the one plugin built from source on x64 but taken pre-built on arm64**: the author's x86_64 binary is `minos 13.0` and this bundle targets 12.0, so the minos guard rejects it (it would fail to load on Monterey — exactly issue #39). It is written in Zig, so the x64 branch fetches a pinned Zig toolchain and builds with `-Dtarget=x86_64-macos.12.0`; `ZIG_VERSION` must satisfy zsmooth's `minimum_zig_version`, and the build needs network access for zsmooth's own Zig dependencies. arm64 keeps the pre-built binary, which is under its 15.0 target. Since 2026-08-28 the x64 branch builds it **twice**, `-Dcpu=haswell` and `-Dcpu=x86_64_v2`, into `vapoursynth/zsmooth/` rather than the plugin directory — see "zsmooth ships once per CPU baseline". - **x64 minimum macOS = 12.0 (Monterey), issue #39**: the only hosted Intel runner is `macos-15-intel` (`macos-13` was retired), so Homebrew bottles come out `minos 14/15` and won't load on 12. The x64 build therefore exports `MACOSX_DEPLOYMENT_TARGET=12.0` and **builds the bundled support libs from source** (zimg, fftw, libdvdread, xz, boost) so they target 12; the OpenCL plugins (`nnedi3cl`/`knlmeanscl`) are compiled against that source boost (`BOOST_ROOT="$SRCLIB"`) for ABI match. vspipe's `doubleToString` is patched off `std::to_chars` (needs 13.3+ libc++). A `minos` verification pass at the end fails the build under `STRICT_MIN_OS=1` (set in `build-deps-macos.yml`) if any bundled Mach-O exceeds 12.0. **arm64 is unchanged (still `minos 15`)** — it has no old runner and the prebuilt arm64 plugins are >12. The app/worker deployment target is **per-arch**: the x64 build targets **12.0** and the arm64 build targets **15.0** (matching its minos-15 deps). `build-macos.yml` resolves the target per matrix arch and threads it to rustc (`MACOSX_DEPLOYMENT_TARGET`) and xcodebuild (which overrides the `Runner.xcodeproj` 12.0 baseline); the `Podfile` reads `VAPOURBOX_DEPLOYMENT_TARGET` (default 12.0). `package-macos.sh` sets the same per-arch target for local builds. - **Code signing**: After `install_name_tool` modifications, binaries must be re-signed: `codesign -s - -f ` (exit code 137 = SIGKILL means invalid signature) - Quarantine removal: `xattr -cr` on deps after download @@ -2973,5 +3145,6 @@ Create the app-specific password at appleid.apple.com → Sign-In and Security | 1.0.0 | 2025-01-15 | Initial release | | … | | (1.1.0–1.6.0 went unrecorded) | | 1.7.0 | 2026-08-01 | Fixes QTGMC Placebo/Very Slow brightening and near-black Draft on arm64, via `Scripts/patches/fmtconv-r31-arm-int-scaler.patch` (root cause: sign constants in fmtconv's non-SIMD integer scaler) plus havsfunc patch 5 as defence in depth; fmtconv r30 → **r31**, now pinned and sourced from GitLab on every platform. **Rebuilt 2026-08-02** to add the **zsmooth** plugin (MIT), providing `core.zsmooth.CCD` plus `Cnr4` and a set of RemoveGrain/TemporalMedian-family filters. Version pinned to 0.19.0 in all three download scripts — keep them in step so the same job can't produce different chroma per OS. Taken pre-built everywhere except macOS x64, which builds it with Zig to reach `minos 12.0` (see the macOS platform notes) | +| 1.10.0 | 2026-08-31 | Fixes **issue #82**: zsmooth now ships **one build per CPU baseline** on x86 (`haswell` and `x86_64_v2`) in `vapoursynth/zsmooth/`, outside the autoload directory, with the worker loading exactly one by path. Upstream builds it for an AVX2 baseline with no runtime dispatch, so the shipped binary died with an illegal instruction (`0xC000001D`) on any pre-2013 CPU the instant a filter ran — silently, since vspipe prints nothing on a native crash. The portable build has no upstream asset, so Windows and Linux compile it with a pinned Zig toolchain (Windows' first from-source plugin; Zig needs no MSVC). macOS x64 builds both, which also makes Intel Macs fast for the first time — that arch had always used Zig's default SSE2 baseline, measured 2-3x slower on CCD/Cnr4. Same zsmooth version (0.19.0), so no output changes on any machine that already worked. Also brings **FFmpeg to 9.0 on all four platforms and pins it**: they had silently diverged (Windows on an unpinned post-9.0 master build, both macOS arches floating on 9.0.1, Linux pinned at 7.1), and BtbN garbage-collecting the n7.1 asset from its rolling `latest` tag 404'd the Linux deps build outright. Each script now verifies the installed binary's series and a push-gate test fails if the three pins disagree | | 1.9.0 | 2026-08-15 | Adds three plugins. **fluxsmooth** (`core.flux.SmoothT` / `SmoothST`), which also unlocks havsfunc's **STPresso** — it calls `core.flux.SmoothT` internally and raised "No attribute with the name flux exists" without it. Pinned to **v2** on every platform: that is the newest tag with a published Windows binary, and `download-deps-windows.ps1` has no from-source path, so macOS/Linux track the version Windows can get rather than letting the same job denoise differently per OS. Built on macOS/Linux by invoking the compiler directly on its single C file rather than through its autotools build, so no new build dependency (autoconf/automake/libtool) is added to CI. Also adds **bifrost** (`core.bifrost.Bifrost`, temporal rainbow/dot-crawl removal, pinned v3.0) and **retinex** (`core.retinex.MSRCP`, shadow-detail lift, pinned r4) — both chosen because their *newest* release ships a Windows binary, so no version skew, and both link nothing beyond system libraries. bifrost is another single C file compiled directly, but it includes `` so the scripts stage a small include root whose parent is passed to `-I`; retinex is an ordinary meson build resolving headers through pkg-config | | 1.8.0 | 2026-08-07 | VapourSynth **R73 → R78** on every platform, which moves Windows to a Python 3.12 wheel layout and makes `deps//vapoursynth/` the Python package itself on macOS/Linux (see the R78 sections). Adds the **akarin** plugin (LGPL-3.0, statically links LLVM 22.1.2) supplying an LLVM JIT for `std.Expr`, routed in via havsfunc **patch 7** and the templates' `_expr()` helper — worth **4.1x** on arm64 QTGMC Slow, since VapourSynth's own Expr JIT is x86-only. **Not** shipped on macos-x64, whose only wheel would raise the Intel floor to macOS 14 (issue #39). Fixes the **nnedi3** build on linux-arm64, which had never produced a binary (`-mfpu=neon` and `HWCAP_ARM_*` are both 32-bit-ARM-only), and drops the plugin from linux-x64's expected list to match the other x86 bundles. **BestSource removed** — nothing had called it since the pipe source replaced it. Linux now needs **glibc 2.39** (ubuntu-24.04), so Ubuntu 22.04 and Debian 12 can no longer run it | diff --git a/Scripts/deps-expected-plugins.json b/Scripts/deps-expected-plugins.json index 8e9631c..e2f398f 100644 --- a/Scripts/deps-expected-plugins.json +++ b/Scripts/deps-expected-plugins.json @@ -1,5 +1,5 @@ { - "_comment": "Required VapourSynth plugin filenames per platform \u2014 the contract for a COMPLETE deps bundle. The package-deps-* scripts assert every file listed here exists in the staged bundle before zipping and FAIL the build if any are missing, so a silently-failed download (e.g. a dead upstream URL) becomes a red build instead of an incomplete bundle shipping. Plugin directory: windows-x64 = vapoursynth/vs-plugins, macos/linux = vapoursynth/plugins. Lists exclude data files (nnedi3 weights) and runtime libs (fftw); they cover the VapourSynth plugin binaries only. Update this when adding or removing a plugin.", + "_comment": "Required VapourSynth plugin filenames per platform — the contract for a COMPLETE deps bundle. The package-deps-* scripts assert every file listed here exists in the staged bundle before zipping and FAIL the build if any are missing, so a silently-failed download (e.g. a dead upstream URL) becomes a red build instead of an incomplete bundle shipping. Plugin directory: windows-x64 = vapoursynth/vs-plugins, macos/linux = vapoursynth/plugins. Lists exclude data files (nnedi3 weights) and runtime libs (fftw); they cover the VapourSynth plugin binaries only. Update this when adding or removing a plugin. An entry containing a '/' is a path relative to the BUNDLE ROOT rather than a filename in the plugin directory: zsmooth ships one build per CPU baseline in vapoursynth/zsmooth/, deliberately outside the autoload directory, because both builds register the same namespace and the worker loads exactly one by path (DependencyLocator::zsmooth_plugin).", "windows-x64": [ "AddGrain.dll", "CAS.dll", @@ -32,8 +32,9 @@ "libtemporalmedian.dll", "libzstd.dll", "neo-f3kdb.dll", - "vsznedi3.dll", - "zsmooth.dll" + "vapoursynth/zsmooth/zsmooth-haswell.dll", + "vapoursynth/zsmooth/zsmooth-x86_64_v2.dll", + "vsznedi3.dll" ], "macos-arm64": [ "libaddgrain.dylib", @@ -69,7 +70,7 @@ "libttempsmooth.dylib", "libvivtc.dylib", "libznedi3.dylib", - "libzsmooth.dylib" + "vapoursynth/zsmooth/libzsmooth.dylib" ], "macos-x64": [ "libaddgrain.dylib", @@ -102,7 +103,8 @@ "libttempsmooth.dylib", "libvivtc.dylib", "libznedi3.dylib", - "libzsmooth.dylib" + "vapoursynth/zsmooth/libzsmooth-haswell.dylib", + "vapoursynth/zsmooth/libzsmooth-x86_64_v2.dylib" ], "linux-x64": [ "libaddgrain.so", @@ -136,7 +138,8 @@ "libttempsmooth.so", "libvivtc.so", "libznedi3.so", - "libzsmooth.so" + "vapoursynth/zsmooth/libzsmooth-haswell.so", + "vapoursynth/zsmooth/libzsmooth-x86_64_v2.so" ], "linux-arm64": [ "libaddgrain.so", @@ -171,6 +174,6 @@ "libttempsmooth.so", "libvivtc.so", "libznedi3.so", - "libzsmooth.so" + "vapoursynth/zsmooth/libzsmooth.so" ] } diff --git a/Scripts/download-deps-linux.sh b/Scripts/download-deps-linux.sh index 1263596..10db031 100755 --- a/Scripts/download-deps-linux.sh +++ b/Scripts/download-deps-linux.sh @@ -53,6 +53,42 @@ NPROC=$(nproc) PYTHON_VERSION="3.12.8" PYTHON_MAJOR_MINOR="3.12" +# FFmpeg major.minor series. THIS MUST MATCH the same pin in +# download-deps-macos.sh and download-deps-windows.ps1 — +# app/test/ffmpeg_version_pin_test.dart fails the push gate if the three drift. +# A per-OS FFmpeg is a per-OS encoder: the arguments in pipeline_executor.rs +# would then mean subtly different things depending on where the job ran. +# +# Pinning the SERIES rather than a full version is deliberate. Every upstream +# here publishes "latest build of the X.Y branch" and garbage-collects older +# series, so a full-version pin cannot be held for long — that is precisely how +# the previous n7.1 pin turned into a 404. +FFMPEG_SERIES="9.0" + +# Fail the build if the FFmpeg we installed is not the series we pinned. +# Without this, an upstream that silently redirects "latest" to a new series +# reintroduces cross-platform skew with nothing to notice it. +assert_ffmpeg_series() { + local bin="$1" + local reported + # No backslashes in this parser on purpose: a bracket expression [.] + # matches a literal dot without one. Written with sed backrefs first, + # the escapes were mangled into control characters in transit and the + # check silently reported an empty version. + reported=$("$bin" -version 2>/dev/null | head -1 | grep -oE "version n?[0-9]+[.][0-9]+" | grep -oE "[0-9]+[.][0-9]+" | head -1) + if [ -z "$reported" ]; then + echo " ERROR: could not read a version from $bin" >&2 + exit 1 + fi + if [ "$reported" != "$FFMPEG_SERIES" ]; then + echo " ERROR: FFmpeg is $reported but this bundle pins $FFMPEG_SERIES." >&2 + echo " Upstream moved the URL to a different series. Update FFMPEG_SERIES" >&2 + echo " in ALL THREE download-deps-* scripts together, or platforms drift." >&2 + exit 1 + fi + echo " FFmpeg $reported (matches the pinned $FFMPEG_SERIES series)" +} + echo "=== VapourBox Linux Dependencies Builder ===" echo "Architecture: $ARCH" echo "Platform: $PLATFORM_DIR" @@ -447,13 +483,16 @@ if [ "$FORCE" = true ] || [ ! -f "$DEPS_DIR/ffmpeg/ffmpeg" ]; then # gracefully at init. (The previous John Van Sickle build had no hw accel.) BTBN_BASE="https://github.com/BtbN/FFmpeg-Builds/releases/download/latest" if [ "$ARCH" = "x86_64" ]; then - FFMPEG_URL="$BTBN_BASE/ffmpeg-n7.1-latest-linux64-gpl-7.1.tar.xz" + FFMPEG_URL="$BTBN_BASE/ffmpeg-n${FFMPEG_SERIES}-latest-linux64-gpl-${FFMPEG_SERIES}.tar.xz" else - FFMPEG_URL="$BTBN_BASE/ffmpeg-n7.1-latest-linuxarm64-gpl-7.1.tar.xz" + FFMPEG_URL="$BTBN_BASE/ffmpeg-n${FFMPEG_SERIES}-latest-linuxarm64-gpl-${FFMPEG_SERIES}.tar.xz" fi - echo " Downloading static FFmpeg (BtbN, hardware-enabled)..." - curl -L -o "$BUILD_DIR/ffmpeg.tar.xz" "$FFMPEG_URL" + echo " Downloading static FFmpeg $FFMPEG_SERIES (BtbN, hardware-enabled)..." + # -f, or a 404 body is written to the tarball and surfaces much later as an + # unexplained `tar: Error is not recoverable`. That is exactly how the n7.1 + # asset ageing out of BtbN's rolling `latest` tag presented. + curl -fL -o "$BUILD_DIR/ffmpeg.tar.xz" "$FFMPEG_URL" echo " Extracting..." tar -xJf "$BUILD_DIR/ffmpeg.tar.xz" -C "$BUILD_DIR" @@ -468,6 +507,7 @@ if [ "$FORCE" = true ] || [ ! -f "$DEPS_DIR/ffmpeg/ffmpeg" ]; then cp "$FFMPEG_DIR/ffmpeg" "$DEPS_DIR/ffmpeg/" cp "$FFMPEG_DIR/ffprobe" "$DEPS_DIR/ffmpeg/" chmod +x "$DEPS_DIR/ffmpeg/ffmpeg" "$DEPS_DIR/ffmpeg/ffprobe" + assert_ffmpeg_series "$DEPS_DIR/ffmpeg/ffmpeg" echo " Downloaded FFmpeg" else echo " FFmpeg already exists, skipping" @@ -984,24 +1024,50 @@ build_plugin "tmedian" \ "libtmedian.so" \ "$PLUGIN_BUILD_ENV meson setup build --buildtype=release && ninja -C build" -# zsmooth (core.zsmooth.CCD - chroma denoiser; also Cnr4 and a set of -# RemoveGrain/TemporalMedian-family filters). +# zsmooth — one build per CPU baseline +# +# core.zsmooth.CCD (also Cnr4 and a set of RemoveGrain/TemporalMedian-family +# filters). Upstream publishes only `haswell` (an AVX2 baseline) and `znver4` +# x86 builds, compiled throughout with NO runtime dispatch — so on a pre-2013 +# CPU the library loads fine and then dies with an illegal instruction the +# instant a filter runs. That is issue #82 (reported on Windows, but this +# bundle took the same haswell asset), and it is silent: vspipe prints nothing. +# +# So x86 ships both builds outside the autoload directory and the worker loads +# exactly one by path (DependencyLocator::zsmooth_plugin). They cannot share a +# directory: each registers the namespace `zsmooth`, so whichever autoloads +# second is rejected. aarch64 has a single NEON baseline and needs no split. # -# Taken pre-built rather than built from source: zsmooth is written in Zig, and -# adding a Zig toolchain to every deps build for one plugin is not worth it. The -# author publishes a binary for every platform/arch VapourBox targets. +# Measured at 720x576: `x86_64` is 2.0x slower than haswell on CCD and 3.0x on +# Cnr4, `x86_64_v2` 1.4x on both — which is why the portable build is v2 +# (SSE4.2/POPCNT, everything from Nehalem 2009 on) and why the fast build is +# still shipped rather than dropped for one portable binary. # # Keep ZSMOOTH_VERSION in step across download-deps-{macos,linux}.sh and # download-deps-windows.ps1 — a version skew would make the same job produce # different chroma per OS. ZSMOOTH_VERSION="0.19.0" +# Must satisfy zsmooth's build.zig.zon `minimum_zig_version` (0.15.2 for 0.19.0). +ZIG_VERSION="0.15.2" +ZSMOOTH_DIR="$DEPS_DIR/vapoursynth/zsmooth" +mkdir -p "$ZSMOOTH_DIR" + +echo "" +echo "=== Installing zsmooth ===" + +# The pre-built asset: haswell on x86, the only build on aarch64. case "$ARCH" in - aarch64|arm64) ZSMOOTH_ASSET="zsmooth-aarch64-linux-gnu.zip" ;; - *) ZSMOOTH_ASSET="zsmooth-x86_64-linux-gnu.zip" ;; + aarch64|arm64) + ZSMOOTH_ASSET="zsmooth-aarch64-linux-gnu.zip" + ZSMOOTH_PREBUILT="$ZSMOOTH_DIR/libzsmooth.so" + ;; + *) + ZSMOOTH_ASSET="zsmooth-x86_64-linux-gnu.zip" + ZSMOOTH_PREBUILT="$ZSMOOTH_DIR/libzsmooth-haswell.so" + ;; esac -echo "" -echo "=== Downloading zsmooth ===" -if [ "$FORCE" = true ] || [ ! -f "$PLUGINS_DIR/libzsmooth.so" ]; then + +if [ "$FORCE" = true ] || [ ! -f "$ZSMOOTH_PREBUILT" ]; then rm -rf "$BUILD_DIR/zsmooth" mkdir -p "$BUILD_DIR/zsmooth" if curl -sL -o "$BUILD_DIR/zsmooth/zsmooth.zip" \ @@ -1009,9 +1075,9 @@ if [ "$FORCE" = true ] || [ ! -f "$PLUGINS_DIR/libzsmooth.so" ]; then && unzip -q -o "$BUILD_DIR/zsmooth/zsmooth.zip" -d "$BUILD_DIR/zsmooth"; then so_path=$(find "$BUILD_DIR/zsmooth" -name "*.so" -type f 2>/dev/null | head -1) if [ -n "$so_path" ]; then - cp "$so_path" "$PLUGINS_DIR/libzsmooth.so" - patchelf --set-rpath '$ORIGIN:$ORIGIN/../../lib' "$PLUGINS_DIR/libzsmooth.so" 2>/dev/null || true - echo " Downloaded pre-built zsmooth -> libzsmooth.so" + cp "$so_path" "$ZSMOOTH_PREBUILT" + patchelf --set-rpath '$ORIGIN:$ORIGIN/../../lib' "$ZSMOOTH_PREBUILT" 2>/dev/null || true + echo " Downloaded pre-built zsmooth -> $(basename "$ZSMOOTH_PREBUILT")" BUILT_PLUGINS+=("zsmooth") else echo " Failed: no .so in the zsmooth archive" @@ -1023,9 +1089,51 @@ if [ "$FORCE" = true ] || [ ! -f "$PLUGINS_DIR/libzsmooth.so" ]; then fi rm -rf "$BUILD_DIR/zsmooth" else - echo " zsmooth already exists, skipping" + echo " $(basename "$ZSMOOTH_PREBUILT") already exists, skipping" fi +# The portable x86 build has no upstream asset and must be compiled. Zig brings +# its own libc and builds zsmooth's fftw dependency itself, so this adds no apt +# package — only network access, since `zig build` fetches zsmooth's own Zig +# dependencies. +case "$ARCH" in + aarch64|arm64) : ;; + *) + ZSMOOTH_V2="$ZSMOOTH_DIR/libzsmooth-x86_64_v2.so" + if [ "$FORCE" = true ] || [ ! -f "$ZSMOOTH_V2" ]; then + echo " Building zsmooth $ZSMOOTH_VERSION (x86_64_v2, runs without AVX2)..." + # Subshell so a failure cannot abort the script under `set -e`; the + # file check below decides whether it worked. + ( + set -e + cd "$BUILD_DIR" + rm -rf zig-toolchain zsmooth-src zig.tar.xz + curl -fsSL -o zig.tar.xz \ + "https://ziglang.org/download/${ZIG_VERSION}/zig-x86_64-linux-${ZIG_VERSION}.tar.xz" + mkdir -p zig-toolchain + tar -xf zig.tar.xz -C zig-toolchain --strip-components=1 + git clone --depth 1 --branch "$ZSMOOTH_VERSION" \ + https://github.com/adworacz/zsmooth.git zsmooth-src + cd zsmooth-src + "$BUILD_DIR/zig-toolchain/zig" build \ + -Doptimize=ReleaseFast -Dcpu=x86_64_v2 + cp zig-out/lib/libzsmooth.so "$ZSMOOTH_V2" + ) || true + + if [ -f "$ZSMOOTH_V2" ]; then + patchelf --set-rpath '$ORIGIN:$ORIGIN/../../lib' "$ZSMOOTH_V2" 2>/dev/null || true + echo " Built zsmooth -> $(basename "$ZSMOOTH_V2")" + else + echo " Warning: failed to build the portable zsmooth" + FAILED_PLUGINS+=("zsmooth-x86_64_v2") + fi + rm -rf "$BUILD_DIR/zig-toolchain" "$BUILD_DIR/zsmooth-src" "$BUILD_DIR/zig.tar.xz" + else + echo " $(basename "$ZSMOOTH_V2") already exists, skipping" + fi + ;; +esac + # DeScratch (core.descratch.DeScratch - vertical scratch removal) # Built from source: the repo carries the VapourSynth + AviSynthPlus headers as # submodules, so a recursive clone is required (build_plugin can't fetch those). diff --git a/Scripts/download-deps-macos.sh b/Scripts/download-deps-macos.sh index 2a09693..5abe82f 100755 --- a/Scripts/download-deps-macos.sh +++ b/Scripts/download-deps-macos.sh @@ -90,6 +90,42 @@ BUILD_DIR="/tmp/vapourbox-build-$$" # Python version to embed PYTHON_VERSION="3.12.8" + +# FFmpeg major.minor series. THIS MUST MATCH the same pin in +# download-deps-linux.sh and download-deps-windows.ps1 — +# app/test/ffmpeg_version_pin_test.dart fails the push gate if they drift. +# A per-OS FFmpeg is a per-OS encoder: the arguments in pipeline_executor.rs +# would then mean subtly different things depending on where the job ran. +FFMPEG_SERIES="9.0" +# The exact evermeet build for x64. Full version rather than series because +# evermeet publishes per-version URLs and keeps them reachable; must be within +# $FFMPEG_SERIES, which assert_ffmpeg_series checks. +FFMPEG_MACOS_X64_VERSION="9.0.1" + +# Fail the build if the FFmpeg we installed is not the series we pinned. +# Without this, an upstream that moves `latest` onto a new series reintroduces +# cross-platform skew with nothing to notice it — which is the whole reason +# Linux sat two majors behind for months. +assert_ffmpeg_series() { + local bin="$1" + local reported + # No backslashes in this parser on purpose: a bracket expression [.] + # matches a literal dot without one. Written with sed backrefs first, + # the escapes were mangled into control characters in transit and the + # check silently reported an empty version. + reported=$("$bin" -version 2>/dev/null | head -1 | grep -oE "version n?[0-9]+[.][0-9]+" | grep -oE "[0-9]+[.][0-9]+" | head -1) + if [ -z "$reported" ]; then + echo " ERROR: could not read a version from $bin" >&2 + exit 1 + fi + if [ "$reported" != "$FFMPEG_SERIES" ]; then + echo " ERROR: FFmpeg is $reported but this bundle pins $FFMPEG_SERIES." >&2 + echo " Upstream moved onto a different series. Update FFMPEG_SERIES in" >&2 + echo " ALL THREE download-deps-* scripts together, or platforms drift." >&2 + exit 1 + fi + echo " FFmpeg $reported (matches the pinned $FFMPEG_SERIES series)" +} PYTHON_MAJOR_MINOR="3.12" echo "=== VapourBox macOS Dependencies Builder ===" @@ -567,14 +603,21 @@ if [ "$ARCH" = "x86_64" ]; then # evermeet.cx ships static x86_64 ffmpeg/ffprobe that link only system # frameworks (verified self-contained), so no dylib wrangling is needed. # This is the canonical pre-built source for Intel macOS ffmpeg. - echo " Downloading static x86_64 FFmpeg from evermeet.cx..." - curl -sL "https://evermeet.cx/ffmpeg/getrelease/ffmpeg/zip" -o "$BUILD_DIR/ffmpeg.zip" - curl -sL "https://evermeet.cx/ffmpeg/getrelease/ffprobe/zip" -o "$BUILD_DIR/ffprobe.zip" + # + # Pinned to an exact version rather than `getrelease`, which is whatever is + # newest and would silently drift onto a different series from the other + # platforms. evermeet keeps old versions reachable (verified: 7.1, 8.0 and + # 9.0.1 all still resolve), so a full-version pin is durable here — unlike + # BtbN, whose rolling tag garbage-collects old series. + echo " Downloading static x86_64 FFmpeg $FFMPEG_MACOS_X64_VERSION from evermeet.cx..." + curl -fsSL "https://evermeet.cx/ffmpeg/ffmpeg-${FFMPEG_MACOS_X64_VERSION}.zip" -o "$BUILD_DIR/ffmpeg.zip" + curl -fsSL "https://evermeet.cx/ffmpeg/ffprobe-${FFMPEG_MACOS_X64_VERSION}.zip" -o "$BUILD_DIR/ffprobe.zip" unzip -q -o "$BUILD_DIR/ffmpeg.zip" -d "$DEPS_DIR/ffmpeg/" unzip -q -o "$BUILD_DIR/ffprobe.zip" -d "$DEPS_DIR/ffmpeg/" chmod +x "$DEPS_DIR/ffmpeg/ffmpeg" "$DEPS_DIR/ffmpeg/ffprobe" codesign -s - -f "$DEPS_DIR/ffmpeg/ffmpeg" 2>/dev/null || true codesign -s - -f "$DEPS_DIR/ffmpeg/ffprobe" 2>/dev/null || true + assert_ffmpeg_series "$DEPS_DIR/ffmpeg/ffmpeg" echo " Installed evermeet.cx FFmpeg" else # arm64: Homebrew's ffmpeg is dynamically linked against ~17 Homebrew dylibs @@ -584,13 +627,20 @@ else # frameworks, ~60 MB) - the same source the 1.3.0 deps shipped, and the # arm64 analogue of the evermeet.cx static build used for x64 above. echo " Downloading static arm64 FFmpeg from martin-riedl.de..." - curl -sL "https://ffmpeg.martin-riedl.de/redirect/latest/macos/arm64/release/ffmpeg.zip" -o "$BUILD_DIR/ffmpeg.zip" - curl -sL "https://ffmpeg.martin-riedl.de/redirect/latest/macos/arm64/release/ffprobe.zip" -o "$BUILD_DIR/ffprobe.zip" + # + # This host only publishes `latest` plus opaque build-id paths whose + # retention is unknown, so pinning a URL is not available. Take latest and + # let assert_ffmpeg_series below fail the build if it has moved off the + # pinned series — a red build naming the drift is better than silently + # shipping a different FFmpeg here than everywhere else. + curl -fsSL "https://ffmpeg.martin-riedl.de/redirect/latest/macos/arm64/release/ffmpeg.zip" -o "$BUILD_DIR/ffmpeg.zip" + curl -fsSL "https://ffmpeg.martin-riedl.de/redirect/latest/macos/arm64/release/ffprobe.zip" -o "$BUILD_DIR/ffprobe.zip" unzip -q -o "$BUILD_DIR/ffmpeg.zip" -d "$DEPS_DIR/ffmpeg/" unzip -q -o "$BUILD_DIR/ffprobe.zip" -d "$DEPS_DIR/ffmpeg/" chmod +x "$DEPS_DIR/ffmpeg/ffmpeg" "$DEPS_DIR/ffmpeg/ffprobe" codesign -s - -f "$DEPS_DIR/ffmpeg/ffmpeg" 2>/dev/null || true codesign -s - -f "$DEPS_DIR/ffmpeg/ffprobe" 2>/dev/null || true + assert_ffmpeg_series "$DEPS_DIR/ffmpeg/ffmpeg" echo " Installed static arm64 FFmpeg from martin-riedl.de" fi @@ -1439,25 +1489,65 @@ build_plugin "retinex" \ "libretinex.dylib" \ "meson setup build --buildtype=release && ninja -C build" -# zsmooth (core.zsmooth.CCD - chroma denoiser; also Cnr4 and a set of -# RemoveGrain/TemporalMedian-family filters). +# zsmooth — one build per CPU baseline +# +# core.zsmooth.CCD (also Cnr4 and a set of RemoveGrain/TemporalMedian-family +# filters). Upstream publishes only `haswell` (an AVX2 baseline) and `znver4` +# x86 builds, compiled throughout with NO runtime dispatch, so on a pre-2013 +# CPU they die with an illegal instruction the instant a filter runs — issue +# #82, silently, because vspipe prints nothing on a native crash. +# +# x86_64 therefore ships TWO builds outside the autoload directory and the +# worker loads exactly one by path (DependencyLocator::zsmooth_plugin); they +# cannot share a directory, because each registers the namespace `zsmooth` and +# whichever autoloads second is rejected. arm64 has one NEON baseline and needs +# no split. +# +# Note this arch was ALREADY affected in the other direction: the x64 build +# below has always been compiled at Zig's default baseline (SSE2), which +# measures 2.0x slower than haswell on CCD and 3.0x on Cnr4. Every Intel Mac +# that can run macOS 12 is at least Nehalem and most are Haswell or newer, so +# building both here makes the common case fast for the first time as well as +# keeping the oldest ones working. # # Keep ZSMOOTH_VERSION in step across download-deps-{macos,linux}.sh and # download-deps-windows.ps1 — a version skew would make the same job produce # different chroma per OS. ZSMOOTH_VERSION="0.19.0" +ZSMOOTH_DIR="$DEPS_DIR/vapoursynth/zsmooth" +mkdir -p "$ZSMOOTH_DIR" if [ "$ARCH" = "arm64" ]; then # arm64 takes the author's build: it is minos 13, comfortably under this - # arch's 15.0 target. - download_prebuilt_plugin "zsmooth" "libzsmooth.dylib" \ - "https://github.com/adworacz/zsmooth/releases/download/${ZSMOOTH_VERSION}/zsmooth-aarch64-macos.zip" + # arch's 15.0 target. One build, no variants. + if [ "$FORCE" = true ] || [ ! -f "$ZSMOOTH_DIR/libzsmooth.dylib" ]; then + tmp="$BUILD_DIR/prebuilt-zsmooth" + rm -rf "$tmp"; mkdir -p "$tmp" + zs_url="https://github.com/adworacz/zsmooth/releases/download/${ZSMOOTH_VERSION}/zsmooth-aarch64-macos.zip" + if curl -sL "$zs_url" -o "$tmp/plugin.zip" && unzip -q -o "$tmp/plugin.zip" -d "$tmp"; then + found=$(find "$tmp" -name "*.dylib" -type f 2>/dev/null | head -1) + if [ -n "$found" ]; then + cp "$found" "$ZSMOOTH_DIR/libzsmooth.dylib" + install_name_tool -id "@loader_path/libzsmooth.dylib" "$ZSMOOTH_DIR/libzsmooth.dylib" 2>/dev/null || true + codesign -s - -f "$ZSMOOTH_DIR/libzsmooth.dylib" 2>/dev/null || true + echo " Downloaded pre-built zsmooth" + else + echo " Warning: no dylib in the zsmooth archive" + FAILED_PLUGINS+=("zsmooth") + fi + else + echo " Warning: failed to fetch pre-built zsmooth" + FAILED_PLUGINS+=("zsmooth") + fi + rm -rf "$tmp" + else + echo " zsmooth already exists, skipping" + fi else - # x64 builds from source (issue #39). The author's x86_64 build is minos - # 13.0, and this bundle targets 12.0, so the pre-built binary is rejected by - # the minos guard at the end of this script — it would fail to load on - # Monterey with a dyld error, which is the exact failure #39 was opened for. - # Same reason zimg/fftw/boost are built from source in the x64 branch above. + # x64 builds from source, for two reasons: the author's x86_64 build is + # minos 13.0 and this bundle targets 12.0, so the minos guard at the end of + # this script rejects it (issue #39 — it would fail to load on Monterey); + # and upstream ships no build that runs without AVX2 at all. # # zsmooth is written in Zig, so this needs a Zig toolchain. It is fetched # here rather than installed globally: it is used for this one plugin, and @@ -1466,48 +1556,108 @@ else # 0.19.0. `zig build` also fetches zsmooth's own Zig dependencies # (vapoursynth headers, fftw), so this step needs network access. ZIG_VERSION="0.15.2" - echo "" - echo "=== Building zsmooth from source (x64, targeting macOS $MACOS_MIN_VERSION) ===" - if [ "$FORCE" = true ] || [ ! -f "$PLUGINS_DIR/libzsmooth.dylib" ]; then + # The fftw fork zsmooth depends on, pinned to the same ref its + # build.zig.zon names. Keep in step when ZSMOOTH_VERSION moves: + # a mismatch here would silently build a different fftw. + FFTW_FORK_TAG="v3.3.11-2" + # Zig needs a full x.y version here: "x86_64-macos.12" is rejected as an + # invalid OS version, "x86_64-macos.12.0" is accepted. Tolerate a bare + # major from a $MACOS_MIN_VERSION override. + case "$MACOS_MIN_VERSION" in + *.*) ZIG_MACOS_MIN="$MACOS_MIN_VERSION" ;; + *) ZIG_MACOS_MIN="${MACOS_MIN_VERSION}.0" ;; + esac + + ZIG_BIN="" + # x86_64_v2 is SSE4.2/POPCNT — every Intel Mac that can run macOS 12. + # haswell is the fast path for 2013-and-later machines. Order matters only + # for the log; the worker picks by CPUID at job time. + for zs_target in haswell x86_64_v2; do + out="$ZSMOOTH_DIR/libzsmooth-${zs_target}.dylib" + if [ "$FORCE" = false ] && [ -f "$out" ]; then + echo " zsmooth ($zs_target) already exists, skipping" + continue + fi + echo "" + echo "=== Building zsmooth $zs_target (x64, targeting macOS $MACOS_MIN_VERSION) ===" # Subshell so a failure here can't abort the whole script under `set -e`; # the file check below decides whether it worked. ( set -e cd "$BUILD_DIR" - rm -rf zig-toolchain zsmooth - curl -fsSL -o zig.tar.xz \ - "https://ziglang.org/download/${ZIG_VERSION}/zig-x86_64-macos-${ZIG_VERSION}.tar.xz" - mkdir -p zig-toolchain - tar -xf zig.tar.xz -C zig-toolchain --strip-components=1 + if [ -z "$ZIG_BIN" ]; then + rm -rf zig-toolchain zig.tar.xz + curl -fsSL -o zig.tar.xz \ + "https://ziglang.org/download/${ZIG_VERSION}/zig-x86_64-macos-${ZIG_VERSION}.tar.xz" + mkdir -p zig-toolchain + tar -xf zig.tar.xz -C zig-toolchain --strip-components=1 + fi + rm -rf zsmooth fftw-patched git clone --depth 1 --branch "$ZSMOOTH_VERSION" \ https://github.com/adworacz/zsmooth.git zsmooth + + # zsmooth's Zig fftw port declares HAVE_MEMALIGN on every non-Windows + # target, but macOS has no memalign() — it is declared in , + # which the SAME file already knows macOS lacks (HAVE_MALLOC_H is + # gated on !is_mac). fftw's kalloc.c only reaches that branch when + # MIN_ALIGNMENT is 32, i.e. when AVX is enabled, so the bug is + # invisible at the SSE-level baselines and kills ONLY the haswell + # build, with a clang implicit-declaration error inside a dependency. + # HAVE_POSIX_MEMALIGN is already true, so clearing this falls through + # to posix_memalign, which macOS does have. + # + # Patched via a local path dependency rather than by editing Zig's + # global package cache: path deps take no hash, so this is + # deterministic and cannot be invalidated by a cache wipe. + git clone --depth 1 --branch "$FFTW_FORK_TAG" \ + https://github.com/adworacz/fftw.git fftw-patched + if ! grep -q '.HAVE_MEMALIGN = if (!is_windows) true else null,' \ + fftw-patched/build.zig; then + echo " ERROR: the fftw HAVE_MEMALIGN line is not what the patch expects." >&2 + echo " Upstream may have fixed it — re-check before removing this patch." >&2 + exit 1 + fi + # `is_mac` is already defined in that file. + sed -i.bak \ + 's/\.HAVE_MEMALIGN = if (!is_windows) true else null,/.HAVE_MEMALIGN = if (!is_windows and !is_mac) true else null,/' \ + fftw-patched/build.zig + grep -q '.HAVE_MEMALIGN = if (!is_windows and !is_mac) true else null,' \ + fftw-patched/build.zig || { echo " ERROR: fftw memalign patch did not apply" >&2; exit 1; } + cd zsmooth - # Zig needs a full x.y version here: "x86_64-macos.12" is rejected - # as an invalid OS version, "x86_64-macos.12.0" is accepted. Tolerate - # a bare major from a $MACOS_MIN_VERSION override. - case "$MACOS_MIN_VERSION" in - *.*) ZIG_MACOS_MIN="$MACOS_MIN_VERSION" ;; - *) ZIG_MACOS_MIN="${MACOS_MIN_VERSION}.0" ;; - esac + # Repoint the fftw dependency at the patched clone. A path dependency + # carries no hash field, so the url+hash pair is replaced wholesale. + "$PYTHON_BIN" - <<'ZONEOF' +import io, re +p = "build.zig.zon" +s = io.open(p, encoding="utf-8").read() +pat = re.compile(r'\.fftw = \.\{[^}]*\}', re.S) +if not pat.search(s): + raise SystemExit("ERROR: no .fftw dependency block in zsmooth build.zig.zon") +s = pat.sub('.fftw = .{ .path = "../fftw-patched" }', s, count=1) +io.open(p, "w", encoding="utf-8").write(s) +print(" fftw repointed to the patched local clone") +ZONEOF + "$BUILD_DIR/zig-toolchain/zig" build \ -Doptimize=ReleaseFast \ - -Dtarget="x86_64-macos.${ZIG_MACOS_MIN}" - cp zig-out/lib/libzsmooth.dylib "$PLUGINS_DIR/libzsmooth.dylib" + -Dtarget="x86_64-macos.${ZIG_MACOS_MIN}" \ + -Dcpu="$zs_target" + cp zig-out/lib/libzsmooth.dylib "$out" ) || true - if [ -f "$PLUGINS_DIR/libzsmooth.dylib" ]; then - install_name_tool -id "@loader_path/libzsmooth.dylib" \ - "$PLUGINS_DIR/libzsmooth.dylib" 2>/dev/null || true - codesign -s - -f "$PLUGINS_DIR/libzsmooth.dylib" 2>/dev/null || true - echo " Built zsmooth -> libzsmooth.dylib" + if [ -f "$out" ]; then + ZIG_BIN="$BUILD_DIR/zig-toolchain/zig" + install_name_tool -id "@loader_path/$(basename "$out")" "$out" 2>/dev/null || true + codesign -s - -f "$out" 2>/dev/null || true + echo " Built zsmooth -> $(basename "$out")" else - echo " Warning: failed to build zsmooth" - FAILED_PLUGINS+=("zsmooth") + echo " Warning: failed to build zsmooth ($zs_target)" + FAILED_PLUGINS+=("zsmooth-$zs_target") fi - rm -rf "$BUILD_DIR/zig-toolchain" "$BUILD_DIR/zsmooth" "$BUILD_DIR/zig.tar.xz" - else - echo " zsmooth already exists, skipping" - fi + rm -rf "$BUILD_DIR/zsmooth" "$BUILD_DIR/fftw-patched" + done + rm -rf "$BUILD_DIR/zig-toolchain" "$BUILD_DIR/zig.tar.xz" fi # ============================================================================ diff --git a/Scripts/download-deps-windows.ps1 b/Scripts/download-deps-windows.ps1 index 4ed0d49..8141295 100644 --- a/Scripts/download-deps-windows.ps1 +++ b/Scripts/download-deps-windows.ps1 @@ -149,7 +149,17 @@ Write-Host "" Write-Host "[2/7] Downloading FFmpeg..." -ForegroundColor Yellow $FFmpegZip = Join-Path $TempDir "ffmpeg.zip" -$FFmpegUrl = "https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-win64-gpl.zip" +# FFmpeg major.minor series. THIS MUST MATCH the same pin in +# download-deps-{linux,macos}.sh — app/test/ffmpeg_version_pin_test.dart fails +# the push gate if the three drift. A per-OS FFmpeg is a per-OS encoder: the +# arguments in pipeline_executor.rs would then mean subtly different things +# depending on where the job ran. +# +# This was `ffmpeg-master-latest-win64-gpl.zip` until 2026-08-31 — an UNPINNED +# moving target, which is how Windows came to ship a post-9.0 master build +# while Linux was still pinned at 7.1. +$FFmpegSeries = "9.0" +$FFmpegUrl = "https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-n$FFmpegSeries-latest-win64-gpl-$FFmpegSeries.zip" if (-not (Test-Path "$FullTargetDir\ffmpeg\ffmpeg.exe")) { Download-File -Url $FFmpegUrl -OutFile $FFmpegZip @@ -167,7 +177,20 @@ if (-not (Test-Path "$FullTargetDir\ffmpeg\ffmpeg.exe")) { Remove-Item $FFmpegZip -Force Remove-Item $FFmpegTempDir -Recurse -Force - Write-Host " FFmpeg installed" -ForegroundColor Green + + # Fail the build if what landed is not the series we pinned. Without this, + # an upstream URL change reintroduces cross-platform skew unnoticed. + $InstalledFFmpeg = Join-Path (Join-Path $FullTargetDir "ffmpeg") "ffmpeg.exe" + $FFmpegVersionLine = & $InstalledFFmpeg -version 2>$null | Select-Object -First 1 + if ($FFmpegVersionLine -notmatch '^ffmpeg version n?(\d+\.\d+)') { + throw "Could not read a version from the installed ffmpeg.exe (got: $FFmpegVersionLine)" + } + if ($Matches[1] -ne $FFmpegSeries) { + throw ("FFmpeg is $($Matches[1]) but this bundle pins $FFmpegSeries. Upstream moved " + + "the URL to a different series. Update the series in ALL THREE " + + "download-deps-* scripts together, or platforms drift.") + } + Write-Host " FFmpeg $($Matches[1]) installed (matches the pinned $FFmpegSeries series)" -ForegroundColor Green } else { Write-Host " FFmpeg already installed" -ForegroundColor Gray } @@ -372,18 +395,9 @@ $PluginsZip = @( Name = "knlmeanscl" Url = "https://github.com/Khanattila/KNLMeansCL/releases/download/v1.1.1/KNLMeansCL-v1.1.1.zip" Check = "KNLMeansCL.dll" - }, - @{ - # core.zsmooth.CCD - chroma denoiser (also Cnr4 and a set of - # RemoveGrain/TemporalMedian-family filters). Written in Zig, so it is - # taken pre-built on every platform rather than adding a Zig toolchain to - # the deps builds. Keep the version in step with ZSMOOTH_VERSION in - # download-deps-{macos,linux}.sh: a skew would make the same job produce - # different chroma per OS. - Name = "zsmooth" - Url = "https://github.com/adworacz/zsmooth/releases/download/0.19.0/zsmooth-x86_64-windows.zip" - Check = "zsmooth.dll" } + # zsmooth is NOT here: it ships as two CPU-specific builds outside the + # autoload directory. See section 4a below. ) foreach ($Plugin in $Plugins7z) { @@ -495,6 +509,124 @@ if ($BadArch.Count -gt 0) { } Write-Host " All plugin DLLs are x64" -ForegroundColor Green +# ============================================================================= +# 4a. zsmooth — one build per CPU baseline +# ============================================================================= +# core.zsmooth.CCD (also Cnr4 and a set of RemoveGrain/TemporalMedian-family +# filters). Upstream publishes only `haswell` (an AVX2 baseline) and `znver4` +# builds, compiled throughout with NO runtime dispatch — so on a pre-2013 x86 +# CPU the DLL loads fine and then dies with an illegal instruction +# (0xC000001D) the instant a filter runs. That is issue #82, reported on a +# Celeron J4105 and a Core i7 870, and it is silent: vspipe prints nothing, so +# the encode surfaces as ffmpeg reading an empty pipe. +# +# Both builds are therefore shipped and the worker loads exactly one by path +# (DependencyLocator::zsmooth_plugin). They cannot both sit in vs-plugins — +# each registers the namespace `zsmooth`, so whichever autoloads second is +# rejected — hence the separate directory, which is deliberately not on the +# plugin path. +# +# Why not just ship the portable build for everyone: measured on this plugin at +# 720x576, `x86_64` is 2.0x slower than haswell on CCD and 3.0x on Cnr4 +# (`x86_64_v2` 1.4x and 1.4x). Paying that on every modern machine to serve the +# rare old one is the wrong trade; picking at runtime costs ~4 MB of zip. +# +# Keep ZSMOOTH_VERSION in step with download-deps-{macos,linux}.sh: a skew +# would make the same job produce different chroma per OS. +Write-Host "" +Write-Host "[4a/8] Installing zsmooth (per-CPU builds)..." -ForegroundColor Yellow + +$ZsmoothVersion = "0.19.0" +# Must satisfy zsmooth's build.zig.zon `minimum_zig_version`; 0.15.2 for 0.19.0. +$ZigVersion = "0.15.2" +$ZsmoothDir = "$FullTargetDir\vapoursynth\zsmooth" +if (-not (Test-Path $ZsmoothDir)) { + New-Item -ItemType Directory -Force -Path $ZsmoothDir | Out-Null +} + +# The AVX2 build comes pre-built from upstream. +$HaswellPath = "$ZsmoothDir\zsmooth-haswell.dll" +if (-not (Test-Path $HaswellPath)) { + Write-Host " Downloading zsmooth $ZsmoothVersion (haswell/AVX2)..." -ForegroundColor Gray + try { + $ZsZip = Join-Path $TempDir "zsmooth-haswell.zip" + $ZsExtract = Join-Path $TempDir "zsmooth-haswell-extract" + Download-File -Url "https://github.com/adworacz/zsmooth/releases/download/$ZsmoothVersion/zsmooth-x86_64-windows.zip" -OutFile $ZsZip + Expand-Archive -Path $ZsZip -DestinationPath $ZsExtract -Force + $Dll = Get-ChildItem -Path $ZsExtract -Recurse -Filter "zsmooth.dll" | Select-Object -First 1 + if (-not $Dll) { throw "no zsmooth.dll in the upstream archive" } + Copy-Item $Dll.FullName $HaswellPath -Force + Remove-Item $ZsZip -Force -ErrorAction SilentlyContinue + Remove-Item $ZsExtract -Recurse -Force -ErrorAction SilentlyContinue + Write-Host " Installed: zsmooth-haswell.dll" -ForegroundColor Gray + } catch { + Write-Host " Failed: $_" -ForegroundColor Red + } +} else { + Write-Host " zsmooth-haswell.dll already installed" -ForegroundColor Gray +} + +# The portable build has no upstream asset and must be compiled. This is the +# first from-source build in this script; Zig cross-compiles with its own libc +# and builds zsmooth's fftw dependency itself, so it needs no MSVC — only git +# and network access (zig build fetches zsmooth's own Zig dependencies). +$V2Path = "$ZsmoothDir\zsmooth-x86_64_v2.dll" +if (-not (Test-Path $V2Path)) { + Write-Host " Building zsmooth $ZsmoothVersion (x86_64_v2, runs without AVX2)..." -ForegroundColor Gray + try { + $ZigDir = Join-Path $TempDir "zig-toolchain" + $ZigZip = Join-Path $TempDir "zig.zip" + $ZsSrc = Join-Path $TempDir "zsmooth-src" + Remove-Item $ZigDir, $ZsSrc -Recurse -Force -ErrorAction SilentlyContinue + + Download-File -Url "https://ziglang.org/download/$ZigVersion/zig-x86_64-windows-$ZigVersion.zip" -OutFile $ZigZip + Expand-Archive -Path $ZigZip -DestinationPath $ZigDir -Force + $ZigExe = (Get-ChildItem -Path $ZigDir -Recurse -Filter "zig.exe" | Select-Object -First 1).FullName + if (-not $ZigExe) { throw "zig.exe not found in the toolchain archive" } + + # No `2>&1` on a native command: this script runs with + # $ErrorActionPreference = "Stop", and in PowerShell 5.1 redirecting a + # native executable's stderr wraps every line in a NativeCommandError — + # so git's ordinary "Cloning into ..." progress becomes a terminating + # error and the build "fails" having worked. Let git write where it + # likes and judge it by its exit code. + & git -c advice.detachedHead=false clone --quiet --depth 1 --branch $ZsmoothVersion https://github.com/adworacz/zsmooth.git $ZsSrc + if ($LASTEXITCODE -ne 0) { throw "git clone of zsmooth failed (exit $LASTEXITCODE)" } + + Push-Location $ZsSrc + try { + # -Dcpu=x86_64_v2 is SSE4.2/POPCNT: everything from Nehalem (2009) + # on, which covers both CPUs in issue #82. Plain `x86_64` would add + # pre-2009 chips at roughly half the CCD/Cnr4 throughput again. + & $ZigExe build -Doptimize=ReleaseFast -Dtarget=x86_64-windows-gnu -Dcpu=x86_64_v2 + if ($LASTEXITCODE -ne 0) { throw "zig build failed (exit $LASTEXITCODE)" } + } finally { + Pop-Location + } + + $Built = Get-ChildItem -Path (Join-Path $ZsSrc "zig-out") -Recurse -Filter "zsmooth.dll" | Select-Object -First 1 + if (-not $Built) { throw "zig build produced no zsmooth.dll" } + Copy-Item $Built.FullName $V2Path -Force + Remove-Item $ZigZip -Force -ErrorAction SilentlyContinue + Remove-Item $ZigDir, $ZsSrc -Recurse -Force -ErrorAction SilentlyContinue + Write-Host " Built: zsmooth-x86_64_v2.dll" -ForegroundColor Gray + } catch { + Write-Host " Failed: $_" -ForegroundColor Red + } +} else { + Write-Host " zsmooth-x86_64_v2.dll already installed" -ForegroundColor Gray +} + +# A missing build here is not a warning to scroll past: without the AVX2 one +# every modern machine loses the pass, and without the portable one issue #82 +# comes straight back. deps-expected-plugins.json also covers both, so the +# packaging step would fail — this just fails nearer the cause. +$MissingZsmooth = @(@($HaswellPath, $V2Path) | Where-Object { -not (Test-Path $_) }) +if ($MissingZsmooth.Count -gt 0) { + throw "zsmooth build(s) missing: $(($MissingZsmooth | Split-Path -Leaf) -join ', ')" +} +Write-Host " zsmooth: both CPU builds present" -ForegroundColor Green + # ============================================================================= # 4b. FFTW Library (required by DFTTest) # ============================================================================= diff --git a/Scripts/package-deps-linux.sh b/Scripts/package-deps-linux.sh index 3fc0ff2..2151e84 100755 --- a/Scripts/package-deps-linux.sh +++ b/Scripts/package-deps-linux.sh @@ -138,11 +138,17 @@ EOF echo " Verifying required plugins for $ARCH_NAME..." local MANIFEST="$PROJECT_ROOT/Scripts/deps-expected-plugins.json" local MISSING - MISSING=$(python3 - "$MANIFEST" "linux-$ARCH_NAME" "$PACKAGE_DIR/vapoursynth/plugins" <<'PY' + MISSING=$(python3 - "$MANIFEST" "linux-$ARCH_NAME" "$PACKAGE_DIR/vapoursynth/plugins" "$PACKAGE_DIR" <<'PY' import json, os, sys -manifest, key, plugin_dir = sys.argv[1], sys.argv[2], sys.argv[3] +manifest, key, plugin_dir, bundle_root = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4] expected = json.load(open(manifest)).get(key, []) -print("\n".join(f for f in expected if not os.path.isfile(os.path.join(plugin_dir, f)))) +# An entry with a "/" is bundle-root-relative, not a plugin-directory filename: +# zsmooth ships one build per CPU baseline OUTSIDE the autoload directory (both +# register the same namespace, so the worker loads exactly one by path), and a +# guard that only looked in plugins/ would stop covering it. +def target(f): + return os.path.join(bundle_root, *f.split("/")) if "/" in f else os.path.join(plugin_dir, f) +print("\n".join(f for f in expected if not os.path.isfile(target(f)))) PY ) if [ -n "$MISSING" ]; then diff --git a/Scripts/package-deps-macos.sh b/Scripts/package-deps-macos.sh index 37bd0d5..6ed9c78 100755 --- a/Scripts/package-deps-macos.sh +++ b/Scripts/package-deps-macos.sh @@ -145,11 +145,17 @@ EOF echo " Verifying required plugins for $ARCH_NAME..." local MANIFEST="$PROJECT_ROOT/Scripts/deps-expected-plugins.json" local MISSING - MISSING=$(python3 - "$MANIFEST" "macos-$ARCH_NAME" "$PACKAGE_DIR/vapoursynth/plugins" <<'PY' + MISSING=$(python3 - "$MANIFEST" "macos-$ARCH_NAME" "$PACKAGE_DIR/vapoursynth/plugins" "$PACKAGE_DIR" <<'PY' import json, os, sys -manifest, key, plugin_dir = sys.argv[1], sys.argv[2], sys.argv[3] +manifest, key, plugin_dir, bundle_root = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4] expected = json.load(open(manifest)).get(key, []) -print("\n".join(f for f in expected if not os.path.isfile(os.path.join(plugin_dir, f)))) +# An entry with a "/" is bundle-root-relative, not a plugin-directory filename: +# zsmooth ships one build per CPU baseline OUTSIDE the autoload directory (both +# register the same namespace, so the worker loads exactly one by path), and a +# guard that only looked in plugins/ would stop covering it. +def target(f): + return os.path.join(bundle_root, *f.split("/")) if "/" in f else os.path.join(plugin_dir, f) +print("\n".join(f for f in expected if not os.path.isfile(target(f)))) PY ) if [ -n "$MISSING" ]; then diff --git a/Scripts/package-deps-windows.ps1 b/Scripts/package-deps-windows.ps1 index c8750d8..c91ec44 100644 --- a/Scripts/package-deps-windows.ps1 +++ b/Scripts/package-deps-windows.ps1 @@ -152,7 +152,14 @@ Write-Host "[4b/5] Verifying required plugins..." -ForegroundColor Yellow $ManifestPath = Join-Path $ProjectRoot "Scripts\deps-expected-plugins.json" $ExpectedPlugins = (Get-Content $ManifestPath -Raw | ConvertFrom-Json)."windows-x64" $StagedPluginDir = Join-Path "$PackageDir\vapoursynth" "vs-plugins" -$MissingPlugins = @($ExpectedPlugins | Where-Object { -not (Test-Path (Join-Path $StagedPluginDir $_)) }) +# An entry with a '/' is bundle-root-relative, not a plugin-directory filename. +# zsmooth ships one build per CPU baseline outside the autoload directory (both +# register the same namespace, so the worker loads exactly one by path), and a +# guard that only ever looked in vs-plugins would stop covering it. +$MissingPlugins = @($ExpectedPlugins | Where-Object { + $Target = if ($_ -match '/') { Join-Path $PackageDir ($_ -replace '/', '\') } else { Join-Path $StagedPluginDir $_ } + -not (Test-Path $Target) +}) if ($MissingPlugins.Count -gt 0) { Write-Host "ERROR: bundle is missing $($MissingPlugins.Count) required plugin(s):" -ForegroundColor Red $MissingPlugins | ForEach-Object { Write-Host " - $_" -ForegroundColor Red } diff --git a/app/assets/deps-version.json b/app/assets/deps-version.json index 6166813..4477365 100644 --- a/app/assets/deps-version.json +++ b/app/assets/deps-version.json @@ -1,6 +1,6 @@ { - "version": "1.9.0", - "releaseTag": "deps-v1.9.0", - "releaseDate": "2026-08-15", + "version": "1.10.0", + "releaseTag": "deps-v1.10.0", + "releaseDate": "2026-08-28", "githubRepo": "StuartCameronCode/VapourBox" } diff --git a/app/test/attribution_test.dart b/app/test/attribution_test.dart index fea86c2..738d61b 100644 --- a/app/test/attribution_test.dart +++ b/app/test/attribution_test.dart @@ -92,10 +92,20 @@ const _pluginToNotice = { const _mustNotAppear = ['ffms2', 'BestSource']; String _stem(String filename) { - var s = filename.toLowerCase(); + // A manifest entry may be a bundle-relative path rather than a bare filename + // (zsmooth ships outside the autoload directory), and may carry a CPU-target + // suffix because it ships once per baseline. Credit is owed to the project, + // not to each build of it, so both are normalised away. + var s = filename.toLowerCase().split('/').last; final dot = s.lastIndexOf('.'); if (dot > 0) s = s.substring(0, dot); if (s.startsWith('lib') && s.length > 3) s = s.substring(3); + for (final variant in const ['-haswell', '-x86_64_v2']) { + if (s.endsWith(variant)) { + s = s.substring(0, s.length - variant.length); + break; + } + } return s; } diff --git a/app/test/ffmpeg_version_pin_test.dart b/app/test/ffmpeg_version_pin_test.dart new file mode 100644 index 0000000..d6688a9 --- /dev/null +++ b/app/test/ffmpeg_version_pin_test.dart @@ -0,0 +1,130 @@ +// The three download-deps-* scripts must pin the SAME FFmpeg series. +// +// A per-OS FFmpeg is a per-OS encoder. Everything in `pipeline_executor.rs` +// — the accumulated `-vf` chain, the colour metadata flags, `-ss`/`-frames:v` +// trimming, the hardware-encoder options — is interpreted by whichever binary +// the bundle happens to carry, so a version skew means the same job produces +// subtly different output depending on where it ran. That is the same class of +// bug the fmtconv and zsmooth version pins exist to prevent, and it is harder +// to spot because nothing fails. +// +// This is not hypothetical. Measured 2026-08-31, before this test existed: +// +// Windows master N-125978 (post-9.0) BtbN `master-latest`, UNPINNED +// macOS x64 9.0.1 evermeet `getrelease`, UNPINNED +// macOS arm64 9.0.1 martin-riedl `latest`, UNPINNED +// Linux 7.1 BtbN `n7.1`, pinned — two majors behind +// +// Three of the four floated on "latest" URLs and the fourth was pinned to a +// series that upstream then garbage-collected, which 404'd the Linux deps build +// outright. Both failure modes are caught here: an unpinned URL has no series +// to find, and a pin that drifts stops matching its siblings. +import 'dart:io'; + +import 'package:flutter_test/flutter_test.dart'; +import 'package:path/path.dart' as p; + +String _repoRoot() { + var dir = Directory.current; + while (!File(p.join(dir.path, 'CLAUDE.md')).existsSync()) { + final parent = dir.parent; + if (parent.path == dir.path) { + throw StateError('could not find the repo root from ${Directory.current.path}'); + } + dir = parent; + } + return dir.path; +} + +/// The series each script pins, by the assignment it declares. +/// +/// Deliberately reads the *variable*, not the URL: the URLs differ per platform +/// and per host (BtbN asset names, evermeet version paths, a martin-riedl +/// redirect), and the variable is the single thing they are all built from. +({String script, String series}) _pin(String root, String file, RegExp pattern) { + final text = File(p.join(root, 'Scripts', file)).readAsStringSync(); + final match = pattern.firstMatch(text); + expect( + match, + isNotNull, + reason: 'Scripts/$file declares no FFmpeg series pin matching ' + '${pattern.pattern}. If the URL was changed to an unpinned "latest" ' + 'build, pin it instead — that is how the platforms drifted apart before.', + ); + return (script: file, series: match!.group(1)!); +} + +void main() { + final root = _repoRoot(); + + group('FFmpeg version pins', () { + test('all three download scripts pin the same series', () { + final pins = <({String script, String series})>[ + _pin(root, 'download-deps-linux.sh', RegExp(r'^FFMPEG_SERIES="([^"]+)"', multiLine: true)), + _pin(root, 'download-deps-macos.sh', RegExp(r'^FFMPEG_SERIES="([^"]+)"', multiLine: true)), + _pin(root, 'download-deps-windows.ps1', RegExp(r'^\$FFmpegSeries = "([^"]+)"', multiLine: true)), + ]; + + final distinct = pins.map((e) => e.series).toSet(); + expect( + distinct, + hasLength(1), + reason: 'the deps scripts pin different FFmpeg series: ' + '${pins.map((e) => "${e.script}=${e.series}").join(", ")}. ' + 'Bump them together or the same job encodes differently per OS.', + ); + }); + + test('the series is a bare major.minor, not a moving target', () { + // "master", "latest" or a full version would each defeat the point: the + // first two are unpinned, and a full version cannot be held because BtbN + // garbage-collects old series from its rolling tag. + final linux = _pin( + root, + 'download-deps-linux.sh', + RegExp(r'^FFMPEG_SERIES="([^"]+)"', multiLine: true), + ); + expect( + linux.series, + matches(RegExp(r'^\d+\.\d+$')), + reason: 'FFMPEG_SERIES should be a major.minor like "9.0", got ' + '"${linux.series}"', + ); + }); + + test("macOS's exact x64 pin lies within the shared series", () { + // evermeet publishes per-version URLs and retains them, so that arch pins + // a full version. It still has to be inside the series everyone else is + // on, or macOS x64 silently diverges from macOS arm64. + final text = File(p.join(root, 'Scripts', 'download-deps-macos.sh')).readAsStringSync(); + final series = RegExp(r'^FFMPEG_SERIES="([^"]+)"', multiLine: true).firstMatch(text)!.group(1)!; + final exact = RegExp(r'^FFMPEG_MACOS_X64_VERSION="([^"]+)"', multiLine: true).firstMatch(text); + expect(exact, isNotNull, reason: 'download-deps-macos.sh no longer pins an exact x64 version'); + expect( + exact!.group(1)!, + startsWith('$series.'), + reason: 'the pinned evermeet build ${exact.group(1)} is not in the ' + '$series series the other platforms use', + ); + }); + + test('every script verifies what it actually installed', () { + // The pins above are intent; these assertions are what makes upstream + // moving the URL a red build instead of silent skew. Losing them would + // leave the pins as comments. + for (final entry in { + 'download-deps-linux.sh': 'assert_ffmpeg_series', + 'download-deps-macos.sh': 'assert_ffmpeg_series', + 'download-deps-windows.ps1': r'$FFmpegSeries', + }.entries) { + final text = File(p.join(root, 'Scripts', entry.key)).readAsStringSync(); + expect( + text.contains(entry.value), + isTrue, + reason: '${entry.key} no longer checks the installed FFmpeg against ' + 'its pin, so a silent upstream change would go unnoticed', + ); + } + }); + }); +} diff --git a/app/test/support/worker_harness.dart b/app/test/support/worker_harness.dart index 3343f14..88f5d28 100644 --- a/app/test/support/worker_harness.dart +++ b/app/test/support/worker_harness.dart @@ -238,16 +238,48 @@ class WorkerHarness { /// start because a probe misbehaved would be worse than one that says /// "unknown". static Future describeCpu() async { + final probe = await _probeCpu(); + if (probe == null) return 'unknown'; + final features = (probe['features'] as List).cast(); + return '${probe['arch']} [${features.isEmpty ? 'none detected' : features.join(' ')}]'; + } + + /// The instruction set extensions the worker process can actually execute, + /// as a set. Empty when the probe could not run at all. + /// + /// A test that needs to know whether a CPU-specific binary is safe to load + /// must ask this rather than parse `/proc/cpuinfo` or `sysctl` itself: a + /// second implementation of the same decision is how the interface and the + /// pipeline come to disagree, and the worker's answer is the one that governs + /// what the pipeline loads. Used by the zsmooth build check in + /// `vapoursynth_integration_test`. + static Future> cpuFeatures() async { + final probe = await _probeCpu(); + if (probe == null) return {}; + return (probe['features'] as List).cast().toSet(); + } + + /// True when the worker reports an x86 architecture (so the x86-specific + /// zsmooth builds are the relevant ones). + static Future isX86() async { + final probe = await _probeCpu(); + final arch = probe?['arch'] as String?; + return arch != null && (arch.startsWith('x86') || arch == 'amd64'); + } + + /// Never throws: this is diagnostic, and it is asked for before + /// `ensureReady()` in at least one test, so it resolves the worker itself. + static Future?> _probeCpu() async { try { - final result = await Process.run(_workerPath!, ['--probe-cpu']); - if (result.exitCode != 0) return 'unknown (probe exit ${result.exitCode})'; - final json = jsonDecode(result.stdout as String) as Map; - final features = (json['features'] as List).cast(); - return '${json['arch']} [${features.isEmpty ? 'none detected' : features.join(' ')}]'; - } catch (e) { + final worker = _workerPath ?? _resolveWorker(); + if (worker == null) return null; + final result = await Process.run(worker, ['--probe-cpu']); + if (result.exitCode != 0) return null; + return jsonDecode(result.stdout as String) as Map; + } catch (_) { // An older worker predates --probe-cpu and exits non-zero on the unknown - // flag; that is reported above rather than thrown. - return 'unknown ($e)'; + // flag, which lands here as "unknown" rather than as a failure. + return null; } } diff --git a/app/test/vapoursynth_integration_test.dart b/app/test/vapoursynth_integration_test.dart index a34666a..a452920 100644 --- a/app/test/vapoursynth_integration_test.dart +++ b/app/test/vapoursynth_integration_test.dart @@ -10,6 +10,8 @@ import 'dart:io'; import 'package:flutter_test/flutter_test.dart'; import 'package:path/path.dart' as path; +import 'support/worker_harness.dart'; + void main() { late String depsDir; late String vspipePath; @@ -117,10 +119,13 @@ core = vs.core # incomplete or stale deps install looks like from the user's side. `zsmooth` # (Chroma Denoise / CCD) was added to the bundle after this list was written and # went uncovered, so a bundle without it passed the suite and failed the filter. +# It is no longer in THIS list because it is deliberately not autoloaded — it +# ships once per CPU baseline and the worker loads one by path. The test below +# covers it. required = ['std', 'resize', 'mv', 'znedi3', 'eedi3m', 'fmtc', 'dfttest', 'neo_f3kdb', 'cas', 'dctf', 'deblock', 'rgvs', 'ctmf', 'warp', 'misc', 'grain', 'tcanny', - 'zsmooth', 'descratch', 'vivtc', 'ttmpsm', 'tmedian', + 'descratch', 'vivtc', 'ttmpsm', 'tmedian', 'fft3dfilter', 'flux', 'bifrost', 'retinex', 'bwdif', 'fb', 'removedirt', 'dedot', 'lghost'] @@ -167,6 +172,77 @@ else: expect(result.exitCode, 0, reason: 'stderr: ${result.stderr}'); expect(result.stdout.toString(), contains('All plugins loaded')); }); + + test('the zsmooth build for this CPU loads and runs', () async { + // zsmooth is deliberately OUTSIDE the autoload directory: upstream builds + // it for an AVX2 baseline with no runtime dispatch, so that binary dies + // with an illegal instruction on a pre-2013 CPU the instant a filter runs + // (issue #82). The bundle ships one build per CPU baseline and the worker + // loads exactly one by path. + // + // The Rust side (test_154) proves the generated script asks for the right + // file on any hardware; only running it proves the file is there and + // executes — and only ever for the CPU that ran it, which is why the + // choice itself is asserted in Rust and not here. + final zsmoothDir = Directory(path.join(depsDir, 'vapoursynth', 'zsmooth')); + expect( + zsmoothDir.existsSync(), + isTrue, + reason: 'deps bundle has no vapoursynth/zsmooth directory: ' + '${zsmoothDir.path} — a bundle older than deps 1.10.0, or a failed ' + 'zsmooth build', + ); + + // Asked of the worker, never derived here: loading the AVX2 build on a + // CPU without AVX2 is the crash this whole split exists to prevent, so a + // second-guessed answer is worse than none. + final features = await WorkerHarness.cpuFeatures(); + final isX86 = await WorkerHarness.isX86(); + final ext = Platform.isWindows + ? 'dll' + : Platform.isMacOS + ? 'dylib' + : 'so'; + final prefix = Platform.isWindows ? '' : 'lib'; + // Same preference order as DependencyLocator::zsmooth_candidates. With no + // probe (worker not built), only the portable build is considered — it + // runs everywhere, so the check degrades rather than risking the crash. + final candidates = [ + if (isX86 && features.contains('avx2')) '${prefix}zsmooth-haswell.$ext', + if (isX86) '${prefix}zsmooth-x86_64_v2.$ext', + '${prefix}zsmooth.$ext', + ]; + final chosen = candidates + .map((f) => File(path.join(zsmoothDir.path, f))) + .where((f) => f.existsSync()) + .firstOrNull; + expect( + chosen, + isNotNull, + reason: 'no zsmooth build this CPU can run in ${zsmoothDir.path}: ' + 'looked for $candidates, found ' + '${zsmoothDir.listSync().map((e) => path.basename(e.path)).toList()} ' + '(CPU features: $features)', + ); + + final script = ''' +import vapoursynth as vs +core = vs.core +core.std.LoadPlugin(r"${chosen!.path}") +clip = core.std.BlankClip(width=160, height=120, format=vs.YUV420P8, length=2) +# CCD is what the Chroma Denoise pass is made of and what #82 was reported +# against. Constructing the node is not enough: the fault is in the kernel, so a +# frame has to be rendered. +clip = core.zsmooth.CCD(clip, threshold=4, scale=1) +clip.get_frame(0) +print("zsmooth OK") +clip.set_output() +'''; + + final result = await _runVspipeScript(vspipePath, script, depsDir: depsDir); + expect(result.exitCode, 0, reason: 'stderr: ${result.stderr}'); + expect(result.stdout.toString(), contains('zsmooth OK')); + }); }); group('pipe_source Tests', () { diff --git a/worker/src/dependency_locator.rs b/worker/src/dependency_locator.rs index 23de2eb..4ad660d 100644 --- a/worker/src/dependency_locator.rs +++ b/worker/src/dependency_locator.rs @@ -574,6 +574,83 @@ impl DependencyLocator { } } + /// Directory holding the zsmooth builds, which are deliberately NOT in the + /// autoload directory. + /// + /// Upstream publishes zsmooth only for `haswell` (an AVX2 baseline, no + /// runtime dispatch) and `znver4`, so the bundled binary hard-crashes with + /// an illegal instruction on any pre-2013 x86 CPU the moment a zsmooth + /// filter runs — issue #82, on a Celeron J4105 and a Core i7 870. A second + /// `x86_64_v2` build covers those machines, and it cannot simply sit beside + /// the first: both register the namespace `zsmooth`, so whichever autoloads + /// second is rejected. Exactly one is therefore loaded explicitly, by + /// [`Self::zsmooth_plugin`], from here. + pub fn zsmooth_dir(&self) -> PathBuf { + self.platform_dir().join("vapoursynth").join("zsmooth") + } + + /// The zsmooth build this machine can actually execute, or `None` when the + /// bundle predates the split. + /// + /// `None` is not a failure: deps bundles up to 1.9.0 ship a single zsmooth + /// inside the autoload directory, and on those the generated script must + /// emit no `LoadPlugin` at all and let autoload do what it has always done. + /// That keeps a newer worker working against an older bundle, which matters + /// because the app can be upgraded before the deps download completes. + /// + /// The choice is made here, in the worker, rather than in the script for the + /// same reason as `script_generator::ctmf_opt`: it is a property of the + /// machine, and no preceding pass can change the answer. + pub fn zsmooth_plugin(&self) -> Option { + let dir = self.zsmooth_dir(); + for name in Self::zsmooth_candidates() { + let path = dir.join(name); + if path.exists() { + return Some(path); + } + } + None + } + + /// Candidate filenames in preference order: the fastest build this CPU can + /// run first, then the one that runs anywhere. + /// + /// Never fall back the other way. Choosing haswell where AVX2 is absent is + /// not a slow job, it is a dead one. + fn zsmooth_candidates() -> Vec { + let ext = if cfg!(target_os = "windows") { + "dll" + } else if cfg!(target_os = "macos") { + "dylib" + } else { + "so" + }; + let prefix = if cfg!(target_os = "windows") { "" } else { "lib" }; + + let mut variants: Vec<&str> = Vec::new(); + #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] + { + if std::is_x86_feature_detected!("avx2") { + variants.push("haswell"); + } + variants.push("x86_64_v2"); + } + // A single build on non-x86: aarch64 has one NEON baseline and upstream + // publishes no variants for it. + variants.push(""); + + variants + .into_iter() + .map(|v| { + if v.is_empty() { + format!("{}zsmooth.{}", prefix, ext) + } else { + format!("{}zsmooth-{}.{}", prefix, v, ext) + } + }) + .collect() + } + /// Get the NNEDI3CL weights path. pub fn nnedi3cl_weights_path(&self) -> PathBuf { #[cfg(target_os = "windows")] @@ -850,6 +927,62 @@ impl DependencyLocator { mod tests { use super::*; + #[test] + fn zsmooth_never_offers_a_build_this_cpu_cannot_run() { + // The whole point of the split: naming the AVX2 build on a machine + // without AVX2 is not a slow job, it is `0xC000001D` and a dead one + // (issue #82, on a Celeron J4105 and a Core i7 870). This assertion + // runs on every platform whatever hardware CI draws, which is the + // durable half of the guard — the end-to-end test can only confirm + // opportunistically, and GitHub's fleet is a mixed draw. + let candidates = DependencyLocator::zsmooth_candidates(); + assert!(!candidates.is_empty(), "there must always be a candidate"); + + let haswell = candidates.iter().any(|c| c.contains("haswell")); + #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] + { + assert_eq!( + haswell, + std::is_x86_feature_detected!("avx2"), + "the haswell build may be offered only where AVX2 exists" + ); + // The fallback has to be present on x86 regardless, or a non-AVX2 + // machine has nothing to load. + assert!( + candidates.iter().any(|c| c.contains("x86_64_v2")), + "x86 must always offer the x86_64_v2 fallback: {candidates:?}" + ); + if haswell { + assert!( + candidates[0].contains("haswell"), + "where AVX2 exists the fastest build must be preferred: {candidates:?}" + ); + } + } + #[cfg(not(any(target_arch = "x86", target_arch = "x86_64")))] + { + assert!(!haswell, "no x86 build may be offered off x86: {candidates:?}"); + } + } + + #[test] + fn zsmooth_candidates_use_this_platforms_library_naming() { + // A wrong prefix or extension makes every candidate miss, which + // degrades silently to the autoload path — i.e. to the bug. + let candidates = DependencyLocator::zsmooth_candidates(); + let (prefix, ext) = if cfg!(target_os = "windows") { + ("", ".dll") + } else if cfg!(target_os = "macos") { + ("lib", ".dylib") + } else { + ("lib", ".so") + }; + for c in &candidates { + assert!(c.starts_with(&format!("{prefix}zsmooth")), "bad prefix: {c}"); + assert!(c.ends_with(ext), "bad extension: {c}"); + } + } + #[test] fn test_platform_suffix() { let locator = DependencyLocator { diff --git a/worker/src/main.rs b/worker/src/main.rs index 13c7719..5a73020 100644 --- a/worker/src/main.rs +++ b/worker/src/main.rs @@ -523,7 +523,8 @@ fn run_worker( let knlm_available = deps.as_ref().map(|d| d.knlm_available()).unwrap_or(true); let script_generator = ScriptGenerator::new()? .with_opencl_available(opencl_available) - .with_knlm_available(knlm_available); + .with_knlm_available(knlm_available) + .with_zsmooth_plugin(deps.as_ref().and_then(|d| d.zsmooth_plugin())); let script_path = script_generator .generate(&job) .with_context(|| "Failed to generate VapourSynth script")?; diff --git a/worker/src/pipeline_executor.rs b/worker/src/pipeline_executor.rs index cc9e7c0..ffb54cd 100644 --- a/worker/src/pipeline_executor.rs +++ b/worker/src/pipeline_executor.rs @@ -52,9 +52,13 @@ fn progress_end_is_ours(current_frame: i32) -> bool { current_frame > 0 } -/// Format an exit status for error messages, including signal info on Unix. +/// Format an exit status for error messages, including signal info on Unix and +/// the NTSTATUS meaning of a Windows crash code. fn format_exit_status(status: &std::process::ExitStatus) -> String { if let Some(code) = status.code() { + if let Some(explanation) = windows_crash_explanation(code) { + return format!("exit code {} — {}", code, explanation); + } return format!("exit code {}", code); } #[cfg(unix)] @@ -76,6 +80,34 @@ fn format_exit_status(status: &std::process::ExitStatus) -> String { "unknown status".to_string() } +/// Plain-language meaning of a Windows NTSTATUS exit code, if it is one. +/// +/// A native crash inside a plugin kills vspipe before it writes the Y4M header, +/// so the *encoder* ffmpeg is what reports a failure ("Header too large") and +/// the only trace of the real cause is this number. Reported as a bare +/// `exit code -1073741795`, it names nothing at all: issue #82 took a round trip +/// with the reporter to establish that it meant a plugin needing AVX2 on a CPU +/// without it, and issue #83's CTMF crash reads almost identically. +/// +/// Deliberately not `#[cfg(windows)]`: these are Windows codes, but a Unix +/// process exit code is 0-255 so there is nothing to collide with, and decoding +/// them everywhere keeps the mapping testable on every platform. +fn windows_crash_explanation(code: i32) -> Option<&'static str> { + match code as u32 { + 0xC000_0005 => Some( + "access violation inside vspipe or a plugin (a native crash, not a script error)", + ), + 0xC000_001D => Some( + "illegal instruction: a plugin used a CPU feature this machine does not have. \ + Run the worker with --probe-cpu to see what it has", + ), + 0xC000_00FD => Some("stack overflow inside vspipe or a plugin"), + 0xC000_0409 => Some("fail-fast: a plugin detected memory corruption and aborted"), + 0xC000_0094 => Some("integer divide by zero inside vspipe or a plugin"), + _ => None, + } +} + /// True if the process was terminated by SIGPIPE (Unix only; always false /// elsewhere). A vspipe SIGPIPE means the downstream consumer (ffmpeg) closed /// the pipe — usually because ffmpeg itself failed, so the ffmpeg error is the @@ -1148,7 +1180,8 @@ impl PipelineExecutor { // FPS as rational let script_generator = ScriptGenerator::new()? .with_opencl_available(self.deps.opencl_available()) - .with_knlm_available(self.deps.knlm_available()); + .with_knlm_available(self.deps.knlm_available()) + .with_zsmooth_plugin(self.deps.zsmooth_plugin()); let (fps_num, fps_den) = script_generator.frame_rate_to_rational(frame_rate); let preview_params = PreviewParams { @@ -1351,6 +1384,39 @@ mod tests { use crate::models::{AudioCodec, AudioQuality, ChromaSubsampling, EncodingSettings, QTGMCParameters, VideoCodec}; use uuid::Uuid; + /// The two codes that have actually cost debugging time must be named in + /// the error, and an ordinary exit code must not be dressed up as a crash. + /// + /// `-1073741795` was all the reporter of issue #82 had to go on, and it took + /// a round trip to establish that it meant "a plugin needs AVX2 and this CPU + /// has none". `-1073741819` is the CTMF crash of issue #83. A tool exiting + /// 1 or 255 is not a fault code and must read as it always did. + #[test] + fn windows_crash_codes_are_named_in_the_error() { + let illegal = windows_crash_explanation(0xC000_001D_u32 as i32) + .expect("0xC000001D must be explained"); + assert!( + illegal.contains("illegal instruction") && illegal.contains("CPU feature"), + "the AVX2 case must say what it means: {illegal}" + ); + assert!( + illegal.contains("--probe-cpu"), + "and how to find out what this machine has: {illegal}" + ); + + let violation = windows_crash_explanation(0xC000_0005_u32 as i32) + .expect("0xC0000005 must be explained"); + assert!(violation.contains("access violation"), "got: {violation}"); + + // Not fault codes: a real exit status from a tool that ran and failed. + for ordinary in [0, 1, 2, 130, 141, 255, -1] { + assert!( + windows_crash_explanation(ordinary).is_none(), + "{ordinary} is an exit code, not a crash" + ); + } + } + /// A leftover `progress=end` must not end a run that has produced no frames. /// /// This is the bug that produced "the UI sits on processing with a spinner diff --git a/worker/src/script_generator.rs b/worker/src/script_generator.rs index d8d68e1..3bcbcea 100644 --- a/worker/src/script_generator.rs +++ b/worker/src/script_generator.rs @@ -69,6 +69,12 @@ pub struct ScriptGenerator { /// `knlm.KNLMeansCL: CL_INVALID_VALUE` / a missing-namespace error. Defaults /// to true; callers set it from `DependencyLocator::knlm_available()`. knlm_available: bool, + /// Absolute path to the zsmooth build this machine can execute, from + /// `DependencyLocator::zsmooth_plugin()`. `None` means the bundle predates + /// the per-CPU split and still autoloads a single zsmooth, so no + /// `LoadPlugin` is emitted — see that method for why that has to keep + /// working. + zsmooth_plugin: Option, } /// Parameters for preview script generation. @@ -95,7 +101,13 @@ impl ScriptGenerator { pub fn new() -> Result { let template = Self::load_template()?; let preview_template = Self::load_preview_template()?; - Ok(Self { template, preview_template, opencl_available: true, knlm_available: true }) + Ok(Self { + template, + preview_template, + opencl_available: true, + knlm_available: true, + zsmooth_plugin: None, + }) } /// Set whether OpenCL is available (probe result from `DependencyLocator`). @@ -113,6 +125,18 @@ impl ScriptGenerator { self } + /// Set the zsmooth build to load explicitly (from + /// `DependencyLocator::zsmooth_plugin()`). + /// + /// Both the encode and the preview path must be given the same value: they + /// are separate scripts, and a preview that loaded a different build than + /// the render would show a different picture than it produced — the same + /// class of split the field-order derivation exists to prevent. + pub fn with_zsmooth_plugin(mut self, plugin: Option) -> Self { + self.zsmooth_plugin = plugin; + self + } + /// Generate a .vpy script file for the given job. pub fn generate(&self, job: &VideoJob) -> Result { let pipeline = job.effective_pipeline(); @@ -136,6 +160,7 @@ impl ScriptGenerator { // Start with preview template and substitute preview-specific params let mut script = self.preview_template.clone(); + script = self.substitute_zsmooth(script); // Pipe source directory (same as main pipeline) let pipe_source_dir = Self::pipe_source_dir().unwrap_or_else(|_| env::temp_dir()); @@ -256,9 +281,30 @@ impl ScriptGenerator { } } + /// Emit (or elide) the explicit `LoadPlugin` for zsmooth. + /// + /// One function for both scripts on purpose: the encode and the preview must + /// load the same build, and doing this twice is how they would drift. + fn substitute_zsmooth(&self, script: String) -> String { + match self.zsmooth_plugin.as_ref() { + Some(path) => { + // The template uses r"..." so backslashes are literal, exactly + // as {{PIPE_SOURCE_DIR}} relies on. + let script = script.replace("{{ZSMOOTH_PLUGIN}}", &path.to_string_lossy()); + script + .replace("{{#LOAD_ZSMOOTH}}\n", "") + .replace("{{/LOAD_ZSMOOTH}}\n", "") + .replace("{{#LOAD_ZSMOOTH}}", "") + .replace("{{/LOAD_ZSMOOTH}}", "") + } + None => remove_block("{{#LOAD_ZSMOOTH}}", "{{/LOAD_ZSMOOTH}}", script), + } + } + /// Substitute parameters in a script string. fn substitute_parameters(&self, template: &str, job: &VideoJob, pipeline: &ProcessingPipeline, _input_path: &str) -> String { let mut script = template.to_string(); + script = self.substitute_zsmooth(script); // Pipe source parameters — FFmpeg decodes, pipes raw frames to VapourSynth via stdin let pipe_source_dir = Self::pipe_source_dir().unwrap_or_else(|_| env::temp_dir()); diff --git a/worker/templates/pipeline_template.vpy b/worker/templates/pipeline_template.vpy index eb1fd4b..17a2b0d 100644 --- a/worker/templates/pipeline_template.vpy +++ b/worker/templates/pipeline_template.vpy @@ -15,6 +15,16 @@ core = vs.core # 1GB default, can be adjusted based on system memory core.max_cache_size = 1024 +{{#LOAD_ZSMOOTH}} +# zsmooth is loaded here rather than autoloaded, because the bundle ships more +# than one build of it and only one may register the namespace. Upstream +# compiles it for an AVX2 baseline with no runtime dispatch, so that build dies +# with an illegal instruction on a pre-2013 CPU (issue #82); the worker picks the +# fastest build this machine can execute and substitutes its path. The block is +# absent on a deps bundle old enough to autoload a single zsmooth. +core.std.LoadPlugin(r"{{ZSMOOTH_PLUGIN}}") +{{/LOAD_ZSMOOTH}} + # Load input video from stdin pipe (FFmpeg decodes → raw frames → VapourSynth) # This eliminates FFMS2 indexing which blocks on large/NAS files. import os diff --git a/worker/templates/preview_template.vpy b/worker/templates/preview_template.vpy index f85d56d..2642ab7 100644 --- a/worker/templates/preview_template.vpy +++ b/worker/templates/preview_template.vpy @@ -14,6 +14,13 @@ core = vs.core # Configure cache size for optimal performance with temporal filters core.max_cache_size = 1024 +{{#LOAD_ZSMOOTH}} +# Same explicit zsmooth load as the encode template, from the same worker-side +# choice — a preview that loaded a different build than the render would show a +# different picture than it produced. See pipeline_template.vpy for why. +core.std.LoadPlugin(r"{{ZSMOOTH_PLUGIN}}") +{{/LOAD_ZSMOOTH}} + # Load raw frames piped from FFmpeg via stdin sys.path.insert(0, r"{{PIPE_SOURCE_DIR}}") from pipe_source import create_pipe_clip diff --git a/worker/tests/filter_integration_test.rs b/worker/tests/filter_integration_test.rs index c149aa5..0bb0ace 100644 --- a/worker/tests/filter_integration_test.rs +++ b/worker/tests/filter_integration_test.rs @@ -2768,7 +2768,10 @@ fn test_77_chroma_denoise_absent_when_disabled() { let mut job = create_base_job("test_77_chroma_denoise_off"); job.processing_pipeline = Some(ProcessingPipeline::default()); let script = std::fs::read_to_string(generator.generate(&job).unwrap()).unwrap(); - assert!(!script.contains("zsmooth"), "a disabled pass must emit nothing"); + assert!( + !script.contains("core.zsmooth."), + "a disabled pass must emit no filter call" + ); // And it sits between noise reduction and dehalo in the pass order. let pipeline = ProcessingPipeline { @@ -5648,3 +5651,120 @@ fn test_153_ctmf_opt_is_a_level_the_cpu_can_actually_run() { assert_eq!(opt, expected, "opt must follow what the CPU actually has"); } } + +/// Generate both scripts with an explicit zsmooth build selected, the way the +/// worker does once a bundle carries the per-CPU split. +fn generate_both_scripts_with_zsmooth( + job: &VideoJob, + plugin: Option, +) -> (String, String) { + let generator = ScriptGenerator::new() + .expect("create generator") + .with_zsmooth_plugin(plugin); + let encode_path = generator.generate(job).expect("generate encode script"); + let encode = std::fs::read_to_string(&encode_path).expect("read encode script"); + + let params = PreviewParams { + width: job.input_width.unwrap_or(720), + height: job.input_height.unwrap_or(480), + pix_fmt: job + .input_pixel_format + .clone() + .unwrap_or_else(|| "yuv420p".to_string()), + num_frames: 11, + fps_num: 30000, + fps_den: 1001, + output_index: 5, + }; + let preview_path = generator + .generate_preview(job, ¶ms) + .expect("generate preview script"); + let preview = std::fs::read_to_string(&preview_path).expect("read preview script"); + + let _ = std::fs::remove_file(&encode_path); + let _ = std::fs::remove_file(&preview_path); + (encode, preview) +} + +fn chroma_denoise_job(id: &str) -> VideoJob { + let mut job = create_base_job(id); + job.qtgmc_parameters.enabled = false; + job.processing_pipeline = Some(ProcessingPipeline { + deinterlace: QTGMCParameters { enabled: false, ..Default::default() }, + chroma_denoise: ChromaDenoiseParameters { + enabled: true, + ..Default::default() + }, + ..ProcessingPipeline::default() + }); + job +} + +#[test] +fn test_154_zsmooth_is_loaded_explicitly_from_the_chosen_build() { + // zsmooth is bundled twice — upstream builds it for an AVX2 baseline with no + // runtime dispatch, so that binary dies with an illegal instruction + // (0xC000001D) on a pre-2013 CPU the instant a filter runs (issue #82). Both + // builds register the namespace `zsmooth`, so neither may autoload and + // exactly one is loaded by path. + // + // Both scripts, because the reporter in #82 hit the preview first. + create_output_dir(); + let job = chroma_denoise_job("test_154_zsmooth_load"); + let chosen = std::path::PathBuf::from("/deps/vapoursynth/zsmooth/zsmooth-x86_64_v2.dll"); + let (encode, preview) = generate_both_scripts_with_zsmooth(&job, Some(chosen.clone())); + + for (name, script) in [("encode", &encode), ("preview", &preview)] { + assert!( + script.contains(&format!("core.std.LoadPlugin(r\"{}\")", chosen.display())), + "{name} script must load the chosen zsmooth build explicitly" + ); + assert!( + script.contains("core.zsmooth.CCD("), + "{name} script should still call the filter" + ); + // An unsubstituted marker is valid Python nowhere and would fail the job + // with a SyntaxError that reads like a template bug. + for leftover in ["{{#LOAD_ZSMOOTH}}", "{{/LOAD_ZSMOOTH}}", "{{ZSMOOTH_PLUGIN}}"] { + assert!( + !script.contains(leftover), + "{name} script left {leftover} unsubstituted" + ); + } + // The load has to precede the first use, or the namespace is missing + // when the filter is constructed. + let load = script.find("core.std.LoadPlugin(r\"").expect("load present"); + let use_ = script.find("core.zsmooth.CCD(").expect("call present"); + assert!(load < use_, "{name} script loads zsmooth after using it"); + } +} + +#[test] +fn test_155_a_bundle_without_the_split_still_autoloads_zsmooth() { + // Deps bundles up to 1.9.0 ship one zsmooth inside the autoload directory. + // A worker that emitted a LoadPlugin for a path those bundles do not have + // would fail every job on them — and the app can be upgraded before the + // deps download finishes, so that window is real. No path selected must + // therefore mean no LoadPlugin, leaving the script as it was before the + // split existed. + create_output_dir(); + let job = chroma_denoise_job("test_155_zsmooth_autoload"); + let (encode, preview) = generate_both_scripts_with_zsmooth(&job, None); + + for (name, script) in [("encode", &encode), ("preview", &preview)] { + assert!( + !script.contains("LoadPlugin"), + "{name} script must not load a plugin the bundle may not have" + ); + assert!( + script.contains("core.zsmooth.CCD("), + "{name} script should still call the filter, via autoload" + ); + for leftover in ["{{#LOAD_ZSMOOTH}}", "{{/LOAD_ZSMOOTH}}", "{{ZSMOOTH_PLUGIN}}"] { + assert!( + !script.contains(leftover), + "{name} script left {leftover} unsubstituted" + ); + } + } +}