Add fvdb.nn.Prune and generative shape completion / VAE examples - #753
Conversation
Add a Prune module to fvdb.nn (the MinkowskiPruning analog): prunes grid topology and its aligned features by a per-voxel boolean mask, built on GridBatch.pruned_grid + JaggedTensor.rmask, differentiable through the kept feature rows. Add two examples mapping one-to-one onto MinkowskiEngine's canonical generative examples, built on the generative transposed convolution semantics from openvdb#726 (ConvolutionPlan.from_grid_batch_transposed with target_grid=None): - examples/shape_completion.py (analog of ME examples/completion.py): sparse encoder-decoder completing a slab-cropped shape, per-level occupancy classifiers, teacher-forced pruning, additive U-Net skips via inject_from. GT level targets come from a conv_grid(2, 2) pyramid queried with coords_in_grid. - examples/shape_vae.py (analog of ME examples/vae.py): shape VAE with a jagged global-pooling latent bottleneck, dense-neck decoder seeded from the latent plus a learned positional embedding, KL + per-level BCE loss, and prior-sampling demo. Both examples train on bundled fvdb-example-data meshes in ~1-2 minutes and run headlessly under tests/test_examples.py. Addresses openvdb#741. Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
notebooks/04_shape_vae.ipynb trains the shape VAE from examples/shape_vae.py and explores its latent space: reconstructions, prior sampling, batched latent interpolation, and an ipywidgets-based interactive latent explorer (blend between shapes, add prior noise) with a static fallback when ipywidgets is unavailable (e.g. in CI). Executed outputs are stored so the notebook renders on GitHub (reconstruction IoU 0.999; tqdm progress spam stripped). Runs under pytest --nbmake in ~87 s on GPU. Adds ipywidgets to the fvdb_learn environment for the interactive path. Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
|
Added |
Switch examples/shape_vae.py and the sampling notebook from the 6 bundled meshes to the 254-model Google Scanned Objects 'Shoe' subset (CC-BY 4.0) now in fvdb-example-data: bump the pinned data revision and add a load_gso_shoes() loader to fvdb.utils.examples. One object category with real intra-class variation (runners, flats, cleats, boots) makes latent interpolation and prior sampling meaningful, mirroring the role of ModelNet40 chairs in MinkowskiEngine's vae.py. Training now runs random minibatches over the pre-voxelized dataset (GridBatch indexing); the ground-truth conv_grid pyramid is built once and sub-indexed per batch. The notebook interpolates a ballet flat into a tall boot and uses a side-on camera. Generative training remains bound by per-iteration generated-topology grid construction (conv_transpose_grid); tracked upstream as openvdb#755. Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
|
Dataset upgrade: the shape VAE (example + notebook) now trains on the 254-model Google Scanned Objects Shoe subset (CC-BY 4.0, added to fvdb-example-data in voxel-foundation/fvdb-example-data#1) instead of the 6 bundled meshes — one category with real intra-class variation, mirroring the role of ModelNet40 chairs in ME's CI note: Perf note: generative training is bound by per-iteration generated-topology grid construction ( |
Replace the matplotlib point scatters in the shape-VAE notebook with lit voxel-cube renders: exposed cube faces from pcu.voxel_grid_geometry (interior faces between adjacent voxels culled via duplicate-centroid removal), per-face Lambertian shading, per-panel auto-fit equal-aspect bounds, and unswapped z-up axes (the GSO scans follow the Gazebo z-up convention; the previous y-up swap was showing the shapes sole-on). Shapes are now clearly readable as shoes - laces, soles, and boot shafts are visible in the dataset, reconstruction, sampling, and interpolation figures. Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
examples/wip/structure_prediction_net.py targeted a removed API surface (VDBTensor, fvnn.ReLU, fvnn.FillFromGrid, per-module conv backends) and undeclared dependencies (torch_cudamanaged, hardcoded local data paths), so it no longer runs. Its intent - hierarchical structure prediction with per-level occupancy classifiers, losses against a coarsened ground-truth pyramid, U-Net skips onto predicted topology, and a dense neck - is now covered by the maintained, CI-tested examples shape_completion.py and shape_vae.py, which use the generative transposed-convolution path from openvdb#726 plus fvdb.nn.Prune. Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
harrism
left a comment
There was a problem hiding this comment.
Analysis by Claude Code, run by @harrism.
Review: fvdb.nn.Prune + shape completion / VAE examples
Two defects below. The layer's core logic is sound — gradient flow through rmask is autograd-tracked and test_prune_gradient pins the exact expected gradient; all-false and all-true masks are covered by test_prune_forward_all_false_all_true, and pruneGrid preserves grid_count with zero-voxel grids; the voxel-order alignment uses the same pruned_grid + rmask(mask.jdata) idiom already established in ClipGrid.cu. The examples' fvdb API usage checks out against the real signatures, and the encoder/decoder channel pyramid in shape_completion.py has no off-by-one.
I don't think we currently have a prescribed way to approach that (I'm assuming changing topology over time). I'd be happy to include an example based on what you're working on if you'd like to include it, perhaps in |
…ell out VAE Prune.forward now raises ValueError when data or mask is not partitioned per grid like the input grid. The underlying rmask and pruneGrid ops only check flat element counts, so a mismatch previously returned misaligned features silently (and an oversized per-grid mask indexed out of bounds). The _trace_fvdb_nn_forward decorator now uses functools.wraps, so every fvdb.nn module keeps its forward docstring and signature. Prune documents its call arguments in the class docstring so they render with autoclass. The shape VAE notebook title spells out Variational Autoencoder and links the Kingma and Welling paper. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
# Batched leaf-mask topology construction for generated conv grids Fixes #755. ## Summary `conv_grid` / `conv_transpose_grid` with generated targets (and `refined_grid` / `coarsened_grid` / the stride-1 dilate-pad paths generally) built their output **one batch member at a time**: each member paid a full NanoVDB `RefineGrid`/`CoarsenGrid`/`DilateGrid`/`PadGrid` build (3+ stream synchronizations each, a host-side speculative root refinement with a pageable D2H readback, a `GridHandle` constructor with 2 blocking memcpys + raw `cudaMalloc`/`cudaFree`), plus a host-built proxy grid + H2D per **empty** member, and finally `nanovdb::cuda::mergeGridHandles` added another synchronization **per grid**. That is ~1.5 ms of fixed overhead x B members x 21 plans per training iteration in generative workloads where topology changes every step (#741 / #753) — construction cost scaled linearly in batch size and left the GPU at ~35% utilization. This PR batches the leaf-mask topology machinery across the whole grid batch (extending #712's leaf-mask direction — no return to coordinate staging), in four self-contained commits: ### 1. Batched topology builder + refine/coarsen passes New `src/fvdb/detail/utils/nanovdb/BatchedTopologyBuilder.cuh`: one batched pass builds ALL grids at once. Per pass: - one emission kernel over all source leaves of all members produces candidate output leaves as (root-tile sort key, in-tile node key, origin, 512-bit mask) slots, segmented per grid — mask bit math is NanoVDB's own (`RefineLeafMasksFunctor::refineMask`, `CoarsenLeafMasksFunctor::coarsenMask`); - two stable segmented radix sorts put each grid's slots in **canonical NanoVDB node order** (root tiles by the PointsToGrid offset-shifted key, then x-major upper/lower child offsets), so produced grids enumerate voxels identically to every other build path (pinned by elementwise `torch.equal` tests); - head-flag + scan passes dedup nodes and derive per-grid counts and parent linkage; root tiles are derived **on device** from the unique upper keys (the host-side speculative `refineRoot` and its readback disappear); - **one** `cudaStreamSynchronize` reads back per-grid node counts, one buffer is allocated, and batched kernels (transcribed from `tools::cuda::TopologyBuilder`'s functors with `(gridIndex, localIndex)` indexing) write every grid's headers (`mGridIndex=g, mGridCount=B`), nodes, leaf `mOffset`/`mPrefixSum`, and bboxes. Empty members become valid empty grids inline. `fineGridHandleFromCoarseCUDA` / `coarseGridHandleFromFineCUDA` route through it for all batch sizes; multi-pass factors (4, 8) chain passes, mirroring the previous per-pass semantics. Checksums are disabled on the output (matching `ops::contiguousGridHandle` and `mergeGridHandles`). CPU / PrivateUse1 paths and non-power-of-two coordinate fallbacks unchanged; masked subdivision still prunes upstream, then batched-refines. ### 2. Batched box-dilate passes (stride-1 K>1, k3s2) A `BoxDilate` pass computes the Minkowski sum with an axis-aligned unit box via pure per-axis bit shifts (scatter formulation, up to 27 target leaves per source leaf, deduplicated by the same back-end). This covers `DilateGrid`'s 26-neighbor dilation (`[-1,1]^3`) and both `PadGrid` octants (`{-1,0}^3`, `{0,1}^3`), so the stride-1 uniform-K conv/conv-transpose paths and the k3s2 transpose (refine + negative pad) become batched pass sequences. The per-member `perItemGridHandle` drivers are deleted from both conv builders. ### 3. Identity-plan fast path (issue's secondary item) `ConvolutionPlan.from_grid_batch[_transposed](1, 1, g[, g])` short-circuits to the matmul backend without building tensor-valued transform diagnostics (which are trivially exact when source and target share `GridBatchData`), preserving channel-pair validation, backend-name rejection, and the general path for distinct-but-equal-looking grids: ~1.2 ms -> **0.08 ms** per call. ### 4. Grid-construction cheap wins - `makeGridBatchData`: `leafBatchIndices` via one `repeat_interleave` instead of B x `torch::full` + `torch::cat` (B+1 dispatches on every grid construction). - `voxelSizesTensor` / `voxelOriginsTensor`: accessor fill instead of per-element ATen indexing (6 dispatches per grid per call, on every plan construction's transform validation). ## Measurements RTX PRO 6000 Blackwell, synthetic shell batches (~7.8k voxels/member, resolution 64), median of 20 CUDA-event-timed iterations (`src/benchmarks/convolution/benchmark_conv_grid_build.py`): | op | before B=16 | after B=16 | before B=48 | after B=48 | |---|---|---|---|---| | `conv_transpose_grid` k2s2 | 9.0 ms | **0.80 ms** | 25.6 ms | **0.94 ms** | | `conv_grid` k2s2 | 8.7 ms | **0.80 ms** | 23.4 ms | **0.94 ms** | | `conv_grid` k3s1 | 9.8 ms | **0.83 ms** | 25.1 ms | **1.00 ms** | | plan `from_grid_batch(2,2,g)` | 8.9 ms | **1.27 ms** | 24.8 ms | **1.62 ms** | | plan `from_grid_batch_transposed(2,2,g)` | 10.2 ms | **1.94 ms** | 29.4 ms | **3.27 ms** | | 4-level plan-pyramid rebuild (8 plans + 3 conv_grids) | 114 ms | **17.4 ms** | 309 ms | **27.0 ms** | | identity plan `from_grid_batch(1,1,g,g)` | ~1.2 ms | **0.08 ms** | — | — | Construction cost is now near-flat in batch size (B=1: within noise of the old single-grid path). The issue's per-iteration plan-construction share (~70 ms of a 165 ms shape-VAE iteration at B=16) drops to single-digit milliseconds. Follow-ups (out of scope, same back-end): an ijk emission front-end to resolve `BuildGridFromIjk.cu`'s per-member FIXME (`from_ijk`/`from_points`/shifted-geometry fallbacks), and `dilated_grid`/`BuildPaddedGrid`'s standalone per-member loops. ## Test plan - New `tests/unit/test_batched_topology_builder.py` (18 tests): elementwise (`torch.equal` on `ijk.jdata`, `num_voxels`, per-member bboxes) equivalence against `from_ijk`-built expected topologies — pinning canonical node order — plus per-member coordinate-set equality against the CPU paths, across: mixed member sizes, empty members (first/middle/last/all), coordinates straddling +-4096 root-tile boundaries and negative octants (where the sort-key and stored `Tile::key` encodings order differently), single-grid and 16-grid batches, factors 2 and 4, masked refine, `conv_grid`/`conv_transpose_grid` K in {2,3,4,5} at stride 1, k3s2 transpose, k2s2 vs per-member, and a refine->coarsen round trip. - Existing suites, all green (724 passed, 1 skipped): `pytest unit/test_conv_semantics_integration.py unit/test_conv_default.py unit/test_conv_transpose_default.py unit/test_batching.py unit/test_basic_ops.py unit/test_sliced_batch.py unit/test_conv_semantics.py unit/test_conv_ground_truth.py unit/test_nn_modules.py` — includes the elementwise ijk order pins, sliced-view coverage, resource-stats path pinning, and the matmul/identity plan contract tests. - Benchmark: `python src/benchmarks/convolution/benchmark_conv_grid_build.py` (numbers above; `--gso` runs the issue's verbatim GSO repro). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Jonathan Swartz <jonathan@jswartz.info> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
pruned_grid built each batch member separately: a host sync on the mask sum, a nanovdb PruneGrid build, and a final mergeGridHandles, at about 2 ms per grid. In the shape VAE decoder that was 28 ms of a 51 ms forward at batch 16, half the decoder. Add a Prune op to the batched topology builder. Slot = source leaf; the emission kernel ANDs the source leaf mask with the per-voxel keep flags, located through the mask's own joffsets plus the leaf's value offset, so sliced batch views work unchanged. The mapping is injective, so it skips the duplicate-mask combine like Refine. Fully pruned leaves become dead slots and fully pruned members become valid empty grids inline. pruned_grid at batch 16 (216k voxels): 30.7 ms -> 0.7 ms, flat across batch sizes. Shape VAE training step: 89 ms -> 67 ms. Tests pin the batched result elementwise against from_ijk of the kept coordinates and against the CPU path over the tricky batches, plus identity/all-false/emptied-member masks and a sliced batch view. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
The notebook's training cell exceeded nbmake's 300 s per-cell default on the CI doc-test runner at 1500 iterations. A learning-rate sweep (3e-4, 1e-3, 3e-3) converged identically, so iteration count is the only lever; 1000 iterations keeps most of the result (loss 0.27 vs 0.25) at two thirds of the time. The notebook reads NUM_ITERATIONS from the example, so one constant covers both. Notebook re-executed with the batched prune build: the training cell now runs in 65 s. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
torch grid_sample's CUDA backward accumulates into a half gradient with atomic adds, so the fp16 reference was noisy and order dependent and the 16-channel dual-grid variant flaked in CI. Run the reference in fp32 for half inputs and cast the result back, matching fVDB's fp32 accumulation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
|
Closing in favor of #785: fvdb's batched topology construction is moving onto a batch-capable upstream NanoVDB The |
|
Reopened, this shouldn't have been closed, we should just remove the batched pruned grid component. |
This reverts commit 1abf3d5. Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
…-generative-examples Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
|
Reopened with the batched |
harrism
left a comment
There was a problem hiding this comment.
Re-review at d3deac2. Both earlier threads are addressed: _check_partitioned_like compares tensor count, row count and joffsets against the grid (int64 on the same device on both sides, so no false positives for grid-built masks or sliced views), covered by test_prune_rejects_mismatched_partitioning; functools.wraps plus the class docstring fix the rendering, pinned by test_prune_forward_keeps_docstring. The batched-prune experiment is reverted net-zero over every file it touched, and the branch merges clean onto main. test_sample.py's fp32 reference is a strengthening, not a loosening.
One gap left, inline; a one-line guard and a test case, then this is good to go. Two non-blocking notes: notebooks/04_shape_vae.ipynb still carries the output from the reverted batched-prune build (1000/1000 [01:05...]) while the PR body now says ~90 s, so the stored cell misrepresents what ships; and the torch.equal on CUDA offsets returns a Python bool, so each Prune.forward adds two device-to-host syncs (eight per decoder step at four levels). pruned_grid already syncs on the mask sum, so it is marginal, but worth knowing given the launch-bound framing.
A nested data tensor (ldim == 2) passed every existing check: num_tensors counts leaves and joffsets holds flat leaf boundaries, so neither sees the outer list structure, and rmask propagates it unchanged. The result paired a pruned grid batch with a JaggedTensor whose outer list count no longer matched grid_count. _check_partitioned_like now requires ldim == 1, and test_prune_rejects_mismatched_partitioning covers the nested case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
…offsets torch.equal on CUDA offsets forces a device-to-host sync, two per Prune.forward. jagged_like and ConvolutionPlan outputs share the grid's offsets tensor object, so an identity match proves the partitioning for free. The equal now runs only for tensors built independently of the grid, which keeps the steady-state decoder path at zero extra syncs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
harrism
left a comment
There was a problem hiding this comment.
Re-verified at 9c5033f. The ldim guard is applied to both data and mask ahead of the count/row/offset checks, and the added nested case in test_prune_rejects_mismatched_partitioning fails without it (it builds the exact same-offsets nested layout and asserts on a real prune call). The identity short-circuit on the offsets object is a better answer to the sync note than leaving it: it holds for jagged_like and for ConvolutionPlan outputs whose input came from the grid, a false positive is impossible, and a miss only costs the torch.equal. All 18 prune tests pass on CPU and CUDA against this head's Python with the unchanged C++ ops.
Two non-blocking leftovers: the committed output in notebooks/04_shape_vae.ipynb (1000/1000 [01:05...]) is still from the reverted batched-prune build while the body says ~90 s, so re-execute or leave knowingly; and one comment wording, inline.
One thing found on the way that is not this PR's: JaggedTensor.from_data_offsets_and_list_ids accepts a permuted single-column jlidx without canonicalizing it, producing a flat tensor whose lshape disagrees with its offsets; fed to Prune that yields pruned_grid.num_voxels == [0, 2] against pruned_data.lshape == [2, 0]. Pre-existing constructor invariant, will file separately.
ConvolutionPlan's matmul backend wraps output as data.jagged_like(out), so plan outputs share the offsets of the input they were built from, not the target grid's directly. The identity short-circuit is unchanged; the comment now states the actual provenance chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
80bff4b to
6d82045
Compare
Addresses #741.
Summary
Scope note: an earlier revision also batched
GridBatch.pruned_gridon CUDA through the leaf-mask builder from #757. That commit is reverted here so this PR is only the layer and the examples; batched pruning follows on the NanoVDBTopologyBuilderengine in #785.fvdb.nn.Prune— theMinkowskiPruninganalog: prunes grid topology and its aligned features by a per-voxel boolean mask (pruned_grid+rmask, differentiable through kept rows). Registered in docs; 13 unit tests.examples/shape_completion.py/examples/shape_vae.py— one-to-one analogs of MinkowskiEngine'scompletion.pyandvae.py, built on the generative transposed convolution from Unify sparse convolution and transpose semantics #726 (from_grid_batch_transposed(..., target_grid=None)) plusPrune.notebooks/04_shape_vae.ipynb— trains the VAE and explores its latent space: reconstructions, prior sampling, latent interpolation (ballet flat → boot), and an optional ipywidgets explorer. Shapes render as lit voxel surfaces.load_gso_shoes()loader and data-revision bump; completion uses the dragon + happy meshes.examples/wip/structure_prediction_net.py(targeted a removed API surface; superseded by the new examples).Design notes
conv_grid(2, 2)pyramid queried withcoords_in_grid— notcoarsened_grid, whose block-centroid lattice differs from whatconv_transpose_grid(2, 2)inverts.keep |= target) and per-level BCE are reproduced verbatim; eval decodes without forcing.Training-loop profile
The first CI run failed on the notebook's training cell exceeding nbmake's 300 s per-cell default, which prompted a profile of the VAE step at batch 16 on an RTX PRO 6000.
pruned_gridis the remaining per-grid loop: 28 ms of a 51 ms decoder forward at batch 16 (one NanoVDBPruneGridper member plusmergeGridHandles). Batching it is deferred to the NanoVDB-based batched topology engine tracked in Batch NanoVDB's TopologyBuilder upstream and make it fvdb's batched topology engine #785, so this PR leavespruned_gridas it is on main; the training step is about 89 ms.NUM_ITERATIONSdrops from 1500 to 1000. Two thirds of the time keeps most of the result (final loss 0.27 vs 0.25), and at ~89 ms per step the notebook cell runs in about 90 s, inside nbmake's 300 s limit. The notebook reads the constant from the example, so one change covers both.Results / verification
test_basic_ops,test_basic_ops_single,test_prune_single_voxel,test_sliced_batch,test_inject,test_nn_modules: 83 passed.clang-formatandblack --line-length 120clean.🤖 Generated with Claude Code