The scan collects every entry before anything else starts:
/// TODO(next): stream entries through a channel instead of collecting,
/// so the UI can render results while the scan runs.
pub fn scan(opts: &ScanOptions, progress: Arc<ScanProgress>) -> Result<Vec<FileEntry>> {
let mut out = Vec::new();Two costs.
Memory. A FileEntry holds a PathBuf plus five small fields. A
home directory of two million files is a few hundred MB held for the
whole scan, and a full-disk scan is worse.
Waiting. The user watches a file counter climb, then nothing until
the walk, the graph pass, classification and hashing have all finished.
The results exist long before they are shown — a safe browser cache
found in the first second is not rendered until the last file is hashed.
Streaming entries through a channel would fix the second and most of the
first. It is a real design change, not a refactor: report::build_with
takes Vec<FileEntry> and makes several passes over it — the graph
needs to see every entry before it can answer anything, and dedup needs
every size before it knows which files to hash. So the honest version is
probably incremental findings — classify and emit as you walk, then
revise verdicts once the graph is complete — rather than a straight
pipe.
Worth writing the approach in a comment before the code. Happy to
discuss shape in this issue first.
crates/diskern-core/src/scanner.rs:62
The scan collects every entry before anything else starts:
Two costs.
Memory. A
FileEntryholds aPathBufplus five small fields. Ahome directory of two million files is a few hundred MB held for the
whole scan, and a full-disk scan is worse.
Waiting. The user watches a file counter climb, then nothing until
the walk, the graph pass, classification and hashing have all finished.
The results exist long before they are shown — a
safebrowser cachefound in the first second is not rendered until the last file is hashed.
Streaming entries through a channel would fix the second and most of the
first. It is a real design change, not a refactor:
report::build_withtakes
Vec<FileEntry>and makes several passes over it — the graphneeds to see every entry before it can answer anything, and dedup needs
every size before it knows which files to hash. So the honest version is
probably incremental findings — classify and emit as you walk, then
revise verdicts once the graph is complete — rather than a straight
pipe.
Worth writing the approach in a comment before the code. Happy to
discuss shape in this issue first.
crates/diskern-core/src/scanner.rs:62