Skip to content

Restore exported location history - #173

Merged
parawanderer merged 5 commits into
parawanderer:mainfrom
hamza-lzr:feat/import-history-102
Sep 12, 2026
Merged

parawanderer merged 5 commits into
parawanderer:mainfrom
hamza-lzr:feat/import-history-102

Conversation

@hamza-lzr

Copy link
Copy Markdown
Contributor

Fixes #102

Summary

  • add stable beacon_id identity and exact restore metadata to Android history exports
  • import Android Export History ZIPs through a validated RFC 4180 parser and one atomic Room merge
  • add the My Devices picker, result/error dialogs, latest-location refresh, and map-change notification
  • reject legacy name-only exports rather than risk attaching reports to the wrong tag

Import behavior

  • existing database rows win on (beacon_id, timestamp)
  • the first duplicate row in an archive wins
  • unknown or removed tags are skipped and counted
  • malformed rows are counted; structurally damaged archives fail before persistence
  • database failures roll back the entire import

Verification

  • ./gradlew.bat testDebugUnitTest
  • targeted managed-device history import tests (10/10)
  • ./gradlew.bat testAll
  • ./gradlew.bat testAllOnDevice (0 failures, 22 intentional skips)
  • python scripts/add_strings.py --check
  • debug APK build

The full suites above passed before the final rebase. After rebasing onto current main, string completeness and diff checks pass; a local JVM rerun could not start because this session no longer has an Android SDK path configured.

@hamza-lzr

Copy link
Copy Markdown
Contributor Author

@parawanderer

parawanderer added a commit that referenced this pull request Sep 2, 2026
Two gaps, both invisible from inside.

**Nothing said the file is agent-owned.** The header names its audience -
"rules for agents working on OpenTagViewer" - and rule 10 has one table
row obliging you to add a rule when a constraint would cost somebody an
afternoon. Neither says the file is maintained by agents, that no human
reviews it, or that keeping it current is part of the work rather than a
favour. An agent picking up work here had to infer all of that, and both
inferences are load-bearing: if nobody is filtering, the agent writing is
the only check, and if the next agent starts from this file then anything
omitted is an afternoon they pay for.

That now sits directly under the title, where it is read before anything
it governs, rather than in a rule two hundred lines down.

**And nothing warned that the register does not travel.** @parawanderer
has not read this file, nor most of docs/, most docstrings, or most
commit messages. Agents wrote them. So an agent reading them cannot tell
house style from its own predecessors' output, and assumes the former.

That showed up this week. A review of #173 was drafted in this file's
voice - argued, at length, with the reasoning as the payload - and its
reader was a first-time contributor who wanted three line numbers and a
fix. Right register, wrong reader.

The loop has nobody in it: agents write the docs, a later agent reads
them as evidence of what the maintainer wants, and writes more of the
same. Nothing in the repository contradicts that, because nothing in the
repository is written by the maintainer.

Rule 16 states the boundary. The reason it argues at length is the same
reason it says not to: the length is for an agent about to undo a
constraint, and a person reading a review is not that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@parawanderer

Copy link
Copy Markdown
Owner

Thanks for listing what you ran locally, and for the edge case handling in the DAO and parser.

CI is approved and green: 656 tests, 0 failures. I checked the report artifact rather than the tick; your 16 new tests ran rather than being skipped.

Three findings. None were caught by CI and none could have been.

1. Peak memory during import

HistoryImporter.java:191 accumulates every row into ReadResult.rows. HistoryImportDao.java:merge then builds a HashSet<Long> of every existing timestamp per beacon, and holds both across the transaction.

Ceiling is about 70,000 rows per tag-year: ~100/day from the Find My network, plus @ubrt's local sightings throttled by LocalFixWorthKeeping to 25 m moved or 15 minutes elapsed.

1 tag, 1 year 5 tags, 1 year
row list ~20 MB ~50 MB
timestamp sets ~4 MB ~10 MB

No android:largeHeap, and Chaquopy's CPython is already resident.

Do not replace the preload with per-row EXISTS. beacon_id is not a left-most prefix of any index on LocationReport; the only index is (hash_id, beacon_id, timestamp). Every check would full-scan. One scan per beacon is correct.

Fix: intern beaconId in HistoryImporter.parseRow with a HashMap<String, String> pool. Currently allocates a fresh 36-char UUID string per row.

An index on (beacon_id, timestamp) removes the preload entirely, but needs a migration, so separate PR.

2. Crash on rotation, and no progress UI

MyDevicesListActivity.java:667 subscribes without retaining the disposable. historyImported and historyImportFailed both call MaterialAlertDialogBuilder(this).show(). Activity destroyed before completion gives BadTokenException. The destroyed activity is also retained for the duration, on top of the memory in point 1.

Duration for ~180k rows:

Stage Time
parse into objects 3-5 s
timestampsFor full scan, 5 beacons 1-3 s
SHA-256 per row 1-2 s
180k inserts, TEXT primary key plus composite index, one transaction 15-40 s

30-60 s with nothing on screen.

Fix, progress: MaterialAlertDialogBuilder.setView with a CircularProgressIndicator, non-cancellable. Already used at activity_fetch_from_icloud.xml:80. ProgressDialog is deprecated. Determinate is available for the merge phase since rows.size() is known after parsing.

Fix, rotation: disposing in onDestroy cancels a 60 s import. Hold the subscription in a ViewModel instead. AppleLoginActivity.java:191 uses ViewModelProvider with AppleLoginViewModel. Acceptable as a follow-up issue if you would rather not grow this PR.

Espresso coverage for the progress UI

ImportingHistoryFromTheDeviceListTest cannot cover this. It writes a real ZIP and runs the real importer against a real database, so the import completes in milliseconds and "in progress" is not observable.

Add a seam: AppDependencies already has replaceGeocoder, replaceICloud, replaceAuthService, replaceLogRedactor, replaceBundleBuilder, all cleared by reset(). Add an importer there so a test can install a fake that blocks on a latch.

HistoryImporter already supports this: the package-private HistoryImporter(sink, clock) constructor is what your JVM test uses. Only MyDevicesListActivity.java:670 reaches past it to construct the database-backed one inline.

Assert the indicator appears, updates, and is gone on every path including each failure. Two traps from AGENTS.md: use Eventually.check rather than a bare onView when waiting, and a GONE view still matches withId, so "gone" is matches(not(isDisplayed())) or doesNotExist(), never an expected NoMatchingViewException.

3. Unexpected exceptions reported as a damaged file

HistoryImporter.java:99:

} catch (RuntimeException error) {
    // Commons CSV reports some malformed record shapes while its iterator advances.
    throw new HistoryImportException(
            HistoryImportException.Reason.INVALID_ARCHIVE,
            "History CSV structure is invalid",
            error);
}

The Commons CSV case is real: it throws from inside Iterator.next(), so a bad record surfaces during the for (CSVRecord record : parser) loop. RuntimeException is wider. An NPE in readCsvEntry lands here too, and the distinction is gone after this line.

MyDevicesListActivity.java:721:

} else {
    title = R.string.history_import_failed_title;
    message = R.string.history_import_failed_message;
}

This is the unknown-cause branch. It should reach ErrorReportActivity, the existing "an error occurred" screen that carries the failure verbatim and links the issue template. Instead it tells the user to find a different file.

onBundleExportFailed at MyDevicesListActivity.java:555 already does this correctly, and ExportingTagsReachesTheBugPageTest covers it.

Changes:

  • Add a HistoryImportException.Reason meaning unexpected, separate from INVALID_ARCHIVE.
  • Route it from historyImportFailed's else to ErrorReportActivity.
  • Pass describe(rootOf(error)), not describe(error). Rx wraps what a map throws; the wrapper puts "RuntimeException" on the page. See the comment at MyDevicesListActivity.java:575.
  • Add an EXTRA_BODY string for the import case via scripts/add_strings.py, all ten locales.
  • Add an import equivalent of ExportingTagsReachesTheBugPageTest, asserting the page is reached for an unexpected failure and not reached for a malformed archive.

Interactively co-authored by Claude Code and @parawanderer

@hamza-lzr

Copy link
Copy Markdown
Contributor Author

Addressed the three findings in 402e95d:

  • interned repeated beacon IDs per archive while keeping the one timestamp scan per beacon;
  • added a non-cancellable progress dialog, bounded merge updates, an AppDependencies importer seam, and progress/error-path Espresso coverage;
  • separated unexpected reader failures from damaged archives and routed unknown failures, with their root cause, to ErrorReportActivity using localized import copy.

The activity now dismisses its dialog before touching a destroyed window. ViewModel ownership across rotation is tracked separately in #175.

The refreshed fork workflows are currently waiting for approval.

@parawanderer

Copy link
Copy Markdown
Owner

CI is red, and the run reads as flaky when it is not. Two symptoms, one cause.

What the run says

HistoryImportProgressLayoutTest > itInflatesAndMeasuresInBothThemes  FAILED
  android.view.InflateException: Binary XML file line #18 in
  dev.wander.android.opentagviewer.debug:layout/history_import_progress:
  Error inflating class <unknown>

testEmulator Tests 509/659 completed. (22 skipped) (1 failed)
##[error]The operation was canceled.

The suite stopped at 509 of 659 and sat there for 30 minutes until timeout-minutes: 45 killed the job. Check the whole suite actually ran went red, which is what caught it. "Cancelled" here means timed out, not aborted.

The layout is not the problem

Your CircularProgressIndicator is identical to the one already working at activity_fetch_from_icloud.xml:80, down to trackColor="?attr/colorSurfaceVariant" and trackCornerRadius="2dp". The theme is Theme.Material3.DynamicColors.DayNight.NoActionBar, so the attribute resolves, and five other layouts use it.

HistoryImportProgressLayoutTest also follows SystemColorsLayoutTest correctly: androidx.appcompat.view.ContextThemeWrapper over createConfigurationContext, then LayoutInflater.from(themed).

What is different

SystemColorsLayoutTest inflates history_list_item, inline_top_toolbar and my_device_list_item. None contains a Material progress indicator, so this is the first one inflated outside an Activity in this repo.

A bare ContextThemeWrapper does not install the AppCompat inflater factory that an Activity gets. MaterialAlertDialogBuilder and getLayoutInflater() both have it, which is why the dialog itself is likely fine at runtime, but worth confirming on a device rather than assuming.

Why the hang follows from it

The dialog layout cannot inflate, so the progress dialog never appears, so aLongImportShowsAndUpdatesProgressBeforeItsResult waits on R.id.history_import_progress forever. One inflation failure, 30 minutes of dead emulator, one timeout.

Fix the inflation and both go away. Options: inflate through an Activity context in the test, or drop the layout test and rely on the Espresso one which uses the real dialog path.

Everything else in 402e95d looks right. The intern pool, Reason.UNEXPECTED routed to ErrorReportActivity with describe(rootOf(error)), runOnUiThread around the progress callback with the isFinishing/isDestroyed guards, and the latch-driven fake through AppDependencies.replaceHistoryImporter. build and Static checks are both green, so the JVM side passes.


Interactively co-authored by Claude Code and @parawanderer

CI on this branch ran for 45 minutes and was killed. Two faults, and the
one that mattered was not the one that showed up red.

**The suite hung in readFailureUsesTheGenericFailureMessage.** The per-
test logcats put it beyond doubt: every other test in that class finished
by 17:51:24, and that one's logcat was written at 18:21:36, 176 KB of
device output, at the moment the job timed out.

Opening the document happens in the Activity, outside importArchive, so a
file that has gone arrives as a bare IOException rather than a
HistoryImportException. historyImportFailed reads that as something this
app did not anticipate and opens ErrorReportActivity - which is right for
a defect here and wrong for a file the picker handed back. The dialog the
test waits for never appeared, and inRoot(isDialog()) against a screen
with no dialog is the case Espresso retries internally for seconds at a
time. AGENTS.md describes that under Eventually; seven tests once took
six and a half minutes to it.

READ_FAILED already existed for this and was not being used. Opening is
now wrapped in it, so the failure reaches the plain dialog rather than
the report page, and the test passes because the behaviour is right.

**And the layout test inflated by a route the app never takes.** It built
a themed context by hand and called LayoutInflater.from on it, which is
the SystemColorsLayoutTest pattern - but nothing there inflates a
Material progress indicator, and an AppCompatActivity installs a factory
a bare ContextThemeWrapper does not. It threw InflateException while the
dialog worked on a device.

It now inflates through TestHostActivity's own inflater, with
cloneInContext carrying a night configuration for the second render, so
both themes are covered without touching global state. Rule 12 makes this
exact point about drawables: load them the way the app loads them.

The progress dialog itself was never broken.
aLongImportShowsAndUpdatesProgressBeforeItsResult passed at 17:51:22.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@parawanderer

Copy link
Copy Markdown
Owner

Correction to my last comment: the hang was not a cascade from the inflation failure. aLongImportShowsAndUpdatesProgressBeforeItsResult passed at 17:51:22; the per-test logcats show the run then sat in readFailureUsesTheGenericFailureMessage until the job timed out. Two separate faults, and I attributed both to one.

0bd848a fixes both:

  • MyDevicesListActivity wraps a file that will not open as HistoryImportException(Reason.READ_FAILED, ...) instead of letting the raw IOException through to the bug page. That was the hang — the test waited on a failure message the user never got.
  • HistoryImportProgressLayoutTest inflates through an Activity's own LayoutInflater (ActivityScenario + cloneInContext for the dark render), which is what installs the AppCompat factory the Material progress indicator needs.

CI is green on all checks now, so this is going in.

@hamza-lzr — from @parawanderer: thanks for taking the time to make this contribution to this project!


Interactively co-authored by Claude Code and @parawanderer

parawanderer#139 landed while this was in review and both touch `AppDependencies`. Conflict resolved by
taking main's version and re-applying this branch's five additions: the two imports, the
`historyImporterFactory` field, `historyImporter(context)`, `replaceHistoryImporter` and the
line in `reset()`.

`OpenTagViewerDatabase` merged cleanly - this branch adds `historyImportDao()` and no schema, so
it sits beside parawanderer#139's migrations 6-7, 7-8 and 8-9 rather than competing with them. Database
version is 9 and every migration is registered.
parawanderer#139 added `LocationReport.provenance` and said in its own docstring that the column exists
because the history is exported - "without it the CSV hands somebody a file where their own
phone's positions sit unlabelled among Apple's, and nothing in the file says which is which".
The writer was never updated, and this branch's importer builds its rows without the field at
all, which is a `NOT NULL` column: Room refuses the insert and takes the whole merge transaction
with it. Five instrumented tests fail on it, and none of them could have failed before, because
each PR was tested against a main without the other.

So the round trip now carries it:

- `BeaconLocationReport` gains the field, because the export reads models rather than rows, and
  the three mappings in `BeaconRepository` carry it in both directions
- `HistoryCsvWriter` writes a `provenance` column. An Apple row is a stranger's iPhone estimating
  a position to within a hundred metres or worse; a `local` row is this phone hearing the tag
  directly and recording its own position as the tag's. Unlabelled, the second reads as the first
- `HistoryImporter` requires the column and refuses a row whose value is neither, rather than
  storing a third kind of report that nothing downstream has a branch for
- `HistoryImportDao.merge` sets it, which is the crash

**The column is required rather than optional**, and that is free exactly once: the history
export shipped in no release - `util/export/` does not exist in `android-app-v1.0.5` - so there
are no archives in the field to stay compatible with. Adding it after 1.1.0 would have meant
accepting both shapes forever.

The two round-trip tests now carry it, and the first uses `local` on purpose: it is the value a
missing column does not fall back to, so it cannot pass against a writer or reader that defaults
the field into place.
@parawanderer
parawanderer merged commit aac7de1 into parawanderer:main Sep 12, 2026
3 checks passed
parawanderer added a commit that referenced this pull request Sep 12, 2026
#173 and #154 landed while this was open. Both sides had appended to strings.xml, in all ten
locales, at the same point in the file - so every conflict is additive and both sets are kept.
`add_strings.py --check` passes at 370 strings across 9 translated locales.
@parawanderer parawanderer added this to the App version 1.1.0 milestone Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Import a history CSV back, so a reset does not lose it permanently

2 participants