978 Migrate Local Users, Datasets, Folders and Files - #1058

Open

tcnichol wants to merge 99 commits into

mainfrom

978-migrate-users

Commits on Mar 28, 2024

adding new dependency and class
tcnichol
committed
511923d
starting a user migration
tcnichol
committed
f763cb4
adding script for migrate metadata definitions
tcnichol
committed
84d43bc
adding option of temporary password (for use with users who are being migrated from old clowder instance
tcnichol
committed
e82e023
reverting changes
tcnichol
committed
dcdbea1
temporary - for users that are migrated
tcnichol
committed
79e56b4

Commits on Apr 23, 2024

creates users
tcnichol
committed
33f1cab

Commits on Apr 25, 2024

does not actually create the dataset
tcnichol
committed
93509e7
using beanie might not work here due to async issues
tcnichol
committed
aba43a3
using nest asyncio to fix problem with even loop closing
tcnichol
committed
727685f

Commits on Apr 26, 2024

entry from right table for file bytes
tcnichol
committed
d4e9526

Commits on May 20, 2024

dependencies for migration, adding file entry and indexing files
tcnichol
committed
8c7d6bd
wrong api key
tcnichol
committed
2ba7df5

Commits on May 21, 2024

api key works for user,
tcnichol
committed
f624784
file upload works, sloppy needs fixed
tcnichol
committed
adc6c63
we are only getting partial files, not sure how to fix yet
tcnichol
committed
171b2b3

Commits on May 26, 2024

files upload right now
tcnichol
committed
1f6a2ed
delete file
tcnichol
committed
2bf100c

Commits on May 27, 2024

adding folders
tcnichol
committed
4eccae1

Commits on May 30, 2024

folder hierarchy - this should work
tcnichol
committed
c01ce96

Commits on May 31, 2024

works for now, folders need to be fixed
tcnichol
committed
c2ab289

Commits on Jun 1, 2024

previous is no longer selectable when moving to my datasets. we start on the first page
tcnichol
committed
2d7cdce
Merge branch 'main' into 978-migrate-users
tcnichol
committed
65599bb
new folder hierarchy method works
tcnichol
committed
5be97bf

Commits on Jun 2, 2024

need a license for datasets
tcnichol
committed
f479284
method for getting all the folders
tcnichol
committed
cb4b835

Commits on Jun 3, 2024

need to remove logging, but folders are created and files uploaded to correct folders
tcnichol
committed
e885a2a
should work now
tcnichol
committed
b4db0f4
fixing typos
tcnichol
committed
72e0845
formatting
tcnichol
committed
a6cd25a
sample .env
tcnichol
committed
54747ca
formatting
tcnichol
committed
55ff21a

Commits on Jul 13, 2024

Merge branch 'main' into 978-migrate-users
tcnichol
committed
e216405
refactoring migrate_user
tcnichol
committed
851503e

Commits on Jul 15, 2024

formatting
tcnichol
committed
074154e

Commits on Aug 5, 2024

Merge branch 'main' into 978-migrate-users
tcnichol
committed
909b8f6

Commits on Aug 6, 2024

Merge branch 'release/v2.0-beta-3' into 978-migrate-users
tcnichol
committed
b309023

Commits on Aug 14, 2024

Merge remote-tracking branch 'origin/main' into 978-migrate-users
tcnichol
committed
58ada46

Commits on Aug 22, 2024

rename
tcnichol
committed
62d5b02
Merge branch 'release/v2.0-beta-3' into 978-migrate-users
longshuicy
committed
8772325

Commits on Aug 26, 2024

migrate metadata definition; modify migration script (#1184)

longshuicy
and
tcnichol
authored
ec56aa4

Commits on Aug 27, 2024

Commits on Aug 29, 2024

Migrate metadata (#1192)
longshuicy
authored
2a52c7c

Commits on Sep 3, 2024

1187 migrate collections and spaces as v2 metadata. (#1190)

authored
4e68f21

Commits on Aug 16, 2025

Merge branch 'main' into 978-migrate-users
tcnichol
committed
4d5281a
removed - already processed
tcnichol
committed
70ea8af

Commits on Aug 17, 2025

new readme, using tomli and a config file to limit or set options for migration
tcnichol
committed
92f4f2c
adding example for config.toml
tcnichol
committed
80d41b0
exclude users - users we do not migrate (or their files)
tcnichol
committed
bc67fc7
adding example env
tcnichol
committed
d86a756
proper path to .env file
tcnichol
committed
a453068
excluding items
tcnichol
committed
c64be06
check to see if we should migrate a dataset
tcnichol
committed
c7e218c
using the excluded space ids
tcnichol
committed
c4d3cdb
fix - if license absent in v1 assign a default license for v2
tcnichol
committed
ca80dbd
fix type in license
tcnichol
committed
f98edda
adding TODO items
tcnichol
committed
5a84349
working on this, need a better way to check what collections a dataset is contained in
tcnichol
committed
1932362

Commits on Aug 18, 2025

Commits on Aug 19, 2025

log metadata def - errors are when there are duplicates
tcnichol
committed
dbe3016

Commits on Aug 21, 2025

get top level collections
tcnichol
committed
73609d2

Commits on Aug 22, 2025

method stubs for new way of processing
tcnichol
committed
093bfb0
more method stubs
tcnichol
committed
8477cba

Commits on Aug 23, 2025

Commits on Aug 24, 2025

remove unused methods
tcnichol
committed
a5dd8ea
too many folders created
tcnichol
committed
f534160

Commits on Aug 25, 2025

progress for dataset folders and files
tcnichol
committed
a00e600

Commits on Aug 26, 2025

the folder hierarchy part is not working for v2
tcnichol
committed
2fb33c4

Commits on Aug 27, 2025

new route should return all folders in a dataset
tcnichol
committed
10f2026

Commits on Aug 28, 2025

Commits on Aug 29, 2025

new method should work for uploading files to either a dataset or a folder, most of these will be uploaded to folders following the 'collection becomes dataset' model
tcnichol
committed
1a5e2b7

Commits on Sep 1, 2025

partial fix?
tcnichol
committed
04fc4b2

Commits on Sep 2, 2025

messy, but works, can post the data
tcnichol
committed
c76e75e

Commits on Sep 3, 2025

worked for one collection with datasets and folders
tcnichol
committed
c45d8f0
other collection also works now
tcnichol
committed
9ca8850
get list of datasets in collections, we need this for migration
tcnichol
committed
e2be15b
this should work for datasets now as well
tcnichol
committed
c94dd6d
single dataset works, except metadata
tcnichol
committed
88e2534
removing unused methods
tcnichol
committed
7e139cb
adding todo with dataset metadata to a folder
tcnichol
committed
5b2562d

Commits on Sep 8, 2025

migrate now adds a metadata file to a folder
tcnichol
committed
0a8e764
running pipenv run black
tcnichol
committed
19b0f02
running pipenv run black for app
tcnichol
committed
0fa7465
fix typo that got in
tcnichol
committed
80ae7cf
ran codegen?
tcnichol
committed
a034ca5
ran codegen?
tcnichol
committed
1f3e5ed

Commits on Sep 9, 2025

more readme instructions
tcnichol
committed
34c868b

Commits on Nov 5, 2025

fixing lint
tcnichol
committed
27a1355