show file size of archived courses when unarchiving

[Alex-Jordan]

When at the Unarchive courses page, the list of courses now leaves off the ".tar.gz" and includes the size of the file to be unarchived. This will help me identify inefficiencies with how RS users are handling their courses. If I am about to unarchive a particularly large course, I will then go into it and see what can be trimmed. Usually, the faculty have put course archive files in their course. Or they are using images inefficiently. Or they copied entire folder trees for backup before editing local pg files.

I did this in a way to make it extensible. Perhaps there would be interest in also showing the date the archive file was last modified. Perhaps this is of tangential interest to @drdrew42 with #3043 in that the displayed course name could be the actual internal course name, not the file name with .tar.gz truncated off.

[Alex-Jordan]

This now has a few styling changes too, in separate commits.

[drgrice1]

I think it would be better to just update the listArchivedCourses method (instead of adding the statArchivedCourses method), and use the same format for showing archive files in the UI in the two places that is done (and the only two places where the listArchivedCourses method is used before this pull request). The two places are when unarchiving courses and on the main course admin page that lists courses and archived courses.

[drgrice1]

Part of the reason that I suggest just updating the listArchivedCourses method instead of adding the statArchivedCourses method is due to the messy structure of the created statArchivedCourses method. It first calls listArchivedCourses. That constructs a Mojo::File object representing the archive directory, lists all files in that directory with the .tar.gz extension, and returns the basename of all of those files. Then the statArchivedCourses method again creates a Mojo::File object representing the archive directory. Preprends it as a pure string (making it completely useless to construct the Mojo::File object) to the deconstructed Mojo::File basename of the listed archive files from the listArchiveCourses method. Note that if you have a Mojo::File object representing a directory, then you can get a Mojo::File object representing anything in that directory with the child method, and you can stat a file using the stat method.

If the two methods were combined much of that inefficiency and code clutter could be cleaned up. You just have to update lines 68-70 of templates/ContentGenerator/CourseAdmin.html.ep to use the updated return value.

Actually, I have put in a pull request to this branch that does the things I have mentioned.

[Alex-Jordan]

Considering #3043, what are your thoughts on making the keys be the actual course ID from inside each course archive file? For example (referencing the discussion in #3043) having drew => {filename => 'alex.tar.gz'}?

[drgrice1]

Yeah, that would probably be a good idea considering the change in #3043. You can use basically the same Tar::Archive code @drdrew42 uses in that pull request to get that from the tar ball. Of course, not extracting the tar ball after it though.

[Alex-Jordan]

This now has @drdrew's code from #3043 that gets the internal top folder name for the course ID instead of the file's name.

You can rename an archive file, and see the right thing at the Unarchive page. However you still can't actually unarchive such a file until #3043 is merged too.

[drgrice1]

There are a few changes to the latest additions needed.

[Alex-Jordan]

Thanks for those notes. I applied them and squashed them in with the commit that moved to using the tarball's top folder.

[drgrice1]

This looks good. I will give my approval, but note that this should not be merged until after #3043.

[pstaabp]

Looks good, but again, don't merge until #3043

[drgrice1]

This can be merged now that #4043 has been.

[somiaj]

Now that this is in production, I notice that on my server (I have 60 archives and older hardware) it takes 30 seconds of server at 100cpu time to list the archives. In my case the .tar.gz name and the archive name are the same, so there is a lot of unneeded computation to look in each .tar.gz. Part of the issue is since the archives are listed on the index page of the admin course, this happens a lot (not just when visiting the unarchive page).

I don't think it is worth looking in each and every archive when listing them. I would just assume the file name is correct, maybe have a tool that checks archives and lets the user know if the filename doesn't match the directory in the archive and let them rename them.

[Alex-Jordan]

I did not study @drdrew42's code that identifies the top level folder name in the archive. The regular tar utility lets you do tar --exclude='*/*' -tf ... to get the top level files. Might that (or whatever the perl module's counterpart to that) be faster?

Looking over Archive::Tar now. Maybe using get_files is overkill. There is a contains_file that could maybe be used to check if the top level folder is the same as the name of the archive. And if not, then use list_files (not get_files) to find the actual name.

[somiaj]

@Alex-Jordan I can test things out if you like (haven't looked at the code so you'll have to send patches).

I still think it is unneeded overhead in lots of cases on the main "Course List" page. Maybe only do that on the "unarchive courses" page and could add some warning if the file name differs from the actual course, while on the may "Course List" page just trust the filename.

[somiaj]

I attempted to use list_files instead of get_files along with your suggest of testing with contains_file for an initial test. I didn't notice any difference. I also tried just using list_files as the only call and that didn't seem to make it noticeably faster.

I also removed all calls to $arch, but left the creating of the archive object, Archive::Tar->new($_), and it seems to be about the same speed. So the problem is in the creation of the object itself. I may have to play around to see if calling tar directly would drastically speed this up.

[Alex-Jordan]

I cheated and used Claude. Does this patch speed things up or relieve your CPU? The idea here is what I described before. Claude's testing benchmarked a 600x speedup.

I am cooking dinner and haven't actually tested that this functions. Like, that it still shows the actual name of the top level folder.

diff --git a/lib/WeBWorK/Utils/CourseManagement.pm b/lib/WeBWorK/Utils/CourseManagement.pm
index 581cf10d7..e90a2bfc1 100644
--- a/lib/WeBWorK/Utils/CourseManagement.pm
+++ b/lib/WeBWorK/Utils/CourseManagement.pm
@@ -107,6 +107,22 @@ sub listCourses {
    }
 }

+sub _archiveTopLevelDir {
+   my ($archivePath, $arch) = @_;
+
+   (my $guess = path($archivePath)->basename) =~ s/\.tar\.gz$//i;
+
+   return $guess if $arch->contains_file("$guess/") || $arch->contains_file($guess);
+
+   my %top_level;
+   for my $file ($arch->list_files) {
+       (my $first = $file) =~ s{/.*}{}s;
+       $top_level{$first} = 1 if length $first;
+   }
+   return undef unless keys %top_level == 1;
+   return (keys %top_level)[0];
+}
+
 =item listArchivedCourses($ce)

 Lists the courses which have been archived (end in .tar.gz). The courses found
@@ -141,16 +157,11 @@ sub listArchivedCourses {
        }
        my $basename = $_->basename;
        my $arch     = Archive::Tar->new($_);
-       my %top_level;
-       for my $file ($arch->get_files) {
-           (my $first = $file->full_path) =~ s{/.*}{}s;
-           $top_level{$first} = 1 if length $first;
-       }
-       unless (keys %top_level == 1) {
+       my $currCourseID = _archiveTopLevelDir($_, $arch);
+       unless (defined $currCourseID) {
            warn "The archive $basename does not contain a single top-level course directory.\n";
            next;
        }
-       my ($currCourseID) = keys %top_level;
        my $round = 10**($unit_idx > 0 ? $unit_idx - 1 : 0);
        $return{$currCourseID} = {
            filename => $basename,
@@ -1009,14 +1020,9 @@ sub unarchiveCourse {
    # source course ID must come from there and not from the caller-supplied name
    # -- otherwise a renamed .tar.gz restores files under one name while the
    # database dump is sought under another.
-   my %top_level;
-   for my $file ($arch->get_files) {
-       (my $first = $file->full_path) =~ s{/.*}{}s;
-       $top_level{$first} = 1 if length $first;
-   }
+   my $currCourseID = _archiveTopLevelDir($archivePath, $arch);
    die "The archive $archivePath does not contain a single top-level course directory.\n"
-       unless keys %top_level == 1;
-   my ($currCourseID) = keys %top_level;
+       unless defined $currCourseID;

    ##### step 2: move a conflicting course away #####

[Alex-Jordan]

I don't think this is actually working. It is showing the file name of an archive that has a different name for its top level folder. I will investigate later tonight, and hopefully find a quick patch. Or abandon it and just list the file names in the first place on the Course Listings page. That's what this PR originally set out to do, but it coincided with the time that @drdrew42 was doing the other thing over in Unarchive, so we took the same approach here. But yes, if it slows the Course Listings page down too much, that won't be acceptable.

[Alex-Jordan]

Er wait, I think it might be working. I was looking at the wrong place. If you are testing, let me know what you find.

[Alex-Jordan]

Hmm, I packed my dev server with a bunch of archive courses. And with or without this change, the course listings page is taking about 2.-2.8 seconds to load. My 2.20 production server, with far more archives, takes only 1.1 second on average.

I'd like to hear what @drgrice1 thinks. But it seems like this may need to either not list the archived courses, or revert to just printing the files.

[somiaj]

@Alex-Jordan The provided patch didn't speed it up, for me the creating of the archive object is what seems to take time. My server is quite old (15-20 years) so no ssd, etc. Might just be a hardware limitation, but it works just fine, unsure why this is taking longer for 60 archived files compared to what you have tested.

[drgrice1]

As I told @somiaj in Slack, I noticed this show down on my test server immediately after this pull request went in. I also tested commenting out all but the line that opens the tar archive file (so skipping all processing after that), and then using the file name for the hash key, and the slow down still occurs. So the mere act of constructing the Archive::Tar object is the slow down.

One solution would be to do what @somiaj suggested in Slack, and cache the results of parsing the archive files. The first time the page loads it would still take a while, but after that it would load fast. I was planning to implement this. Basically, save the archive filename, last modified time, and identified course name in a JSON file in the root of the admin course. Then read that, check modified times and update as needed.

[Alex-Jordan]

Are we opposed to just using tar as a system utility? I wonder if it could be faster. Alex Jordan Mathematics Instructor Portland Community College

…

On Tue, Aug 4, 2026, 7:54 PM Jaimos Skriletz ***@***.***> wrote: *somiaj* left a comment (openwebwork/webwork2#3047) <https://urldefense.com/v3/__https://github.com/openwebwork/webwork2/pull/3047*issuecomment-5186983528__;Iw!!Ka_JY85zDv0FFw!ngLSakbyKaDGjxOu1HYjp4XskPRwT3H6nx1TsP7r71AMqKFe23f8GGdI_JaLUjV3HZKMMlTrYGAha9yByXlNS_0VHoQ$> @Alex-Jordan <https://urldefense.com/v3/__https://github.com/Alex-Jordan__;!!Ka_JY85zDv0FFw!ngLSakbyKaDGjxOu1HYjp4XskPRwT3H6nx1TsP7r71AMqKFe23f8GGdI_JaLUjV3HZKMMlTrYGAha9yByXlNjuxsf68$> The provided patch didn't speed it up, for me the creating of the archive object is what seems to take time. My server is quite old (15-20 years) so no ssd, etc. Might just be a hardware limitation, but it works just fine, unsure why this is taking longer for 60 archived files compared to what you have tested. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/openwebwork/webwork2/pull/3047?email_source=notifications&email_token=ABEDOAAIJWBIWM4DUCIMFC35IKOVFA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJYGY4TQMZVGI4KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM*issuecomment-5186983528__;Iw!!Ka_JY85zDv0FFw!ngLSakbyKaDGjxOu1HYjp4XskPRwT3H6nx1TsP7r71AMqKFe23f8GGdI_JaLUjV3HZKMMlTrYGAha9yByXlNu_ZhOaM$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABEDOAGZOKVMT57FM5ARS3D5IKOVFAVCNFSNUABDKJSXA33TNF2G64TZHMZTANRUGU3TGO2JONZXKZJ3GQ4DIMRSGYZDSOJTUF3AE__;!!Ka_JY85zDv0FFw!ngLSakbyKaDGjxOu1HYjp4XskPRwT3H6nx1TsP7r71AMqKFe23f8GGdI_JaLUjV3HZKMMlTrYGAha9yByXlNhLKncCw$> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://urldefense.com/v3/__https://github.com/notifications/mobile/ios/ABEDOAFDKQBTXGKOKLTPICD5IKOVFA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJYGY4TQMZVGI4KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y__;!!Ka_JY85zDv0FFw!ngLSakbyKaDGjxOu1HYjp4XskPRwT3H6nx1TsP7r71AMqKFe23f8GGdI_JaLUjV3HZKMMlTrYGAha9yByXlN3S9Ezm8$> and Android <https://urldefense.com/v3/__https://github.com/notifications/mobile/android/ABEDOABPRCDSZVBD6YPGUA35IKOVFA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJYGY4TQMZVGI4KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI__;!!Ka_JY85zDv0FFw!ngLSakbyKaDGjxOu1HYjp4XskPRwT3H6nx1TsP7r71AMqKFe23f8GGdI_JaLUjV3HZKMMlTrYGAha9yByXlNbsUx8DQ$>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[drgrice1]

I am opposed to using a system utility, and actually doubt that it would be faster.

[somiaj]

I did a test with calling tar -tf $path --exclude="*/*" directly. It was still slow. It might have been slightly faster, but not worth it. Also this doesn't work reliably. Some of the archives do include the 'top level directory' as a file and others don't, so some archives that returned the correct thing and others it didn't return anything at all.

To me caching the data is probably the best as archives don't change that often. Maybe the cache should also be updated when an archive is created and not just when listing them.

The only other option I can think of is trust the file name for the archive list, and have some utility to check archives on the "unarchive course" page to let users know if the file name differs from the course ID. Though this would probably have to be manually run or done via a cache system to make that page not load so slowly with large archives.

Since it sounds like @drgrice1 is working on a caching method, that sounds good to me.