Job queue improve perms, design, and docs

docs/guides/generating-mosaics.md

@@ -1,28 +1,75 @@
 # Generating mosaics
 
-Once layers are grouped into layersets, and each layerset is trimmed in the [multimask](./trimming.md) interface, the final step is to collapse the content into a single mosaic per layerset, like finally gluing down all the individual clippings in a collage. You'll find these within within a map's **Mosaic** → **Derivatives** (see [New Iberia, 1885](https://oldinsurancemaps.net/map/sanborn03375_001#mosaic)).
+## Understanding mosaic "derivatives"
+
+Once layers are grouped into layersets, and each layerset is trimmed in the [MultiMask](./trimming.md) interface, the final step is to collapse the content into a single mosaic per layerset, like finally glueing down all the individual clippings in a collage.
+
+![masked detroit 1897 vol. 2](../images/detroit-vol2-1897-masked.png)
+
+This screenshot of [Detroit, Mich., 1897, vol. 2](https://oldinsurancemaps.net/map/sanborn03985_005), shows 106 individual layers that have been georeferenced and masked. Creating a derivative from this mosaic produces a single layer, in this case a Cloud Optimized GeoTIFF.
+
+![cog mosaic for detroit 1897 vol. 2](../images/detroit-vol2-1897-cog.png)
+
+This greatly improves the utility of what you have created by allowing you to do things like:
+
+- Download a single mosaic file for an entire volume
+- Load a single layer of the entire volume into web applications like HistoryForge or OpenHistoricalMap
+
+It also speeds up parts of OIM that use mosaic derivatives when they are available.
+
+## Accessing derivatives
+
+Within within a map's **Mosaic** → **Derivatives** (see [New Iberia, 1885](https://oldinsurancemaps.net/map/sanborn03375_001#mosaic)) section you'll find a list of all "layersets" within that map (usually just Main Content, but other categories could appear here as well if they have more than one layer).
 
 Currently, OIM supports the creation of two different derivatives per layerset:
 
 - **Cloud Optimized GeoTIFFs (COG)**
     - This is the primary output, and when present, is used within OIM in certain contexts
-- **Static XYZ Tileset**
-    - This is a rendering of the latest COG mosaic to actual PNG tiles in a {z}/{x}/{y}.png folder structure.
+- **XYZ Tileset**
+    - This is a rendering of the latest COG mosaic to actual PNG tiles in a {z}/{x}/{y}.png folder structure
+
+These two formats, in turn, allow us to provide other formats and web services as well.
 
 !!! note
 
     You may be wondering what happens to Key Maps and other categories of maps, i.e. other layersets. Though these are handled on the backend exactly like the Main Content layerset, they are only shown in the derivatives list if they have more than one layer.
 
 ## Keeping derivatives up-to-date
 
-The creation of a single geotiff (or static XYZ tileset) is effectively a snapshot of the work that has been performed on a map's layers and masks up to that point. Timestamps are displayed showing when masks were last updated, and when derivatives were last generated.
+The creation of a single geotiff (or static XYZ tileset) is effectively a snapshot of the work that has been performed on a map's layers and masks up to that point. The derivatives list will show a timestamp for the last time an edit was made to the multimask, as well as a small status tag for each format showing the status of its last mosaic job.
+
+### `up-to-date`
+
+![up-to-date cog derivative](../images/mosaic-cog-up-to-date.png)
+
+A derivative is **up-to-date** when there has been no updates to the multimask or layers since it was created. You can hover the tag to see when it was last run.
+
+
+### `stale`
+
+![stale cog derivative](../images/mosaic-cog-stale.png)
+
+If a derivative is marked as **stale**, that means that the masks or layers used to create it have been changed since the last job was run. In this case you can click the **build COG** button to queue the re-creation of this derivative.
+
+!!! note
+
+    Because **XYZ Tilesets** are generated _from_ COGs, the COG derivative must be up-to-date, or queued, before you will be able to queue the creation of a tileset.
+
+### `queued`
+
+![queued cog derivative](../images/mosaic-cog-queued.png)
+
+Because the process to create a COG (or XYZ tileset) from dozens or hundreds of layers can take several hours, we use a **queueing** system to ensure the server is never overloaded. Queued mosaic operations are processed with a first in, first out policy. You can see the status of mosaicking processes (queued, running, or completed) in [Jobs](https://oldinsurancemaps.net/jobs).
+
+### `running`
 
-![up-to-date cog derivative with timestamps](../images/mosaic-cog-fresh.png)
+![queued cog derivative](../images/mosaic-cog-running.png)
 
-However, this means that if the ground control points are subsequently updated for a layer, or the masks are adjusted, the derivative artifact will be out-of-date. In this case, the interface will show timestamps of out-of-date artifacts in <span style="color:red">red</span> and a button will be available allowing you to queue this artifact to be rebuilt.
+Once a job has been picked up by a runner, it will change from queued to **running** for as long as the process takes, eventually returning to **up-to-date** when the process is completed.
 
-![out-of-date cog derivative with red timestamp](../images/mosaic-cog-stale.png)
+## How long does it take for a job to complete?
 
-Because the process to create a single geotiff (or xyz tileset) from dozens or hundreds of layers can take several hours, we use a queueing system to ensure the server is never overloaded. You can see all mosaicking processes (queued, running, or completed) in [Jobs](https://oldinsurancemaps.net/jobs).
+This depends on the layerset itself, and which format of derivative you are creating.
 
-In the current setup, only two jobs are allowed to run at any time, and queued mosaic operations are processed with a first in, first out policy.
+- COG: The greater the **geographic extent** of all layers (imagine drawing a big rectangle that encompasses **all** of the layers in the layerset), the longer the job (and larger the resulting file).
+- XYZ Tileset: The greater the **geographic coverage** of all layers in the mosaic (imagine all of the mask boundaries dissolved into a big polygon, or multi-polygon), the longer the job (and higher the number of tiles in the tileset).

ohmg/frontend/svelte_components/src/components/base/Link.svelte

@@ -9,18 +9,19 @@
   export let download = false;
   export let title = '';
   export let classes = [];
+  export let style = '';
 
   const classStr = classes.join(' ');
 </script>
 
 {#if external || rightArrow || download}
   {#if download }
-  <a {href} class={classStr} target={external ? '_blank' : null} {title} download>
+  <a {href} class={classStr} style={style} target={external ? '_blank' : null} {title} download>
     <slot></slot>
     <DownloadSimple />
   </a>
   {:else}
-  <a {href} class={classStr} target={external ? '_blank' : null} {title}>
+  <a {href} class={classStr} style={style} target={external ? '_blank' : null} {title}>
     <slot></slot>
     {#if external}
       <ArrowSquareOut />
@@ -30,7 +31,7 @@
   </a>
   {/if}
 {:else}
-  <a {href} class={classStr} {title}>
+  <a {href} style={style} class={classStr} {title}>
     <slot></slot>
   </a>
 {/if}

ohmg/frontend/svelte_components/src/components/map/MosaicDerivatives.svelte

@@ -8,6 +8,8 @@
     import ModalInfo from '../base/ModalInfo.svelte';
     import LoadingEllipsis from '../shared/LoadingEllipsis.svelte';
     import DerivativeDD from '../shared/DerivativeDD.svelte';
+    import MosaicStatus from '../shared/tags/MosaicStatus.svelte';
+    import ArrowsClockwise from 'phosphor-svelte/lib/ArrowsClockwise';
 
     export let CONTEXT;
     export let mapId;
@@ -49,52 +51,29 @@
                 i.masksDateDisplay = i.multimask_date ? new Date(i.multimask_date*1000).toLocaleString() : null;
                 i.xyzStaticArchiveURL = i.xyz_tiles_url ? `${i.xyz_tiles_url}/archive.tar.gz` : null;
                 i.xyzStaticTilesURL = i.xyz_tiles_url ? `${i.xyz_tiles_url}/{z}/{x}/{y}.png` : null;
+
+                i.enableCogQueueBtn = CONTEXT.user.perms.includes("core.queue_mosaic_cog")
+                i.cogQueueBtnTitle = CONTEXT.user.perms.includes("core.queue_mosaic_cog") ?
+                    "Click to queue COG build" : "You do not have permission for this action"
+
                 i.cogStale = false;
-                i.cogDateDisplay = "---";
-                i.showCogQueueBtn = false;
-                if (i.latest_cog_job) {
-                    if (i.latest_cog_job.stage == "completed") {
-                        i.cogDate = new Date(i.latest_cog_job.date_started * 1000).toLocaleString();
-                        i.cogDateDisplay = `last updated: ${i.cogDate}`;
-                        i.cogStale = i.multimask_date ? i.latest_cog_job.date_started < i.multimask_date : false
-                        i.showCogQueueBtn = i.cogStale;
-                    } else {
-                        i.cogDateDisplay = i.latest_cog_job.stage;
-                        if (i.latest_cog_job.stage != "queued" && i.latest_cog_job.stage != "running") {
-                            i.showCogQueueBtn = true;
-                        }
-                    }
-                } else {
-                    i.showCogQueueBtn = true;
-                    i.cogDateDisplay = "not generated";
+                if (i.latest_cog_job && i.latest_cog_job.stage == "completed") {
+                    i.cogStale = i.multimask_date ? i.latest_cog_job.date_started < i.multimask_date : false
                 }
 
                 // same thing now but for the XYZ tileset build
-                i.showXyzQueueBtn = false;
-                i.enableXyzQueue = false;
-                if (i.mosaic_cog_url && !i.cogStale) {
-                    i.enableXyzQueue = true
+                i.xyzQueueBtnTitle = "Click to queue XYZ tileset build";
+                i.enableXyzQueue = true;
+                if (!i.latest_cog_job || i.cogStale) {
+                    i.enableXyzQueue = false;
+                    i.xyzQueueBtnTitle = "COG must be up-to-date before tileset can be built";
                 }
-
-                i.xyzStale = false;
-                i.xyzDateDisplay = "---"
-                if (i.latest_xyz_job) {
-                    if (i.latest_xyz_job.stage == "completed") {
-                        i.xyzDate = new Date(i.latest_xyz_job.date_started * 1000).toLocaleString();
-                        i.xyzDateDisplay = `last updated: ${i.xyzDate}`;
-                        i.xyzStale = i.multimask_date ? i.latest_xyz_job.date_started < i.multimask_date : false;
-                        i.showXyzQueueBtn = i.xyzStale;
-                    } else {
-                        i.xyzDateDisplay = i.latest_xyz_job.stage;
-                        if (i.latest_xyz_job.stage != "queued" && i.latest_xyz_job.stage != "running") {
-                            i.showXyzQueueBtn = true;
-                            i.enableXyzQueue = true;
-                        }
-                    }
-                } else {
-                    i.showXyzQueueBtn = true;
-                    i.xyzDateDisplay = "not generated"
+                if (!CONTEXT.user.perms.includes("core.queue_mosaic_xyz")) {
+                    i.enableXyzQueue = false;
+                    i.xyzQueueBtnTitle = "You do not have permission for this action";
                 }
+
+
                 return i
             });
             loading = false;
@@ -128,20 +107,21 @@
 <ModalConfirm id="modal-confirm-cog-queue"
     yesAction={() => {submitQueueRequest('queue-cog-creation')}}
 >
-    <p>Submit mosaic COG generation to queue?</p>
+    <p>Submit mosaic COG generation to queue? If the current COG is up-to-date, there is no need to rebuild it.</p>
 </ModalConfirm>
 <ModalConfirm id="modal-confirm-xyz-queue"
     yesAction={() => {submitQueueRequest('queue-tileset-creation')}}
 >
-    <p>Submit XYZ tileset generation to queue?</p>
+    <p>Submit XYZ tileset generation to queue? If the current XYZ tileset is up-to-date, there is no need to rebuild it.</p>
 </ModalConfirm>
 
 <div>
-    <p>Once layers have been trimmed in the <strong>MultiMask</strong> they can be combined into a single
-    mosaic. We provide access to this mosaic in the form of file downloads, web service endpoints, and direct integrations
-    into other platforms. Read more <Link href="https://docs.oldinsurancemaps.net/guides/generating-mosaics" rightArrow={true}>in the docs</Link></p>
-    <p>If the MultiMask has been updated <em>after</em> one of these artifacts was generated, dates will be shown in
-    red and you can queue that mosaic to be rebuilt. <button class="is-text-link" on:click={initLayersets}>refresh table</button></p>
+    <p>Once layers have been trimmed in the <strong>MultiMask</strong> they are combined into a single
+    mosaic output, or "derivative". You can access these derivatives here in the form of file downloads, web service endpoints, and direct integrations
+    into other platforms.</p>
+    <p>To queue the creation (or recreation) of mosaics for each layerset, use the summary table below. If a derivative has been created, but
+    the MultiMask has since been edited, it will be marked as <strong>stale</strong> it should be queued for a rebuild.
+    Read more <Link href="https://docs.oldinsurancemaps.net/guides/generating-mosaics" rightArrow={true}>in the docs</Link></p>
 </div>
 {#if loading}
 <LoadingEllipsis />
@@ -152,24 +132,31 @@
         <span>
             {`${ls.name} (${ls.layers_masked_ct}/${ls.layers.length} layers masked)`}
         </span>
-        {#if ls.multimask_date}
-        <span class="mask-timestamp">
-            masks last updated: {ls.masksDateDisplay}
-        </span>
-        {/if}
+        <div class="dl-title-right">
+            {#if ls.multimask_date}
+            <span class="mask-timestamp">
+                masks last edit: {ls.masksDateDisplay}
+            </span>
+            {/if}
+            <button style="color:white" on:click={initLayersets}><ArrowsClockwise/></button>
+        </div>
     </h4>
     <dl style="margin-bottom: 1em;">
         <dt class="derivative-subheader">
             Cloud Optimized GeoTIFF
-            <span class="timestamp{ls.cogStale ? ' stale' : ''}">
-                {ls.cogDateDisplay}
-                {#if ls.showCogQueueBtn}
-                <button class="is-text-link" on:click={() => {
+            <div class="derivative-subheader-right">
+                {#if ls.latest_cog_job}
+                <MosaicStatus job={ls.latest_cog_job} maskDate={ls.multimask_date}/>
+                {/if}
+                <button
+                    class="button is-small is-link"
+                    disabled={!ls.enableCogQueueBtn}
+                    title={ls.cogQueueBtnTitle}
+                    on:click={() => {
                         layersetToQueueForCog=ls.id;
                         openModal('modal-confirm-cog-queue')
-                    }}>queue rebuild</button>
-                {/if}
-            </span>
+                    }}>build COG</button>
+            </div>
         </dt>
         <dt>Direct download (.tif)</dt>
         <DerivativeDD
@@ -183,7 +170,7 @@
             linkType="copytext"
             naMessage="requires COG"
         />
-        <dt>XYZ tile endpoint (dynamic)</dt>
+        <dt>Dynamic XYZ endpoint</dt>
         <DerivativeDD
             linkUrl={ls.dynamicXyzUrl}
             linkType="copytext"
@@ -196,41 +183,37 @@
             naMessage="requires COG"
         />
         <dt class="derivative-subheader">
-            Static XYZ Tileset
-            <span class="timestamp{ls.xyzStale ? ' stale' : ''}">
-                {ls.xyzDateDisplay}
-                {#if ls.showXyzQueueBtn}
-                <button class="is-text-link"
+            XYZ Tileset
+            <div class="derivative-subheader-right">
+                {#if ls.latest_xyz_job}
+                <MosaicStatus job={ls.latest_xyz_job} maskDate={ls.multimask_date}/>
+                {/if}
+                <button
+                    class="button is-small is-link"
                     disabled={!ls.enableXyzQueue}
-                    title={ls.enableXyzQueue ? 
-                        'Queue creation of XYZ tileset' :
-                        'COG must be rebuilt before tileset can be created'}
+                    title={ls.xyzQueueBtnTitle}
                     on:click={() => {
                         layersetToQueueForCog=ls.id;
                         openModal('modal-confirm-xyz-queue')
-                    }}>queue rebuild</button>
-                {/if}
-            </span>
+                    }}>build XYZ tileset</button>
+            </div>
         </dt>
         <dt>Direct download (gzipped tarfile)</dt>
         <DerivativeDD
             linkUrl={ls.xyzStaticArchiveURL}
             linkType="download"
             naMessage="not yet generated"
         />
-        <dt>Tiles endpoint</dt>
+        <dt>Static XYZ endpoint</dt>
         <DerivativeDD
             linkUrl={ls.xyzStaticTilesURL}
             linkType="copytext"
             naMessage="not yet generated"
         />
         <dt class="derivative-subheader">
-            Extensions...
-            <span class="timestamp">
-                always current, unless noted
-            </span>
+            Extensions
         </dt>
-        <dt>Open in OpenHistoricalMap editor (uses XYZ tile endpoint)</dt>
+        <dt>Open in OpenHistoricalMap editor (uses dynamic XYZ endpoint)</dt>
         <DerivativeDD
             linkUrl={ls.ohmUrl}
             linkType="external"
@@ -261,6 +244,11 @@
         align-items: baseline;
         flex-wrap: wrap;
     }
+    .dl-title-right {
+        display: flex;
+        align-items: center;
+        gap: .5em
+    }
     span.mask-timestamp {
         font-size: .9em;
     }
@@ -271,6 +259,7 @@
         padding: .25em .5em;
         display: flex;
         justify-content: space-between;
+        align-items: center;
         flex-wrap: wrap;
         font-weight: 700;
         font-size: .85em;
@@ -279,10 +268,9 @@
     dt.derivative-subheader {
         background-color: rgb(188, 241, 253);
     }
-    .timestamp {
-        color: rgb(128, 128, 128);
-    }
-    .timestamp.stale {
-        color: red;
+    .derivative-subheader-right {
+        display: flex;
+        align-items: center;
+        gap: .5em;
     }
 </style>

ohmg/frontend/svelte_components/src/components/shared/tags/MosaicStatus.svelte

@@ -0,0 +1,26 @@
+<script>
+    import Link from "../../base/Link.svelte";
+    export let job;
+    export let maskDate;
+
+    const stale = maskDate > job.date_started;
+    const dateDisplay = new Date(job.date_started * 1000).toLocaleString()
+    const title = `latest build: ${dateDisplay}`
+
+</script>
+
+{#if job.stage == "completed"}
+    {#if stale}
+    <div class="tag is-danger" title={title}>
+        stale
+    </div>
+    {:else}
+    <div class="tag is-success" title={title}>
+        up-to-date
+    </div>
+    {/if}
+{:else}
+    <div class={`tag ${job.stage == "queued" ? 'is-info' : 'is-warning'}`} title={title}>
+        <Link href="/jobs" style="color:black;" external={true} title="Open running jobs page...">{job.stage}</Link>
+    </div>
+{/if}