From f1abcec81693dd70718038424585ec889156e5bc Mon Sep 17 00:00:00 2001 From: evanwill Date: Tue, 8 Sep 2026 17:48:29 -0700 Subject: [PATCH 1/3] rewrite download_by_csv to use only ruby libraries, removing wget --- docs/rake_tasks/download_by_csv.md | 56 +++- rakelib/download_by_csv.rake | 462 ++++++++++++++++++++++++++--- 2 files changed, 470 insertions(+), 48 deletions(-) diff --git a/docs/rake_tasks/download_by_csv.md b/docs/rake_tasks/download_by_csv.md index b4e4c8f3..273f863a 100644 --- a/docs/rake_tasks/download_by_csv.md +++ b/docs/rake_tasks/download_by_csv.md @@ -1,21 +1,17 @@ # download_by_csv -`rake download_by_csv` allows you to download a list of files from a CSV using Wget. +`rake download_by_csv` downloads a list of files from a CSV. -Please ensure you have Wget installed and available on your terminal! -Check by typing `wget --version` in your terminal. -If you need to install on Windows, see [Add more to Git Bash](https://evanwill.github.io/_drafts/notes/gitbash-windows.html). +The task uses only the Ruby standard library, so there is nothing to install beyond the normal project setup (`bundle install`). -Requirements: - -- wget +*Note:* earlier versions of this task required Wget, which is no longer needed. Using defaults: -- Create a CSV named "download.csv" with columns "url" (the full link to the objects you want to download) and "filename_new" (optional, the name you want to save/rename the downloaded objects as). Make sure it is UTF-8 (not from Excel). +- Create a CSV named "download.csv" with columns "url" (the full link to the objects you want to download) and "filename_new" (optional, the name you want to save/rename the downloaded objects as). Make sure it is UTF-8 (not from Excel). - Put "download.csv" into the root of this repository (i.e. same place as the Rakefile). - Open terminal and type `rake download_by_csv` -- Items included in the "download.csv" will be downloaded using wget, renamed, and output in new folder "download/". +- Items included in the "download.csv" will be downloaded, renamed, and output in new folder "download/". The options can be changed by passing arguments with the rake command. @@ -25,9 +21,45 @@ The options can be changed by passing arguments with the rake command. | download_link | the column name that is the full link to the objects you want to download | "url" | | download_rename | the column name of the new filename for the downloads (optional, if you don't provide one, it will use what ever the url uses) | "filename_new" | | output_dir | the name of the new folder to download the files | "download/" | +| delay | seconds to wait between requests, to keep the load on the server you are downloading from reasonable | 1 | + +The order follows [:csv_file,:download_link,:download_rename,:output_dir,:delay]. +For example, + +`rake download_by_csv["other_down.csv","item_link","new_name","download_folder",2]` + +To download as fast as possible, set the delay to 0. +Please be considerate with collections held by other institutions! +A short delay is often the difference between a download that finishes and one that gets blocked part way through. + +## How the download works + +**Redirects.** +Permalinks are followed automatically, up to ten hops per item, including relative locations and hops that change host or scheme. +This covers the usual DOI, Handle, ARK, CONTENTdm, and repository "download" links. +Any session cookie set along the way is sent to the following hops, which many repository platforms require. +The final URL of the chain is used to work out the filename. + +**Filenames.** +If the "filename_new" column has a value, it is used. +Otherwise the name comes from the server's content disposition header, then from the last part of the final URL, and finally from the row number plus an extension guessed from the file type. +Names are always cleaned up so a download can only ever be written inside your output folder. + +**Resuming.** +Files that already exist in the output folder are skipped, so you can run the task again after fixing errors without downloading everything a second time. +Each file is written to a temporary ".part" file and only given its real name once the download finishes, so an interrupted run never leaves a broken or empty object behind. + +**Errors.** +Timeouts, dropped connections, and temporary server errors (such as 429 and 503) are tried up to three times, waiting longer between each attempt and honoring the server's own "retry after" header. +A row that still fails does not stop the run. +When the task finishes, any failures are listed in "download_errors.csv" inside the output folder, with the URL, the intended filename, and the reason. +That file can be used as the input CSV to retry just those items: +`rake download_by_csv["download/download_errors.csv"]` -The order follows [:csv_file,:download_link,:download_rename,:output_dir]. -For example, +## Troubleshooting -`rake download_by_csv["other_down.csv","item_link","new_name","download_folder"]` +- **"the server returned a web page"** means the URL gave back HTML rather than the object. Usually the link points at an item landing page instead of the file itself, or the object requires a login. +- **403 or 404 errors** on links that work in your browser usually mean the repository requires a session or referring page. Try the direct file URL rather than the permalink. +- **certificate errors** mean the server's HTTPS setup is broken or out of date. Check whether the collection is also available over a different host. +- **downloads that stall** are usually rate limiting. Raise the delay, for example `rake download_by_csv["download.csv","url","filename_new","download/",5]`. diff --git a/rakelib/download_by_csv.rake b/rakelib/download_by_csv.rake index 922e8fb1..66ffd649 100644 --- a/rakelib/download_by_csv.rake +++ b/rakelib/download_by_csv.rake @@ -1,57 +1,447 @@ +# frozen_string_literal: true + ############################################################################### # TASK: download_by_csv # -# read csv, download using wget +# read csv, download objects using the Ruby standard library ############################################################################### +require 'net/http' +require 'uri' +require 'openssl' +require 'time' + +# Helpers for the download_by_csv task. +# Kept in a module since all files in "rakelib" share one namespace. +module CBDownload + # network settings + MAX_REDIRECTS = 10 + MAX_ATTEMPTS = 3 + MAX_RETRY_WAIT = 120 + OPEN_TIMEOUT = 15 + READ_TIMEOUT = 60 + WRITE_TIMEOUT = 60 + CHUNK_LOG_SECONDS = 0.5 + # many library servers reject the default ruby user agent + USER_AGENT = 'Mozilla/5.0 (compatible; CollectionBuilder download_by_csv)' + # response codes worth trying again + RETRY_STATUS = [408, 425, 429, 500, 502, 503, 504].freeze + # network errors worth trying again + RETRY_ERRORS = [ + Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNRESET, Errno::ECONNABORTED, + Errno::EPIPE, Errno::ETIMEDOUT, Errno::EHOSTUNREACH, EOFError, SocketError, + OpenSSL::SSL::SSLError + ].freeze + # guess an extension when neither the url nor the headers give a filename + EXTENSIONS = { + 'image/jpeg' => '.jpg', 'image/png' => '.png', 'image/tiff' => '.tif', + 'image/gif' => '.gif', 'image/webp' => '.webp', 'image/svg+xml' => '.svg', + 'application/pdf' => '.pdf', 'audio/mpeg' => '.mp3', 'audio/wav' => '.wav', + 'audio/x-wav' => '.wav', 'audio/ogg' => '.ogg', 'video/mp4' => '.mp4', + 'video/quicktime' => '.mov', 'video/webm' => '.webm', 'text/plain' => '.txt', + 'text/html' => '.html', 'text/xml' => '.xml', 'application/xml' => '.xml', + 'application/json' => '.json', 'application/zip' => '.zip' + }.freeze + + class Error < StandardError; end + + # an http response that is not a success or a redirect + class HTTPError < Error + attr_reader :code, :retry_after + + def initialize(response) + @code = response.code.to_i + @retry_after = CBDownload.parse_retry_after(response['retry-after']) + super("server returned #{response.code} #{response.message}") + end + end + + @last_request_at = nil + + ############################################################################# + # download one row + ############################################################################# + + # returns [status, path] where status is :downloaded or :skipped + def self.fetch_to_file(url, output_dir, rename, index, total, delay) + puts "[#{index}/#{total}] #{url}" + + # when the new name is known up front, skip before touching the network + if rename + dest = File.join(output_dir, sanitize_filename(rename, "item_#{index}")) + if File.exist?(dest) + puts " '#{dest}' already exists, skipping!" + return [:skipped, dest] + end + end + + throttle(delay) + + with_retries do + with_response(url) do |response, final_uri| + name = rename || filename_from(response, final_uri, index) + dest = File.join(output_dir, sanitize_filename(name, "item_#{index}")) + if File.exist?(dest) + puts " '#{dest}' already exists, skipping!" + next [:skipped, dest] + end + + warn_if_html(response, dest) + written = write_stream(response, dest) + puts " saved '#{dest}' (#{human_size(written)})" + [:downloaded, dest] + end + end + end + + ############################################################################# + # http + ############################################################################# + + # request url, following redirects, and yield the successful response + def self.with_response(url) + uri = parse_uri(url) + cookies = {} + result = nil + + (MAX_REDIRECTS + 1).times do + unless %w[http https].include?(uri.scheme.to_s) + raise Error, "unsupported url scheme '#{uri.scheme}'" + end + + redirect = nil + @last_request_at = Time.now + + Net::HTTP.start(uri.host, uri.port, + use_ssl: uri.scheme == 'https', + open_timeout: OPEN_TIMEOUT, + read_timeout: READ_TIMEOUT, + write_timeout: WRITE_TIMEOUT, + max_retries: 0) do |http| + request = Net::HTTP::Get.new(uri) + request['User-Agent'] = USER_AGENT + request['Accept'] = '*/*' + # ask for the bytes as they are so streamed binaries are never gzipped + request['Accept-Encoding'] = 'identity' + request['Cookie'] = cookie_header(cookies) unless cookies.empty? + if uri.user + request.basic_auth(percent_decode(uri.user), percent_decode(uri.password.to_s)) + end + + http.request(request) do |response| + collect_cookies(cookies, response) + if response.is_a?(Net::HTTPRedirection) + redirect = next_location(uri, response) + elsif response.is_a?(Net::HTTPSuccess) + result = yield(response, uri) + else + raise HTTPError, response + end + end + end + + return result if redirect.nil? + + uri = redirect + end + + raise Error, "too many redirects (more than #{MAX_REDIRECTS})" + end + + # run a request, trying again on temporary failures + def self.with_retries + attempt = 0 + begin + attempt += 1 + yield + rescue HTTPError => e + raise if !RETRY_STATUS.include?(e.code) || attempt >= MAX_ATTEMPTS + + wait = e.retry_after || backoff(attempt) + raise if wait > MAX_RETRY_WAIT + + report_retry(e.message, wait, attempt) + sleep wait + retry + rescue *RETRY_ERRORS => e + raise if attempt >= MAX_ATTEMPTS + + wait = backoff(attempt) + report_retry("#{e.class}: #{e.message}", wait, attempt) + sleep wait + retry + end + end + + def self.report_retry(message, wait, attempt) + puts " #{message}, trying again in #{wait.round}s (attempt #{attempt + 1} of #{MAX_ATTEMPTS})" + end + + def self.backoff(attempt) + 2**attempt + end + + # resolve the next url in a redirect chain, handling relative locations + def self.next_location(uri, response) + location = response['location'].to_s.strip + raise Error, "redirect (#{response.code}) with no location header" if location.empty? + + target = uri.merge(parse_uri(location)) + if uri.scheme == 'https' && target.scheme == 'http' + puts " WARNING: redirect downgrades from https to http" + end + puts " redirected to #{target}" + target + end + + # keep session cookies through the whole chain, some repositories set one + # on the permalink host and expect it back on the object host + def self.collect_cookies(cookies, response) + fields = response.get_fields('set-cookie') + return if fields.nil? + + fields.each do |raw| + name, value = raw.split(';', 2).first.to_s.strip.split('=', 2) + cookies[name] = value if name && !name.empty? && value + end + end + + def self.cookie_header(cookies) + cookies.map { |name, value| "#{name}=#{value}" }.join('; ') + end + + def self.parse_retry_after(value) + value = value.to_s.strip + return nil if value.empty? + return value.to_i if value.match?(/\A\d+\z/) + + seconds = (Time.httpdate(value) - Time.now).ceil + seconds.positive? ? seconds : nil + rescue ArgumentError + nil + end + + # wait so that requests are at least delay seconds apart + def self.throttle(delay) + return if delay <= 0 || @last_request_at.nil? + + elapsed = Time.now - @last_request_at + sleep(delay - elapsed) if elapsed < delay + end + + ############################################################################# + # urls and filenames + ############################################################################# + + def self.parse_uri(url) + URI.parse(url.to_s.strip) + rescue URI::InvalidURIError + # some servers send locations containing spaces or other raw characters + URI.parse(escape_unsafe(url.to_s.strip)) + end + + def self.escape_unsafe(url) + url.gsub(%r{[^A-Za-z0-9\-._~:/?\#\[\]@!$&'()*+,;=%]}) do |char| + char.bytes.map { |byte| format('%%%02X', byte) }.join + end + end + + def self.percent_decode(value) + decoded = value.to_s.gsub(/%[0-9A-Fa-f]{2}/) { |match| match[1, 2].hex.chr } + decoded = decoded.dup.force_encoding(Encoding::UTF_8) + decoded.valid_encoding? ? decoded : value.to_s + end + + # work out a filename from the headers, then the final url, then the row + def self.filename_from(response, uri, index) + extension = EXTENSIONS.fetch(content_type(response), '') + name = filename_from_disposition(response['content-disposition']) + name = percent_decode(File.basename(uri.path.to_s)) if name.nil? || name.empty? + return "item_#{index}#{extension}" if name.nil? || name.empty? + + # urls like "/download" or "/objects/1234" resolve to a name with no + # extension, so take one from the content type + name += extension if File.extname(name).empty? + name + end + + def self.filename_from_disposition(header) + header = header.to_s + return nil if header.empty? + + # rfc 5987 form, filename*=UTF-8''name.jpg + match = header.match(/filename\*\s*=\s*[^']*'[^']*'([^;]+)/i) + return percent_decode(match[1].strip) if match + + match = header.match(/filename\s*=\s*"([^"]*)"/i) || header.match(/filename\s*=\s*([^;]+)/i) + match ? match[1].strip : nil + end + + # never let a header or url write outside of the output directory + def self.sanitize_filename(name, fallback) + name = name.to_s.split(/[?#]/).first.to_s + name = File.basename(name.tr('\\', '/')).strip + name = name.gsub(/[\x00-\x1f<>:"|*]/, '_') + name = '' if ['.', '..'].include?(name) + return fallback if name.empty? + + if name.length > 150 + extension = File.extname(name)[0, 20].to_s + name = File.basename(name, File.extname(name))[0, 150 - extension.length] + extension + end + name + end + + def self.content_type(response) + response['content-type'].to_s.split(';').first.to_s.strip.downcase + end + + def self.warn_if_html(response, dest) + return unless ['text/html', 'application/xhtml+xml'].include?(content_type(response)) + return if ['.html', '.htm', '.xhtml'].include?(File.extname(dest).downcase) + + puts ' WARNING: the server returned a web page, this may be an error or login page' + end + + ############################################################################# + # writing + ############################################################################# + + # stream to a part file so a failed download never leaves a broken object + def self.write_stream(response, dest) + part = "#{dest}.part" + expected = response['content-length'].to_i + written = 0 + shown = false + logged_at = Time.now + + File.open(part, 'wb') do |file| + response.read_body do |chunk| + file.write(chunk) + written += chunk.bytesize + next unless expected.zero? || expected > 1_000_000 + next unless Time.now - logged_at > CHUNK_LOG_SECONDS + + print_progress(written, expected) + shown = true + logged_at = Time.now + end + end + clear_progress if shown + + if expected.positive? && written != expected + raise Error, "incomplete download, expected #{expected} bytes but got #{written}" + end + + File.rename(part, dest) + written + rescue StandardError, Interrupt + File.delete(part) if File.exist?(part) + raise + end + + def self.print_progress(written, expected) + if expected.positive? + percent = (written * 100.0 / expected).round + print "\r #{percent}% of #{human_size(expected)}" + else + print "\r #{human_size(written)}" + end + $stdout.flush + end + + def self.clear_progress + print "\r#{' ' * 40}\r" + $stdout.flush + end + + def self.human_size(bytes) + return "#{bytes} B" if bytes < 1024 + return "#{(bytes / 1024.0).round(1)} KB" if bytes < 1_048_576 + + "#{(bytes / 1_048_576.0).round(1)} MB" + end + + # record failures so they can be fed back into the task as a csv + def self.write_error_csv(failures, output_dir, link_column, rename_column) + path = File.join(output_dir, 'download_errors.csv') + CSV.open(path, 'wb') do |csv| + csv << [link_column, rename_column, 'error'] + failures.each { |failure| csv << [failure[:url], failure[:rename], failure[:error]] } + end + path + end +end + desc "download objects and rename using csv" -task :download_by_csv, [:csv_file,:download_link,:download_rename,:output_dir] do |_t, args| +task :download_by_csv, [:csv_file, :download_link, :download_rename, :output_dir, :delay] do |_t, args| # set default arguments args.with_defaults( csv_file: 'download.csv', download_link: 'url', download_rename: 'filename_new', - output_dir: 'download/' + output_dir: 'download/', + delay: '1' ) + # rake arguments always arrive as strings + delay = args.delay.to_f + delay = 0.0 if delay.negative? + # check for csv file - if !File.exist?(args.csv_file) + unless File.exist?(args.csv_file) puts "CSV file does not exist! No files downloaded and exiting." - else - # read csv file - csv_text = File.read(args.csv_file, :encoding => 'utf-8') - csv_contents = CSV.parse(csv_text, headers: true) - - # Ensure that the output directory exists. - FileUtils.mkdir_p(args.output_dir) unless Dir.exist?(args.output_dir) - - # iterate on csv rows - csv_contents.each do |item| - # check for download url - if item[args.download_link] - # check for rename - if item[args.download_rename] - # check if file already exists - name_new = File.join(args.output_dir, item[args.download_rename]) - if File.exist?(name_new) - puts "new filename '#{name_new}' already exists, skipping!" - next - end - puts "downloading" - # call wget - system('wget','-O', name_new, item[args.download_link]) - else - puts "downloading" - # call wget - system('wget', item[args.download_link], "-P", args.output_dir) - end - else - puts "no download url!" - end + next + end + + # read csv file + csv_text = File.read(args.csv_file, :encoding => 'utf-8') + csv_contents = CSV.parse(csv_text, headers: true) + + # check for the download url column + unless csv_contents.headers.include?(args.download_link) + puts "CSV does not have a '#{args.download_link}' column! No files downloaded and exiting." + next + end + has_rename = csv_contents.headers.include?(args.download_rename) + + # Ensure that the output directory exists. + FileUtils.mkdir_p(args.output_dir) unless Dir.exist?(args.output_dir) + + total = csv_contents.size + downloaded = 0 + skipped = 0 + failures = [] + + puts "downloading #{total} rows from '#{args.csv_file}' into '#{args.output_dir}'" + puts "waiting #{delay}s between requests" if delay.positive? + + # iterate on csv rows + csv_contents.each_with_index do |item, index| + # check for download url + url = item[args.download_link].to_s.strip + if url.empty? + puts "[#{index + 1}/#{total}] no download url!" + next end - puts "done downloading." + # check for rename + rename = has_rename ? item[args.download_rename].to_s.strip : '' + rename = nil if rename.empty? + begin + status, = CBDownload.fetch_to_file(url, args.output_dir, rename, index + 1, total, delay) + status == :skipped ? skipped += 1 : downloaded += 1 + rescue StandardError => e + puts " ERROR: #{e.message}" + failures << { url: url, rename: rename, error: e.message } + end end + puts "done downloading. #{downloaded} downloaded, #{skipped} skipped, #{failures.length} failed." + + unless failures.empty? + error_csv = CBDownload.write_error_csv(failures, args.output_dir, args.download_link, args.download_rename) + puts "failed downloads written to '#{error_csv}'" + end end From 8b0ab192aca8631677c8a322287b8fa787f3d9e9 Mon Sep 17 00:00:00 2001 From: EvanWill Date: Wed, 9 Sep 2026 13:21:41 -0700 Subject: [PATCH 2/3] small tweak fixes --- rakelib/download_by_csv.rake | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/rakelib/download_by_csv.rake b/rakelib/download_by_csv.rake index 66ffd649..4fb2a2c1 100644 --- a/rakelib/download_by_csv.rake +++ b/rakelib/download_by_csv.rake @@ -29,8 +29,8 @@ module CBDownload # network errors worth trying again RETRY_ERRORS = [ Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNRESET, Errno::ECONNABORTED, - Errno::EPIPE, Errno::ETIMEDOUT, Errno::EHOSTUNREACH, EOFError, SocketError, - OpenSSL::SSL::SSLError + Errno::EPIPE, Errno::ETIMEDOUT, Errno::EHOSTUNREACH, Errno::ECONNREFUSED, + Errno::ENETUNREACH, EOFError, SocketError, OpenSSL::SSL::SSLError ].freeze # guess an extension when neither the url nor the headers give a filename EXTENSIONS = { @@ -158,7 +158,7 @@ module CBDownload raise if !RETRY_STATUS.include?(e.code) || attempt >= MAX_ATTEMPTS wait = e.retry_after || backoff(attempt) - raise if wait > MAX_RETRY_WAIT + raise Error, "#{e.message} (retry-after #{wait}s exceeds max wait of #{MAX_RETRY_WAIT}s)" if wait > MAX_RETRY_WAIT report_retry(e.message, wait, attempt) sleep wait @@ -281,7 +281,7 @@ module CBDownload def self.sanitize_filename(name, fallback) name = name.to_s.split(/[?#]/).first.to_s name = File.basename(name.tr('\\', '/')).strip - name = name.gsub(/[\x00-\x1f<>:"|*]/, '_') + name = name.gsub(/[\x00-\x1f<>:"|*?]/, '_') name = '' if ['.', '..'].include?(name) return fallback if name.empty? From 3d7d4fd0a82fc798ec1127bd12f6a1141a9261c3 Mon Sep 17 00:00:00 2001 From: EvanWill Date: Wed, 9 Sep 2026 14:04:54 -0700 Subject: [PATCH 3/3] improve rake docs with use cases --- docs/rake_tasks/download_by_csv.md | 14 ++++++++++---- docs/rake_tasks/generate_derivatives.md | 2 ++ docs/rake_tasks/rename_by_csv.md | 7 +++++++ docs/rake_tasks/rename_lowercase.md | 1 + docs/rake_tasks/resize_images.md | 1 + 5 files changed, 21 insertions(+), 4 deletions(-) diff --git a/docs/rake_tasks/download_by_csv.md b/docs/rake_tasks/download_by_csv.md index 273f863a..2aa02b45 100644 --- a/docs/rake_tasks/download_by_csv.md +++ b/docs/rake_tasks/download_by_csv.md @@ -1,10 +1,13 @@ # download_by_csv -`rake download_by_csv` downloads a list of files from a CSV. +`rake download_by_csv` downloads a list of files (URLs) from a CSV. -The task uses only the Ruby standard library, so there is nothing to install beyond the normal project setup (`bundle install`). +Optionally, the task can rename the files it downloads so you can normalize filenames at the same time. +This is sometimes required when download filenames would be the same, such as downloading from a IIIF server where all items are "default.jpg". -*Note:* earlier versions of this task required Wget, which is no longer needed. +If errors are encountered, the task outputs "download_errors.csv" providing information about the items that were unsuccessful. + +This task is helpful to set up a self-contained "objects" folder for a project by downloading external resources from a repository or S3 bucket. Using defaults: @@ -21,7 +24,7 @@ The options can be changed by passing arguments with the rake command. | download_link | the column name that is the full link to the objects you want to download | "url" | | download_rename | the column name of the new filename for the downloads (optional, if you don't provide one, it will use what ever the url uses) | "filename_new" | | output_dir | the name of the new folder to download the files | "download/" | -| delay | seconds to wait between requests, to keep the load on the server you are downloading from reasonable | 1 | +| delay | seconds to wait between requests, to keep the load on the server you are downloading from reasonable and avoid rate limits | 1 | The order follows [:csv_file,:download_link,:download_rename,:output_dir,:delay]. For example, @@ -34,6 +37,9 @@ A short delay is often the difference between a download that finishes and one t ## How the download works +The task uses only the Ruby standard library, so there is nothing to install beyond the normal project setup (`bundle install`). +*Note:* earlier versions of this task required Wget, which is no longer needed. + **Redirects.** Permalinks are followed automatically, up to ten hops per item, including relative locations and hops that change host or scheme. This covers the usual DOI, Handle, ARK, CONTENTdm, and repository "download" links. diff --git a/docs/rake_tasks/generate_derivatives.md b/docs/rake_tasks/generate_derivatives.md index c958ca39..daaf7457 100644 --- a/docs/rake_tasks/generate_derivatives.md +++ b/docs/rake_tasks/generate_derivatives.md @@ -42,3 +42,5 @@ rake generate_derivatives[,,70] The mini_magick Gem is used to interface with ImageMagick so it supports both current version 7 and legacy versions (which are common on Linux). The image_optim Gem is used to optimize images using the optimization libraries provided by the image_optim_pack Gem. image_optim_pack does not provide binaries for Windows, so optimization is skipped when using the rake task on Windows. + +ImageMagick (all image formats) and Ghostscript (PDF rendering) are essential software commonly used for batch processing, so although they are external dependencies, they may be useful for other tasks on your computer. diff --git a/docs/rake_tasks/rename_by_csv.md b/docs/rake_tasks/rename_by_csv.md index a39a8956..751b1563 100644 --- a/docs/rake_tasks/rename_by_csv.md +++ b/docs/rake_tasks/rename_by_csv.md @@ -2,6 +2,13 @@ This task allows you to rename a batch of files using a spreadsheet. +This is helpful for normalizing filenames for use in a project and on the web, allowing you to use spreadsheet tools to batch generate the new names, often directly in your CB metadata CSV. +For example, you might keep the old filename in "original_filename" for archival purposes, then create a new "filename" field based on the "objectid" of the item and the standardize file extensions. + +Be sure to include file extensions in both your old and new filename columns! +However, keep in mind it does NOT convert file types, just renames, so changing the file extension should be done with care. +Changing the file extension can be useful to normalize situations such as `.jpeg` to `.jpg` or inconsistent case (`.PDF` to `.pdf`). + Using defaults: - Create a CSV named "rename.csv" with the columns "filename_old" (the exact matching current filename, not including directory) and "filename_new" (the new name you want, not including directory). Make sure it is UTF-8 (not from Excel). diff --git a/docs/rake_tasks/rename_lowercase.md b/docs/rake_tasks/rename_lowercase.md index a109b9ef..e1c011cd 100644 --- a/docs/rake_tasks/rename_lowercase.md +++ b/docs/rake_tasks/rename_lowercase.md @@ -1,6 +1,7 @@ # rename_lowercase This task takes a folder of files and copies them to a new folder with all filenames downcased. +This is commonly used to normalize the filenames for use on the web, where most servers are case sensitive, to simplify matching up with metadata CSV data. Using default: diff --git a/docs/rake_tasks/resize_images.md b/docs/rake_tasks/resize_images.md index 59110384..a4f084c0 100644 --- a/docs/rake_tasks/resize_images.md +++ b/docs/rake_tasks/resize_images.md @@ -2,6 +2,7 @@ This task resizes all images (.jpeg, .jpg, .png, .tif, or .tiff) in a folder of files within this repository. It outputs the resized images to a new folder in this repository, with all filenames and extensions lowercased, and optionally converted into another image format. +This is commonly used if you have a batch of scanned full sized TIF images and need to generate reasonable sized JPEG access copies for the web. Requirements: