Skip to content

Instantly share code, notes, and snippets.

@rayvoelker

rayvoelker/00-README.md

Last active March 6, 2026 22:28
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save rayvoelker/5aae40b92a5758a8713360931b2df2e5 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save rayvoelker/5aae40b92a5758a8713360931b2df2e5 to your computer and use it in GitHub Desktop.
Download ZIP
ilsaux Framework & Archive Plan - CHPL Sierra ILS
Raw
framework--archive-plan.md

ILS Auxiliary Server Archive & Migration Plan

Generated: 2026-03-06T16:16:45

Report Classification

Active-Critical (6 reports, 344.6 MB, 295 files)

Running in 2026, shelf list reports -- document fully, migrate first

Report Full Name Last Run Size Credentials
slitemdata Item Data Inconsistency Report 2026-03-06 344.2 MB db_connection, host_config, user_config
sllastcopymissing Shelflist - Last Copy Missing 2026-03-01 69.4 KB db_connection, host_config, user_config
slmainmissing Shelflist - Main Missing 2026-03-01 54.6 KB db_connection, host_config, user_config
slmissing Shelflist - Missing 2026-01-15 219.4 KB db_connection, host_config, user_config
slonorder Shelflist - Circulated On-Order 2026-02-15 60.8 KB db_connection, host_config, user_config
slrepair Shelflist - Repair 2026-03-01 90.0 KB db_connection, host_config, user_config

Active (16 reports, 32.3 GB, 9,413 files)

Running in 2026, non-shelf-list -- document, evaluate for migration

Report Full Name Last Run Size Credentials
activeusersmonthly Active Users Count - Monthly 2026-03-01 53.6 KB db_connection, host_config, user_config
bookclubrenewal Book Club Renewal Report 2026-03-06 82.5 KB db_connection, host_config, user_config
deletedbibs Deleted Bibs Report 2026-03-01 32.2 GB db_connection, host_config, password, user_config
dupeitembarcodes Duplicate Barcodes 2026-03-06 2.7 MB db_connection, host_config, user_config
excusedlossbookclub Excused Lost Bookclub 2026-03-01 569.8 KB db_connection, host_config, user_config
maindisplays Shelflist - Main Displays 2026-03-02 173.8 KB db_connection, host_config, user_config
newdvds New DVDs 2026-03-01 84.9 MB db_connection, host_config, user_config
newlongintransit Shelflist - New Long In-Transit 2026-02-15 99.8 KB db_connection, host_config, user_config
nolongernew Shelflist - No Longer New 2026-02-15 87.5 KB db_connection, host_config, user_config
olddvds Old DVDs 2026-03-01 40.7 MB db_connection, host_config, user_config
oldlongintransit Shelflist - Old Long In-Transit 2026-02-15 100.2 KB db_connection, host_config, user_config
removedupfines Remove Duplicate Fines 2026-03-06 210.4 KB db_connection, host_config, user_config
turn13 Turn 13 2026-03-06 34.7 KB db_connection, host_config, user_config
turn18 Turn 18 2026-03-06 63.9 KB db_connection, host_config, user_config
unpaidmanualcharges Unpaid Manual Charges 2026-03-01 18.2 MB db_connection, host_config, user_config
userdata User Data Entry Error Report 2026-03-06 18.6 MB db_connection, host_config, user_config

Inactive-Recent (2 reports, 410.6 KB, 2,926 files)

Last run 2020-2025 -- review with stakeholders before archiving

Report Full Name Last Run Size Credentials
generic Generic Report 2021-08-16 87.3 KB db_connection, host_config, user_config
multipleintransitmessages Multiple Intransit Messages 2023-07-26 323.3 KB db_connection, host_config, user_config

Inactive-Legacy (6 reports, 3.1 GB, 541 files)

Last run before 2020 -- archive only, low priority

Report Full Name Last Run Size Credentials
autoextend Expr Date Auto Extend - cohort: morning 2014-01-10 23.6 MB db_connection, host_config, user_config
birthdatesync Birthdate Sync 2014-01-24 102.7 KB db_connection, host_config, user_config
holdsnocopies 2019-06-22 19.9 MB db_connection, host_config, user_config
novelistisbnexport Novelist ISBN Export 2017-08-27 3.0 GB db_connection, host_config, user_config
replacelabels Replacement Labels 2015-08-13 421.9 KB none
slintransit Shelflist - In-Transit 2017-12-18 40.0 KB db_connection, host_config, user_config

Obsolete (21 reports, 488.4 MB, 840 files)

No output found or commented out -- archive as historical record

Report Full Name Last Run Size Credentials
90dayunfilledholds never 164.3 KB db_connection, host_config, user_config
activeusers never 20.8 KB db_connection, host_config, user_config
boopsie never 4.0 KB none
cleanup never 674.0 KB db_connection, host_config, user_config
collectiondetail never 4.4 MB db_connection, host_config, user_config
collectionhq never 2.3 KB none
complimentarycards never 56.1 MB none
educator_outpatron_renewals never 3.0 KB db_connection
fineamnesty never 33.5 MB db_connection, host_config, user_config
gap_finder never 14.3 KB db_connection, host_config, user_config
intransitstats In-Transit Stats never 1.7 MB db_connection, host_config, user_config
knownissues never 195.5 KB none
labels never 17.2 MB db_connection
old_perl_scripts never 70.7 MB none
orangeboy never 135.9 MB db_connection, host_config, user_config
patron_update never 51.2 KB password
renewaldata never 36.4 KB db_connection, host_config, secret, user_config
reporter never 251.3 KB password
scanhistory never 163.4 MB none
systemwideholds never 4.2 MB db_connection, host_config, user_config
teleforms never 1.4 KB db_connection, host_config, secret, user_config

Historical Directories

These are not report directories but contain historical data:

  • Symphony_Hist: 12.7 GB -- archive as historical record
  • Symphony_Bincustom: 113.8 MB -- archive as historical record
  • webpac: 67.3 MB -- archive as historical record

Credential Rotation Requirements

Total credential references found: 316

Type Count Action
host_config 105 Review
user_config 100 Review
db_connection 76 Rotate immediately
password 33 Rotate immediately
secret 2 Review

All .cfg files contain plaintext credentials and must NOT be migrated as-is.

Size Breakdown

Total ilsaux size: 49.6 GB

Category Size Percentage
Active (critical + other) 32.7 GB 65.9%
Inactive (recent + legacy) 3.1 GB 6.2%
Obsolete 488.4 MB 1.0%
Non-report dirs 13.4 GB 27.0%

Migration Priorities

Priority 1: Active-Critical (Shelf List Reports)

  1. slitemdata -- Item Data Inconsistency Report
  2. sllastcopymissing -- Shelflist - Last Copy Missing
  3. slmainmissing -- Shelflist - Main Missing
  4. slmissing -- Shelflist - Missing
  5. slonorder -- Shelflist - Circulated On-Order
  6. slrepair -- Shelflist - Repair

These reports are actively running and serve shelf list operations. Document fully, test migration, coordinate with staff.

Priority 2: Active (Other Reports)

  1. activeusersmonthly -- Active Users Count - Monthly
  2. bookclubrenewal -- Book Club Renewal Report
  3. deletedbibs -- Deleted Bibs Report
  4. dupeitembarcodes -- Duplicate Barcodes
  5. excusedlossbookclub -- Excused Lost Bookclub
  6. maindisplays -- Shelflist - Main Displays
  7. newdvds -- New DVDs
  8. newlongintransit -- Shelflist - New Long In-Transit
  9. nolongernew -- Shelflist - No Longer New
  10. olddvds -- Old DVDs
  11. oldlongintransit -- Shelflist - Old Long In-Transit
  12. removedupfines -- Remove Duplicate Fines
  13. turn13 -- Turn 13
  14. turn18 -- Turn 18
  15. unpaidmanualcharges -- Unpaid Manual Charges
  16. userdata -- User Data Entry Error Report

Running in production. Evaluate each for continued need.

Priority 3: Inactive-Recent

Review with stakeholders. Some may need reactivation, others can be archived.

Priority 4: Legacy & Historical

Archive for reference. No migration needed.

Recommendations

  1. Credential rotation: All plaintext credentials must be rotated before any migration
  2. Config modernization: Replace Config::Simple .cfg with environment variables or vault
  3. Consolidate git: Merge per-report repos into monorepo for easier management
  4. Archive Symphony data: The 13 GB of historical Symphony logs can be compressed and cold-stored
  5. Document sl-reports first: These are the highest-value, most-used reports
  6. Test framework: The generic-cron.sh pattern is sound but should be modernized (systemd timers, structured logging)
Raw
framework--config-file-format.md

Config File Format (.cfg)

Reports use Config::Simple to read .cfg files. Format: key value or key=value (one per line).

Common Configuration Keys

Key Used By (count) Description
SierraModulesLocation 43 Path to Sierra:: Perl modules
DatabaseHost 38 Database or server hostname
DatabasePort 38 Database name
DatabaseUser 38 Database name
DatabasePass 38 Database name
OutputDir 16 Output file path
FTPUser 16 Database or service username
FTPPass 16 FTP-related setting
FTPHost 15 Database or server hostname
PatronAPIHost 5 Database or server hostname
CsvFilePrefix 3 Output file path
EmailRecipient 2
PDFOutputDir 2 Output file path
PatronApiHost 2 Database or server hostname
FtpHost 1 Database or server hostname
FtpUser 1 Database or service username
FtpPass 1 FTP-related setting
HostName 1 Database or server hostname
PatronAPIUser 1 Database or service username
PatronAPIPass 1
FTPServer 1 Database or server hostname
FTPDir 1 FTP-related setting
EmailFrom 1
PatronFilePrefix 1 Output file path
CircFilePrefix 1 Output file path
ClientKey 1
ClientSecret 1
XlsxFilePrefix 1 Output file path

Security Note

Many .cfg files contain plaintext credentials (database passwords, FTP credentials). These are NOT documented here and must be rotated as part of any migration. See credential-locations.csv for an inventory of affected files.

Raw
framework--generic-cron-framework.md

Generic Cron Framework

File: Reports/generic/generic-cron.sh Lines: 154

Overview

The generic-cron.sh script is the execution framework for all ilsaux reports. Each report has a thin wrapper (<name>-cron.sh) that sets variables and sources this script.

Execution Flow

  1. Variable Setup -- Date variables (TODAY, WEEKAGO, MONTHAGO, YEARAGO)
  2. Defaults -- Sets REPORTNAME, LOGFILE, JSONFILE, KEEPPERIOD, LINK if not provided by wrapper
  3. Run Report -- cd to report dir, run perl ./$SOURCEFILE >> $LOGFILE-$TODAY.txt
  4. JSON Metadata -- Creates timestamped JSON with fullName, name, date, timeStarted, timeFinished, logFile, link
  5. Cleanup -- Deletes old log files and JSON based on KEEPPERIOD (WEEK/MONTH/YEAR)
  6. MESA Integration -- Copies JSON + log to /var/www/html/mesa/, cleans old files, rebuilds index

Required Variables (set by wrapper)

Variable Required Default Description
REPORTNAME Yes generic Directory name and base filename
FULLNAME Yes Generic Report Human-readable name for JSON/MESA
SOURCEFILE Yes SierraGenericReport.pl Perl script filename
LINK No nil URL for the report output
KEEPPERIOD No YEAR Retention: WEEK, MONTH, or YEAR
LOGFILE No $REPORTNAME-log Log file basename
JSONFILE No $REPORTNAME JSON metadata basename

Cron Wrapper Pattern

Every report follows this pattern:

#!/bin/bash

REPORTNAME=slmainmissing
FULLNAME="Shelflist - Main Missing"
SOURCEFILE=SierraShelfListMainMissing.pl
LINK="http://[REDACTED-HOST]/ils/shelflists/mainmissing.asp"
KEEPPERIOD=MONTH

source ~/Reports/generic/generic-cron.sh

JSON Metadata Format

{
  "fullName": "Shelflist - Main Missing",
  "name": "slmainmissing",
  "date": "2026-01-15",
  "timeStarted": "1737000000",
  "timeFinished": "1737000300",
  "logFile": "slmainmissing-log-20260115.txt",
  "link": "http://[REDACTED-HOST]/ils/shelflists/mainmissing.asp"
}

MESA Dashboard Integration

  • JSON metadata copied to /var/www/html/mesa/finished/
  • Log files copied to /var/www/html/mesa/logs/
  • json-wn.pl generates /var/www/html/mesa/upcoming.json
  • json-index.pl generates /var/www/html/mesa/finished/index.json
  • Old MESA files cleaned after 32 days

Retention Periods

Period Log Cleanup JSON Cleanup
WEEK 7 days 8 days
MONTH 30 days 32 days
YEAR 365 days 366 days

Historical Note

The script contains commented-out FTP code that previously transferred files to [REDACTED-HOST]. This was replaced by direct file copy to the MESA web directory on the same server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment