Was this page useful?

Document Sync & Backup Architecture

Purpose

This page explains how Club documents and website files are:

  • synchronised between IONOS HiDrive and the VPS
  • audited for consistency and usage
  • backed up independently of the website

The aim is to provide a simple, transparent, and recoverable system.

Core Principles

  • Single source of truth (documents live in one place)
  • Separation of functions (documents vs backups vs audit)
  • No hidden processes
  • Recoverability over complexity
  • Visibility for administrators

System Overview

1. Document Storage (Authoritative Source)

IONOS HiDrive (Shared space):

/WEBSYNC/CLUBDOCS/
  AGMS/
  Policies/
  Minutes/
  Notices/

Managed via:

  • HiDrive Web Interface

This is the master copy of all Club documents.

2. Website Document Mirror (Grav VPS)

/httpdocs/user/data/clubdocs/

  • Local mirror of IONOS documents
  • Used by Grav to serve files to members
  • Not edited directly on the VPS

3. Sync & Audit Logs (VPS)

/httpdocs/user/data/logs/clubdocs-audit/

  • pull_clubdocs.log → latest sync run
  • clubdocs-audit-report.txt → latest audit results

These are overwritten on each run and represent the current system state.

4. VPS Backup Storage (IONOS Personal Space)

/users/comradesclub/vpsbackup/

  • Stores backups of website files
  • Separate from shared document storage
  • Used for recovery only

Data Flow

A. Document Sync (IONOS → VPS)

IONOS (source) → VPS (mirror)

  • Tool: sftp (batch mode)
  • Direction: one-way
  • Authority: IONOS

Behaviour:

  • full directory structure is traversed each run
  • files are downloaded as required
  • local mirror is refreshed to match source

Note:

This is not a true incremental sync in the rsync sense, but performs efficiently for the current scale.

B. Link Audit (VPS)

VPS local mirror → Grav pages

  • Tool: custom audit script
  • Purpose: validate document usage

Checks:

  • files not referenced by any page
  • links pointing to missing files

Produces:

clubdocs-audit-report.txt

C. Website Backup (VPS → IONOS)

VPS → IONOS (backup)

  • Tool: rsync
  • Direction: one-way
  • Authority: VPS runtime state

Behaviour:

  • only changed files are transferred
  • efficient incremental backup
  • full mirror maintained remotely

Scripts

1. Document Sync + Audit Script

Location:

~/scripts/sync/pull_clubdocs_v1.0.sh

Purpose:

  • Pull documents from IONOS
  • Run link audit
  • Produce current status logs

Key features:

  • SSH key authentication (no password)
  • Full directory sync via SFTP
  • Integrated audit execution
  • Single log output per run

2. Link Audit Script

Location:

~/scripts/sync/clubdocs-audit_v1.0.sh

Purpose:

  • Analyse document usage within Grav

Outputs:

  • list of all files in local mirror
  • unreferenced files
  • missing file links

3. Backup Script

Location:

~/scripts/backup/backup_httpdocs_to_hidrive_v1.0.sh

Purpose:

  • Back up website files to IONOS personal storage
  • Back up Plesk local backup dumps to a separate IONOS personal storage folder

Key features:

  • rsync incremental transfer
  • efficient after initial run
  • independent recovery path
  • separate targets for website files and Plesk full-server dumps
  • skips Plesk dumps if a Plesk backup appears to still be running

Outputs:

  • /users/comradesclub/site-backups/coulsdonclubmembers.info/httpdocs/
  • /users/comradesclub/site-backups/coulsdonclubmembers.info/plesk-dumps/

4. Codex Hub Live Copy Script

Location:

~/scripts/export_localhost_sync_v1.0.sh

Purpose:

  • Copy the essential live website files into codex-hub
  • Keep a current working copy available for comparison with localhost
  • Exclude live accounts, sessions, cache, temporary files, and runtime logs
  • Write a change report each time the copy is refreshed

Outputs:

  • /var/www/vhosts/coulsdonclubmembers.info/codex-hub/LOCALHOST_SYNC_MANIFEST.txt
  • /var/www/vhosts/coulsdonclubmembers.info/codex-hub/LOCALHOST_SYNC_CHECKSUMS.txt
  • /var/www/vhosts/coulsdonclubmembers.info/codex-hub/LOCALHOST_SYNC_CHANGE_REPORT.txt
  • /site-admin/codex-hub-sync

The script copies live files into Codex Hub only. It does not deploy code to the live website.

5. Codex Hub Cron Installer

Location:

~/scripts/sync/install_codex_hub_cron_v1.0.sh

Purpose:

  • Install or refresh the scheduled Codex Hub sync job
  • Manage only its own marked cron block
  • Leave unrelated cron jobs untouched

Default schedule:

Every 30 minutes.

Manual command:

~/scripts/sync/install_codex_hub_cron_v1.0.sh

Optional hourly schedule:

~/scripts/sync/install_codex_hub_cron_v1.0.sh "0 * * * *"

Security Model

  • No passwords stored in scripts
  • SSH key authentication used for:
    • SFTP document sync
    • rsync backups

HiDrive separation:

  • Shared (Common) → documents
  • Personal → backups

Operational Model

Document Updates

  1. Documents uploaded to IONOS
  2. Sync script runs
  3. VPS mirror updated
  4. Audit runs
  5. Grav serves latest version

Website Backup

  1. Backup script runs
  2. Files copied to IONOS personal storage
  3. Backup available independently

Important Behaviour

Sync Behaviour

  • Each run refreshes the full structure
  • Directory traversal occurs every time
  • Transfer volume remains low in practice

Audit Behaviour

  • Audit reflects current state only
  • No historical tracking retained
  • Designed for operational clarity

Folder Structure

  • Entire folder tree is mirrored
  • New folders appear automatically
  • No manual configuration required

Key Decisions

IONOS as source of truth

  • centralised document management
  • accessible to multiple users
  • independent of VPS

Separate document sync, audit, and backup

  • avoids confusion
  • improves reliability
  • allows independent recovery

Overwrite logs each run

  • reflects current system state only
  • avoids accumulation of historical noise
  • supports simple monitoring

Use of SFTP for documents

  • aligns with HiDrive shared space
  • ensures visibility across tools
  • avoids rsync limitations in shared space

Limitations

  • Sync does not delete removed files from VPS
  • SFTP sync is not fully incremental
  • Backup does not currently include database

Future Improvements

  • controlled deletion workflow
  • database backup integration
  • optional alerting

Summary

This system provides:

  • a clear document pipeline
  • an integrated audit mechanism
  • a separate and reliable backup system

It is designed to be:

  • understandable
  • maintainable
  • resilient
  • suitable for long-term handover