Skip to content

feat: Configuration backup/restore system with secure test authentication#41

Merged
patchmemory merged 10 commits intomainfrom
feat/config-backup-restore
Feb 8, 2026
Merged

feat: Configuration backup/restore system with secure test authentication#41
patchmemory merged 10 commits intomainfrom
feat/config-backup-restore

Conversation

@patchmemory
Copy link
Owner

@patchmemory patchmemory commented Feb 8, 2026

Summary

Implements complete configuration backup/restore functionality with zip-based file backups and comprehensive test authentication infrastructure.

Key Features

  • ✅ Zip-based Backup System: Complete file-based backups (SQLite DBs, .env, etc.)
  • ✅ Professional UI: Modal interface for viewing/managing/restoring backups
  • ✅ Direct Downloads: Clickable backup files with secure static file serving
  • ✅ Restore & Delete: Confirmation dialogs for safe backup management
  • ✅ Security First: Minimal public routes, proper authentication everywhere
  • ✅ All Tests Passing: 414/414 tests passing (100%)

Implementation Details

Backend (scidk/core/backup_manager.py):

  • Uses SQLite backup API for consistent database snapshots
  • Creates zip archives with metadata (timestamp, reason, created_by)
  • Supports pre-restore backups for safety
  • Proper cleanup and error handling

API Endpoints (scidk/web/routes/api_settings.py):

  • GET /api/settings/export - Download backup as zip
  • POST /api/settings/import - Upload and restore from zip
  • GET /api/settings/backups - List all available backups
  • DELETE /api/settings/backups/<id> - Delete specific backup
  • GET /api/backups/<filename> - Static file serving (with path traversal protection)

UI (scidk/ui/templates/index.html):

  • Export button - downloads backup immediately
  • Import button - file upload dialog
  • View Backups button - opens modal with table of all backups
  • Modal features: Download links, Restore button, Delete button, all with confirmations

Test Infrastructure (tests/conftest.py):

  • New authenticate_test_client() helper for tests that create their own app
  • Auto-authentication in client fixture when auth is enabled
  • Updated 14+ test files to use proper authentication
  • Security maintained - only /api/health remains public

Security Considerations

  • ✅ Backup files served with path traversal protection
  • ✅ Authentication required for all backup operations
  • ✅ Only legitimate public route: /api/health (for health checks)
  • ✅ Session-based auth using existing auth manager
  • ✅ Tests use proper authentication, no security bypasses

Test Results

414 passed, 2 skipped in 110s
  • All unit tests passing
  • All integration tests passing
  • 2 E2E tests skipped (require BASE_URL)
  • No failures

User Experience

  1. Click "Export Configuration" → backup downloads immediately
  2. Click "View Backups" → see table of all backups with metadata
  3. Click filename → download that backup
  4. Click "Restore" → confirm and restore (creates pre-restore backup first)
  5. Click "Delete" → confirm and remove backup file
  6. Click "Import Configuration" → upload zip to restore

Related Task

Completes task:security/config/config-export-import (RICE score: 24)

🤖 Generated with Claude Code

patchmemory and others added 10 commits February 8, 2026 12:01
@patchmemory @claude
feat(security): implement configuration export/import functionality
6c8963e 
- Add ConfigManager class for exporting and importing all SciDK settings
- Support for complete or selective section export (general, neo4j, chat, interpreters, plugins, rclone, integrations, security)
- Sensitive data handling: option to exclude or include passwords/API keys in export
- Automatic backup creation before import operations
- Configuration validation and preview of changes before applying
- Backup management: create, list, get, restore, and delete backups
- Audit trail with timestamps, reason, created_by, and notes for all backups

API Endpoints:
- GET /api/settings/export - Export configuration as JSON
- POST /api/settings/import/preview - Preview changes without applying
- POST /api/settings/import - Import configuration with validation
- GET /api/settings/backups - List all backups
- GET /api/settings/backups/:id - Get specific backup
- POST /api/settings/backups - Create manual backup
- POST /api/settings/backups/:id/restore - Restore from backup
- DELETE /api/settings/backups/:id - Delete backup

UI Updates:
- Add Export Configuration button in Settings > General
- Add Import Configuration button with file picker
- Add View Backups button to list recent backups
- Import preview shows diff of changes before applying
- Automatic page reload suggestion after successful import
- Success/error status messages for all operations

Tests:
- 19 unit tests for ConfigManager export/import/backup operations
- E2E tests for full export-import-restore cycle
- API endpoint tests for all configuration management operations

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@patchmemory
chore(dev): update submodule pointer for completed task
fc6f236
@patchmemory
fix(ui): add import/backup buttons to actual settings page in index.html
1313040 
The settings were migrated to index.html (home page), but the configuration
import/export UI was only added to the old settings.html template. This adds
the complete UI with all three buttons and JavaScript handlers to index.html.

Changes:
- Add Import Configuration and View Backups buttons to General section
- Replace mock export implementation with real API calls
- Add import functionality with preview and confirmation
- Add backup viewing functionality
- Update section title from 'Configuration Export' to 'Configuration Management'
@patchmemory
feat(tests): add cleanup function for test users in conftest
f86ec8f 
Adds _cleanup_test_users_from_db() to automatically remove test users
from the database before test runs, preventing accumulation of test
users that show up in the UI when running scidk-serve after tests.

The cleanup function:
- Removes users matching test patterns (test%, Test%, demo%, temp%, etc)
- Removes users created by 'system' with test-like usernames
- Cleans up associated auth records (sessions, failed attempts, audit logs)
- Follows the same pattern as existing cleanup functions for scans and labels

This prevents issues where test users like 'testuser' remain in the
production database and appear in the user management UI.
@patchmemory
fix(ui): add authentication credentials to config export/import API c…
9371d7f 
…alls

The export/import/backup API endpoints require authentication but the
JavaScript fetch calls were not including credentials, causing 401 errors
when auth is enabled.

Changes:
- Add 'credentials: same-origin' to all fetch requests
- Improve error messages to show actual error from API response
- Parse JSON response before checking status for better error reporting

This fixes the 'Export failed: Export failed' error when trying to export
configuration with authentication enabled.
@patchmemory
refactor(config): replace JSON export with complete zip-based backup …
f9aee19 
…system

Complete redesign of configuration export/import to use file-based backups
instead of trying to serialize individual settings. This is much simpler,
more reliable, and captures everything.

New BackupManager:
- Creates zip archives of all important files (databases, .env, etc.)
- Uses SQLite backup API for consistent database snapshots
- Includes metadata (timestamp, reason, created_by, notes)
- Supports listing, restoring, and deleting backups
- Human-readable file sizes

Changes:
- Add backup_manager.py with complete zip-based backup system
- Replace /api/settings/export to return zip file instead of JSON
- Update /api/settings/import to accept zip file upload (multipart/form-data)
- Update JavaScript to download zip files and upload them for restore
- Change file input from .json to .zip
- Remove complex JSON serialization/deserialization logic
- No more schema mismatch issues with table_formats or other tables

Benefits:
- Captures complete state including all databases
- No schema mismatch errors
- Simpler implementation (no field-by-field export/import)
- Automatic backups before restore
- Works with any future schema changes

The old config_manager.py remains but is no longer used by the UI.
@patchmemory
fix(backup): add missing imports and improve user handling in export …
db003cf 
…endpoint

Fixes:
- Add missing 'g' and 'send_file' imports to api_settings.py
- Add 'os' import for file operations
- Fix g.current_user access to use hasattr() check
- Remove redundant 'from flask import send_file' inside function
- Improve View Backups display to show filename, size, and better formatting
- Fix timestamp parsing to handle ISO format (not Unix timestamp)

These fixes resolve the 500 error when exporting configuration.
@patchmemory @claude
feat(ui): improve backup management UX with clickable modal interface
0ec9f6a 
- Replace simple alert() with comprehensive modal for viewing backups
- Add table showing Date, Filename, Size, Reason, By, and Actions columns
- Add download links on backup filenames
- Add Restore and Delete buttons for each backup with confirmation dialogs
- Add static file serving route for /backups/<filename> to enable direct downloads
- Improve error handling in export to avoid false error messages
- Address user feedback: "option to just click on the available backups"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@patchmemory @claude
fix(tests): add authentication helper for tests without breaking secu…
8ab5058 
…rity

- Add authenticate_test_client() helper in conftest for tests that create their own app/client
- Keep PUBLIC_ROUTES minimal (only /api/health legitimately needs to be public)
- Fix test_interpreters_* tests to use authentication helper
- Fix test_interpreters_registry_api to use client fixture
- Do NOT weaken security by making more API routes public

379 tests passing, 35 failures remaining (was 39)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@patchmemory @claude
fix(tests): fix all remaining test authentication issues
2cb0b21 
- Batch fix 13+ test files to use authenticate_test_client helper
- Fix make_client_with_rclone helper to return authenticated client
- Fix state_backend_toggle tests that create multiple apps
- Keep security intact - NO additional public routes added

✅ All 414 tests now passing (only 2 skipped E2E tests)
✅ Security maintained - only /api/health is public (for legitimate health checks)
✅ Backup management feature complete and working

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@patchmemory patchmemory merged commit 9b7296b into main Feb 8, 2026
1 check passed
@patchmemory patchmemory deleted the feat/config-backup-restore branch February 8, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@patchmemory