[SPIKE] Adding all implementations for SqlServer, MySql and Postgresql persistence

[johnsimons]

SQL Persistence Implementation for ServiceControl

Summary

Implements SQL persistence layer for ServiceControl with support for SQL Server, PostgreSQL, and MySQL. Introduces Entity Framework Core-based implementation as an alternative to the existing RavenDB persistence.

Key Changes

Core Persistence Infrastructure (ServiceControl.Persistence.Sql.Core)

• Entity models and EF Core configurations for all ServiceControl data stores
• Base ServiceControlDbContextBase with 16 entity sets
• Abstract base classes for provider-specific implementations (BasePersistence, SqlPersisterSettings)
• Database migrator interface for schema management

Provider-Specific Implementations

• SQL Server - SqlServerDatabaseMigrator.cs:1
• PostgreSQL - PostgreSqlDatabaseMigrator.cs:1
• MySQL - MySqlDatabaseMigrator.cs:1
• Each with generated EF migrations and provider-specific DbContext

Data Store Implementations

All 19+ persistence interfaces implemented including:

• Message storage (ErrorMessageDataStore, BodyStorage)
• Retry management (RetryBatchesDataStore, RetryDocumentDataStore, RetryHistoryDataStore)
• Monitoring (MonitoringDataStore, CustomChecksDataStore)
• Configuration (EndpointSettingsStore, QueueAddressStore, NotificationsManager)

Testing

• Spiked using Testcontainers-based integration tests for all three SQL providers

Design Decisions

Data Modeling

• GUIDs for primary keys: Ensures global uniqueness and prevents ID conflicts across distributed instances
• Flat table structure: No foreign key relationships between tables to maximize flexibility and simplify queries
• JSON + denormalization hybrid: Complex nested objects stored as JSON with critical fields extracted to columns for indexing and efficient filtering
  • Example: FailedMessageEntity stores full ProcessingAttemptsJson but denormalizes LastProcessedAt, MessageType, ExceptionType for query performance
• Separate message body storage: Message bodies stored in dedicated MessageBodies table to keep primary tables lean and improve query performance
• Singleton entities: Some entities (e.g., RetryHistoryEntity, NotificationsSettingsEntity) use fixed IDs as they represent single configuration records

Query Optimization

• Strategic indexes on frequently-filtered columns (status, timestamps, endpoint names)
• Denormalized fields avoid JSON parsing in WHERE clauses
• Binary message bodies separated from metadata to reduce table bloat

Best Way to Review

1. Start with the abstractions (understand the architecture)

• BasePersistence.cs:1 - DI registration
• SqlPersisterSettings.cs:1 - Configuration model
• IDatabaseMigrator.cs:1 - Migration contract

2. Review the data model (entities represent the domain)

• Entities/ - 16 entity classes
• EntityConfigurations/ - EF configurations with indexes
• Pay attention to JSON columns vs denormalized fields in FailedMessageEntity.cs:1
• Note the message body separation in MessageBodyEntity.cs:1

3. Check one provider implementation (SQL Server recommended)

• SqlServerPersistence.cs:1
• SqlServerDbContext.cs:1
• Migrations in Migrations/

4. Review key data store implementations (focus on complex ones)

• ErrorMessageDataStore.cs:1 - Most complex queries, JSON deserialization logic
• RetryBatchesDataStore.cs:1 - State machine logic
• BodyStorage.cs:1 - Binary data handling in separate table

5. Spot-check tests (verify integration test coverage)

• SqlServerPersistenceTests.cs
• Similar for PostgreSQL and MySQL

Migration Notes

• EF migrations generated for all three providers with identical schema
• No automatic migration from RavenDB - this is a fresh persistence option
• Connection string and provider selection via configuration