# Data Safety, Durability & Recovery Architecture — SyntricDB

For any production database engine, **performance without data durability is a liability**. SyntricDB is engineered with a **Safety-First Core Architecture** that guarantees full ACID compliance, crash resilience under power failure, and zero-loss crash recovery across SQL tables, key-value stores, and HNSW vector indexes.

This document details SyntricDB's Write-Ahead Logging (WAL) mechanics, Multi-Version Concurrency Control (MVCC), fault injection testing, and split-brain defenses under network partitions.

---

## 1. Write-Ahead Logging (WAL) Architecture

SyntricDB enforces the fundamental rule of database durability: **No data page is mutated on disk until the corresponding transaction record has been sequentially committed to physical storage in the Write-Ahead Log (WAL).**

```
 [Client Transaction]
          │
          ▼
┌───────────────────┐      Sequential Write & fsync
│ Write-Ahead Log   │ ──────────────────────────────► [ Disk Physical Log Segment ]
└───────────────────┘
          │ (Commit Ack)
          ▼
┌───────────────────┐      Async Flush
│ In-Memory MemTable│ ──────────────────────────────► [ Immutable SSTable Data File ]
└───────────────────┘
```

### Key WAL Specifications
1. **Append-Only Log Segments**: All SQL `INSERT`/`UPDATE`/`DELETE`, RESP `SET`/`HDEL`, and Vector insertions are appended as sequential log entries with strict 64-bit Log Sequence Numbers (LSN).
2. **CRC32 Checksum Validation**: Every record header includes a 32-bit CRC checksum. During startup recovery, corrupted or incomplete records resulting from sudden power loss are instantly detected and safely truncated without corrupting prior state.
3. **Fsync Durability Modes**:
   * `fsync_mode = ALWAYS` *(Maximum Security)*: Issues synchronous OS `fsync()` system call on every single committed transaction block (RPO = 0).
   * `fsync_mode = EVERY_SEC` *(Balanced Default)*: Flushes log buffers to disk every 1,000ms, providing sub-millisecond throughput with a maximum bounded loss of 1 second during total server destruction.
   * `fsync_mode = OS_MANAGED`: Relies on OS kernel page cache flushing for benchmark-only evaluation environments.

---

## 2. ACID Transactions & Concurrency Control

SyntricDB unifies relational tables, vector indexes, and KV pairs under a unified transactional storage engine powered by **MVCC (Multi-Version Concurrency Control)**.

### Isolation Guarantees
* **Read Committed**: Readers view data committed before the read query began. Non-blocking snapshot reads eliminate read/write lock contention.
* **Serializable (Snapshot Isolation)**: Optimistic Concurrency Control (OCC) tracks read-set and write-set version ranges. If a concurrent write conflict occurs during commit validation, the transaction aborts with a retryable error (`3A000: transaction_rollback`).

### Unified Atomicity Across Engine Primitives
In SyntricDB, inserting a SQL record containing vector embeddings updates both the relational schema and the HNSW graph index **in a single atomic transaction**. If graph insertion fails (e.g., dimension mismatch), the entire SQL insertion is rolled back instantly in memory and logged as aborted in the WAL.

---

## 3. Crash Recovery & Resilience Sequence

When SyntricDB boots after a graceful shutdown or sudden crash (e.g., `kill -9`, power loss, kernel panic), it executes an automatic 4-phase recovery protocol:

```
[System Boot]
     │
     ▼
Phase 1: Log Segment Inspection & LSN Bounding
     │
     ▼
Phase 2: Checksum Verification & Partial-Write Truncation
     │
     ▼
Phase 3: REDO Replay (Rebuilding MemTables & Unflushed HNSW Graph Nodes)
     │
     ▼
Phase 4: UNDO Cleanup (Rolling back uncommitted transaction IDs)
     │
     ▼
[Database Ready — Port 5432 & 6379 Active]
```

1. **Phase 1 (LSN Bounding)**: Reads the last persistent checkpoint file to locate the exact LSN offset where disk pages were last safely flushed.
2. **Phase 2 (CRC Validation)**: Scans log entries past the checkpoint LSN. If an unwritten or torn write (incomplete sector) is encountered at the tail end of the WAL file, it is automatically truncated to the last valid CRC32 block.
3. **Phase 3 (REDO Replay)**: Replays all committed transactions from the WAL into active memory structures, recreating index graphs and table rows up to the exact instant of failure.
4. **Phase 4 (UNDO Rollback)**: Purges any active transactions that were still open when the server abruptly stopped, ensuring 100% data consistency.

---

## 4. Network Partition Defenses (Raft Consensus & Split-Brain Prevention)

In distributed multi-node deployments, SyntricDB utilizes **Raft Consensus Protocol** for leader election and log replication.

* **Majority Quorum Enforcement**: Write operations require acknowledgement from a strict majority quorum (`N/2 + 1`) of cluster nodes before returning success to the client application.
* **Split-Brain Immunity**: In the event of a network partition splitting a 5-node cluster into `{Node1, Node2}` and `{Node3, Node4, Node5}`, the minority partition automatically rejects writes because it cannot achieve quorum. The majority partition continues serving traffic cleanly.
* **Leader Lease & Monotonic Terms**: Term numbers prevent stale leaders from accepting writes after partition healing.

---

## 5. Durability & Fault-Injection Testing Framework

To rigorously validate our durability promises, SyntricDB undergoes automated continuous fault-injection testing inspired by Jepsen testing methodology:

### Test Suites Included in Repository
* **Sudden Process Termination (`KillNineTest.java`)**: Spawns thousands of concurrent write threads, issues random `kill -9` signals to the database JVM process, re-launches SyntricDB, and verifies zero missing or corrupt transactions.
* **Disk Disk-Full & Power Outage Simulations**: Injects IOExceptions mid-write to verify WAL transactional rollbacks and checksum boundary enforcement.
* **Network Flapping & Partition Injection (`RaftPartitionTest.java`)**: Simulates asymmetric network latency, dropped TCP packets, and sudden node isolates to guarantee linearizability and zero stale writes.

---

## 6. Backup & Point-in-Time Recovery (PITR)

SyntricDB supports hot backup without stopping engine execution:

* **Physical Hot Backups**: Issue `SYNTRICDB CHECKPOINT 's3://my-backups/daily/'` via SQL or CLI. SyntricDB creates an instantaneous read-only snapshot using copy-on-write file handles.
* **Point-in-Time Recovery (PITR)**: Combine base snapshots with archived WAL log segments to recover database state to any precise nanosecond in history.