How Redis persist data

Two methods can be used together, separately, or not at all in some circumstances.

Snapshotting

snapshotting that takes the data as it exists at one moment in time and writes it to disk.

so the configuration will be:

save 60 1000 # how often to perform an automatic snapshot
stop-writes-on-bgsave-error no # whether to compress the snapshot
rdbcompression yes # whether to keep accepting writes on failure
dbfilename dump.rdb # what to name the snapshot on disk
dir /var/lib/redis

Methods to initiate a snapshot:

  1. BGSAVE command: fork sub-process and do snapshot(expect windows)

  2. SAVE command: stop world snapshot generate

  3. save 60 10000 config will trigger a BGSAVE operation if 10,000 writes have occurred within 60 seconds

  4. SHUTDOWN command or it receives a standard TERM signal, will wait to Save finish

  5. Receive Sync command in replication master

You must remember that if a crash were to happen, you’d lose any data changed since the last snapshot when using snapshots. The question should be “how much time we’re willing to lose if something crashes between dumps”.

Sometimes, to recover from data loss, we need to know what we lost in the first place. To know what we lost, we need to keep a record of our progress while processing logs.

BGSAVE’s fork mechanism may cause high memory occupation(although there is COW), sometimes maybe choose SAVE.

AOF

AOF, or append-only file, works by copying incoming write commands to disk as they happen.

appendonly no # whether to use it
appendfsync everysec # how often to sync writes to disk
no-appendfsync-on-rewrite no # whether to sync during AOF compaction
auto-aof-rewrite-percentage 100 # and how often to compact the AOF
auto-aof-rewrite-min-size 64mb
dir /var/lib/redis

It writes OS buffer and sync by appendfsync:

  • alwaysEvery: write command to Redis results in a write to disk. This slows Redis down substantially if used.

  • everysecOnce: per second, explicitly syncs write commands to disk.

  • no: let the operating system control syncing to disk.

To solve the growing AOF problem, we can use BGREWRITEAOF, which will rewrite the AOF to be as short as possible by removing redundant commands. BGREWRITEAOF works similarly to the snapshotting BGSAVE: performing a fork and subsequently rewriting the append-only log in the child, even worse, because AOF can grow to be many times the size of a dump (if left uncontrolled), when the AOF is rewritten, the OS needs to delete the AOF, which can cause the system to hang for multiple seconds while it’s deleting an AOF of tens of gigabytes.

Misc