Skip to content

db sanitize: scrub PII and trim databases for development - #76

Open
dreispt wants to merge 17 commits into
masterfrom
feature/db-sanitize
Open

db sanitize: scrub PII and trim databases for development#76
dreispt wants to merge 17 commits into
masterfrom
feature/db-sanitize

Conversation

@dreispt

@dreispt dreispt commented Oct 29, 2026

Copy link
Copy Markdown
Owner

Summary

  • osh db sanitize scrubs a database for development in two steps —
    anonymize rewrites PII to deterministic placeholders (partners,
    users, bank accounts, employees; the bundled
    osh/data/anonymize/ profile) and trim deletes low-value rows
    (the bundled osh/data/trim/NN_<table>.sql scripts: logs, mail/SMS
    queues, audit trails, old chatter). Both run by default;
    --no-anonymize/--no-trim pick one.
  • Layered script resolution replaces copy-at-init: neutralize,
    anonymize and trim scripts resolve at run time from
    osh/data/<kind>/ < ~/.config/osh/<kind>/ < .osh/<kind>/, merged
    by filename, later layers overriding. osh init stops copying —
    centrally maintained fixes reach every project immediately, and
    existing .osh/neutralize/ copies keep working as overrides.
  • osh backup restore --anonymize --trim [--keep-days N] runs the same
    steps after neutralization; --script/--skip select scripts on
    both commands, osh db sanitize --list shows the resolved set.
  • Script variables: --var NAME=VALUE and --keep-days inject
    osh.<name> session settings readable via current_setting() —
    even inside DO $$ blocks where psql :var can't reach. Ambient
    env vars reach scripts through the new psql(env=…) pass-through +
    \getenv; $OSH_KEEP_DAYS fills --keep-days.
  • Safety: target must exist, warns on non-neutralized databases,
    confirmation unless --force, --dry-run previews; reports
    pg_database_size before/after and reclaimed space.
  • docs/SANITIZE.md is now a user manual (design notes + roadmap kept
    as trailer sections).

Test plan

  • tests/test_db_sanitize.py — command-level runs on real databases:
    dry-run preview, force run, unneutralized warning, --var
    injection end-to-end, restore flag wiring
  • tests/test_sql_scripts.py — layered resolution, overrides,
    --script/--skip filtering, dry-run
  • tests/test_init.py + test_restore_plugin.py — no-copy init and
    neutralize migration regressions
  • Verified on a real Odoo 19 database (dry-run sizes + script
    resolution)

Assisted-by: Devin:SWE-2 High

dreispt added 17 commits October 29, 2026 08:56
@dreispt
docs: db sanitize design — anonymize + trim with layered SQL scripts
e5a3ab8 
`osh db sanitize` scrubs PII (anonymize) and deletes low-value rows
from big tables (trim), also exposed as `osh backup restore
--anonymize --trim`. SQL scripts resolve at run time from layered
sources — bundled < user (~/.config/osh) < project (.osh) — replacing
the copy-at-init model used by today's neutralize scripts, which
migrate to the same resolver. Every script-running command grows
--list/--script/--skip.

Assisted-by: Devin:SWE-2 High
@dreispt
core: resolve SQL scripts from layered sources instead of init-time c…
bc28d75 
…opies

Neutralize scripts were copied into .osh/neutralize/ at `osh init`, so
centrally maintained scripts in ~/.config/osh/ only reached a project
on re-init. Scripts now resolve at run time from layered sources —
bundled osh/data/<kind>/ < user ~/.config/osh/<kind>/ < project
.osh/<kind>/ — merged by filename and run in sorted order, with a
same-named file in a later layer overriding. Existing .osh/neutralize/
copies keep working as project-layer entries.

`osh init` stops copying; the bundled scripts move into
osh/data/neutralize/ under their resolved names. The layered resolver
also powers the script selection flags the sanitize feature adds.

Assisted-by: Devin:SWE-2 High
@dreispt
db_sanitize: add osh db sanitize and restore --anonymize/--trim
6f7e566 
`osh db sanitize` scrubs a database for development use in two steps —
anonymize rewrites PII to deterministic placeholders (the bundled
osh/data/anonymize/ profile), trim deletes low-value rows from big
tables (the bundled watchlist, overridable via [trim] in
.osh/config.toml). Both run by default; the database must exist and a
non-neutralized target warns and asks for confirmation.

`osh backup restore --anonymize --trim [--keep-days N]` runs the same
steps after neutralization. Script steps use the layered resolver;
`osh db sanitize --list` shows the resolved set and `--script`/`--skip`
select from it on both commands.

Assisted-by: Devin:SWE-2 High
@dreispt
docs: update for db sanitize and run-time script resolution
7037a98 
`osh init` no longer installs neutralize scripts (they resolve at run
time), the commands table lists `osh db sanitize`, GETTING-STARTED
mentions it, and the SANITIZE design tasks are checked off.

Assisted-by: Devin:SWE-2 High
@dreispt
db_sanitize: trim is bundled SQL scripts, driven by osh.keep_days
1c0d253 
The trim policies become real scripts under osh/data/trim/NN_<table>.sql
— same layered resolution, same guards, so users read and override them
like any other script. Statements inside DO blocks can't see psql
variables, so --keep-days/[trim] keep_days reach the SQL as
current_setting('osh.keep_days') via a SET prelude run_sql_scripts now
injects. --table selects scripts by table name; reclaimed is measured
on pg_database_size before/after.

Assisted-by: Devin:SWE-2 High
@dreispt
core: rename sqlscripts.py to sql_scripts.py
ca60e62 
Matches the codebase's snake_word module convention (cli_utils,
odoo_layout, format_detect).

Assisted-by: Devin:SWE-2 High
@dreispt
db_sanitize: let SQL scripts see env vars via psql \\getenv and OSH_K…
756d240 
…EEP_DAYS

psql() and run_sql_scripts() gain an env= pass-through into the
runtime process environment, so scripts can read ambient variables with
psql's \\getenv (psql 14+). keep_days now also resolves from
$OSH_KEEP_DAYS between the flag and the [trim] config value.

Assisted-by: Devin:SWE-2 High
@dreispt
db_sanitize: --var NAME=VALUE injects variables into SQL scripts
48b61fe 
`osh db sanitize --var domain=example.test` sets the osh.<name> session
settings before each script runs, so user and project scripts read them
with current_setting('osh.<name>') — inside DO blocks too. keep_days
flows through the same path.

Assisted-by: Devin:SWE-2 High
@dreispt
docs: SANITIZE.md as a user manual, design notes as trailer
6dccb9d 
Rewritten around usage — the steps, flags, script layers, variable
injection, bundled scripts, restore integration — with rationale,
rejected alternatives and deferred work kept under Design notes and
Roadmap at the end. Stale watchlist/policy wording updated to the
shipped script-based trim.

Assisted-by: Devin:SWE-2 High
@dreispt
db_sanitize: anonymize masks PII with x, keeps website
0439870 
Fields like phone, street or VAT are masked at their original length
instead of NULLed — the format survives, the data doesn't. website is
not PII and is left untouched.

Assisted-by: Devin:SWE-2 High
@dreispt
db_sanitize: split anonymize into identity and address levels
6d84516 
010_identity.sql carries names, emails, phones, logins and identifier
numbers — including OCA partner_firstname's firstname/lastname; running
it alone gives a partial scrub. 050_address.sql holds the street/city/
zip masks and the is_anonymized marker, so the flag only lands after
the full set.

Assisted-by: Devin:SWE-2 High
@dreispt
db_sanitize: guard age-trim scripts on the date column, not just the …
33d296d 
…table

mail_notification has no create_date in Odoo versions where
_log_access=False, so the DELETE failed mid-run. The guards now check
information_schema for the column too; notifications still get cleaned
via the cascade on mail_message. Also fixes stale help wording
(watchlist, old script name).

Assisted-by: Devin:SWE-2 High
@dreispt
db_sanitize: positional STEPS select which groups to run
2eb8ff2 
`osh db sanitize trim` runs only the trim group, `osh db sanitize
anonymize` only anonymize — validated names, unknown steps are a usage
error. Deselecting stays with --no-anonymize/--no-trim.

Assisted-by: Devin:SWE-2 High
@dreispt
db_sanitize: document STEPS values in --help
ea08173 
The usage line now shows [anonymize|trim] and the help text names the
two groups and how to deselect them.

Assisted-by: Devin:SWE-2 High
@dreispt
db_sanitize: sharper UX — group levels, per-table report, SQL preview
1a886b1 
- `osh db sanitize identity`/`address` select the anonymize levels;
  groups validate as anonymize/trim/identity/address
- The prompt names the plan ('anonymize + trim') and the report shows
  per-table before->after sizes; a zero-reclaimed run suggests
  --vacuum/'osh db vacuum'
- --dry-run prints the actual SQL each script would execute
- --table matches table stems (auditlog -> auditlog_log)

Assisted-by: Devin:SWE-2 High
@dreispt
db_sanitize: --table matches the covering script's stem either way
a78f9da 
--table selects the scripts named for a table — a script-name
convention, not a pg_class check. Matching now works bidirectionally:
'auditlog' selects 030_auditlog_log.sql, and 'auditlog_log_line'
selects it too since the script covers that table. Help text spells
out that --table picks scripts covering the named tables.

Assisted-by: Devin:SWE-2 High
@dreispt
db_sanitize: --table folds into --script with relaxed matching
2afcbc0 
--script/--skip now accept full filenames, stems, or the NN_<name>
tail (mail_message, identity), with prefix matching on the tail
(auditlog -> 030_auditlog_log.sql). That covers --table's job, so the
flag is gone — one selection mechanism instead of two.

Assisted-by: Devin:SWE-2 High
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@dreispt