SIGN IN SIGN UP
apache / superset UNCLAIMED

Apache Superset is a Data Visualization and Data Exploration Platform

0 0 150 TypeScript

feat(scripts): add --dirty-duplicates-pct to seed_junction_load.py

Extends the stress-test seed script with an optional duplicate-row
injection step, used to measure the empirical cost of the migration's
``_dedupe_by_min_id`` phase.

Usage: after running the normal seed at a given scale, add
``--dirty-duplicates-pct 5`` (or any non-zero value) to inject that
percentage of duplicate ``(fk1, fk2)`` rows into each non-UNIQUE
junction (slice_user, dashboard_user, dashboard_roles —
dashboard_slices is skipped because its UNIQUE constraint, present
both pre- and post-migration, rejects duplicates).

Pre-condition: requires the DB to be at the pre-migration revision
(33d7e0e21daa). The post-migration composite PK rejects duplicates,
so attempting to inject on the upgraded schema errors out.

Empirical result on MySQL @ 10M dashboard_slices + ~2.1M other
junction rows + 105K injected duplicates (5% on the 3 non-UNIQUE
tables):
  Upgrade time: 1m 36s vs clean baseline 1m 37s
  → dedupe cost is within measurement noise; the table-scan that
    the migration already performs dominates whether or not
    duplicates exist.

This empirically confirms what the cost-model predicted: the
``_dedupe_by_min_id`` GROUP BY scan is the dominant cost of that
phase, and the actual per-duplicate DELETE is negligible.

NULL-FK injection deliberately skipped — would require altering the
six non-UNIQUE FK columns from NOT NULL back to nullable (the
migration's downgrade keeps them NOT NULL by design), which adds
per-backend ALTER complexity for a code path that's structurally
identical in cost shape (DELETE WHERE col IS NULL is the same scan
shape as the dedupe scan).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
M
Mike Bridge committed
0a9fa1ac85371c149885290e36528a87cb5be166
Parent: 58a1a1a