Rehashing macros (#415)
* Working Macro to re-hash a hub-satellite * Added example execution * Added macro to rehash multiple satellites in one go * Added macro to rehash multiple satellites in one go * Added macros to rehash one or many hubs * Started macro to rehash a single link, still WIP * Working Macro for single link rehash * Macro to hash multiple links finished * Implemented MA Sat Rehash macros * Working macro for hashing entire RDV, but column drop not yet working * Added "drop_old_values" as parameter * Use existing rehashed key * started additional documentation inside macros * Documentation for all Macros * Added specific overwrite macros * changed macro call for rename_col * adjusted documentation * commented out jinja comments * removed default value * removed default key * removed description for drop_old_values * Changed folder structure and added single_entities macros for bigquery * Removed some unnecessary code and added dropping of old link and renaming of new link table * Changed Macro name to be adapter-specific * Fixed Macro to correctly handle multi source hashkeys * Changed logging logic to respect output_logs flag * Refactored rehash_single_ma_satellite macro for BiQuery, split by database and added top-level dispatcher * Added top-level macro for remaining single entities * fix: Fixed Typo in top-level Macros * Fix: Replace render() to avoid BigQuery backticks * feat: Added rehashing macro for nh_satellite in bigquery * feat: Added rehashing macro for standard satellite in bigquery * Rehash single hub * Rehash without temp table * simplifying copy statement * Rehash single link * Exclude unknown and error keys * Add rehash check * Rehash single fabric ma satellite * Add NULL = NULL for ma keys, correct relation in drop * Rehash single Satellite * Rehash single nh satellite * Fixing hashdiff logic * Update logs and variable names * Parse columns to drop * log and project prefix * add passed arguments, correct join on Update sql * Rehash macros for redshift * Using sp_rename for renaming fabric columns * Excluding ghost records from rehashing, check if hub rehashed for link * Excluding ghost records from rehashing * Adding ma keys to update * Remove ma key from group by * removing ma keys from group by * Rehash macros for postgres * Snowflake MA parent hk columns, hashdiff true for satellite * 91 translate migration macros for databricks (#393) * Feat: Created Migration Macros for Databricks * Fix: Fixed Timestamp Issue * Fix: Multi Active Satellites pass the test now * 221 translate rehashing for exasol (#395) * v1 exasol rehash (doesn't work yet) * Fixed exasol rehash * ma sat rehash v2 * Fixed update statement * 175 translate rehashing macros for synapse (#391) * initial code * deleting extra file * Using sp_rename for synapse macros * 213 translate rehashing for oracle (#413) * Oracle specific rehashing macros. Rehashing macros are using MERGE INTO Statements instead of UPDATE-Statements like the other database adapters, since UPDATE-Statements don't support FROM-clause. * Added oracle specific changes * Alignment with other databases * Update and rename rehash_satellites.sql to rehash_single_satellite.sql * renaming rehash folder * remove all overwrites, leave only snowflake * Revert "remove all overwrites, leave only snowflake" This reverts commit 645c779e22bbdf907ea22fd45e13ca6d01011b18. * create custom macro instead of overwrite * Rehashing for bigquery (#421) * Update rehashing * change selected parent hashkey * correct add column syntax * delete logs * always drop deprecated table * Fix multi active hashing * delete second alias * delete log * update logs * aliasing sat columns * correct whitespace in examples --------- Co-authored-by: Tim Kirschke <tkirschke@scalefree.com> Co-authored-by: jstendel <joshua.stendel@gmail.com> Co-authored-by: Tobias Pahlke <101293313+Tobias-Pah@users.noreply.github.com> Co-authored-by: kgruenhagen <89976277+kgruenhagen@users.noreply.github.com> Co-authored-by: Theo Kiehn <162969167+tkiehn@users.noreply.github.com>
M
mszerencse committed
6352e20613e4cf9667c290f16ca72fb7310fc2d2
Parent: 3a4f8e8
Committed by GitHub <noreply@github.com>
on 3/2/2026, 1:54:18 PM