Nodes are meant to document the tasks we accomplish, however for the most part, we have backup versions of every data pipeline (primary and secondary). Previously this was documented by the mirrors Edge relationship, but this would overpopulate our table.
Instead, have primary be the default classification for a priority attribute in every node. When populating the database, we will create the primary runs first, then copy the node entries to update relevant attribute for the secondary run.
Nodes are meant to document the tasks we accomplish, however for the most part, we have backup versions of every data pipeline (primary and secondary). Previously this was documented by the
mirrorsEdge relationship, but this would overpopulate our table.Instead, have primary be the default classification for a priority attribute in every node. When populating the database, we will create the primary runs first, then copy the node entries to update relevant attribute for the secondary run.