CLI: openfn reconcile

[josephjclark]

Idea for a new projects command to reconcile/merge a local project on the filesystem against the remote project

This is basically a merge helper

The problem is:

• I make changes to workflow A on my file system
• In the app, someone else makes changes to workflow B
• I want to be able to reflect those remote changes on workflow B on my file system

In other words, I want to fetch the latest remote version so I'm up to date, and selectively checkout only some workflows.

The danger is that if I force deploy, I'm also force deploying the old workflows and overwriting changes. I could do deploy --workflows A to only deploy one workflow. But my file system is still out of sync - the fs and project are not consistent, and sooner or later I'll hit problems.

The idea of reconcile is a sort of expand-and-merge.

It'll run a checkout. For each workflow that is different to the filesystem, we ask the user whether to checkout the remote, local or both. If both, we duplicate the workflow (probably we do ./workflows/A.app/ and the user must manually copy any diffs into their local workflow, the delete the duplication.

Once reconciliation is complete, we do openfn project reconcile complete and update the local project file. Now the FS and project are in sync. When running deploy, we can safely push the local version to the server. Hmm, note that on deploy, when fetching, the local project file is now ahead of the server... so that's kinda fun.

Note that this all works at the workflow level - there's also conflicts within a workflow. Which I think is an extension of this strategy.

[josephjclark]

Thinking through this stuff has reminded me to be strict about the project file. It must always represent the connected app, not the local file system. This removes one area of possible conflict. I'm refocusing development around that idea.

The idea of a reconcile command might still be useful for resolving merge conflicts.

EDIT: But wait, this doesn't help us with the case of basically a partial merge

• On my filesystem I edit workflow A
• In the app I edit workflow B
• When I deploy workflow A, SOMEONE needs to be smart enough not to override the changes in workflow B. This would essentially revert the changes made in the app.

I think right now we'll just throw an error because the workflows have diverged? But really they've only diverged one way: we should be able to recognise that the local app is behind, resolve the two workflows and push the difference.

This has to be part of the merge function. When merging from staging into main, you only merge the workflows where staging is ahead (ie, staging and main have a common version history, but staging has versions that main does not have). If main has commits that staging doesn't have, this is fine - you just ignore the local changes. It's only when both staging and main have versions - a real divergence - that you have a problem.

If true, then you only need to reconcile for these diverged workflows.