Proposal check (before implementing)

[svdgriendt]

It's been a while, but that doesn't mean I haven't been thinking about this project 😉. Before I go spend a lot of effort into getting something going that's maybe not a good idea, I first want to run it by you. After typing all of this, I realized this is a lot of text, which shows even more to me that having a general direction of alignment is essential.

The thing I would like to include is a form of a node walker. That is, you process a query (a SQL string) with Protobuf and it will give you back some statements. You then may be interested in particular kinds of statements/nodes in the set of "trees" that was built of the query. In my case, I would like to know all relations that are used in a CREATE TABLE so that I can build a dependency graph for example. Manually walking the "tree" is cumbersome, but "subscribing" to some of the statements/nodes that you're interested in is much more manageable I can imagine.

The goal is to have this source generated, because you definitely don't want to manually upkeep this. I've done a lightweight test already, and it's definitely possible to do the things proposed below. So, this is more about structure and if you're OK with such a feature to be included. An important note is that all the Protobuf generated classes are public sealed partial and thus open for extension within the same project (hence source generator).

1. interface INodeExtensions
   This one has a public property (or method) for IEnumerable<Node?> Children.
2. public sealed partial class CreateStmt : INodeExtensions
   And for all the other nodes/statements that are out there, where each one will include all of its properties that implement Node are included in Children. In case of CreateStatement this could be [Constraints, InhRelation, Nnconstraints, Options, TableELts] for example. We can't leave this out, because next up needs it to have a "generic" way to walk the "tree". Or we must build something similar as 3 instead, which would then be specifically only for Children and not open for more things marked as "added by us on top of Protobuf" (I personally like this marker system).
3. public sealed partial class Node to have INodeExtensions? property (or method)
   This has to switch on the NodeCase and then simply return the matching property. In the case of CreateStmt that would mean the same named property Node.CreateStmt. But note that not all exist or are exactly the same cased (but this I already solved in my lightweight test).
4. public sealed NodeIterator : IEnumerator<Node>
   Implementation is trivial, because we can use Node.INodeExtensions (property or method) and then simply use Children to keep on navigating the "tree".
5. public sealed NodeEnumerable : IEnumerable<Node>
   Implementation is trivial, because we can use NodeIterator which does all the heavy lifting.

The next part is a bit tricky for me on what would work best really. I have considered the following options really. Because I think in most cases you do want to bundle a few things, the visitor pattern is the best match. But I could also see the strategy pattern to be chosen, as probably bundling is a different concern that you shouldn't mix in here, as it's already complex enough (you can see that with this wall of text 😬).

6. A visitor pattern with an interface and a base implementation with only virtual and empty implementations of the methods. One method for each node/statement. Consumers can then inherit for this base override only the methods they're interested in. They then also automatically get the correct type, e.g., CreateStmt or Alias etc.
   This would mean Node gets a method Accept(IVisitor) and basically has the same switch as in 3, but then does visitor.Visit(Node.CreateStmt) for example.
7. A mediator pattern which takes in some filter, e.g., a params NodeCase[]. It can then walk the "tree", check the NodeCase and raise Notify(Node) when it matches. You then have to wire up an event or callback lambda that's then triggered within this Notify method.
   This does mean that a consumer still has to extract the correct property. I mean, you will get a Node that matches with any of the given NodeCase, but you don't know which one. You're then forced to implement your own (partial/filtered) swich from 3.
8. A strategy pattern which has an Execute<T>(T) method. We then detect the T and match it with a NodeCase (we could pre-build a FrozenDictionary<Type, NodeCase> for that). Consumers can then register one or more strategies, which are then very specific to one node/statement only.
9. Some kind of source generator option. I have not much experience with this, but you can bundle a source generator together with a NuGet package. We could then go for a few routes. Both would require the FrozenDictionary<Type, NodeCode> to be properly generated.
   9.1. [NodeCaseFilter([NodeCase.CreateStmt, NodeCase.Alias])] public partial class MyThing {}. You then get partial method that has an empty implementation, which you can then render explicit yourself and have the concrete implementation. It will also source generate a method that takes in a Node, a partial/filtered case like in 3 and invoke the correct source generated partial method.
   9.2. Also a public partial class MyThing, maybe with a marker attribute on it to be able to detect it quickly. Then define on a method of your own liking an attribute to mark it as a "handler". We then search all these methods in that class and source generate that same method that takes in Node as with 9.1, but then forward it to our own method.

In addition of the above, I have the idea to also include a "stack" of sorts alongside of the Node you would "receive". That's needed in my opinion, because if you're in a ColumnDef, you've lost the context in which you are. You can force the consumer to keep track of this, or we could offer it alongside.

As you can see, I put in some thought into this, but this is also heavily biased to what I would like to implement in my own project. Sadly, I can't do it as easily there (I could with decorator pattern), because of the sealed partial class nature, else I would have done some try-outs there and would have asked you to check it there and I would adapt a version for here based on your feedback. I'm not sure if this in the right format to discuss this, but I have no other way available to me at this moment.

So, what do you think of each point? Is this something we should include in here? And which pattern do you think fits best if this is to be included here?

[svdgriendt]

I figured some actual pseudo code would help with the imagination. After giving it some more thought, at the expense of a harder implementation in this repository, I figured the source generators route would be the best really. It feels the most natural to me for the scenario I wish to implement, which is to figure out all "relations/objects" that are used in a query, e.g., tables and functions.

What we got already today

public enum NodeOneofCase
{
  CreateStatement,
  SelectStatement
  // All the other ones
}

public sealed partial class Node
{
  public NodeOneofCase NodeCase { get; init; }
  public CreateStatementNode? CreateStatement { get; init; }
  public SelectStatementNode? SelectStatement { get; init; }
  // All the other ones
}

public sealed partial class CreateStatementNode
{
  public required Node SelectStatement { get; init; }
  // All the other ones
}

public sealed partial class SelectStatementNode
{
  // All the other ones
}

What we source generate on top of those

public sealed partial class Node
{
  public INpgqueryNodeExtensions? GetNpgqueryNodeExtensions()
  {
    return NodeCase switch
    {
      NodeOneofCase.CreateStatement => CreateStatement,
      NodeOneofCase.SelectStatement => SelectStatement,
      // All the other ones
      _ => null
    };
  }
}

public sealed partial class CreateStatementNode : INpgqueryNodeExtensions
{
  public IEnumerable<Node?> Children => [SelectStatement]; // All the other ones
}

public sealed partial class SelectStatementNode : INpgqueryNodeExtensions
{
  public IEnumerable<Node?> Children => []; // All the other ones
}

What we manually add ourselves (not source generated)

public interface INpgqueryNodeExtensions
{
  IEnumerable<Node?> Children { get; }
}

public sealed class NodeIterator : IEnumerator<Node>
{
  private readonly Node _root;
  private readonly Stack<Node> _stack = new();
  private bool _disposed = false;

  public Node Current { get; private set; }
  object? IEnumerator.Current => Current;

  public NodeIterator(Node root)
  {
    _root = Current = root;
    _stack.Push(_root);
  }

  public bool MoveNext()
  {
    ObjectDisposedException.ThrowIf(_disposed, this);

    if (_stack.Count == 0)
      return false;

    Current = _stack.Pop();
    foreach (var child in Current.GetNpgqueryNodeExtensions()?.Children ?? [])
      if (child is not null)
        _stack.Push(child);

    return true;
  }

  public void Reset()
  {
    ObjectDisposedException.ThrowIf(_disposed, this);

    Current = _root;
    _stack.Clear();
    _stack.Push(_root);
  }

  public void Dispose()
  {
    _disposed = true;
    _stack.Clear();
  }
}

public sealed class NodeEnumerable : IEnumerable<Node>
{
  private readonly Node  _root;

  public NodeEnumerable(Node root) => _root = root;

  public IEnumerator<Node> GetEnumerator() => new NodeIterator(_root);

  IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}

[AttributeUsage(AttributeTargets.Class)]
public sealed class NodeWalkerAttribute : Attribute
{
  public NodeOneofCase[] Cases { get; }

  public NodeWalkerAttribute(NodeOneofCase[] cases)
  {
    Cases = cases;
  }
}

What a consumer then is able to do

[NodeWalker([NodeOneofCase.CreateStatement, NodeOneofCase.SelectStatement])]
public partial class MyNodeVisitor
{
  protected partial bool Visit(CreateStatementNode createStatementNode)
  {
    Console.WriteLine("Create statement");
    return true;
  }

  protected partial bool Visit(SelectStatementNode selectStatementNode)
  {
    Console.WriteLine("Select statement");
    return true;
  }
}

With the source generator we include in the NuGet package

public partial class MyNodeVisitor
{
  public void Visit(Node node)
  {
    foreach (var visitedNode in new NodeEnumerable(node))
    {
      var @continue = visitedNode.NodeCase switch
      {
        NodeOneofCase.CreateStatement => Visit(visitedNode.CreateStatement!),
        NodeOneofCase.SelectStatement => Visit(visitedNode.SelectStatement!),
        _ => true
      };

      if (!@continue)
        break;
    }
  }

  protected partial bool Visit(CreateStatementNode createStatementNode);
  protected partial bool Visit(SelectStatementNode selectStatementNode);
}

Given this example

var node = new Proposal.Node
{
  NodeCase = Proposal.NodeOneofCase.CreateStatement,
  CreateStatement = new Proposal.CreateStatementNode
  {
    SelectStatement = new Proposal.Node
    {
      NodeCase = Proposal.NodeOneofCase.SelectStatement,
      SelectStatement = new Proposal.SelectStatementNode
      {

      }
    }
  }
};
new Proposal.MyNodeVisitor().Visit(node);

It would then print

Create satatement
Select statement