Extending QueryForge

Extending QueryForge

Three extension points, in increasing order of effort: a value accessor for an unusual in-memory source, a dialect for a database engine, and a whole provider for a data source that is neither.

---

Writing a value accessor

The smallest extension point. An accessor tells the in-memory engine how to read a named column off a row.

Func<TRow, string, object?> accessor;   // (row, columnName) => value

The default reads properties by name, case-insensitively, cached per type. Supply your own for anything else:

var result = rows.ToQueryResult(query, (row, column) => column switch
{
    "Name"    => row.Profile.DisplayName,
    "Country" => row.Address?.CountryCode,
    "Age"     => row.BirthDate is null ? null : Years(row.BirthDate.Value),
    _         => null
});

Rules an accessor must follow:

1. Return null for a name you do not know, rather than throwing. A stray column must not break a whole query.
2. Normalize DBNull to null. The engine treats them the same, but only after your accessor has returned.
3. Be pure and thread-safe. It is called once per row per column reference.
4. Be fast. It is on the hot path of every filter, sort and grouping. A switch is much faster than reflection for large collections.

Supplying an accessor switches off column-existence checking, since QueryForge cannot know which names you can resolve. Every name is taken at face value and nothing is dropped for being unknown. Add an allow-list with validation if the source is client-facing.

Two ready-made accessors ship in InMemoryAccessors — see In-Memory Provider.

---

Writing a dialect

A dialect teaches the Dapper provider a database's syntax. Everything structural is already handled by SqlQueryCompiler, so the work is bounded.

The contract

Implement PepperX.QueryForge.Dapper.Compiler.ISqlDialect — thirteen members, all documented under Dapper: Dialects.

A worked example: Firebird

using PepperX.QueryForge.Dapper;
using PepperX.QueryForge.Dapper.Compiler;

public sealed class FirebirdDialect : ISqlDialect
{
    // Firebird is not in the DapperDatabaseProvider enum, so it registers under the nearest
    // member — see "Adding an engine that is not in the enum" below.
    public DapperDatabaseProvider ProviderType => DapperDatabaseProvider.MSSQL;

    public string DefaultSchema => string.Empty;

    public string QuoteIdentifier(string identifier)
    {
        ArgumentException.ThrowIfNullOrWhiteSpace(identifier);

        return "\"" + identifier.Replace("\"", "\"\"") + "\"";
    }

    public string ParameterReference(string name) => "@" + name;

    // Firebird 3+ supports the SQL standard form.
    public string PagingClause(int offset, int size)
        => $"OFFSET {offset} ROWS FETCH NEXT {size} ROWS ONLY";

    public bool RequiresOrderByForPaging => false;
    public string OrderByFallback => "1";

    // Firebird defaults to nulls last ascending, so placement is stated explicitly.
    public string NullOrdering(SortOrder order)
        => order == SortOrder.Descending ? " NULLS LAST" : " NULLS FIRST";

    public string EscapeLikeValue(string value)
        => value.Replace(@"\", @"\\").Replace("%", @"\%").Replace("_", @"\_");

    public string LikeEscapeClause => @" ESCAPE '\'";

    public bool SupportsTableValuedFunctions => true;    // selectable stored procedures
    public bool SupportsStoredProcedures => true;

    public string BuildSource(
        string schema, string name, DapperObjectType type, IReadOnlyList<string> argumentReferences)
    {
        var qualified = QuoteIdentifier(name);

        return type == DapperObjectType.TVF
            ? $"{qualified}({string.Join(", ", argumentReferences)})"
            : qualified;
    }

    public string BuildStoredProcedureCall(
        string schema, string name, IReadOnlyList<KeyValuePair<string, string>> argumentNames)
        => $"SELECT * FROM {QuoteIdentifier(name)}({string.Join(", ", argumentNames.Select(a => a.Value))})";
}

Register it

builder.Services.AddQueryForgeDapper();
builder.Services.AddQueryForgeDialect(new FirebirdDialect());

Order matters — AddQueryForgeDialect throws InvalidOperationException if called first. Registration also updates the static registry behind IDbConnection.QueryForgeAsync, so the custom dialect is available on that path too.

Overriding a built-in dialect

Registering a dialect whose ProviderType matches a built-in one replaces it. The built-in dialects are sealed, so implement ISqlDialect and delegate for the members you are not changing:

public sealed class NullsLastPostgresDialect : ISqlDialect
{
    private readonly PostgreSqlDialect _inner = new();

    // The one behaviour being changed: match PostgreSQL's own default instead of QueryForge's.
    public string NullOrdering(SortOrder order) => string.Empty;

    // Everything else forwards.
    public DapperDatabaseProvider ProviderType => _inner.ProviderType;
    public string DefaultSchema => _inner.DefaultSchema;
    public string QuoteIdentifier(string identifier) => _inner.QuoteIdentifier(identifier);
    public string ParameterReference(string name) => _inner.ParameterReference(name);
    public string PagingClause(int offset, int size) => _inner.PagingClause(offset, size);
    public bool RequiresOrderByForPaging => _inner.RequiresOrderByForPaging;
    public string OrderByFallback => _inner.OrderByFallback;
    public string EscapeLikeValue(string value) => _inner.EscapeLikeValue(value);
    public string LikeEscapeClause => _inner.LikeEscapeClause;
    public bool SupportsTableValuedFunctions => _inner.SupportsTableValuedFunctions;
    public bool SupportsStoredProcedures => _inner.SupportsStoredProcedures;

    public string BuildSource(string s, string n, DapperObjectType t, IReadOnlyList<string> a)
        => _inner.BuildSource(s, n, t, a);

    public string BuildStoredProcedureCall(string s, string n, IReadOnlyList<KeyValuePair<string, string>> a)
        => _inner.BuildStoredProcedureCall(s, n, a);
}

Overriding null ordering breaks the cross-provider parity guarantee on purpose. Do it knowingly.

Adding an engine that is not in the enum

DapperDatabaseProvider is closed:

public enum DapperDatabaseProvider { MSSQL, MySQL, PostgreSQL, Oracle, SQLite }

and engine detection maps a fixed set of connection type names to those members:

Connection type name	Member
SqlConnection	MSSQL
NpgsqlConnection	PostgreSQL
MySqlConnection, MariaDbConnection	MySQL
OracleConnection	Oracle
SqliteConnection	SQLite

An engine outside that set has no member of its own today. The practical options:

1. Reuse the nearest member (as the Firebird example does) and accept that the enum value no longer names the real engine. It works — the enum is only a registry key — but the value is misleading in logs, and you lose the built-in dialect for whichever member you took.
2. Route explicitly rather than by connection type, by resolving the executor yourself. This requires internal access and is not a supported public path today.
3. Open an issue so a member and a connection-name mapping can be added. This is the right answer for an engine anyone else would want.

Be honest about which you chose in your own documentation — option 1 in particular is a compromise, not a clean extension.

Testing a dialect

Dialects are pure. Test them directly, and add yours to the repository's dialect portability suite, which asserts the invariants every dialect must satisfy:

[Fact]
public void Quoting_neutralises_the_closing_delimiter()
    => Assert.Equal("\"a\"\"b\"", new FirebirdDialect().QuoteIdentifier("a\"b"));

[Fact]
public void Values_are_never_inlined()
{
    var sql = new SqlQueryCompiler(new FirebirdDialect()).CompileRows(query, columns);

    Assert.DoesNotContain("Germany", sql.Text);
    Assert.Equal("Germany", sql.Parameters["p0"]);
}

See Testing.

---

Writing a new provider

For a data source that is neither a relational database nor an in-memory collection — a document store, a search index, a remote API.

What a provider is

Something that takes a Query and returns a QueryResult<TModel> obeying Query Semantics. That is the whole contract.

Reuse the shared layer

The core package exposes everything you need so you do not re-derive the rules:

using PepperX.QueryForge.Querying;

ConditionSemantics.IsExecutable(condition);      // is this condition usable at all
ConditionSemantics.IsDisjunction(logic);         // OR vs AND
ConditionSemantics.IsNegated(logic);             // AndNot / OrNot
ConditionSemantics.Unwrap(value);                // JsonElement / DBNull → CLR value
ConditionSemantics.IsPatternOperator(op);        // needs LIKE-style escaping

ValueCoercion.TryCoerce(value, targetType, out var coerced);

QueryValueComparer.Instance.Compare(x, y);
QueryValueComparer.Instance.AreEqual(x, y);

HierarchyBuilder.Build(rows, groupByColumns, valueAccessor);
ProjectionShaper.ShapeAll(rows, selectColumns, alsoKeep);
PropertyAccessor.For<TModel>();
PropertyAccessor.Exists<TModel>(columnName);

Using these is what makes a provider agree with the others rather than merely resemble them.

Skeleton

public static class MyStoreQueryExtensions
{
    public static async Task<QueryResult<TModel>> ToQueryResultAsync<TModel>(
        this IMyStore<TModel> store, Query query, CancellationToken ct = default)
    {
        // 1. Resolve the whitelist — whatever "a real column" means for your source.
        var known = store.KnownFields;

        // 2. Filter, honouring usability, group logic and negation.
        var filter = BuildNativeFilter(query.Criteria, known);

        // 3. Count before paging.
        var total = await store.CountAsync(filter, ct);

        // 4. Sort — nulls first ascending, last descending.
        var sorted = ApplySorts(filter, query.SortColumns, known);

        // 5. Page.
        var size   = query.Paging.Size   > 0 ? query.Paging.Size   : 12;
        var number = query.Paging.Number > 0 ? query.Paging.Number : 1;

        // 6. Grouped or flat.
        var groups = query.GroupByColumns.Where(g => known.Contains(g.ColumnName)).ToList();

        if (groups.Count == 0)
        {
            var rows = await sorted.Skip((number - 1) * size).Take(size).ToListAsync(ct);

            return new QueryResult<TModel>
            {
                Meta = new QueryResultMeta(
                    new QueryResultMetaTotal(total, PageCount(total, size)),
                    QueryResultType.Flat),
                Models = ProjectionShaper.ShapeAll(rows, query.SelectColumns)
            };
        }

        // Grouped: count distinct outermost keys, page them, fetch every row under them,
        // then hand the rows to the shared builder.
        // …
    }

    private static int PageCount(int total, int size)
        => size <= 0 ? 0 : (int)Math.Ceiling(total / (double)size);
}

The checklist

A provider is correct when all of these hold:

• Unusable conditions are dropped, never turned into match-nothing clauses.
• Unknown column names are dropped in filters, sorts, groupings and projections.
• Group Logic joins conditions; AndNot/OrNot negate the group.
• Criteria Logic joins groups; its Not suffix is ignored.
• Three-valued logic is honoured — a comparison against null is unknown and survives negation.
• Equals/NotEquals against null are IS NULL/IS NOT NULL.
• Between is inclusive at both ends and needs both bounds.
• Text operators escape wildcards in the caller's value.
• Nulls sort first ascending, last descending.
• Non-positive Size/Number fall back to 12 and 1.
• Meta.Total.Rows is the filtered count when flat, the distinct outermost key count when grouped.
• Meta.Total.Pages is ceil(total / size).
• A page past the end returns empty with accurate totals.
• Grouping columns survive SelectColumns.
• A projection naming nothing real is ignored rather than blanking rows.
• HierarchyBuilder is used for the tree — do not hand-roll it.
• Both collections on QueryResult<T> are non-null.

Prove it

Inherit the two shared suites and supply a RunAsync:

public sealed class MyStoreConformanceTests : QueryForgeConformanceTests
{
    protected override Task<QueryResult<Widget>> RunAsync(Query query) => /* … */;
}

public sealed class MyStoreSalesTests : SalesScenarioTests
{
    protected override Task<QueryResult<SalesOrder>> RunAsync(Query query) => /* … */;
}

125 tests then tell you exactly where you diverge. That is the definition of done, and it is the same bar the three shipped providers are held to. See Testing.

---

Contributing an extension back

Dialects for engines other people use, and providers for common data sources, are welcome. Open an issue first to agree the shape — particularly for anything needing a DapperDatabaseProvider member, since that is a public-API change.

A contribution is ready when it passes both shared suites, adds itself to the dialect portability suite where applicable, and documents any divergence from Cross-Provider Parity.