Feat: Allow some control of table naming at the physical layer (#4982)

Author: erindru

docs/guides/configuration.md

@@ -320,10 +320,14 @@ The cache directory is automatically created if it doesn't exist. You can clear
 
 SQLMesh creates schemas, physical tables, and views in the data warehouse/engine. Learn more about why and how SQLMesh creates schema in the ["Why does SQLMesh create schemas?" FAQ](../faq/faq.md#schema-question).
 
-The default SQLMesh behavior described in the FAQ is appropriate for most deployments, but you can override where SQLMesh creates physical tables and views with the `physical_schema_mapping`, `environment_suffix_target`, and `environment_catalog_mapping` configuration options. These options are in the [environments](../reference/configuration.md#environments) section of the configuration reference page.
+The default SQLMesh behavior described in the FAQ is appropriate for most deployments, but you can override *where* SQLMesh creates physical tables and views with the `physical_schema_mapping`, `environment_suffix_target`, and `environment_catalog_mapping` configuration options.
+
+You can also override *what* the physical tables are called by using the `physical_table_naming_convention` option. 
+
+These options are in the [environments](../reference/configuration.md#environments) section of the configuration reference page.
 
 #### Physical table schemas
-By default, SQLMesh creates physical tables for a model with a naming convention of `sqlmesh__[model schema]`.
+By default, SQLMesh creates physical schemas for a model with a naming convention of `sqlmesh__[model schema]`.
 
 This can be overridden on a per-schema basis using the `physical_schema_mapping` option, which removes the `sqlmesh__` prefix and uses the [regex pattern](https://docs.python.org/3/library/re.html#regular-expression-syntax) you provide to map the schemas defined in your model to their corresponding physical schemas.
 
@@ -436,6 +440,104 @@ Given the example of a model called `my_schema.users` with a default catalog of
     - Using `environment_suffix_target: catalog` only works on engines that support querying across different catalogs. If your engine does not support cross-catalog queries then you will need to use `environment_suffix_target: schema` or `environment_suffix_target: table` instead.
     - Automatic catalog creation is not supported on all engines even if they support cross-catalog queries. For engines where it is not supported, the catalogs must be managed externally from SQLMesh and exist prior to invoking SQLMesh.
 
+#### Physical table naming convention
+
+Out of the box, SQLMesh has the following defaults set:
+
+ - `environment_suffix_target: schema`
+ - `physical_table_naming_convention: schema_and_table`
+ - no `physical_schema_mapping` overrides, so a `sqlmesh__<model schema>` physical schema will be created for each model schema
+
+This means that given a catalog of `warehouse` and a model named `finance_mart.transaction_events_over_threshold`, SQLMesh will create physical tables using the following convention:
+
+```
+# <catalog>.sqlmesh__<schema>.<schema>__<table>__<fingerprint>
+
+warehouse.sqlmesh__finance_mart.finance_mart__transaction_events_over_threshold__<fingerprint>
+```
+
+This deliberately contains some redundancy with the *model* schema as it's repeated at the physical layer in both the physical schema name as well as the physical table name.
+
+This default exists to make the physical table names portable between different configurations. If you were to define a `physical_schema_mapping` that maps all models to the same physical schema, since the model schema is included in the table name as well, there are no naming conflicts.
+
+##### Table only
+
+Some engines have object name length limitations which cause them to [silently truncate](https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS) table and view names that exceed this limit. This behaviour breaks SQLMesh, so we raise a runtime error if we detect the engine would silently truncate the name of the table we are trying to create.
+
+Having redundancy in the physical table names does reduce the number of characters that can be utilised in model names. To increase the number of characters available to model names, you can use `physical_table_naming_convention` like so:
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    physical_table_naming_convention: table_only
+    ```
+
+=== "Python"
+
+    ```python linenums="1"
+    from sqlmesh.core.config import Config, ModelDefaultsConfig, TableNamingConvention
+
+    config = Config(
+        model_defaults=ModelDefaultsConfig(dialect=<dialect>),
+        physical_table_naming_convention=TableNamingConvention.TABLE_ONLY,
+    )
+    ```
+
+This will cause SQLMesh to omit the model schema from the table name and generate physical names that look like (using the above example):
+```
+# <catalog>.sqlmesh__<schema>.<table>__<fingerprint>
+
+warehouse.sqlmesh__finance_mart.transaction_events_over_threshold__<fingerprint>
+```
+
+Notice that the model schema name is no longer part of the physical table name. This allows for slightly longer model names on engines with low identifier length limits, which may be useful for your project.
+
+In this configuration, it is your responsibility to ensure that any schema overrides in `physical_schema_mapping` result in each model schema getting mapped to a unique physical schema.
+
+For example, the following configuration will cause **data corruption**:
+
+```yaml
+physical_table_naming_convention: table_only
+physical_schema_mapping:
+  '.*': sqlmesh
+```
+
+This is because every model schema is mapped to the same physical schema but the model schema name is omitted from the physical table name.
+
+##### MD5 hash
+
+If you *still* need more characters, you can set `physical_table_naming_convention: hash_md5` like so:
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    physical_table_naming_convention: hash_md5
+    ```
+
+=== "Python"
+
+    ```python linenums="1"
+    from sqlmesh.core.config import Config, ModelDefaultsConfig, TableNamingConvention
+
+    config = Config(
+        model_defaults=ModelDefaultsConfig(dialect=<dialect>),
+        physical_table_naming_convention=TableNamingConvention.HASH_MD5,
+    )
+    ```
+
+This will cause SQLMesh generate physical names that are always 45-50 characters in length and look something like:
+
+```
+# sqlmesh_md5__<hash of what we would have generated using 'schema_and_table'>
+
+sqlmesh_md5__d3b07384d113edec49eaa6238ad5ff00
+
+# or, for a dev preview
+sqlmesh_md5__d3b07384d113edec49eaa6238ad5ff00__dev
+```
+
+This has a downside that now it's much more difficult to determine which table corresponds to which model by just looking at the database with a SQL client. However, the table names have a predictable length so there are no longer any surprises with identfiers exceeding the max length at the physical layer.
+
 #### Environment view catalogs
 
 By default, SQLMesh creates an environment view in the same [catalog](../concepts/glossary.md#catalog) as the physical table the view points to. The physical table's catalog is determined by either the catalog specified in the model name or the default catalog defined in the connection.

docs/reference/configuration.md

@@ -16,32 +16,44 @@ This section describes the other root level configuration parameters.
 
 Configuration options for SQLMesh project directories.
 
-| Option             | Description                                                                                                        |     Type     | Required |
-| ------------------ | ------------------------------------------------------------------------------------------------------------------ | :----------: | :------: |
-| `ignore_patterns`  | Files that match glob patterns specified in this list are ignored when scanning the project folder (Default: `[]`) | list[string] |    N     |
-| `project`          | The project name of this config. Used for [multi-repo setups](../guides/multi_repo.md).                            | string       |    N     |
+| Option             | Description                                                                                                                 |     Type     | Required |
+| ------------------ | --------------------------------------------------------------------------------------------------------------------------- | :----------: | :------: |
+| `ignore_patterns`  | Files that match glob patterns specified in this list are ignored when scanning the project folder (Default: `[]`)          | list[string] |    N     |
+| `project`          | The project name of this config. Used for [multi-repo setups](../guides/multi_repo.md).                                     | string       |    N     |
 | `cache_dir`        | The directory to store the SQLMesh cache. Can be an absolute path or relative to the project directory. (Default: `.cache`) | string       |    N     |
+| `log_limit`        | The default number of historical log files to keep (Default: `20`)                                                          | int          |    N     |
 
-### Environments
+### Database (Physical Layer)
 
-Configuration options for SQLMesh environment creation and promotion.
+Configuration options for how SQLMesh manages database objects in the [physical layer](../concepts/glossary.md#physical-layer).
 
 | Option                        | Description                                                                                                                                                                                                                                                                                        | Type                 | Required |
 |-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------:|:--------:|
 | `snapshot_ttl`                | The period of time that a model snapshot not a part of any environment should exist before being deleted. This is defined as a string with the default `in 1 week`. Other [relative dates](https://dateparser.readthedocs.io/en/latest/) can be used, such as `in 30 days`. (Default: `in 1 week`) | string               | N        |
+| `physical_schema_override`    | (Deprecated) Use `physical_schema_mapping` instead. A mapping from model schema names to names of schemas in which physical tables for the corresponding models will be placed.                                                                                                                    | dict[string, string] | N        |
+| `physical_schema_mapping`     | A mapping from regular expressions to names of schemas in which physical tables for the corresponding models [will be placed](../guides/configuration.md#physical-table-schemas). (Default physical schema name: `sqlmesh__[model schema]`)                                                        | dict[string, string] | N        |
+| `physical_table_naming_convention`| Sets which parts of the model name are included in the physical table names. Options are `schema_and_table`, `table_only` or `hash_md5` - [additional details](../guides/configuration.md#physical-table-naming-convention). (Default: `schema_and_table`)                                     | string               | N        |
+
+### Environments (Virtual Layer)
+
+Configuration options for how SQLMesh manages environment creation and promotion in the [virtual layer](../concepts/glossary.md#virtual-layer).
+
+| Option                        | Description                                                                                                                                                                                                                                                                                        | Type                 | Required |
+|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------:|:--------:|
 | `environment_ttl`             | The period of time that a development environment should exist before being deleted. This is defined as a string with the default `in 1 week`. Other [relative dates](https://dateparser.readthedocs.io/en/latest/) can be used, such as `in 30 days`. (Default: `in 1 week`)                      | string               | N        |
 | `pinned_environments`         | The list of development environments that are exempt from deletion due to expiration                                                                                                                                                                                                               | list[string]         | N        |
-| `time_column_format`          | The default format to use for all model time columns. This time format uses [python format codes](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) (Default: `%Y-%m-%d`)                                                                                        | string               | N        |
 | `default_target_environment`  | The name of the environment that will be the default target for the `sqlmesh plan` and `sqlmesh run` commands. (Default: `prod`)                                                                                                                                                                   | string               | N        |
-| `physical_schema_override`    | (Deprecated) Use `physical_schema_mapping` instead. A mapping from model schema names to names of schemas in which physical tables for the corresponding models will be placed.                                                                                                                    | dict[string, string] | N        |
-| `physical_schema_mapping`     | A mapping from regular expressions to names of schemas in which physical tables for the corresponding models [will be placed](../guides/configuration.md#physical-table-schemas). (Default physical schema name: `sqlmesh__[model schema]`)                                                        | dict[string, string] | N        |
-| `environment_suffix_target`   | Whether SQLMesh views should append their environment name to the `schema` or `table` - [additional details](../guides/configuration.md#view-schema-override). (Default: `schema`)                                                                                                                 | string               | N        |
-| `gateway_managed_virtual_layer`   | Whether SQLMesh views of the virtual layer will be created by the default gateway or model specified gateways - [additional details](../guides/multi_engine.md#gateway-managed-virtual-layer). (Default: False)                                                                                                                 | boolean               | N        |
-| `infer_python_dependencies`   | Whether SQLMesh will statically analyze Python code to automatically infer Python package requirements. (Default: True)                                                                                                                 | boolean               | N        |
+| `environment_suffix_target`   | Whether SQLMesh views should append their environment name to the `schema`, `table` or `catalog` - [additional details](../guides/configuration.md#view-schema-override). (Default: `schema`)                                                                                                      | string               | N        |
+| `gateway_managed_virtual_layer`   | Whether SQLMesh views of the virtual layer will be created by the default gateway or model specified gateways - [additional details](../guides/multi_engine.md#gateway-managed-virtual-layer). (Default: False)                                                                                | boolean              | N        |
 | `environment_catalog_mapping` | A mapping from regular expressions to catalog names. The catalog name is used to determine the target catalog for a given environment.                                                                                                                                                             | dict[string, string] | N        |
-| `log_limit`                   | The default number of logs to keep (Default: `20`)                                                                                                                                                                                                                                                 | int                  | N        |
 
-### Model defaults
+### Models
+
+| Option                        | Description                                                                                                                                                                                                                                                                                        | Type                 | Required |
+|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------:|:--------:|
+| `time_column_format`          | The default format to use for all model time columns. This time format uses [python format codes](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) (Default: `%Y-%m-%d`)                                                                                        | string               | N        |
+| `infer_python_dependencies`   | Whether SQLMesh will statically analyze Python code to automatically infer Python package requirements. (Default: True)                                                                                                                                                                            | boolean              | N        |
+| `model_defaults`              | Default [properties](./model_configuration.md#model-defaults) to set on each model. At a minimum, `dialect` must be set.                                                                                                                                                                           | dict[string, any]    | Y        |
 
 The `model_defaults` key is **required** and must contain a value for the `dialect` key.
 

sqlmesh/core/config/__init__.py

@@ -2,7 +2,10 @@
     AutoCategorizationMode as AutoCategorizationMode,
     CategorizerConfig as CategorizerConfig,
 )
-from sqlmesh.core.config.common import EnvironmentSuffixTarget as EnvironmentSuffixTarget
+from sqlmesh.core.config.common import (
+    EnvironmentSuffixTarget as EnvironmentSuffixTarget,
+    TableNamingConvention as TableNamingConvention,
+)
 from sqlmesh.core.config.connection import (
     AthenaConnectionConfig as AthenaConnectionConfig,
     BaseDuckDBConnectionConfig as BaseDuckDBConnectionConfig,