Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/anomalies/details/insights.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ Click the **Download :material-download:** button to export all source records a
![source-records-download](../../assets/anomalies/details/insights/source-records-download.png)

!!! note
The download includes only the records that were captured during the scan. The number of available records depends on the **maximum source records per anomaly** configured in the [scan settings](../../operations/scan/scan.md#configuration). If you need more records, increase the limit and re-run the scan.
The download includes only the records that were captured during the scan. The number of available records depends on the **maximum source records per anomaly** configured in the [scan settings](../../operations/scan/how-tos/scan-settings.md). If you need more records, increase the limit and re-run the scan.

## Tickets

Expand Down
4 changes: 2 additions & 2 deletions docs/anomalies/details/source-record.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ If the Anomaly Type is **Record**, you will find the highlighted row(s) in the s

## Source Record Visualization

The number of source records displayed per anomaly is determined by the **Maximum Source Examples per Anomaly** setting, which can be configured during [scan setup](../../operations/scan/scan.md#configuration){:target="_blank"}. The available limits are 10, 100, 1,000, or 10,000 records. The interface includes sticky headers that remain visible when scrolling through large datasets, making navigation easier during data review.
The number of source records displayed per anomaly is determined by the **Maximum Source Examples per Anomaly** setting, which can be configured during [scan setup](../../operations/scan/how-tos/scan-settings.md){:target="_blank"}. The available limits are 10, 100, 1,000, or 10,000 records. The interface includes sticky headers that remain visible when scrolling through large datasets, making navigation easier during data review.

![record](../../assets/anomalies/details/source-record/visualization.png)

Expand All @@ -34,7 +34,7 @@ Click the **Download :material-download:** button to export all source records a
![source-records-download](../../assets/anomalies/details/insights/source-records-download.png)

!!! note
The download includes only the records that were captured during the scan. The number of available records depends on the **maximum source records per anomaly** configured in the [scan settings](../../operations/scan/scan.md#configuration){:target="_blank"}. If you need more records, increase the limit and re-run the scan.
The download includes only the records that were captured during the scan. The number of available records depends on the **maximum source records per anomaly** configured in the [scan settings](../../operations/scan/how-tos/scan-settings.md){:target="_blank"}. If you need more records, increase the limit and re-run the scan.

## Masked Fields in Source Records

Expand Down
2 changes: 1 addition & 1 deletion docs/anomalies/detection.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Authored checks can range from simple, template-based checks to more complex rul
The Scan operation asserts rigorous quality checks to identify any anomalies within the data. This step ensures data integrity and reliability by recording the analyzed data in your configured enrichment datastore, facilitating continuous data quality improvement.

!!! note
For more information, please refer to the documentation [Scan Operation](../operations/scan/scan.md).
For more information, please refer to the documentation [Scan Operation](../operations/scan/getting-started.md).

**6. Anomaly Analysis**

Expand Down
4 changes: 2 additions & 2 deletions docs/anomalies/fingerprints.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ The lack of a persistent identifier means **Qualytics** cannot distinguish betwe

To handle recurring anomalies in truncate-and-reload tables, configure your scan to use fingerprint-based duplicate handling.

Follow the steps in the [scan operation configuration](../operations/scan/scan.md#configuration) to reach the correct settings. Then, under **Step 8 → Scan Settings**, open the [anomaly options section](https://userguide.qualytics.io/source-datastore/scan/#configuration:~:text=Step%208%3A%20Configure%20the%20Scan%20Settings) and enable both duplicate-handling options:
Follow the steps in the [scan operation configuration](../operations/scan/how-tos/scan-settings.md) to reach the correct settings. Then, under **Step 8 → Scan Settings**, open the [anomaly options section](https://userguide.qualytics.io/source-datastore/scan/#configuration:~:text=Step%208%3A%20Configure%20the%20Scan%20Settings) and enable both duplicate-handling options:

- **Archive Duplicate Anomalies:** When the same 127 anomalies appear again after the table reload, Qualytics recognizes their fingerprints and automatically marks them as duplicates rather than new anomalies.
- **Reactivate Recurring Anomalies:** If an anomaly was previously archived or resolved but reappears in subsequent scans, Qualytics reactivates the original anomaly record, maintaining full historical context.
Expand All @@ -88,4 +88,4 @@ Enable these settings in Scan Settings of your Scan Operation:
- **Archive Duplicate Anomalies**
- **Reactivate Recurring Anomalies**

Set an appropriate **Anomaly Rollup Threshold** based on your data volume and tolerance for grouped anomalies.
Set an appropriate **Maximum Record Anomalies per Check** based on your data volume and tolerance for grouped anomalies.
3 changes: 3 additions & 0 deletions docs/anomalies/types.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ A shape anomaly identifies an anomalous structure within the analyzed data. The
!!! note
Sometimes, shape anomalies only affect a subset of the dataset. This means that only certain rows exhibit the structural issue, rather than the entire dataset.

!!! note "Shape anomalies from the rollup threshold"
Shape anomalies can also be created when the number of individual record anomalies for a single check exceeds the **Maximum Record Anomalies per Check** threshold. When this happens, remaining violations are consolidated into a single rolled-up shape anomaly that preserves the total violation count, rather than producing one record anomaly per violation. The threshold is configurable in the scan Advanced Options; see [Scan Settings](../operations/scan/how-tos/scan-settings.md#anomaly-options).

## Example Use Case

**Scenario**
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/assets/operations/scan/step-1-side-menu.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-10-incremental.png
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-14-next-button.png
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-16-anomalyy.png
Diff not rendered.
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-19-run-now.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-2-details-page.png
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-21-hourly.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-22-daily.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-23-weekly.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-24-monthly.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-25-advanced.png
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-27-schedule.png
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-29-activity.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-3-run.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-30-running.png
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-32-warning.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-33-success.png
Diff not rendered.
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-36-result.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-37-drop-down.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-4-all-operation.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-5-specific.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-6-tag.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-7-next.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-8-select-check.png
Diff not rendered.
Binary file removed docs/assets/operations/scan/step-9-nextt.png
Diff not rendered.
2 changes: 1 addition & 1 deletion docs/container/actions-on-container.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,5 @@ The **Run** button provides options to execute operations on datasets, such as p
| No. | Options | Description |
| :---- | :---- | :---- |
| **1.** | Profile | **Profile** allows you to run a profiling operation to analyze the data structure, gather metadata, set thresholds, and define record limits for comprehensive dataset profiling.<br>**Note:** For profile operation, please refer to the [Profile Operation documentation](../operations/profile/profile.md){target="_blank"}. |
| **2.** | Scan | **Scan** allows you to perform data quality checks, configure scan strategies, and detect anomalies in the dataset.<br>**Note:** For scan operation, please refer to the [Scan Operation documentation](../operations/scan/scan.md){target="_blank"}. |
| **2.** | Scan | **Scan** allows you to perform data quality checks, configure scan strategies, and detect anomalies in the dataset.<br>**Note:** For scan operation, please refer to the [Scan Operation documentation](../operations/scan/getting-started.md){target="_blank"}. |
| **3.** | External Scan | **External Scan** allows you to upload a file and validate its data against predefined checks in the selected table.<br>**Note:** For external scan, please refer to the [ External Scan documentation](../operations/external-scan/external-scan.md){target="_blank"}. |
2 changes: 1 addition & 1 deletion docs/container/manage-tables-and-files/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@ Under **Run**, choose the type of operation you want to perform:

To understand how a profile operation is performed, you can follow the remaining steps from the documentation [Profile Operation.](../../operations/profile/profile.md#configuration){target="_blank"}.

To understand how a scan operation is performed, you can follow the remaining steps from the documentation [Scan Operation.](../../operations/scan/scan.md#configuration){target="_blank"}.
To understand how a scan operation is performed, you can follow the remaining steps from the documentation [Scan Operation.](../../operations/scan/getting-started.md){target="_blank"}.
2 changes: 1 addition & 1 deletion docs/container/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Totals are calculated from sampled data, not the full dataset. Values may differ
After a Sync operation, each container is assigned a status (**Available**, **Changed**, **Inaccessible**, or **Unloadable**) based on its current state in the datastore.

!!! note
For the full list and definitions, refer to [Container Statuses](../operations/sync/sync.md#container-statuses){target="_blank"} on the Sync Operation page. If a container is marked **Unloadable** after 3 consecutive scan or profile failures, see [Unloadable Container Error](../operations/scan/scan.md#unloadable-container-error){target="_blank"} for resolution steps (run a Sync for tables, views, or file patterns; for computed assets, force an edit on the asset and re-save it with Validate then Save).
For the full list and definitions, refer to [Container Statuses](../operations/sync/sync.md#container-statuses){target="_blank"} on the Sync Operation page. If a container is marked **Unloadable** after 3 consecutive scan or profile failures, see [Unloadable Container Error](../operations/scan/troubleshooting.md#unloadable-container-error){target="_blank"} for resolution steps (run a Sync for tables, views, or file patterns; for computed assets, force an edit on the asset and re-save it with Validate then Save).

## Actions on Container

Expand Down
2 changes: 1 addition & 1 deletion docs/data-quality-checks/authored-check.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ If the validation fails, a red message will appear saying **"Failed Validation"*
![failed](../assets/data-quality-checks/authored-check/failed.png)

!!! note "Container marked as Unloadable"
If validation fails with a message like `Container '<CONTAINER_NAME>' is marked as Unloadable. No attempt was made to load the container due to multiple consecutive failures in prior operations.`, the underlying container has been skipped after 3 consecutive scan or profile failures. For tables, views, and file patterns, run a [Sync Operation](../operations/sync/sync.md) on the datastore to reset the status. For computed assets (Computed Tables, Computed Files, Computed Joins), force an edit on the asset (click **Edit**), then click **Validate** and **Save** to re-evaluate the definition. See [Unloadable Container Error](../operations/scan/scan.md#unloadable-container-error) for the full resolution steps.
If validation fails with a message like `Container '<CONTAINER_NAME>' is marked as Unloadable. No attempt was made to load the container due to multiple consecutive failures in prior operations.`, the underlying container has been skipped after 3 consecutive scan or profile failures. For tables, views, and file patterns, run a [Sync Operation](../operations/sync/sync.md) on the datastore to reset the status. For computed assets (Computed Tables, Computed Files, Computed Joins), force an edit on the asset (click **Edit**), then click **Validate** and **Save** to re-evaluate the definition. See [Unloadable Container Error](../operations/scan/troubleshooting.md#unloadable-container-error) for the full resolution steps.

**Step 5:** Once you have a successful validation, click the **"Save"** button.

Expand Down
24 changes: 16 additions & 8 deletions docs/explore/activity.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

**Activity** in Qualytics provides a comprehensive view of all operations, helping users monitor and analyze the performance and workflows across various source datastores. Activities are categorized into **Runs** and **Schedule** operations, offering distinct insights into executed and scheduled activities.

The Rerun and Resume options depend on the type of operation. [Profile](../operations/profile/profile.md) and [Scan](../operations/scan/scan.md) support both because the system can remember where it stopped and continue from there. [Sync](../operations/sync/sync.md), [Export](../operations/export-operation/export-operation.md), and [Materialize](../operations/materialize-operation/materialize-operation.md) only support Rerun, since the system can't pick up from where it left off and must start over. External Scan doesn't support either option, as they don't apply to it.
The Rerun and Resume options depend on the type of operation. [Profile](../operations/profile/profile.md) and [Scan](../operations/scan/getting-started.md) support both because the system can remember where it stopped and continue from there. [Sync](../operations/sync/sync.md), [Export](../operations/export-operation/export-operation.md), and [Materialize](../operations/materialize-operation/materialize-operation.md) only support Rerun, since the system can't pick up from where it left off and must start over. External Scan doesn't support either option, as they don't apply to it.

Let’s get started 🚀

Expand All @@ -16,7 +16,7 @@ Let’s get started 🚀

![activity](../assets/explore/activity/activity-light.png)

You will be navigated to the **Activity** tab and here you'll see a list of operations [sync](../operations/sync/sync.md), [profile](../operations/profile/profile.md), [scan](../operations/scan/scan.md), and [external scan](../operations/external-scan/external-scan.md) across different source datastores.
You will be navigated to the **Activity** tab and here you'll see a list of operations [sync](../operations/sync/sync.md), [profile](../operations/profile/profile.md), [scan](../operations/scan/getting-started.md), and [external scan](../operations/external-scan/external-scan.md) across different source datastores.

![list](../assets/explore/activity/list-light.png)

Expand All @@ -28,7 +28,7 @@ Activities are divided into two categories: Runs and Schedule Operations. Runs p

### Runs

Runs provide a complete record of all executed operations across various source datastores. This section enables users to monitor and review activities such as [sync](../operations/sync/sync.md), [profile](../operations/profile/profile.md), [scan](../operations/scan/scan.md), and [external scan](../operations/external-scan/external-scan.md). Each run displays key details like the operation type, status, execution time, duration, and triggering method, offering a clear overview of system performance and data processing workflows.
Runs provide a complete record of all executed operations across various source datastores. This section enables users to monitor and review activities such as [sync](../operations/sync/sync.md), [profile](../operations/profile/profile.md), [scan](../operations/scan/getting-started.md), and [external scan](../operations/external-scan/external-scan.md). Each run displays key details like the operation type, status, execution time, duration, and triggering method, offering a clear overview of system performance and data processing workflows.

![run](../assets/explore/activity/runs-light.png)

Expand All @@ -37,9 +37,17 @@ Runs provide a complete record of all executed operations across various source
| 1. | Select Source Datastore | Select specific source datastores to focus on their operations. |
| 2. | Search | This feature helps users quickly find specific identifiers. |
| 3. | Sort By | **Sort By** option helps users organize the list of performed operations by criteria like Duration and Created Date for quick access. |
| 4. | Filter | The filter lets users easily refine the list of performed operations by choosing a specific Type [Scan](../operations/scan/scan.md), [Sync](../operations/sync/sync.md), [Profile](../operations/profile/profile.md), [External Scan](../operations/external-scan/external-scan.md), etc. along with Status (Success, Failure, Running, and Aborted) or **Has Logs** to view operations that completed with logs. |
| 4. | Filter | The filter lets users easily refine the list of performed operations by choosing a specific Type [Scan](../operations/scan/getting-started.md), [Sync](../operations/sync/sync.md), [Profile](../operations/profile/profile.md), [External Scan](../operations/external-scan/external-scan.md), etc. along with Status (Success, Failure, Running, and Aborted) or **Has Logs** to view operations that completed with logs. |
| 5. | Activity Heatmap | The **Activity Heatmap** shows daily activity levels, with color intensity indicating operation counts. Hovering over a square reveals details for that day. |
| 6. | Operation List | Shows a list of operations [**sync**](../operations/sync/sync.md), [**profile**](../operations/profile/profile.md), [**scan**](../operations/scan/scan.md), and [**external scan**](../operations/external-scan/external-scan.md), etc performed across various source datastores. |
| 6. | Operation List | Shows a list of operations [**sync**](../operations/sync/sync.md), [**profile**](../operations/profile/profile.md), [**scan**](../operations/scan/getting-started.md), and [**external scan**](../operations/external-scan/external-scan.md), etc performed across various source datastores. |

#### Auto-Resolved indicator on Scan operations

When a Scan operation ran with [Auto Resolve Anomalies](../operations/scan/deep-dive/read-strategies.md#auto-resolve-on-full-scans) enabled, the operation row shows a green **Anomalies Auto-Resolved** pill joined to the existing **Anomalies** pill, with the count of previously open anomalies this scan automatically resolved. Expanding the operation row reveals the same value as a stat tile in the summary card.

<!-- TODO screenshot: assets/explore/activity/auto-resolved-pill.png — Scan operation row with the joined Anomalies / Auto-Resolved pills -->

The pill and stat are hidden when the scan ran as Incremental, when **Auto Resolve Anomalies** was turned off, or when the scan did not auto-resolve any anomaly.

### Activity Heatmap

Expand Down Expand Up @@ -98,7 +106,7 @@ If the operation is a **Materialize** or **Export** run, users can click the **V

### Schedule

The Schedule section provides a complete record of all scheduled operations across various source datastores. This section enables users to monitor and review scheduled operations such as [sync](../operations/sync/sync.md), [profile](../operations/profile/profile.md), and [scan](../operations/scan/scan.md). Each scheduled operation includes key details like operation type, scheduled time, and triggering method, giving users a clear overview of system performance and data workflows.
The Schedule section provides a complete record of all scheduled operations across various source datastores. This section enables users to monitor and review scheduled operations such as [sync](../operations/sync/sync.md), [profile](../operations/profile/profile.md), and [scan](../operations/scan/getting-started.md). Each scheduled operation includes key details like operation type, scheduled time, and triggering method, giving users a clear overview of system performance and data workflows.

![schedule](../assets/explore/activity/schedule-light.png)

Expand All @@ -107,8 +115,8 @@ The Schedule section provides a complete record of all scheduled operations acro
| 1. | Selected Source Datastores | Select specific source datastores to focus on their operations. |
| 2. | Search | This feature helps users quickly find specific identifiers. |
| 3. | Sort By | **Sort By** option helps users organize the list of scheduled operations by criteria like Created Date and Operations for quick access. |
| 4. | Filter | The filter lets users easily refine the list of scheduled operations by choosing a specific operation type: [Scan](../operations/scan/scan.md), [Sync](../operations/sync/sync.md), [Profile](../operations/profile/profile.md), etc. to view. |
| 5. | Operation List | Shows the list of scheduled operations such as [sync](../operations/sync/sync.md), [profile](../operations/profile/profile.md), [scan](../operations/scan/scan.md), etc across various source datastores. |
| 4. | Filter | The filter lets users easily refine the list of scheduled operations by choosing a specific operation type: [Scan](../operations/scan/getting-started.md), [Sync](../operations/sync/sync.md), [Profile](../operations/profile/profile.md), etc. to view. |
| 5. | Operation List | Shows the list of scheduled operations such as [sync](../operations/sync/sync.md), [profile](../operations/profile/profile.md), [scan](../operations/scan/getting-started.md), etc across various source datastores. |

#### Deactivate Schedule Operation

Expand Down
Loading
Loading