Skip to content

Commit f5adaa4

Browse files
author
DavertMik
committed
improved reporter docs
1 parent a7683e7 commit f5adaa4

1 file changed

Lines changed: 56 additions & 59 deletions

File tree

‎docs/reports.md‎

Lines changed: 56 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -5,15 +5,23 @@ title: Reporters
55

66
# Reporters
77

8-
CodeceptJS prints test results to the console by default (see [CLI output](#cli-output)). For an HTML report, a pull-request comment, JUnit XML, or a hosted dashboard, it is recommeded to use **[Testomat.io Reporter](https://github.com/testomatio/reporter)**. It sends results to whichever destinations you turn on with steps, screenshots, videos, traces, and logs.
8+
CodeceptJS prints test results to the console by default (see [CLI output](#cli-output)). For an HTML report, a pull-request comment, JUnit XML, or a hosted dashboard, use **[Testomat.io Reporter](https://github.com/testomatio/reporter)**.
9+
10+
Testomat.io Reporter is open source. It can send create reports of various types:
11+
12+
- **Local files** — HTML, CSV, and Markdown reports written to disk.
13+
- **CI/CD pipeline reports** — reports added as comments to GitHub, GitLab, or Bitbucket pull requests.
14+
- **Cloud Services** — a hosted dashboard on [app.testomat.io](https://testomat.io) to keep track of results history and get analytics.
15+
16+
You choose the destinations. The reporter collects steps, screenshots, videos, traces, and logs once and sends them everywhere you enabled.
917

1018
### Install
1119

1220
```sh
1321
npm install @testomatio/reporter --save-dev
1422
```
1523

16-
Enable reporter plugin:
24+
Enable the reporter plugin and turn on the local reports you want:
1725

1826
```js
1927
// codecept.conf.js
@@ -22,122 +30,109 @@ plugins: {
2230
enabled: true,
2331
require: '@testomatio/reporter/codecept',
2432
html: true,
25-
markdown: true,
26-
csv: true,
27-
reportDir: 'output/report',
33+
// markdown: true,
34+
// csv: true,
35+
// reportDir: 'output/report',
2836
},
2937
}
3038
```
3139

32-
The local reports above are enabled directly from CodeceptJS config. If `reportDir` is omitted, reports are written to `output/report` using the CodeceptJS `output` directory.
40+
> `html`, `markdown`, and `csv` are config switches for local reports. Local reports are stored in filesystem in `output/report` dir by default.
41+
42+
### Local reports and remote destinations
3343

34-
### Enable an output
44+
Local reports are turned on with config switches and saved to disk:
3545

36-
Each output can also be enabled with environment variables. Run your tests as usual and one run feeds every output you enabled.
46+
| Config switch | File in `reportDir` |
47+
| --- | --- |
48+
| `html: true` | `testomatio-report.html` |
49+
| `markdown: true` | `testomatio-report.md` |
50+
| `csv: true` | `report.csv` |
3751

38-
| To get… | Set | Details |
39-
| --- | --- | --- |
40-
| HTML report | `TESTOMATIO_HTML_REPORT_SAVE=1` | [HTML Report](#html-report) |
41-
| Markdown report | `TESTOMATIO_MARKDOWN_REPORT_SAVE=1` | [Markdown Report](#markdown-report) |
42-
| Run Result on [app.testomat.io](https://testomat.io) | `TESTOMATIO` (project API key) | [Cloud Report](#cloud-report) |
43-
| A comment on a GitHub Pull Request | `GH_PAT` (`${{ github.token }}` in Actions) | [GitHub Report](#github-report) |
44-
| A comment on a GitLab Merge Request | `GITLAB_PAT` (token with `api` scope) | [GitLab Report](#gitlab-report) |
45-
| A comment on a Bitbucket Pull Request | `BITBUCKET_ACCESS_TOKEN` (repo access token) | [Bitbucket Report](#bitbucket-report) |
52+
Remote destinations need an access token. A token is a secret, so pass it through an environment variable or CI secret instead of committing it to the config file:
4653

47-
Screenshots and videos in these reports are uploaded to your own storage — see [Artifacts](#artifacts).
54+
| Destination | Token variable |
55+
| --- | --- |
56+
| Run result on [app.testomat.io](https://testomat.io) | `TESTOMATIO` |
57+
| Comment on a GitHub Pull Request | `GH_PAT` (`${{ github.token }}` in Actions) |
58+
| Comment on a GitLab Merge Request | `GITLAB_PAT` (token with `api` scope) |
59+
| Comment on a Bitbucket Pull Request | `BITBUCKET_ACCESS_TOKEN` (repo access token) |
4860

49-
Put the variables on CI when running tests:
61+
One run feeds every destination you enabled. On CI, keep the report switches in the config and pass only the tokens as environment variables:
5062

5163
```yaml
5264
- run: npx codeceptjs run
5365
env:
54-
TESTOMATIO_HTML_REPORT_SAVE: 1 # → output/reports/testomatio-report.html
55-
TESTOMATIO_HTML_REPORT_FOLDER: output/reports # keep it with the rest of output/
56-
GH_PAT: ${{ github.token }} # → PR comment
57-
# TESTOMATIO: ${{ secrets.TESTOMATIO }} # → testomat.io run
66+
GH_PAT: ${{ github.token }} # → Print report as PR comment
67+
# TESTOMATIO: ${{ secrets.TESTOMATIO }} # → Send report to testomat.io
5868
- uses: actions/upload-artifact@v4
5969
if: always()
6070
with:
6171
name: codeceptjs-output
6272
path: output/
6373
```
6474
65-
The GitHub pipe also needs the job to grant `permissions: pull-requests: write`.
75+
The GitHub destination also needs the job to grant `permissions: pull-requests: write`.
6676

77+
For the full list of options and environment variables, see the [reporter configuration reference](https://github.com/testomatio/reporter/blob/master/docs/configuration.md).
6778

6879
### HTML Report
6980

70-
A single self-contained HTML file with the run summary and, per test, its steps, screenshots, logs, and error. It needs no API key and no service, so it works anywhere — open it locally or attach it to a CI build.
81+
A local, self-contained HTML file saved to `output/report/testomatio-report.html`. It holds the run summary and, per test, its steps, screenshots, logs, and error. It needs no API key and no service, so it works anywhere — open it locally or attach it to a CI build.
7182

7283
![HTML report](./images/testomatio-html-report.png)
7384

74-
- Preferred in CodeceptJS 4: enable `html: true` in `plugins.testomatio` and run `npx codeceptjs run`
85+
Enable it with `html: true` in `plugins.testomatio` and run `npx codeceptjs run`. To change the file name or folder, see the [HTML pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/html.md).
7586

76-
- `TESTOMATIO_HTML_REPORT_SAVE=1` — enable the report
77-
- `TESTOMATIO_HTML_REPORT_FOLDER=output/reports` — keep it inside CodeceptJS's `output/` dir (default folder is `html-report`)
78-
- `TESTOMATIO_HTML_FILENAME` — file name, must end in `.html` (default `testomatio-report.html`)
87+
### CSV Report
7988

80-
### Cloud Report
89+
A local CSV file saved to `output/report/report.csv`, with one row per test — suite, title, and status. Use it in spreadsheets or data pipelines.
8190

82-
Sends the run to [app.testomat.io](https://testomat.io) — a hosted dashboard with run history, flaky-test detection, parallel-run merging, re-running failed tests, and notifications. Free for small teams.
91+
Enable it with `csv: true` in `plugins.testomatio`. See the [CSV pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/csv.md).
8392

84-
![Testomat.io report](https://user-images.githubusercontent.com/220264/151728836-b52d2b2b-56e1-4640-8d3a-b39de817b1fd.png)
93+
### Testomat.io Cloud Report
94+
95+
A remote destination: the run is sent to [app.testomat.io](https://testomat.io), a hosted dashboard with run history, flaky-test detection, parallel-run merging, re-running failed tests, and notifications. Free for small teams.
8596

86-
- `TESTOMATIO` — project API key; enables the pipe
87-
- `TESTOMATIO_CREATE=1` — create tests in Testomat.io that were not imported beforehand
88-
- `TESTOMATIO_TITLE` — report title
89-
- `TESTOMATIO_RUNGROUP_TITLE` — add the run to a group (e.g. `"Build ${BUILD_ID}"`)
90-
- `TESTOMATIO_PUBLISH=1` — make the report publicly accessible
97+
![Testomat.io report](https://user-images.githubusercontent.com/220264/151728836-b52d2b2b-56e1-4640-8d3a-b39de817b1fd.png)
9198

92-
More options (shared runs, rungroups, run management): [Testomat.io pipe](https://github.com/testomatio/reporter/blob/master/docs/pipes/testomatio.md).
99+
Set the `TESTOMATIO` environment variable to your project API key and run the tests. Run titles, run groups, shared runs, and publishing options: [Testomat.io pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/testomatio.md).
93100

94-
To view artifacts on cloud they must be uploaded to S3 storages. Images from [`screenshot`](/plugins#screenshot) plugin, videos from the [`screencast`](/plugins#screencast) plugin (or the Playwright helper's `video` and `trace`). Can be used with any S3 provider: AWS S3, Cloudflare R2, Google Cloud Storage (interoperability mode), DigitalOcean Spaces, MinIO.
101+
To view artifacts on the cloud, upload them to S3 storage. Images come from the [`screenshot`](/plugins#screenshot) plugin, videos from the [`screencast`](/plugins#screencast) plugin (or the Playwright helper's `video` and `trace`). Any S3 provider works: AWS S3, Cloudflare R2, Google Cloud Storage (interoperability mode), DigitalOcean Spaces, MinIO.
95102

96103
### GitHub Report
97104

98-
Posts a comment to the Pull Request: run status, pass/fail/skip counts, stack traces of the failures, screenshots, and the slowest tests. Re-running the workflow replaces the previous comment.
105+
A remote destination: a comment on the Pull Request with run status, pass/fail/skip counts, stack traces of the failures, screenshots, and the slowest tests. Re-running the workflow replaces the previous comment.
99106

100107
![GitHub report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/github.png)
101108

102-
- `GH_PAT` — GitHub token; `${{ github.token }}` works in Actions
103-
- the job must grant `permissions: pull-requests: write`
104-
- `GH_KEEP_OUTDATED_REPORTS=1` — keep previous comments instead of deleting them
109+
Set `GH_PAT` to a GitHub token (`${{ github.token }}` works in Actions) and grant the job `permissions: pull-requests: write`. More options: [GitHub pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/github.md).
105110

106111
### GitLab Report
107112

108-
Posts a comment to the Merge Request with the same summary. It needs Merge Request context, so run it in merge-request pipelines.
113+
A remote destination: a comment on the Merge Request with the same summary. Run it in merge-request pipelines (`$CI_PIPELINE_SOURCE == "merge_request_event"`).
109114

110115
![GitLab report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/gitlab.png)
111116

112-
- `GITLAB_PAT` — Personal or Project Access Token with `api` scope
113-
- run in merge-request pipelines (`$CI_PIPELINE_SOURCE == "merge_request_event"`)
114-
- `GITLAB_KEEP_OUTDATED_REPORTS=1` — keep previous comments
115-
- `GITLAB_REMOVE_ALL_OUTDATED_REPORTS=1` — remove all previous comments, not just the latest
117+
Set `GITLAB_PAT` to a Personal or Project Access Token with `api` scope. More options: [GitLab pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/gitlab.md).
116118

117119
### Bitbucket Report
118120

119-
Posts a comment to the Pull Request with the same summary. Comments are created only in `pull-requests` pipelines.
121+
A remote destination: a comment on the Pull Request with the same summary. Comments are created only in `pull-requests` pipelines.
120122

121123
![Bitbucket report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/bitbucket.png)
122124

123-
- `BITBUCKET_ACCESS_TOKEN` — repository access token with `Pull requests: Write` and `Repository: Read`
124-
- run in `pull-requests` pipelines
125-
- `BITBUCKET_KEEP_OUTDATED_REPORTS=1` — keep previous comments
125+
Set `BITBUCKET_ACCESS_TOKEN` to a repository access token with `Pull requests: Write` and `Repository: Read`. More options: [Bitbucket pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/bitbucket.md).
126126

127127
### Markdown Report
128128

129-
- Preferred in CodeceptJS 4: enable `markdown: true` in `plugins.testomatio` and run `npx codeceptjs run`
129+
A local, self-contained Markdown file saved to `output/report/testomatio-report.md`. It renders in PR comments, CI job summaries, and Slack, and is convenient for AI agents reading test results. It needs no API key.
130130

131131
![Markdown report](./images/testomatio-markdown-report.png)
132132

133-
A single self-contained Markdown file — renders in PR comments, CI job summaries, and Slack, and is convenient for AI agents reading test results. Needs no API key.
134-
135-
- `TESTOMATIO_MARKDOWN_REPORT_SAVE=1` — enable the report
136-
- `TESTOMATIO_MARKDOWN_REPORT_FOLDER=output/reports` — keep it inside CodeceptJS's `output/` dir (default folder is `md-report`)
137-
- `TESTOMATIO_MARKDOWN_FILENAME` — file name, must end in `.md` (default `testomatio-report.md`)
138-
- `TESTOMATIO_TITLE` — document title (default `Test Results`)
133+
Enable it with `markdown: true` in `plugins.testomatio`. To change the file name, folder, or title, see the [Markdown pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/markdown.md).
139134

140-
On GitHub Actions, append it to the job summary: `cat output/reports/testomatio-report.md >> "$GITHUB_STEP_SUMMARY"`.
135+
On GitHub Actions, append it to the job summary: `cat output/report/testomatio-report.md >> "$GITHUB_STEP_SUMMARY"`.
141136

142137
## JUnit XML
143138

@@ -189,6 +184,8 @@ By default CodeceptJS prints test names and failures. Add `--steps` to see each
189184

190185
```sh
191186
npx codeceptjs run --steps
187+
npx codeceptjs run --debug
188+
npx codeceptjs run --verbose
192189
```
193190

194191
`dry-run` lists tests and steps without running them:

0 commit comments

Comments
 (0)