Skip to content

Commit 9da1079

Browse files
yaRnMcDonutsyaRnMcDonuts
authored
Merge pull request #2280 from richardTingle/#2279-screenshot-tests
#2279 screenshot tests
2 parents 36aac25 + e04ed8a commit 9da1079

26 files changed

+2417
-1
lines changed

‎.github/workflows/main.yml‎

Lines changed: 43 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,49 @@ on:
5656
types: [published]
5757

5858
jobs:
59-
59+
ScreenshotTests:
60+
name: Run Screenshot Tests
61+
runs-on: ubuntu-latest
62+
permissions:
63+
contents: read
64+
steps:
65+
- uses: actions/checkout@v4
66+
- name: Set up JDK 17
67+
uses: actions/setup-java@v4
68+
with:
69+
java-version: '17'
70+
distribution: 'temurin'
71+
- name: Install Mesa3D
72+
run: |
73+
sudo apt-get update
74+
sudo apt-get install -y mesa-utils libgl1-mesa-dri libgl1 libglx-mesa0 xvfb
75+
- name: Set environment variables for Mesa3D
76+
run: |
77+
echo "LIBGL_ALWAYS_SOFTWARE=1" >> $GITHUB_ENV
78+
echo "MESA_LOADER_DRIVER_OVERRIDE=llvmpipe" >> $GITHUB_ENV
79+
- name: Start xvfb
80+
run: |
81+
sudo Xvfb :99 -ac -screen 0 1024x768x16 &
82+
export DISPLAY=:99
83+
echo "DISPLAY=:99" >> $GITHUB_ENV
84+
- name: Verify Mesa3D Installation
85+
run: |
86+
glxinfo | grep "OpenGL"
87+
- name: Validate the Gradle wrapper
88+
uses: gradle/actions/wrapper-validation@v3
89+
- name: Test with Gradle Wrapper
90+
run: |
91+
./gradlew :jme3-screenshot-test:screenshotTest
92+
- name: Upload Test Reports
93+
uses: actions/upload-artifact@master
94+
if: always()
95+
with:
96+
name: screenshot-test-report
97+
retention-days: 30
98+
path: |
99+
**/build/reports/**
100+
**/build/changed-images/**
101+
**/build/test-results/**
60102
# Build the natives on android
61103
BuildAndroidNatives:
62104
name: Build natives for android

‎jme3-screenshot-tests/README.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# jme3-screenshot-tests
2+
3+
This module contains tests that compare screenshots of the JME3 test applications to reference images. The tests are run using
4+
the following command:
5+
6+
```
7+
./gradlew :jme3-screenshot-test:screenshotTest
8+
```
9+
10+
This will create a report in `jme3-screenshot-test/build/reports/ScreenshotDiffReport.html` that shows the differences between the reference images and the screenshots taken during the test run. Note that this is an ExtentReport.
11+
12+
This is most reliable when run on the CI server. The report can be downloaded from the artifacts section of the pipeline (once the full pipeline has completed). If you go into
13+
the Actions tab (on GitHub) and find your pipeline you can download the report from the Artifacts section. It will be called screenshot-test-report.
14+
15+
## Machine variability
16+
17+
It is important to be aware that the tests are sensitive to machine variability. Different GPUs may produce subtly different pixel outputs
18+
(that look identical to a human user). The tests are run on a specific machine and the reference images are generated on that machine. If the tests are run on a different machine, the images may not match the reference images and this is "fine". If you run these on your local machine compare the differences by eye in the report, don't wory about failing tests.
19+
20+
## Parameterised tests
21+
22+
By default, the tests use the class and method name to produce the screenshot image name. E.g. org.jmonkeyengine.screenshottests.effects.TestExplosionEffect.testExplosionEffect_f15.png is the testExplosionEffect test at frame 15. If you are using parameterised tests this won't work (as all the tests have the same function name). In this case you should specify the image name (including whatever parameterised information to make it unique). E.g.
23+
24+
```
25+
screenshotTest(
26+
....
27+
).setFramesToTakeScreenshotsOn(45)
28+
.setBaseImageFileName("some_unique_name_" + theParameterGivenToTest)
29+
.run();
30+
)
31+
```
32+
33+
## Non-deterministic (and known bad) tests
34+
35+
By default, screenshot variability will cause the pipeline to fail. If a test is non-deterministic (e.g. includes randomness) or
36+
is a known accepted failure (that will be fixed "at some point" but not now) that can be non-desirable. In that case you can
37+
change the behaviour of the test such that these are marked as warnings in the generated report but don't fail the test
38+
39+
```
40+
screenshotTest(
41+
....
42+
).setFramesToTakeScreenshotsOn(45)
43+
.setTestType(TestType.NON_DETERMINISTIC)
44+
.run();
45+
)
46+
```
47+
48+
## Accepting new images
49+
50+
It may be the case that a change makes an improvement to the library (or the test is entirely new) and the new image should be accepted as the new reference image. To do this, copy the new image to the `src/test/resources` directory. The new image can be found in the `build/changed-images` directory, however it is very important that the image come from the reference machine. This can be obtained from the CI server. The job runs only if there is an active pull request (to one of the mainline branches; e.g. master or 3.7). If you go into the Actions tab and find your pipeline you can download the report and changed images from the Artifacts section.

‎jme3-screenshot-tests/build.gradle‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
plugins {
2+
id 'java'
3+
}
4+
5+
repositories {
6+
mavenCentral()
7+
}
8+
9+
dependencies {
10+
implementation project(':jme3-desktop')
11+
implementation project(':jme3-core')
12+
implementation project(':jme3-effects')
13+
implementation project(':jme3-terrain')
14+
implementation project(':jme3-lwjgl3')
15+
implementation project(':jme3-plugins')
16+
17+
implementation 'com.aventstack:extentreports:5.1.1'
18+
implementation platform('org.junit:junit-bom:5.9.1')
19+
implementation 'org.junit.jupiter:junit-jupiter'
20+
testRuntimeOnly project(':jme3-testdata')
21+
}
22+
23+
tasks.register("screenshotTest", Test) {
24+
useJUnitPlatform{
25+
filter{
26+
includeTags 'integration'
27+
}
28+
}
29+
}
30+
31+
32+
test {
33+
useJUnitPlatform{
34+
filter{
35+
excludeTags 'integration'
36+
}
37+
}
38+
}

‎jme3-screenshot-tests/src/main/java/org/jmonkeyengine/screenshottests/testframework/App.java‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
/*
2+
* Copyright (c) 2024 jMonkeyEngine
3+
* All rights reserved.
4+
*
5+
* Redistribution and use in source and binary forms, with or without
6+
* modification, are permitted provided that the following conditions are
7+
* met:
8+
*
9+
* * Redistributions of source code must retain the above copyright
10+
* notice, this list of conditions and the following disclaimer.
11+
*
12+
* * Redistributions in binary form must reproduce the above copyright
13+
* notice, this list of conditions and the following disclaimer in the
14+
* documentation and/or other materials provided with the distribution.
15+
*
16+
* * Neither the name of 'jMonkeyEngine' nor the names of its contributors
17+
* may be used to endorse or promote products derived from this software
18+
* without specific prior written permission.
19+
*
20+
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
21+
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
22+
* TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
23+
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
24+
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
25+
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
26+
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
27+
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
28+
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
29+
* NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
30+
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
31+
*/
32+
package org.jmonkeyengine.screenshottests.testframework;
33+
34+
import com.jme3.app.SimpleApplication;
35+
import com.jme3.app.state.AppState;
36+
import com.jme3.app.state.VideoRecorderAppState;
37+
import com.jme3.math.ColorRGBA;
38+
39+
/**
40+
* The app used for the tests. AppState(s) are used to inject the actual test code.
41+
* @author Richard Tingle (aka richtea)
42+
*/
43+
public class App extends SimpleApplication {
44+
45+
public App(AppState... initialStates){
46+
super(initialStates);
47+
}
48+
49+
@Override
50+
public void simpleInitApp(){
51+
getViewPort().setBackgroundColor(ColorRGBA.Black);
52+
setTimer(new VideoRecorderAppState.IsoTimer(60));
53+
}
54+
55+
}

‎jme3-screenshot-tests/src/main/java/org/jmonkeyengine/screenshottests/testframework/ExtentReportExtension.java‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
/*
2+
* Copyright (c) 2024 jMonkeyEngine
3+
* All rights reserved.
4+
*
5+
* Redistribution and use in source and binary forms, with or without
6+
* modification, are permitted provided that the following conditions are
7+
* met:
8+
*
9+
* * Redistributions of source code must retain the above copyright
10+
* notice, this list of conditions and the following disclaimer.
11+
*
12+
* * Redistributions in binary form must reproduce the above copyright
13+
* notice, this list of conditions and the following disclaimer in the
14+
* documentation and/or other materials provided with the distribution.
15+
*
16+
* * Neither the name of 'jMonkeyEngine' nor the names of its contributors
17+
* may be used to endorse or promote products derived from this software
18+
* without specific prior written permission.
19+
*
20+
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
21+
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
22+
* TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
23+
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
24+
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
25+
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
26+
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
27+
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
28+
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
29+
* NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
30+
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
31+
*/
32+
package org.jmonkeyengine.screenshottests.testframework;
33+
34+
import com.aventstack.extentreports.ExtentReports;
35+
import com.aventstack.extentreports.ExtentTest;
36+
import com.aventstack.extentreports.reporter.ExtentSparkReporter;
37+
import com.aventstack.extentreports.reporter.configuration.Theme;
38+
import org.junit.jupiter.api.extension.AfterAllCallback;
39+
import org.junit.jupiter.api.extension.BeforeAllCallback;
40+
import org.junit.jupiter.api.extension.BeforeTestExecutionCallback;
41+
import org.junit.jupiter.api.extension.ExtensionContext;
42+
import org.junit.jupiter.api.extension.TestWatcher;
43+
44+
import java.util.Optional;
45+
46+
/**
47+
* This creates the Extent report and manages the test lifecycle
48+
*
49+
* @author Richard Tingle (aka richtea)
50+
*/
51+
public class ExtentReportExtension implements BeforeAllCallback, AfterAllCallback, TestWatcher, BeforeTestExecutionCallback{
52+
private static ExtentReports extent;
53+
private static final ThreadLocal<ExtentTest> test = new ThreadLocal<>();
54+
55+
@Override
56+
public void beforeAll(ExtensionContext context) {
57+
if(extent==null){
58+
ExtentSparkReporter spark = new ExtentSparkReporter("build/reports/ScreenshotDiffReport.html");
59+
spark.config().setTheme(Theme.STANDARD);
60+
spark.config().setDocumentTitle("Screenshot Test Report");
61+
spark.config().setReportName("Screenshot Test Report");
62+
extent = new ExtentReports();
63+
extent.attachReporter(spark);
64+
}
65+
}
66+
67+
@Override
68+
public void afterAll(ExtensionContext context) {
69+
/*
70+
* this writes the entire report after each test class. This sucks but I don't think there is
71+
* anywhere else I can hook into the lifecycle of the end of all tests to write the report.
72+
*/
73+
extent.flush();
74+
}
75+
76+
@Override
77+
public void testSuccessful(ExtensionContext context) {
78+
getCurrentTest().pass("Test passed");
79+
}
80+
81+
@Override
82+
public void testFailed(ExtensionContext context, Throwable cause) {
83+
getCurrentTest().fail(cause);
84+
}
85+
86+
@Override
87+
public void testAborted(ExtensionContext context, Throwable cause) {
88+
getCurrentTest().skip("Test aborted " + cause.toString());
89+
}
90+
91+
@Override
92+
public void testDisabled(ExtensionContext context, Optional<String> reason) {
93+
getCurrentTest().skip("Test disabled: " + reason.orElse("No reason"));
94+
}
95+
96+
@Override
97+
public void beforeTestExecution(ExtensionContext context) {
98+
String testName = context.getDisplayName();
99+
test.set(extent.createTest(testName));
100+
}
101+
102+
public static ExtentTest getCurrentTest() {
103+
return test.get();
104+
}
105+
}

‎jme3-screenshot-tests/src/main/java/org/jmonkeyengine/screenshottests/testframework/PixelSamenessDegree.java‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
/*
2+
* Copyright (c) 2024 jMonkeyEngine
3+
* All rights reserved.
4+
*
5+
* Redistribution and use in source and binary forms, with or without
6+
* modification, are permitted provided that the following conditions are
7+
* met:
8+
*
9+
* * Redistributions of source code must retain the above copyright
10+
* notice, this list of conditions and the following disclaimer.
11+
*
12+
* * Redistributions in binary form must reproduce the above copyright
13+
* notice, this list of conditions and the following disclaimer in the
14+
* documentation and/or other materials provided with the distribution.
15+
*
16+
* * Neither the name of 'jMonkeyEngine' nor the names of its contributors
17+
* may be used to endorse or promote products derived from this software
18+
* without specific prior written permission.
19+
*
20+
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
21+
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
22+
* TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
23+
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
24+
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
25+
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
26+
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
27+
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
28+
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
29+
* NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
30+
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
31+
*/
32+
package org.jmonkeyengine.screenshottests.testframework;
33+
34+
import com.jme3.math.ColorRGBA;
35+
36+
/**
37+
* @author Richard Tingle (aka richtea)
38+
*/
39+
public enum PixelSamenessDegree{
40+
SAME(1, null),
41+
NEGLIGIBLY_DIFFERENT(1, ColorRGBA.Green),
42+
SUBTLY_DIFFERENT(10, ColorRGBA.Blue),
43+
44+
MEDIUMLY_DIFFERENT(20, ColorRGBA.Yellow),
45+
46+
VERY_DIFFERENT(60,ColorRGBA.Orange),
47+
48+
EXTREMELY_DIFFERENT(100,ColorRGBA.Red);
49+
50+
private final int maximumAllowedDifference;
51+
52+
private final ColorRGBA colorInDebugImage;
53+
54+
PixelSamenessDegree(int maximumAllowedDifference, ColorRGBA colorInDebugImage){
55+
this.colorInDebugImage = colorInDebugImage;
56+
this.maximumAllowedDifference = maximumAllowedDifference;
57+
}
58+
59+
public ColorRGBA getColorInDebugImage(){
60+
return colorInDebugImage;
61+
}
62+
63+
public int getMaximumAllowedDifference(){
64+
return maximumAllowedDifference;
65+
}
66+
}

0 commit comments

Comments
 (0)