My playwright.config.ts includes:
use: {
...
screenshot: 'only-on-failure',
}
and test failures result in screenshots being saved in /test-results when they fail locally. But when the tests fail when run in Github Actions, no screenshots are taken. So it's impossible for me to tell what's going wrong in my tests, which pass fine locally.
The only CI-specific parts of my config are:
/* Fail the build on CI if you accidentally left test.only in the source code. */
forbidOnly: !!process.env.CI,
/* Retry on CI only */
retries: process.env.CI ? 2 : 0,
/* Reporter to use. See https://playwright.dev/docs/test-reporters */
reporter: process.env.CI ? 'github' : 'list',
ETA: my action.yml attempts to upload the /test-results folder but it's always completely empty, as no screenshots were taken:
- uses: actions/upload-artifact#v2
if: always()
with:
name: playwright-test-results
path: test-results/
We can upload the test-results folder as a run artifact using GitHub's "upload-artifact" action post suite execution.
Alternatively, a step can be added in the workflow after the test run to upload the images to the desired location.
// playwright.config.ts
import { PlaywrightTestConfig, devices } from '#playwright/test';
const config: PlaywrightTestConfig = {
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 1,
use: {
trace: 'on',
screenshot: 'only-on-failure', // Capture screenshot after each test failure.
video: 'retain-on-failure', //Record video only when retrying a test for the first time.
headless: true,
viewport: { width: 1280, height: 720 },
baseURL: "https:xxxxx",
launchOptions: {
slowMo: 50,
logger: {
isEnabled: (name, severity) => name === 'browser',
log: (name, severity, message, args) => console.log(`${name} ${message}`)
}
},
actionTimeout: 10 * 1000,
navigationTimeout: 30 * 1000
},
expect: {
timeout: 10 * 1000
},
timeout: 90000,
outputDir: `${__dirname}/build`,
projects: [
{
name: 'chromium',
use: { ...devices['Desktop Chrome'] }
}
],
reporter: [
['junit', { outputFile: 'build/results.xml' }],
['html', { outputFolder: 'build/html-report', open: 'never' }],
['list']
]
};
export default config;
// .gitlab-ci.yml
.e2e:
stage: postdeploy
image: xxxxx
script:
- pnpm xxx
- pnpm nx test e2e // "test": "playwright test --output build --workers 2",
after_script:
- ls -l tests/e2e
- cp -r tests/e2e/build ./results
artifacts:
when: always
expire_in: never
name: 'test-artifacts'
paths:
- results/html-report
exclude:
- results/results.xml
reports:
junit: results/results.xml
Related
In my Cypress testing suite, I'm getting the following issue and I'm not understanding how to fix it. I understood is a path issue but cannot figure out what should be the right path. The issue is only with my smoke test but the rest of my scopes is working fine.
I'm not sure what is wrong here, probably I configured the paths wrong but whatever I do seems not working.
The spec files are existing and there are configured as follows in a smoke.json file
{
"specPattern": [
"authentication/log-in.cy.js",
"candidate/study-sign-up.cy.js",
"candidate/click-all-tabs.cy.js"
]
}
The folder structure is as follows
Details on the CY configuration
const { defineConfig } = require('cypress');
module.exports = defineConfig({
e2e: {
chromeWebSecurity: false,
viewportWidth: 1280,
viewportHeight: 800,
screenshotsFolder: 'artifacts',
video: false,
reporter: 'junit',
reporterOptions: {
mochaFile: 'results/test-results-[hash].xml',
},
retries: 1,
defaultCommandTimeout: 60000,
setupNodeEvents(on, config) {
return require('./cypress/plugins/index.js')(on, config);
},
},
});
FYI, none of the other Stackoverflow questions/answers have resolved this for me.
In an Angular project, we're using Protractor Cucumber Framework for our E2E tests.
I can't figure out how to run only one single test via tags.
You're supposed to be able to edit the tags inside of the cucumberOpts property of the protractor.conf.js file. But when I add a tag #testOnlyThis there, then add that tag to a test in a .feature file, then run npm run e2e:ci (which, according to package.json, runs "protractor ./e2e/protractor.conf.js"), Protractor still runs every single E2E test in our suite. Other changes made in the protractor.conf.js file take effect, but editing the tags seems to have zero effect.
What gives?
protractor.conf.js
// #ts-check
// Protractor configuration file, see link for more information
// https://github.com/angular/protractor/blob/master/lib/config.ts
const path = require('path');
const fs = require('fs');
const cucumberJunit = require('protractor-cucumber-junit/lib/cucumber_junit');
const downloadsPath = path.resolve(__dirname, 'downloads');
const reportingPath = path.resolve(__dirname, 'reporting/protractor-cucumber-framework');
let startDate;
/**
* #type { import("protractor").Config }
*/
exports.config = {
allScriptsTimeout: 20000,
specs: ['./src/features/**/**/**/**/**/*.feature'],
resultJsonOutputFile: 'reporting/results.json',
capabilities: {
browserName: 'chrome',
shardTestFiles: true,
maxInstances: 1,
chromeOptions: {
prefs: {
'plugins.always_open_pdf_externally': true,
download: {
directory_upgrade: true,
prompt_for_download: false,
default_directory: downloadsPath,
},
},
args: [
'--no-sandbox',
'--test-type=browser',
'--disable-gpu',
'--log-level=1',
'--disable-dev-shm-usage',
// '--disk-cache-dir=null',
],
},
},
directConnect: true,
SELENIUM_PROMISE_MANAGER: false,
noGlobals: true,
baseUrl: 'https://mybaseurl.com',
framework: 'custom',
frameworkPath: require.resolve('protractor-cucumber-framework'),
cucumberOpts: {
require: ['./src/step-definitions/*steps.ts'],
tags: ['#testOnlyThis', '~#ignore'],
format: ['json:./reporting/protractor-cucumber-framework/results.json'],
retry: 2,
},
onPrepare() {
require('ts-node').register({
project: require('path').join(__dirname, './tsconfig.json'),
});
const chai = require('chai');
const chaiAsPromised = require('chai-as-promised');
chai.use(chaiAsPromised);
},
beforeLaunch() {
startDate = new Date().getTime();
if (!fs.existsSync(downloadsPath)) {
fs.mkdirSync(downloadsPath);
}
if (!fs.existsSync(reportingPath)) {
fs.mkdirSync(reportingPath, { recursive: true });
}
console.log(`process.env.E2E_LANGUAGE is set to: '${process.env.E2E_LANGUAGE}'`);
},
afterLaunch() {
const endDate = new Date().getTime();
const duration = (endDate - startDate) / (60 * 1000);
console.log(
`ALL TESTS EXECUTION TIME: ${Math.floor(duration)}m${Math.round((duration % 1) * 60)}s`,
);
const file = fs.readFileSync('reporting/results.json', 'utf-8');
// #ts-ignore
const xml = cucumberJunit(file);
fs.writeFileSync('e2e/reporting/results.xml', xml);
fs.rmdirSync(reportingPath, { recursive: true });
},
};
Hi Try declaring your tags like this, if that helps
cucumberOpts: {
tags: '#Smoke,#Intgr'
}
I´m running into a bug where, when using Quasar together with Knex, it is demanding a package for a driver that's not even in use:
ERROR Failed to compile with 1 errors 8:34:02 AM
This dependency was not found:
* mssql/package.json in ../api/node_modules/knex/lib/dialects/mssql/index.js
To install it, you can run: npm install --save mssql/package.json
This seems to be a problem with webpack, as per those other issues:
https://github.com/knex/knex/issues/1128
https://github.com/knex/knex/issues/1446#issuecomment-537715431
https://github.com/quasarframework/quasar/issues/6553
Neither the technique of creating plugins to filter out the dependencies, nor the configuration of externals seems to work:
quasar.conf.js (please, take a look at the extendWebpack section):
// Configuration for your app
// https://quasar.dev/quasar-cli/quasar-conf-js
const webpack = require('webpack');
module.exports = function (ctx) {
return {
// Quasar looks for *.js files by default
sourceFiles: {
router: 'src/router/index.ts',
store: 'src/store/index.ts'
},
// app boot file (/src/boot)
// --> boot files are part of "main.js"
// https://quasar.dev/quasar-cli/cli-documentation/boot-files
boot: [
'i18n'
],
// https://quasar.dev/quasar-cli/quasar-conf-js#Property%3A-css
css: [
'app.scss'
],
// https://github.com/quasarframework/quasar/tree/dev/extras
extras: [
// 'ionicons-v4',
// 'mdi-v4',
// 'fontawesome-v5',
// 'eva-icons',
// 'themify',
// 'line-awesome',
// 'roboto-font-latin-ext', // this or either 'roboto-font', NEVER both!
'roboto-font', // optional, you are not bound to it
'material-icons' // optional, you are not bound to it
],
// https://quasar.dev/quasar-cli/quasar-conf-js#Property%3A-framework
framework: {
config: {
dark: 'auto'
},
iconSet: 'material-icons', // Quasar icon set
lang: 'en-us', // Quasar language pack
// Possible values for "all":
// * 'auto' - Auto-import needed Quasar components & directives
// (slightly higher compile time; next to minimum bundle size; most convenient)
// * false - Manually specify what to import
// (fastest compile time; minimum bundle size; most tedious)
// * true - Import everything from Quasar
// (not treeshaking Quasar; biggest bundle size; convenient)
all: 'auto',
components: [],
directives: [],
// Quasar plugins
plugins: []
},
// https://quasar.dev/quasar-cli/cli-documentation/supporting-ie
supportIE: false,
// Full list of options: https://quasar.dev/quasar-cli/quasar-conf-js#Property%3A-build
build: {
vueRouterMode: 'hash', // available values: 'hash', 'history'
// showProgress: false,
// gzip: true,
// analyze: true,
// Options below are automatically set depending on the env, set them if you want to override
// preloadChunks: false,
// extractCSS: false,
// https://quasar.dev/quasar-cli/cli-documentation/handling-webpack
extendWebpack (cfg) {
cfg.module.rules.push({
enforce: 'pre',
test: /\.(js|vue)$/,
loader: 'eslint-loader',
exclude: /node_modules/,
options: {
formatter: require('eslint').CLIEngine.getFormatter('stylish')
}
})
// Contornando problema onde o webpack tenta, indevidamente, forçar o uso dos drivers não utilizados do Knex
// Vide: https://github.com/knex/knex/issues/1446#issuecomment-537715431
/*
// force unused dialects to resolve to the only one we use
// and for whom we have the dependencies installed
cfg.plugins.push(new webpack.ContextReplacementPlugin(/knex\/lib\/dialects/, /postgres\/index.js/));
// pg optionally tries to require pg-native
// replace it by a noop (real module from npm)
cfg.plugins.push(new webpack.NormalModuleReplacementPlugin(/pg-native/, 'noop2'));
*/
if (!cfg.externals) {
cfg.externals = {};
}
Object.assign(cfg.externals, {
// Possible drivers for knex - we'll ignore them
'sqlite3': 'sqlite3',
'mariasql': 'mariasql',
'mssql': 'mssql',
'mysql': 'mysql',
'oracle': 'oracle',
'strong-oracle': 'strong-oracle',
'oracledb': 'oracledb',
'pg': 'pg',
'pg-query-stream': 'pg-query-stream'
});
}
},
// Full list of options: https://quasar.dev/quasar-cli/quasar-conf-js#Property%3A-devServer
devServer: {
https: false,
port: 8080,
open: true // opens browser window automatically
},
// animations: 'all', // --- includes all animations
// https://quasar.dev/options/animations
animations: [],
// https://quasar.dev/quasar-cli/developing-ssr/configuring-ssr
ssr: {
pwa: false
},
// https://quasar.dev/quasar-cli/developing-pwa/configuring-pwa
pwa: {
workboxPluginMode: 'GenerateSW', // 'GenerateSW' or 'InjectManifest'
workboxOptions: {}, // only for GenerateSW
manifest: {
name: 'Quasar App',
short_name: 'Quasar App',
description: 'A Quasar Framework app',
display: 'standalone',
orientation: 'portrait',
background_color: '#ffffff',
theme_color: '#027be3',
icons: [
{
src: 'statics/icons/icon-128x128.png',
sizes: '128x128',
type: 'image/png'
},
{
src: 'statics/icons/icon-192x192.png',
sizes: '192x192',
type: 'image/png'
},
{
src: 'statics/icons/icon-256x256.png',
sizes: '256x256',
type: 'image/png'
},
{
src: 'statics/icons/icon-384x384.png',
sizes: '384x384',
type: 'image/png'
},
{
src: 'statics/icons/icon-512x512.png',
sizes: '512x512',
type: 'image/png'
}
]
}
},
// Full list of options: https://quasar.dev/quasar-cli/developing-cordova-apps/configuring-cordova
cordova: {
// noIosLegacyBuildFlag: true, // uncomment only if you know what you are doing
id: 'org.cordova.quasar.app'
},
// Full list of options: https://quasar.dev/quasar-cli/developing-capacitor-apps/configuring-capacitor
capacitor: {
hideSplashscreen: true
},
// Full list of options: https://quasar.dev/quasar-cli/developing-electron-apps/configuring-electron
electron: {
bundler: 'packager', // 'packager' or 'builder'
packager: {
// https://github.com/electron-userland/electron-packager/blob/master/docs/api.md#options
// OS X / Mac App Store
// appBundleId: '',
// appCategoryType: '',
// osxSign: '',
// protocol: 'myapp://path',
// Windows only
// win32metadata: { ... }
},
builder: {
// https://www.electron.build/configuration/configuration
appId: 'q-frontend'
},
// More info: https://quasar.dev/quasar-cli/developing-electron-apps/node-integration
nodeIntegration: true,
extendWebpack (cfg) {
// do something with Electron main process Webpack cfg
// chainWebpack also available besides this extendWebpack
}
}
}
}
Is there anyway to work around this?
I've the following grunt file which runs the mocha tests OK (I get results of the test after running grunt.js)Now I want to add a code and I use the https://github.com/taichi/grunt-istanbul module. but when I run the grunt.js nothing happen,any idea?
What I want is simply after that mocha test are running it will run the code coverage with some reports? any new code coverage will be great
This is my project structure
myApp
-server.js
-app.js
-test
-test1.spec
-test2.spec
-test-reports
-grunt.js
-utils
-file1.js
-file2.js
-controller
-file1.js
-file2.js
This is the code inside the grunt for what I've tried
module.exports = function (grunt) {
var path = require('path');
process.env.RESOURCE_PATH_PREFIX = "../";
var d = new Date();
var datestring = d.getDate() + "-" + (d.getMonth() + 1) + "-" + d.getFullYear() + " " +
d.getHours() + ":" + d.getMinutes();
var npmCommand = path.dirname(process.execPath).concat('/npm');
var reportDir = "./test-reports/" + datestring;
grunt.initConfig({
jasmine_nodejs: {
// task specific (default) options
options: {
specNameSuffix: ["-spec.js"],
helperNameSuffix: "helper.js",
useHelpers: false,
stopOnFailure: false,
reporters: {
console: {
colors: true,
},
junit: {
savePath: "./test-reports",
filePrefix: "testresult",
}
}
},
test: {
specs: [
"test/*",
]
},
makeReport: {
src: './test-reports/coverage.json',//should I create this file?or its created automatically ?
options: {
type: ['lcov', 'html'],
dir: reportDir,
print: 'detail'
}
},
coverage: {
APP_DIR_FOR_CODE_COVERAGE: "./utils/*.js",//HERE IS THE PATH OF THE UTILS Folder which all the js login which I want to test there
clean: ['build'],
instrument: {
files: tasks,//WHAT IS TASKS????
options: {
lazy: true,
basePath: 'build/instrument/'//I DONT KNOW WHAT IT IS???
}
},
reloadTasks: {
rootPath: 'build/instrument/tasks'//SHOULD I USE IT????
},
storeCoverage: {
options: {
dir: reportDir
}
}
}
});
grunt.loadNpmTasks('grunt-jasmine-nodejs');
grunt.loadNpmTasks('grunt-istanbul');
grunt.registerTask('default', ['jasmine_nodejs']);
grunt.registerTask('cover', ['instrument', 'test',
'storeCoverage', 'makeReport']);
};
If there is mocha code coverage which can help it will be great either, I want that after I run the test I will able to see report with all the code coverage.
I want that the coverage will be done for folder utils and controller (all the files there) how should I config that?
UPDATE
This is what I use for jasmin and I think I should change to mocha
jasmine_nodejs: {
// task specific (default) options
options: {
specNameSuffix: ["-spec.js"], // also accepts an array
helperNameSuffix: "helper.js",
useHelpers: false,
stopOnFailure: false,
reporters: {
console: {
colors: true,
cleanStack: 1, // (0|false)|(1|true)|2|3
verbosity: 4, // (0|false)|1|2|3|(4|true)
listStyle: "indent", // "flat"|"indent"
activity: false
},
junit: {
savePath: "./test-reports",
filePrefix: "testresult",
consolidate: true,
useDotNotation: true
}
}
},
test: {
// target specific options
options: {
useHelpers: false
},
// spec files
specs: [
"test/*",
]
}
},
How should I change it?
The syntax of my test are similar(jasmine/mocha) and what I want is simply to run my test and after run the code coverage
I'll give you an indirect answer. I've gotten code coverage to work before, but with a different plugin (and with mocha). I'm not sure if you're open to swapping out jasmine for mocha but I will say that I struggled with various code coverage plugins before coming across this one. I think you'll agree the configuration is both concise and obvious.
The plugin you want is grunt-mocha-istbanbul, and here is a sample configuration for your Gruntfile:
module.exports = function(grunt) {
grunt.initConfig({
clean: ['coverage'],
mocha_istanbul: {
coverage: {
src: 'test',
options: {
timeout: 20000,
'report-formats': 'html',
print: 'summary',
check: {
lines: 90,
statements: 90,
functions: 100,
branches: 80
}
}
}
}
});
grunt.loadNpmTasks('grunt-contrib-clean');
grunt.loadNpmTasks('grunt-mocha-istanbul');
grunt.registerTask('default', ['clean', 'mocha_istanbul']);
}
I want to create a Gruntfile.js to run bunch of phantomjs tests, when I execute > grunt run-test from commandline, it runs a bunch of tests. I created a Gruntfile.js and package.json which works ok and it reads bunch of tests from a directory. Now my problem is that when I write a phantomjs test and run the "grunt", it gives me this error:
Error: Cannot find module 'system'
Error: Cannot find module 'phantom'
However phantomjs is installed before by using npm install phantomjs
Example of phantomtest which gives me the above error:
var system = require('system');
if (system.args.length === 1) {
console.log('Try to pass some args when invoking this script!');
} else {
system.args.forEach(function (arg, i) {
console.log(i + ': ' + arg);
});
}
phantom.exit();
When I run phantomjs test1 (name of the test file) it runs the test so I think maybe I should append "phantomjs" somewhere in the Gruntfile. Any idea?
Gruntfile.js
'use strict';
module.exports = function(grunt) {
// Project configuration.
grunt.initConfig({
jshint: {
all: [
'Gruntfile.js',
'tests/*.js',
'<%= nodeunit.tests %>',
],
options: {
jshintrc: '.jshintrc',
},
},
// Before generating any new files, remove any previously-created files.
clean: {
tests: ['tmp'],
},
// Configuration to be run (and then tested).
testArgs: {
configFile:"test/testConf.js",
options: {
args: {
params: {
number: 1,
bool: true,
str: "string",
nil: null, // Null is not supported.
obj: {
array: [1, 2, 3],
undef: undefined
}
},
capabilities: {
'browserName': 'chrome'
},
rootElement:"body",
specs:["test/argsTest.js"],
verbose:true
}
}
},
testDebug: {
configFile:"test/testConf.js",
options: {
debug:true,
args: {
specs:["test/debugTest.js"],
}
}
},
// Unit tests.
nodeunit: {
tests: ['tests/*_test.js'],
},
});
// Actually load this plugin's task(s).
grunt.loadTasks('tests');
// These plugins provide necessary tasks.
grunt.loadNpmTasks('grunt-contrib-jshint');
grunt.loadNpmTasks('grunt-contrib-clean');
grunt.loadNpmTasks('grunt-contrib-nodeunit');
// Whenever the "test" task is run, first clean the "tmp" dir, then run this
// plugin's task(s), then test the result.
grunt.registerTask('test', ['clean']);
// By default, lint and run all tests.
grunt.registerTask('default', ['jshint', 'test']);
};