We Keep Coding

UI5 Custom Middleware often cannot parse JSON bodies when accessed via Karma testing suite

Ok, this is a very specific problem I have been struggling with for over a week now. I am developing a SAP UI5 application where UI testing is done via Opa5/QUnit. Serving the application directly via npm does work without problems, however, using Karma (targeting a headless approach) two problems have surfaced which seem to be caused by the used custom Middleware:
res.status() / res.header() do not work (only native node methods like res.setHeader())
Using a body parser (no matter wheter express.json() or deprecated bodyparser.json()), the parser middleware seems to be working forever until the browser request fails after exact 20 or 40 seconds (Chrome only shows the "Stalled" timebar). This happens very often, but not always.
While there is a workaround for the first issue (but still - would be interesting to know why this happens only with Karma) I can't find a solution for the failing requests. I tried changing the browser Karma uses, changing from HTML to script mode, including several plugins and also analyzed packets via Wireshark because browsers show no difference at all between normal and Karma execution.
Through Wireshark I found out that the Karma browser keeps closing Websockets after requests are done while the normal browser doesn't (even when application is served via Karma). Also, in rare cases of working POST JSON requests, content length or processing time do not seem to have an effect.
karma.conf.js:
module.exports = function(config) {
"use strict";
config.set({
frameworks: ['ui5'],
reporters: ["progress"],
browsers: ["Chrome_without_security"],
ui5: {
mode: "html",
testpage: "webapp/test/integration/opaTests.qunit.html",
configPath: "ui5-testing.yaml",
},
customLaunchers: {
Chrome_without_security: {
base: 'Chrome',
flags: ['--disable-web-security', '--no-sandbox']
}
},
singleRun: true,
browserNoActivityTimeout: 400000,
//logLevel: config.LOG_DEBUG,
});
};
ui5-testing.yaml:
specVersion: '2.1'
metadata:
name: grunt-build
type: application
framework:
name: SAPUI5
version: "1.84.0"
libraries:
- name: sap.m
- name: sap.ui.core
- name: sap.ui.layout
- name: sap.ui.support
development: true
- name: sap.ui.table
- name: sap.ui.unified
#- name: sap.ui.model
- name: sap.ushell
development: true
- name: themelib_sap_fiori_3
optional: true
- name: themelib_sap_belize
optional: true
#- name: themelib_sap_bluecrystal
# optional: true
- name: sap.f
- name: sap.tnt
resources:
configuration:
paths:
webapp: /webapp
server:
customMiddleware:
- name: proxy
beforeMiddleware: serveResources
configuration:
testing: true
---
specVersion: '2.1'
kind: extension
type: server-middleware
metadata:
name: proxy
middleware:
path: lib/middleware/proxy.js
proxy.js:
const express = require('express')
module.exports = function ({
resources,
middlewareUtil,
options
}) {
require('dotenv').config();
const axios = require('axios')
var admin = require('firebase-admin');
const app = express();
...
app.use(express.json());
app.use((req, res, next) => { // Most POST Requests with application/json header do not enter this!
...
}
return app;
};
Requesting method (example):
upsert: function (aElements, iTimeout) {
let that = this;
return new Promise((resolve, reject) => {
let sBody = JSON.stringify(aElements);
let xhr = new XMLHttpRequest();
xhr.open('POST', UPSERT_URL, true);
xhr.onload = function (oResponse) {
that.proceedResponse(oResponse, this)
.then(() => resolve())
.catch(iStatus => reject(iStatus));
};
xhr.onerror = function (oError) {
reject(oError);
};
xhr.ontimeout = function (oError) {
console.error(`The request for ${UPSERT_URL} timed out.`);
reject(oError);
};
xhr.timeout = iTimeout;
xhr.setRequestHeader("Content-Type", "application/json");
xhr.send(sBody);
});
},
Normal call: ui5 serve -p 8080 -o /test/integration/opaTests.qunit.html --config ui5-testing.yaml
Karma call: karma start
Maybe someone is able to help me here, thank you very much!

Grunt-express and static routes : Server should provide a function called "listen" that acts as http.Server.listen

I am trying to set-up Grunt to start my express server, using grunt-express.
After reading the docs and this SO question, I still can't figure it out. I've tried several combinations for my Grunt file.
Nonetheless, each time I'm getting the Server should provide a function called "listen" that acts as http.Server.listen error
Here is my code :
Gruntfile.js
module.exports = function (grunt) {
'use strict';
var path = require('path');
grunt.initConfig({
pkg: grunt.file.readJSON('package.json'),
app: {
basePath: 'public',
[...],
serverPath: 'backend'
},
express: {
all: {
options: {
port: 3000,
hostname: '*', //()=>Localhost
bases: '<%= app.basePath %>',
server: '<%= app.serverPath %>/Server.js',
livereload: true
}
}
}
});
grunt.registerTask('server', [
'express',
'open',
'watch'
]);
};
Project structure :
App_Root/
-Backend/
--Server.js
--BackofficeRouter.js
-Public/
--index.html
Server.js :
var express = require('express');
var backofficeRouter = require('./backofficeRouter.js');
var constants = require('./../public/constants/ConstantsModule.js');
var app = express();
var appRoot = require('app-root-path');
app.use('/backoffice', backofficeRouter);
app.use(express.static(appRoot + '/public'));
app.listen(process.env.PORT || 3000);
BackofficeRouter.js :
backofficeRouter.get('/', function(req, res) {
res.sendFile(path.join(appRoot + '/public/index.html'));
});
I don't really get how grunt-express interacts with my Server.js file.
I've tried to set the default grunt-express config in my Gruntfile, thinking it would take into account what I have in my Server.js. But it seems like grunt-express' config overrides everything.
Any hints on where I could have made a mistake ?
Thank you for your help.

The Server should provide a function called "listen" that acts as http.Server.listen error suggests that the grunt task is expecting a server instance that contains the method 'listen', so you should remove this line from your server
app.listen(process.env.PORT || 3000);
and replace it for
module.exports = app;
that way, the grunt task will receive a configured instance of express that contains the listen method it is looking for.

Module 'httpMock' is not available

This is error message on a protractor test use protractor http mock:
JavascriptError: javascript error: [$injector:nomod] Module 'httpMock'
is not available! You eit her misspelled the module name or forgot to
load it. If registering a module ensure that you specify the
dependencies as the second argument.
conf.js:
// An example configuration file.
exports.config = {
directConnect: true,
// Selenium server
SeleniumAddress: 'http://localhost:4444/wd/hub',
// Capabilities to be passed to the webdriver instance.
capabilities: {
'browserName': 'chrome'
},
//baseUrl: 'http://develop.garbo.livebranches.com/sv-SE/',
//Framework to use. Jasmine 2 is recommended.
framework: 'jasmine2',
//frameworks: ['mocha', 'jasmine'],
// Spec patterns are relative to the current working directly when
// protractor is called.
//specs: ['testmain.js','testlogin.js'],
//specs: ['testmain.js','testteaPartyList.js','testpositionSearchIndex.js','testpositionList.js'],
specs: ['testlogin.js'],
//Options to be passed to Jasmine.
jasmineNodeOpts: {
defaultTimeoutInterval: 250000
},
mocks: {
dir: '../node_modules/protractor-http-mock',
//dir: 'mocks',
default: []
},
//=====login begin =====
onPrepare: function() {
require("protractor-http-mock").config = {
rootDirectory: '../node_modules/protractor-http-mock/lib',
//rootDirectory: __dirname,
protractorConfig: "conf.js", // name of the config here
};
}
//=====login end========
};
testlogin.js
describe('angularjs homepage', function() {
//browser.ignoreSynchronization = true;
it('should login', function() {
var mock = require('protractor-http-mock');
var todoList;
beforeEach(function() {
var url ='http://dev.etest.com:285/Actor/tbUsers/LoginAndGet';
var req = {Mobile:'14500000006',Password:'123456'};
var rep = {UserId:164,AccountId:328,Token:'328:dc91d536ab424aa0b8d7f1ecaf64c55b',Id:328};
mock([{
request: {
path: url,
method: 'POST',
data:req,
},
response: {
data: rep,
}
}]);
});
afterEach(function() {
mock.teardown();
});
browser.get('http://localhost:2024/daNiuJob/www/ionicWeb/index.html#/login');
console.log('mock='+mock);
element(by.model('data.userName')).sendKeys('14500000006');
element(by.model('data.password')).sendKeys('123456');
var btnlogin = element(by.id('Regist')).element(by.tagName('a'));
expect(browser.getTitle()).toEqual('userlogin');
browser.getTitle().then(function(text){
console.log('title='+text);
});
//cause mock error
expect(mock.requestsMade()).toEqual([
{ url : 'http://dev.etest.com:285/Actor/tbUsers/LoginAndGet', method : 'GET' },
]);
btnlogin.click();
browser.sleep(8000);
});
});
Why can't find httpMock, thank!
note:
C:\Users\HQ-XXX\AppData\Roaming\npm\node_modules\protractor\node_modules\protractor-http-mock
This is path of 'protractor-http-mock'

You should be giving the path of the http-mock module folder and not lib folder inside it. Change your rootDirectory path of protractor-http-mock inside onPrepare() function to -
rootDirectory: 'C:\Users\HQ-XXX\AppData\Roaming\npm\node_modules\protractor\node_modules\protrac‌​tor-http-mock ',
If at all you need to provide a relative path then change it as below -
rootDirectory: '..\node_modules\protrac‌​tor-http-mock ',
Hope this helps.

We had the same issue and it was related to the page reloading at the beginning of every spec.
This was caused by a faulty config of html5mode and the browser.get so it did a redirect at the beginning from foo.bar/ to foo.bar/#/ which unloads all injected protractor code.

Grunt Environment Variables Don't Get Set Until All Tasks Loaded

I am using the npm modules grunt env and load-grunt-config in my project. grunt env handles environment variables for you, while load-grunt-config handles, well, loads the grunt configuration for you. You can put your tasks into other files, then load-grunt-config will bundle them up and have grunt load & consume them for you. You can also make an aliases.js file, with tasks you want to combine together into one task, running one after another. It's similar to the grunt.registerTask task in the original Gruntfile.js. I put all my grunt tasks inside a separate grunt/ folder under the root folder with the main Gruntfile, with no extra subfolders, as suggested by the load-grunt-config README.md on Github. Here is my slimmed-down Gruntfile:
module.exports = function(grunt) {
'use strict';
require('time-grunt')(grunt);
// function & property declarations
grunt.initConfig({
pkg: grunt.file.readJSON('package.json')
});
require('load-grunt-config')(grunt, {
init: true,
loadGruntConfig: {
scope: 'devDependencies',
pattern: ['grunt-*', 'time-grunt']
}
});
};
In theory, setting all these files up the correct way for load-grunt-config to load should be exactly the same as just having a Gruntfile.js. However, I seem to have run into a little snag. It seems the environment variables set under the env task do not get set for the subsequent grunt tasks, but are set by the time node processes its tasks, in this case an express server.
grunt env task:
module.exports = {
// environment variable values for developers
// creating/maintaining site
dev: {
options: {
add: {
NODE_ENV: 'dev',
MONGO_PORT: 27017,
SERVER_PORT: 3000
}
}
}
};
grunt-shell-spawn task:
// shell command tasks
module.exports = {
// starts up MongoDB server/daemon
mongod: {
command: 'mongod --bind_ip konneka.org --port ' + (process.env.MONGO_PORT || 27017) + ' --dbpath C:/MongoDB/data/db --ipv6',
options: {
async: true, // makes this command asynchronous
stdout: false, // does not print to the console
stderr: true, // prints errors to the console
failOnError: true, // fails this task when it encounters errors
execOptions: {
cwd: '.'
}
}
}
};
grunt express task:
module.exports = {
// default options
options: {
hostname: '127.0.0.1', // allow connections from localhost
port: (process.env.SERVER_PORT || 3000), // default port
},
prod: {
options: {
livereload: true, // automatically reload server when express pages change
// serverreload: true, // run forever-running server (do not close when finished)
server: path.resolve(__dirname, '../backend/page.js'), // express server file
bases: 'dist/' // watch files in app folder for changes
}
}
};
aliases.js file (grunt-load-config's way of combining tasks so they run one after the other):
module.exports = {
// starts forever-running server with "production" environment
server: ['env:prod', 'shell:mongod', 'express:prod', 'express-keepalive']
};
part of backend/env/prod.js (environment-specific Express configuration, loaded if NODE_ENV is set to "prod", modeled after MEAN.JS):
'use strict';
module.exports = {
port: process.env.SERVER_PORT || 3001,
dbUrl: process.env.MONGOHQ_URL || process.env.MONGOLAB_URI || 'mongodb://konneka.org:' + (process.env.MONGO_PORT || 27018) + '/mean'
};
part of backend/env/dev.js (environment-specific Express configuration for dev environment, loaded if the `NODE_ENV variable is not set or is set to "dev"):
module.exports = {
port: process.env.SERVER_PORT || 3000,
dbUrl: 'mongodb://konneka.org:' + (process.env.MONGO_PORT || 27017) + '/mean-dev'
};
part of backend/page.js (my Express configuration page, also modeled after MEAN.JS):
'use strict';
var session = require('express-session');
var mongoStore = require('connect-mongo')(session);
var express = require('express');
var server = express();
...
// create the database object
var monServer = mongoose.connect(environ.dbUrl);
// create a client-server session, using a MongoDB collection/table to store its info
server.use(session({
resave: true,
saveUninitialized: true,
secret: environ.sessionSecret,
store: new mongoStore({
db: monServer.connections[0].db, // specify the database these sessions will be saved into
auto_reconnect: true
})
}));
...
// listen on port related to environment variable
server.listen(process.env.SERVER_PORT || 3000);
module.exports = server;
When I run grunt server, I get:
$ cd /c/repos/konneka/ && grunt server
Running "env:prod" (env) task
Running "shell:mongod" (shell) task
Running "express:prod" (express) task
Running "express-server:prod" (express-server) task
Web server started on port:3000, hostname: 127.0.0.1 [pid: 3996]
Running "express-keepalive" task
Fatal error: failed to connect to [konneka.org:27018]
Execution Time (2014-08-15 18:05:31 UTC)
loading tasks 38.3s █████████████████████████████████ 79%
express-server:prod 8.7s ████████ 18%
express-keepalive 1.2s ██ 2%
Total 48.3s
Now, I can't seem to get the database connected in the first place, but ignore that for now. Notice that the server is started on port 3000, meaning that during execution of the grunt express:prod task, SERVER_PORT is not set so the port gets set to 3000. There are numerous other examples like this, where an environment variable is not set so my app uses the default. However, notice that session tries to connect to the database on port 27018 (and fails), so MONGO_PORT does get set eventually.
If I had just tried the grunt server task, I could chalk it up to load-grunt-config running the tasks in parallel instead of one after the other or some other error, but even when I try the tasks one-by-one, such as running grunt env:prod shell:mongod express-server:prod express-keepalive, I get similar (incorrect) results, so either grunt or grunt env run the tasks in parallel, as well, or something else is going on.
What's going on here? Why are the environment variables not set correctly for later grunt tasks? When are they eventually set, and why then rather than some other time? How can I make them get set for grunt tasks themselves rather than after, assuming there even is a way?

The solution is rather obvious once you figure it out, so let's start at the beginning:
The problem
You're using load-grunt-config to load a set of modules (objects that define tasks) and combine them into one module (object) and pass it along to Grunt. To better understand what load-grunt-config is doing, take a moment to read through the source (it's just three files). So, instead of writing:
// filename: Gruntfile.js
grunt.initConfig({
foo: {
a: {
options: {},
}
},
bar: {
b: {
options: {},
}
}
});
You can write this:
// filename: grunt/foo.js
module.exports = {
a: {
options: {},
}
}
// filename: grunt/bar.js
module.exports = {
b: {
options: {},
}
}
// filename: Gruntfile.js
require('load-grunt-config')(grunt);
Basically, this way you can split up a Grunt configuration into multiple files and have it be more "maintainable". But what you'll need to realize is that these two approaches are semantically equivalent. That is, you can expect them to behave the same way.
Thus, when you write the following*:
(* I've reduced the problem in an attempt to make this answer a bit more general and to reduce noise. I've excluded things like loading the tasks and extraneous option passing, but the error should still be the same. Also note that I've changed the values of the environment variables because the default was the same as what was being set.)
// filename: grunt/env.js
module.exports = {
dev: {
options: {
add: {
// These values are different for demo purposes
NODE_ENV: 'dev',
MONGO_PORT: 'dev_mongo_port',
SERVER_PORT: 'dev_server_port'
}
}
}
};
// filename: grunt/shell.js
module.exports = {
mongod: {
command: 'mongod --port ' + (process.env.MONGO_PORT || 27017)
}
};
// filename: grunt/aliases.js
module.exports = {
server: ['env:prod', 'shell:mongod']
};
// filename: Gruntfile.js
module.exports = function (grunt) {
require('load-grunt-config')(grunt);
};
You can consider the above the same as below:
module.exports = function (grunt) {
grunt.initConfig({
env: {
dev: {
options: {
add: {
NODE_ENV: 'dev',
MONGO_PORT: 'dev_mongo_port',
SERVER_PORT: 'dev_server_port'
}
}
}
},
shell: {
mongod: {
command: 'mongod --port ' + (process.env.MONGO_PORT || 27017)
}
}
});
grunt.registerTask('server', ['env:dev', 'shell:mongod']);
};
Now do you see the problem? What command do you expect shell:mongod to run? The correct answer is:
mongod --port 27017
Where what you want to be executed is:
mongo --port dev_mongo_port
The problem is that when (process.env.MONGO_PORT || 27017) is evaluated the environment variables have not yet been set (i.e. before the env:dev task has been run).
A solution
Well let's look at a working Grunt configuration before splitting it across multiple files:
module.exports = function (grunt) {
grunt.initConfig({
env: {
dev: {
options: {
add: {
NODE_ENV: 'dev',
MONGO_PORT: 'dev_mongo_port',
SERVER_PORT: 'dev_server_port'
}
}
}
},
shell: {
mongod: {
command: 'mongod --port ${MONGO_PORT:-27017}'
}
}
});
grunt.registerTask('server', ['env:dev', 'shell:mongod']);
};
Now when you run shell:mongod, the command will contain ${MONGO_PORT:-27017} and Bash (or just sh) will look for the environment variable you would have set in the task before it (i.e. env:dev).
Okay, that's all well and good for the shell:mongod task, but what about the other tasks, Express for example?
You'll need to move away from environment variables (unless you want to set them up before invoking Grunt. Why? Take this Grunt configuration for example:
module.exports = function (grunt) {
grunt.initConfig({
env: {
dev: {
options: {
add: {
NODE_ENV: 'dev',
MONGO_PORT: 'dev_mongo_port',
SERVER_PORT: 'dev_server_port'
}
}
}
},
express: {
options: {
hostname: '127.0.0.1'
port: (process.env.SERVER_PORT || 3000)
},
prod: {
options: {
livereload: true
server: path.resolve(__dirname, '../backend/page.js'),
bases: 'dist/'
}
}
}
});
grunt.registerTask('server', ['env:dev', 'express:prod']);
};
What port will the express:prod task configuration contain? 3000. What you need is for it to reference the value you've defined in the above task. How you do this is up to you. You could:
Separate the env configuration and reference its values
module.exports = function (grunt) {
grunt.config('env', {
dev: {
options: {
add: {
NODE_ENV: 'dev',
MONGO_PORT: 'dev_mongo_port',
SERVER_PORT: 'dev_server_port'
}
}
}
});
grunt.config('express', {
options: {
hostname: '127.0.0.1'
port: '<%= env.dev.options.add.SERVER_PORT %>'
}
});
grunt.registerTask('server', ['env:dev', 'express:prod']);
};
But you'll notice that the semantics of the env task don't hold up here due to it no longer representing a task's configuration. You could use an object of your own design:
module.exports = function (grunt) {
grunt.config('env', {
dev: {
NODE_ENV: 'dev',
MONGO_PORT: 'dev_mongo_port',
SERVER_PORT: 'dev_server_port'
}
});
grunt.config('express', {
options: {
hostname: '127.0.0.1'
port: '<%= env.dev.SERVER_PORT %>'
}
});
grunt.registerTask('server', ['env:dev', 'express:prod']);
};
Pass grunt an argument to specify what config it should use
Have multiple configuration files (e.g. Gruntfile.js.dev and Gruntfile.js.prod) and rename them as needed
Read a development configuration file (e.g. grunt.file.readJSON('config.development.json')) if it exists and fall back to a production configuration file if it doesn't exist
Some better way not listed here
But all of the above should achieve the same end result.

This seems to be the essence of what you are trying to do, and it works for me. The important part was what I mentioned in my comment -- chaining the environment task before running the other tasks.
Gruntfile.js
module.exports = function(grunt) {
// Do grunt-related things in here
grunt.loadNpmTasks('grunt-env');
grunt.initConfig({
env: {
dev: {
PROD : 'http://production.server'
}
}
});
grunt.registerTask('printEnv', 'prints a message with an env var', function() { console.log('Env var in subsequent grunt task: ' + process.env.PROD) } );
grunt.registerTask('prod', ['env:dev', 'printEnv']);
};
Output of grunt prod
Running "env:dev" (env) task
Running "printEnv" task
Env var in subsequent grunt task: http://production.server
Done, without errors.

Where can I save log files to with Node.JS in Express framework?

I have a node.js application where I am using Winston to track some logging. I want to have a server side log in addition to the client side. In the express framework is it just the files in the public folder that are accessible to the client and if I want to do a server side log where the client users can not access it can I create a folder outside of public?

Exactly, just point the log file to be outside the /public folder and whatever other folders you have configured in app.use(express.static(path.join(__dirname, 'public_folder')));
You can configure were the winston tranports will log the files by setting its proper attributes, for instance, in the file transport you have to set the filename option.
The following is a config.js file from I load all configuration for my logging activities (server side and client requests):
config.json
{
"transports": {
"console": {
"enabled": true,
"colorize" : true,
"timestamp": false
},
"file" : {
"enabled": true,
"colorize" : true,
"filename" : "/var/log/vifros.log",
"timestamp": true
},
"mongodb": {
"enabled": true,
"db": "vifros"
}
}
}
EDIT LOGGING- logger.js -- (see comment):
var winston = require('winston');
require('winston-mongodb').MongoDB; // Monkeypatch Winston for MongoDb transport.
var config = require('../config');
/*
* Enable initially all transports for initial app startup and then disable them according to settings.
*
* Add console transport.
*/
winston.remove(winston.transports.Console); // To being able to reconfigure it.
if (config.logging.transports.console.enabled) {
winston.add(winston.transports.Console, config.logging.transports.console);
}
/*
* File transport.
*/
if (config.logging.transports.file.enabled) {
winston.add(winston.transports.File, config.logging.transports.file);
}
/*
* Enable saving logs to mongodb.
*/
if (config.logging.transports.mongodb.enabled) {
winston.add(winston.transports.MongoDB, config.logging.transports.mongodb);
}
exports.logger = winston;
exports.tags = {
init : 'init',
api_request: 'api_request',
db : 'db',
validation : 'validation',
cross_rel : 'cross_relationship',
os : 'os'
};