We Keep Coding

TestCafe: awaiting ClientFunction to resolve in expect method leads to unexpected behavior

I'm currently facing an issue where awaiting a TestCafe ClientFunction within a assertion query chain leads to the behavior demonstrated by the code below:
const getPageUrl = ClientFunction(() => window.location.href.toString());
// This test works fine
test("Test that works as expected", async t => {
await t
.click(pageElements.someNaviButton) // navigates me to the site with url extension /some/url
.expect(getPageUrl()).contains("/some/url")
});
// This test works fine as well
test("Another test that works as expected", async t => {
await t.click(pageElements.someNaviButton) // navigates me to the site with url extension /some/url
const myPageUrl = await getWindowLocation();
await t.expect(myPageUrl).contains("/some/url")
});
// This test fails
test("Test that does not work as expected", async t => {
await t
.click(pageElements.someNaviButton)
.expect(await getPageUrl()).contains("/some/url")
});
According to TestCafe's documentation, one has to await asynchronous ClientFunctions. This leads me to what irritates me: I expect the second test to pass, which it does. Awaiting the method that encapsulates the ClientFunction within the expect method of the third test seems to be problematic, but why is that?
Additional info that could be of interest: I'm using TestCafe version 1.14.0.

Please consider that a test actions chain is a single expression. So, in the third test, getPageUrl is calculated before the chain execution. At that moment, the test is still on the start page.
The first test passes because getPageUrl is calculated lazily via TestCafe Smart Assertion Query Mechanism. This is the most proper way to write this kind of tests.
The second test passes because you broke the test actions chain.

How to trace possible async race conditions in my test code? [duplicate]

This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Closed 3 years ago.
I am learning modern JavaScript, and am writing a little API. I plan to host it in MongoDB Stitch, which is a serverless lambda-like environment. I am writing functions in the way that this system requires, and then adding Jest functions to be run locally and in continuous integration.
I am learning Jest as I go, and for the most part, I like it, and my prior experience with Mockery in PHP is making it a fairly painless experience. However, I have an odd situation where my lack of knowledge of JavaScript is stopping my progress. I have a failing test, but it is intermittent, and if I run all of the tests, sometimes it all passes, and sometimes the test that fails changes from one to another. This behaviour, coupled with my using async-await makes me think I am experiencing a race condition.
Here is my SUT:
exports = async function(delay) {
/**
* #todo The lambda only has 60 seconds to run, so it should test how
* long it has been running in the loop, and exit before it gets to,
* say, 50 seconds.
*/
let query;
let ok;
let count = 0;
for (let i = 0; i < 5; i++) {
query = await context.functions.execute('getNextQuery');
if (query) {
ok = context.functions.execute('runQuery', query.phrase, query.user_id);
if (ok) {
await context.functions.execute('markQueryAsRun', query._id);
count++;
}
} else {
break;
}
// Be nice to the API
await sleep(delay);
}
return count;
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
};
The content.functions object is a global in Stitch, but as I show below, it is mocked inside Jest. Stitch is not involved in these tests at all.
As you can see, getNextQuery and markQueryAsRun are awaited, as they are defined as async.
Here is my main test:
// Main SUT
const findAndRunQueries = require('./_source');
// Utility test classes
const MongoTester = require('../../test/mongo-tester');
const StitchFuncMocking = require('../../test/stitch-func-mocking');
// Other functions for the integration test
const getNextQuery = require('../getNextQuery/_source');
const markQueryAsRun = require('../markQueryAsRun/_source');
describe('Some integration tests for findAndRunQueries', () => {
const mongoTester = new MongoTester('findAndRunQueries-integration');
const stitchFuncMocking = new StitchFuncMocking();
beforeAll(async () => {
await mongoTester.connect();
console.log('Connect');
});
afterAll(async () => {
await mongoTester.disconnect();
console.log('Disconnect');
});
beforeEach(async () => {
// Set up global values
global.context = {};
global.context.services = mongoTester.getStitchContext();
global.context.functions = stitchFuncMocking.getFunctionsObject(jest);
// Delete existing mocks
jest.clearAllMocks();
stitchFuncMocking.clearMocks();
// Connect some real implementations
stitchFuncMocking.setGlobalMock('getNextQuery', getNextQuery);
stitchFuncMocking.setGlobalMock('markQueryAsRun', markQueryAsRun);
// Truncate all collections in use
await mongoTester.emptyCollections(['queries']);
console.log('Init mocks and clear collections');
});
test('end-to-end test with no queries', async () => {
expect(await findAndRunQueries(0)).toBe(0);
});
test('end-to-end test with one successful query', async () => {
// Here is a query entry
await mongoTester.getDatabase().collection('queries').insertOne({
"user_id": 1,
"phrase": 'hello',
"last_run_at": null,
"enabled": true
});
var d = await mongoTester.getDatabase().collection('queries').findOne({});
console.log(d);
// We need to mock runQuery, as that calls an external API
stitchFuncMocking.setGlobalMock('runQuery', () => 123);
// Let's see if we can run a call sucessfully
expect(await findAndRunQueries(0)).toBe(1);
// #todo Check that a log entry has been made
});
});
From this code you can see that getNextQuery and markQueryAsRun are wired to their real implementations (since this is an integration test) but runQuery is a mock, because I don't want this test to make HTTP calls.
For brevity, I am not showing the above code, as I don't think it is needed to answer the question. I am also not showing all of MongoTester or any of StitchFuncMocking (these connect to an in-memory MongoDB instance and simplify Jest mocking respectively).
For database-level tests, I run this MongoTester utility function to clear down collections:
this.emptyCollections = async function(collections) {
// Interesting note - how can I do deleteMany without async, but
// wait for all promises to finish before the end of emptyCollections?
collections.forEach(async (collectionName) => {
let collection = this.getDatabase().collection(collectionName);
await collection.deleteMany({});
});
};
This is how I am running the test:
sh bin/test-compile.sh && node node_modules/jest/bin/jest.js -w 1 functions/findAndRunQueries/
The compilation step can be ignored (it just converts the exports to module.exports, see more here). I then run this test plus a unit test inside the functions/findAndRunQueries/ folder. The -w 1 is to run a single thread, in case Jest does some weird parallelisation.
Here is a good run (containing some noisy console logging):
root#074f74105081:~# sh bin/test-compile.sh && node node_modules/jest/bin/jest.js -w 1 functions/findAndRunQueries/
PASS functions/findAndRunQueries/findAndRunQueries.integration.test.js
● Console
console.log functions/findAndRunQueries/findAndRunQueries.integration.test.js:18
Connect
console.log functions/findAndRunQueries/findAndRunQueries.integration.test.js:42
Init mocks and clear collections
console.log functions/findAndRunQueries/findAndRunQueries.integration.test.js:42
Init mocks and clear collections
console.log functions/findAndRunQueries/findAndRunQueries.integration.test.js:60
{ _id: 5e232c13dd95330804a07355,
user_id: 1,
phrase: 'hello',
last_run_at: null,
enabled: true }
PASS functions/findAndRunQueries/findAndRunQueries.test.js
Test Suites: 2 passed, 2 total
Tests: 6 passed, 6 total
Snapshots: 0 total
Time: 0.783s, estimated 1s
Ran all test suites matching /functions\/findAndRunQueries\//i.
In the "end-to-end test with one successful query" test, it inserts a document and then passes some assertions. However, here is another run:
root#074f74105081:~# sh bin/test-compile.sh && node node_modules/jest/bin/jest.js -w 1 functions/findAndRunQueries/
FAIL functions/findAndRunQueries/findAndRunQueries.integration.test.js
● Console
console.log functions/findAndRunQueries/findAndRunQueries.integration.test.js:18
Connect
console.log functions/findAndRunQueries/findAndRunQueries.integration.test.js:42
Init mocks and clear collections
console.log functions/findAndRunQueries/findAndRunQueries.integration.test.js:42
Init mocks and clear collections
console.log functions/findAndRunQueries/findAndRunQueries.integration.test.js:60
null
● Some integration tests for findAndRunQueries › end-to-end test with one successful query
expect(received).toBe(expected) // Object.is equality
Expected: 1
Received: 0
64 |
65 | // Let's see if we can run a call sucessfully
> 66 | expect(await findAndRunQueries(0)).toBe(1);
| ^
67 |
68 | // #todo Check that a log entry has been made
69 | });
at Object.test (functions/findAndRunQueries/findAndRunQueries.integration.test.js:66:44)
PASS functions/findAndRunQueries/findAndRunQueries.test.js
Test Suites: 1 failed, 1 passed, 2 total
Tests: 1 failed, 5 passed, 6 total
Snapshots: 0 total
Time: 0.918s, estimated 1s
Ran all test suites matching /functions\/findAndRunQueries\//i.
The null in the log output indicates that the insert failed, but I do not see how that is possible. Here is the relevant code (reproduced from above):
await mongoTester.getDatabase().collection('queries').insertOne({
"user_id": 1,
"phrase": 'hello',
"last_run_at": null,
"enabled": true
});
var d = await mongoTester.getDatabase().collection('queries').findOne({});
console.log(d);
I assume the findOne() returns a Promise, so I have awaited it, and it is still null. I also awaited the insertOne() as I reckon that probably returns a Promise too.
I wonder if my in-RAM MongoDB database might not be performing like a real Mongo instance, and I wonder if I should spin up a Docker MongoDB instance for testing.
However, perhaps there is just an async thing I have not understood? What can I dig into next? Is it possible that the MongoDB write concerns are set to the "don't confirm write before returning control"?
Also perhaps worth noting is that the "Disconnect" message does not seem to pop up. Am I not disconnecting from my test MongoDB instance, and could that cause a problem by leaving an in-memory server in a broken state?

The problem in my project was not initially shown in the question - I had omitted to show the emptyCollections method, believing the implementation to be trivial and not worth showing. I have now updated that in the question.
The purpose of that method is to clear down collections between tests, so that I do not accidentally rely on serial effects based on test run order. The new version looks like this:
this.emptyCollections = async function(collections) {
const promises = collections.map(async (collectionName, idx) => {
await this.getDatabase().collection(collectionName).deleteMany({})
});
await Promise.all(promises);
};
So, what was going wrong with the old method? It was just a forEach wrapping awaited database operation Promises - not much to go wrong, right?
Well, it turns out that plenty can go wrong. I have learned that async functions can be run in a for, and they can be run in a for in, but the Array implementation of forEach does not do any internal awaiting, and so it fails. This article explains some of the differences. It sounds like if there is one takeaway, it is "don't try to run async code in a forEach loop".
The effect of my old function was that the collection emptying was not finished even after that method had run, and so when I did other async operations - like inserting a document! - there would be a race that determined which one executed first. This partially explains the intermittent test failure.
As it turns out, I had made some provision while debugging to use a different Mongo database per test file, in case Jest was doing some parallelisation. In fact it was, and when I removed the feature to use different databases, it failed again - this time because there was a race condition between tests (and their respective beforeEach functions) to set up the state of a single database.

Why do I keep facing a Mocha "timeout error"; Also node keeps telling me to resolve my promise?

I keep getting a timeout error, it keeps telling me to enusre I have called done(), even though I have.
const mocha = require('mocha');
const assert = require('assert');
const Student = require('../models/student.js');
describe('CRUD Tests',function(){
it('Create Record',function(done){
var s = new Student({
name: "Yash"
});
s.save().then(function(){
assert(s.isNew === false);
done();
});
});
});
Result is -
CRUD Tests
1) Create Record
0 passing (2s) 1 failing
1) CRUD Tests
Create Record:
Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it
resolves.
(/home/yash/Documents/Development/Node/MongoCRUD/test/CRUD_test.js)

Note that as written, your unit test ignores the fact that save() might reject instead of resolving. Whenever you use this done construct, make sure your unit test handles an error scenario, like this:
s.save().then(function() {
assert(s.isNew === false);
done();
}).catch(error => {
done(error);
});
Alternatively, since Mocha has built-in support for promises, you can remove the done parameter and return the promise directly, like this:
it('Create Record', function() {
// ...
return s.save().then(function() {
assert(s.isNew === false);
});
});
The advantage with this approach is a reject promise will automatically fail the test, and you don't need any done() calls.

I guess that your mocha runs without being connected to your database. So .save() is waiting for a connection which it never get and your mocha timeout.
You can initialize your software system before to run any Mocha test.
For example, connect a database.
// ROOT HOOK Executed before the test run
before(async () => {
// connect to the database here
});
// ROOT HOOK Excuted after every tests finished
after(async () => {
// Disconnect from the database here
});

Jest unit testing - Async tests failing Timeout

I got the following error message "Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL." On top of that, I also received an error message stating that 0 assertions were executed while expecting 2 assertions.
I've tried extending the timeout to 10 seconds using jest.setTimeout(10000), which should be more than sufficient time to execute that code, but the problem persisted.
I know m.employeeGetAll() works because when I test my web app using the browser, I can see the list of employees in the view.
Here's how my test looks like
it('Lists all employees successfully', () => {
expect.assertions(2);
return m.employeeGetAll().then(result => { //m.employeeGetAll() returns a promise
expect(result).toBeDefined();
expect(result.length).toBe(3);
});
});

The problem I've discovered, was the way async code works.
What cannot be seen in the code snippet was the call to mongoose.connection.close(); at the very end of my test file.
This call must be done inside afterEach() or afterAll() function for Jest unit testing framework. Otherwise, the connection to the database will be closed before the tests could complete since all of the calls in my controller methods are asynchronous; This leads to no promise ever being returned and the code goes into timeout.
Since I'm using beforeAll() and afterAll(), to load data from database once before the start of all tests and to clear the database at the end of all tests, I've included the call to connect to DB using mongoose inside beforeAll() as well.
Hope this helps someone who's also stuck in my situation.

with async you have to call done
it('Lists all employees successfully', (done) => {
expect.assertions(2);
return m.employeeGetAll().then(result => { //m.employeeGetAll() returns a promise
expect(result).toBeDefined();
expect(result.length).toBe(3);
done();
});
});

Async Mocha: `done` called, but next test never runs

I have an unconventional setup I can't change. It looks something like this:
test POSTs to server
server POSTs to blockchain
blockchain updates
syncing script updates the database
It is absolutely imperative that I do not run the next test until the test before it has completed that entire workflow, which typically takes 2-3 seconds. Here is an example of a test written for that flow with supertest and chai:
it('should create a user', done => {
request(server)
.post(`${API}/signup`)
.set('Content-Type', 'application/json')
.send(`{
"email":"${USER_EMAIL}",
"password":"${USER_PASSWORD}"
}`)
.expect(200)
.expect(res => {
expect(res.body.role).to.equal('user');
expect(res.body.id).to.match(ID_PATTERN);
})
.end(_wait(done));
});
That _wait function is the key issue here. If I write it very naively with a setTimeout, it will work:
const _wait = cb => () => setTimeout(cb, 5000);
However, this isn't a great solution since the blockchain is very unpredictable, and can sometimes take much more than 2-3 seconds. What would be much better is to watch the database for changes. Thankfully the database is written in Rethink, which provides cursor objects that update on a change. So that should be easy, and look something like this:
var _wait = cb => () => {
connector.exec(db => db.table('chain_info').changes())
.then(cursor => {
cursor.each((err, change) => {
cb(err);
return false;
});
});
};
This setup breaks the tests. As near as I can tell done does get called. Any console logs in and around it fire, and the test itself is logged as completed, but the next test never starts, and eventually everything times out:
Manager API Workflow:
Account Creation:
✓ should create a user (6335ms)
1) should login an administrator
1 passing (1m)
1 failing
1) Manager API Workflow: Account Creation: should login an administrator:
Error: timeout of 60000ms exceeded. Ensure the done() callback is being called in this test.
Any assistance would be greatly appreciated. I am using Mocha 3.1.2, Chai 3.5.0, Supertest 2.0.1, and Node 6.9.1.