We Keep Coding

Why does jest time out when running a test on mock AWS file read?

I am trying to write a unit test (jest) to test the execution of a service. Currently, the test times out and I am positive it has to do with the point at with I create the read stream. Am I mocking s3.getObject.createReadStream() correctly?
handler.ts
const XLSX = require("xlsx");
const AWS = require("aws-sdk");
import { Pool } from "pg";
class Service {
client: any;
workbook: any = "";
constructor(client) {
this.workbook = "";
this.client = client;
}
async createWorkbookFromS3(filePath: string, id: string) {
const fileLocation = id + filePath;
const params = {
Bucket: "bucket",
Key: fileLocation,
};
function getBufferFromS3(callback) {
const buffers = [];
const s3 = new AWS.S3();
const stream = s3.getObject(params).createReadStream();
stream.on("data", (data) => buffers.push(data));
stream.on("end", () => callback(null, Buffer.concat(buffers)));
stream.on("error", (error) => callback(error));
}
function getBufferFromS3Promise() {
return new Promise((resolve, reject) => {
getBufferFromS3((error, s3buffer) => {
if (error) {
return reject(error)
};
return resolve(s3buffer);
});
});
}
try {
const buffer = await getBufferFromS3Promise();
this.workbook = XLSX.read(buffer);
} catch (e) {
if (e) {
this.client.release()
console.error(e);
throw new Error("Unable to read the xlsx from S3");
}
}
}
}
const decoder = async (event: any) => {
const id: string = event.body["id"];
const filePath = `${id}.xlsx`;
try {
switch (event.path) {
case "/vsiDecoder/draft":
try {
const filePath: string = event.body["filePath"];
const dbCredentials = {"user":"postgres","password":"password","host":"awsconnectihoststring","port": "0000" ,"database":"db"}
const pool = new Pool(JSON.parse(dbCredentials['SecretString']));
const client = await pool.connect();
const vsiObj = new Service(client);
await vsiObj.createWorkbookFromS3(filePath, id);
vsiObj.client.release()
pool.end();
} catch (e) {
throw new Error(e)
}
return {
statusCode: 200,
message: "The document has been successfully drafted.",
};
default:
return {
message: "Bad request",
statusCode: 400,
};
}
} catch (e) {
console.log(e)
return{
statusCode: 400,
message: e.message,
}
}
};
module.exports = {Service, decoder }
handler.test.ts
const decoder = require('./handler.ts')
import { Pool } from 'pg';
const mockgetObject = {
createReadStream: jest.fn(() => {
return {
on: jest.fn()
}
})
}
const mockS3Instance = {
putObject: jest.fn().mockReturnThis(),
promise: jest.fn().mockReturnThis(),
catch: jest.fn(),
getObject: jest.fn(() => mockgetObject)
}
jest.mock('aws-sdk', () => {
return {
S3: jest.fn(() => mockS3Instance),
}
});
jest.mock('pg', () => {
const mClient = {
connect: jest.fn().mockReturnThis(),
query: jest.fn().mockReturnThis(),
end: jest.fn().mockReturnThis(),
release: jest.fn().mockReturnThis(),
};
return { Pool: jest.fn(() => mClient) };
});
describe('Service Test', () => {
let client;
beforeEach(() => {
client = new Pool();
jest.setTimeout(30000);
});
afterEach(() => {
jest.clearAllMocks();
});
it('should executre decoder', async () => {
const id = 'id';
const filePath = 'filePath'
const service = new decoder.Service(client);
await service.createWorkbookFromS3(id, filePath)
const events = { 'body': { 'id': '1234', "path": "/decoder/draft", "vsiFilePath": "path" } };
const response = '{"statusCode":200,"message":"The document has been succesfully drafted."}';
expect((await decoder.decoder(events)).body).toEqual(response);
})
})
error:
Service Test › should executre decoder
: Timeout - Async callback was not invoked within the 30000 ms timeout specified by jest.setTimeout.Timeout - Async callback was not invoked within the 30000 ms timeout specified by jest.setTimeout.Error:
Thanks to any help! I do believe that I am successfully mocking createReadStream but if there is a way to mock getBufferFromS3Promise that would be acceptable instead.

In your createReadStream mock, the on function has no implementation. However, in the code for getBufferFromS3(callback), the on function is responsible for setting up the event handlers that execute the callback function, which is where the Promise from getBufferFromS3Promise will be resolved or rejected. Because that promise is never resolved or rejected, your asynchronous test code never finishes.
You need to implement a mock of the on function that will execute the event handler callbacks somehow. If you are only testing the success case you could just execute the handler synchronously within on since the 'end' event happens to be hooked up before the 'error' event. A better approach might be to mock the event emitter by having on save the handlers, and adding a triggerEvent function to the stream mock that allows you to imperatively trigger the different stream events within your test.

How to make this nodejs lambda wait until the job is done?

I'm a newbie in nodejs.
I have an AWS lambda that reads a queue and delete messages after processing them:
const AWS = require('aws-sdk');
AWS.config.logger = console;
const sqs = new AWS.SQS({apiVersion: '2012-11-05'});
const redriveQueueUrl = 'https://myqueue";
async function deleteMessage (messageToDelete) {
let params = {
QueueUrl: redriveQueueUrl,
ReceiptHandle: messageToDelete.ReceiptHandle
};
sqs.deleteMessage(params, function (err, data) {
if (err) console.log(err, err.stack);
else console.log(data);
});
}
exports.handler = async (event) => {
var params = {
QueueUrl: redriveQueueUrl,
MaxNumberOfMessages: '2',
};
let sqsResponse = await sqs.receiveMessage(params).promise();
let messages = sqsResponse.Messages
for (let index in messages) {
// do something here later.
await deleteMessage(messages[index]);
};
const response = {
statusCode: 200,
body: JSON.stringify('End of Lambda!'),
};
return response;
};
This code correctly reads a given queue but does not delete any message.
How can fix this problem?

sqs.deleteMessage needs to return a promise. Otherwise calling await sqs.deleteMessage() will return immediately.
Try this:
async function deleteMessage (messageToDelete) {
let params = {
QueueUrl: redriveQueueUrl,
ReceiptHandle: messageToDelete.ReceiptHandle
};
return new Promise((resolve, reject)=>{
sqs.deleteMessage(params, function (err, data) {
if (err) return reject(err);
else return resolve(data);
});
})
}

Combine two callbacks into one return

So I have this code:
module.exports.getEstimate = (event, context, callback) => {
var data = JSON.parse(event.body);
lalamove.getQuotation(data ,context, function(err, llm_data){
callback(null,llm_data)
});
};
So it calls lalamove.getQuotation function and returns an object:
{ "totalFee": "108", "totalFeeCurrency": "PHP" }
Now, I have added a new function, that returns this object:
{ "totalFee": "10", "totalFeeCurrency": "PHP" }
from a different function so I thought I should push them in one array and then that is when I would call the callback but it does not work, this is what I have tried
module.exports.getEstimate = (event, context, callback) => {
var data = JSON.parse(event.body);
var response = []
lalamove.getQuotation(data ,context, function(err, llm_data){
const llm_obj = { "lalamove": llm_data }
response.push(llm_obj);
});
inhouse.getQuotation(data ,context, function(err, ih_data){
const ih_obj = {"inhouse": ih_data }
response.push(ih_obj);
});
callback(null,response);
};
and what I want to be the response is like this:
["lalamove": { "totalFee": "108", "totalFeeCurrency": "PHP" },
"inhouse": { "totalFee": "10", "totalFeeCurrency": "PHP" }]
what am I doing wrong?

Your callback(null,response) will not wait for those two callback functions to finish. You can use Promise and use Promise.all(objs).then(function) to wait for all promises finish and run.

Welcome to World's Javascript world - Callback hell.
We have some options for your case: Callback hell, async lib, Promise, async/await...
Callback hell: Call a async function in a callback
module.exports.getEstimate = (event, context, callback) => {
var data = JSON.parse(event.body);
var response = []
lalamove.getQuotation(data, context, function (err, llm_data) {
const llm_obj = { "lalamove": llm_data }
response.push(llm_obj);
// lalamove.getQuotation done!
// call next action
inhouse.getQuotation(data, context, function (err, ih_data) {
const ih_obj = { "inhouse": ih_data }
response.push(ih_obj);
// inhouse.getQuotation done!
// call the last action
callback(null, response);
});
});
};
Async lib: async
You can use waterfall function to do actions in order, and parallel if order is not matter.
module.exports.getEstimate = (event, context, callback) => {
var data = JSON.parse(event.body);
var response = []
async.parallel([
function (next) {
lalamove.getQuotation(data, context, function (err, llm_data) {
// TODO: check err object
const llm_obj = { "lalamove": llm_data }
response.push(llm_obj);
// lalamove.getQuotation done!
// do next action
next();
});
},
function (next) {
inhouse.getQuotation(data, context, function (err, ih_data) {
const ih_obj = { "inhouse": ih_data }
response.push(ih_obj);
// inhouse.getQuotation done!
// do next action
next()
});
}
], function (err) {
// TODO: check err object
// call the last action
callback(null, response);
});
};

Try wrapping two quotation calls in Promise, then utilise Promise.all to wait for both of them to be completed, then return the result to the callback
module.exports.getEstimate = (event, context, callback) => {
let data = JSON.parse(event.body);
// wrap quotation calls in `Promise`
Promise.all([
new Promise(resolve => lalamove.getQuotation(data, context, (err, lalamove) => resolve({ lalamove }))),
new Promise(resolve => inhouse.getQuotation (data, context, (err, inhouse ) => resolve({ inhouse }))),
]).then(response => {
// return the result back to `callback`
callback(null, response);
})
};

You could also try using util.promisify and the async / await syntax.
For example:
const util = require("util");
module.exports.getEstimate = async (event, context, callback) => {
let data = JSON.parse(event.body);
try {
let response = await Promise.all([ util.promisify(lalamove.getQuotation)(data, context),
util.promisify(inhouse.getQuotation)(data, context) ]);
callback(null, response);
} catch (err) {
callback(err);
}
};
We can also do something similar, but without async / await:
const util = require("util");
const getEstimate = (event, context, callback) => {
let data = JSON.parse(event.body);
Promise.all([util.promisify(lalamove.getQuotation)(data, context),
util.promisify(inhouse.getQuotation)(data, context)])
.then(response => callback(null, response))
.catch(err => callback(err));
};

Node.js - Mock result of a promise

I want to mock the result of a function within a node module so that i can run assertions.
Considering the following node module:
const doPostRequest = require('./doPostRequest.js').doPostRequest;
const normalizeSucessResult = require('./normalizer.js').normalizeSucessResult;
const normalizeErrorResult = require('./normalizer.js').normalizeErrorResult;
exports.doPost = (params, postData) => {
return doPostRequest(params, postData).then((res) => {
const normalizedSuccessResult = normalizeSucessResult(res);
return normalizedSuccessResult;
}).catch((err) => {
const normalizedErrorResult = normalizeErrorResult(err);
return normalizedErrorResult;
})
}
The function doPostRequest returns a promise. How can i fake the return value of this promise so that i can assert if normalizeSucessResult has been called?
So for i have tried:
const normalizeSucessResult = require('./normalizer.js');
const doPostRequest = require('./doPostRequests.js');
const doPost = require('./doPost.js');
it('runs a happy flow scenario', async () => {
let normalizeSucessResultStub = sinon.stub(normalizeSucessResult, 'normalizeSucessResult');
let postData = { body: 'Lorum ipsum' };
let params = { host: 'someUrl', port: 433, method: 'POST', path: '/' };
sinon.stub(doPostRequest, 'doPostRequest').resolves("some response data"); //Fake response from doPostRequest
return doPost.doPost(params, postData).then((res) => { //res should be equal to some response data
expect(normalizeSucessResultStub).to.have.been.calledOnce;
expect(normalizeSucessResultStub).to.have.been.with("some response data");
});
});
The doPostRequest module looks like this:
const https = require('https')
module.exports.doPostRequest = function (params, postData) {
return new Promise((resolve, reject) => {
const req = https.request(params, (res) => {
let body = []
res.on('data', (chunk) => {
body.push(chunk)
})
res.on('end', () => {
try {
body = JSON.parse(Buffer.concat(body).toString())
} catch (e) {
reject(e)
}
resolve(body)
})
})
req.on('error', (err) => {
reject(err)
})
if (postData) {
req.write(JSON.stringify(postData))
}
req.end()
})
}

You can use Promise.resolve to return a promise with any given value.
Promise.resolve(“hello world”);

For stub your func you need to do like this
sinon.stub({doPostRequest}, 'doPostRequest').resolves("some response data")

Okay, i figured it out. The function doPostRequest was loaded using require, on the top of the file using const doPostRequest = require('./doPostRequest.js').doPostRequest;
In order to mock the data that comes back from a function that is loaded using require i had to use a node module called mock-require. There are more modules that can take care of this (proxyquire is a populair one) but i picked mock-require (i did not have a specific reason for choosing mock-require).
For anyone else that is stuck with a similar problem, try mock-require to mock the respose from files that are loaded using require.

Node-scheduler error after implement test

After I have successfully implemented one of methods to fetch some data and run test
Code:
const fetchData = async (url) => {
const response = await axios.get(url);
const contentType = response.headers['content-type'];
if (typeof response.data === 'object') {
return JSON.stringify(response.data);
}
throw new Error('Content-Type is unrecognized');
};
module.exports = fetchData;
And test:
describe('fetchData', () => {
it('should return json string response data on successful request', async () => {
const responseData = await fetchData(url);
const expectedData = JSON.stringify({ key1: 'value1' });
assert.deepEqual(responseData, expectedData, 'Response data doesn\'t match');
});
However, I wanted to implement scheduling to my method. I implemented in by using node-scheduler npm module.
After my modification
scheduler.scheduleJob({ start: startTime, end: endtTime }, async () => {
const fetchData = async (url) => {
const response = await axios.get(url);
}
Tests are failing immadietly, furthermore I noticed that error log is going continuously, therefore I have to kill test.
Does anyone have an idea why adding simple scheduler makes my error not working? I am using:
Node v.8.11.4
chai-as-promised
nock