We Keep Coding

How to get HTTP responseBody using Selenium, CDP, JavaScript

I am an AQA and testing an app. According to the test, after the button is clicked, I need to get the responseBody returned from the server, like we have in devtools - network tab. I have tried multiple Java and Python code examples found here, tried to transform them to JavaScript, but nothing worked for me. I've been trying smth like this:
try {
const url = 'http://someUrl';
const driver = await new Builder().forBrowser('chrome').build();
const cdpConnection = await driver.createCDPConnection('page');
await cdpConnection.execute('Network.responseReceived()', response => {
// Network.getResponseBody(), etc.
const res = response.getResponse();
console.log(res);
};
await driver.get(url);
await driver.quit();
} catch (error) {
console.log(error);
}

Network.responseReceived is an Event. So you have to listen to the message from the underlying CDP connection.
wsConnection.on('message', message => {
let data = JSON.parse(message);
if (data.method === 'Network.loadingFinished') {
// ... load response body here
}
});
I use Network.loadingFinished event instead of Network.responseReceived, as the response is then completely loaded.
The problem is, that the CDPConnection class is not properly implemented yet: CDPConnection.js#L18 It doesn't return any promise. This is message-based communication, though it adds the Message ID, to retrieve later the Response Message from the WebSocket, but it doesn't handle that response message here webdriver.js#L1239
Until it is implemented you can use custom CDPConnection class. Here is the TypeScript implementation.
let ID = 0;
type TAwaiter = {
id: number
resolve: (value: any) => void
reject: (reason?: any) => void
};
export class BiDiCDPConnection {
private requests: Map<number, TAwaiter> = new Map();
constructor(private wsConnection, private sessionId: string) {
wsConnection.on('message', this.onMessage.bind(this));
wsConnection.on('close', this.onClose.bind(this));
wsConnection.on('error', this.rejectAll.bind(this));
}
execute <T = any> (method, params, onMessageSent: (err) => any = null): Promise<T> {
let message = {
sessionId: this.sessionId,
method,
params,
id: ++ID,
};
let listener = {
id: message.id,
resolve: null,
reject: null,
};
let promise = new Promise<T>((resolve, reject) => {
listener.resolve = resolve;
listener.reject = reject;
});
this.requests.set(listener.id, listener);
this.wsConnection.send(JSON.stringify(message), onMessageSent)
return promise;
}
private onMessage (message: Buffer) {
let params = JSON.parse(message.toString());
let { id, result } = params;
if (id != null && this.requests.has(id)) {
this.requests.get(id)?.resolve?.(result);
this.requests.delete(id);
}
}
private onClose () {
this.rejectAll(new Error(`CDPConnection: The underlying connection was closed`));
}
private rejectAll(error: Error) {
let awaiters = this.requests.values();
this.requests = new Map();
for (let awaiter of awaiters) {
awaiter.reject(error);
}
}
}
Then you initialize the class and use it for your calls, after you create the inner cdp connection, as createCDPConnection establishes the WebSocket connection.
const cdpConnection = await driver.createCDPConnection('page');
const wsConnection = driver._wsConnection;
const bidiCdpConnection = new BiDiCDPConnection(wsConnection, driver.sessionId);
wsConnection.on('message', message => {
let data = JSON.parse(message);
if (data.method === 'Network.loadingFinished') {
let response = await bidiCdpConnection.execute('Network.getResponseBody', {
requestId: data.params.requestId,
});
console.log(response)
}
});
I use this to monitor (selenium-query/BrowserNetworkMonitor.ts) and intercept (selenium-query/BrowserNetworkInterceptor.ts) requests. You can take and modify those classes for your initial needs.

I am close to getting this to work... but not quite there. See my code here: https://github.com/SeleniumHQ/seleniumhq.github.io/issues/1155
If anyone can figure out the last step I'm missing, that'd be so amazing.
ie.
let test = await cdpConnection.execute('Fetch.getResponseBody', {
requestId: obj.params.requestId,
});
console.log(test); // ------> THIS RETURNS UNDEFINED !!!!

chained await in class chained methods

context: Two javascript classes in separate files, each integrating a different external service and being called in a express.js router.
See "problematic code" below:
route
routes.post('/aws', upload.single('file'), async (req, res) => {
const transcribeParams = JSON.parse(req.body.options)
const bucket = 'bucket-name'
const data = await ( await ( await awsTranscribe.Upload(req.file, bucket)).CreateJob(transcribeParams)).GetJob()
res.send(data)
})
S3 class
class AmazonS3 {
constructor() {
this.Upload = this.Upload
}
async Upload(file, bucket) {
const uploadParams = {
Bucket: bucket,
Body: fs.createReadStream(file.path),
Key: file.filename,
}
this.data = await s3.upload(uploadParams).promise()
return this
}
}
Transcribe class
class Transcribe extends AwsS3 {
constructor() {
super()
this.CreateJob = this.CreateJob
this.GetJob = this.GetJob
}
async CreateJob(params) {
if(this.data?.Location) {
params.Media = { ...params.Media, MediaFileUri: this.data.Location }
}
this.data = await transcribeService.startTranscriptionJob(params).promise()
return this
}
async GetJob(jobName) {
if(this.data?.TranscriptionJob?.TranscriptionJobName) {
jobName = this.data.TranscriptionJob.TranscriptionJobName
}
this.data = await transcribeService.getTranscriptionJob({TranscriptionJobName: jobName}).promise()
return this
}
}
problem: the problem is with the chained awaits in the router file:
await ( await ( await awsTranscribe.Upload...
Yes, it does work, but it would be horrible for another person to maintain this code in the future.
How can i make so it would be just
awsTranscribe.Upload(req.file, bucket).CreateJob(transcribeParams).GetJob() without the .then?

The problem is with the chained awaits in the router file: await ( await ( await awsTranscribe.Upload...
No, that's fine. In particular it would be trivial to refactor it to separate lines:
routes.post('/aws', upload.single('file'), async (req, res) => {
const transcribeParams = JSON.parse(req.body.options)
const bucket = 'bucket-name'
const a = await awsTranscribe.Upload(req.file, bucket);
const b = await b.CreateJob(transcribeParams);
const c = await b.GetJob();
res.send(c);
});
Your actual problem is that a, b, and c all refer to the same object awsTranscribe. Your code would also "work" if it was written
routes.post('/aws', upload.single('file'), async (req, res) => {
const transcribeParams = JSON.parse(req.body.options)
const bucket = 'bucket-name'
await awsTranscribe.Upload(req.file, bucket);
await awsTranscribe.CreateJob(transcribeParams);
await awsTranscribe.GetJob();
res.send(awsTranscribe);
});
The horrible thing is that you are passing your data between these methods through the mutable awsTranscribe.data property - even storing different kinds of data in it at different times! One could change the order of method calls and it would completely break in non-obvious and hard-to-debug ways.
Also it seems that multiple requests share the same awsTranscribe instance. This will not work with concurrent requests. Anything is possible from just "not working" to responding with the job data from a different user (request)! You absolutely need to fix that, then look at ugly syntax later.
What you really should do is get rid of the classes. There's no reason to use stateful objects here, this is plain procedural code. Write simple functions, taking parameters and returning values:
export async function uploadFile(file, bucket) {
const uploadParams = {
Bucket: bucket,
Body: fs.createReadStream(file.path),
Key: file.filename,
};
const data = s3.upload(uploadParams).promise();
return data.Location;
}
export async function createTranscriptionJob(location, params) {
params = {
...params,
Media: {
...params.Media,
MediaFileUri: location,
},
};
const data = await transcribeService.startTranscriptionJob(params).promise();
return data.TranscriptionJob;
}
async function getTranscriptionJob(job) {
const jobName = job.TranscriptionJobName;
return transcribeService.getTranscriptionJob({TranscriptionJobName: jobName}).promise();
}
Then you can import and call them as
routes.post('/aws', upload.single('file'), async (req, res) => {
const transcribeParams = JSON.parse(req.body.options)
const bucket = 'bucket-name'
const location = await uploadFile(req.file, bucket);
const job = await createTranscriptionJob(location, transcribeParams);
const data = await getTranscriptionJob(job);
res.send(c);
});

I got interested in whether it was possible to take an object with several async methods and somehow make them automatically chainable. Well, you can:
function chain(obj, methodsArray) {
if (!methodsArray || !methodsArray.length) {
throw new Error("methodsArray argument must be array of chainable method names");
}
const methods = new Set(methodsArray);
let lastPromise = Promise.resolve();
const proxy = new Proxy(obj, {
get(target, prop, receiver) {
if (prop === "_promise") {
return function() {
return lastPromise;
}
}
const val = Reflect.get(target, prop, receiver);
if (typeof val !== "function" || !methods.has(prop)) {
// no chaining if it's not a function
// or it's not listed as a chainable method
return val;
} else {
// return a stub function
return function(...args) {
// chain a function call
lastPromise = lastPromise.then(() => {
return val.apply(obj, args);
//return Reflect.apply(val, obj, ...args);
});
return proxy;
}
}
}
});
return proxy;
}
function delay(t) {
return new Promise(resolve => {
setTimeout(resolve, t);
});
}
function log(...args) {
if (!log.start) {
log.start = Date.now();
}
const delta = Date.now() - log.start;
const deltaPad = (delta + "").padStart(6, "0");
console.log(`${deltaPad}: `, ...args)
}
class Transcribe {
constructor() {
this.greeting = "Hello";
}
async createJob(params) {
log(`createJob: ${this.greeting}`);
return delay(200);
}
async getJob(jobName) {
log(`getJob: ${this.greeting}`);
return delay(100);
}
}
const t = new Transcribe();
const obj = chain(t, ["getJob", "createJob"]);
log("begin");
obj.createJob().getJob()._promise().then(() => {
log("end");
});
There's a placeholder for your Transcribe class that has two asynchronous methods that return a promise.
Then, there's a chain() function that returns a proxy to an object that makes a set of passed in method names be chainable which allows you to then do something like this:
const t = new Transcribe();
// make chainable proxy
const obj = chain(t, ["getJob", "createJob"]);
obj.createJob().getJob()
or
await obj.createJob().getJob()._promise()
I wouldn't necessarily say this is production-ready code, but it is an interesting feasibility demonstration and (for me) a chance to learn more about a Javascript proxy object.
Here's a different approach that (instead of the proxy object) adds method stubs to a promise to make things chainable:
function chain(orig, methodsArray) {
let masterP = Promise.resolve();
function addMethods(dest) {
for (const m of methodsArray) {
dest[m] = function(...args) {
// chain onto master promise to force sequencing
masterP = masterP.then(result => {
return orig[m].apply(orig, ...args);
});
// add methods to the latest promise befor returning it
addMethods(masterP);
return masterP;
}
}
}
// add method to our returned promise
addMethods(masterP);
return masterP;
}
function delay(t) {
return new Promise(resolve => {
setTimeout(resolve, t);
});
}
function log(...args) {
if (!log.start) {
log.start = Date.now();
}
const delta = Date.now() - log.start;
const deltaPad = (delta + "").padStart(6, "0");
console.log(`${deltaPad}: `, ...args)
}
class Transcribe {
constructor() {
this.greeting = "Hello";
this.cntr = 0;
}
async createJob(params) {
log(`createJob: ${this.greeting}`);
++this.cntr;
return delay(200);
}
async getJob(jobName) {
log(`getJob: ${this.greeting}`);
++this.cntr;
return delay(100);
}
}
const t = new Transcribe();
log("begin");
chain(t, ["getJob", "createJob"]).createJob().getJob().then(() => {
log(`cntr = ${t.cntr}`);
log("end");
});
Since this returns an actual promise (with additional methods attached), you can directly use .then() or await with it without the separate ._promise() that the first implementation required.
So, you can now do something like this:
const t = new Transcribe();
chain(t, ["getJob", "createJob"]).createJob().getJob().then(() => {
log(`cntr = ${t.cntr}`);
});
or:
const t = new Transcribe();
await chain(t, ["getJob", "createJob"]).createJob().getJob();
log(`cntr = ${t.cntr}`);
And, here's a third version where it creates a thenable object (a pseudo-promise) with the added methods on it (if it bothers you to add methods to an existing promise):
function chain(orig, methodsArray) {
if (!methodsArray || !methodsArray.length) {
throw new Error("methodsArray argument must be array of chainable method names");
}
let masterP = Promise.resolve();
function makeThenable() {
let obj = {};
for (const m of methodsArray) {
obj[m] = function(...args) {
// chain onto master promise to force sequencing
masterP = masterP.then(result => {
return orig[m].apply(orig, ...args);
});
return makeThenable();
}
}
obj.then = function(onFulfill, onReject) {
return masterP.then(onFulfill, onReject);
}
obj.catch = function(onReject) {
return masterP.catch(onReject);
}
obj.finally = function(onFinally) {
return masterP.finally(onFinally);
}
return obj;
}
return makeThenable();
}
function delay(t) {
return new Promise(resolve => {
setTimeout(resolve, t);
});
}
function log(...args) {
if (!log.start) {
log.start = Date.now();
}
const delta = Date.now() - log.start;
const deltaPad = (delta + "").padStart(6, "0");
console.log(`${deltaPad}: `, ...args)
}
class Transcribe {
constructor() {
this.greeting = "Hello";
this.cntr = 0;
}
async createJob(params) {
log(`createJob: ${this.greeting}`);
++this.cntr;
return delay(200);
}
async getJob(jobName) {
log(`getJob: ${this.greeting}`);
++this.cntr;
return delay(100);
}
}
const t = new Transcribe();
log("begin");
chain(t, ["getJob", "createJob"]).createJob().getJob().then(() => {
log(`cntr = ${t.cntr}`);
log("end");
});

Async is out of sync

I have a button that makes an http.get on click, and then displays some data, although my async/awaits seem to not be synchronized correctly.
For a reason I haven't quite figured out yet, this.tabRows = file.contents will be called before loadDateFolderFileContents() completes. Am I missing an await somewhere?
async ngAfterViewInit() {
if (!this.drawn) {
if(!file.contents){
await this.dataLakeService.loadDateFolderFileContents(file.parentFolder);
}
this.tabRows = file.contents;
this.redraw();
}
}
public async loadDateFolderFileContents(dateFolder: folder) {
await date.files.map(async file => {
file.contents = await this.getFileContents(dateFolder.name, file.name);
});
}
public async loadDateFolderFileContents(dateName: string, fileName: string): Promise<any> {
var url = this.buildUrl(dateName, fileName);
var fileContents = {};
await this.http.get<any>(url).toPromise()
.then(contents => {
fileContents = contents;
})
.catch(e => {
console.log(`Error retreiving file contents from url ${url}. Exception:${e}`);
});
return fileContents;
}

If you use await in a map, map will always return an array of promise. This is because asynchronous functions always return promises.
Since map always return promises (if you use await), you have to wait for the array of promises to get resolved. You can do this with await Promise.all(arrayOfPromises), after all promises are resolved, Promise.all returns a single promise.
below is your updated code
async ngAfterViewInit() {
if (!this.drawn) {
if(!file.contents){
await this.dataLakeService.loadDateFolderFileContents(file.parentFolder);
}
this.tabRows = file.contents;
this.redraw();
}
}
public async loadDateFolderFileContents(dateFolder: folder) {
await Promise.all(date.files.map(async file => {
file.contents = await this.getFileContents(dateFolder.name, file.name);
}));
}
public async loadDateFolderFileContents(dateName: string, fileName: string): Promise<any> {
var url = this.buildUrl(dateName, fileName);
var fileContents = {};
await this.http.get<any>(url).toPromise()
.then(contents => {
fileContents = contents;
})
.catch(e => {
console.log(`Error retreiving file contents from url ${url}. Exception:${e}`);
});
return fileContents;
}

map is not async so instead of:
await date.files.map(async file => {
...
try
await Promise.all(date.files.map(async file => {
...)

I think this is what you are trying to do:
async ngAfterViewInit() {
if (!this.drawn) {
if(!file.contents){
await this.dataLakeService.loadDateFolderFileContents(file.parentFolder);
}
this.tabRows = file.contents;
this.redraw();
}
}
public async loadDateFolderFileContents(dateFolder: folder) {
await Promise.allSettled(date.files.map(async file => {
file.contents = await this.getFileContents(dateFolder.name, file.name);
return file;
}));
}
public async loadDateFolderFileContents(dateName: string, fileName: string): Promise<any> {
var url = this.buildUrl(dateName, fileName);
var fileContents = {};
try {
const contents = await this.http.get<any>(url).toPromise();
fileContents = contents;
return fileContents;
} catch(e){
console.log(`Error retreiving file contents from url ${url}. Exception:${e}`);
}
}

Converting an Image url to base64 in Angular

I am struggling trying to convert a given image url to base64... in my case i have a String with the image's path
var imgUrl = `./assets/logoEmpresas/${empresa.logoUrl}`
how can i convert the given image url in a base64 directly?... i tried this post.
Converting an image to base64 in angular 2
but this post is getting the image from a form... how can i adapt it?

You can use this to get base64 image
async function getBase64ImageFromUrl(imageUrl) {
var res = await fetch(imageUrl);
var blob = await res.blob();
return new Promise((resolve, reject) => {
var reader = new FileReader();
reader.addEventListener("load", function () {
resolve(reader.result);
}, false);
reader.onerror = () => {
return reject(this);
};
reader.readAsDataURL(blob);
})
}
Then call it like this
getBase64ImageFromUrl('your url')
.then(result => testImage.src = result)
.catch(err => console.error(err));

works like charm in pdfMake and angular
You can use this function to create generate a base64 image
toDataURL = async (url) => {
console.log("Downloading image...");
var res = await fetch(url);
var blob = await res.blob();
const result = await new Promise((resolve, reject) => {
var reader = new FileReader();
reader.addEventListener("load", function () {
resolve(reader.result);
}, false);
reader.onerror = () => {
return reject(this);
};
reader.readAsDataURL(blob);
})
return result
};
and then call it like this
imageSrcString = await this.toDataURL(imageSrc)

If we're doing this in Angular, we may as well make use of HttpClient and a Service.
Let's go ahead and add the HttpClientModule into our related Module, we'll need this in order to use HttpClient.
#NgModule({
imports: [HttpClientModule],
...
})
export class AppModule {}
Then let's create a generic Image Service, and then ask Angular to inject the HttpClient into our Service.
#Injectable()
export class ImageService {
constructor(private http: HttpClient) { }
}
Once that's done we can actually create our function in our service
imageUrlToBase64(urL: string) {
return this.http.get(urL, {
observe: 'body',
responseType: 'arraybuffer',
})
.pipe(
take(1),
map((arrayBuffer) =>
btoa(
Array.from(new Uint8Array(arrayBuffer))
.map((b) => String.fromCharCode(b))
.join('')
)
),
)
}
When we use http.get and provide arraybuffer as our response type, Angular interprets the body of our request as an ArrayBuffer. What that means is that we'll now have our image as an array of bytes. All we need to do is then convert our ArrayBuffer to a base64 string. If you'd like to view alternative options, this SO Question has good answers.
// taken from above
map(
btoa(
Array.from(new Uint8Array(arrayBuffer))
.map((b) => String.fromCharCode(b))
.join('')
)
)
Now that the function is done, we can shift to usage:
#Component()
export class AppComponent {
base64Image: string;
constructor(private imageService: ImageService) {
this.imageService.imageUrlToBase64('https://picsum.photos/200/300').subscribe(
base64 => {
this.base64Image = base64
})
}
}
We'll now have access to the image as a base64

How to write an async function that resolves when `data` event emitter fires

I am using node-serialport to communicate with a piece of hardware. It just writes a command and receives a response.
https://serialport.io/docs/en/api-parsers-overview
The following code works:
const port = new SerialPort(path);
const parser = port.pipe(new Readline({ delimiter: '\r', encoding: 'ascii' }));
const requestArray = [];
parser.on('data', (data) => {
// get first item in array
const request = requestArray[0];
// remove first item
requestArray.shift();
// resolve promise
request.promise.resolve(data);
});
export const getFirmwareVersion = async () => {
let resolvePromise;
let rejectPromise;
const promise = new Promise((resolve, reject) => {
resolvePromise = resolve;
rejectPromise = reject;
});
const title = 'getFirmwareVersion';
const cmd = 'V\r';
requestArray.push({
title,
cmd,
promise: {
resolve: resolvePromise,
reject: rejectPromise
}
});
await v2Port.write(cmd);
return promise;
};
Then from my app (which is written in electron/react) I can call the function:
<Button onClick={() => {
let data = await _api.getFirmwareVersion();
console.log('done waiting...');
console.log(data);
}>
Click Me
</Button>
Is there anyway I can refactor this code to make it more succinct?
Is there a way to get the Promise from the async function, rather than having to make a new Promise?
Is there a way to tap into the Transform Stream that already exists and pipe the Promise in there somehow?
I'm also new to async/await, and wanted to avoid using callbacks, especially in the React/Redux side of things.
I aim to have a lot of these endpoints for the api (i.e. getFirmwareVersion, getTemperature, etc...). So I want to make the code as concise as possible. I don't want the UI to have any underlying knowledge of how the API is getting the data. It just needs to request it like any other API and wait for a response.

Oh, I think I get it. The parser is receiving data constantly. So when a request comes, you wait for the next data and send it when it arrives. I suggest you to write an intermediate class.
Like this:
const SerialPort = require('serialport')
const Readline = require('#serialport/parser-readline')
const { EventEmitter } = require('events');
class SerialPortListener extends EventEmitter {
constructor(path) {
super();
this.serialPortPath = path;
}
init() {
this.serialPort = new SerialPort(this.serialPortPath);
const parser = this.serialPort.pipe(new Readline({ delimiter: '\r', encoding: 'ascii' }));
parser.on('data', data => this.emit('data', data));
}
}
Then you could modify the getFirmwareVersion like this:
const serialPortListener = new SerialPortListener(path);
serialPortListener.init();
export const getFirmwareVersion = () => {
return new Promise((resolve, reject) => {
serialPortListener.once('data', async (data) => {
try {
const cmd = 'V\r';
await v2Port.write(cmd);
resolve(data);
} catch (ex) {
reject(ex);
}
});
});
};

Based on help from Mehmet, here is what I ended up with:
const _port = new SerialPort(path);
const _parser = _port.pipe(new Readline({ delimiter: '\r', encoding: 'ascii' }));
const waitForData = async () => {
return new Promise((resolve, reject) => {
const timeoutId = setTimeout(() => reject('Write Timeout'), 500);
_parser.once('data', (data) => {
clearTimeout(timeoutId);
resolve(data);
});
});
};
const createAPIFunction = (cmdTemplate, validationString) => {
return async (config) => {
try {
// replace {key} in template with config[key] props
const cmd = cmdTemplate.replace(/{(\w+)}/g, (_, key) => {
return config[key];
});
_port.write(cmd + '\r');
const data = await waitForData();
// validate data
if (data.startsWith(validationString)) {
// is valid
return data;
} else {
// invalid data
throw new Error('Invalid Data Returned');
}
} catch (err) {
throw err;
}
};
};
export const getFirmwareVersion = createAPIFunction('V', 'V1');
export const enableSampling = createAPIFunction('G1{scope}', 'G11');