We Keep Coding

Http requests being dropped in Chrome Extension

Summary:
I've built a chrome extension that reaches out to external API to fetch some data. Sometimes that data returns quickly, sometimes it takes 4 seconds or so. I'm often doing about 5-10 in rapid succession (this is a scraping tool).
Previously, a lot of requests were dropped because the service worker in V3 of Manifest randomly shuts down. I thought I had resolved that. Then I realized there was a race condition because local storage doesn't have a proper queue.
Current Error - Even with all these fixes, requests are still being dropped. The external API returns the correct data successfully, but it seems like the extension never gets it. Hoping someone can point me in the right direction.
Relevant code attached, I imagine it will help someone dealing with these queue and service worker issues.
Local Storage queue
let writing: Map<string, Promise<any>> = new Map();
let updateUnsynchronized = async (ks: string[], f: Function) => {
let m = await new Promise((resolve, reject) => {
chrome.storage.local.get(ks, res => {
let m = {};
for (let k of ks) {
m[k] = res[k];
}
maybeResolveLocalStorage(resolve, reject, m);
});
});
// Guaranteed to have not changed in the meantime
let updated = await new Promise((resolve, reject) => {
let updateMap = f(m);
chrome.storage.local.set(updateMap, () => {
maybeResolveLocalStorage(resolve, reject, updateMap);
});
});
console.log(ks, 'Updated', updated);
return updated;
};
export async function update(ks: string[], f: Function) {
let ret = null;
// Global lock for now
await navigator.locks.request('global-storage-lock', async lock => {
ret = await updateUnsynchronized(ks, f);
});
return ret;
}
Here's the main function
export async function appendStoredScrapes(
scrape: any,
fromHTTPResponse: boolean
) {
let updated = await update(['urlType', 'scrapes'], storage => {
const urlType = storage.urlType;
const scrapes = storage.scrapes;
const {url} = scrape;
if (fromHTTPResponse) {
// We want to make sure that the url type at time of scrape, not time of return, is used
scrapes[url] = {...scrapes[url], ...scrape};
} else {
scrapes[url] = {...scrapes[url], ...scrape, urlType};
}
return {scrapes};
});
chrome.action.setBadgeText({text: `${Object.keys(updated['scrapes']).length}`});
}
Keeping the service worker alive
let defaultKeepAliveInterval = 20000;
// To avoid GC
let channel;
// To be run in content scripts
export function contentKeepAlive(name : string) {
channel = chrome.runtime.connect({ name });
channel.onDisconnect.addListener(() => contentKeepAlive(name));
channel.onMessage.addListener(msg => { });
}
let deleteTimer = (chan : any) => {
if (chan._timer) {
clearTimeout(chan._timer);
delete chan._timer;
}
}
let backgroundForceReconnect = (chan : chrome.runtime.Port) => {
deleteTimer(chan);
chan.disconnect();
}
// To be run in background scripts
export function backgroundKeepAlive(name : string) {
chrome.runtime.onConnect.addListener(chan => {
if (chan.name === name) {
channel = chan;
channel.onMessage.addListener((msg, chan) => { });
channel.onDisconnect.addListener(deleteTimer);
channel._timer = setTimeout(backgroundForceReconnect, defaultKeepAliveInterval, channel);
}
});
}
// "Always call sendResponse() in your chrome.runtime.onMessage listener even if you don't need
// the response. This is a bug in MV3." — https://stackoverflow.com/questions/66618136/persistent-service-worker-in-chrome-extension
export function defaultSendResponse (sendResponse : Function) {
sendResponse({ farewell: 'goodbye' });
}
Relevant parts of background.ts
backgroundKeepAlive('extension-background');
let listen = async (request, sender, sendResponse) => {
try {
if (request.message === 'SEND_URL_DETAIL') {
const {url, website, urlType} = request;
await appendStoredScrapes({url}, false);
let data = await fetchPageData(url, website, urlType);
console.log(data, url, 'fetch data returned background');
await appendStoredScrapes(data, true);
defaultSendResponse(sendResponse);
} else if (request.message === 'KEEPALIVE') {
sendResponse({isAlive: true});
} else {
defaultSendResponse(sendResponse);
}
} catch (e) {
console.error('background listener error', e);
}
};
chrome.runtime.onMessage.addListener(function (request, sender, sendResponse) {
listen(request, sender, sendResponse);
});

Node JS SSE: RxJs observable error gets ignored while sharing a subscription

There is an SSE endpoint that shares a subscription if the consumer with the same key is already subscribed. If there is an active subscription the data is being polled from another client.
The problem is that the outer subscription never seems to catch the error and delegate it to the router in order to close the connection with the client: polling stops, but connection stays active.
I think the issue is how I start the subscription that is to be shared... but I can't think of a way to resolve this in another way currently.
Router (SSE) / outer subscription:
...
const clientId = Date.now();
const newClient = {
id: clientId,
res,
};
clients.push(newClient);
const sub = subscriptionService.listenToChanges(req.context, categoryIds).subscribe({
next: (data) => {
if (JSON.stringify(data) !== '{}') {
newClient.res.write(`data: ${JSON.stringify(data)}\n\n`);
} else {
newClient.res.write(': poke...\n\n');
}
},
error: () => {
// we never get here...
next(new InternalError());
clients = clients.filter((c) => c.id !== clientId);
res.end();
},
complete: () => {
res.end();
clients = clients.filter((c) => c.id !== clientId);
},
});
req.on('close', () => {
subscriptionService.stopListening(req.context);
sub.unsubscribe();
clients = clients.filter((c) => c.id !== clientId);
});
...
SubscriptionService
...
#trace()
public listenToChanges(ctx: Context, ids: string[]): Observable<{ [key: string]: Data }> {
const key = ctx.user?.email || ClientTypeKey.Anon;
try {
if (this.pool$[key]) {
return this.pool$[key];
}
this.poolSource[key] = new BehaviorSubject<{ [p: string]: Data }>({});
this.pool$[key] = this.poolSource[key].asObservable();
this.fetchData(ctx, ids);
return this.pool$[key].pipe(
catchError((e) => throwError(e)), // we never get here...
);
} catch (e) {
throw new Error(`Subscription Service Listen returned an error: "${e}"`);
}
}
...
private fetchData(ctx: Context, ids: string[]): void {
const key = ctx.user?.email || ClientTypeKey.Anon;
const sub = this.service.getData(ctx, ids)
.pipe(
catchError((e) => throwError(e)),
).subscribe(
(r) => this.poolSource[key].next(r),
(e) => throwError(e), // last time the error is caught
);
this.subscriptions[key] = sub;
}
...
Polling Service
...
#trace()
public getData(ctx: Context, ids: string[]): Observable<{[key: string]: Data}> {
try {
const key = ctx.user?.email || ClientTypeKey.Anon;
const pollingInterval = config.get('services.pollingInterval') || 10000;
return interval(pollingInterval).pipe(
startWith(0),
switchMap(() => this.getConfig(ctx, !!this.cachedData[key])),
map((r) => this.getUpdatedData(ctx, r.data, ids)),
catchError((e) => throwError(e)),
);
} catch (e) {
throw new Error(`Get Data returned an error: "${e}"`);
}
}
...

throwError doesn't actually throw an error, but rather creates an observable that emits an error.
From the docs:
[throwError] Creates an observable that will create an error instance and push it to the consumer as an error immediately upon subscription.
This is why using it inside subscribe does not work as intended. You should simply throw:
.subscribe(
(r) => this.poolSource[key].next(r),
(e) => throw new Error(e)
);
It seems like you have some unnecessary complexity in the way you are calling fetchData() in order to subscribe and push the result into a BehaviorSubject. I don't know all your requirements, but it seems like maybe you don't need the BehaviorSubject at all.
Instead of subscribing in fetchData(), you could simply return the observable and add that into your pool$ array, or maybe even get rid of fetchData() altogether:
public listenToChanges(ctx: Context, ids: string[]): Observable<{ [key: string]: Data }> {
const key = ctx.user?.email || ClientTypeKey.Anon;
try {
if (!this.pool$[key]) {
this.pool$[key] = this.service.getData(ctx, ids).pipe(
catchError((e) => throwError(e))
);
}
return this.pool$[key];
} catch (e) {
throw new Error(`Subscription Service Listen returned an error: "${e}"`);
}
}
Notes:
with the above simplification, maybe you no longer need the outer try/catch
this isn't a complete solution and may require some tweaks in other places of your code. I just wanted to point out, what seems like unnecessary complexity.

resolve after new Promise did nothing (console.log -> undefined)

here is my Promise Function, I go through each blob in Azure BlobStorage, then I read each blob. console.log(download) delivers the values as JSON.
But to close the return new Promise function, I want that resolve should return the JSON data from the reading the blobstream. But here in my case, resolve leads to nothing.
In Angular Service.ts file the code looks like this:
getData(): Promise<JsonData[]> {
return new Promise(async resolve => {
const containerName = "blobcontainer";
const containerClient = this.blobServiceClient.getContainerClient(containerName);
//list blobs
let i = 1;
async function main() {
i = 1;
for await (const blob of containerClient.listBlobsFlat()) {
console.log(`Blob ${i++}: ${blob.name}`);
const blockBlobClient = containerClient.getBlockBlobClient(blob.name);
//console.log(blockBlobClient)
const downloadBlockBlobResponse = await blockBlobClient.download(0);
const download = await blobToString(await downloadBlockBlobResponse.blobBody)
//console.log(downloadBlockBlobResponse)
console.log(download)
}
}
async function blobToString(blob: Blob): Promise<string> {
const fileReader = new FileReader();
return new Promise((resolve, reject) => {
fileReader.onloadend = (ev: any) => {
JSON.parse(ev.target!.result)
resolve(JSON.parse(ev.target!.result));
};
fileReader.onerror = reject;
fileReader.readAsText(blob);
});
}
const _blob = await main().catch((err) => {
console.error('message'); return null
});
resolve(_blob) //resolve should return downloaded JSON file, but it didn't
})
}
Then in the component file, I want to retrieve the data from the resolve, which should return the JSON string variables like "name", "timestamp", "value"- But in my case, you receive metadata from the blob and not the contents. Means the service.ts file isn't correctly programmed:
xy.component.ts
export class xyComponent implements OnInit {
#Input() title: string;
//jsondatas: Array<JsonData> = [];
jsondata: JsonData;
name: String;
timestamp: string;
value: number;
//constructor() { }
private jsonlistService: JsonDataService;
jsondatas: JsonData[]=null;
constructor(private jsonService: JsonDataService) {
this.jsonlistService = jsonService;
}
ngOnInit(): void {
this.jsonlistService.getData()
.then(results => this.jsondatas = results);
console.log(this.jsonService)
}
}
EDIT:
Even if I return download at the main function, resolve from main() doesn't deliver the json string.
Second EDIT:
here is the snippets how to return data:
async function main() {
i = 1;
for await (const blob of containerClient.listBlobsFlat()) {
console.log(`Blob ${i++}: ${blob.name}`);
const blockBlobClient = containerClient.getBlockBlobClient(blob.name);
//console.log(blockBlobClient)
const downloadBlockBlobResponse = await blockBlobClient.download(0);
const download = await blobToString(await downloadBlockBlobResponse.blobBody)
//console.log(downloadBlockBlobResponse)
console.log(download)
return download
}
}
But I didn't receive the downloaded file, error is still the same.
Would be very nice if you could help me

You doesn't return anything from main. Just return the answer.

How to write an async function that resolves when `data` event emitter fires

I am using node-serialport to communicate with a piece of hardware. It just writes a command and receives a response.
https://serialport.io/docs/en/api-parsers-overview
The following code works:
const port = new SerialPort(path);
const parser = port.pipe(new Readline({ delimiter: '\r', encoding: 'ascii' }));
const requestArray = [];
parser.on('data', (data) => {
// get first item in array
const request = requestArray[0];
// remove first item
requestArray.shift();
// resolve promise
request.promise.resolve(data);
});
export const getFirmwareVersion = async () => {
let resolvePromise;
let rejectPromise;
const promise = new Promise((resolve, reject) => {
resolvePromise = resolve;
rejectPromise = reject;
});
const title = 'getFirmwareVersion';
const cmd = 'V\r';
requestArray.push({
title,
cmd,
promise: {
resolve: resolvePromise,
reject: rejectPromise
}
});
await v2Port.write(cmd);
return promise;
};
Then from my app (which is written in electron/react) I can call the function:
<Button onClick={() => {
let data = await _api.getFirmwareVersion();
console.log('done waiting...');
console.log(data);
}>
Click Me
</Button>
Is there anyway I can refactor this code to make it more succinct?
Is there a way to get the Promise from the async function, rather than having to make a new Promise?
Is there a way to tap into the Transform Stream that already exists and pipe the Promise in there somehow?
I'm also new to async/await, and wanted to avoid using callbacks, especially in the React/Redux side of things.
I aim to have a lot of these endpoints for the api (i.e. getFirmwareVersion, getTemperature, etc...). So I want to make the code as concise as possible. I don't want the UI to have any underlying knowledge of how the API is getting the data. It just needs to request it like any other API and wait for a response.

Oh, I think I get it. The parser is receiving data constantly. So when a request comes, you wait for the next data and send it when it arrives. I suggest you to write an intermediate class.
Like this:
const SerialPort = require('serialport')
const Readline = require('#serialport/parser-readline')
const { EventEmitter } = require('events');
class SerialPortListener extends EventEmitter {
constructor(path) {
super();
this.serialPortPath = path;
}
init() {
this.serialPort = new SerialPort(this.serialPortPath);
const parser = this.serialPort.pipe(new Readline({ delimiter: '\r', encoding: 'ascii' }));
parser.on('data', data => this.emit('data', data));
}
}
Then you could modify the getFirmwareVersion like this:
const serialPortListener = new SerialPortListener(path);
serialPortListener.init();
export const getFirmwareVersion = () => {
return new Promise((resolve, reject) => {
serialPortListener.once('data', async (data) => {
try {
const cmd = 'V\r';
await v2Port.write(cmd);
resolve(data);
} catch (ex) {
reject(ex);
}
});
});
};

Based on help from Mehmet, here is what I ended up with:
const _port = new SerialPort(path);
const _parser = _port.pipe(new Readline({ delimiter: '\r', encoding: 'ascii' }));
const waitForData = async () => {
return new Promise((resolve, reject) => {
const timeoutId = setTimeout(() => reject('Write Timeout'), 500);
_parser.once('data', (data) => {
clearTimeout(timeoutId);
resolve(data);
});
});
};
const createAPIFunction = (cmdTemplate, validationString) => {
return async (config) => {
try {
// replace {key} in template with config[key] props
const cmd = cmdTemplate.replace(/{(\w+)}/g, (_, key) => {
return config[key];
});
_port.write(cmd + '\r');
const data = await waitForData();
// validate data
if (data.startsWith(validationString)) {
// is valid
return data;
} else {
// invalid data
throw new Error('Invalid Data Returned');
}
} catch (err) {
throw err;
}
};
};
export const getFirmwareVersion = createAPIFunction('V', 'V1');
export const enableSampling = createAPIFunction('G1{scope}', 'G11');

Proper way of creating a EventEmitter that works with Promises in the background

I'm creating a "class" that emits events such as error, data, downloadFile and initialize. Each event is fired after a request is made, and each event is fired by a method that has the same name:
class MyClass extends EventEmitter {
constructor(data) {
this.data = data
this.initialize()
.then(this.downloadFile)
.then(this.data)
.catch(this.error)
}
initialize() {
const req = superagent.post('url...')
superagent.send(data)
const res = await req // this will actually fire the request
this.emit('initialize')
this.url = res.body
return res
}
downloadFile() {
const req = superagent.put(this.url)
const res = await req; // this will actually fire the request
req.on('progress', (progress) => this.emit('downloadFile', progress)
//
// save to disk
//
return res
}
data() {
// Next in the sequence. And will fire the 'data' event: this.emit('data', data)
}
error(err) {
this.emit('error', err)
}
}
After that I have the data method to be called. My doubt is: Is there a design pattern to call the events in sequence without using Promises? Currently I'm using chaining, but I'm feeling that this isn't the best approach, maybe I'm wrong.
this.initialize()
.then(this.downloadFile)
.then(this.data)
.catch(this.error)
But I feel that could be a better approach.
Answers for bergi's questions:
a) Why are you using class syntax?
Because it's easier to inherit from EventEmitter and personally I think it's more readable than using a constructor
functin, e.g:
function Transformation(data) {
this.data = data
}
// Prototype stuffs here
b) How this code is going to be used
I'm creating a client to interact with my API. The ideia is that the user can see what is happening in the background. E.g:
const data = {
data: {},
format: 'xls',
saveTo: 'path/to/save/xls/file.xls'
}
const transformation = new Transformation(data)
// Events
transformation.on('initialize', () => {
// Here the user knows that the transformation already started
})
transformation.on('fileDownloaded', () => {
// Here the file has been downloaded to disk
})
transformation.on('data', (data) => {
// Here the user can see details of the transformation -
// name,
// id,
// size,
// the original object,
// etc
})
transformation.on('error', () => {
// Here is self explanatory, if something bad happens, this event will be fired
})
c) What it is supposed to do?
The user will be able to transform a object with data into a Excel.

It sounds like the transformation object you are creating is used by the caller solely for listening to the events. The user does not need a class instance with properties to get or methods to call. So don't make one. KISS (keep it super simple).
function transform(data) {
const out = new EventEmitter();
async function run() {
try {
const url = await initialise();
const data = await downloadFile(url);
out.emit('data', data);
} catch(err) {
out.emit('error', err);
}
}
async function initialise() {
const req = superagent.post('url...')
superagent.send(data)
const res = await req // this will actually fire the request
out.emit('initialize')
return res.body
}
async function downloadFile(url) {
const req = superagent.put(url)
req.on('progress', (progress) => out.emit('downloadFile', progress)
const res = await req; // this will actually fire the request
//
// save to disk
//
return data;
}
run();
return out;
}
It might be even simpler to leave out the (once-only?) data and error events and just to return a promise, alongside the event emitter for progress notification:
return {
promise: run(), // basically just `initialise().then(downloadFile)`
events: out
};

If you want another way to call the events in sequence, and if you're using a Node.js version that supports ES7, you can do the following :
class MyClass extends EventEmitter {
constructor(data) {
this.data = data;
this.launcher();
}
async launcher() {
try {
await this.initialize();
await this.downloadFile();
await this.data();
}
catch(err) {
this.error(err);
}
}
initialize() {
const req = superagent.post('url...');
superagent.send(data);
this.emit('initialize');
this.url = req.body;
return req;
}
downloadFile() {
const req = superagent.put(this.url);
req.on('progress', (progress) => this.emit('downloadFile', progress)
//
// save to disk
//
return req;
}
data() {
// Next in the sequence. And will fire the 'data' event: this.emit('data', data)
}
error(err) {
this.emit('error', err)
}
}
Explanation : instead of awaiting for your Promises inside your functions, just return the Promises and await for them at root level.