How to create/postmessage to web workers in an array - javascript

Question is simple:
Example:
For (iterate based on amount of cores){
Let worker = workers[I]
Worker.postmessage
}
End of example .
Disclaimer: This example only shows what is expected of the end result and is in no means in what is considered "working condition" . Also note that the method used above does not return a worker for "workers[iterator]" instead just undefined.
Objective: Create working methods:
1: make array of unknown amount of workers(based on cores).
2: once that array is built, post a message to each worker and have a returned result(other than undefined).
Note: I do have a hypothesis of why it does not work:
1: web workers are created and are only accessable through the event that created them and its only acception is the onmessage "event" handler .
in defiance of my hypothesis there is such things that would say neigh to what is written above for example , like thread.js that allows for thread pooling and other procedures.
This is the main reason of why I ask , because I do know it is possible but would like a simple answer.
Thanks for your time .

Here is an example:
function createWorker (workerScript) {
const blob = new Blob([`(${workerScript})(self)`], {type: 'application/javascript'});
return new Worker(URL.createObjectURL(blob));
};
function workerCode (self) {
self.onmessage = function (message) {
postMessage(`Data from worker: ${message.data}`);
};
};
// assuming that you will send only one message to the worker,
// and that the worker will produce only one message too.
function workerPromise (worker, message) {
const promise = new Promise((resolve, reject) => {
worker.onmessage = resolve;
}).then(message => message.data);
worker.postMessage(message);
return promise;
}
(async () => {
const workers = [];
for (let i = 0; i < navigator.hardwareConcurrency; i++) {
workers.push(createWorker(workerCode));
}
const results = await Promise.all(
workers.map((w, index) => workerPromise(w, `worker ${index}`))
);
console.log(results);
})();

Start with the "Examples" section under: https://developer.mozilla.org/en-US/docs/Web/API/NavigatorConcurrentHardware/hardwareConcurrency
Modified example:
// in Main thread
const numberOfCPUCores = window.navigator.hardwareConcurrency;
const workerList = [];
const cpuWorkerMessageHandler = event => {
// process message from a worker by accessing: event.data
}
for (let i = 0; i < numberOfCPUCores; i++) {
const newWorker = new Worker('cpuworker.js');
newWorker.addEventListener("message", cpuWorkerMessageHandler);
workerList.push(newWorker);
}
// then, when done with all processing, terminate all workers
workerList.forEach(w => w.terminate());

Related

Run Go-WebAssembly before onmessage event in a Web Worker

I'm trying to include a Go-WebAssembly function inside a JavaScript Web Worker, and the problem is that the event onmessage from the worker runs before the WebAssembly loads so everytime I call the WebAssembly function I got an error: "yourFunction is not defined". I hope you can help me figuring out how to solve this problem or you can give me ideas how to implement this. Thanks !
A simplified version of my code:
main.go
package main
import (
"fmt"
"log"
"syscall/js"
)
func myGoFunction(this js.Value, i []js.Value) interface{} {
//Do some hard work
fmt.Println(i[0])
return true
}
func main() {
js.Global().Set("myGoFunction", js.FuncOf(myGoFunction))
<-make(chan bool)
}
main.js
const doSomething = () => {
if (myArray.length > 0)
worker.postMessage({ value: myArray.shift() })
}
const init = () => {
if (worker) worker.terminate()
worker = new Worker('worker.js')
worker.postMessage({ a: A, b: B, bool: true })
worker.onmessage = doSomething
}
init()
worker.js
importScripts('wasm_exec.js');
const go = new Go();
WebAssembly.instantiateStreaming(fetch("main.wasm"), go.importObject).then((result) => {
go.run(result.instance);
});
onmessage = (e) => {
const {settings} = e.data
if (settings) {
//set some values
} else {
for (let i= 0; i < 1000000; i++)
someArray[i] = calculate(i)
postMessage({someArray})
}
}
const calculate = (i) => {
//Do more
//Here is where I call the go function
myGoFunction(i)
}
Something I did to see if the myGoFunction is loading, is to put the WebAssembly.instantiateStreaming into a promise then call the onmessage but of course this will load the WebAssembly.instantiateStreaming millions of times and the job is done but extremely slow. Or maybe I implemented the promises the wrong way. I don't know, please help. :D
You can store the promise returned from WebAssembly.instantiateStreaming() and await it in your onmessage handler:
const waInit = WebAssembly.instantiateStreaming(fetch("main.wasm"), go.importObject).then((result) => {
go.run(result.instance);
});
onmessage = async (e) => {
await waInit; // now WA is ready
const {settings} = e.data
// The rest of your handler

Javascript object retaining "old" properties, can't override?

I have the following code:
const readDataFromSql = () => {
// going to have to iterate through all known activities + load them here
let sql = "[...]"
return new Promise((resolve, reject) => {
executeSqlQuery(sql).then((dict) => {
let loadedData = [];
for (let key in dict) {
let newItemVal = new ItemVal("reading hw", 7121, progress.DONE);
loadedData.push(newItemVal);
}
resolve(loadedData);
});
});
}
ItemVal implementation:
class ItemVal {
constructor(name, time, type) {
this.name = name
this.time = time
this.type = type
}
}
Let's assume that newItemVal = "reading hwj", 5081, progress.PAUSED when readDataFromSql() first runs.
readDataFromSql() is then again called after some state changes -- where it repulls some information from a database and generates new values. What is perplexing, however, is that when it is called the second time, newItemVal still retains its old properties (attaching screenshot below).
Am I misusing the new keyword?
From what I can see in your example code, you are not mutating existing properties but creating a new object with the ItemVal constructor function and adding them to an array, that you then return as a resolved promise. Are you sure the examples you give a correct representation of what you are actually doing
Given that, I'm not sure what could be causing the issue you are having, but I would at least recommend a different structure for your code, using a simpler function for the itemVal.
Perhaps with this setup, you might get an error returned that might help you debug your issue.
const itemVal = (name, time, type) => ({ name, time, type })
const readDataFromSql = async () => {
try {
const sql = "[...]"
const dict = await executeSqlQuery(sql)
const loadedData = dict.map((key) =>
ItemVal("reading hw", 7121, progress.DONE)
)
return loadedData
} catch (error) {
return error
}
};
If the issue is not in the function, then I would assume that the way you handle the data, returned from the readDataFromSql function, is where the issue lies. You need to then share more details about your implementation.
const readDataFromSql = async () => {
let sql = "[...]"
------> await executeSqlQuery(sql).then((dict) => {
Use the await keyword instead of creating a new promise.
I did some modification and found that below code is working correctly, and updating the new values on each call.
const readDataFromSql = () => {
return new Promise((resolve, reject) => {
let loadedData = [];
let randomVal = Math.random();
let newItemVal = new ItemVal(randomVal*10, randomVal*100, randomVal*1000);
loadedData.push(newItemVal);
resolve(loadedData);
});
}
Could you recheck if you are using below line in the code, as it will instantiate object with same properties again and again.
let newItemVal = new ItemVal("reading hw", 7121, progress.DONE);
You can modify your code as below to simplify the problem.
const readDataFromSql = async () => {
// going to have to iterate through all known activities + load them here
let sql = "[...]" // define sql properly
let result = await executeSqlQuery(sql);
let loadedData = [];
for (let row in result) {
let newItemVal = new ItemVal(row.name, row.time, row.type);
loadedData.push(newItemVal);
}
return loadedData;
}
class ItemVal {
constructor(name, time, type) {
this.name = name
this.time = time
this.type = type
}
}
What you are talking about is an issue related to Object mutation in Redux, however, you didn't add any redux code. Anyway, you might be making some mistake while recreating(not mutating) the array.
General solution is the use spread operator as:
loadedData = [ ...loadedData.slice(0) , ...newloadedData]
In Dropdown.js line 188 instead of console.log-ing your variable write debugger;
This will function as a breakpoint. It will halt your code and you can inspect the value by hovering your mouse over the code BEFORE the newItemVal is changed again.
I can see in your screenshot that the newItemVal is modified again after you log it.

Axios request not being made in electron app

I am new to javascript async/await etc. The code below has been stripped of any unimportant parts.
The function addAllocationToQueue is invoked from a button press in an electron app. This makes an axios request and creates a new allocation item which belongs to a user. After this has completed, populateAllocationQueue is invoked which updates the DOM to reflect the new allocation item that's been added, but this is where I am having trouble... In this method I am looping through the allocations in the allocationQueue array. for each of the allocations I wish to make a new api call which will return the user that this allocation belongs to, but the problem is that the axios request inside the for loop is not firing.
Am i handling the async functions correctly?
const axios = require('axios')
var currentSandboxId = '1';
var allocationQueue = []
const setCurrentSandboxId = async (sbId) => {
currentSandboxId = sbId;
const response = await axios.get(`http://localhost:3000/allocations/all-for-sandbox/${currentSandboxId}`)
allocationQueue = response.data
await populateAllocationQueue()
}
const addAllocationToQueue = async () => {
const response = await axios.post('http://localhost:3000/allocations/', {
sandboxId: currentSandboxId,
branchName: 'test-branch',
userId: '123456789'
})
await populateAllocationQueue()
}
const populateAllocationQueue = async () => {
for(let j = 0; j < allocationQueue.length; j++) {
let allocation = allocationQueue[j];
//some DOM manipulation
const res = await axios.get('http://localhost:3000/users', { _id: allocation.createdBy })
console.log('Never gets called - seems to hang on the above request without error message')
}
}
setCurrentSandboxId('6');

How to query SQL in a loop in node js

SCENARIO:
Using mssql I'm connecting to sql and retrieving a list of ids, then based on those id I want to run stored procedures. What I'm currently doing is running the first stored proc, storing the id's in an array, then I'm running a for loop calling another module, where I pass the id to run a stored proc. This works fine when I've got a single id, but fails with 'Global connection already exists. Call sql.close() first.' when I try to run multiple ones.
How do I create connect to sql, run my query, then run the next one? What's the best approach?
The code below runs the stored proc with ids and causes the above error.
exports.runStoredProc = function (query,id) {
sql.connect(config.config).then(()=>{
return sql.query`${query} ${id}`
}).then(res=> {
do something with the response
}).catch(error => {
console.log(error)
})
}
Looks like the connection still exists when the below bit of code runs it using next id. I thought that creating a Promise will force to await execution before it runs the above bit of code again?
let toRun = result.recordset.length
let gen = async num => {
for(let i=0;i<num;i++) {
var resp = result.recordset[i].id
console.log(i, resp)
var sp = report
var reportId = await new Promise(() => db.runStoredProc(sp,resp))
}
}
gen(toRun).then(() => console.log("done!"))
You need to return Promise from runStoredProc
exports.runStoredProc = function (query,id) {
return sql.connect(config.config).then(()=>{
return sql.query`${query} ${id}`
}).then(res=> {
do something with the response
}).catch(error => {
console.log(error)
})
}
and no need to wrap db.runStoredProc in loop
let toRun = result.recordset.length
let gen = async num => {
for(let i=0;i<num;i++) {
var resp = result.recordset[i].id
console.log(i, resp)
var sp = report
var reportId = await db.runStoredProc(sp,resp)
}
}
gen(toRun).then(() => console.log("done!"))

node.js resolve promise and return value

I use the Microsoft bot framework to come up with a "simple" PoC bot. I used a tutorial as a basis and extend it.
I've a couple of basic functions for differet intents (ie. greetings, goodbye, etc) and one with some more logic in it (reqstatus).
The simple ones (ie greeting.js) return the answer nicely but the more complex one doesn't (reqstatus.js). Running the main code of reqstatus.js (without the first "const getReqStatus = (entity) => {") in a standalone script works.
server.js (main) -> see call in "if (intent) {"...
const getFeelings = require('./intents/feelings.js')
const getGoodbyes = require('./intents/goodbyes.js')
const getGreetings = require('./intents/greetings.js')
const getHelp = require('./intents/help.js')
const getReqStatus = require('./intents/reqstatus.js')
...
const bot = new builder.UniversalBot(connector)
// Intents based on definitions on recast
const INTENTS = {
feelings: getFeelings,
goodbyes: getGoodbyes,
greetings: getGreetings,
help: getHelp,
reqstatus: getReqStatus,
}
// Event when Message received
bot.dialog('/', (session) => {
recastClient.textRequest(session.message.text)
.then(res => {
const intent = res.intent()
const entity = res.get('request_number')
console.log(`UserName: ${session.message.user.name}`)
console.log(`Msg: ${session.message.text}`)
console.log(`Intent: ${intent.slug}`)
if (intent) {
INTENTS[intent.slug](entity)
.then(res => session.send(res))
.catch(err => session.send(err))
}
})
.catch(() => session.send('Sorry I didn\'t get that. '))
})
...
greetings.js -> Returns the string ok
const getGreetings = () => {
const answers = ['Hi, my name is SuperBot. Nice to meet you!', ]
return Promise.resolve((answers))
}
module.exports = getGreetings
reqstatus.js -> Does not return anything
const getReqStatus = (entity) => {
var request = require('request');
var request_number = entity.toLowerCase()
var output = [];
// Processing
var lineReader = require('readline').createInterface({
input: fs.createReadStream('netreqs.csv')
});
lineReader.on('line', function (line) {
var jsonFromLine = {};
var lineSplit = line.split(';');
jsonFromLine.req = lineSplit[0];
jsonFromLine.req_count = lineSplit[1];
jsonFromLine.req_type = lineSplit[2];
//...
var req_lowever = jsonFromLine.req.toLowerCase()
if (req_lowever == request_number) {
output.push( `Your request ${jsonFromLine.req} was received`);
// simplified
}
});
// Output
lineReader.on('close', function (line) {
if (output == '') {
output.push( `I was not able to find a request like ${request_number}.`);
}
console.log(output); // list output
return Promise.resolve(output);
});
}
module.exports = getReqStatus
I also tried to put getReqStatus in a function but that also didn't work.
After a lot of trying and googling I'm still stuck and wanted to ask the experts here. Thanks a lot in advance.
I think that the problem is that your getReqStatus isn't really returning anything. In your example getGreetings function you're actually returning Promise.resolve(answers) as the return value of that function.
However, in your getReqStatus function, you just set up a listener lineReader close event:
lineReader.on('close', function (line) {
if (output == '') {
output.push( `I was not able to find a request like ${request_number}.`);
}
console.log(output); // list output
return Promise.resolve(output);
});
You're returning a Promise resolved inside the anonymous callback function you're passing to lineReader.on() as second parameter. That is not the return value from the getReqStatus function itself, so that getReqStatus is not returning anything, as expected.
The code of that function runs correctly as standalone code, as you say, just because it sets the listener properly and it does what it has to do. However, that code just doesn't return a Promise when wrapped in a function.
What you would need is to return a Promise that wraps the lineReader.on close handler, like:
function getReqStatus(){
//...code
return new Promise( function(resolve , reject ){
lineReader.on('close', function (line) {
if (output == '') {
output.push( `I was not able to find a request like ${request_number}.`);
}
console.log(output); // list output
return resolve(output);
});
});
}
I say would because I really don't know if this code will work, I don't have any kind of experience with the Microsoft Bot framework and not used at all with the readline module. However, even if this doesn't solve your problem, I hope it will help you a bit understanding why your function doesn't return a Promise and how could you fix it.

Categories