Nodeschool's Stream-Adventure #12 Duplexer redux - javascript

It's been days since I'm trying several ways to solve this exercice. I give up. What am I doing wrong here ? Why am I getting "write after end" error if the event I'm using is the "end" event and I don't write anything to the pipe ? I'm just changing an object after it's use. Or at least I think so.
var duplexer2 = require("duplexer2");
var through = require('through2');
module.exports = function (counter) {
var countries = {};
var duplex = duplexer2(through.obj(function (obj, encoding, done) {
if (obj.country in countries)
countries[obj.country]++;
else
countries[obj.country] = 1;
done();
}), counter);
duplex.on("finish", function() {
counter.setCounts(countries);
});
counter.pipe(duplex);
return duplex;
};
If I substitute the line counter.setCounts(countries); for console.log(countries) I see that it's populated correctly.
Problem text: http://pastebin.com/vAM4vKZg
I have read this exercice test file and counldn't get any clue from it as it only compares the objects to see if they're correct.

Your solution is almost correct. Just remove the counter.pipe(duplex); line and it works. You do not need to pipe anything to the duplex stream you created, the stream adventure is going to do that to check if it works :).

Related

Unable to add async / await and then unable to export variable. Any help appreciated

Background: Been trying for the last 2 day to resolve this myself by looking at various examples from both this website and others and I'm still not getting it. Whenever I try adding callbacks or async/await I'm getting no where. I know this is where my problem is but I can't resolve it myself.
I'm not from a programming background :( Im sure its a quick fix for the average programmer, I am well below that level.
When I console.log(final) within the 'ready' block it works as it should, when I escape that block the output is 'undefined' if console.log(final) -or- Get req/server info, if I use console.log(ready)
const request = require('request');
const ready =
// I know 'request' is deprecated, but given my struggle with async/await (+ callbacks) in general, when I tried switching to axios I found it more confusing.
request({url: 'https://www.website.com', json: true}, function(err, res, returnedData) {
if (err) {
throw err;
}
var filter = returnedData.result.map(entry => entry.instrument_name);
var str = filter.toString();
var addToStr = str.split(",").map(function(a) { return `"trades.` + a + `.raw", `; }).join("");
var neater = addToStr.substr(0, addToStr.length-2);
var final = "[" + neater + "]";
// * * * Below works here but not outside this block* * *
// console.log(final);
});
// console.log(final);
// returns 'final is not defined'
console.log(ready);
// returns server info of GET req endpoint. This is as it is returning before actually returning the data. Not done as async.
module.exports = ready;
Below is an short example of the JSON that is returned by website.com. The actual call has 200+ 'result' objects.
What Im ultimately trying to achieve is
1) return all values of "instrument_name"
2) perform some manipulations (adding 'trades.' to the beginning of each value and '.raw' to the end of each value.
3) place these manipulations into an array.
["trades.BTC-26JUN20-8000-C.raw","trades.BTC-25SEP20-8000-C.raw"]
4) export/send this array to another file.
5) The array will be used as part of another request used in a websocket connection. The array cannot be hardcoded into this new request as the values of the array change daily.
{
"jsonrpc": "2.0",
"result": [
{
"kind": "option",
"is_active": true,
"instrument_name": "26JUN20-8000-C",
"expiration_timestamp": 1593158400000,
"creation_timestamp": 1575305837000,
"contract_size": 1,
},
{
"kind": "option",
"is_active": true,
"instrument_name": "25SEP20-8000-C",
"expiration_timestamp": 1601020800000,
"creation_timestamp": 1569484801000,
"contract_size": 1,
}
],
"usIn": 1591185090022084,
"usOut": 1591185090025382,
"usDiff": 3298,
"testnet": true
}
Looking your code we find two problems related to final and ready variables. The first one is that you're trying to console.log(final) out of its scope.
The second problem is that request doesn't immediately return the result of your API request. The reason is pretty simple, you're doing an asynchronous operation, and the result will only be returned by your callback. Your ready variable is just the reference to your request object.
I'm not sure about what is the context of your code and why you want to module.exports ready variable, but I suppose you want to export the result. If that's the case, I suggest you to return an async function which returns the response data instead of your request variable. This way you can control how to handle your response outside the module.
You can use the integrated fetch api instead of the deprecated request. I changed your code so that your component exports an asynchronous function called fetchData, which you can import somewhere and execute. It will return the result, updated with your logic:
module.exports = {
fetchData: async function fetchData() {
try {
const returnedData = await fetch({
url: "https://www.website.com/",
json: true
});
var ready = returnedData.result.map(entry => entry.instrument_name);
var str = filter.toString();
var addToStr = str
.split(",")
.map(function(a) {
return `"trades.` + a + `.raw", `;
})
.join("");
var neater = addToStr.substr(0, addToStr.length - 2);
return "[" + neater + "]";
} catch (error) {
console.error(error);
}
}
}
I hope this helps, otherwise please share more of your code. Much depends on where you want to display the fetched data. Also, how you take care of the loading and error states.
EDIT:
I can't get responses from this website, because you need an account as well as credentials for the api. Judging your code and your questions:
1) return all values of "instrument_name"
Your map function works:
var filter = returnedData.result.map(entry => entry.instrument_name);
2)perform some manipulations (adding 'trades.' to the beginning of each value and '.raw' to the end of each value.
3) place these manipulations into an array. ["trades.BTC-26JUN20-8000-C.raw","trades.BTC-25SEP20-8000-C.raw"]
This can be done using this function
const manipulatedData = filter.map(val => `trades.${val}.raw`);
You can now use manipulatedData in your next request. Being able to export this variable, depends on the component you use it in. To be honest, it sounds easier to me not to split this logic into two separate components - regarding the websocket -.

Any way to kill String.prototype.match after specified time passed? [duplicate]

Is it possible to cancel a regex.match operation if takes more than 10 seconds to complete?
I'm using an huge regex to match a specific text, and sometimes may work, and sometimes can fail...
regex: MINISTÉRIO(?:[^P]*(?:P(?!ÁG\s:\s\d+\/\d+)[^P]*)(?:[\s\S]*?))PÁG\s:\s+\d+\/(\d+)\b(?:\D*(?:(?!\1\/\1)\d\D*)*)\1\/\1(?:[^Z]*(?:Z(?!6:\s\d+)[^Z]*)(?:[\s\S]*?))Z6:\s+\d+
Working example: https://regex101.com/r/kU6rS5/1
So.. i want cancel the operation if takes more than 10 seconds. Is it possible? I'm not finding anything related in sof
Thanks.
You could spawn a child process that does the regex matching and kill it off if it hasn't completed in 10 seconds. Might be a bit overkill, but it should work.
fork is probably what you should use, if you go down this road.
If you'll forgive my non-pure functions, this code would demonstrate the gist of how you could communicate back and forth between the forked child process and your main process:
index.js
const { fork } = require('child_process');
const processPath = __dirname + '/regex-process.js';
const regexProcess = fork(processPath);
let received = null;
regexProcess.on('message', function(data) {
console.log('received message from child:', data);
clearTimeout(timeout);
received = data;
regexProcess.kill(); // or however you want to end it. just as an example.
// you have access to the regex data here.
// send to a callback, or resolve a promise with the value,
// so the original calling code can access it as well.
});
const timeoutInMs = 10000;
let timeout = setTimeout(() => {
if (!received) {
console.error('regexProcess is still running!');
regexProcess.kill(); // or however you want to shut it down.
}
}, timeoutInMs);
regexProcess.send('message to match against');
regex-process.js
function respond(data) {
process.send(data);
}
function handleMessage(data) {
console.log('handing message:', data);
// run your regex calculations in here
// then respond with the data when it's done.
// the following is just to emulate
// a synchronous computational delay
for (let i = 0; i < 500000000; i++) {
// spin!
}
respond('return regex process data in here');
}
process.on('message', handleMessage);
This might just end up masking the real problem, though. You may want to consider reworking your regex like other posters have suggested.
Another solution I found here:
https://www.josephkirwin.com/2016/03/12/nodejs_redos_mitigation/
Based on the use of VM, no process fork.
That's pretty.
const util = require('util');
const vm = require('vm');
var sandbox = {
regex:/^(A+)*B/,
string:"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC",
result: null
};
var context = vm.createContext(sandbox);
console.log('Sandbox initialized: ' + vm.isContext(sandbox));
var script = new vm.Script('result = regex.test(string);');
try{
// One could argue if a RegExp hasn't processed in a given time.
// then, its likely it will take exponential time.
script.runInContext(context, { timeout: 1000 }); // milliseconds
} catch(e){
console.log('ReDos occurred',e); // Take some remedial action here...
}
console.log(util.inspect(sandbox)); // Check the results

How do you prevent a NodeJS server from potentially exposing function code?

Let's imagine you're building a banking app backend. You want to respond to a user with a string that returns the balance but you forgot to add ().
class User {
constructor() {console.log("ctor")}
balance() { console.log("secret balance code")}
}
Then when referencing the user, instead of writing this:
const userA = new User();
return `Your balance is $${userA.balance()}`;
I accidentally write this:
const userA = new User();
return `Your balance is $${userA.balance}`;
Which sadly outputs:
'Your balance is balance() { console.log("secret balance code")}'
Which leaks the source code.
You do not need to worry about it, if you forget something, then testing will help to find it. Nobody deploy in production without testing when he has a serious project. It is better to write tests than to try to correct language behavior.
One workaround is to override all functions' toString like so:
> Function.prototype.toString = () => {return "bla"}
[Function]
> '' + new User().balance
'bla'
When responding to a request, you're undoubtedly going to be running the response through some sort of serializer. JSON, CBOR, etc. Handle it on that layer.
Fortunately for you, if you're returning JSON data, it's already handled:
JSON.stringify(someFunction);
// undefined
If you really are returning plain text strings, you can still have such a layer that ensures you're not putting out functions.
I've a solution which is definitely slower than raw templates, but here it goes.
So basically I just send a context object which has all the string I want to resolve. And before the actual string replacement, I just check for the types of arguments.
function resolveTemplates(str, args){
if(args && Array.isArray(args) && args.length){
args.forEach((argument) => {
// check here for any unwanted types
if(typeof arg === 'function'){
throw new Error('Cannot send function to create raw Strings')
}
})
}
const rx = /\{([^{}]*)\}/g;
let match = {};
let matches = [];
while(match = rx.exec(str)){
matches.push(match)
}
matches.reverse();
matches.forEach(function(match){
const key = match[1];
const index = match.index;
str = str.slice(0, index) + args[key] + str.slice(index + 2 + key.length)
})
return str;
}
resolveTemplates('Hello! My name is {firstName} {lastName}', {firstName: 'Shobhit', lastName: 'Chittora'})
PS: Instead of throwing errors for functions as arguments, you can call the functions. But binding the functions to the correct context can be a overhead to think about and generally not suggested.

indexeddb Invalid state error by call-stack?

Again, i got some question on indexeddb. I´m getting a
InvalidStateError: A Mutation operation was attempted on a database
that did not allow mutations.
and also an
AbortError
Here is my code:
DB_LINK.prototype.pushStoreNumeric = function ()
{
// Saving Values
var _temp = 0;
var _version = this.link.version;
var _name = this.link.name;
var that = this;
var _objectStoreNames = this.link.objectStoreNames;
// Close DB
this.link.close();
this.state = 4;
// Reopen Database
this.req = indexedDB.open(_name,_version+1); // Abort error here
this.req.onupgradeneeded = function () {
that.state = 1;
// Get Number of object stores
_temp = _objectStoreNames.length;
if(_temp != 0)
{
// Already object stores: read highest value
_temp = parseInt(_objectStoreNames[_objectStoreNames.length - 1]);
}
that.link.createObjectStore(_temp); // InvalidStateError here
};
I have marked per comment where the errors occur.
The InvalidStateError occures first, the AbortError follows.
I am calling this function inside another onsuccess function of the same database. Might this be the problem?
What is this.link? That's probably the problem. You need to be doing createObjectStore on the database instance created by the indexedDB.open request. So either this.req.result.createObjectStore or (if you change to this.req.onupgradeneeded = function (e) {) you could use e.target.result.createObjectStore.
More generally, I can't really comment on what your code is supposed to be doing because I can only see a snippet, but it looks really weird how you are incrementing the version every time this is called. Probably you don't actually want to be doing that. You might want to read a bit more documentation.

Attempt to use Promises to delete records

Here is a simple task I would like to accomplish on Parse.com with Cloud Code.
The task consists to delete a Unit and what is related to it.
One Unit has several Sentences related to it and each Sentence has one or more Translations.
So when the task is performed, the Unit as well as the Sentence and Translations should be deleted.
I have a strong feeling I should be using Promises (and chain them up) in order to do what I want in a good manner.
Below is the code I wrote, but it works only partially (The translations are deleted, not the rest).
Parse.Cloud.define("deleteUnitAndDependencies", function(request, response) {
var unitListQuery = new Parse.Query("UnitList");
unitListQuery.equalTo("objectId", request.params.unitID);
unitListQuery.equalTo("ownerID", request.params.userID);
unitListQuery.find().then(function(resUnit) {
var sentenceListQuery = new Parse.Query("SentenceList");
sentenceListQuery.equalTo("unit", resUnit[0]);
return sentenceListQuery.find();
}).then(function(resSentence) {
var translatListQuery = new Parse.Query("TranslatList");
for (i = 0; i < resSentence.length; i++) {
var query = new Parse.Query("TranslatList");
query.equalTo("sentence", resSentence[i]);
translatListQuery = Parse.Query.or(translatListQuery, query);
}
return translatListQuery.find();
}).then(function(resTranslat) {
for (iT = 0; iT < resTranslat.length; iT++) {
resTranslat[iT].destroy({});
}
});
});
I surely need to add some lines of code like:
resSentence[x].destroy({});
and:
resUnit[0].destroy({});
The problem is that I do not quite see where is the adequate place for that.
Collect the objects to be deleted then use Parse.Object.destroyAll(someArray); to delete all at once.
In cases like this I like to use a scope variable to hold things for later use.
var scope = {
sentences: [],
units: []
};
// later inside then block...
scope.sentences.push(resSentence[i]);
// ...now we have them collected safely
.then(function() {
return Parse.Object.destroyAll(scope.sentences);
})

Categories