We Keep Coding

How to do find all or get all in dynamo db in NodeJS

I want to get all the data from a table in Dynamo DB in nodejs, this is my code
const READ = async (payload) => {
const params = {
TableName: payload.TableName,
};
let scanResults = [];
let items;
do {
items = await dbClient.scan(params).promise();
items.Items.forEach((item) => scanResults.push(item));
params.ExclusiveStartKey = items.LastEvaluatedKey;
} while (typeof items.LastEvaluatedKey != "undefined");
return scanResults;
};
I implemented this and this is working fine, but our code review tool is flagging red that this is not optimized or causing some memory leak, I just cannot figure out why, I have read somewhere else that scanning API from dynamo DB is not the most efficient way to get all data in node or is there something else that I am missing to optimize this code

DO LIKE THIS ONLY IF YOUR DATA SIZE IS VERY LESS (less than 100 items or data size less than 1MB, that's I prefer and in that case you don't need a do-while loop)
Think about the following scenario, What about in case in future, more and more items will add in to DynamoDB table? - This will return all your data and put into the scanResults variable right? This will impact the memory. Also, DynamoDB scan operation is expensive - in terms of both memory and cost
It's perfectly okay to use SCAN operation if the data is very less. Otherwise, go with pagination (I always prefer this). If there are 1000's of items, then who will look in to all these in a single shot? So use pagination instead.
Lets take another scenario, If your requirement is to retrieve all the data for doing some analytics or aggregation. Then better store the aggregate data upfront into the table (same or different DynamoDB table) as an item or use some analytics database.
If your requirement is something else, elaborate it in the question.

API can't handle my request because of template literals to make the API dynamic

For a school project, I have to make a quiz app. It is possible to chose a difficulty, a category and an amount of desired questions. The api is a url which can be modified easily by changing some values. For example: https://quizapi.io/api/v1/questions?apiKey=MYAPIKEY&limit=15&difficulty=hard&category=cms. If you would just change the php to code in the url, you would get a max amount of 15 questions on a hard difficulty about HTML and CSS. I think you see where this is going.
However. I have setup my code that the difficulty, category and amount are stored in localstorage and they are fetched when the quiz is started. At the moment, I get the amout of questions I desire but I can't change my difficulty or category because probably Template Literals aren't working in a fetch api.. Maybe someone can give me an idea or maybe I'm making a mistake in my current code
let storageDif = localStorage.getItem("mD");
console.log(storageDif.toString());
let storageCat = localStorage.getItem("mC");
console.log(storageCat);
let geslideVragen = localStorage.getItem("slider");
let MAX_VRAGEN = geslideVragen;
console.log(MAX_VRAGEN);
let vragen = [];
fetch(`https://quizapi.io/api/v1/questions?apiKey=kAFKilHLeEcfLkGE2H0Ia9uTIp1rYHDTIYIHs9qf&limit=15&difficulty=hard&category=${storageCat}`)
.then((res) => {
return res.json();
})
.then((loadedQuestions) => {
for (let i = 0; i < MAX_VRAGEN; i++) {
vragen = loadedQuestions;
console.log(vragen[i].question);
};
startGame();
})
.catch( err => {
console.error(err);
});

I'm sure you found out by now that you're only interpolating the category. To get it to be correctly, you'd need to do this:
`https://quizapi.io/api/v1/questions?apiKey=kAFKilHLeEcfLkGE2H0Ia9uTIp1rYHDTIYIHs9qf&limit=${MAX_VRAGEN}&difficulty=${storageDif}&category=${storageCat}`
That being said, you should never expose your API keys this way, because especially for cloud services, it can easily cost you over 5 digits in a single day if someone decided to use it for their own means. There are plenty of scrapers that scour GitHub for exposed API keys for illegitimate uses.
Also, should apply a check to make sure all values are present using an if() statement so that it doesn't fetch anything if a value is undefined.

How I can calculate page fully load time with Pupppeteer?

I'm trying to get the page fully load time in seconds with puppeteer in Node, for this I do some research on the API and other questions and create the following code:
/* First Configuration */
puppeteer.launch({
defaultViewport: { width: 1600, height: 800 }
}).then(async browser => {
const page = await browser.newPage();
await page.setCacheEnabled(false);
await page.goto('https://stackoverflow.com', {waitUntil: 'networkidle0'});
/* Get Page Metrics */
const perf = await page.metrics();
console.log(JSON.stringify(perf));
/* Get Page Evaluate */
const timing = await page.evaluate(() => {
const result = {};
for (const key of Object.keys(window.performance.timing.__proto__))
result[key] = window.performance.timing[key];
return result;
});
console.log(JSON.stringify(timing));
/* Show Results on Browser Close */
await browser.close().then(() => {
var fullyLoadEvaluate = (timing.loadEventEnd - timing.navigationStart);
console.log('Fully Load Time (Page Evaluate): ' + fullyLoadEvaluate);
var fullyLoadMetrics = (perf.LayoutDuration + perf.RecalcStyleDuration + perf.ScriptDuration + perf.TaskDuration);
console.log('Fully Load Time (Page Metrics): ' + fullyLoadMetrics);
/* Send Response to Server */
res.send('Check The Console');
});
});
Basically I use two codes to return metrics, One of them is page.metrics() that return the following data:
{"Timestamp":961736.600171,"Documents":8,"Frames":4,"JSEventListeners":375,"Nodes":8654,"LayoutCount":27,"RecalcStyleCount":31,"LayoutDuration":0.705517,"RecalcStyleDuration":0.144379,"ScriptDuration":0.527385,"TaskDuration":1.812213,"JSHeapUsedSize":11082496,"JSHeapTotalSize":20344832}
And the last one page.evaluate(), return the following:
{"navigationStart":1556722407938,"unloadEventStart":0,"unloadEventEnd":0,"redirectStart":0,"redirectEnd":0,"fetchStart":1556722407938,"domainLookupStart":1556722408247,"domainLookupEnd":1556722408548,"connectStart":1556722408548,"connectEnd":1556722408737,"secureConnectionStart":1556722408574,"requestStart":1556722408738,"responseStart":1556722408940,"responseEnd":1556722409087,"domLoading":1556722408957,"domInteractive":1556722409995,"domContentLoadedEventStart":1556722409995,"domContentLoadedEventEnd":1556722410190,"domComplete":1556722412584,"loadEventStart":1556722412584,"loadEventEnd":1556722412589,"toJSON":{}}
In my example I'm testing the site https://stackoverflow.com. Like webpagetest.org and getmetrix.com, I'm trying to get Page Fully Load Time.
I know this kind of value is inconsistent, but I wonder if the values ​​I'm calculating are right, and which of the two results seems to be more correct ? Fully Load Time (Page Evaluate) or Fully Load Time (Page Metrics) ?

You can use page.metrics() to compare two points in time (e.g. before and after page.goto). The page.evaluate approach to read the data from the performance API is also a good alternative. As I already pointed out in the comment, it is not defined what should be considered a "full page load". Both approaches are valid.
It's even more complex
There are a number of thing which people might consider a page to be loaded:
DOMContentLoaded event fired
Load event fired
Time it takes from navigation start until all resources embedded in the document (like images are loaded)
Time it takes from navigation start until all resources are loaded
Time until there are not more ongoing network requests.
...
You also have to consider whether whether you want network related phases (like DNS) to be part of the measurement. Here is an example request (generated with the Chrome DevTools Network tab) showing how complex a single request might be:
There is also a document explaining each of these phases.
Simple approach
The simplest way to measure the load time would just to start measuring when the navigaiton starts and stop measuring after the page is loaded. This could be done like this:
const t1 = Date.now();
await page.goto('https://example.com');
const diff1 = Date.now() - t1;
console.log(`Time: ${diff1}ms`);
Note that there are also other APIs (page.metrics, process.hrtime, perf_hooks) to get more precise timestamps.
You can also pass options to the page.goto function to change the resolving of the promise to something like this (quoted from the docs):
Consider navigation to be finished when there are no more than 0 network connections for at least 500ms
For that, you would have to use the setting networkidle0:
await page.goto('https://example.com', { waitUntil: 'networkidle0' });
There are also other events in the docs linked above you could use.
More complex: Use the Performance API
To get more precise results, you can use the Performance API as you already did in your code. Instead of going through the prototype of window.performance you can also use the functions performance.getEntries() or performance.toJSON() like this:
const perfData = await page.evaluate(() =>
JSON.stringify(performance.toJSON(), null, 2)
);
That way, you get data that looks like this:
{
"timeOrigin": 1556727036740.113,
"timing": {
"navigationStart": 1556727036740,
"unloadEventStart": 0,
"unloadEventEnd": 0,
"redirectStart": 0,
"redirectEnd": 0,
"fetchStart": 1556727037227,
"domainLookupStart": 1556727037230,
"domainLookupEnd": 1556727037280,
"connectStart": 1556727037280,
"connectEnd": 1556727037348,
"secureConnectionStart": 1556727037295,
"requestStart": 1556727037349,
"responseStart": 1556727037548,
"responseEnd": 1556727037805,
"domLoading": 1556727037566,
"domInteractive": 1556727038555,
"domContentLoadedEventStart": 1556727038555,
"domContentLoadedEventEnd": 1556727038570,
"domComplete": 1556727039073,
"loadEventStart": 1556727039073,
"loadEventEnd": 1556727039085
},
"navigation": {
"type": 0,
"redirectCount": 0
}
}
So if you want to know how long it took from navigationStart to loadEventStart you subtract one value from the other one (e.g. 1556727039073 - 1556727036740 = 2333 ms).
So which one to take?
This is up to your decision. In general, it is a good idea to use the Load event as a starting point. Waiting until all requests are finished might actually never happen because there are constantly resources being loaded in the background. Using networkidle2 as waitUntil option might be an alternative in case you don't want to use the load event.
In the end, however, it comes down to your use case which metric to use.

Insert multiple entries to SQL Server with Node.js

I am rewriting an old API for which I am trying to insert multiple values at once into a MSSQL-Server (2008) database using the node module mssql. Now, I am capable of doing this somehow, but I want to this following best practices. I've done my research and tried a lot of things to accomplish my target. However, I was not able to find a single solution which works just right.
Before
You may wonder:
Well, you are rewriting this API, so there must be a way this has been done before and that was working?
Sure, you're right, it was working before, but... not in a way I'd feel comfortable with using in the rewrite. Let me show you how it was done before (little bit of abstraction added of course):
const request = new sql.Request(connection);
let query = "INSERT INTO tbl (col1, col2, col3, col4) VALUES ";
for (/*basic for loop w/ counter variable i*/) {
query += "(1, #col2" + [i] + ", #col3" + [i] + ", (SELECT x FROM y WHERE z = #someParam" + [i] + "))";
// a check whether to add a comma or not
request.input("col2" + [i], sql.Int(), values[i]);
// ...
}
request.query(query, function(err, recordset) {
// ...
}
While this is working, again, I don't quite think this could be called anything like 'best practice'. Also this shows the biggest problem: a subselect is used to insert a value.
What I tried so far
The easy way
At first I tried the probably easiest thing:
// simplified
const sQuery = "INSERT INTO tbl (col1, col2, col3, col4) VALUES (1, #col2, #col3, (SELECT x FROM y WHERE z = #col4));";
oPool.request().then(oRequest => {
return oRequest
.input("col2", sql.Int(), aValues.map(oValue => oValue.col2))
.input("col3", sql.Int(), aValues.map(oValue => oValue.col3))
.input("col4", sql.Int(), aValues.map(oValue => oValue.col4))
.query(sQuery);
});
I'd say, this was a pretty good guess and actually working relative fine.
Except for the part, that ignores every item after the first one... which makes this pretty useless. So, I tried...
Request.multiple = true
...and I thought, it would do the job. But - surprise - it doesn't, still only the first item is inserted.
Using '?' for parameters
At this point I really started the search for a solution, as the second one was only a quick search in the modules documentation.
I stumbled upon this answer and tried it immediately.
Didn't take long for my terminal to spit out a
RequestError: Incorrect syntax near '?'.
So much for that.
Bulk inserting
Some further research led to bulk inserting.
Pretty interesting, cool feature and excellent updating of the question with the solution by the OP!
I had some struggle getting started here, but eventually it looked really good: Multiple records were inserted and the values seemed okay.
Until I added the subquery. Using it as value for a column declared didn't cause any error, however when checking the values of the table, it simply displayed a 0 as value for this column. Not a big surprise at all, but everybody can dream, right?
The lazy way
I don't really know what to think about this:
// simplified
Promise.all(aValues.map(oValue => {
return oPool.request().then(oRequest =>
oRequest
.input("col2", sql.Int, oValue.col2)
.input("col3", sql.Int, oValue.col3)
.input("col4", sql.Int, oValue.col4)
.query(sQuery);
});
});
It does the job, but if any of the request fails for whichever reason, the other, non-failing inserts, will still be executed, even though this should not be possible.
Lazy + Transaction
As continuing even if some fail was the major problem with the last method, I tried building a transaction around it. All querys are successful? Good, commit. Any query has an errpr? Well, just rollback than. So I build a transaction, moved my Promise.all construct into it and tried again.
Aaand the next error pops up in my terminal:
TransactionError: Can't acquire connection for the request. There is another request in progress.
If you came this far, I don't need to tell you what the problem is.
Summary
What I didn't try yet (and I don't think I will try this) is using the transaction way and calling the statements sequentially. I do not believe that this is be the way to go.
And I also don't think the lazy way is the one that should be used, as it uses single requests for every record to insert, when this could somehow be done using only one request. It's just that this somehow is, I don't know, not in my head right now. So please, if you have anything that could help me, tell me.
Also, if you see anything else that's wrong with my code, feel free to point it out. I am not considering myself as a beginner, but I also don't think that learning will ever end. :)

The way I solved this was using PQueue library with concurrency 1. Its slow due to concurrency of one but it works with thousands of queries:
const transaction = sql.transaction();
const request = transaction.request();
const queue = new PQueue({ concurrency: 1 });
// being transaction
await transaction.begin();
for (const query of queries) {
queue.add(async () => {
try {
await request.query(query);
} catch (err) {
// stop pending transactions
await queue.clear();
await queue.onIdle();
// rollback transaction
await transaction.rollback();
// throw error
throw err;
}
});
}
// await queue
await queue.onIdle();
// comit transaction
await transaction.commit();

JSON.stringify is very slow for large objects

I have a very big object in javascript (about 10MB).
And when I stringify it, it takes a long time, so I send it to backend and parse it to an object( actually nested objects with arrays), and that takes long time too but it's not our problem in this question.
The problem:
How can I make JSON.stringify faster, any ideas or alternatives, I need a javaScript solution, libraries I can use or ideas here.
What I've tried
I googled a lot and looks there is no better performance than JSON.stringify or my googling skills got rusty!
Result
I accept any suggestion that may solve me the long saving (sending to backend) in the request (I know its big request).
Code Sample of problem (details about problem)
Request URL:http://localhost:8081/systemName/controllerA/update.html;jsessionid=FB3848B6C0F4AD9873EA12DBE61E6008
Request Method:POST
Status Code:200 OK
Am sending a POST to backend and then in JAVA
request.getParameter("BigPostParameter")
and I read it to convert to object using
public boolean fromJSON(String string) {
if (string != null && !string.isEmpty()) {
ObjectMapper json = new ObjectMapper();
DateFormat dateFormat = new SimpleDateFormat(YYYY_MM_DD_T_HH_MM_SS_SSS_Z);
dateFormat.setTimeZone(TimeZone.getDefault());
json.setDateFormat(dateFormat);
json.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);
WebObject object;
// Logger.getLogger("JSON Tracker").log(Level.SEVERE, "Start");
try {
object = json.readValue(string, this.getClass());
} catch (IOException ex) {
Logger.getLogger(JSON_ERROR).log(Level.SEVERE, "JSON Error: {0}", ex.getMessage());
return false;
}
// Logger.getLogger("JSON Tracker").log(Level.SEVERE, "END");
return this.setThis(object);
}
return false;
}
Like This
BigObject someObj = new BigObject();
someObj.fromJSON(request.getParameter("BigPostParameter"))
P.S : FYI this line object = json.readValue(string, this.getClass());
is also very very very slow.
Again to summarize
Problem in posting time (stringify) JavaScript bottle nick.
Another problem parsing that stringified into an object (using jackson), and mainly I have svg tags content in that stringified object as a style column, and other columns are strings, int mainly

As commenters said - there is no way to make parsing faster.
If the concern is that the app is blocked while it's stringifying/parsing then try to split data into separate objects, stringily them and assemble back into one object before saving on the server.
If loading time of the app is not a problem you could try to ad-hoc incremental change on top of the existing app.
... App loading
Load map data
Make full copy of the data
... End loading
... App working without changes
... When saving changes
diff copy with changed data to get JSON diff
send changes (much smaller then full data)
... On server
apply JSON diff changes on the server to the full data stored on server
save changed data
I used json-diff https://github.com/andreyvit/json-diff to calc changes, and there are few analogs.

Parsing is a slow process. If what you want is to POST a 10MB object, turn it into a file, a blob, or a buffer. Send that file/blob/buffer using formdata instead of application/json and application/x-www-form-urlencoded.
Reference
An example using express/multer

Solution
Well just as most big "repeatable" problems go, you could use async!
But wait, isn't JS still single-threaded even when it does async... yes... but you can use Service-Workers to get true async and serialize an object way faster by parallelizing the process.
General Approach
mainPage.js
//= Functions / Classes =============================================================|
// To tell JSON stringify that this is already processed, don't touch
class SerializedChunk {
constructor(data){this.data = data}
toJSON() {return this.data}
}
// Attach all events and props we need on workers to handle this use case
const mapCommonBindings = w => {
w.addEventListener('message', e => w._res(e.data), false)
w.addEventListener('error', e => w._rej(e.data), false)
w.solve = obj => {
w._state && await w._state.catch(_=>_) // Wait for any older tasks to complete if there is another queued
w._state = new Promise((_res, _rej) => {
// Give this object promise bindings that can be handled by the event bindings
// (just make sure not to fire 2 errors or 2 messages at the same time)
Object.assign(w, {_res, _rej})
})
w.postMessage(obj)
return await w._state // Return the final output, when we get the `message` event
}
}
//= Initialization ===================================================================|
// Let's make our 10 workers
const workers = Array(10).fill(0).map(_ => new Worker('worker.js'))
workers.forEach(mapCommonBindings)
// A helper function that schedules workers in a round-robin
workers.schedule = async task => {
workers._c = ((workers._c || -1) + 1) % workers.length
const worker = workers[workers._c]
return await worker.solve(task)
}
// A helper used below that takes an object key, value pair and uses a worker to solve it
const _asyncHandleValuePair = async ([key, value]) => [key, new SerializedChunk(
await workers.schedule(value)
)]
//= Final Function ===================================================================|
// The new function (You could improve the runtime by changing how this function schedules tasks)
// Note! This is async now, obviously
const jsonStringifyThreaded = async o => {
const f_pairs = await Promise.all(Object.entries(o).map(_asyncHandleValuePair))
// Take all final processed pairs, create a new object, JSON stringify top level
final = f_pairs.reduce((o, ([key, chunk]) => (
o[key] = chunk, // Add current key / chunk to object
o // Return the object to next reduce
), {}) // Seed empty object that will contain all the data
return JSON.stringify(final)
}
/* lot of other code, till the function that actually uses this code */
async function submitter() {
// other stuff
const payload = await jsonStringifyThreaded(input.value)
await server.send(payload)
console.log('Done!')
}
worker.js
self.addEventListener('message', function(e) {
const obj = e.data
self.postMessage(JSON.stringify(obj))
}, false)
Notes:
This works the following way:
Creates a list of 10 workers, and adds a few methods and props to them
We care about async .solve(Object): String which solves our tasks using promises while masking away callback hell
Use a new method: async jsonStringifyThreaded(Object): String which does the JSON.stringify asynchronously
We break the object into entries and solve each one parallelly (this can be optimized to be recursive to a certain depth, use best judgement :))
Processed chunks are cast into SerializedChunk which the JSON.stringify will use as is, and not try to process (since it has .toJSON())
Internally if the number of keys exceeds the workers, we round-robin back to the first worker and overschedule them (remember, they can handle queued tasks)
Optimizations
You may want to consider a few more things to improve performance:
Use of Transferable Objects which will decrease the overhead of passing objects to service workers significantly
Redesign jsonStringifyThreaded() to schedule more objects at deeper levels.

You can explore libraries like fast-json-stringify which use a template schema and use it while converting the json object, to boost the performance. Check the below article.
https://developpaper.com/how-to-improve-the-performance-of-json-stringify/