We Keep Coding

Why do I get a 400 error trying to get data from an html page using fetch API?

Hi I'm doing a coding challenge and I'm trying to fetch data from an html page but I keep getting the 400 error. I don't see where my syntax would be wrong. The Promise returns an empty string as PromiseResult. Why do I not get the data from https://adventofcode.com/2021/day/2/input?
fetch('https://adventofcode.com/2021/day/2/input', {
method: 'GET',
mode: 'no-cors',
credentials: 'omit',
headers: {
'Content-Type': 'text/plain'
}
})
.then((response) => {
response.text();
})
.then((html) => {
var parser = new DOMParser();
var doc = parser.parseFromString(html, 'text/html');
})
.catch((err) => { console.log(err) })

After visting https://adventofcode.com/2021/day/2/input I get information that
Puzzle inputs differ by user. Please log in to get your puzzle input.
Please check if you are providing your login data correctly.

mode: 'no-cors',
This says that you are not going to do anything that requires permission to be granted using CORS. Anything that does require CORS will be quietly ignored.
You need CORS permission to read data across origins. Since you have said you aren't going to use CORS, you don't have permission, so you don't get any data.
This is why you get an empty string.
credentials: 'omit',
The input endpoint requires your credentials so that it can give you your input, which is unique to each user. Since told the browser not to send any, the end point doesn't understand the request.
As an aside:
headers: {
'Content-Type': 'text/plain'
}
This is just nonsense.
It claims that the request includes a body consisting of plain text.
You are making a GET request. There is no body on the request at all.
Advent of code expects you to manually download your input data. It doesn't expect your solution to fetch it from the AoC website.
JS solutions are generally run in Node.js (rather than a web browser) where they can use the fs module to read the local copy of the input file. (Tip: I find it handy to copy/paste the sample data into a sample file to test my results against the worked example on each day).

Node.js Express send huge data to client vanilla JS

In my application I read huge data of images, and send the whole data to the client:
const imagesPaths = await getFolderImagesRecursive(req.body.rootPath);
const dataToReturn = await Promise.all(imagesPaths.map((imagePath) => new Promise(async (resolve, reject) => {
try {
const imageB64 = await fs.readFile(imagePath, 'base64');
return resolve({
filename: imagePath,
imageData: imageB64,
});
} catch {
return reject();
}
})));
return res.status(200).send({
success: true,
message: 'Successfully retreived folder images data',
data: dataToReturn,
});
Here is the client side:
const getFolderImages = (rootPath) => {
return fetch('api/getFolderImages', {
method: 'POST',
headers: { 'Content-type': 'application/json' },
body: JSON.stringify({ rootPath }),
});
};
const getFolderImagesServerResponse = await getFolderImages(rootPath);
const getFolderImagesServerData = await getFolderImagesServerResponse.json();
When I do send the data I get failure due to the huge data. Sending the data just with res.send(<data>) is impossible. So, then, how can I bypass this limitation - and how should I accept the data in the client side with the new process?

The answer to your problem requires some read :
Link to the solution
One thing you probably haven’t taken full advantage of before is that webserver’s http response is a stream by default.
They just make it easier for you to pass in synchron data, which is parsed to chunks under the hood and sent as HTTP packages.
We are talking about huge files here; naturally, we don’t want them to be stored in any memory, at least not the whole blob. The excellent solution for this dilemma is a stream.
We create a readstream with the help of the built-in node package ‘fs,’ then pass it to the stream compatible response.send parameter.
const readStream = fs.createReadStream('example.png');
return response.headers({
'Content-Type': 'image/png',
'Content-Disposition': 'attachment; filename="example.png"',
}).send(readStream);
I used Fastify webserver here, but it should work similarly with Koa or Express.
There are two more configurations here: naming the header ‘Content-Type’ and ‘Content-Disposition.’
The first one indicates the type of blob we are sending chunk-by-chunk, so the frontend will automatically give the extension to it.
The latter tells the browser that we are sending an attachment, not something renderable, like an HTML page or a script. This will trigger the browser’s download functionality, which is widely supported. The filename parameter is the download name of the content.
Here we are; we accomplished minimal memory stress, minimal coding, and minimal error opportunities.
One thing we haven’t mentioned yet is authentication.
For the fact, that the frontend won’t send an Ajax request, we can’t expect auth JWT header to be present on the request.
Here we will take the good old cookie auth approach. Cookies are set automatically on every request header that matches the criteria, based on the cookie options. More info about this in the frontend implementation part.
By default, cookies arrive as semicolon separated key-value pairs, in a single string. In order to ease out the parsing part, we will use Fastify’s Cookieparser plugin.
await fastifyServer.register(cookieParser);
Later in the handler method, we simply get the cookie that we are interested in and compare it to the expected value. Here I used only strings as auth-tokens; this should be replaced with some sort of hashing and comparing algorithm.
const cookies = request.cookies;
if (cookies['auth'] !== 'authenticated') {
throw new APIError(400, 'Unauthorized');
}
That’s it. We have authentication on top of the file streaming endpoint, and everything is ready to be connected by the frontend.

How to use the RapidAPI in Excel VBA

I'm new to using RapidAPI. I want to pull live Cricket Scores from RapidAPI with Excel VBA, but the programming language isn't available on the platform.
I would like to know if there is anyway I can view the json results directly through a browser. This is what I read in their documentation https://docs.rapidapi.com/docs/getting-started-with-rapidapi-sdks but doesn't seem to solve my problem
"What If the Programming Language I'm Using Isn't Available?
Whether the programming language you're using isn't available or you prefer to use another request library, have no fear! All of the APIs available on RapidAPI are exposed through a REST API endpoint. All you'll need to do is take the information provided on the documentation provided. Here's an example of what that would look like using the 'request' npm module"
var options = {
method: 'POST',
url: API_URL,
headers: {
'cache-control': 'no-cache',
'Content-Type': 'application/json',
'X-RapidAPI-Key': API_KEY,
'header1': 'header-value-1'
},
qs: {
parameter1: 'parameter-value-1'
}
};
request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(body);
});
I'm looking for a better idea of working with RapidAPI in Excel VBA. If I can get the json response directly in a browser that solves my problem. But if that's not possible, any example to setup a Node.JS server offline and possibly display the json response in a browser may also work. Thanks in advance

Const Covid19URL As String = "https://covid-193.p.rapidapi.com/history?day=2020-03-25&country=Malaysia"
Const APIKey As String = "brx2HKOJvP6iQ14WxeumLmnhx2L2MAZz"
Sub Main()
Dim xmlhttp As New MSXML2.XMLHTTP60
Dim xmlresponse As New DOMDocument60
xmlhttp.Open "GET", Covid19URL, False
xmlhttp.setRequestHeader "x-rapidapi-host", "covid-193.p.rapidapi.com"
xmlhttp.setRequestHeader "x-rapidapi-key", APIKey
xmlhttp.send
Cells(2, 12).Value = xmlhttp.responseText
'xmlresponse.LoadXML (xmlhttp.responseText)
'"response":[
'{"country":"Malaysia","cases":{"new":"+172","active":1596,"critical":64,"recovered":183,"total":1796},"deaths":{"new":"+1","total":17},"day":"2020-03-25","time":"2020-03-25T09:15:07+00:00"},
'{"country":"Malaysia","cases":{"new":"+172","active":1597,"critical":64,"recovered":183,"total":1796},"deaths":{"new":null,"total":16},"day":"2020-03-25","time":"2020-03-25T07:15:06+00:00"},
'{"country":"Malaysia","cases":{"new":"+106","active":1425,"critical":64,"recovered":183,"total":1624},"deaths":{"new":"+2","total":16},"day":"2020-03-25","time":"2020-03-25T06:15:05+00:00"}
Set xmlresponse = Nothing
Set xmlhttp = Nothing
End Sub

Send Array in post request using node.js using application/x-www-form-urlencoded

I tried to send post request to API and the post parameters should be array,
this is how to send it in cURL
curl http://localhost:3000/check_amounts
-d amounts[]=15 \
-d amounts[]=30
I tried to do that in Node.js using request module
request.post('http://localhost:3000/check_amounts', {
form: {
'amounts[]': 15 ,
'amounts[]': 30
}
}, function(error, response, body) {
console.log(body)
res.json(body);
});
but the second amount override the first one and the API gets the result as following: amounts = [30]
Then I tried to send it in different way
request.post('http://localhost:3000/check_amounts', {
form: {
'amounts[]': [ 15 , 30]
}
}, function(error, response, body) {
console.log(body)
res.json(body);
});
but the result was not as an expected amounts = [{"0":15},{"1":30}]
Note : the header should contains 'Content-Type': 'application/x-www-form-urlencoded' not 'application/json'
Does any one have solution to this problem?

It's quite easy if you read the manual of request. All you should do is replace the form by querystring rather than object, in your case this should be:
amounts=15&amounts=30
the only thing I'm not sure is the above expression works in your web server. As I know it works well in java struts. So if not you may try
amounts[]=15&amounts[]=30 instead. Hope it help.

How to insert couchdb documents in bulk?

I'm tasked with modifying a legacy app so that users can upload payroll adjustments in bulk. Currently they have to fill out a form and input the data item by item, hitting submit after each one. I'm giving them the ability to upload a CSV file containing tons of adjustments at once.
On the server they are inserting items into couch one by one, like this:
function wsapiPOST(req, res, next) {
var path = req.path.substr(6)
, url = couchPath + path + requestUtil.buildQueryString(req);
request.post({
url: url,
headers: { 'content-type': 'application/json' },
body: JSON.stringify(req.body)
},function (err, resp, body) {
if (err) {
if (resp) {
res.writeHead(resp.statusCode);
res.end(body);
} else { // This would happen if the request timed out
res.writeHead(408);
res.end('timeout');
}
}
}).pipe(res);
}
The couch URL is built dynamically.
req.body contains the properties for a single item.
I'm new to couch but I'm not sure how to send multiple documents for insertion in a single operation. I could throw the request.post call into a loop as is, but I imagine that's not going to be very performant.
I just need pointed in the right direction for bulk insertion into couch via its REST API. Thanks!

You can use the bulk document API to insert (and even update) multiple documents.

You can use nano - there is a bulk insert option.