Intercepting requests in Puppeteer - updated postData not being sent - javascript

So after solving yesterday's problem (linked below) another one has popped up.
Setting a FormData key/value on click with Puppeteer/NodeJS
The issue I'm having today is that I'm trying to modify an existing key in the FormData that's being sent to the server. The request gets intercepted correctly, I parse the FormData using the "qs" query string parsing package, then try to modify an existing key and use "NEW VALUE" in it's place. When I log it everything looks fine. However when I run chrome with headless=false and inspect the traffic I can see that the request is being sent without "NEW VALUE". It's as if I'm just calling continue() without passing any parameters in. I've attached the code below.
await this.page.setRequestInterception(true);
this.page.on("request", interceptedRequest => {
const formDataObj = qs.parse(interceptedRequest.postData());
if (interceptedRequest.url() === "https://www.someurl.com" && formDataObj.email) {
formDataObj.existingKey= "NEW VALUE";
const options = {
'method': 'POST',
'postData': qs.stringify(formDataObj),
'headers': {
...interceptedRequest.headers(),
'Content-Type': 'application/x-www-form-urlencoded'
},
};
console.log("Sending with new data");
console.log(qs.stringify(formDataObj));
interceptedRequest.continue(options);
} else {
interceptedRequest.continue();
}
Not sure where to go from here. I've done a lot of reading and tried some things like different versions of Puppeteer, passing ONLY the FormData (as a string) in to the continue() call and using some flags like "--enable-feature=NetworkService" (suggested on github) when launching chrome but nothing is working. Any help would be greatly appreciated!

Related

FastAPI returns "Error 422: Unprocessable entity" when I send multipart form data with JavaScript Fetch API

I have some issue with using Fetch API JavaScript method when sending some simple formData like so:
function register() {
var formData = new FormData();
var textInputName = document.getElementById('textInputName');
var sexButtonActive = document.querySelector('#buttonsMW > .btn.active');
var imagesInput = document.getElementById('imagesInput');
formData.append('name', textInputName.value);
if (sexButtonActive != null){
formData.append('sex', sexButtonActive.html())
} else {
formData.append('sex', "");
}
formData.append('images', imagesInput.files[0]);
fetch('/user/register', {
method: 'POST',
data: formData,
})
.then(response => response.json());
}
document.querySelector("form").addEventListener("submit", register);
And on the server side (FastAPI):
#app.post("/user/register", status_code=201)
def register_user(name: str = Form(...), sex: str = Form(...), images: List[UploadFile] = Form(...)):
try:
print(name)
print(sex)
print(images)
return "OK"
except Exception as err:
print(err)
print(traceback.format_exc())
return "Error"
After clicking on the submit button I get Error 422: Unprocessable entity. So, if I'm trying to add header Content-Type: multipart/form-data, it also doesn't help cause I get another Error 400: Bad Request. I want to understand what I am doing wrong, and how to process formData without such errors?
The 422 response body will contain an error message about which field(s) is missing or doesn’t match the expected format. Since you haven't provided that (please do so), my guess is that the error is triggered due to how you defined the images parameter in your endpoint. Since images is expected to be a List of File(s), you should instead define it using the File type instead of Form. For example:
images: List[UploadFile] = File(...)
^^^^
When using UploadFile, you don't have to use File() in the default value of the parameter. Hence, the below should also work:
images: List[UploadFile]
Additionally, in the frontend, make sure to use the body (not data) parameter in the fetch() function to pass the FormData object (see example in MDN Web Docs). For instance:
fetch('/user/register', {
method: 'POST',
body: formData,
})
.then(res => {...
Please have a look at this answer, as well as this answer, which provide working examples on how to upload multiple files and form data to a FastAPI backend, using Fetch API in the frontend.
As for manually specifying the Content-Type when sending multipart/form-data, you don't have to (and shouldn't) do that, but rather let the browser set the Content-Type—please take a look at this answer for more details.
So, I found that I has error in this part of code:
formData.append('images', imagesInput.files[0]);
Right way to upload multiple files is:
for (const image of imagesInput.files) {
formData.append('images', image);
}
Also, we should use File in FastAPI method arguments images: List[UploadFile] = File(...) (instead of Form) and change data to body in JS method. It's not an error, cause after method called, we get right type of data, for example:
Name: Bob
Sex: Man
Images: [<starlette.datastructures.UploadFile object at 0x7fe07abf04f0>]

HTTP POST to Google Sheets (via Apps Script) not importing JSON body message contents?

I've designed a web app that posts order information to Google Sheets via HTTP POST and Google Apps Script. However, I am struggling to POST and extract the order information from the body JSON. I have tried 2 different methods and they both output the similar information but with the same data type - FileUpload. I've even attempted to import it as parameters as FormData() but repeatedly appending parameters didn't feel quite right.
Method 1
function method1() {
var xml = new XMLHttpRequest();
var data = JSON.stringify({ firstName: "Jay", lastName: "Smith" });
xml.open("POST", url, true);
/* xml.setRequestHeader("Content-type", "application/json"); */
xml.onreadystatechange = function () {
if (xml.readyState === 4 && xml.status === 200) {
alert(xml.responseText);
}
};
xml.send(data);
}
Output
{postData=FileUpload, contextPath=, contentLength=38.0, parameters={}, parameter={}, queryString=}
Notice how the setRequestHeader() function is commented. No output would occur if it was uncommented making the content type set to JSON. Why will it not work?
Method 2
function method2() {
const body = {
firstName: "Jay",
lastName: "Smith",
};
const options = {
method: "POST",
body: JSON.stringify(body),
};
fetch(url, options).then((res) => res.json());
}
Output
{contextPath=, parameters={}, parameter={}, queryString=, postData=FileUpload, contentLength=38.0}
Method 3 (Should I even consider this?)
function method3() {
const formData = new FormData();
formData.append("firstName", "Jay");
formData.append("lastName", "Smith");
fetch(url, {
method: "POST",
body: formData,
}).then((res) => res.json());
}
Output
{contentLength=243.0, parameter={firstName=Jay, lastName=Smith}, queryString=, contextPath=, parameters={lastName=[Ljava.lang.Object;#6c4aece6, firstName=[Ljava.lang.Object;#3c461e46}}
The content length is considerably larger... this does not seem like an effective concept.
Google Apps Script doPost() Function
function doPost(e) {
const sheet = SpreadsheetApp.openById(SHEETID).getSheetByName('Testing Inputs');
sheet.getRange(3,1).setValue(e); // To display POST output
const data = JSON.parse(request.postData.contents)
sheet.getRange(5,1).setValue(data['firstName']) // Not writing "Jay"?
sheet.getRange(5,2).setValue(data['lastName']) // Not writing "Smith"?
}
In general, it appears data is coming through as body message content judging by the contentLength output but is it not JSON considering the postData is FileUpload? That may cause my Apps Script nor to be able to extract it right? Also, I thought I read that using fetch() is considerably better than using XMLHttpRequest() so should I be using my Method 2? Still not sure what am I doing wrong to populate it in individual cells in sheets using Apps Script?
Any help is appreciated!
Output Using sheet.getRange(3,1).setValue(JSON.stringify(e))
{"parameter":{},"postData":{"contents":"{\"firstName\":\"Jay\",\"lastName\":\"Smith\"}","length":38,"name":"postData","type":"text/plain"},"parameters":{},"contentLength":38,"queryString":"","contextPath":""}
Based on the output provided by sheet.getRange(3,1).setValue(JSON.stringify(e)) (I'm not sure which of your methods generated this, but that method looks like it's the right one), your doPost as written should work, with one modification: I don't see a variable request defined, so I assume you intend request to be the http request object, which in an Apps Script web app is defined by the e parameter passed to the doPost (documentation). Therefore change this line
const data = JSON.parse(request.postData.contents)
to this one
const data = JSON.parse(e.postData.contents)

Axios in React Native - Cannot POST a Blob or File

I'm trying to post the raw data of a picture, using Axios, after taking it with react-native-image-picker.
I successfully generated a blob using this piece of code:
const file = await fetch(response.uri);
const theBlob = await file.blob();
If I inspect the metadata of the blob it's all right: MIME type, size, and so on.
However, when I try to POST it using Axios, using:
axios({
method: "POST",
url: "https://my-api-endpoint-api.example.org",
data: theBlob,
});
what I receive on the API side is this strange JSON payload:
{"_data":{"lastModified":0,"name":"rn_image_picker_lib_temp_77cb727c-5056-4cb9-8de1-dc5e13c673ec.jpg","size":1635688,"offset":0,"type":"image/jpeg","blobId":"83367ee6-fa11-4ae1-a1df-bf1fdf1d1f57","__collector":{}}}
The same code is working fine on React, and I have the same behavior trying with a File object instead of a Blob one.
I see in other answers that I could use something else than Axios, like RNFetchBlob.fetch(), but since I'm using shared functions between the React website and the React Native app, I'd really prefer an approach that allows me to use Axios and Blobs.
Is there some way to work around it?
Updated answer
As pointed out by #T.J.Crowder in the comments, there is a cleaner approach that works around the issue of the React Native host environment, without touching anything else on the code.
It's enough to add this on the index.js file, before everything else:
Blob.prototype[Symbol.toStringTag] = 'Blob'
File.prototype[Symbol.toStringTag] = 'File'
I leave my original answer here under since it's a working alternative if one doesn't want to mess up with the prototype.
Original answer
The described behavior happens because the host environment of React Native does not handle the Blob type nicely: it will actually become just an object.
In fact, if you try to render toString.call(new Blob()) in a component, you'll see [object Blob] in a browser, but [object Object] in React Native.
The matter is that the default transformRequest implementation of Axios will use exactly this method (toString.call) to check if you're passing a Blob or some generic object to it. If it sees you're passing a generic object, it applies a JSON.stringify to it, producing the strange JSON you're seeing POSTed.
This happens exactly here: https://github.com/axios/axios/blob/e9965bfafc82d8b42765705061b9ebe2d5532493/lib/defaults.js#L61
Actually, what happens here is that utils.isBlob(data) at line 48 returns false, since it really just applies toString on the value and checks if it is [object Blob], which as described above is not the case.
The fastest workaround I see here, since you're sure you're passing a Blob, is just to override transformRequest with a function that just returns the data as it is, like this:
axios({
method: "POST",
url: "https://my-api-endpoint-api.example.org",
data: theBlob,
transformRequest: (d) => d,
});
This will just make the request work.
I actually had this problem recently (using Expo SDK 43 on iPhone). I remember using axios over fetch because I had problems with uploading blobs with fetch in the past. But I tried it here and it just worked.
The context in this use case is downloading a giphy from url and then putting it on s3 with a signed request. Worked on both web and phone.
const blob = await fetch(
giphyUrl
).then((res) => res.blob());
fetch(s3SignedPutRequest, { method: "PUT", body: blob });
You can send file into server using formData through axios API like below :
const file = await fetch(response.uri);
const theBlob = await file.blob();
var formData = new FormData();
theBlob.lastModifiedDate = new Date();
theBlob.name = "file_name";
formData.append("file", theBlob);
axios.post('https://my-api-endpoint-api.example.org', formData, {
headers: {
'Content-Type': 'multipart/form-data'
}
})

Route to URL with javascript fetch method

My question seems pretty basic, but I came across a lot of documentation and question on this forum without getting any proper way to get the work done.
I have a secured webapp, in which I handle redirections programatically to send authentification headers with each request. Thus, instead of href links, I have buttons, which trigger the following function :
route_accessor.mjs
const access = (path = '') => {
const token = localStorage.getItem('anov_auth_token');
const dest = `http://localhost:8080/${path}`;
const headers = new Headers();
if (token) headers.append('authorization', token);
fetch(
dest,
{
method: 'GET',
headers,
mode: 'cors',
redirect: 'follow'
}
)
.then(response => {
if (response.url.includes('error/403')) {
localStorage.removeItem('anov_auth_token');
}
// Here I need to redirect to the response page
})
.catch(error => {
console.error(error);
});
};
export default access;
Then, I have NodeJs backend, which determines where I should go, either :
My requested page
A 403 page (if I sent a wrong token)
Login page (if I havent sent any token)
Backend works perfectly so far. The problem is, once I made my request, I can't display the result as I'd like. Here is what I tried. The following code takes place where I put a comment inside route_accessor.mjs.
Using window.location
window.location.href = response.url;
I tried every variant, such as window.location.replace(), but always went into the same issue : those methods launch a second request to the requested url, without sending the headers. So I end up in an infinite 403 redirection loop when token is acceptable by my server.
I tried methods listed in the following post : redirect after fetch response
Using document.write()
A second acceptable answer I found was manually updating page content and location. The following almost achieve what I want, with a few flaws :
response.text().then(htmlResponse => {
document.open();
document.write(htmlResponse);
document.close();
// This doesn't do what I want, event without the second argument
window.document.dispatchEvent(new Event("DOMContentLoaded", {
bubbles: true,
cancelable: true
}));
});
With this, I get my view updated. However, URL remains the same (I want it to change). And though every script is loaded, I have few DOMContentLoaded event to make my page fully functionnal, but they aren't triggered, at all. I can't manage to dispatch a new DOMContentLoaded properly after my document is closed.
Some other minor problems come, such as console not being cleared.
Conclusion
I am stuck with this issue for quite a time right now, and all my researches havent lead me to what I am looking for so far. Maybe I missed an important point here, but anyway...
This only concerns get requests.
Is their a proper way to make them behave like a link tag, with a single href, but with additional headers ? Can I do this only with javascript or is their a limitation to it ?
Thanks in advance for any helpful answer !

Empty values when posting FormData-object via fetch-api in Internet Explorer 11

I have written a React-component which should be used for all forms in my application. When a certain button is clicked I make some validation and finally I want to post the form to the server.
This is what this part currently looks like:
// get what should be submitted
const formData = new FormData(theForm)); // theForm is the DOM-element
// post the form-data element
fetch(theForm.action,
{
credentials: "include", //pass cookies, for authentication
method: theForm.method,
headers: {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
//"Content-Type": "application/x-www-form-urlencoded"
},
body: formData
})
.then(res => console.dir(res))
.catch(reason => console.error(reason));
The shown snippet work fine in Chrome. However, in IE11 it is not.
On the other hand, when uncommenting the Content-Type header, it will also break in Chrome.
As found https://stackoverflow.com/a/46642899/615288 it is always "multipart/form-data". But even if I set it to multipart/form-data the values are not send to the server.
I am using the FormData polyfill from https://github.com/jimmywarting/FormData as well as whatwg-fetch.
I don't see what is going on here as FormData should work in IE since version 9.
Sidenote: When commenting out the whole header-part it still works in Chrome as the browser seems to guess the correct ones (as it can be seen in the developer-tools)
Somebody reported this to me today in the repo.
https://github.com/jimmywarting/FormData/issues/44
Apparently "IE fail to set Content-Type header on XHR whose body is a typed Blob" that's why you get wrong content-type header. updating the version to might 3.0.7 fix this
I had this problem, but after much frustration I found out that appending an extra field to the FormData object suddenly all the fields appeared on the server.
const formData = new FormData(theForm);
// this somehow makes it work
formData.append('fakefield', 'nothing to see here!');
fetch(theForm.action).then(/* do the stuff */);
I don't know why it works, but before I added that line the server received {} as the request body and afterwards it received { myformfield: 'myvalue', myotherfield: 'myothervalue', fakefield: 'nothing to see here!' }
Hope this can save someone a bit of frustation :)

Categories