Why does FormData appends extra empty arrays? - javascript

I am trying to upload multiple images via axios to backend server on a new application. It worked perfectly on my previous applications I have developed. When I did some inspections, the problem was there were a couple of arrays with empty values added to the formdata. It didn't happen on the previous apps I developed.
This is the code to upload the files:
const formData = new FormData();
for(var i =0;i<data.files.length;i++){
let name = data.files[i].name
let ext = data.files[i].name.split('.')
ext = ext[ext.length-1].toLowerCase() == 'dcm' ? 'DCM':'IMG'
let content = data.files[i]
console.log({name,ext,content})
formData.append('types',ext)
formData.append('names',name)
formData.append('images',content)
}
const response = await axios.post(`${API_BASE_URL}images/`,
formData,
{
headers: {
"Content-Type": 'application/x-www-form-urlencoded',
"Authorization": TOKEN
}
})
return response;
This is the request payload of the old application
And this is the request payload of the new application
I use the same exact code for both of the new and old app. Notice that in the new app somehow the formdata appends a lot of arrays with empty values, while in the old app there's no such thing.
Can anybody tell me what happened here?
EDIT: It seems that axios's current version (0.27.1) sends data differently than the previous axios version I used (0.26.0). It worked fine when I downgraded the version number. Still curious what happened though...

Related

Intercepting requests in Puppeteer - updated postData not being sent

So after solving yesterday's problem (linked below) another one has popped up.
Setting a FormData key/value on click with Puppeteer/NodeJS
The issue I'm having today is that I'm trying to modify an existing key in the FormData that's being sent to the server. The request gets intercepted correctly, I parse the FormData using the "qs" query string parsing package, then try to modify an existing key and use "NEW VALUE" in it's place. When I log it everything looks fine. However when I run chrome with headless=false and inspect the traffic I can see that the request is being sent without "NEW VALUE". It's as if I'm just calling continue() without passing any parameters in. I've attached the code below.
await this.page.setRequestInterception(true);
this.page.on("request", interceptedRequest => {
const formDataObj = qs.parse(interceptedRequest.postData());
if (interceptedRequest.url() === "https://www.someurl.com" && formDataObj.email) {
formDataObj.existingKey= "NEW VALUE";
const options = {
'method': 'POST',
'postData': qs.stringify(formDataObj),
'headers': {
...interceptedRequest.headers(),
'Content-Type': 'application/x-www-form-urlencoded'
},
};
console.log("Sending with new data");
console.log(qs.stringify(formDataObj));
interceptedRequest.continue(options);
} else {
interceptedRequest.continue();
}
Not sure where to go from here. I've done a lot of reading and tried some things like different versions of Puppeteer, passing ONLY the FormData (as a string) in to the continue() call and using some flags like "--enable-feature=NetworkService" (suggested on github) when launching chrome but nothing is working. Any help would be greatly appreciated!

Axios in React Native - Cannot POST a Blob or File

I'm trying to post the raw data of a picture, using Axios, after taking it with react-native-image-picker.
I successfully generated a blob using this piece of code:
const file = await fetch(response.uri);
const theBlob = await file.blob();
If I inspect the metadata of the blob it's all right: MIME type, size, and so on.
However, when I try to POST it using Axios, using:
axios({
method: "POST",
url: "https://my-api-endpoint-api.example.org",
data: theBlob,
});
what I receive on the API side is this strange JSON payload:
{"_data":{"lastModified":0,"name":"rn_image_picker_lib_temp_77cb727c-5056-4cb9-8de1-dc5e13c673ec.jpg","size":1635688,"offset":0,"type":"image/jpeg","blobId":"83367ee6-fa11-4ae1-a1df-bf1fdf1d1f57","__collector":{}}}
The same code is working fine on React, and I have the same behavior trying with a File object instead of a Blob one.
I see in other answers that I could use something else than Axios, like RNFetchBlob.fetch(), but since I'm using shared functions between the React website and the React Native app, I'd really prefer an approach that allows me to use Axios and Blobs.
Is there some way to work around it?
Updated answer
As pointed out by #T.J.Crowder in the comments, there is a cleaner approach that works around the issue of the React Native host environment, without touching anything else on the code.
It's enough to add this on the index.js file, before everything else:
Blob.prototype[Symbol.toStringTag] = 'Blob'
File.prototype[Symbol.toStringTag] = 'File'
I leave my original answer here under since it's a working alternative if one doesn't want to mess up with the prototype.
Original answer
The described behavior happens because the host environment of React Native does not handle the Blob type nicely: it will actually become just an object.
In fact, if you try to render toString.call(new Blob()) in a component, you'll see [object Blob] in a browser, but [object Object] in React Native.
The matter is that the default transformRequest implementation of Axios will use exactly this method (toString.call) to check if you're passing a Blob or some generic object to it. If it sees you're passing a generic object, it applies a JSON.stringify to it, producing the strange JSON you're seeing POSTed.
This happens exactly here: https://github.com/axios/axios/blob/e9965bfafc82d8b42765705061b9ebe2d5532493/lib/defaults.js#L61
Actually, what happens here is that utils.isBlob(data) at line 48 returns false, since it really just applies toString on the value and checks if it is [object Blob], which as described above is not the case.
The fastest workaround I see here, since you're sure you're passing a Blob, is just to override transformRequest with a function that just returns the data as it is, like this:
axios({
method: "POST",
url: "https://my-api-endpoint-api.example.org",
data: theBlob,
transformRequest: (d) => d,
});
This will just make the request work.
I actually had this problem recently (using Expo SDK 43 on iPhone). I remember using axios over fetch because I had problems with uploading blobs with fetch in the past. But I tried it here and it just worked.
The context in this use case is downloading a giphy from url and then putting it on s3 with a signed request. Worked on both web and phone.
const blob = await fetch(
giphyUrl
).then((res) => res.blob());
fetch(s3SignedPutRequest, { method: "PUT", body: blob });
You can send file into server using formData through axios API like below :
const file = await fetch(response.uri);
const theBlob = await file.blob();
var formData = new FormData();
theBlob.lastModifiedDate = new Date();
theBlob.name = "file_name";
formData.append("file", theBlob);
axios.post('https://my-api-endpoint-api.example.org', formData, {
headers: {
'Content-Type': 'multipart/form-data'
}
})

Node.js Express send huge data to client vanilla JS

In my application I read huge data of images, and send the whole data to the client:
const imagesPaths = await getFolderImagesRecursive(req.body.rootPath);
const dataToReturn = await Promise.all(imagesPaths.map((imagePath) => new Promise(async (resolve, reject) => {
try {
const imageB64 = await fs.readFile(imagePath, 'base64');
return resolve({
filename: imagePath,
imageData: imageB64,
});
} catch {
return reject();
}
})));
return res.status(200).send({
success: true,
message: 'Successfully retreived folder images data',
data: dataToReturn,
});
Here is the client side:
const getFolderImages = (rootPath) => {
return fetch('api/getFolderImages', {
method: 'POST',
headers: { 'Content-type': 'application/json' },
body: JSON.stringify({ rootPath }),
});
};
const getFolderImagesServerResponse = await getFolderImages(rootPath);
const getFolderImagesServerData = await getFolderImagesServerResponse.json();
When I do send the data I get failure due to the huge data. Sending the data just with res.send(<data>) is impossible. So, then, how can I bypass this limitation - and how should I accept the data in the client side with the new process?
The answer to your problem requires some read :
Link to the solution
One thing you probably haven’t taken full advantage of before is that webserver’s http response is a stream by default.
They just make it easier for you to pass in synchron data, which is parsed to chunks under the hood and sent as HTTP packages.
We are talking about huge files here; naturally, we don’t want them to be stored in any memory, at least not the whole blob. The excellent solution for this dilemma is a stream.
We create a readstream with the help of the built-in node package ‘fs,’ then pass it to the stream compatible response.send parameter.
const readStream = fs.createReadStream('example.png');
return response.headers({
'Content-Type': 'image/png',
'Content-Disposition': 'attachment; filename="example.png"',
}).send(readStream);
I used Fastify webserver here, but it should work similarly with Koa or Express.
There are two more configurations here: naming the header ‘Content-Type’ and ‘Content-Disposition.’
The first one indicates the type of blob we are sending chunk-by-chunk, so the frontend will automatically give the extension to it.
The latter tells the browser that we are sending an attachment, not something renderable, like an HTML page or a script. This will trigger the browser’s download functionality, which is widely supported. The filename parameter is the download name of the content.
Here we are; we accomplished minimal memory stress, minimal coding, and minimal error opportunities.
One thing we haven’t mentioned yet is authentication.
For the fact, that the frontend won’t send an Ajax request, we can’t expect auth JWT header to be present on the request.
Here we will take the good old cookie auth approach. Cookies are set automatically on every request header that matches the criteria, based on the cookie options. More info about this in the frontend implementation part.
By default, cookies arrive as semicolon separated key-value pairs, in a single string. In order to ease out the parsing part, we will use Fastify’s Cookieparser plugin.
await fastifyServer.register(cookieParser);
Later in the handler method, we simply get the cookie that we are interested in and compare it to the expected value. Here I used only strings as auth-tokens; this should be replaced with some sort of hashing and comparing algorithm.
const cookies = request.cookies;
if (cookies['auth'] !== 'authenticated') {
throw new APIError(400, 'Unauthorized');
}
That’s it. We have authentication on top of the file streaming endpoint, and everything is ready to be connected by the frontend.

POST image to django rest API always returns 'No file was submitted'

I am trying to get this working now for days -.-
Using a simple NodeJS express server, I want to upload an image to a Django instance through Post request, but I just can't figure out, how to prepare the request and embed the file.
Later I would like to post the image, created from a canvas on the client side,
but for testing I was trying to just upload an existing image from the nodeJS server.
app.post('/images', function(req, res) {
const filename = "Download.png"; // existing local file on server
// using formData to create a multipart/form-data content-type
let formData = new FormData();
let buffer = fs.readFileSync(filename);
formData.append("data", buffer); // appending the file a buffer, alternatively could read as utf-8 string and append as text
formData.append('name', 'var name here'); // someone told me, I need to specify a name
const config = {
headers: { 'content-type': 'multipart/form-data' }
}
axios.post("http://django:8000/images/", formData, config)
.then(response => {
console.log("success!"); // never happens :(
})
.catch(error => {
console.log(error.response.data); // no file was submitted
});
});
What am I doing wrong or did I just miss something?
EDIT
I just found a nice snippet with a slighlty other approach on the npm form-data page, on the very bottom (npmjs.com/package/form-data):
const filename = "Download.png"; // existing local file on server
let formData = new FormData();
let stream = fs.createReadStream(filename);
formData.append('data', stream)
let formHeaders = formData.getHeaders()
axios.post('http://django:8000/images/', formData, {
headers: {
...formHeaders,
},
})
.then(response => {
console.log("success!"); // never happens :(
})
.catch(error => {
console.log(error.response.data); // no file was submitted
});
sadly, this doesn't change anything :( I still receive only Bad Request: No file was submitted
I don't really have much Django code just a basic setup using the rest_framework with an image model:
class Image(models.Model):
data = models.ImageField(upload_to='images/')
def __str__(self):
return "Image Resource"
which are also registered in the admin.py,
a serializer:
from .models import Image
class ImageSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = Image
fields = ('id', 'data')
using automatic URL routing.
I wrote a simple test script and put the same image on the django server, to verify that image uploads works, and it does:
import requests
url = "http://127.0.0.1:8000/images/"
file = {'data': open('Download.png', 'rb')}
response = requests.post(url, files=file)
print(response.status_code) # 201
I had a similar problem: I used the same Django endpoint to upload a file using axios 1) from the client side and 2) from the server side. From the client side it worked without any problem, but from the server side, the request body was always empty.
My solution was to use the following code:
const fileBuffer = await readFile(file.filepath)
const formData = new FormData()
formData.append('file', fileBuffer, file.originalFilename)
const response = await fetch(
urlJoin(BACKEND_URL),
{
method: 'POST',
body: formData,
headers: {
...formData.getHeaders(),
},
}
)
A few relevant references that I found useful:
This blog post, even though it seems the author manages to send form data from the server side using axios, I did not manage to reproduce it on my case.
This issue report in the axio repository, where one comment suggests to use fetch.
In your node.js express server instead of adding the image to the form data, try directly sending the stream in the API post.
const filename = "Download.png"; // existing local file on server
//let formData = new FormData();
let stream = fs.createReadStream(filename);
//formData.append('data', stream)
let formHeaders = formData.getHeaders()
axios.post('http://django:8000/images/', stream, {
headers: {
...formHeaders,
},
})
.then(response => {
console.log("success!"); // never happens :(
})
.catch(error => {
console.log(error.response.data); // no file was submitted
});
I still didn't manage to get this working with axios so I tried another package for sending files as post requests, namely unirest, which worked out of the box for me.
Also it is smaller, requires less code and does everything I needed:
const filename = "Download.png"; // existing local file on server
unirest
.post(url)
.attach('data', filename) // reads directly from local file
//.attach('data', fs.createReadStream(filename)) // creates a read stream
//.attach('data', fs.readFileSync(filename)) // 400 - The submitted data was not a file. Check the encoding type on the form. -> maybe check encoding?
.then(function (response) {
console.log(response.body) // 201
})
.catch((error) => console.log(error.response.data));
If I have some spare time in the future I may look into what was wrong with my axios implementation or someone does know a solution pls let me know :D

how do I send a request using multipart/form-data?

I have an app in ReactJs, using Axios and Papaparse.
I have a page where a user drop a csv file in a box, I automatically download the csv, update and make some change in the data, and send a new csv file to a server.
I did all until I arrive to the part where I need to create a new csv, and upload it to the server.
Here is my code currently :
const data = papaparse.unparse(destinationUpdateData, {
header: true,
skipEmptyLines: true
});
// data is a string in csv format
const file = new File([data as BlobPart], "destination.csv", { type: "text/csv" });
// I get a File type.
const paramsDestination = {
project_id: this.props.projectId,
datastore_id: 'DESTINATIONS',
file: file,
id: ["1", "2","3"]
}
// the data I would like to send is build
Axios.post(`new_item_file_attachment`, params, {headers: {"Content-Type": "multipart/form-data"}})
//I send to the server
The thing is, my server is expecting a request with a content Type of multipart/form-data, but I don't get how manually set my parameter to match this type.
The api call currently don't work, because the data is passed like a json, and the server reject it.
Is it possible to achieve it ?
I tried using FormData, but I can't see how to send boolean and array
Not 100% familiar with Axios but it should be something like this:
var params = new FormData();
params.append("project_id", this.props.projectId);
params.append("datastore_id", 'DESTINATIONS');
params.append("file", file);
params.append("id", JSON.stringify(["1", "2","3"])); // Arrays must be stringified
Axios.post(`new_item_file_attachment`, params)
You definitely need to put everything in FormData object. Last time I was doing this, I also had to remove the "Content-Type": "multipart/form-data" from the header. I believe the correct header should get filled in automatically. Try it with and without that stating a header and let me know if either works.
Here is my solution.
const data = new FormData();
data.append("project_id", id);
data.append("file", file);
axios.post(url, data);
Try and comments when some errors occur.

Categories