We Keep Coding

FastAPI returns "Error 422: Unprocessable entity" when I send multipart form data with JavaScript Fetch API

I have some issue with using Fetch API JavaScript method when sending some simple formData like so:
function register() {
var formData = new FormData();
var textInputName = document.getElementById('textInputName');
var sexButtonActive = document.querySelector('#buttonsMW > .btn.active');
var imagesInput = document.getElementById('imagesInput');
formData.append('name', textInputName.value);
if (sexButtonActive != null){
formData.append('sex', sexButtonActive.html())
} else {
formData.append('sex', "");
}
formData.append('images', imagesInput.files[0]);
fetch('/user/register', {
method: 'POST',
data: formData,
})
.then(response => response.json());
}
document.querySelector("form").addEventListener("submit", register);
And on the server side (FastAPI):
#app.post("/user/register", status_code=201)
def register_user(name: str = Form(...), sex: str = Form(...), images: List[UploadFile] = Form(...)):
try:
print(name)
print(sex)
print(images)
return "OK"
except Exception as err:
print(err)
print(traceback.format_exc())
return "Error"
After clicking on the submit button I get Error 422: Unprocessable entity. So, if I'm trying to add header Content-Type: multipart/form-data, it also doesn't help cause I get another Error 400: Bad Request. I want to understand what I am doing wrong, and how to process formData without such errors?

The 422 response body will contain an error message about which field(s) is missing or doesn’t match the expected format. Since you haven't provided that (please do so), my guess is that the error is triggered due to how you defined the images parameter in your endpoint. Since images is expected to be a List of File(s), you should instead define it using the File type instead of Form. For example:
images: List[UploadFile] = File(...)
^^^^
When using UploadFile, you don't have to use File() in the default value of the parameter. Hence, the below should also work:
images: List[UploadFile]
Additionally, in the frontend, make sure to use the body (not data) parameter in the fetch() function to pass the FormData object (see example in MDN Web Docs). For instance:
fetch('/user/register', {
method: 'POST',
body: formData,
})
.then(res => {...
Please have a look at this answer, as well as this answer, which provide working examples on how to upload multiple files and form data to a FastAPI backend, using Fetch API in the frontend.
As for manually specifying the Content-Type when sending multipart/form-data, you don't have to (and shouldn't) do that, but rather let the browser set the Content-Type—please take a look at this answer for more details.

So, I found that I has error in this part of code:
formData.append('images', imagesInput.files[0]);
Right way to upload multiple files is:
for (const image of imagesInput.files) {
formData.append('images', image);
}
Also, we should use File in FastAPI method arguments images: List[UploadFile] = File(...) (instead of Form) and change data to body in JS method. It's not an error, cause after method called, we get right type of data, for example:
Name: Bob
Sex: Man
Images: [<starlette.datastructures.UploadFile object at 0x7fe07abf04f0>]

how do I send a request using multipart/form-data?

I have an app in ReactJs, using Axios and Papaparse.
I have a page where a user drop a csv file in a box, I automatically download the csv, update and make some change in the data, and send a new csv file to a server.
I did all until I arrive to the part where I need to create a new csv, and upload it to the server.
Here is my code currently :
const data = papaparse.unparse(destinationUpdateData, {
header: true,
skipEmptyLines: true
});
// data is a string in csv format
const file = new File([data as BlobPart], "destination.csv", { type: "text/csv" });
// I get a File type.
const paramsDestination = {
project_id: this.props.projectId,
datastore_id: 'DESTINATIONS',
file: file,
id: ["1", "2","3"]
}
// the data I would like to send is build
Axios.post(`new_item_file_attachment`, params, {headers: {"Content-Type": "multipart/form-data"}})
//I send to the server
The thing is, my server is expecting a request with a content Type of multipart/form-data, but I don't get how manually set my parameter to match this type.
The api call currently don't work, because the data is passed like a json, and the server reject it.
Is it possible to achieve it ?
I tried using FormData, but I can't see how to send boolean and array

Not 100% familiar with Axios but it should be something like this:
var params = new FormData();
params.append("project_id", this.props.projectId);
params.append("datastore_id", 'DESTINATIONS');
params.append("file", file);
params.append("id", JSON.stringify(["1", "2","3"])); // Arrays must be stringified
Axios.post(`new_item_file_attachment`, params)
You definitely need to put everything in FormData object. Last time I was doing this, I also had to remove the "Content-Type": "multipart/form-data" from the header. I believe the correct header should get filled in automatically. Try it with and without that stating a header and let me know if either works.

Here is my solution.
const data = new FormData();
data.append("project_id", id);
data.append("file", file);
axios.post(url, data);
Try and comments when some errors occur.

Swagger Nodejs - Response Validation Failed

I am building the API with Swagger and NodeJS, the annoying problem I have faced so far is Swagger validates the response, and it's not always smooth.
My case:
In file .yaml: I checked the yaml syntax with Swagger Editor => File Yaml is correct.
/user/createNew:
x-swagger-router-controller: xxxxxxx
post:
tags:
- User
summary: Create New User
# used as the method name of the controller
operationId: createNewUser
parameters:
- name: NewUserReq
in: body
required: true
description: Email register
schema:
$ref: "#/definitions/NewUserReq"
responses:
"201":
description: Successful
schema:
# a pointer to a definition
$ref: "#/definitions/CreateUserResp"
# responses may fall through to errors
default:
description: Error
schema:
$ref: "#/definitions/ErrorResponse"
CreateUserResp:
properties:
status:
type: integer
description: Response status
response:
$ref: "#/definitions/MsgResponse"
MsgResponse:
required:
- resp_msg
properties:
resp_msg:
type: string
To check the response format, I generated the NodeJs file from Swagger Editor
examples['application/json'] = {
"response" : {
"resp_msg" : "aeiou"
},
"status" : 123
};
In controller file .js:
function createNewUser(req,res){
....
var resp = new Object();
resp.resp_msg=data.email;
final_response.status = 200;
final_response.response = resp;
console.log("createNewUser::Query succeffully", JSON.stringify(final_response));
//{"status":200,"response":{"resp_msg":"test#gmail.com"}}
res.set('Content-Type', 'application/json');
res.json(final_response);
}
Try to run API with Postman, the error happens with log below:
Error: Response validation failed: failed schema validation
at throwErrorWithCode (/var/app/current/node_modules/swagger-express-mw/node_modules/swagger-node-runner/node_modules/swagger-tools/lib/validators.js:121:13)
at Object.module.exports.validateAgainstSchema (/var/app/current/node_modules/swagger-express-mw/node_modules/swagger-node-runner/node_modules/swagger-tools/lib/validators.js:176:7)
at /var/app/current/node_modules/swagger-express-mw/node_modules/swagger-node-runner/node_modules/swagger-tools/middleware/swagger-validator.js:141:22
at /var/app/current/node_modules/swagger-express-mw/node_modules/swagger-node-runner/node_modules/swagger-tools/node_modules/async/lib/async.js:356:13
at async.forEachOf.async.eachOf (/var/app/current/node_modules/swagger-express-mw/node_modules/swagger-node-runner/node_modules/swagger-tools/node_modules/async/lib/async.js:233:13)
at _asyncMap (/var/app/current/node_modules/swagger-express-mw/node_modules/swagger-node-runner/node_modules/swagger-tools/node_modules/async/lib/async.js:355:9)
at Object.map (/var/app/current/node_modules/swagger-express-mw/node_modules/swagger-node-runner/node_modules/swagger-tools/node_modules/async/lib/async.js:337:20)
at validateValue (/var/app/current/node_modules/swagger-express-mw/node_modules/swagger-node-runner/node_modules/swagger-tools/middleware/swagger-validator.js:134:11)
at ServerResponse.res.end (/var/app/current/node_modules/swagger-express-mw/node_modules/swagger-node-runner/node_modules/swagger-tools/middleware/swagger-validator.js:252:9)
at ServerResponse.send (/var/app/current/node_modules/express/lib/response.js:205:10)
I cannot figure out what caused the error, I double checked the structure of JSON response.
Very appreciate for any suggestion.

You have put required: true which means you have to pass that parameter otherwise Response Validation Failed will appear.

I tried to reproduce but it gives me some errors. However, I tried removing MsgResponse and defining CreateUserResp as a single definition, and it worked:
CreateUserResp:
type: object
properties:
response:
type: object
properties:
resp_msg:
type: string
description: The response object
status:
type: number
description: The status code
It seems like you're not defining response as an object, and just adding some properties to it.

After trying some experiments, I made it work. My solution simply is changing response code "201" to "200" in yaml file, and it worked. I don't know why I left the response code "201". Anyway, I am still new with this, and don't know if it's a best practice or not. I am open to receive any better suggestion.

You can get this error if you don't specify in the header the data type sent within the request body
Content-Type : application/json

cordova-plugin-file-transfer: How do you upload a file to S3 using a signed URL?

I am able to upload to S3 using a file picker and regular XMLHttpRequest (which I was using to test the S3 setup), but cannot figure out how to do it successfully using the cordova file transfer plugin.
I believe it is either to do with the plugin not constructing the correct signable request, or not liking the local file uri given. I have tried playing with every single parameter from headers to uri types, but the docs aren't much help, and the plugin source is bolognese.
The string the request needs to sign match is like:
PUT
1391784394
x-amz-acl:public-read
/the-app/317fdf654f9e3299f238d97d39f10fb1
Any ideas, or possibly a working code example?

A bit late, but I just spent a couple of days struggling with this so in case anybody else is having problems, this is how managed to upload an image using the javascript version of the AWS SDK to create the presigned URL.
The key to solving the problem is in the StringToSign element of the XML SignatureDoesNotMatch error that comes back from Amazon. In my case it looked something like this:
<StringToSign>
PUT\n\nmultipart/form-data; boundary=+++++org.apache.cordova.formBoundary\n1481366396\n/bucketName/fileName.jpg
</StringToSign>
When you use the aws-sdk to generate a presigned URL for upload to S3, internally it will build a string based on various elements of the request you want to make, then create an SHA1 hash of it using your AWS secret. This hash is the signature that gets appended to the URL as a parameter, and what doesn't match when you get the SignatureDoesNotMatch error.
So you've created your presigned URL, and passed it to cordova-plugin-file-transfer to make your HTTP request to upload a file. When that request hits Amazon's server, the server will itself build a string based on the request headers etc, hash it and compare that hash to the signature on the URL. If the hashes don't match then it returns the dreaded...
The request signature we calculated does not match the signature you provided. Check your key and signing method.
The contents of the StringToSign element I mentioned above is the string that the server builds and hashes to compare against the signature on the presigned URL. So to avoid getting the error, you need to make sure that the string built by the aws-sdk is the same as the one built by the server.
After some digging about, I eventually found the code responsible for creating the string to hash in the aws-sdk. It is located (as of version 2.7.12) in:
node_modules/aws-sdk/lib/signers/s3.js
Down the bottom at line 168 there is a sign method:
sign: function sign(secret, string) {
return AWS.util.crypto.hmac(secret, string, 'base64', 'sha1');
}
If you put a console.log in there, string is what you're after. Once you make the string that gets passed into this method the same as the contents of StringToSign in the error message coming back from Amazon, the heavens will open and your files will flow effortlessly into your bucket.
On my server running node.js, I originally created my presigned URL like this:
var AWS = require('aws-sdk');
var s3 = new AWS.S3(options = {
endpoint: 'https://s3-eu-west-1.amazonaws.com',
accessKeyId: "ACCESS_KEY",
secretAccessKey: "SECRET_KEY"
});
var params = {
Bucket: 'bucketName',
Key: imageName,
Expires: 60
};
var signedUrl = s3.getSignedUrl('putObject', params);
//return signedUrl
This produced a signing string like this, similar to the OP's:
PUT
1481366396
/bucketName/fileName.jpg
On the client side, I used this presigned URL with cordova-plugin-file-transfer like so (I'm using Ionic 2 so the plugin is wrapped in their native wrapper):
let success = (result: any) : void => {
console.log("upload success");
}
let failed = (err: any) : void => {
let code = err.code;
alert("upload error - " + code);
}
let ft = new Transfer();
var options = {
fileName: filename,
mimeType: 'image/jpeg',
chunkedMode: false,
httpMethod:'PUT',
encodeURI: false,
};
ft.upload(localDataURI, presignedUrlFromServer, options, false)
.then((result: any) => {
success(result);
}).catch((error: any) => {
failed(error);
});
Running the code produced the signature doesn't match error, and the string in the <StringToSign> element looks like this:
PUT
multipart/form-data; boundary=+++++org.apache.cordova.formBoundary
1481366396
/bucketName/fileName.jpg
So we can see that cordova-plugin-file-transfer has added in its own Content-Type header which has caused a discrepancy in the signing strings. In the docs relating to the options object that get passed into the upload method it says:
headers: A map of header name/header values. Use an array to specify more than one value. On iOS, FireOS, and Android, if a header named Content-Type is present, multipart form data will NOT be used. (Object)
so basically, if no Content-Type header is set it will default to multipart form data.
Ok so now we know the cause of the problem, it's a pretty simple fix. On the server side I added a ContentType to the params object passed to the S3 getSignedUrl method:
var params = {
Bucket: 'bucketName',
Key: imageName,
Expires: 60,
ContentType: 'image/jpeg' // <---- content type added here
};
and on the client added a headers object to the options passed to cordova-plugin-file-transfer's upload method:
var options = {
fileName: filename,
mimeType: 'image/jpeg',
chunkedMode: false,
httpMethod:'PUT',
encodeURI: false,
headers: { // <----- headers object added here
'Content-Type': 'image/jpeg',
}
};
and hey presto! The uploads now work as expected.

I run into such issues with this plugin
The only working way I found to upload a file with a signature is the method of Christophe Coenraets : http://coenraets.org/blog/2013/09/how-to-upload-pictures-from-a-phonegap-app-to-amazon-s3/
With this method you will be able to upload your files using the cordova-plugin-file-transfer
First, I wanted to use the aws-sdk on my server to sign with getSignedUrl()
It returns the signed link and you only have to upload to it.
But, using the plugin it always end with 403 : signatures don't match
It may be related to the content length parameter but I didn't found for now a working solution with aws-sdk and the plugin

How to check if there is any payload (POST) data in a request with hapijs?

This is what I have:
server.route({
method: 'POST',
path: '/somePath/',
config: {
payload:{
output: 'stream'
}
});
I expect a short XML and that is why I using 'stream' instead of 'data'. I am using FakeToe as the XML parser as it is stream based and in my handler function I call request.payload.pipe(parser). All I want to check is if there is any POST data in the request and if not send a bad request back to the user.
I have thought about getting the size of data in the request.payload stream but there does not seem to be straightforward of doing that without pulling first and then pushing back data into the stream, which I don't like.
Everything works perfectly when the data is there, but if not FakeToe does not call the callback function with an error saying that the stream is empty. Also, I do not want to call request.payload.read method myself just to check if there is any data.
Any way to do that in hapijs?

You can use Joi to validate the presence of a payload. It is a schema and validation module provided by Hapi. It will automatically validate any requests received at an endpoint and reply with an appropriate badRequest message if the validation fails.
var Joi = require('Joi');
server.route({
method: 'POST',
path: '/somePath/',
config: {
payload:{
output: 'stream'
}
validate: {
payload: Joi.any().required()
}
});
More about this here and here.
EDIT:
As per the discussion in this GitHub Issue, a validation object like
Joi.object({
pipe: Joi.func().required()
}).unknown();
Can be used to test whether the payload is actually a stream.

I suppose that you have content-length header ? You could check that from request.headers instead of reading the payload.