We Keep Coding

Sending pdf (or any file) over in a POST request as json string and then writing it down by fs

I am creating a website and I need to make a backend POST server. I have created 90% of it, but the issue I am facing is that I need to send JSON as well as a pdf (normally sent by multipart/form-data, but would need a different route). What I am trying to do is transform the file into base64string, send it over in a request, and then restore it back and write it to a file. This whole thing happens, and the PDF even returns scrambled data, but the PDF is just a blank page when written down even after being converted back to binary and being written
HTML-side JS code:
async function post(endpoint){
let binaryCV;
let CV = document.getElementById("upfile").files[0];
var reader = new FileReader();
reader.onload = (readerEvt)=>{
var binaryString = readerEvt.target.result;
binaryCV = binaryString;
let xhr = new XMLHttpRequest();
let object = {
name: document.getElementById("ecaName").value,
email: document.getElementById("ecaEmail").value,
phone: document.getElementById("ecaTel").value,
class: document.getElementById("ecaClass").value,
institute: document.getElementById("ecaInstitute").value,
paragraph:{
experience: document.getElementById("ecaExp").value,
why: document.getElementById("ecaWhyPart").value,
changes: document.getElementById("ecaWhatChanges").value,
},
CV: binaryCV,
}
xhr.open("POST", `http://localhost:8080/apply/internship/${endpoint}`,true);
xhr.setRequestHeader('Content-Type', 'application/json');
xhr.send(JSON.stringify(object));
};
await reader.readAsBinaryString(CV);
/*xhr.onload = function() {
if (this.readyState == 4 && this.status == 200) {
document.getElementById("demo").innerHTML = xhttp.responseText;
}
};*/
}
Server side JS callback:
app.post('/apply/internship/econoxe', async (req, res)=>{
res.sendStatus(200);
let CV = req.body.CV;
fs.writeFileSync(path.join(__dirname,`../CV/${req.body.email}.pdf`),CV)
console.log(req.body);
})
All this returns 100% blank PDFs (with larger file size than original for some reason) no matter what PDF I upload
Please help
If you know any other way to do what I mean in one request and route, please tell!

I found the answer!I had to change
fs.writeFileSync(path.join(__dirname,`../CV/${req.body.email}.pdf`),CV)
to
fs.writeFileSync(path.join(__dirname,`../CV/${req.body.email}.pdf`),CV,"binary")
And this would fix the issue

Download PDF file With FTP using XMLHttpRequest and generic handler

I am trying to download PDF file from FTP server with Jquery Ajax request. I referred http://www.dave-bond.com/blog/2010/01/JQuery-ajax-progress-HMTL5/.
My Jquery ajax call is as below
$.ajax({
xhr: function () {
var xhr = new window.XMLHttpRequest();
//Download progress
xhr.addEventListener("progress", function (evt) {
console.log("Event :"+evt.lengthComputable);
if (evt.lengthComputable) {
var percentComplete = evt.loaded / evt.total;
//Do something with download progress
console.log(percentComplete);
}
}, false);
return xhr;
},
type: 'POST',
url: "Downloader.ashx",
success: function (data) {
//Do something success-ish
}
});
And My C# generic handler code to download file is as below
public void ProcessRequest(HttpContext context)
{
DownLoadFilesFromFTp("MyFile.pdf", "Foldername");
}
public bool DownLoadFilesFromFTp(string fileName,string ftpFolder)
{
//Create FTP Request.
try
{
string Ftp_Host = System.Configuration.ConfigurationManager.AppSettings["Ftp_Host"];
string Ftp_UserName = System.Configuration.ConfigurationManager.AppSettings["Ftp_UserName"];
string Password = System.Configuration.ConfigurationManager.AppSettings["Password"];
string downloadpath= System.Configuration.ConfigurationManager.AppSettings["downloadpath"];
//Fetch the Response and read it into a MemoryStream object.
string ftpurl = Ftp_Host + ftpFolder + "/" + fileName;
FtpWebRequest reqFTP;
reqFTP = (FtpWebRequest)FtpWebRequest.Create(new Uri(ftpurl));
reqFTP.Credentials = new NetworkCredential(Ftp_UserName, Password);
reqFTP.KeepAlive = false;
reqFTP.Method = WebRequestMethods.Ftp.DownloadFile;
reqFTP.UseBinary = true;
reqFTP.Proxy = null;
reqFTP.UsePassive = false;
FtpWebResponse response = (FtpWebResponse)reqFTP.GetResponse();
Stream responseStream = response.GetResponseStream();
FileStream writeStream = null;
//if (fileName.Substring(fileName.Length - 3, 3) == "pdf" || fileName.Substring(fileName.Length - 3, 3) == "PDF")
//{
writeStream = new FileStream(downloadpath + fileName, FileMode.Create);
//}
int Length = 2048; // 2048;
Byte[] buffer = new Byte[Length];
int bytesRead = responseStream.Read(buffer, 0, Length);
while (bytesRead > 0)
{
writeStream.Write(buffer, 0, bytesRead);
bytesRead = responseStream.Read(buffer, 0, Length);
}
responseStream.Close();
writeStream.Close();
response.Close();
return true;
}
catch (WebException wEx)
{
return false;
}
catch (Exception ex)
{
return false;
}
}
When I run a code files downloads to a folder without any issues and on Ajax call
if (evt.lengthComputable) {
}
When I console evt i got below result
Always returns false so i am unable to track a progress.
1) is there anything wrong with the code ?
2) Any alternative way to show progress bar while downloading pdf

For the bytes uploaded, it is quite easy to show progress bar. Just monitor the xhr.upload.onprogress event. The browser knows the size of the files it has to upload and the size of the uploaded data, so it can provide the progress info.
For the bytes downloaded, it is a little bit more difficult, because the only thing that the browser knows in this case is the size of the bytes it is receiving.
The reason of evt.lengthComputable is 0 is that the browser doesn't
know how many bytes will be sent in the server request.
There is a solution for this, it's sufficient to set a Content-Length header on the server like below, in order to get the total size of the bytes the browser is going to receive.
// prepare the response to the client. resp is the client Response
var resp = HttpContext.Current.Response;
// Add Content-Length of the file to headers
// if the headers is not set then the evt.loaded will be 0
resp.AddHeader("Content-Length", "lengthOfYourFile");

Your code JS side look fine.
I am not C# programmer, but i observed that C# server side, download the file ftp and save it to disk server, but never response/send the PDF binary to JS SIDE?
From JS side is 0 bytes download. and evt.lengthComputable is alway false/0.

XMLHttpRequest chunked response, only read last response in progress

I'm sending chunked data from a NodeJS application back to the browser. The chunks are really json strings. Problem I'm having is that every time the onprogress function is called, it adds on a string of the complete data. Meaning that response chunk number two, is appended to response chunk number one, and so on. I'd like to get ONLY the "just now" received chunk.
Here's the code:
console.log("Start scan...");
var xhr = new XMLHttpRequest();
xhr.responseType = "text";
xhr.open("GET", "/servers/scan", true);
xhr.onprogress = function () {
console.log("PROGRESS:", xhr.responseText);
}
xhr.send();
So really, the contents of xhr.responseText contains, when the third response comes, also the response text for the first and the second response. I've checked what the server is sending, and it doesn't look like there's a problem there. Using Node with Express, and sending with res.send("...") a couple of times. Also headers are set like so:
res.setHeader('Transfer-Encoding', 'chunked');
res.setHeader('X-Content-Type-Options', 'nosniff');
res.set('Content-Type', 'text/json');

This index based approach works for me:
var last_index = 0;
var xhr = new XMLHttpRequest();
xhr.open("GET", "/servers/scan");
xhr.onprogress = function () {
var curr_index = xhr.responseText.length;
if (last_index == curr_index) return;
var s = xhr.responseText.substring(last_index, curr_index);
last_index = curr_index;
console.log("PROGRESS:", s);
};
xhr.send();
Inspired by https://friendlybit.com/js/partial-xmlhttprequest-responses/

Catch too long form submission

I am developing an HTML page which has simple HTML form (nothing special) being submitted by button. There is a couple of situations when form submitted and response comes too long (if whenever really comes back). How can i organize the form the way it fires some callback when waiting a response is too long? We could show up some notice for user, indicating our server is overloaded in that situation.
Here is request that being sent by form:
POST http://example.com/search HTTP/1.1
Host: example.com
Proxy-Connection: keep-alive
Content-Length: 83
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://example.com
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://example.com/
Accept-Encoding: gzip, deflate
Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4
Cookie: [cookie definition omitted]
[form data omitted]
Is Proxy-Connection: keep-alive influence the process somehow? Googling led me to https://github.com/caxap/jquery-safeform plugin, but it is for a little bit different purpose.

It depends on what type of UI you want to present to the user. You can simply lower the timeout at the server level and if it can't finish the response in time, it will abort. However, the user experience is pretty harsh, as they'll just get a generic timeout error that likely won't even be from your site. (They'll have to click back or something to get back to your site.)
If you just want to display a message after a certain amount of time has passed, you can attach to the form's submit event and use setTimeout to display an alert or something:
$('#MyForm').on('submit', function () {
setTimeout(30000, function () { // number is milliseconds
alert('Sorry this is taking so long.');
});
});
Finally, if there's some method of tracking the progress of the action that's being completed server-side, you could use something like web sockets or long-polling via AJAX to provide a progress bar or status update of some sort. That's a bit more complex, though, and will require some research on your part.

There are two approaches, I will write two separate answers.
XMLHttpRequest progress approach
(for modern browsers)
Just send data and read uploading progress from XMLHttpRequest:
//-------------upload ------------------
var lastProgressIndex = -1;
//is the file api available in this browser
//only override *if* available.
if (new XMLHttpRequest().upload) {
$("#upload-files").click(function () {
upload_files($("#files-to-upload")[0].files, lastProgressIndex++);
return false;
});
$("#files-to-upload").change(function () {
upload_files($("#files-to-upload")[0].files, lastProgressIndex++);
return false;
});
$("#upload-files").hide();
}
function resetFormElement(elem) {
elem.wrap('<form>').closest('form').get(0).reset();
elem.unwrap();
}
function clear_upload() {
//https://stackoverflow.com/questions/1043957/clearing-input-type-file-using-jquery
var upload = $("#files-to-upload");
//upload.replaceWith(upload = upload.clone(true));
resetFormElement(upload);
}
//accepts the input.files parameter
function upload_files(filelist) {
if (typeof filelist !== "undefined") {
for (var i = 0, l = filelist.length; i < l; i++) {
upload_file(filelist[i], lastProgressIndex++);
}
}
clear_upload();
}
//each file upload produces a unique POST
function upload_file(file, index) {
//TODO - vytvor progress bar podle indexu
$("#progresscontainer").append('<div id="progressbar' + index + '" class="progressbar"><div id="progresslabel' + index + '" class="progressbarlabel"></div></div>')
var progressBarSelector = "#progressbar" + index;
var progressLabelSelector = "#progresslabel" + index;
var fileName = file.name;
var xhr = new XMLHttpRequest();
xhr.upload.addEventListener("progress", function (evt) {
if (evt.lengthComputable) {
//update the progress bar
var percent = Math.floor((evt.loaded / evt.total) * 100) + "%";
//TODO http://www.binaryintellect.net/articles/859d32c8-945d-4e5d-8c89-775388598f62.aspx
$(progressBarSelector).css({
width: percent
});
$(progressLabelSelector).html(fileName + ' ' + percent);
}
}, false);
// File uploaded
xhr.addEventListener("load", function () {
$(progressLabelSelector).html(fileName + " uploaded");
AddImageToGallery(GetFilenameWithoutExt(fileName));
$(progressBarSelector).fadeOut(500, function () {
$(progressBarSelector).remove();
});
}, false);
var guid = $("#Identification").val();
xhr.open("post", "/uploadurl/uploadfile/" + guid, true);
// Set appropriate headers
// We're going to use these in the UploadFile method
// To determine what is being uploaded.
xhr.setRequestHeader("Content-Type", "multipart/form-data");
xhr.setRequestHeader("X-File-Name", file.name);
xhr.setRequestHeader("X-File-Size", file.size);
xhr.setRequestHeader("X-File-Type", file.type);
// Send the file
xhr.send(file);
}
And server part:
private UploadedFile[] RetrieveFileFromRequest()
{
List<UploadedFile> uploadedFiles = new List<UploadedFile>();
if (Request.Files.Count > 0)
{ //they're uploading the old way
for (int i = 0; i < Request.Files.Count; i++)
{
var file = Request.Files[0];
string contentType = file.ContentType;
string filename = file.FileName;
UploadedFile uploadedFile = SaveUploadedFile(file.InputStream, file.ContentLength, filename, contentType);
uploadedFiles.Add(uploadedFile);
}
}
else if (Request.ContentLength > 0)
{
string filename = Request.Headers["X-File-Name"];
string contentType = Request.Headers["X-File-Type"];
UploadedFile uploadedFile = SaveUploadedFile(Request.InputStream, Request.ContentLength, filename, contentType);
uploadedFiles.Add(uploadedFile);
}
return uploadedFiles.ToArray();
}
These sources are modification of the original article. There is related stackoverflow question.

How can I organize the form the way it fires some callback when waiting a response is too long?
This is based on your algorithm. You can estimate the time like brute force calculation and show the result before beginning to perform. It's not an optimized solution if you start processing and then break it when too many time is spent!
Update: If you can't estimate like above, At least, write an asynch controller method which is useful when an action must perform several independent long running operations. Follow this:
Write an Ajax Form with unobtrusive-ajax
Define a function for OnBegin form option to call a timer
The timer's time out event will show another thing to user.
Write an asynch controller method for the operation which
contains a timer to stop processing when it takes more
than ?? seconds.

There are two approaches, I will write two separate answers.
Sending thread/UI thread approach
You will need 2 javascript "threads".
Sending thread:
With first open websocket and start sending data to server. On the server side update some variable how many data was sent and how many remaining.
UI thread:
Using Ajax ask server frequently (HTTP request with JSON format is fine), how many data was send and how many remaining and render the progressbar in javascript.
Finnaly, when UI thread request receive information that upload is ready, remove progressbar with jquery and show message or redirect page.
With this approach you can easily show send bytes, estimation time remaining, upload speed.
So every javascript thread has corresponding server tread. And both server threads share the structure with dataSend and dataRemaining dependent on your server technology.
Bellow You can find part of my code which upload several images and show preview after finish is finished:
intervalID = window.setInterval(function()
{
var request = new Sys.Net.WebRequest();
request.set_url('UploadProgress.ashx?DJUploadStatus=' + token + '&ts=' + new Date().getTime());
request.set_httpVerb('GET');
request.add_completed(function(executor) {
// the progress is returned as xml
var e = executor.get_xml().documentElement;
var empty = e.getAttribute('empty');
if (!empty)
{
// extract the attributes I am interested in
var percent = e.getAttribute('progress');
var file = e.getAttribute('file');
var kbs = Math.round(parseInt(e.getAttribute('bytes')) / 1024);
var size = Math.round(parseInt(e.getAttribute('size')) / 1024);
// update the progressbar to the new value
progressBar.set_percentage(percent);
// upadte the message
updateMessage('info', 'Uploading ' + file + ' ... ' + kbs + ' of ' + size + ' KB');
if (percent == 100)
{
// clear the interval
window.clearInterval(intervalID);
}
}
});
// invoke the request
request.invoke();
}, 1000);

You could check for WebTimeout exception from server side then use SignalR to actively send timeout messsage back to client side:
var request = (HttpWebRequest)WebRequest.Create("http://www.your_third_parties_page.com");
request.Timeout = 1000; //Timeout after 1000 ms
try
{
using (var stream = request.GetResponse().GetResponseStream())
using (var reader = new StreamReader(stream))
{
Console.WriteLine(reader.ReadToEnd());
}
}
catch (WebException ex)
{
//Check for timeout exception
if (ex.Status == WebExceptionStatus.Timeout)
{
//If timeout then send SignalR ajax message to inform the clients...
var context = GlobalHost.ConnectionManager.GetHubContext<YourHub>();
context.Clients.All.addNewMessageToPage("This method has been processing too long!");
}
}
See more how to setup SignalR for asp.net here

Why does XMLHttpRequest's upload onprogress callback get called only twice for a 15 MB file?

My code creates an XHR and sets its upload.onprogress callback to increment a counter. The problem is that, when I upload a 15 MB file (which takes over 10 seconds to upload), the onprogress callback fires only twice.
From this other question I learned that, according to the spec, a task should be queued to fire a progress event "about every 50ms or for every byte transmitted, whichever is least frequent." So, I expect to see many more events than just two.
I am testing on Chrome on Linux. In case it is relevant, the XHR code is run inside an iframe proxy (to communicate across domains).
var numProgressEventsFired = 0;
xhr = new XMLHttpRequest();
xhr.upload.onprogress = function(e) {
numProgressEventsFired += 1;
};
xhr.onload = function() {
console.info("numProgressEventsFired: " + numProgressEventsFired); // Always 2.
if (xhr.status === 200) {
console.info("Upload completed.");
} else {
console.error("Upload failed. Error code: " + xhr.status);
}
};
xhr.open("POST", url, true);
xhr.setRequestHeader("Content-Type", "multipart/form-data; boundary=" + boundary);
...
xhr.sendAsBinary(binaryDataMessage);