We Keep Coding

Using pdf.js to render a PDF but it doesn't work and I don't get any error messages to help me debug the issue

I'm trying to build a Flask app where I upload pdf's and I'm working on previewing them before submitting to the back-end.
The script I'm using is as follows:
const imageUploadValidation = (function () {
"use strict";
pdfjsLib.GlobalWorkerOptions.workerSrc =
"https://mozilla.github.io/pdf.js/build/pdf.js";
const onFilePicked = function (event) {
// Select file Nodelist containing 1 file
const files = event.target.files;
const filename = files[0].name;
if (filename.lastIndexOf(".") <= 0) {
return alert("Please add a valid file!");
}
const fileReader = new FileReader();
fileReader.onload = function (e) {
const pdfData = e.target.result;
let loadingTask = pdfjsLib.getDocument({ data: pdfData })
loadingTask.promise.then(function (pdf) {
console.log("PDF loaded", pdf);
pdf.getPage(1).then((page) => {
console.log("page loaded", page);
// var scale = 1.5;
// var viewport = page.getViewport({ scale: scale });
var iframe = document.getElementById("image-preview");
iframe.src = page
// var context = canvas.getContext("2d");
// canvas.height = viewport.height;
// canvas.width = viewport.width;
// var renderContext = {
// canvasContext: context,
// viewport: viewport,
// };
// var renderTask = page.render(renderContext);
// renderTask.promise.then(function () {
// console.log("Page rendered");
// });
});
})
.catch((error) => {
console.log(error);
});
};
const pdf = fileReader.readAsArrayBuffer(files[0]);
console.log("read as Data URL", pdf);
};
const Constructor = function (selector) {
const publicAPI = {};
const changeHandler = (e) => {
// console.log(e)
onFilePicked(e);
};
publicAPI.init = function (selector) {
// Check for errors.
const fileInput = document.querySelector(selector);
if (!selector || typeof selector !== "string") {
throw new Error("Please provide a valid selector");
}
fileInput.addEventListener("change", changeHandler);
};
publicAPI.init(selector);
return publicAPI;
};
return Constructor;
})();
imageUploadValidation("form input[type=file]");
The loading task promise never seems to run. Everything seems to work up until that point. I'm not familiar with this Promise syntax, so I can't be sure if the problem is there or how I'm passing in the pdf file.
P.S. The commented out code is the original way I had this setup, what
s uncommented was just me testing a different way.

Check Datatype
First you might want to check what your getting back from your FileReader, specifically what is the datatype for pdfData. If you have a look at the documentation (direct link) getDocument is expecting a Unit8Array or a binary string.
Add Missing Parameters
The next problem you have is your missing required parameters in your call to getDocument. Here is the minimum required arguments:
var args = {
url: 'https://example.com/the-pdf-to-load.pdf',
cMapUrl: "./cmaps/",
cMapPacked: true,
}
I have never used the data argument in place of the url but as long as you supply the correct datatype you should be fine. Notice that cMapUrl should be a relative or absolute path to the cmap folder. PDFJS often needs these files to actually interpret a PDF file. Here are all the files from the demo repository (GitHub pages): cmaps You'll need to add these to your project.
Instead of using data I would recommend uploading your files as blobs and then all you have to do is supply the blob URL as url. I am not familiar with how to do that, I just know its possible in modern browsers.
Where Is Your Viewer / You Don't Need iFrame or Canvas
PDFJS just needs a div to place the PDF inside of. It's picky about some of the CSS rules, for exmaple it MUST be positioned absolute, otherwise PDFJS generates the pages as 0px height.
I don't see PDFViewer or PDFLinkService in your code. It looks like you are trying to build the entire viewer from scratch yourself. This is no small endeavor. When you get loadingTask working correctly the response should be handled something like this:
loadingTask.promise.then(
// Success function.
function( doc ) {
// viewer is holding: new pdfjsViewer.PDFViewer()
// linkService is: new pdfjsViewer.PDFLinkService()
viewer.setDocument( doc );
linkService.setDocument( doc );
},
// Error function.
function( exception ) {
// What type of error occurred?
if ( exception.name == 'PasswordException' ) {
// Password missing, prompt the user and try again.
elem.appendChild( getPdfPasswordBox() );
} else {
// Some other error, stop trying to load this PDF.
console.error( exception );
}
/**
* Additional exceptions can be reversed engineered from here:
* https://github.com/mozilla/pdf.js/blob/master/examples/mobile-viewer/viewer.js
*/
}
);
Notice that PDFViewer does all the hard work for you. PDFLinkService is needed if you want links in the PDF to work. You really should checkout the live demo and the example files.
Its a lot of work but these example files specifically can teach you all you need to know about PDFJS.
Example / Sample Code
Here is some sample code from a project I did with PDFJS. The code is a bit advanced but it should help you reverse engineer how PDFJS is working under the hood a bit better.
pdfObj = An object to store all the info and objects for this PDF file. I load multiple PDFs on a single page so I need this to keep them separate from each other.
updatePageInfo = My custome function that is called by PDFJS's eventBus when the user changes pages in the PDF; this happens as they scroll from page to page.
pdfjsViewer.DownloadManager = I allow users to download the PDFs so I need to use this.
pdfjsViewer.EventBus = Handles events like loading, page changing, and so on for the PDF. I am not 100% certain but I think the PDFViewer requires this.
pdfjsViewer.PDFViewer = What handles actually showing your PDF to users. container is the element on the page to render in, remember it must be positioned absolute.
// Create a new PDF object for this PDF.
var pdfObj = {
'container': elem.querySelector('.pdf-view-wrapper'),
'document': null,
'download': new pdfjsViewer.DownloadManager(),
'eventBus': new pdfjsViewer.EventBus(),
'history': null,
'id': id,
'linkService': null,
'loaded': 0,
'loader': null,
'pageTotal': 0,
'src': elem.dataset.pdf,
'timeoutCount': 0,
'viewer': null
};
// Update the eventBus to dispatch page change events to our own function.
pdfObj.eventBus.on( 'pagechanging', function pagechange(evt) {
updatePageInfo( evt );
} );
// Create and attach the PDFLinkService that handles links and navigation in the viewer.
var linkService = new pdfjsViewer.PDFLinkService( {
'eventBus': pdfObj.eventBus,
'externalLinkEnabled': true,
'externalLinkRel': 'noopener noreferrer nofollow',
'externalLinkTarget': 2 // Blank
} );
pdfObj.linkService = linkService;
// Create the actual PDFViewer that shows the PDF to the user.
var pdfViewer = new pdfjsViewer.PDFViewer(
{
'container': pdfObj.container,
'enableScripting': false, // Block embeded scripts for security
'enableWebGL': true,
'eventBus': pdfObj.eventBus,
'linkService': pdfObj.linkService,
'renderInteractiveForms': true, // Allow form fields to be editable
'textLayerMode': 2
}
);
pdfObj.viewer = pdfViewer;
pdfObj.linkService.setViewer( pdfObj.viewer );

Generate logs for web application in Javascript

I have a use case in which I would like to generate logs for my JS web application, which could be stored as a file on the client-side (stored on the user's machine).
Therefore, I want to know that what approach can I follow for generating logs in JS?

If you mean you want to do this from browser-hosted JavaScript code, I'm afraid you can't. Browser-based JavaScript code can't write to arbitrary files on the user's computer. It would be a massive security problem.
You could keep a "log" in web storage, but note that web storage has size limits so you wouldn't want to let it grow too huge.
Here's a barebones logging function that adds to a log in local storage:
function log(...msgs) {
// Get the current log's text, or "" if there isn't any yet
let text = localStorage.getItem("log") || "";
// Add this "line" of log data
text += msgs.join(" ") + "\r\n";
// Write it back to local storage
localStorage.setItem("log", text);
}
Obviously you can then build on that in a bunch of different ways (log levels, date/time logging, etc.).

You can use local storage to simulate file :
Create id for each line of your "file" and store the number of the last line
function logIntoStorage (pMsg) {
if (!pMsg) pMsg = "pMsg is not here !";
if ((typeof pMsg) != "string") pMsg = "pMsg is Not a string:"+(typeof pMsg);
let logNb = "logNb";
let padLong = 7;
let strLg = "0";
let lg = 0;
let maxSize = 50; // max nb of lines in the log
// Reading log num line
strLg = localStorage.getItem(logNb);
if(!strLg) { //logNb not stored yet
lg = 0;
strLg = "0";
localStorage.setItem(logNb, lg.toString(10)); // store the number of cur line
} else { // Read logNb from storage
strLg = localStorage.getItem(logNb);
lg = parseInt(strLg,10);
}
if (lg >= maxSize) {
lg = maxSize; // size limit nb lines.
pMsg = "LIMIT SIZE REACHED";
}
// log msg into localStorage at logLine:0000####
let s = ("0000000000000000"+strLg).substr(-padLong); // padding zeros
localStorage.setItem("logLine:"+s, pMsg);
if (lg >= maxSize) return;
lg++; // point to the next line
localStorage.setItem(logNb, lg.toString(10));
}

In modern Chrome you can actually "stream" data to the user's disk, after they give permission, thanks to the File System Access API.
To do so, you have to request for a file to save to, calling showSaveFilePicker().
Once you get the user's approval you'll receive a handle from where you'll be able to get a WriteableStream.
Once you are done writing, you just have to .close() the writer.
onclick = async () => {
if( !("showSaveFilePicker" in self) ) {
throw new Error( "unsupported browser" );
}
const handle = await showSaveFilePicker();
const filestream = await handle.createWritable();
const writer = await filestream.getWriter();
// here we have a WritableStream, with direct access to the user's disk
// we can write to it as we wish
writer.write( "hello" );
writer.write( " world" );
// when we're done writing
await writer.ready;
writer.close();
};
Live example.

Calculate SHA-1 checksum of local html5 video file using JavaScript

When a video on my local storage—let's say it's currently located at file:///home/user/video.m4v—is opened by dragging it into a new tab in Chrome, how can I calculate the SHA-1 checksum for the file using JavaScript?
Purpose:
I am planning to write a Chrome extension which will store the calculated checksum of videos (files with extensions matching a pattern) as localStorage objects in order to save the playback position of video upon tab close and then restore it when the file is loaded again, even if the location or filename of the video is changed.

You need a crypto library for this. A well known one is Google CryptoJS.
I found this as an specific example for your task: https://gist.github.com/npcode/11282867
After including the crypto-js source:
function sha1sum() {
var oFile = document.getElementById('uploadFile').files[0];
var sha1 = CryptoJS.algo.SHA1.create();
var read = 0;
var unit = 1024 * 1024;
var blob;
var reader = new FileReader();
reader.readAsArrayBuffer(oFile.slice(read, read + unit));
reader.onload = function(e) {
var bytes = CryptoJS.lib.WordArray.create(e.target.result);
sha1.update(bytes);
read += unit;
if (read < oFile.size) {
blob = oFile.slice(read, read + unit);
reader.readAsArrayBuffer(blob);
} else {
var hash = sha1.finalize();
console.log(hash.toString(CryptoJS.enc.Hex)); // print the result
}
}
}
I wouldn't recommend to calculate a hash over the whole video file as it can be pretty resource consuming depending on the file size. Maybe you can use just the meta information or reconsider about the filename and filepath again?

Web APIs have progressed considerably since I asked this question. Calculating a hex digest is now possible using the built-in SubtleCrypto.digest().
TS Playground link
function u8ToHex (u8: number): string {
return u8.toString(16).padStart(2, '0');
}
/** Ref: https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/digest#supported_algorithms */
const supportedAlgorithms = [
'SHA-1',
'SHA-256',
'SHA-384',
'SHA-512',
] as const;
type SupportedAlgorithm = typeof supportedAlgorithms[number];
type Message = string | Blob | BufferSource;
async function hexDigest (
algorithm: SupportedAlgorithm,
message: Message,
): Promise<string> {
let buf: BufferSource;
if (typeof message === 'string') buf = new TextEncoder().encode(message);
else if (message instanceof Blob) buf = await message.arrayBuffer();
else buf = message;
const hash = await crypto.subtle.digest(algorithm, buf);
return [...new Uint8Array(hash)].map(u8ToHex).join('');
}

How to write a file from an ArrayBuffer in JS

I am trying to write a file uploader for Meteor framework.
The principle is to split the fileon the client from an ArrayBuffer in small packets of 4096 bits that are sent to the server through a Meteor.method.
The simplified code below is the part of the client that sends a chunk to the server, it is repeated until offset reaches data.byteLength :
// data is an ArrayBuffer
var total = data.byteLength;
var offset = 0;
var upload = function() {
var length = 4096; // chunk size
// adjust the last chunk size
if (offset + length > total) {
length = total - offset;
}
// I am using Uint8Array to create the chunk
// because it can be passed to the Meteor.method natively
var chunk = new Uint8Array(data, offset, length);
if (offset < total) {
// Send the chunk to the server and tell it what file to append to
Meteor.call('uploadFileData', fileId, chunk, function (err, length) {
if (!err) {
offset += length;
upload();
}
}
}
};
upload(); // start uploading
The simplified code below is the part on the server that receives the chunk and writes it to the file system :
var fs = Npm.require('fs');
var Future = Npm.require('fibers/future');
Meteor.methods({
uploadFileData: function(fileId, chunk) {
var fut = new Future();
var path = '/uploads/' + fileId;
// I tried that with no success
chunk = String.fromCharCode.apply(null, chunk);
// how to write the chunk that is an Uint8Array to the disk ?
fs.appendFile(path, chunk, 'binary', function (err) {
if (err) {
fut.throw(err);
} else {
fut.return(chunk.length);
}
});
return fut.wait();
}
});
I failed to write a valid file to the disk, actually the file is saved but I cannot open it, when I see the content in a text editor, it is similar to the original file (a jpg for example) but some characters are different, I think that could be an encoding problem as the file size is not the same, but I don't know how to fix that...

Saving the file was as easy as creating a new Buffer with the Uint8Array object :
// chunk is the Uint8Array object
fs.appendFile(path, Buffer.from(chunk), function (err) {
if (err) {
fut.throw(err);
} else {
fut.return(chunk.length);
}
});

Building on Karl.S answer, this worked for me, outside of any framework:
fs.appendFileSync(outfile, Buffer.from(arrayBuffer));

Just wanted to add that in newer Meteor you could avoid some callback hell with async/await. Await will also throw and push the error up to client
Meteor.methods({
uploadFileData: async function(file_id, chunk) {
var path = 'somepath/' + file_id; // be careful with this, make sure to sanitize file_id
await fs.appendFile(path, new Buffer(chunk));
return chunk.length;
}
});

How to calculate md5 hash of a file using javascript

Is there a way to calculate the MD5 hash of a file before the upload to the server using Javascript?

While there are JS implementations of the MD5 algorithm, older browsers are generally unable to read files from the local filesystem.
I wrote that in 2009. So what about new browsers?
With a browser that supports the FileAPI, you can read the contents of a file - the user has to have selected it, either with an <input> element or drag-and-drop. As of Jan 2013, here's how the major browsers stack up:
FF 3.6 supports FileReader, FF4 supports even more file based functionality
Chrome has supported the FileAPI since version 7.0.517.41
Internet Explorer 10 has partial FileAPI support
Opera 11.10 has partial support for FileAPI
Safari - I couldn't find a good official source for this, but this site suggests partial support from 5.1, full support for 6.0. Another article reports some inconsistencies with the older Safari versions
How?
See the answer below by Benny Neugebauer which uses the MD5 function of CryptoJS

I've made a library that implements incremental md5 in order to hash large files efficiently.
Basically you read a file in chunks (to keep memory low) and hash it incrementally.
You got basic usage and examples in the readme.
Be aware that you need HTML5 FileAPI, so be sure to check for it.
There is a full example in the test folder.
https://github.com/satazor/SparkMD5

it is pretty easy to calculate the MD5 hash using the MD5 function of CryptoJS and the HTML5 FileReader API. The following code snippet shows how you can read the binary data and calculate the MD5 hash from an image that has been dragged into your Browser:
var holder = document.getElementById('holder');
holder.ondragover = function() {
return false;
};
holder.ondragend = function() {
return false;
};
holder.ondrop = function(event) {
event.preventDefault();
var file = event.dataTransfer.files[0];
var reader = new FileReader();
reader.onload = function(event) {
var binary = event.target.result;
var md5 = CryptoJS.MD5(binary).toString();
console.log(md5);
};
reader.readAsBinaryString(file);
};
I recommend to add some CSS to see the Drag & Drop area:
#holder {
border: 10px dashed #ccc;
width: 300px;
height: 300px;
}
#holder.hover {
border: 10px dashed #333;
}
More about the Drag & Drop functionality can be found here: File API & FileReader
I tested the sample in Google Chrome Version 32.

The following snippet shows an example, which can archive a throughput of 400 MB/s while reading and hashing the file.
It is using a library called hash-wasm, which is based on WebAssembly and calculates the hash faster than js-only libraries. As of 2020, all modern browsers support WebAssembly.
const chunkSize = 64 * 1024 * 1024;
const fileReader = new FileReader();
let hasher = null;
function hashChunk(chunk) {
return new Promise((resolve, reject) => {
fileReader.onload = async(e) => {
const view = new Uint8Array(e.target.result);
hasher.update(view);
resolve();
};
fileReader.readAsArrayBuffer(chunk);
});
}
const readFile = async(file) => {
if (hasher) {
hasher.init();
} else {
hasher = await hashwasm.createMD5();
}
const chunkNumber = Math.floor(file.size / chunkSize);
for (let i = 0; i <= chunkNumber; i++) {
const chunk = file.slice(
chunkSize * i,
Math.min(chunkSize * (i + 1), file.size)
);
await hashChunk(chunk);
}
const hash = hasher.digest();
return Promise.resolve(hash);
};
const fileSelector = document.getElementById("file-input");
const resultElement = document.getElementById("result");
fileSelector.addEventListener("change", async(event) => {
const file = event.target.files[0];
resultElement.innerHTML = "Loading...";
const start = Date.now();
const hash = await readFile(file);
const end = Date.now();
const duration = end - start;
const fileSizeMB = file.size / 1024 / 1024;
const throughput = fileSizeMB / (duration / 1000);
resultElement.innerHTML = `
Hash: ${hash}<br>
Duration: ${duration} ms<br>
Throughput: ${throughput.toFixed(2)} MB/s
`;
});
<script src="https://cdn.jsdelivr.net/npm/hash-wasm"></script>
<!-- defines the global `hashwasm` variable -->
<input type="file" id="file-input">
<div id="result"></div>

HTML5 + spark-md5 and Q
Assuming your'e using a modern browser (that supports HTML5 File API), here's how you calculate the MD5 Hash of a large file (it will calculate the hash on variable chunks)
function calculateMD5Hash(file, bufferSize) {
var def = Q.defer();
var fileReader = new FileReader();
var fileSlicer = File.prototype.slice || File.prototype.mozSlice || File.prototype.webkitSlice;
var hashAlgorithm = new SparkMD5();
var totalParts = Math.ceil(file.size / bufferSize);
var currentPart = 0;
var startTime = new Date().getTime();
fileReader.onload = function(e) {
currentPart += 1;
def.notify({
currentPart: currentPart,
totalParts: totalParts
});
var buffer = e.target.result;
hashAlgorithm.appendBinary(buffer);
if (currentPart < totalParts) {
processNextPart();
return;
}
def.resolve({
hashResult: hashAlgorithm.end(),
duration: new Date().getTime() - startTime
});
};
fileReader.onerror = function(e) {
def.reject(e);
};
function processNextPart() {
var start = currentPart * bufferSize;
var end = Math.min(start + bufferSize, file.size);
fileReader.readAsBinaryString(fileSlicer.call(file, start, end));
}
processNextPart();
return def.promise;
}
function calculate() {
var input = document.getElementById('file');
if (!input.files.length) {
return;
}
var file = input.files[0];
var bufferSize = Math.pow(1024, 2) * 10; // 10MB
calculateMD5Hash(file, bufferSize).then(
function(result) {
// Success
console.log(result);
},
function(err) {
// There was an error,
},
function(progress) {
// We get notified of the progress as it is executed
console.log(progress.currentPart, 'of', progress.totalParts, 'Total bytes:', progress.currentPart * bufferSize, 'of', progress.totalParts * bufferSize);
});
}
<script src="https://cdnjs.cloudflare.com/ajax/libs/q.js/1.4.1/q.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/spark-md5/2.0.2/spark-md5.min.js"></script>
<div>
<input type="file" id="file"/>
<input type="button" onclick="calculate();" value="Calculate" class="btn primary" />
</div>

You need to to use FileAPI. It is available in the latest FF & Chrome, but not IE9.
Grab any md5 JS implementation suggested above. I've tried this and abandoned it because JS was too slow (minutes on large image files). Might revisit it if someone rewrites MD5 using typed arrays.
Code would look something like this:
HTML:
<input type="file" id="file-dialog" multiple="true" accept="image/*">
JS (w JQuery)
$("#file-dialog").change(function() {
handleFiles(this.files);
});
function handleFiles(files) {
for (var i=0; i<files.length; i++) {
var reader = new FileReader();
reader.onload = function() {
var md5 = binl_md5(reader.result, reader.result.length);
console.log("MD5 is " + md5);
};
reader.onerror = function() {
console.error("Could not read the file");
};
reader.readAsBinaryString(files.item(i));
}
}

Apart from the impossibility to get
file system access in JS, I would not
put any trust at all in a
client-generated checksum. So
generating the checksum on the server
is mandatory in any case. – Tomalak
Apr 20 '09 at 14:05
Which is useless in most cases. You want the MD5 computed at client side, so that you can compare it with the code recomputed at server side and conclude the upload went wrong if they differ. I have needed to do that in applications working with large files of scientific data, where receiving uncorrupted files were key. My cases was simple, cause users had the MD5 already computed from their data analysis tools, so I just needed to ask it to them with a text field.

If sha256 is also fine:
async sha256(file: File) {
// get byte array of file
let buffer = await file.arrayBuffer();
// hash the message
const hashBuffer = await crypto.subtle.digest('SHA-256', buffer);
// convert ArrayBuffer to Array
const hashArray = Array.from(new Uint8Array(hashBuffer));
// convert bytes to hex string
const hashHex = hashArray.map(b => b.toString(16).padStart(2, '0')).join('');
return hashHex;
}

To get the hash of files, there are a lot of options. Normally the problem is that it's really slow to get the hash of big files.
I created a little library that get the hash of files, with the 64kb of the start of the file and the 64kb of the end of it.
Live example: http://marcu87.github.com/hashme/ and library: https://github.com/marcu87/hashme

hope you have found a good solution by now. If not, the solution below is an ES6 promise implementation based on js-spark-md5
import SparkMD5 from 'spark-md5';
// Read in chunks of 2MB
const CHUCK_SIZE = 2097152;
/**
* Incrementally calculate checksum of a given file based on MD5 algorithm
*/
export const checksum = (file) =>
new Promise((resolve, reject) => {
let currentChunk = 0;
const chunks = Math.ceil(file.size / CHUCK_SIZE);
const blobSlice =
File.prototype.slice ||
File.prototype.mozSlice ||
File.prototype.webkitSlice;
const spark = new SparkMD5.ArrayBuffer();
const fileReader = new FileReader();
const loadNext = () => {
const start = currentChunk * CHUCK_SIZE;
const end =
start + CHUCK_SIZE >= file.size ? file.size : start + CHUCK_SIZE;
// Selectively read the file and only store part of it in memory.
// This allows client-side applications to process huge files without the need for huge memory
fileReader.readAsArrayBuffer(blobSlice.call(file, start, end));
};
fileReader.onload = e => {
spark.append(e.target.result);
currentChunk++;
if (currentChunk < chunks) loadNext();
else resolve(spark.end());
};
fileReader.onerror = () => {
return reject('Calculating file checksum failed');
};
loadNext();
});

There is a couple scripts out there on the internet to create an MD5 Hash.
The one from webtoolkit is good, http://www.webtoolkit.info/javascript-md5.html
Although, I don't believe it will have access to the local filesystem as that access is limited.

This is another hash-wasm example, but using the streams API, instead of having to set FileReader:
async function calculateSHA1(file: File) {
const hasher = await createSHA1()
const hasherStream = new WritableStream<Uint8Array>({
start: () => {
hasher.init()
// you can set UI state here also
},
write: chunk => {
hasher.update(chunk)
// you can set UI state here also
},
close: () => {
// you can set UI state here also
},
})
await file.stream().pipeTo(hasherStream)
return hasher.digest('hex')
}

I don't believe there is a way in javascript to access the contents of a file upload. So you therefore cannot look at the file contents to generate an MD5 sum.
You can however send the file to the server, which can then send an MD5 sum back or send the file contents back .. but that's a lot of work and probably not worthwhile for your purposes.