We Keep Coding

new Uint32Array(buff, start_pos, arr_len) returning unexpected results with Node.js

I've been trying to read large binary files that are gzipped from a URL with a node.js program. To simplify the problem, I'm trying to read two unsigned ints from the start of the stream.
It appears that given a large buffer, the following code (incorrectly) returns the same array for a1 and a2 in Node.js, but (correctly) will not return the same array when run from within Chrome against the same URL when called using an XMLHttpRequest:
var bufferPos = 0;
var a1 = new Uint32Array(bData, bufferPos, 1);
bufferPos += 4;
var a2 = new Uint32Array(bData, bufferPos, 1);
Regardless of the content in data, a1 and a2 always appear to be identically long arrays the size of the whole buffer, so I've had to try this which has better results but I then need to convert from an array of values to a Uint32:
var bufferPos = 0;
var a1 = new Uint32Array(bData.slice(bufferPos, (bufferPos+=4)));
var a2 = new Uint32Array(bData.slice(bufferPos, (bufferPos+=4)));
This code is in a callback and the code that builds content and passes the data to bData is in a https.get call that contains this:
//In https.get()
tBuffer = null;
var gunzip = zlib.createGunzip();
resp.pipe(gunzip);
gunzip.on('data', (data) => {
if (tBuffer === null){
tBuffer = data;
} else {
tBuffer = Buffer.concat([tBuffer, data]);
}).on('end', () => {
//callback code in here
callback(tBuffer);
});
Am I handling the response data somehow incorrectly in Node.js that is different from what I would expect in the Chrome XMLHTTPRequest response? The call to gzip was also a pain... am I missing some step there?
Should I not be trying to parse the response content as a buffer in a callback for some reason?
Thanks for the help.

how to get pdf context into html?

for several days now I'm learning html, CSS and now javascript. What I need is a way to get the informations of an pdf document into my html webpage.
I tried several things now and couldnt find the correct answer or informations I need. So here come an use case:
get an .pdf document into a folder
get the information of all .pdf documents of the target folder (with the exact same formatting)
convert those information into html context
get this html context to show on the webpage (images and text)
1 is trivial, I can just drag and drop my documents
2 I'm thinking about something like an array, which then calls the folder to get data into it.
For this I found:
'use strict';
function getFiles(dir) {
fileList = [];
var files = fs.readdirSync(dir);
for (var i in files) {
if (!files.hasOwnProperty(i)) continue;
var name = dir + '/' + files[i];
if (!fs.statSync(name).isDirectory()) {
fileList.push(name);
}
}
return fileList;
}
console.log(getFiles('pathtodirectory'));
Here I'm always getting a reference error, no matter what the path, well I can use only a local path on my pc for now. I'm not 100% sure what everything does, but I think I got it good so far. This function just gets me a list of the documents to work with.
3 That's even more tricky for me now, but I think if I get the data to work with, I may be able to work something out.
4 I think I can do it with a little research
I am happy for any tips or solutions, as I said I'm quite new to all of this :)
regards,
Pascal

'use strict';
function getFiles(dir) {
fileList = []; // <- This becomes a global variable
Should be:
'use strict';
function getFiles(dir) {
var fileList = []; // <- Now it's local to this scope
Because creating implicit global variables are not allowed in strict mode.
Also the getDirSync return an array, so you should treat it as such:
function getFiles(dir) {
fileList = [];
var files = fs.readdirSync(dir);
for (var i = 0; i < files.length; i++) {
var name = dir + '/' + files[i];
if (!fs.statSync(name).isDirectory()) {
fileList.push(name);
}
}
return fileList;
}
Or with .reduce:
function getFiles(dir) {
return fs.readdirSync(dir).reduce(function(arr, file) {
var name = dir + '/' + file;
if (!fs.statSync(name).isDirectory()) {
arr.push(name);
}
return arr;
}, []);
}

Accessing typescript file variable values using gulp

I have several typescript files, some of them export a const named APIS.
I'm trying to access those exports (I want to concatenated all of them to a single file), but it doesn't seem to work. I'm obviously doing something wrong, but I'm not sure what.
For example, I have a folder named services, with 2 files: service1.ts, service2.ts.
service1.ts:
...
export const APIS = [ { "field1" : "blabla" } ];
service2.ts: does not contain the APIS var.
This is my gulpfile.js:
var gulp = require('gulp');
var concat = require('gulp-concat');
var map = require('gulp-map');
gulp.task('default', function() {
return gulp.src('.../services/*.ts')
.pipe(map(function(file) {
return file.APIS;
}))
.pipe(concat('all.js'))
.pipe(gulp.dest('./test/'));
});
When I run this task, I get nothing. When I added console.log(file.APIS); to the map function, I get undefined for all the values (although it is defined in service1.ts!).
This is following to: Extracting typescript exports to json file using gulp
EDIT: OK, so I tried saving the exports in a .js file instead of a .ts file, and now I can access those vars using require:
gulp.task('default', function() {
return gulp.src('./**/*.service.export.js')
.pipe(map(function(file) {
var fileObj = require(file.path);
...
}))
Now if I try console.log(fileObj.APIS); I get the correct values. What I'm still confused about is how I can pass these value on, and create a single file out of all these vars. Is it possible to push them into an array?

This will not work as you think it would work. Gulp itself knows nothing about typescript files, that file is a vinyl-file and has no knowledge about the typescript code within its content.
Edit
Based on your example, you can do something like this:
var gulp = require('gulp');
var concat = require('gulp-concat');
var map = require('gulp-map');
var fs = require('fs');
gulp.task('test', function ()
{
var allConstants = [];
var stream = gulp.src('./**/*.service.export.js')
.pipe(map(function(file)
{
var obj = require(file.path);
if (obj.APIS != null)
allConstants = allConstants.concat(obj.APIS);
return file;
}));
stream.on("end", function (cb)
{
// Do your own formatting here
var content = allConstants.map(function (constants)
{
return Object.keys(constants).reduce(function (aggregatedString, key)
{
return aggregatedString + key + " : " + constants[key];
}, "");
}).join(", ");
fs.writeFile('filename.txt', content, cb);
});
return stream;
});

Suggestion
If you want to collect multiple variables into a single file i.e. a common variables file I suggest gulp-replace.
Steps
Create a file, require it and use tags within that file to place your variables.
Advice
If you are already using services don't create an array. Instead create an object (JSON) where every property is a constant. i.e.
var constants = {
const_1: 0,
const_2: 1,
const_3: 2,
}

save restore local storage to a local file

Is there a way to easily save and restore the local storage to a file in jquery or JavaScript?
There are 3 scenarios for this:
testing with a specific local storage
making a backup of the local storage in some specific situations where this data is critical (we want to save in case local cache is deleted)
Setting up another browser from an existing local storage.

I probably would have just tacked this on as a comment to Nathaniel Johnson's answer, but I don't have the reputation yet! With regard with those methods, here are some more simple versions of his functions:
function getLocalStorage() {
return JSON.stringify(localStorage)
}
function writeLocalStorage(data) {
Object.keys(data).forEach(function(key) { localStorage.setItem(key, data[key])})
}

The process for saving and retrieving local storage has two parts.
First you must be able to retrieve the contents of local storage in a form that is manageable in javascript. Since local storage is a map of key-value pairs the easiest way to this is to turn local storage into a javascript object. Then take this object and turn it into a JSON string. What you do with this string is up to you but I find it easiest to just have the user copy the string into an email.
function getLocalStorage() {
var a = {};
for (var i = 0; i < localStorage.length; i++) {
var k = localStorage.key(i);
var v = localStorage.getItem(k);
a[k] = v;
}
var s = JSON.stringify(a);
return s;
}
When I get the string, I use the following function to turn my local storage into a copy of their local storage. Remember to wipe your local storage clean before duplicating their data with a call to localStorage.clear()
function writeLocalStorage(data) {
var o = JSON.parse(data);
for (var property in o) {
if (o.hasOwnProperty(property)) {
localStorage.setItem(property, o[property]);
}
}
}
The last part of your question is how to protect the data from overwriting. You can't write to a local file, however, you can have copy the data into <textarea> and tell the user how to copy and paste the data into a email or a more direct approach.

This javascript below works for me:
function getLocalstorageToFile(fileName) {
/* dump local storage to string */
var a = {};
for (var i = 0; i < localStorage.length; i++) {
var k = localStorage.key(i);
var v = localStorage.getItem(k);
a[k] = v;
}
/* save as blob */
var textToSave = JSON.stringify(a)
var textToSaveAsBlob = new Blob([textToSave], {
type: "text/plain"
});
var textToSaveAsURL = window.URL.createObjectURL(textToSaveAsBlob);
/* download without button hack */
var downloadLink = document.createElement("a");
downloadLink.download = fileName;
downloadLink.innerHTML = "Download File";
downloadLink.href = textToSaveAsURL;
downloadLink.onclick = function () {
document.body.removeChild(event.target);
};
downloadLink.style.display = "none";
document.body.appendChild(downloadLink);
downloadLink.click();
}

Create two bookmarks in Chrome with names e.g. LS BACKUP and LS RESTORE
Put those two snippets in URLs accordingly
javascript:!function(e){var o=document.createElement("textarea"),t=document.getSelection();o.textContent=e,document.body.appendChild(o),t.removeAllRanges(),o.select(),document.execCommand("copy"),t.removeAllRanges(),document.body.removeChild(o)}(JSON.stringify(localStorage)),alert("Local storage is copied to clipboard");
javascript:!function(){let t=prompt("Input local storage backup string");try{t=JSON.parse(t),Object.keys(t).forEach(r=>{try{localStorage.setItem(r,t[r])}catch(a){alert(`Error occurred with the key "${r}" and value "${t[r]}"`)}})}catch(t){alert("Input is not a valid JSON string")}}();
The first one will copy a JSON string to clipboard.
The second one will prompt you to insert a JSON string, which will be parsed and put to local storage.

Reading all files in a directory

here is the code snippet of what I am working with right now...
/*
Module: inputReader.js, directoryScanner.js (included)
Description: Basic file reader returns string of contents of a file from a set file name
Needs to be done: Add key listener to allow for a dynamic file name
*/
// Declare node.js module dependencies from API
var fs = require('fs'),
wrench = require('wrench'),
util = require('util')
// Define module to be exported as a function(s)
module.exports = {
dirRead: function() {
var rootfolder = './node_modules';
var filteredfiles = [];
var files = [];
var fileextension = '.html';
files = wrench.readdirSyncRecursive(rootfolder)
for (var i = 0; i<files.length; i++) {
if (files[i].indexOf(fileextension) != -1) {
filteredfiles.push(files[i]);
}
}
return filteredfiles;
},
fileRead: function() {
// Call synchronous fileSystem function readFileSync on file name
for(i = 0; i<filteredfiles.length; i++) {
var temp = fs.readFileSync(filteredfiles[i].toString(), 'utf8')
return temp
}
}
};
I am exporting a module of 2 different functions; the first (readDir) that reads a directory and its' subdirectories for a list of files and the second (readFile) which relies on the first (reads the list of files from the first function and actually goes through each file).
The problem is when I try to call that look of filteredfiles in the 2nd function, readFile, I get an error saying filteredfiles is undefined.
I am not sure how to fix this, may someone help me please? (My guess is a scope problem)...
res.send(reader.dirRead()) and then, res.send(reader.fileRead(reader.dirRead()))

filteredfiles is declared locally in dirRead function (so it is not visible for fileRead function). You must declare it in some higher scope, or pass it as a fileRead parameter.