We Keep Coding

Efficient way to get the actual filename in a case-insensitive filesystem in node

This is NOT a dupe of this question. This question is NOT about Windows. It's a general question across OSes.
Is there an efficient way to get the correct case of a filename in node.js other than getting the directory and finding the matching name?
Example: Assume I have a folder with 3 files
+-someFolder
+-fooBar.txt
+-Moo.txt
+-ReadMe.txt
I want a function that passed somefolder/readme.txt returns someFolder/ReadMe.txt.
AFAICT the only way to do that is to call fs.readDir or fs.readDirSync and see if there is a matching file, something like
const fs = require('fs');
const path = require('path');
function getActualFilename(filename) {
if(!fs.existsSync(filename)) {
throw new Error(`${filename} does not exist`);
}
return getActualFilenameImpl(filename);
}
function getActualFilenameImpl(filename) {
const lcFilename = path.basename(filename).toLowerCase();
// handles passing in `c:\\`
if (!lcFilename) {
return filename.toUpperCase();
}
const dirname = path.dirname(filename);
let filenames;
try {
filenames = fs.readdirSync(dirname);
} catch (e) {
// we already verified the path exists above so if this
// happens it means the OS won't let use get a listing (UNC root on windows)
// so it's the best we can do
return filename;
}
const matches = filenames.filter(name => lcFilename === name.toLowerCase());
if (!matches.length) {
throw new Error(`${filename} does not exist`);
}
const realname = matches[0];
if (dirname !== '.') {
if (dirname.endsWith('/') || dirname.endsWith('\\')) {
return path.join(dirname, realname);
} else {
return path.join(getActualFilenameImpl(dirname), realname);
}
} else {
return realname;
}
}
The code above is pretty hacky. Trying in on different things has made it clear there's lots of edge cases. On Windows in particular UNC paths fail since you can't call fs.readdirSync once you get to the network path root. I have no idea what functions to call to figure out where that path separates and then how to get the correct case path for that which is probably an entirely separate set of Windows API calls (like calling whatever functions net use uses to show shares) etc...
I did notice path.dirname stops removing the trailing slash when it gets to a UNC path so using that to try to figure out when stop trying.
Notes:
I get that for example on Linux (and optionally on Mac) the file system may be case sensitive and I'd have to check for that but I'm mostly concerned with Windows and standard macOS and will deal with case-sensitive issues later.
I also get that JavaScript's toLowerCase might not match the OSes concept of case insensitivity so if there is a solution that takes that into account that would also be great!
I get that I could cache results or directory listing for a speed up but was wondering if there is some other function to use that doesn't read the entire directory listing.
I'm actually trying to solve several problems and am open to other suggestions
Problem 1: What filename to store in an app specific database. It seems best to store the actual filename. See #3
Problem 2: Figuring out if 2 filenames reference the same file/folder. So if the user specifies SomeFolder/foobar.txt and somefolder/FOOBAR.txt I don't want that to appear as 2 separate files if they are actually the same file. I need my app to know they reference the same file. I think I can call fs.stat for this and check if the ino field matches?
Problem 3: Related to problem 1, reloading metadata related to the file. If the user specifies SomeFolder/foobar.txt at some point and my app generates metadata related to the file, then at some other point in time they specify somefolder/FOOBAR.txt I need to find the matching metadata. My current thinking is by looking up the actual filename and using that to match with this problem would be solved. Although I suppose if they rename the file from FooBar.txt to foobar.txt it would lose the metadata. Not sure I care about that situation though since if they rename from FooBar.txt to SomethingElse.txt I definitely do not care if I lose the metadata.
That said, maybe I should store the ino as the key in my DB? Not sure I'm comfortable with that idea yet but it's a possibility and would love to know if others do that. Some checking reveals that at least on macOS the ino stays the same across moves and rename on the same drive which would be a good thing for my use case. On the other hand I'd assume ino is only valid per file system so if I have 2 different drives mounted I could get clashing inos. I could use dev and ino as a key as in
const stat = fs.statSync(filename);
const key = `${stat.dev}:${stat.ino}`;
Though I have no idea if stat.dev is always the same with removable storage. I assume it's not. So it seems like filename as key is probably better?

As long as the filesystem doesn't keep a connection between files with the same names in different cases (and I don't know any such filesystem) there can't be a solution other than scanning the directory because there is simply no API provided for this at all at any level.
So you have to either scan manually as you already suggested or by using libraries like glob to find files while ignoring case.
But you say you also have the filenames in a database. So if you can make sure that the filenames in the DB are exactly matching the filenames in the filesystem then you should be able to find the files in different cases by doing case-insensitive DB queries. If it is an SQL database then it should already provide this functionality. If it is a more primitive data store you may add another filename property which is always lower-case so you can match against this to find the real file.

Firefox add-on. How to really dowload image/file?

I'm getting more and more downhearted. For the last three days, I've been trying to add to my simple Firefox add-on a 'download image' feature.
The following is an add-on that creates a right-click contextual menu with a sub-menu:
var contextMenu = require("sdk/context-menu");
var clipboard = require("sdk/clipboard");
var data = require("sdk/self").data;
var myApp_cm = contextMenu.Menu({
label: "Send to myApp",
context: contextMenu.SelectorContext("body"),
items: [
contextMenu.Item({
label: "Send image to MyApp",
context: contextMenu.SelectorContext("img"),
contentScript: 'self.on("click", function (node, data) { ' +
' var link = node.src; ' +
' self.postMessage(link); ' +
'});',
onMessage: function(link) {
//
// Download image from 'link' and run 'myApp.exe' with
// downloaded image as parameter
//
}
})
]
});
I would like to add to the above code a simple download feature as the "Save as..." option of Firefox that downloads the image from the selected URL and runs an EXE with the downloaded image as a parameter.
I read everything I found about this argument starting from Mozilla MDN to all the questions asked at Stackoverflow. But, I never managed to make a single line of code work. I really don't understand why it's so complicated to download a file when this is the browser's job.
For example, I know that from Firefox 26+ I need to use downloads.jsm. So, I copied the following code from MDN.
Components.utils.import("resource://gre/modules/Downloads.jsm");
Components.utils.import("resource://gre/modules/osfile.jsm")
Components.utils.import("resource://gre/modules/Task.jsm");
Task.spawn(function () {
yield Downloads.fetch("http://www.mozilla.org/",OS.Path.join(OS.Constants.Path.tmpDir,"example-download.html"));
console.log("example-download.html has been downloaded.");
}).then(null, Components.utils.reportError);
But, I keep getting the error below:
So I added the string let {Cu, Ci, CC} = require('chrome') but nothing changes.
I'm 43 years old and I'm still learning JavaScript. I'm aware I don't have the same flexibility I had 2 decades ago. But, I remember that programming was much more straightforward. I still love programming but now I often find it quite frustrating.

I have not actually tried it, but I would not not expect the destructuring assignment
let {Cu, Ci, Cc} = require('chrome');
[Note: your CC should be Cc.]
to provide your Add-on SDK code to have access to the complete Components object through referencing it as Components, but only to have the properties (sub-objects) which you have assigned to "aliases" be available through the objects that you have defined using let:
Object ("alias") now available Object full name normally available
to your SDK add-on to Overlay and Restartless add-ons
Cu = Components.utils
Ci = Components.interfaces
Cc = Components.classes
The destructuring assignment should have extracted just the properties (sub-objects) referred to as Cu, Ci, and Cc within requre('chrome').
The code you copied from MDN would need to change to:
Cu.import("resource://gre/modules/Downloads.jsm");
Cu.import("resource://gre/modules/osfile.jsm")
Cu.import("resource://gre/modules/Task.jsm");
Task.spawn(function () {
yield Downloads.fetch("http://www.mozilla.org/",
OS.Path.join(OS.Constants.Path.tmpDir,"example-download.html"));
console.log("example-download.html has been downloaded.");
}).then(null, Cu.reportError);
If you wanted to use Components without using the Cc, Ci, Cu, Cr, and Cm aliases, you would need to use:
let {components} = require('chrome'); // note the lowercase "components"
let Components = components;
With that you could then use your original code:
Components.utils.import("resource://gre/modules/Downloads.jsm");
Components.utils.import("resource://gre/modules/osfile.jsm")
Components.utils.import("resource://gre/modules/Task.jsm");
Task.spawn(function () {
yield Downloads.fetch("http://www.mozilla.org/",
OS.Path.join(OS.Constants.Path.tmpDir,"example-download.html"));
console.log("example-download.html has been downloaded.");
}).then(null, Components.utils.reportError);
For more information, you can see the Chrome Authority page on MDN.

So, I copied the following code from MDN.
This is a big part of your problem. You're copying code without reasoning about it. Code snippets generally have prerequisites and assumptions that must be fulfilled, i.e. they must be evaluated on specific contexts - npm modules won't run in a browser for example.
Those code snippets too have dependencies, e.g. the Components object. The error message warns you about that, so that might be a good hint to read the documentation on require("chrome") and Components.
The second issue is that you're trying to use JSMs in an SDK addon without looking for equivalent SDK APIs first. Note that the top-level MDN Addon page distinguishes several types of extensions, especially SDK and legacy extensions. You're writing an SDK extension.
So for the purpose of downloading images instead of going through the file download manager (Downloads.jsm) you can simply use the request and IO SDK modules to download the file and then child_process to spawn the exe.
Task.spawn(function () {
yield Downloads.fetch("http://www.mozilla.org/",
That's using yield outside a generator function, which is legacy syntax and should not be used.
For chaining you probably should use ES6 promises instead.

Receive EntryChangedEvent on folder

Currently i'm experementing with the the chrome.fileSystem-Api and i was curious about the new EntryChangedEvent that was added in Version 38. But im writing because i do not really know how to receive this event in my app. I tried it like this, what obviously didn't worked :
chrome.fileSystem.chooseEntry({type: 'openDirectory'}, function(folder) {
if (!folder) {
output.textContent = 'No Folder selected.';
return;
}
folder.on("entrychangedevent",function(v){
console.log(v);
});
});
How do i have to change my code so that i really can use receive EntryChangedEvents?
Thanks!
Link to documentation:
https://developer.chrome.com/apps/fileSystem#type-EntryChangedEvent

I actually think that this feature is not implemented yet. I don't know why it is in the documentation, but if you look at the chromium source there is a function called chrome.fileSystem.observeDirectory (which has a callback that should get these events) but when looking at the implementation it just says:
bool FileSystemObserveDirectoryFunction::RunSync() {
NOTIMPLEMENTED();
error_ = kUnknownIdError;
return false;
}
I found this document which is a request for the API:
https://docs.google.com/document/d/1aW-37JOBZgoD2CIPvoBEgw6bog8gFSowpfqtDEKt5Vo/edit#heading=h.w8inspeo32bj
Anyways, great feature and will probably be available soon.

Three issues here:
It's not clear from the documentation what object this listener should be attached to. In your example, you have attached it to the DirectoryEntry representing a directory. While I don't know the answer, that sounds like it's probably wrong, as a DirectoryEntry is only a means for accessing a directory, and is not itself a directory.
The documentation says that this event is fired for only certain filesystems, and I guess neither of us knows whether it is even effective for whatever file systems we're testing with.
In your example, even if the code were right and the event were enabled, you don't have any changes to the directory, so the event wouldn't fire.

Node.js changing exports on the fly

changing exports.X in a function seems to not work...
I want to be able to load settings from a file & access them in Node.js. I have this currently, however, the clients connecting to my node application can edit what's in the settings file. Unfortunately as it stands the Node application has to be restarted for the changes to take effect. Is there a way I can reload the module.exports on the fly?
EDIT:
Settings file is literally a JSON string.
My settings module is 'required' in almost every single file, and there's a lot of files... So reloading it per-file basis is out of the question. I do, however, know precisely when someone makes a change to the settings.

If you are using require to load the settings and only referencing the settings from one module, then doing something along the lines of:
delete require.cache[require.resolve(filename)];
will work for you.
If, on the other hand, multiple modules will be referencing these settings, that approach can become a bit unwieldy and open you up to unforeseen bugs. For example, if any of the modules are holding on to a reference to the required settings file, they would each need to somehow learn that the settings had changed and update their references.
To alleviate (not completely solve) the caching issue, you build your settings interface so that users of it must access either the settings object via a function and/or require that properties are accessed via functions. Even with this model, someone may still decide to cache a setting causing an obscure failure later down the road.
Using the simplest approach of a single getter for the settings object would look something like this:
var settings = require('./settings.json');
// ... watch for changes and reload by invalidating node's cache
module.exports = function() { return settings; }
Usage:
var settings = require('./path/to/settings');
settings().foo;
There are several libraries that do settings. Depending on your needs, I'm partial to nconf.

I'd set up a file watcher here that checks for changes of a JSON file dynamically. It is not recommended practice to change a JS script once the app is running.
Something like:
var _ = require("lodash");
var fs = require("fs");
var result = {};
fs.watch('my-settings.json',function(event,filename){
fs.readFile(filename,function(err,data){
if(err){
// your error catching
}
_.extend(result,JSON.parse(data));
});
});
module.exports = result;
Now, this comes with lots of caveats, first that fs.watch is not always supported by all platforms.
http://nodejs.org/api/fs.html#fs_fs_watch_filename_options_listener
Second, that it's really awkward to change a property like this. The expectation is generally that exports of module not mutate. I'd instead recommend exposing a method whose result can change based on the state of the file, a getter for the resulting data.
Third, a file watcher can be expensive, memory-wise.
This is better code, IMHO:
var _ = require("lodash");
var fs = require("fs");
var filename = 'my-settings.json';
var lastModified;
var mySetting;
module.exports = {
getSettingAsync : function (callback) {
fs.stat(filename,function(err,stat){
if(stat.mtime == lastModified) {
callback(mySetting);
} else {
fs.readFile(filename,function(err,data){
if(err){
// your error catching
}
// this assumes that your data is always correct
mySetting = JSON.parse(data).mySetting;
callback(mySetting);
});
}
});
}
};
In this case, we both check for a JSON file, and expose this as an async method. You could just as easily change the code to use the sync versions if need be and return the value instead of invoking the callback. This version checks when the file was changed, which is cheaper than reading the whole file every time, reads the file if newer and saves you the need to use a potentially buggy file watcher.
By the way, I've not tested this code and it may contain errors as is, but the concept is sound.
But, perhaps the more salient question, why not just store that value in the database?

Dynamically Included Javascript and Dependencies

So, as a sort of exercise for myself, I'm writing a little async script loader utility (think require.js, head.js, yepnope.js), and have run across a little bit of a conundrum. First, the basic syntax is like this:
using("Models/SomeModel", function() {
//callback when all dependencies loaded
});
Now, I want to know, when this call is made, what file I'm in. I could do it with an ajax call, so that I can mark a flag after the content loads, but before I eval it to mark that all using calls are going to be for a specific file, then unset the flag immediately after the eval (I know eval is evil, but in this case it's javascript in the first place, not json, so it's not AS evil). I'm pretty sure this would get what I need, however I would prefer to do this with a script tag for a few reasons:
It's semantically more correct
Easier to find scripts for debugging (unique file names are much easier to look through than anonymous script blocks and debugger statements)
Cross-domain requests. I know I could try to use XDomainRequest, but most servers aren't going to be set up for that, and I want the ability to reference external scripts on CDN's.
I tried something that almost got me what I needed. I keep a list of every time using is called. When one of the scripts loads, I take any of those using references and incorporate them into the correct object for the file that just loaded, and clear the global list. This actually seems to work alright in Firefox and Chrome, but fails in IE because the load events seem to go off at weird times (a jQuery reference swallowed a reference to another type and ended up showing it as a dependency). I thought I could latch on to the "interactive" readystate, but it doesn't appear to ever happen.
So now I come asking if anybody here has any thoughts on this. If y'all want, I can post the code, but it's still very messy and probably hard to read.
Edit: Additional usages
//aliasing and multiple dependencies
using.alias("ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js", "jQuery");
using(["jQuery", "Models/SomeModel"], function() {
//should run after both jQuery and SomeModel have been loaded and run
});
//css and conditionals (using some non-existant variables here)
using.css({ src: "IEFix", conditionally: browser === "MSIE" && version < 9 });
//should include the IEFix.css file if the browser is IE8 or below
and to expound more on my response below, consider this to be file A (and consider the jquery alias from before to be there still):
using(["jQuery", "B"], function() {
console.log("This should be last (after both jQuery and B have loaded)");
console.log(typeof($));
});
Then this would be B:
using("C", function() {
console.log("This should be second");
});
And finally, C:
console.log("This should be first");
The output should be:
This should be first
This should be second
This should be last (after both jQuery and B have loaded)
[Object Object]

Commendable that you are taking on such an educational project.
However, you won't be able to pull it off quite the way you want to do it.
The good news is:
No need to know what file you are in
No need to mess with eval.
You actually have everything you need right there: A function reference. A callback, if you will.
A rough P-code for your using function would be:
function using(modules, callback) {
var loadedModules = []
// This will be an ajax call to load things, several different ways to do it..
loadedModules[0] = loadModule(modules[0]);
loadedModules[1] = loadModule(modules[1]);
// Great, now we have all the modules
// null = value for `this`
callback.apply(null, loadedModules);
}