Nodejs Persistent Variable - javascript

Is there some sort of Session-like variable to hold an array in Nodejs?
What I meant is like something where I can define the name in other scope and be accessed in different scope (i.e: Variable("Array1") is defined in function A but accessed in function B and persists until it is destroyed).
The reason is I am using Meteor for slicing big files into small blobs and pass it back the chunk to the server. I tried to use the combination of fs.WriteFile and fs.AppendFile but somehow the file is mutilated along the way (the file is a video and playback error occurred with the copied file).
I read somewhere that blob can be rebuild by the constructor. However, I would need to pass this to a global or session-like variable in order to do so.
So...how can I use such thing in Nodejs?

There is such thing – it is called database :-)
When you're in Meteor, all files are loaded to a single running environment. Therefore, unlike in plain Node, a global variable created in one file can be accessed in any other one. So you can write
Slices = {};
in one file, and then in another say
Slices['Array1'] = ...
Notice there is no var keyword when defining the Slices object, otherwise it wouldn't be global but scoped to the file.
There is obviously one problem with the above method, and it's persistence over server reload. When the server crashes and restarts, or when you upload a new version, all such variables are recreated and you lose your data.
To prevent this, you need to store your variables in a place where they are retained permanently – a database of some kind. There are several solutions tailored for such runtime variables (such as Redis), but since you're using Meteor the natural solution would be to use the provided Mongo database. So just create a new collection on the server side
Slices = new Meteor.Collection('slices');
and use the usual find, insert, update and remove methods to access your variables.

If everything happens in the same process space, you can use a module as a singleton.
Remember, even if a module is included multiple times, the same copy is returned.
So if you have this module:
module.exports = new Array();
And you include it by several other modules, each one of them will have the same array instance.
You can also have a more complex singleton:
var blocks = {};
module.exports.addBlock = function(name, block) {
blocks[name] = block;
};
module.exports.getBlock = function(name) {
return blocks[name];
};
module.exports.delBlock = function(name) {
delete blocks[name];
};
module.exports.list = function() {
return Object.keys(blocks);
};
In your different files, you would include and use this module like:
var blocks = require('./blocks');
console.log(blocks.list());
Read about module caching here.

Related

Attempting to Import Module in Child Process (Javascript) and Failing

I'm currently running a heavy computation (i.e. generating a Monte Carlo tree), which is an expensive operation. I only have a few seconds to build as big of a tree as I can, so I am using subprocesses in Node.js in order to build multiple trees, and then aggregate their data together to make a more informed decision.
I understand that subprocesses do not share information/memory, and I need to use modules within these subprocesses that are located in a file, called "Epilog.js" on my machine.
When I run functions that are in epilog.js from the main file, it works just fine. But all of my functions that are in my worker threads return absolutely nothing.
I have tested to make sure that the parameters of the functions I am trying to use in "epilog.js" aren't empty, and they're not. The problem isn't in the parameter.
I have also tested to see what happens if I simply don't import, and instead of just outputting an undefined array, I get an error saying that there is no function called "findroles".
//My main thread.
var fs = require('fs');
eval(fs.readFileSync('epilog.js') + '');
var process = fork('./buildGraph.js');
process.send({library});
//My worker thread.
//buildGraph.js
var fs = require('fs');
eval(fs.readFileSync('epilog.js') + '');
// receive message from master process
process.on('message', async(message) => {
library = message["library"];
console.log(findroles(library));
// findroles(library) is a function that is defined in epilog.js,
//and this outputs an array of "roles" given a parameter,library.
// For some reason this function outputs [], rather than giving me
// all of the roles. If I run this exact line from my main thread,
// it doesn't give any errors and outputs the right array:
// e.g. ['red', 'white'].
});
I expect to get not the empty array, but [red, white], as I do if I were to run the same line in the main thread. Does anyone have an idea as to the inconsistency of the functions? I'm very new to node.js and this isn't a class focused too much on software engineering in JavaScript, so I'd appreciate if someone can dumb down what is going on, as this is all very new to me.
If your script does not find the function called findroles then there is a problem with the importing method. Using the eval function for importing is not the normal way of importing modules. Try something like this:
// buildGraph.js
const epilog = require("./epilog.js");
......
console.log(epilog.findroles(library));
then epilog.js
exports.findroles = function (library) {
// function content
}
You can find more info here:
https://www.w3schools.com/nodejs/nodejs_modules.asp
Base on the document and example here, everything seem correct but I think the problem come from this line:
var process = fork('./buildGraph.js');
you might override the original process.
try to change it to
const n = fork('./buildGraph.js');

Unable to refer javascript array from another javascript file

I am trying to create a chart using highcharts in which i am not able to fetch data which i store in another js file.
My main.js file has all the code for creating chart. It has the section of series.
series: [{
name: 'Desktops',
data: '/data/desktop.js',
tooltip: {
valueDecimals: 2
}
}]
Here i want to refer data from another js file. The name of file is desktop.js and it has just the below array
var desktopData = [[1475272800000, 117759], [1475359200000, 106147], [1475445600000, 147747], [1475532000000, 302031], [1475618400000, 520539], [1475704800000, 245353], [1475791200000, 180376], [1475877600000, 78819], [1475964000000, 90466], [1476050400000, 257822], [1476136800000, 284465], [1476223200000, 242898], [1476309600000, 297186], [1476396000000, 268069], [1476482400000, 183149], [1476568800000, 410442], [1476655200000, 1117798], [1476741600000, 1274668], [1476828000000, 1331799], [1476914400000, 1230213], [1477000800000, 888251], [1477087200000, 572050], [1477173600000, 931144], [1477260000000, 1556641], [1477346400000, 1526736], [1477432800000, 1310133], [1477519200000, 1207422], [1477605600000, 785556], [1477692000000, 487264], [1477778400000, 787714], [1477868400000, 942663]];
How can i refer array defined in another file?
Do you use any type of the framework? To load JS file in another file, you need to require it somehow, either by using AMD, ES imports or any other way (be it synchronous or asynchronous).
If you are just adding the files to the HTML (using script tags), then the only thing you have to do is to make sure that the files are loaded (DOMContentLoaded event) and then, you have to attach the data to a global scope.
Global scope in the browsers is called a window and every time when you use variable that was not declared in your scope by var it means that the JS is searching for it in the global scope. In order to assign something to a global scope you can just omit var or explicitely assign it like this:
window.desktopData = [];
// And then in another file, assuming that the data is ready
var myData = window.desktopData
Please also remember that including files in <script> tags usually means that they will block your browser and the order of the files matters (at least when you want to rely on the fact of one file already loaded before another file). I'd suggest using some kind of import system then:
http://requirejs.org/docs/whyamd.html

How does live object creation and partial teardown management work in javascript?

What I would like to do is load javascript to create a library of methods in an object and wait until the object is used for the first time before it is actually defined or compiled. I would like to build references to this object before it is actually fully defined. When I call a method on this object for the first time before the methods on the object are ever defined (meaning the object doesn't actually have methods) I would like to define the object and then call the method. Is there a way to do this using standard syntax such as "MyLibrary.sayHello()" if "sayHello()" is not yet defined on the object.
I imagine it would look like this:
var independentVar = "noCommitments";
var MyLibrary = function(user_ini){
//MyLibrary.init looks like
// (function(ini){
// var a = ini;
// return function(){
// //Notice the method sayHello defines when called,
// // and does not return a reference
// return {
// b:a,c:"c",sayHello:function(z){return "Hello"+a+z}
// }
// }
// })(user_ini);
var d1 = myRequire("MyLibrary.init");
return {
**handleAll : function(){ this = d1(); this.("**calledMethod")}
}
};
var greeting = MyLibrary.sayHello();
alert(greeting);
This is only pseudo-code. If I add a method to cleanup I can then return that object to the uninitialized state of "{**handleAll:function(){/noContext/}}". My application/library has a stub and a link this way and can be used immediately from an undefined state, when building modules this can be useful in order to lower the number of references to a utility, say a post has a menu of functions and those functions are shared by by all posts, -- with a mechanism such as is described here only the "active post"/"post in focus" will reference the utility. It moreless give the ability to activate and de-activate modules. The special part is the modules are already warmed up, they are ready to call functions even though they do not reference them yet, it is similar to live binding but allows the whole user interface to already be defined with functions already stubbed out with the exact name they will have when they are usable. A control mechanism for defaults and debounce is easily found in this model for me.
My question is: Is this type of scripting possible natively or will I have to use some form of compilation like for TypeScript, CoffeeScript or others. I understand it is possible if I pass the method I would like to call as a parameter to a singleton factory. I ultimately would like whole applications that are able to gracefully degrade unused functionality without polluting the code.
What I mean by pollution:
var LibDef = (
function(){
return {
callUndefined:function(methodName){
var returnVal = {}
}
}
})()
var MySingltonLibrary = moduleSinglton.getLibrary("MyLibrary", Lib);
var greeting = MySingltonLibrary.callUndefined("sayHello");
//
// Please use your imagination to consider the complexity in the singlton
The best way that will allow you to tear down an object releasing any space its functions and members consume on the heap and maintain a single reference, that will allow the object to rebuild itself or just rebuild the function that is called is like this - (A very simple model, you may like to use arrays and gradually tear down nested objects internally):
var twentySecondObj(function(window,document){
var base_obj = undefined;
var externalAPI = undefined;
setTimeout(function(){
base_obj = undefined;
},20000);
return function(){
base_obj = (function(base_obj){
if(base_obj === undefined){
return {
property1:"This is property1",
property2:"This is property2"
}
}
})();
externalAPI = (function(){
if(externalAPI === undefined){
return {
property1:base_obj.property1,
property2:base_obj.property2
}
}
})();
return externalAPI;
}
})(window,document);
console.log(twentySecondObj().property1);
On an additional note, you can use getters and setters to observe access to properties and can internally present a facade of both functions and properties which reference a build method like the one above, this way it looks like you are accessing a legit member of the object. There are no options I can think of that will allow you to intercept when attempt to set a new property on an object like: myObj.fooProperty = "foo", and buildup that property into a custom object with a getter and setter, if you have a custom type that needs to be set, then you will have to know it's implimentation details to set it, or call a function passing in the property name and value, or use a method similar to what is shown above.
Here is a link to the proposal for adding weak references to javascript: https://ponyfoo.com/articles/weakref weak-references would alter how this looks, however would not address everything mentioned in this question. Remapping an object when a property is added via some type of deep observer will allow new property members to be enhanced at the time they are set, this would require that the observer ran synchrounously when the property was set, or once the set is complete, the very next statement must be a call to update the object. I will keep posted here for any advances I see that will make the "default handler function" available within javascript in the future.
WeakRef can absolutely be used for recording and handling object usage. I would really like to move object management into webworkers and service workers so they can be maintained through all web endpoints on the domain and do not require to reload across requests. Web frameworks would need to have modified handle to offload all dom changes and updates to worker, essentially a single hook that handles message passing for all hooks. Modload, now must include a message handle name and have task priority meta data so it is properly placed in the least busy or least active worker (slow worker and fast worker) this helps to create an api that can offload to cloud functions, this shpuld give us ability to do more AI, lookups and work offline that is currently handled for most apps in the cloud where more processing power is, and in this way we can gracefully augment local processing with cloud functions only when local resources, or completion times are degraded below acceptable speeds, or above acceptable power policy.
https://v8.dev/features/weak-references

How to parse and load javascript object?

I have one js files . I load it using other javascrupt file using eval() function. I have seen eval is slow and with some other limtation. Since i need to store my JS file object in cache and use it anytime i need after apllication starts. I dont want to do eval() everytime.
Is there anyway to do it in simple way.
var evalObj;
if(evalObj) {
console.log('eval object already evaluated');
_myfunctionInJSFile_(layouts.FormatDate(startTime), threadName, level, categoryName, message);
}
else {
evalObj = eval(fs.readFileSync('./myJSFile', 'utf8'));
console.log('re evaluating object ..' );
_myfunctionInJSFile_(layouts.FormatDate(startTime), threadName, level,message);
}
myJSFile
var _sigmaAlarmHandler_ =function(args)
{
var args = Array.prototype.slice.call(arguments);
args.unshift();
console.log('Alarm : ', args);
}
Either the conditional eval is not working.
In node.js you can simple require your js-file:
var obj = require('./myJSFile');
obj.foo();
./myJSFile.js:
exports.foo = function() {
console.log('foo');
}
This file becomes a module with exported functions, that you need.
It loads once, then every require reuse already loaded module.
If it is not commonjs-compliant (i.e. using module.exports will not work), then you can run it in its own vm:
var vm = require('vm');
vm.runInNewContext(jscode,{/*globalvars*/});
where the second parameter is an object with global vars made available in the context in which the jscode is run. So if the second param is, say, {a:1,b:"foo"} then your jscode will run with the global variable a set to 1 and the global variable b set to "foo".
The jscode itself is a string that you load from a file or elsewhere.
Think of vm.runInNewContext() as "practice safe eval". Well, relatively safe, you can still do some dangerous stuff if you pass in particular vars, like process or file etc.
I used this for the declarative part of cansecurity http://github.com/deitch/cansecurity for nodejs
You can view the sample in the file lib/declarative.js
Here is the API for vm http://nodejs.org/api/vm.html
There are options to run in the same context, etc. But that is very risky.
When you actually run the code, using your example above:
_myfunctionInJSFile_(layouts.FormatDate(startTime), threadName, level,message);
you are looking to pass in 4 params: startTime, threadName, level, message and execute the function. The issue is that you cannot run the function on the current context. You need the function to be defined and run in the file. So you should have something like:
vm.runInNewContext(jscode,{startTime:layouts.FormatDate(startTime),threadName:threadName,level:level,message:message});
And then the jscode should look like
function _myfunctionInJSFile(startTime,threadName,level,message) {
// do whatever you need to do
}
// EXECUTE IT - the above vars are set by the global context provide in vm.runInNewContext
_myfunctionInJSFile(startTime,threadName,level,message);
If you prefer to define the function and have it loaded and run in this context, then just use the commonjs format.
I think i have found the answer for this.
Since my application is running in node js which uses v8 engine platform. When the application starts v8 engine caches all the code/configuration and can be used anytime.
Similarly in my code i will pre-load the JS code using eval and i will do it only once. So on next call i will return only the loaded JS code. Here i need to modify the code to load once.
But main point we have look is that in future if any body has similar requirement they can cache their JS codes using eval (thanks to v8 engine) and use it till your application is running.

Node.js changing exports on the fly

changing exports.X in a function seems to not work...
I want to be able to load settings from a file & access them in Node.js. I have this currently, however, the clients connecting to my node application can edit what's in the settings file. Unfortunately as it stands the Node application has to be restarted for the changes to take effect. Is there a way I can reload the module.exports on the fly?
EDIT:
Settings file is literally a JSON string.
My settings module is 'required' in almost every single file, and there's a lot of files... So reloading it per-file basis is out of the question. I do, however, know precisely when someone makes a change to the settings.
If you are using require to load the settings and only referencing the settings from one module, then doing something along the lines of:
delete require.cache[require.resolve(filename)];
will work for you.
If, on the other hand, multiple modules will be referencing these settings, that approach can become a bit unwieldy and open you up to unforeseen bugs. For example, if any of the modules are holding on to a reference to the required settings file, they would each need to somehow learn that the settings had changed and update their references.
To alleviate (not completely solve) the caching issue, you build your settings interface so that users of it must access either the settings object via a function and/or require that properties are accessed via functions. Even with this model, someone may still decide to cache a setting causing an obscure failure later down the road.
Using the simplest approach of a single getter for the settings object would look something like this:
var settings = require('./settings.json');
// ... watch for changes and reload by invalidating node's cache
module.exports = function() { return settings; }
Usage:
var settings = require('./path/to/settings');
settings().foo;
There are several libraries that do settings. Depending on your needs, I'm partial to nconf.
I'd set up a file watcher here that checks for changes of a JSON file dynamically. It is not recommended practice to change a JS script once the app is running.
Something like:
var _ = require("lodash");
var fs = require("fs");
var result = {};
fs.watch('my-settings.json',function(event,filename){
fs.readFile(filename,function(err,data){
if(err){
// your error catching
}
_.extend(result,JSON.parse(data));
});
});
module.exports = result;
Now, this comes with lots of caveats, first that fs.watch is not always supported by all platforms.
http://nodejs.org/api/fs.html#fs_fs_watch_filename_options_listener
Second, that it's really awkward to change a property like this. The expectation is generally that exports of module not mutate. I'd instead recommend exposing a method whose result can change based on the state of the file, a getter for the resulting data.
Third, a file watcher can be expensive, memory-wise.
This is better code, IMHO:
var _ = require("lodash");
var fs = require("fs");
var filename = 'my-settings.json';
var lastModified;
var mySetting;
module.exports = {
getSettingAsync : function (callback) {
fs.stat(filename,function(err,stat){
if(stat.mtime == lastModified) {
callback(mySetting);
} else {
fs.readFile(filename,function(err,data){
if(err){
// your error catching
}
// this assumes that your data is always correct
mySetting = JSON.parse(data).mySetting;
callback(mySetting);
});
}
});
}
};
In this case, we both check for a JSON file, and expose this as an async method. You could just as easily change the code to use the sync versions if need be and return the value instead of invoking the callback. This version checks when the file was changed, which is cheaper than reading the whole file every time, reads the file if newer and saves you the need to use a potentially buggy file watcher.
By the way, I've not tested this code and it may contain errors as is, but the concept is sound.
But, perhaps the more salient question, why not just store that value in the database?

Categories