process npm module in Node and accessing another file that does computations for me. The problem is that when on. message event/callback I am not sure what it is actually but there i am trying to access global variable and it says it is undefined. If somebody have can have a good explained solution.
_addBlock(newBlock)
{
newBlock.previousHash = this._getLatestBlock().hash;
var child =
childProcess.fork('C:\\Users\\Yoana\\WebstormProjects\\ChildProcess\\mining1.js'
);
child.on('message', function(newBlock)
{
// Receive results from child process
console.log('received: ' , newBlock);
this.chain.push(newBlock);
})
// Send child process some work
child.send(newBlock);
}
It says that this.chain.push is undefined. The method _addBlock is part of a class Blockchain and this.chain is globally accessible.
I'm not sure which model are you using i.e. node.js master/worker architecture with cluster native module or child_process native module with message passing etc., by the way despite of sharing globals it is not recommended (how to handle the shared memory? how to handle protected memory?), you can do in this way:
global.GlobalBotFactory = function() {
if (typeof(instance)=="undefined")
instance = new MyClass(options);
return instance;
}
and then you can reference it in other files like
this.instance = GlobalBotFactory(); // the shared factory instance
But this approach, despite it works, could led to several issues like
concurrent variable modification
shared memory corruption
reader/writer problem
etc. so I strongly suggest to follow a node cluster module with master/worker approach and then message passing:
/// node clustering
const cluster = require('cluster');
const numCPUs = require('os').cpus().length;
if (cluster.isMaster) { // master node
var masterConfig=require('./config/masterconfig.json');
// Fork workers.
var maxCPUs = masterConfig.cluster.worker.num;
maxCPUs=(maxCPUs>=numCPUs)?numCPUs:maxCPUs;
for (let i = 0; i < maxCPUs; i++) {
const worker=cluster.fork();
}
var MasterNode=require('./lib/master');
var master= new MasterNode(masterConfig);
master.start()
.then(done=> {
console.log(`Master ${process.pid} running on ${masterConfig.pubsub.node}`);
})
.catch(error=> {
console.error(`Master ${process.pid} error`,error);
});
}
else if (cluster.isWorker) { // worker node
var workerConfig=require('./config/workerconfig.json');
var WorkerNode=require('./lib/worker');
var worker= new WorkerNode(workerConfig);
worker.start()
.then(done=> {
console.log(`Worker ${process.pid} running on ${workerConfig.pubsub.node}`);
})
.catch(error=> {
console.error(`Worker ${process.pid} error`,error);
});
}
For the message passing part take care since you will deal with async forked process, and in node.js there is not guarantee that a message will be delivered, so you need a ack logic or you can use a pubsub approach (Redis will offer this for free, please check here), by the way here you are
for (var i = 0; i < 2; i++) {
var worker = cluster.fork();
// Receive messages from this worker and handle them in the master process.
worker.on('message', function(msg) {
console.log('Master ' + process.pid + ' received message from worker ' + this.pid + '.', msg);
});
// Send a message from the master process to the worker.
worker.send({msgFromMaster: 'This is from master ' + process.pid + ' to worker ' + worker.pid + '.'});
}
this will fork the workers and listen for incoming messages from the master or other workers. But please keep in mind that the delivery logic it's up to you. See here for more info about subprocess.send.
(note, I'm using Node.js to create a telnet server and handle user input)
I'm having issues using variables/arrays in javascript inside functions in other scripts.
Let me set a simpler example:
var connections = []
is an array in my main.js
Broadcast.js is a function I put in a separate module and attempt to use the broadcast() function in my main.js.
The error I get is stating that connections is undefined. How do get Broadcast.js able to see the connections array in the main.js
~~
For example in my main.js I set an array to handle clients connecting to a server
//point to Telnet library
const Telnet = require('ranvier-telnet');
const logger = require('./logger.js')
var outspeak = []
var connections = []
var clients = []
let server = new Telnet.TelnetServer(rawSocket => {
let telnetSocket = new Telnet.TelnetSocket();
//attaching socket
telnetSocket.attach(rawSocket);
//setting telnet options
telnetSocket.telnetCommand(Telnet.Sequences.WILL, Telnet.Options.OPT_EOR);
//giving clients a name
telnetSocket.name = rawSocket.remoteAddress + ":" + rawSocket.remotePort
//pushing client names to array
clients.push(telnetSocket.name);
//pushing client connections to an array
connections.push(rawSocket);
console.log(`${telnetSocket.name} has connected`)
logger(`${telnetSocket.name} has connected`)
broadcast(telnetSocket.name + " connected.")
telnetSocket.on('data', function (data) {
//broadcast (telnetSocket.name + ">" + data, telnetSocket);
}
function broadcast (message, sender) {
connections.forEach(function (connection) {
//don't want to send it to sender
if (connection === sender) return;
connection.write(`${message} \n`);
});
}
Now inside my main script, I could called that array/push to that array, read from that array, as long as I type out the function inside the main.js file.
And it can easily use the broadcast function.
Now I want to make it more advance and make reduce my lines on my main.js
but once I separate the broadcast function into it's own module.
use strict'
//broadcast function
function broadcast (message, sender) {
connections.forEach(function (connection) {
//don't want to send it to sender
if (connection === sender) return;
connection.write(`${message} \n`);
});
}
module.exports = broadcast
I get a connection undefined error any time I try to invoke that broadcast function. It's like my global variable/array can't be seen by broadcast.js function.
this is how I'm invoking it
// handle input
telnetSocket.on('data', function (data) {
broadcast (telnetSocket.name + ">" + data, telnetSocket);
});
And yes, const broadcast = require('./broadcast.js'); as been added to the file at the top.
Here's the broken code complete:
'use strict'
//point to Telnet library
const Telnet = require('ranvier-telnet');
const logger = require('./logger.js');
const broadcast = require('./broadcast.js');
var connections = []
var clients = []
//had to call message as global variable
//Asan's timestamp functionm
//telnetstuff
console.log("Starting...");
let server = new Telnet.TelnetServer(rawSocket => {
let telnetSocket = new Telnet.TelnetSocket();
//attaching socket
telnetSocket.attach(rawSocket);
//setting telnet options
telnetSocket.telnetCommand(Telnet.Sequences.WILL, Telnet.Options.OPT_EOR);
//giving clients a name
telnetSocket.name = rawSocket.remoteAddress + ":" + rawSocket.remotePort
//pushing client names to array
clients.push(telnetSocket.name);
//pushing client connections to an array
connections.push(rawSocket);
console.log(`${telnetSocket.name} has connected`)
logger(`${telnetSocket.name} has connected`)
broadcast(telnetSocket.name + " connected.")
// handle input
telnetSocket.on('data', function (data) {
broadcast (telnetSocket.name + ">" + data, telnetSocket);
});
//removing client/connection from array
rawSocket.on('end', function () {
clients.splice(clients.indexOf(telnetSocket), 1);
connections.splice(connections.indexOf(rawSocket), 1);
broadcast(telnetSocket.name + " has left.\n");
logger(telnetSocket.name + " has left.");
console.log(telnetSocket.name + " has left.");
});
}).netServer
server.listen(4000);
console.log('ServerRunning...');
logger('>Server started.');
What I'm missing here? Also I apologize in advance this is my first question ever asked and I've gone through as much I could today to even figure out how to ask my question, maybe I'm not using correct lingo/terms? any help is appreciative.
refactor\broadcast.js:5
connections.forEach(function (connection) {
^
ReferenceError: connections is not defined
In nodejs, when you declare a variable not inside any function definitions, it is scoped to the file only. (This is different from browser javascript.) If you want something to be accessible from outside, you need to export it:
module.exports.connections = connections;
Then import it into the other file:
const connections = require(myFile);
This will work as long as you don't try to set the value of the variable in either file, but if you do that they'll end up pointing to separate objects. But mutating it, calling methods on it, etc should work fine.
I have a simple case where I'm requesting a different upstream proxy server from my node.js server. With the increase in load I see the request takes lot of time to execute(though time taken to respond from my upstream proxy server is constant across the requests). To demonstrate the issue i've written a sample program as below. When I execute the below program, the first request takes 118ms to execute and the last one takes 10970ms depending on the website you hit (I've changed the url to google, Try it out with your favourite website). If you observe i'm using async to parallelize my requests.
The question is, what is the reason node.js takes this much time to execute a request when run in parallel. To give some more context on the infra settings(centos 6.5) I have opened up my port range from 1024 to 65535, change the fin_timeout to 15 seconds and enable tw_reuse =1 for sockets in sysctl.conf
var http = require('http');
var uuid = require('node-uuid');
var async = require('async');
function callExternalUrl(){
var uniqueId = uuid.v4();
console.time(uniqueId);
var options = {
host: 'google.com',
port: '80',
path: '/',
method: 'GET'
};
var req = http.request(options, function(res) {
var msg = '';
res.setEncoding('utf8');
res.on('data', function(chunk) {
msg += chunk;
console.timeEnd(uniqueId);
});
res.on('end', function() {
});
});
req.end();
}
function iterateAsync(callback){
var iter = [];
for(var i=0; i<1000; i++){
iter[i] = i;
}
async.each(iter,
function(item, callback) {
callExternalUrl();
},
function(err) {
callback(err);
}
);
}
iterateAsync(function(){console.log('done');});
To give more context below is the code in ruby to do the same. I understand i can't compare these two languages as in apples to apples. But the idea is to show the time it takes to execute the same requests in sequence using ruby. I don't see any increase in the response times for each request going out in sequence. So, I doubt the parallel requests using node is taking more time for the request to respond(and the issue is not from the server to respond but its from sending out the request from the machine itself)
require 'rest_client'
1000.times do |number|
beginning = Time.now
response = RestClient.get 'http://google.com'
puts "Time elapsed #{Time.now - beginning} seconds"
end
For one, you're not calling the async iterator callback function:
function callExternalUrl(asyncCallback) {
...
res.on('end', function() {
asyncCallback();
});
...
}
function iterateAsync(callback) {
var iter = [];
for(var i=0; i<1000; i++){
iter[i] = i;
}
async.each(iter,
function(item, asyncCallback) { // <-- HERE
callExternalUrl(asyncCallback);
},
function(err) {
callback(err);
}
);
}
Also, depending on the Node version you're using, the http module may limit the number of parallel requests being made to a particular hostname:
$ node -pe 'require("http").globalAgent.maxSockets'
On Node 0.10, the default is 5; on Node 0.12, the default is Infinity ("unlimited"). So if you're not on Node 0.12, you should increase that value in your code:
var http = require('http');
http.globalAgent.maxSockets = Infinity;
...
I've tried to run your scenario by using JXcore (fork of Node.JS, and an open source project now on github), which offers multitasking (among many other new features).
var task = function (item) {
var http = require('http');
var uuid = require('node-uuid');
var uniqueId = uuid.v4() + "-" + process.threadId;
console.time(uniqueId);
var options = {
host: 'google.com',
port: '80',
path: '/',
method: 'GET'
};
var req = http.request(options, function (res) {
var msg = '';
res.setEncoding('utf8');
res.on('data', function (chunk) {
msg += chunk;
console.timeEnd(uniqueId);
});
res.on('end', function () {
process.release();
});
});
req.end();
process.keepAlive();
};
jxcore.tasks.setThreadCount(4);
console.time("total");
process.on('exit', function () {
console.timeEnd("total");
});
for (var i = 0; i < 1000; i++)
jxcore.tasks.addTask(task, i);
The sample is not really optimized, but still the total 1000 requests runs with JXcore a little bit faster for me (I was able to measure up to 20% gain on my platform). That may vary depending on the machine, since multitasking is using different threads/instances within one single process (no need for clustering any more). My machine has just 4 threads, that's why I used jxcore.tasks.setThreadCount(4);. You can try with your 32 :)
The way of handling each single request is not significantly different, so I'm not saying that each request takes less time, but the key might be hidden in different queuing mechanism as opposite to "async" module. And of course thanks to multitasking.
Is it possible to use socket.io to dispatch events to the client and not send them to the server? Something like:
socket.emitLocally('some event', data);
Here is why I'm asking:
At the moment my server emits events to all sockets.
io.sockets.emit()
While this works well, there is the possibility of a delay between the user interaction and the response from the server.
This is why I would prefer to use broadcast on the server side and handle things immediatly in the broadcasting client.
socket.broadcast.emit()
The client is based on angular.js and the modules don't know of each other. I can't access the code that is responsible for updating the client directly. I would have to use some kind of event dispatching service which I would inject into relevant modules.
Since this is basically what I am doing with socket.io right now I wonder if I could not simply use their system to do this.
Thank you for any input or suggestions!
This can be done as shown on the socket.io client code on function Emitter.prototype.emit.
Events are saved on socket._callbacks with the prefix $, so for instance "connected" would be under _callbacks as "$connected".
You can replicate the behaviour of said function by using:
// You can use internal (e.g. disconnected) or custom events (e.g. ChatMessage).
var event = "MyEvent";
// Set your args exactly as in your socket.io server emit.
// Note that they will be used on an apply so they must be within an array.
var args = [1, 2, 3];
if(socket._callbacks) { // If the socket has callbacks
var callbacks = socket._callbacks['$' + event]; // Load the callback event
if (callbacks) {
callbacks = callbacks.slice(0);
for (var i = 0, len = callbacks.length; i < len; ++i)
callbacks[i].apply(socket, args);
} // Otherwise no calls
}
In the example above the callback for "MyEvent" will be called with the arguments 1, 2, 3.
A further example on how to use this on an actual connection:
io.on('connection', function (socket) {
socket.on("getSingleUpdate", function(id, data){
alert("Event #"+id+" data "+data);
});
function emitLocalEvent(event/*, arg1, arg2, etc.*/) {
if(socket._callbacks) {
var args = [].slice.call(arguments, 1),
callbacks = socket._callbacks['$' + event];
if (callbacks) {
callbacks = callbacks.slice(0);
for (var i = 0, len = callbacks.length; i < len; ++i)
callbacks[i].apply(socket, args);
}
}
}
// Emitting fake events
emitLocalEvent("getSingleUpdate", 1, "Hello");
emitLocalEvent("getSingleUpdate", 2, "World!");
// You can even receive an array of events via socket.io...
socket.on("getMultipleEvents", function(eventArray) {
for(var i in eventArray)
emitLocalEvent.apply(null, eventArray[i]);
});
});
Assuming you catch events on the client side using
io.on("name",callback);
you could just call
callback();
I hope i could help! =)
Is it possible to have a socket.io client respond to all events without to have specify each event individually?
For example, something like this (which obviously doesn't work right now):
var socket = io.connect("http://myserver");
socket.on("*", function(){
// listen to any and all events that are emitted from the
// socket.io back-end server, and handle them here.
// is this possible? how can i do this?
});
I want this callback function to be called when any / all events are received by the client-side socket.io code.
Is this possible? How?
Updated solution for socket.io-client 1.3.7
var onevent = socket.onevent;
socket.onevent = function (packet) {
var args = packet.data || [];
onevent.call (this, packet); // original call
packet.data = ["*"].concat(args);
onevent.call(this, packet); // additional call to catch-all
};
Use like this:
socket.on("*",function(event,data) {
console.log(event);
console.log(data);
});
None of the answers worked for me, though the one of Mathias Hopf and Maros Pixel came close, this is my adjusted version.
NOTE: this only catches custom events, not connect/disconnect etc
It looks like the socket.io library stores these in a dictionary. As such, don't think this would be possible without modifying the source.
From source:
EventEmitter.prototype.on = function (name, fn) {
if (!this.$events) {
this.$events = {};
}
if (!this.$events[name]) {
this.$events[name] = fn;
} else if (io.util.isArray(this.$events[name])) {
this.$events[name].push(fn);
} else {
this.$events[name] = [this.$events[name], fn];
}
return this;
};
Finally, there is a module called socket.io-wildcard which allows using wildcards on client and server side
var io = require('socket.io')();
var middleware = require('socketio-wildcard')();
io.use(middleware);
io.on('connection', function(socket) {
socket.on('*', function(){ /* … */ });
});
io.listen(8000);
Here you go ...
var socket = io.connect();
var globalEvent = "*";
socket.$emit = function (name) {
if(!this.$events) return false;
for(var i=0;i<2;++i){
if(i==0 && name==globalEvent) continue;
var args = Array.prototype.slice.call(arguments, 1-i);
var handler = this.$events[i==0?name:globalEvent];
if(!handler) handler = [];
if ('function' == typeof handler) handler.apply(this, args);
else if (io.util.isArray(handler)) {
var listeners = handler.slice();
for (var i=0, l=listeners.length; i<l; i++)
listeners[i].apply(this, args);
} else return false;
}
return true;
};
socket.on(globalEvent,function(event){
var args = Array.prototype.slice.call(arguments, 1);
console.log("Global Event = "+event+"; Arguments = "+JSON.stringify(args));
});
This will catch events like connecting, connect, disconnect, reconnecting too, so do take care.
Note: this answer is only valid for socket.io 0.x
You can override socket.$emit
With the following code you have two new functions to:
Trap all events
Trap only events which are not trapped by the old method (it is a default listener)
var original_$emit = socket.$emit;
socket.$emit = function() {
var args = Array.prototype.slice.call(arguments);
original_$emit.apply(socket, ['*'].concat(args));
if(!original_$emit.apply(socket, arguments)) {
original_$emit.apply(socket, ['default'].concat(args));
}
}
socket.on('default',function(event, data) {
console.log('Event not trapped: ' + event + ' - data:' + JSON.stringify(data));
});
socket.on('*',function(event, data) {
console.log('Event received: ' + event + ' - data:' + JSON.stringify(data));
});
As it is in v3.0 documentation:
socket.onAny((event, ...args) => {
console.log(`got ${event}`);
});
The current (Apr 2013) GitHub doc on exposed events mentions a socket.on('anything'). It appears that 'anything' is a placeholder for a custom event name, not an actual keyword that would catch any event.
I've just started working with web sockets and Node.JS, and immediately had a need to handle any event, as well as to discover what events were sent. Can't quite believe this functionality is missing from socket.io.
socket.io-client 1.7.3
As of May 2017 couldn't make any of the other solutions work quite how i wanted - made an interceptor, using at Node.js for testing purposes only:
var socket1 = require('socket.io-client')(socketUrl)
socket1.on('connect', function () {
console.log('socket1 did connect!')
var oldOnevent = socket1.onevent
socket1.onevent = function (packet) {
if (packet.data) {
console.log('>>>', {name: packet.data[0], payload: packet.data[1]})
}
oldOnevent.apply(socket1, arguments)
}
})
References:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function/apply
https://github.com/socketio/socket.io-client/blob/ff4cb3eed04a95c9725b8aaba8b64fa9fa1ca413/lib/socket.js#L257
Because your question was pretty general in asking for a solution, I'll pitch this one that requires no hacking the code, just a change in how you use the socket.
I just decided to have my client app send the exact same event, but with a different payload.
socket.emit("ev", { "name" : "miscEvent1"} );
socket.emit("ev", { "name" : "miscEvent2"} );
And on the server, something like...
socket.on("ev", function(eventPayload) {
myGenericHandler(eventPayload.name);
});
I don't know if always using the same event could cause any issues, maybe collisions of some kind at scale, but this served my purposes just fine.
There is a long discussion about this topic going on at the Socket.IO repository issue page. There are a variety of solutions posted there (e.g., overriding EventEmitter with EventEmitter2). lmjabreu released another solution a couple weeks ago: a npm module called socket.io-wildcard that patches in a wildcard event onto Socket.IO (works with the current Socket.IO, ~0.9.14).
Even though this is a old question, I have the same problem and solved using the native socket in Node.js, which has a event of .on('data'), fired everytime some data comes. So this is what I've done so far:
const net = require('net')
const server = net.createServer((socket) => {
// 'connection' listener.
console.log('client connected')
// The stuff I was looking for
socket.on('data', (data) => {
console.log(data.toString())
})
socket.on('end', () => {
console.log('client disconnected')
})
})
server.on('error', (err) => {
throw err;
})
server.listen(8124, () => {
console.log('server bound');
})
All methods I found (including socket.io-wildcard and socketio-wildcard) didn't work for me. Apparently there is no $emit in socket.io 1.3.5...
After reading socket.io code, I patched up the following which DID work:
var Emitter = require('events').EventEmitter;
var emit = Emitter.prototype.emit;
[...]
var onevent = socket.onevent;
socket.onevent = function (packet) {
var args = ["*"].concat (packet.data || []);
onevent.call (this, packet); // original call
emit.apply (this, args); // additional call to catch-all
};
This might be a solution for others as well. However, ATM I don't exactly understand why nobody else seems to have issues with the existing "solutions"?!? Any ideas? Maybe it's my old node version (0.10.31)...
#Matthias Hopf answer
Updated answer for v1.3.5. There was a bug with args, if you wanna listen on old event and * event together.
var Emitter = require('events').EventEmitter;
var emit = Emitter.prototype.emit;
// [...]
var onevent = socket.onevent;
socket.onevent = function (packet) {
var args = packet.data || [];
onevent.call (this, packet); // original call
emit.apply (this, ["*"].concat(args)); // additional call to catch-all
};
In v4, Socket.IO has Catch-all listeners. For example:
socket.prependAny(() => {
console.log("This will be fired first");
});
I'm using Angular 6 and the npm package: ngx-socket-io
import { Socket } from "ngx-socket-io";
...
constructor(private socket: Socket) { }
...
After connect the socket, I use this code, this is handling all custom events...
const onevent = this.socket.ioSocket.onevent;
this.socket.ioSocket.onevent = function (packet: any) {
const args = packet.data || [];
onevent.call(this, packet); // original call
packet.data = ["*"].concat(args);
onevent.call(this, packet); // additional call to catch-all
};
this.socket.on("*", (eventName: string, data: any) => {
if (typeof data === 'object') {
console.log(`socket.io event: [${eventName}] -> data: [${JSON.stringify(data)}]`);
} else {
console.log(`socket.io event: [${eventName}] -> data: [${data}]`);
}
});