We Keep Coding

"Illegal group end indicator...(not a group)" when decoding gtfs-r data

I'm trying to use a node.js app to regularly decode some gtfs-realtime data. It's mostly running fine, but every few hours I run into an error that crashes my app. The error message in my log says that there is an "Illegal group end indicator for Message .transit_realtime.FeedMessage 7 (not a group)"
I found this question/answer on StackOverflow but it doesn't seem to solve my particular problem. Here is an outline of the code I am using to decode the gtfs-r feed:
//process the response
var processBuffers = function(response) {
var data = [];
response.on('data', function (chunk) {
data.push(chunk);
});
response.on('end', function () {
data = Buffer.concat(data);
var decodedFeedMessage = transit.FeedMessage.decode(data);
allData = decodedFeedMessage.entity;
//continues processing with allData...
});
}
Thanks!

NodeJs crashed issue basically happen every time, everydays that any kind of fatal error trigger. And since your received data from 3-rd party, It will very had to make sure the data always correct to prevent error as well.
The simple solution is using another system to deploy your NodeJS application. I recommend 2 tools that very popular today, PM2 and Passenger . (PM2 is very simple to use). Those tool will help to auto restart your NodeJS application once it crashed
http://pm2.keymetrics.io/
https://www.phusionpassenger.com/library/walkthroughs/deploy/nodejs/ownserver/nginx/oss/install_passenger_main.html

copyFileSync not copying file and not throwing error

I'm running a function which I've written in JavaScript inside a nodejs/Electron client.
This function is meant to copy a file from the users flash drive to their c:/Windows/System32 (The file is being copied there so that it can be ran from Command Prompt manually next time the computer is touched without having to switch directories)
The problem is, the files are not being copied, and copyFileSync is not throwing an error.
Here is the code I'm specifically having a problem with:
try {
console.log('copying t.bat');
fs.copyFileSync(remote.app.getAppPath() + '\\app\\files\\scripts\\files\\t.bat', 'C:\\Windows\\System32\\t.bat');
} catch(err) {
console.log('could not copy t.bat', err);
$('#mfail_title').text('Could not copy t.bat file');
$('#mfail_data').text(err);
UIkit.modal("#master_fail").show();
return false;
}
As you can see, I have copyFileSync inside a TRY CATCH block. I know this code is running because in the console I get copying t.bat, but nothing else.
How can I get my files to copy, or at least throw an error when it cannot?
This client is running inside OOBE mode on various Windows 10 machines, therefore always has administrator access.
I've tried updating to the async version of copyFile, but I'm having the same issue. Here is my code
var source = remote.app.getAppPath() + '\\app\\files\\scripts\\files\\t.bat';
var destination = 'C:\\Windows\\System32\\t.bat';
fs.copyFile(source, destination, (err) => {
if (err) {
console.log(err);
} else {
source = remote.app.getAppPath() + '\\app\\files\\scripts\\files\\p.bat';
destination = 'C:\\Windows\\System32\\p.bat';
fs.copyFile(source, destination, (err) => {
if (err) {
console.log(err);
} else {
source = remote.app.getAppPath() + '\\app\\files\\scripts\\files\\p.bat';
destination = 'C:\\Windows\\System32\\p.bat';
child = spawn("powershell.exe",['-ExecutionPolicy', 'ByPass', '-File', remote.app.getAppPath() + '\\app\\files\\scripts\\' + type + '.ps1']);
}
});
}
});
This should copy a file, then when it's complete it should copy another file, once that is complete, it should run a powershell script.
Each copyFile checks for an error before moving on, but it never throws an error, and the file is never copied.

I had a similar issue earlier, In which an Antivirus(Comodo) was not allowing electron app to access the hard drive.
Copy and other file operations were successful in that case as well, because electron in such case access the corresponding sandbox
Please check this is not the case with you.
You can actually access 'fs' in console from electron and check other things in the file system.

Looks to me as if you're using fs on then renderer process (client side) which will not work (assuming that your fs is the node.js fs module and (*)). Your first script seems to use jQuery (hints for renderer) and the second one uses remote in the first line.
fs can only (*) be used on the main process and you'll need to create an IRC channel and do something like:
ircRenderer.sendSync('copy-file-sync', {from: '/from/path', to: '/to/path'})
and, of course, implement the handler for that quickly invented 'copy-file' channel on the main process.
(*) Edit: I haven't played around a lot with nodeIntegration = true, so fs may or may not work on the renderer process with that flag set on the BrowserWindow. But the irc messaging should definitely work and if not, the problem is outside electron, probably related to file permissions.

WebSocket connection closes after calling child process

Here's the setup: I create simple WebSocket Server using the ws library. I then attach a listener for when the client sends me the URL of a PDF to transform. I download it locally then I call another command to transform it:
const download = require("download");
wss.on("connection", ws => {
ws.onmessage = async msg => {
await download(msg.data, destination, {
filename: fileName
});
transformPDF(ws, msg.data);
};
// ...
});
After that, the transformPDF function calls the spawn command to execute a command line binary. I parse the percentage done from the stdout and then try to emit it to the client. But even before this, the connection has been closed and I'm not sure why:
const { spawn } = require("child_process");
const transformPDF = (ws, url) => {
// ...
const child = spawn("k2pdfopt", settings);
child.stdout.on("data", data => {
// ...
ws.send(percentageDone); // <--- connection is broken before this is called
});
};
I have tried to make the transformPDF function a promise and then awaiting it. I have also tried adding an optional detached option to the spawn process. I'm not really sure why it's closing since I've also successfully replaced the command k2pdfopt with something like a lengthy find, and that worked just fine (although it did batch all of the data in the stdout before calling ws.send).
Any help or insight on why it's closing is much appreciated.

Turns out that when I was creating a child process, it was resetting the Visual Studio Code live-server extension that I had running the index.html. That explains why I was also getting a status code of 1001, which I found out most likely means the client refreshed.
I fixed the issue by simply installing the node package live-server and running my index.html from a different terminal.

Sending data in a timer by JS WebRTC crashes when reloading before the end

I have this simple piece of code in a server.js javascript file served by node:
function multiStep(myConnection, data) {
var i=0;
var myTimer=setInterval(function() {
if (i<data.length){
var element=JSON.stringify(data[i]);
console.log("mando elemento: "+element);
myConnection.send(element);
i++;
}
}, 3000);
}
//require our websocket library
clearInterval(myTimer);
var WebSocketServer = require('ws').Server;
//creating a websocket server at port 9090
var wss = new WebSocketServer({port: 9090});
//when a user connects to our sever
wss.on('connection', function(connection) {
loadJSON(function(response) {
//when server gets a message from a connected user
connection.on('message', function(message){
console.log("Got message from a user:", message);
});
var json = JSON.parse(response);
multiStep(connection, json, 0);
})
});
loadJSON simply loads a json data file from another web site.
When I run the client application the first time or when the timeout has ended everything goes fine. Yet if I reload the page while the timeout is not finished I get a crash when I try to use the connection of the old page on the server with report:
/var/www/html/MATERIALI/phonegap/node_modules/ws/lib/WebSocket.js:219
else throw new Error('not opened');
^ Error: not opened
at WebSocket.send (/var/www/html/MATERIALI/phonegap/node_modules/ws/lib/WebSocket.js:219:16)
at null. (/var/www/html/MATERIALI/phonegap/WebRTC/server.js:36:9)
at wrapper [as _onTimeout] (timers.js:261:14)
at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
As a matter of fact I could simply ignore the old session given the page is reloaded. How do I avoid the server to crash in these circumstances?

Ok, I think I found the solution; function multiStep becomes:
function multiStep(myConnection, data) {
var i=0;
clearInterval(myTimer);
myTimer=setInterval(function() {
if (i<data.length){
var element=JSON.stringify(data[i]);
console.log("mando elemento: "+element);
try {
myConnection.send(element);
console.log("mandato elemento");
} catch(err) {
console.log('Websocket error: %s', err);
}
i++;
} else {
}
}, 3000);
}
And it does not crash any longer.

You need to do some checking along the way. Your code assumes that everything is 100%.
var json = JSON.parse(response);
multiStep(connection, json, 0);
You assume there is data in response (it might be empty, or contain non-json data)
You should also check that json is a valid array before passing it to multiStep
The function multiStep also assumes that data.length will return something numeric
This may not be the complete answer, but it should give you a start on making your code more robust.
It's probably failing at myConnection.send(element); but that is probably only a symptom of your lack of checking along the way (you can also check if myConnection is still valid before you send something to it)
Referring to https://developer.mozilla.org/en-US/docs/Web/API/WebSocket, you should be able to check the myConnection.readyState value:
Ready state constants
These constants are used by the readyState attribute to describe the state of the WebSocket connection.
Constant Value Description
CONNECTING 0 The connection is not yet open.
OPEN 1 The connection is open and ready to communicate.
CLOSING 2 The connection is in the process of closing.
CLOSED 3 The connection is closed or couldn't be opened.
Your code will look like this now:
console.log("mando elemento: "+element);
if (myConnection.readyState === 1)
myConnection.send(element);
else
console.log("web socket not open");

Listening for outside events. Bash to NodeJS bridge

Being inside of a NodeJS process, how can I listen for events from bash?
For example
NodeJS side
obj.on("something", function (data) {
console.log(data);
});
Bash side
$ do-something 'Hello World'
Then in the NodeJS stdout will appear "Hello World" message.
How can I do this?
I guess it's related to signal events.

The problem with using signals is that you can't pass arguments and most of them are reserved for system use already (I think SIGUSR2 is really the only safe one for node since SIGUSR1 starts the debugger and those are the only two that are supposed to be for user-defined conditions).
Instead, the best way that I've found to do this is by using UNIX sockets; they're designed for inter process communication.
The easiest way to setup a UNIX socket in node is by setting up a standard net server with net.createServer() and then simply passing a file path to server.listen() to create the socket at the path you specified. Note: It's important that a file at that path doesn't exist, otherwise you'll get a EADDRINUSE error.
Something like this:
var net = require('net');
var server = net.createServer(function(connection) {
connection.on('data', function(data) {
// data is a Buffer, so we'll .toString() it for this example
console.log(data.toString());
});
});
// This creates a UNIX socket in the current directory named "nodejs_bridge.sock"
server.listen('nodejs_bridge.sock');
// Make sure we close the server when the process exits so the file it created is removed
process.on('exit', function() {
server.close();
});
// Call process.exit() explicitly on ctl-c so that we actually get that event
process.on('SIGINT', function() {
process.exit();
});
// Resume stdin so that we don't just exit immediately
process.stdin.resume();
Then, to actually send something to that socket in bash, you can pipe to nc like this:
echo "Hello World" | nc -U nodejs_bridge.sock

What about using FIFOs?
NodeJS code:
process.stdin.on('readable', function() {
var chunk = process.stdin.read();
if (chunk !== null) {
process.stdout.write('data: ' + chunk);
}
});
NodeJS startup (the 3>/tmp/... is a trick to keep FIFO open):
mkfifo /tmp/nodeJsProcess.fifo
node myProgram.js </tmp/nodeJsProcess.fifo 3>/tmp/nodeJsProcess.fifo
Bash linkage:
echo Hello >/tmp/nodeJsProcess.fifo

The signals described in the page that you've linked are used to send some specific "command" to processes. This is called "Inter Process Communication". You can see here a first definition of IPC.
You can instruct you node.js code to react to a specific signal, as in this example:
// Start reading from stdin so we don't exit.
process.stdin.resume();
process.on('SIGUSR1', function() {
console.log('Got SIGUSR1. Here you can do something.');
});
Please note that the signal is sent to the process, and not to a specific object in the code.
If you need to communicate in a more specific way to the node.js daemon you can listen on another port too, and use it to receive (and eventually send) control commands.