We Keep Coding

Node js Error: Protocol "https:" not supported. Expected "http:"

I am using IBM Bluemix to make a web service for a school project.
My project needs to request a JSON from an API, so I can use the data it provides. I use the http get method for a data set, and I am not sure if it is working properly.
When I run my code, I get the message:
Error: Protocol "https:" not supported. Expected "http:"
What is causing it and how can I solve it?
Here is my .js file:
// Hello.
//
// This is JSHint, a tool that helps to detect errors and potential
// problems in your JavaScript code.
//
// To start, simply enter some JavaScript anywhere on this page. Your
// report will appear on the right side.
//
// Additionally, you can toggle specific options in the Configure
// menu.
function main() {
return 'Hello, World!';
}
main();/*eslint-env node*/
//------------------------------------------------------------------------------
// node.js starter application for Bluemix
//------------------------------------------------------------------------------
// HTTP request - duas alternativas
var http = require('http');
var request = require('request');
// cfenv provides access to your Cloud Foundry environment
// for more info, see: https://www.npmjs.com/package/cfenv
var cfenv = require('cfenv');
//chama o express, que abre o servidor
var express = require('express');
// create a new express server
var app = express();
// serve the files out of ./public as our main files
app.use(express.static(__dirname + '/public'));
// get the app environment from Cloud Foundry
var appEnv = cfenv.getAppEnv();
// start server on the specified port and binding host
app.listen(appEnv.port, '0.0.0.0', function() {
// print a message when the server starts listening
console.log("server starting on " + appEnv.url);
});
app.get('/home1', function (req,res) {
http.get('http://developers.agenciaideias.com.br/cotacoes/json', function (res2) {
var body = '';
res2.on('data', function (chunk) {
body += chunk;
});
res2.on('end', function () {
var json = JSON.parse(body);
var CotacaoDolar = json["dolar"]["cotacao"];
var VariacaoDolar = json["dolar"]["variacao"];
var CotacaoEuro = json["euro"]["cotacao"];
var VariacaoEuro = json["euro"]["variacao"];
var Atualizacao = json["atualizacao"];
obj=req.query;
DolarUsuario=obj['dolar'];
RealUsuario=Number(obj['dolar'])*CotacaoDolar;
EuroUsuario=obj['euro'];
RealUsuario2=Number(obj['euro'])*CotacaoEuro;
Oi=1*VariacaoDolar;
Oi2=1*VariacaoEuro;
if (VariacaoDolar<0) {
recomend= "Recomenda-se, portanto, comprar dólares.";
}
else if (VariacaoDolar=0){
recomend="";
}
else {
recomend="Recomenda-se, portanto, vender dólares.";
}
if (VariacaoEuro<0) {
recomend2= "Recomenda-se, portanto, comprar euros.";
}
else if (VariacaoEuro=0){
recomend2="";
}
else {
recomend2="Recomenda-se,portanto, vender euros.";
}
res.render('cotacao_response.jade', {
'CotacaoDolar':CotacaoDolar,
'VariacaoDolar':VariacaoDolar,
'Atualizacao':Atualizacao,
'RealUsuario':RealUsuario,
'DolarUsuario':DolarUsuario,
'CotacaoEuro':CotacaoEuro,
'VariacaoEuro':VariacaoEuro,
'RealUsuario2':RealUsuario2,
'recomend':recomend,
'recomend2':recomend2,
'Oi':Oi,
'Oi2':Oi2
});
app.get('/home2', function (req,res) {
http.get('https://www.quandl.com/api/v3/datasets/BCB/432.json?api_key=d1HxqKq2esLRKDmZSHR2', function (res3) {
var body = '';
res3.on('data', function (chunk) {
body += chunk;
});
res3.on('end', function () {
var x=json.dataset.data[0][1];
console.log("My JSON is "+x); });
});
});
});
});
});
Here is a print of the error screen I get:

When you want to request an https resource, you need to use https.get, not http.get.
https://nodejs.org/api/https.html

As a side note to anyone looking for a solution from Google... make sure you are not using an http.Agent with an https request or you will get this error.

The reason for this error is that you are trying to call a HTTPS URI from a HTTP client. The ideal solution would have been for a generic module to figure out the URI protocol and take the decision to use HTTPS or HTTP internally.
The way I overcame this problem is by using the switching logic on my own.
Below is some code which did the switching for me.
var http = require('http');
var https = require('https');
// Setting http to be the default client to retrieve the URI.
var url = new URL("https://www.google.com")
var client = http; /* default client */
// You can use url.protocol as well
/*if (url.toString().indexOf("https") === 0){
client = https;
}*/
/* Enhancement : using the URL.protocol parameter
* the URL object , provides a parameter url.protocol that gives you
* the protocol value ( determined by the protocol ID before
* the ":" in the url.
* This makes it easier to determine the protocol, and to support other
* protocols like ftp , file etc)
*/
client = (url.protocol == "https:") ? https : client;
// Now the client is loaded with the correct Client to retrieve the URI.
var req = client.get(url, function(res){
// Do what you wanted to do with the response 'res'.
console.log(res);
});

Not sure why, but the issue for me happened after updating node to version 17, i was previously using version 12.
In my setup, i have node-fetch using HttpsProxyAgent as an agent in the options object.
options['agent'] = new HttpsProxyAgent(`http://${process.env.AWS_HTTP_PROXY}`)
response = await fetch(url, options)
Switching back to node 12 fixed the problem:
nvm use 12.18.3

I got this error while deploying the code.
INFO error=> TypeError [ERR_INVALID_PROTOCOL]: Protocol "https:" not supported. Expected "http:"
at new NodeError (node:internal/errors:372:5)
To fix this issue, I have updated the "https-proxy-agent" package version to "^5.0.0"
Now the error was gone and it's working for me.

Node.js - Reading CSV-file not working with line numbers > 500

I am currently struggling to run my Node.js server.
What I want to do:
Upload a CSV-File from mobile device to my local server and save it on the file system
Read each line of the .csv-File and save each row to my MongoDB database
Uploading and saving the file works flawlessly. Reading the .csv-File and saving each row to the database only works for files with small line numbers.
I don't know the exact number of lines when it stops working. It seems to differ every time I read a file.
Sometimes (if the line numbers are bigger than 1000) the CSV-Reader I use doesn't even start processing the file. Other times he reads only 100-200 lines and then stops.
Here is my code how I upload the file:
var fs = require('fs');
var sys = require("sys");
var url = require('url');
var http = require('http');
http.createServer(function(request, response) {
sys.puts("Got new file to upload!");
var urlString = url.parse(request.url).pathname;
var pathParts = urlString.split("/");
var deviceID = pathParts[1];
var fileName = pathParts[2];
sys.puts("DeviceID: " + deviceID);
sys.puts("Filename: " + fileName);
sys.puts("Start saving file");
var tempFile = fs.createWriteStream(fileName);
request.pipe(tempFile);
sys.puts("File saved");
// Starting a new child process which reads the file
// and inserts each row to the database
var task = require('child_process').fork('databaseInsert.js');
task.on('message', function(childResponse) {
sys.puts('Finished child process!');
});
task.send({
start : true,
deviceID : deviceID,
fileName : fileName
});
sys.puts("After task");
response.writeHead(200, {
"Content-Type" : "text/plain"
});
response.end('MESSAGE');
}).listen(8080);
This works all fine.
Now the code of the child process (databaseInsert.js):
var sys = require("sys");
var yaCSV = require('ya-csv');
var Db = require('mongodb').Db;
var dbServer = require('mongodb').Server;
process.on('message', function(info) {
sys.puts("Doing work in child process");
var fileName = info.fileName;
var deviceID = info.deviceID;
sys.puts("Starting db insert!");
var dbClient = new Db('test', new dbServer("127.0.0.1", 27017, {}), {
w : 1
});
dbClient.open(function(err, client) {
if (err) {
sys.puts(err);
}
dbClient.createCollection(deviceID, function(err, collection) {
if (err) {
sys.puts("Error creating collection: " + err);
} else {
sys.puts("Created collection: " + deviceID);
var csvReader = yaCSV.createCsvFileReader(fileName, {
columnsFromHeader : true,
'separator' : ';'
});
csvReader.setColumnNames([ 'LineCounter', 'Time', 'Activity',
'Latitude', 'Longitude' ]);
var lines = 0;
csvReader.addListener('data', function(data) {
lines++;
sys.puts("Line: " + data.LineCounter);
var docRecord = {
fileName : fileName,
lineCounter : data.LineCounter,
time : data.Time,
activity : data.Activity,
latitude : data.Latitude,
longitude : data.Longitude
};
collection.insert(docRecord, {
safe : true
}, function(err, res) {
if (err) {
sys.puts(err);
}
});
});
}
});
});
process.send('finished');
});
At first I didn't use a child process but I had the same behaviour as I have now. So I tested this.
Hopefully someone who has some experience with Node.js can help me.

I think your issue is that you are trying to read the tempFile while it is still being written to. Right now you are piping the request to the file stream (which proceeds in parallel and asynchronously) and start the reader process. The reader process will then start reading the file in parallel with the write operations. If the reader is faster (it usually will be), it will read the first couple of records but then encounter an end of file and stop reading.
To remedy this, you could only start the reader process after writing has completely finished, i.e., put the part from sys.puts("File.send"); onward into a callback of tempFile.end(...) (see http://nodejs.org/api/stream.html#stream_writable_end_chunk_encoding_callback).
Reading the file while it is still being written to, akin to the tail command in Unix, is fairly hard in my understanding (google for details on how difficult it is to implement a proper tail).

Are you familiar with mongoimport/export?
I used this in the past to export from my db to a csv file...so you can do the opposite after you upload it from the mobile-client to the server.
Its from the shell, but you can write it in code using nodeJS_ChildSpawn

Learning Node - Book - Pages 16-18 - Two Examples

There are two examples in between these pages 16 and 18.
Example 1.3 is a server app.
Example 1.4 is a client app doing GET requests to the server.
When I run the two examples (at the same time) I notice some quite weird behavior
in the client. All requests are executed (i.e. the for loop in the client completes)
but the callbacks of only 5 of them get called. The client doesn't exit and also
doesn't error out. And just no more callbacks are called.
Any ideas what might be happening or how I can troubleshoot this further?
Note: I am running Node.js v0.10.20 on Windows 7.
Server:
var http = require('http');
var fs = require('fs');
// write out numbers
function writeNumbers(res) {
var counter = 0;
// increment, write to client
for (var i = 0; i<100; i++) {
counter++;
res.write(counter.toString() + '\n');
}
}
// create http server
http.createServer(function (req, res) {
var query = require('url').parse(req.url).query;
var app = require('querystring').parse(query).file;
// content header
res.writeHead(200, {'Content-Type': 'text/plain'});
if (!app){
res.end();
console.log('No file argument found in query string.');
return;
}else{
app = app + ".txt";
}
// write out numbers
writeNumbers(res);
// timer/timeout to open file and read contents
setTimeout(function() {
console.log('Opening file: ' + app + '.');
// open and read in file contents
fs.readFile(app, 'utf8', function(err, data) {
res.write('\r\n');
if (err)
res.write('Could not find or open file ' + app + ' for reading.\r\n');
else {
res.write(data);
}
// response is done
res.end();
});
},2000);
}).listen(8124);
console.log('Server running at 8124');
Client:
var http = require('http');
var N = 200;
// The URL we want, plus the path and options we need
var options = {
host: 'localhost',
port: 8124,
path: '/?file=automatic',
method: 'GET'
};
var callback_function = function(response) {
// finished? ok, write the data to a file
console.log('got response back');
};
for (var i = 1; i <= N; i++) {
// make the request, and then end it, to close the connection
http.request(options, callback_function).end();
console.log('done with call # ' + i);
}
--- Experiment Done ---
If I lower N to 10 and also if I do a
global "var i = 1" and then do this thing:
function schedule(){
http.request(options, callback_function).end();
console.log('done with call ' + i);
i++;
if (i<=N){
setTimeout(function(){
schedule();
}, 1000);
}
}
schedule();
instead of the loop in the client, I get similar behavior.
I guess that's what Milimetric meant by "sleep" i.e. just
to make sure I don't hit the server too quickly with too
many simultaneous requests.
But the behavior is not fully identical, it takes several mins
to print 'got response back' on the second set of 5 requests
and then another maybe 5-6 mins for the client to exit.
Still, all that does look weird to me.
C:\PERSONAL\NODE_TEST>node test004.js
done with call 1
got response back
done with call 2
got response back
done with call 3
got response back
done with call 4
got response back
done with call 5
got response back
done with call 6
done with call 7
done with call 8
done with call 9
done with call 10
got response back
got response back
got response back
got response back
got response back
C:\PERSONAL\NODE_TEST>

The problem is that the client doesn't consume the response body sent by the server, so the connection remains (half) open and the http agent only allows 5 concurrent requests per client by default, causing it to hang after 5 requests. The connection will eventually timeout, causing the next 5 requests to be processed.
node.js http.get hangs after 5 requests to remote site
Change your callback function to consume any data sent down the response.
var callback_function = function(response) {
// finished? ok, write the data to a file
console.log('got response back');
response.on('data', function () {});
};

Chrome packaged app UDP sockets not working

I'm trying to get UDP sockets working for a packaged app using Chrome Canary (currently version 25). I am pretty confused by the fact the UDP example here conflicts with the reference documentation here.
The official example uses this line:
chrome.socket.create('udp', '127.0.0.1', 1337, { onEvent: handleDataEvent }, ...
In Canary using this line results in the error:
Uncaught Error: Invocation of form socket.create(string, string,
integer, object, function) doesn't match definition
socket.create(string type, optional object options, function callback)
Not surprising since that matches the documented form of the function. (I guess the example is out of date?) OK, so I try this...
chrome.socket.create('udp', { onEvent: handleDataEvent }, ...
Canary complains:
Uncaught Error: Invalid value for argument 2. Property 'onEvent':
Unexpected property.
Now I'm confused, especially since this parameter is undocumented in the reference. So I just go with this:
chrome.socket.create('udp', {}, ...
Now it creates OK, but the following call to connect...
chrome.socket.connect(socketId, function(result) ...
...fails with this:
Uncaught Error: Invocation of form socket.connect(integer, function)
doesn't match definition socket.connect(integer socketId, string
hostname, integer port, function callback)
...which is not surprising, since now my code doesn't mention a host or port anywhere, so I guess it needs to be in connect. So I change it to the form:
chrome.socket.connect(socketId, address, port, function (result) ...
At last I can connect and write to the socket OK. But this doesn't cover reading.
Can someone show me a working example based on UDP that can send & receive, so I can work from that?
How do I receive data since the example's onEvent handler does not work? How do I ensure I receive any data on-demand as soon as it arrives without blocking?

The Network Communications doc is not up-to-date. See the latest API doc: https://developer.chrome.com/trunk/apps/socket.html. But the doc doesn't state everything clearly.
I looked into Chromium source code and found some useful comments here: https://code.google.com/searchframe#OAMlx_jo-ck/src/net/udp/udp_socket.h&q=file:(%5E%7C/)net/udp/udp_socket%5C.h$&exact_package=chromium
// Client form:
// In this case, we're connecting to a specific server, so the client will
// usually use:
// Connect(address) // Connect to a UDP server
// Read/Write // Reads/Writes all go to a single destination
//
// Server form:
// In this case, we want to read/write to many clients which are connecting
// to this server. First the server 'binds' to an addres, then we read from
// clients and write responses to them.
// Example:
// Bind(address/port) // Binds to port for reading from clients
// RecvFrom/SendTo // Each read can come from a different client
// // Writes need to be directed to a specific
// // address.
For the server UDP socket, call chrome.socket.bind and chrome.socket.recvFrom/chrome.socket.sendTo to interact with clients. For the client UDP socket, call chrome.socket.connect and chrome.socket.read/chrome.socket.write to interact with the server.
Here's an example:
var serverSocket;
var clientSocket;
// From https://developer.chrome.com/trunk/apps/app_hardware.html
var str2ab=function(str) {
var buf=new ArrayBuffer(str.length);
var bufView=new Uint8Array(buf);
for (var i=0; i<str.length; i++) {
bufView[i]=str.charCodeAt(i);
}
return buf;
}
// From https://developer.chrome.com/trunk/apps/app_hardware.html
var ab2str=function(buf) {
return String.fromCharCode.apply(null, new Uint8Array(buf));
};
// Server
chrome.socket.create('udp', null, function(createInfo){
serverSocket = createInfo.socketId;
chrome.socket.bind(serverSocket, '127.0.0.1', 1345, function(result){
console.log('chrome.socket.bind: result = ' + result.toString());
});
function read()
{
chrome.socket.recvFrom(serverSocket, 1024, function(recvFromInfo){
console.log('Server: recvFromInfo: ', recvFromInfo, 'Message: ',
ab2str(recvFromInfo.data));
if(recvFromInfo.resultCode >= 0)
{
chrome.socket.sendTo(serverSocket,
str2ab('Received message from client ' + recvFromInfo.address +
':' + recvFromInfo.port.toString() + ': ' +
ab2str(recvFromInfo.data)),
recvFromInfo.address, recvFromInfo.port, function(){});
read();
}
else
console.error('Server read error!');
});
}
read();
});
// A client
chrome.socket.create('udp', null, function(createInfo){
clientSocket = createInfo.socketId;
chrome.socket.connect(clientSocket, '127.0.0.1', 1345, function(result){
console.log('chrome.socket.connect: result = ' + result.toString());
});
chrome.socket.write(clientSocket, str2ab('Hello server!'), function(writeInfo){
console.log('writeInfo: ' + writeInfo.bytesWritten +
'byte(s) written.');
});
chrome.socket.read(clientSocket, 1024, function(readInfo){
console.log('Client: received response: ' + ab2str(readInfo.data), readInfo);
});
});

How to create a streaming API with NodeJS

How would you go to create a streaming API with Node? just like the Twitter streaming API.
What I want to do ultimately is get the first update from the FriendFeed api, and stream when a new one becomes available (if the id is different), and later on expose it as a web service so I can use it with WebSockets on my website :).
So far I have this:
var sys = require('sys'),
http = require('http');
var ff = http.createClient(80, 'friendfeed-api.com');
var request = ff.request('GET', '/v2/feed/igorgue?num=1',
{'host': 'friendfeed-api.com'});
request.addListener('response', function (response) {
response.setEncoding('utf8'); // this is *very* important!
response.addListener('data', function (chunk) {
var data = JSON.parse(chunk);
sys.puts(data.entries[0].body);
});
});
request.end();
Which only gets the data from FriendFeed, creating the Http server with node is easy but it can't return a stream (or I haven't yet found out how).

You would want to set up a system that keeps track of incoming requests and stores their response objects. Then when it's time to stream a new event from FriendFeed, iterate through their response objects and responses[i].write('something') out to them.
Check out LearnBoost's Socket.IO-Node, you may even just be able to use that project as your framework and not have to code it yourself.
From the Socket.IO-Node example app (for chat):
io.listen(server, {
onClientConnect: function(client){
client.send(json({ buffer: buffer }));
client.broadcast(json({ announcement: client.sessionId + ' connected' }));
},
onClientDisconnect: function(client){
client.broadcast(json({ announcement: client.sessionId + ' disconnected' }));
},
onClientMessage: function(message, client){
var msg = { message: [client.sessionId, message] };
buffer.push(msg);
if (buffer.length > 15) buffer.shift();
client.broadcast(json(msg));
}
});