I am trying to show Nodejs is not good for CPU Intensive tasks.
In this code, I created the async function mySlowFunction that accepts baseNumber.
When I hit http://localhost:8080 with query params 12 it takes 5 sec.
What If 5 subsequent request is being made to this api for query params 11,89,23,12,23? It would be very slow.
In this kind of scenario is it really good to use NodeJs?[i.e for CPU Intensive Task]. If yes, then how can I make this fast?
async function mySlowFunction(baseNumber) {
let result = 0;
for (var i = Math.pow(baseNumber, 7); i >= 0; i--) {
result += Math.atan(i) * Math.tan(i);
};
return result;
}
const express = require('express');
const cookieparser = require('cookie-parser');
const csrf = require('csurf');
const dotenv = require("dotenv").config();
const app = express();
app.use(express.urlencoded({extended:true}));
app.use(express.json());
app.use(cookieparser());
app.get("/",async(req,res)=>{
let data = await mySlowFunction(req.query.base);
res.json({data})
})
app.listen(process.env.PORT, () => {
console.log(`Listening on http://localhost:${process.env.PORT}`);
});
I have a very general answer.
You can precalculate the results if query params are limited.
You can store results in files or database query them.
You can cache results.
If there is no result for query param in db, calculate it and save it to db.
Related
I am facing some kind of issue in the case of concurrent request handling using express. I am using some encrypted value in every request header and want to set it to some global variable from middleware after decrypting it and want to use it in all the controllers, utils across the project.
var app = express();
app.use("/test_test",Controller.testtest);
exports.testtest = async function (req, res) {
globalcons.vals.ssip = req.body.ssip;
var usr = req.body.usr;
await new Promise(resolve => setTimeout(resolve, 8000));
let response = {};
response['status'] = 'success';
response['ssip'] = globalcons.vals.ssip; // this is changing
response['usr'] = usr; //this is not
return apiResponse.successApiResponse(res, response);
}
Here is the problem screenshot
The function I would like this function to run by itself at time intervals. As it is now I have to visit the '/getCompanyInfo' path to trigger it. I would like it to run every minute as if I was visiting the '/getCompanyInfo' path each minute. The app is on Heroku and I would like the function to execute without any pages open.
The original function that is triggered by visiting the path.
const express = require('express');
const app = express();
/**
* getCompanyInfo ()
*/
app.get('/getCompanyInfo', function(req,res){
const companyID = oauthClient.getToken().realmId;
console.log(companyID)
const url = OAuthClient.environment.production ;
oauthClient.makeApiCall({url: url + 'v3/company/0000000000/salesreceipt/8?minorversion=41'})
.then(function(authResponse){
console.log("The response for API call is :"+JSON.parse(JSON.stringify(authResponse)));
res.send(authResponse);
})
.catch(function(e) {
console.error(e);
});
});
One of my attempts here was to put it in a function that executes each minute using node-schedule.
This one doesn't do anything other than print 'This will run once a minute.' to the console.
I tried removing
app.get(function(req,res){
and the
})
below it but that made the app (hosted on Heroku) fail to build.
const express = require('express');
const app = express();
var schedule = require('node-schedule');
var j = schedule.scheduleJob('* * * * *', function(){
console.log('This will run once a minute.');
app.get(function(req,res){
const companyID = oauthClient.getToken().realmId;
console.log(companyID)
const url = OAuthClient.environment.production ;
oauthClient.makeApiCall({url: url + 'v3/company/0000000000/salesreceipt/8?minorversion=41'})
.then(function(authResponse){
console.log("The response for API call is :"+JSON.parse(JSON.stringify(authResponse)));
res.send(authResponse);
})
.catch(function(e) {
console.error(e);
});
});
});
More Context:
It is inside an app I have on Heroku. I would like to set the app to make a requests for JSON data from the API every x time without me having to touch it.
app.get initializes api handler - e.g. this is your api route definition - the thing that will respond when you call GET /getCompanyInfo via web browser or some other client. You should not redefine it regularly with your scheduled action.
The failed build after you've removed the route handler is probably because of the res.send(authResponse); left behind.
You could have something like:
// function that will be used to get the data
const getCompanyInfo = (done) => {
const companyID = oauthClient.getToken().realmId
console.log(companyID)
const url = OAuthClient.environment.production
oauthClient.makeApiCall({url: url + 'v3/company/0000000000/salesreceipt/8?minorversion=41'})
.then((authResponse) => {
console.log("The response for API call is :"+JSON.parse(JSON.stringify(authResponse)))
done(authResponse)
})
.catch((e) => {
console.error(e)
})
}
// this will trigger the function regularly on the specified interval
const j = schedule.scheduleJob('* * * * *', () => {
getCompanyInfo((companyInfo) => {
// ...do whatever you want with the info
})
})
// this will return you the data by demand, when you call GET /getCompanyInfo via browser
app.get('/getCompanyInfo', function(req,res) {
getCompanyInfo((companyInfo) => {
res.send(companyInfo)
})
})
Heroku has an add on called Heroku Scheduler that does what you want. The node-schedule npm package might do the job, but as you mentioned, you probably aren't going to be able to see the execution/results/logs of your jobs that run every 24 hours without making some interface for it on your own.
For your issue, calling app.get doesn't make a lot of sense. That's just telling node about the route. Assuming you have your /getCompanyInfo route up and running, you just need to call it in your scheduled job, not re-register it every time.
You could also just do this (http being the http client you're using):
var j = schedule.scheduleJob('* * * * *', async function(){
console.log('This will run once a minute.');
const result = await http.get('/getCompanyInfo');
console.log(result);
});
I am new to Node and MongoDB, so please forgive me if I sound too naive.
I have a DB class that creates a connection to MongoDB
db.service.js
const mongoose = require("mongoose");
const fs = require("fs");
const dbConfigs = JSON.parse(fs.readFileSync("/configs.json"));
const dbHost = dbConfigs.DB_HOST;
const dbName = dbConfigs.DB_NAME;
const dbPort = dbConfigs.DB_PORT;
const dbUrl = "mongodb://" + dbHost + ":" + dbPort + "/" + dbName;
const dbOptions = {
useNewUrlParser: true
};
let dbConnection = mongoose.createConnection(dbUrl, dbOptions);
exports.getConnection = () => {
if (dbConnection)
return dbConnection;
}
exports.closeConnection = () => {
if (dbConnection)
return dbConnection.close();
}
Next I have another module that creates a Schema for MongoDB
schema.js
const connection = require("./db.service").getConnection();
const Schema = require("mongoose").Schema;
const SampleSchema = new Schema({...})
exports.Sample = connection.model("Sample", SampleSchema);
Then I have another module that makes use of this Schema to save objects
logger.js
const Sample = require("./schema").Sample;
exports.log = () => {
let body = {....};
let sampleObj = new Sample(body);
return sampleObj.save(sampleObj);
The Main module
Main.js
const logger = require("./logger");
for (let i=0; i < 100; i++) {
logger.log();
}
When I run node Main.js, everything is saved, but when I check the MongoDB with the command db.serverStatus.connections, I see 6 connections open.
I don't know how it's getting there. I know the command mongo itself keep a connection open, but where are the other 5 connections coming from?
I have checked this, which seems to suggest that Mongoose opens 5 connections for an application, but my application is firing just one transaction, where is the need to open 5 connection then? Couldn't it be done with just one?
Mongoose is managing the connections itself; it tries to optimize the speed of the request.
You are seeing it like you are using only one transaction, but behind the scene mongoose is probably doing much more than you expect.
You can modify the number of connection mongoose can open using the parameter poolSize.
Example :
const dbOptions = {
useNewUrlParser: true,
poolSize: 1,
};
Try it with poolSize equals to 1 and see how much time it takes to execute your transaction, you'll have your answer.
I'm trying to make a .js file that will constantly have the price of bitcoin updated (every five minutes or so). I've tried tons of different ways to web scrape but they always output with either null or nothing. Here is my latest code, any ideas?
var express = require('express');
var path = require('path');
var request = require('request');
var cheerio = require('cheerio');
var fs = require('fs');
var app = express();
var url = 'https://blockchain.info/charts/';
var port = 9945;
function BTC() {
request(url, function (err, res, body) {
var $ = cheerio.load(body);
var a = $(".market-price");
var b = a.text();
console.log(b);
})
setInterval(BTC, 300000)
}
BTC();
app.listen(port);
console.log('server is running on '+port);
It successfully says what port it's running on, that's not the problem. This example (when outputting) just makes a line break every time the function happens.
UPDATE:
I changed the new code I got from Wartoshika and it stopped working, but im not sure why. Here it is:
function BTCPrice() {
request('https://blockchain.info/de/ticker', (error, response, body) => {
const data = JSON.parse(body);
var value = (parseInt(data.USD.buy, 10) + parseInt(data.USD.sell, 10)) / 2;
return value;
});
};
console.log(BTCPrice());
If I have it console.log directly from inside the function it works, but when I have it console.log the output of the function it outputs undefined. Any ideas?
I would rather use a JSON api to get the current bitcoin value instead of an HTML parser. With the JSON api you get a strait forward result set that is parsable by your browser.
Checkout Exchange Rates API
Url will look like https://blockchain.info/de/ticker
Working script:
const request = require('request');
function BTC() {
// send a request to blockchain
request('https://blockchain.info/de/ticker', (error, response, body) => {
// parse the json answer and get the current bitcoin value
const data = JSON.parse(body);
value = (parseInt(data.THB.buy, 10) + parseInt(data.THB.sell, 10)) / 2;
console.log(value);
});
}
BTC();
Using the value as callback:
const request = require('request');
function BTC() {
return new Promise((resolve) => {
// send a request to blockchain
request('https://blockchain.info/de/ticker', (error, response, body) => {
// parse the json answer and get the current bitcoin value
const data = JSON.parse(body);
value = (parseInt(data.THB.buy, 10) + parseInt(data.THB.sell, 10)) / 2;
resolve(value);
});
});
}
BTC().then(val => console.log(val));
As the other answer stated, you should really use an API. You should also think about what type of price you want to request. If you just want a sort of index price that aggregates prices from multiple exchanges, use something like the CoinGecko API. Also if you need real-time data you need a websocket-based API, not a REST API.
If you need prices for a particular exchange, for example you're building a trading bot for one or more exchanges, you;ll need to communicate with each exchange's websoceket API directly. For that I would recommend something like the Coygo API, a node.js package that connects you directly to each exchange's real-time data feeds. You want something that doesn't add a middleman since that would add latency to your data.
I'm working on an app that uses Node.js as the backend. Currently, I have a web server setup like this:
var express = require('express');
var http = require('http');
var app = module.exports.app = express();
http.createServer(app).listen(appConfig.port, function () {
var logger = app.get('logger');
logger.info('**** STARTING SERVER ****');
});
My app is working just fine. Except, I have now added a request that is takes ~5 minutes. From my understanding, Node defaults to a 2 minute timeout window. I can read the documentation here. However, I don't understand how to set the timeout to a greater value.
Can someone show me how to increase the timeout to 10 minutes?
this should set it. you need a reference to the server object then set the timeout
var express = require('express');
var http = require('http');
var app = module.exports.app = express();
var server = http.createServer(app);
server.setTimeout(10*60*1000); // 10 * 60 seconds * 1000 msecs
server.listen(appConfig.port, function () {
var logger = app.get('logger');
logger.info('**** STARTING SERVER ****');
});