We Keep Coding

Google Cloud Functions, resolveMX is not working with a list of domain

I have the following function. I have a list of domains (very big list, more than 100000), I'm trying to put them in a foreach and resolveMx all of them and save the mx records in another database.
Edit, this is the complete function:
const dns = require('dns');
const {BigQuery} = require('#google-cloud/bigquery');
const bigquery = new BigQuery(project="smartiodomains");
const functions = require('firebase-functions');
exports.getMxRecords = functions.https.onRequest( async (req, res) => {
const query = "SELECT string_field_0 FROM smartiodomains.Domains.sk_domains_table";
const options = {
query: query,
location: 'US',
};
const [job] = await bigquery.createQueryJob(options);
const [rows] = await job.getQueryResults();
const datasetId = 'Domains';
const tableId = 'smartio_records';
var index = 0;
rows.forEach((row) => {
dns.resolveMx(row.string_field_0, function(error,addresses){
if(error){
const rows = [
{domain:row.string_field_0, mx_records: 'No data found.', priority: 'No data found.'}
];
// Insert data into a table
bigquery
.dataset(datasetId)
.table(tableId)
.insert(rows);
res.write("Something");
}else{
res.write("Something else");
addresses.forEach( address => {
const rows = [
{domain:row.string_field_0, mx_records: address.exchange, priority: address.priority}
];
// Insert data into a table
bigquery
.dataset(datasetId)
.table(tableId)
.insert(rows).then((foundErrors) => {
if (foundErrors && foundErrors.insertErrors != undefined) {
console.log('Error: ', err);
}
})
.catch((err) => {
console.error('ERROR:', err);
});
});
}
});
});
});
As #Doug Stevenson suggested i add a response (res.write("Something")). Now i have one error and a warning:
1.- Memory Limit exceeded
2.- TeenyStatisticsWarning: Possible excessive concurrent requests detected. 5000 requests in-flight, which exceeds the configured threshold of 5000. Use the TEENY_REQUEST_WARN_CONCURRENT_REQUESTS environment variable or the concurrentRequests option of teeny-request to increase or disable (0) this warning.
Old error:
With this implementation i got this error in the logs of GCF:
getMxRecordsp5ter5a8u17q { Error: queryMx ETIMEOUT marketingweb.sk
Sorry for my bad english. And thanks for any help.

An HTTP function requires that you send a response to the client after all of the asynchronous work is complete. The function terminates immediately after you send that response. Right now, you're not sending any response, so the function never terminates, and it will always time out. You should send a response after all the calls to dns.resolveMx are fully complete.

Consuming external api with Firebase Functions

I am trying to consume an external api with Firebase Functions, but it gives a time out error when I use OPTIONS, when I use GET to work normally.
I don't want to use const request = require('request'); or var rp = require('request-promise');, as they are obsolete.
What may be wrong, I await help from colleagues.
const express = require('express');
const cors = require('cors');
const app = express();
// Permitir solicitações de origem cruzada automaticamente
app.use(cors({
origin: true
}));
//app.use(cors());
app.get('/criarcliente', (req, res) => {
let uri = "https://api.iugu.com/v1/customers?api_token=<mytoken>";
let headers = {
'Content-Type': 'application/json'
}
let body = {
custom_variables: [{
name: 'fantasia',
value: 'Dolci Technology'
}, {
name: 'vendedor',
value: ''
}],
email: 'teste1#teste1.com',
name: 'John Dolci',
phone: 9999999,
phone_prefix: 66,
cpf_cnpj: '00000000000',
cc_emails: 'test#test.com',
zip_code: '78520000',
number: '49',
street: 'Name Street',
city: 'Guarantã do Norte',
state: 'MT',
district: 'Jardim Araguaia'
}
var options = {
method: 'POST',
uri: uri,
body: body,
headers: headers,
json: true
};
const https = require('https');
var req = https.request(options, (resp) => {
let data = '';
resp.on('data', (chunk) => {
data += chunk;
});
resp.on('end', () => {
var result = JSON.parse(data);
res.send(result);
});
console.log("aqui");
}).on("error", (err) => {
console.log("Error: " + err.message);
});
}); ```

There are two points that I would point out to your case. First, you need to confirm that you are using the Blaze plan in your billing. As clarified in the official pricing documentation, in case you are not and are trying to use a non Google-owned service, you won't be able to do it and this will cause an error.
The second point would be to use the axios library within your application. It's the one that I believe to be most used with Node.js to allow you access to third-party application within Cloud Functions - I would say that it's pretty easy to use as well. You should find a whole example on using it with third-party API here.
To summarize, take a look at your pricing, as it's the most common issue to be caused and give it a try with axios, to confirm that you can avoid the error with it.

Returning a value from async function to outer function Node js [duplicate]

This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 2 years ago.
I am fairly new to the whole asynchronous world. Had to start for implementing Twilio video call through node.
I have been trying to call this server side function which then calls another asynchronous function that returns promise. The then portion works fine and I am able to see output in console. But when I try to call another function in then which has an ajax call to the main application to save some data in database.
Neither am I able to send the returned value in response.send() nor am I able to call a function in then.
Please help!
Server Side - index.js
'use strict';
/**
* Load Twilio configuration from .env config file - the following environment
* variables should be set:
* process.env.TWILIO_ACCOUNT_SID
* process.env.TWILIO_API_KEY
* process.env.TWILIO_API_SECRET
*/
require('dotenv').load();
const express = require('express');
const http = require('http');
const path = require('path');
const { jwt: { AccessToken } } = require('twilio');
const Twilio = require('twilio');
const jquery = require( 'jQuery');
const cors = require('cors');
const VideoGrant = AccessToken.VideoGrant;
// Max. period that a Participant is allowed to be in a Room (currently 14400 seconds or 4 hours)
const MAX_ALLOWED_SESSION_DURATION = 14400;
// Create Express webapp.
const app = express();
// Set up the path for the HealthAssure.
const startPath = path.join(__dirname, '../HAVideoConsultation/public');
app.use('/HAVideoConsultation', express.static(startPath));
/**
* Default to the application.
*/
app.get('/', (request, response) => {
response.redirect('/HAVideoConsultation');
});
/**
* Generate an Access Token for a chat application user - it generates a random
* username for the client requesting a token, and takes a device ID as a query
* parameter.
*/
app.get('/token', function(request, response) {
const { identity } = request.query;
// Create an access token which we will sign and return to the client,
// containing the grant we just created.
const token = new AccessToken(
process.env.TWILIO_ACCOUNT_SID,
process.env.TWILIO_API_KEY,
process.env.TWILIO_API_SECRET,
{ ttl: MAX_ALLOWED_SESSION_DURATION }
);
// Assign the generated identity to the token.
token.identity = identity;
// Grant the access token Twilio Video capabilities.
const grant = new VideoGrant();
token.addGrant(grant);
// Serialize the token to a JWT string.
response.send(token.toJwt());
});
function pushCompositionId(compositionId){
console.log(compositionId);
jquery.ajax({
url:'http://localhost:58674/ABC/XYZ',
type:'GET',
data: {CompositionId:compositionId},
cors: true,
success:function(result){
Console.log('Composition Id pushed successfully.');
},
error:function(err){
console.log(err);
return false;
}
});
}
app.get('/Composition',function(request,response){
const client = new Twilio(process.env.TWILIO_API_KEY,process.env.TWILIO_API_SECRET, {accountSid: process.env.TWILIO_ACCOUNT_SID});
const cid = null;
client.video.compositions
.create({
roomSid: request.query.roomSid,
audioSources: '*',
videoLayout: {
grid : {
video_sources: ['*']
}
},
format: 'mp4'
}).then(composition =>{
console.log("Created Composition with SID=" + composition.sid); // This works properly
cid=composition.sid;
// pushCompositionId(composition.sid); // I want to call this function here
});
response.send(cid); // This does not return proper value
});
// Create http server and run it.
const server = http.createServer(app);
const port = process.env.PORT || 3000;
server.listen(port, function() {
console.log('Express server running on *:' + port);
});
Client side
async function GenerateCompositionId(roomsid){
const compositionid = await fetch(`/Composition?roomSid=${roomsid}`);
}
Server Side function I want to call in then after Composition.sid is generated. If I put this in a try block it gives me error jquery.ajax is not a function. I have included it require and another ajax function on client side is working fine. Why does this not?
function pushCompositionId(compositionId){
jquery.ajax({
url:'http://localhost:58674/ABC/XYZ',
type:'GET',
data: {CompositionId:compositionId},
cors: true,
success:function(result){
Console.log('Composition Id pushed successfully.');
},
error:function(err){
console.log(err);
return false;
}
});
}

In order to send the response when the async calls are done the server-side must be:
var jsdom = require("jsdom");
const { JSDOM } = jsdom;
const { window } = new JSDOM();
const { document } = (new JSDOM('')).window;
global.document = document;
var $ = jQuery = require('jquery')(window);
function pushCompositionId(compositionId,response){
console.log(compositionId);
jquery.ajax({
url:'http://localhost:58674/ABC/XYZ',
type:'GET',
data: {CompositionId:compositionId},
cors: true,
success:function(result){
Console.log('Composition Id pushed successfully.');
response.send(cid); // This does not return proper value
},
error:function(err){
console.log(err);
response.send(false);
}
});
}
app.get('/Composition',function(request,response){
const client = new Twilio(process.env.TWILIO_API_KEY,process.env.TWILIO_API_SECRET, {accountSid: process.env.TWILIO_ACCOUNT_SID});
const cid = null;
client.video.compositions
.create({
roomSid: request.query.roomSid,
audioSources: '*',
videoLayout: {
grid : {
video_sources: ['*']
}
},
format: 'mp4'
}).then(composition =>{
console.log("Created Composition with SID=" + composition.sid); // This works properly
cid=composition.sid;
pushCompositionId(composition.sid,response); // I want to call this function here
});
});
Otherwise you can use await/async to return a sync-like result

Unexpected behavior with knex's select

For the following code, I get a result that is sometimes an array and other times an object. I want to receive the array, even if its empty.
export const GetByPId = async (userId, pId) => knex('table1').where({ userId, pId }).select(
'userId',
'pId',
'id',
'created',
'updated',
);
In my Business object, I await the response
static async LoadByPId(userId, pId) {
const data = await GetByPId(userId, pId);
console.log(`result ${JSON.stringify(data)}`);
}
Once it returned
[{ userId: 1, id: 1 ... - I want this
and the next time it returned
{ userId: 1, id: 1 ... - Don't want this
Whats going on and how can I get it to always return an array?
Update #1
Now it only returns a single result.
Update #2
It went from bad to worse.
Now my other basic functions don't work properly. Knex only works on the first set of parameters and fails for everything else.
For example, if the express server was restarted and send a request for userId: 1 and pId: 1, it works. If I repeat the same request with same parameters, it works. But if I change the parameters (ie userId or pId) to another valid set, it fails. I have to restart the express server before trying any other parameters. I tested this on my app and postman.
My express code looks like follows
router.post('/list', auth, async (req, res) => {
try {
const biz= await BizObj.LoadByPId(req.user.id, req.body.pId);
res.json(biz);
} catch (ex) {
console.log(ex);
res.status(400).json('Unauthorized');
}
});
Update #4
In case my knex config is the problem
development: {
client: 'postgresql',
connection: {
database: 'somedb',
user: 'SOMEUSER',
password: '',
timezone: 'UTC',
},
pool: {
min: 2,
max: 10,
},
migrations: {
tableName: 'knex_migrations',
},
},
Update #5
Single page of code (only missing express setup)
in pg db / SomeObj
id userId
1 1
2 2
3 2
code sample
import knex from 'knex';
import express from 'express';
const config = {
development: {
client: 'pg',
connection: {
database: 'somedb',
user: 'SOMEUSER',
password: '',
timezone: 'UTC',
},
pool: {
min: 2,
max: 10,
},
migrations: {
tableName: 'knex_migrations',
},
},
};
const knexed = knex(config.development);
const SQL = knexed('SomeObj');
const GetAll = async userId => SQL.where({ userId }).select(
'id',
'userId',
);
const GetById = async (userId, id) => SQL.where({ userId, id }).first(
'id',
'userId',
);
class SomeObj {
constructor(data, userId) {
this.userId = userId;
this.id = data.id;
}
static async LoadAll(userId) {
const data = await GetAll(userId);
if (!data || data.length === 0) return null;
return data.map(r => new SomeObj(r, userId));
}
static async Load(userId, id) {
const data = await GetById(userId, id);
if (!data) return null;
return new SomeObj(data, userId);
}
}
const router = express.Router();
router.post('/list', async (req, res) => {
try {
const res1 = await SomeObj.LoadAll(req.body.id); // works and returns array
const res2 = await SomeObj.Load(req.body.id, 2); // fails and returns undefined
res.json({ res1, res2 });
} catch (ex) {
res.status(401).json(ex);
}
});
No second query can be ran. Don't know if I'm missing something simple to close the connection.
Update #6
I swear knex is messing with me. Every time I try something (and revert back to confirm changes are due my new inputs), there are different responses. Now, both res1 and res2 return the correct result for the first request but the second request fails.
Update #7
Runkit example: https://runkit.com/tristargod/runkit-npm-knex
It runs for the first request but fails for all other requests on express server.
Update #8
Refer to https://github.com/tgriesser/knex/issues/2346#issuecomment-346757344 for further details. Thanks Mikael!

knex('table1')
.where({ userId, pId })
.select('userId', 'pId', 'id', 'created', 'updated')
Should return always an array of results. You are doing something else wrong that is not shown in the example.
Example code: https://runkit.com/embed/kew7v2lwpibn
RESPONSE TO UPDATE #7
tldr; Knex query builders are mutable so when re-using them .clone() is necessary. https://runkit.com/mikaelle/5a17c6d99cd063001284a20a
Nice example, from that it was easy to spot the problem 👍
You are reusing the same query builder multiple times without cloning it between the queries. If you would run your code with DEBUG=knex:* environment variable set, you would see that constructed queries are not correct after the first call.
const GetAll = async userId => SQL.clone().where({ userId }).select(
'id',
'userId',
);
const GetById = async (userId, id) => SQL.clone().where({ userId, id }).first(
'id',
'userId',
);

Send Response in 'then' after Promise resolves

I want to display the json i got from the search for localhost:8400/api/v1/search. But I have no idea how.
I'm using the Elasticsearch Javascript Client
my routing:
'use-strict';
const express = require('express');
const elasticsearch = require('../models/elasticsearch.js');
const router = express.Router();
router.get('/api/v1/search', elasticsearch.search);
for accessing the ElasticSearch DB
const es = require('elasticsearch');
let esClient = new es.Client({
host: 'localhost:9200',
log: 'info',
apiVersion: '5.3',
requestTimeout: 30000
})
let indexName = "randomindex";
const elasticsearch = {
search() {
return esClient.search({
index: indexName,
q: "test"
})
.then(() => {
console.log(JSON.stringify(body));
// here I want to return a Response with the Content of the body
})
.catch((error) => { console.trace(error.message); });
}
}
module.exports = elasticsearch;

Firstly, the route handlers for express routes always have (request, response, next) as it's parameters. You can use the response object to send data back to the client.
Instead of passing the elasticsearch.search method as a route handler, you can write your own route handler and call elasticsearch.search in there, so you still have access to the response object. For example:
function handleSearch(req, res, next) {
elasticsearch.search()
.then(function(data) {
res.json(data)
})
.catch(next)
}
And structure your search function like so:
const elasticsearch = {
search() {
return esClient.search({
index: indexName,
q: "test"
})
.then((body) => body) // just return the body from this method
}
}
This way you separate your concerns of querying elastic and handling the request. You also have access to the request object in case you want to pass any query string parameters from your request to your search function.

Since you add elasticsearch.search as the route handler, it will be invoked with some arguments.
Change the signature of the search method to search(req, res).
Then just call res.send(JSON.stringify(body));
See https://expressjs.com/en/4x/api.html#res for more details