We Keep Coding

How to throw custom error for KafkaJS client when initialization fails?

I am using the following code to initialize the Kafka client:
this.kafka = new Kafka({
clientId: <my_client_ID>,
brokers: [
`${process.env.KAFKA_BROKER_1}`,
`${process.env.KAFKA_BROKER_2}`,
`${process.env.KAFKA_BROKER_3}`,
],
retry: {
initialRetryTime: 3000,
retries: 3,
},
});
Now if there's an issue with connecting to the brokers it will throw errors like this:
{"level":"ERROR","timestamp":"2022-10-19T04:21:08.143Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"<broker_1>","clientId":"<my_client_id"}
{"level":"ERROR","timestamp":"2022-10-19T04:21:08.144Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":0,"retryTime":299}
{"level":"ERROR","timestamp":"2022-10-19T04:21:08.143Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"<broker_2>","clientId":"<my_client_id"}
{"level":"ERROR","timestamp":"2022-10-19T04:21:09.447Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":1,"retryTime":564}
{"level":"ERROR","timestamp":"2022-10-19T04:21:08.143Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"<broker_3>","clientId":"<my_client_id"}
{"level":"ERROR","timestamp":"2022-10-19T04:21:11.014Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":2,"retryTime":1008}
Now, I want to change the log message here OR add a custom message after this happens. I am thinking of wrapping it in a try-catch block but for some reason it's not throwing the exception. So:
Is there a way to change this out of the box log to a custom log?
How can I add a custom long if (1) is not possible if the Kafka client fails to initialize?
Thanks.

From what I understand, you want to add custom logs to handle certain error scenarios.
Kafkajs provides a way to create our own custom logs and use them instead of default ones.
here is the reference for the same. You can check your version of kafkajs to avoid compatibility issues. Have added an example below.
{
level: 4,
label: 'INFO', // NOTHING, ERROR, WARN, INFO, or DEBUG
timestamp: '2017-12-29T13:39:54.575Z',
logger: 'kafkajs',
message: 'Started',
// ... any other extra key provided to the log function
}
const { logLevel } = require('kafkajs')
const winston = require('winston')
const toWinstonLogLevel = level => switch(level) {
case logLevel.ERROR:
case logLevel.NOTHING:
return 'error'
case logLevel.WARN:
return 'warn'
case logLevel.INFO:
return 'info'
case logLevel.DEBUG:
return 'debug'
}
const WinstonLogCreator = logLevel => {
const logger = winston.createLogger({
level: toWinstonLogLevel(logLevel),
transports: [
new winston.transports.Console(),
new winston.transports.File({ filename: 'myapp.log' })
]
})
return ({ namespace, level, label, log }) => {
const { message, ...extra } = log
logger.log({
level: toWinstonLogLevel(level),
message,
extra,
})
}
}
const kafka = new Kafka({
clientId: 'my-app',
brokers: ['kafka1:9092', 'kafka2:9092'],
logLevel: logLevel.ERROR,
logCreator: WinstonLogCreator
})
Thanks.

Google cloud dataflow job creation error: "Cannot set worker pool zone. Please check whether the worker_region experiments flag is valid"

I try to create a dataflow job to index a bigquery table into elasticSearchwith the node package google-cloud/dataflow.v1beta3.
The job is working fine when it's created and launched from the google cloud console, but I have the following error when I try it in node:
Error: 3 INVALID_ARGUMENT: (b69ddc3a5ef1c40b): Cannot set worker pool zone. Please check whether the worker_region experiments flag is valid. Causes: (b69ddc3a5ef1cd76): An internal service error occurred.
I tried to specify the experiments params in various ways but I always end up with the same error.
Does anyone managed to get a similar dataflow job working? Or do you have information about dataflow experiments?
Here is the code:
const { JobsV1Beta3Client } = require('#google-cloud/dataflow').v1beta3
const dataflowClient = new JobsV1Beta3Client()
const response = await dataflowClient.createJob({
projectId: 'myGoogleCloudProjectId',
location: 'europe-west1',
job: {
launch_parameter: {
jobName: 'indexation-job',
containerSpecGcsPath: 'gs://dataflow-templates-europe-west1/latest/flex/BigQuery_to_Elasticsearch',
parameters: {
inputTableSpec: 'bigQuery-table-gs-adress',
connectionUrl: 'elastic-endpoint-url',
index: 'elastic-index',
elasticsearchUsername: 'username',
elasticsearchPassword: 'password'
}
},
environment: {
experiments: ['worker_region']
}
}
})
Thank you very much for your help.

After many attempts I manage yesterday to find how to specify the worker region.
It looks like this:
await dataflowClient.createJob({
projectId,
location,
job: {
name: 'jobName',
type: 'Batch',
containerSpecGcsPath: 'gs://dataflow-templates-europe-west1/latest/flex/BigQuery_to_Elasticsearch',
pipelineDescription: {
inputTableSpec: 'bigquery-table',
connectionUrl: 'elastic-url',
index: 'elastic-index',
elasticsearchUsername: 'username',
elasticsearchPassword: 'password',
project: projectId,
appName: 'BigQueryToElasticsearch'
},
environment: {
workerPools: [
{ region: 'europe-west1' }
]
}
}
})
It's not working yet, I need to find the correct way to provide the other parameters, but now the dataflow job is created in the google cloud console.

For anyone who would be struggling with this issue, I finally found how to launch a dataflow job from a template.
There is a function launchFlexTemplate that work the same way as the job creation in the google cloud console.
Here is the final function working correctly:
const { FlexTemplatesServiceClient } = require('#google-cloud/dataflow').v1beta3
const response = await dataflowClient.launchFlexTemplate({
projectId: 'google-project-id',
location: 'europe-west1',
launchParameter: {
jobName: 'job-name',
containerSpecGcsPath: 'gs://dataflow-templates-europe-west1/latest/flex/BigQuery_to_Elasticsearch',
parameters: {
apiKey: 'elastic-api-key', //mandatory but not used if you provide username and password
connectionUrl: 'elasticsearch endpoint',
index: 'elasticsearch index',
elasticsearchUsername: 'username',
elasticsearchPassword: 'password',
inputTableSpec: 'bigquery source table', //projectid:datasetId.table
//parameters to upsert elasticsearch index
propertyAsId: 'table index use for elastic _id',
usePartialUpdate: true,
bulkInsertMethod: 'INDEX'
}
}

Unable to deploy the Node.js AWS Lambda function to Docker

I am developing a REST API with Node.Js. my technology stack is AWS Lambda, API Gateway and RDS (MySQL). Below is my code
roles.js
const mysql = require('mysql');
const con = mysql.createConnection({
host : "*****.rds.amazonaws.com",
user : "*****",
password : "*****",
port : 3306,
database : "******"
});
exports.lambdaHandler = (event, context, callback) => {
// allows for using callbacks as finish/error-handlers
context.callbackWaitsForEmptyEventLoop = false;
const sql = "select * from role";
con.query(sql, function (err, result) {
if (err) throw err;
var response = {
"statusCode": 200,
"headers": {
"Content-Type": "application/json"
},
"body": JSON.stringify(result),
"isBase64Encoded": false
};
callback(null, response)
});
};
exports.selectRoleByIDHandler = (event, context, callback) => {
const { id } = event.queryStringParameters;
console.log("id", id);
// allows for using callbacks as finish/error-handlers
context.callbackWaitsForEmptyEventLoop = false;
const sql = "select * from role where idRole = "+id;
con.query(sql, function (err, result) {
if (err) throw err;
var response = {
"statusCode": 200,
"headers": {
"Content-Type": "application/json"
},
"body": JSON.stringify(result),
"isBase64Encoded": false
};
callback(null, response)
});
};
Below is my template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
node2
Sample SAM Template for node2
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 100
VpcConfig:
SecurityGroupIds:
- sg-sdsdsdsd
SubnetIds:
- subnet-ssdsds
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs14.x
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
Role: !GetAtt LambdaRole.Arn
RoleFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello-world/
Handler: roles.lambdaHandler
Runtime: nodejs14.x
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /roles
Method: get
Role: !GetAtt LambdaRole.Arn
SelectRolesByIDFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello-world/
Handler: roles.selectRoleByIDHandler
Runtime: nodejs14.x
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /selectRoleByIDHandler
Method: get
Role: !GetAtt LambdaRole.Arn
LambdaRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: root
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- ec2:DescribeNetworkInterfaces
- ec2:CreateNetworkInterface
- ec2:DeleteNetworkInterface
- ec2:DescribeInstances
- ec2:AttachNetworkInterface
Resource: '*'
When I try to execute the sam local invoke, i get the following error
Error: You must provide a function logical ID when there are more than one functions in your template. Possible options in your template: ['HelloWorldFunction', 'RoleFunction', 'SelectRolesByIDFunction']
Now, I have 2 questions.
How to solve this issue?
Is it a bad practice to have more than one function in a single file in AWS Lambda?

Looks like you are trying to build Lambda functions via the AWS SDK for Javascipt. Have you looked at the AWS examples in the AWS SDK for JavaScript V3 DEV Guide. There are end to end instructions on how to build Lambda function using the JS API that can help you.
See this topic and the child topics in the TOC:
Cross-service examples for the AWS SDK for JavaScript

I found the issue, I had to invoke this as sam local invoke HelloWorldFunction . The HelloWorldFunction is the function name I need to deploy.

SailsJs websocket with a custom route instead of blueprint?

I am following the official Sails docs. Would like to implement the most basic socket functionality, i.e. client connects to a socket and when server notifies it about a response, executes a script.
The problem is that the socket requests are http and I am getting badRequest.
What's the right way to register a socket route in Sails?
My client code:
io.socket.on('hello', function (data) {
console.log('Socket `' + data.id + '` joined the party!')
})
io.socket.get('/sayhello', function gotResponse(data, jwRes) {
console.log('Server responded with status code ' + jwRes.statusCode + ' and data: ', data);
});
The controller:
module.exports = {
exits: {
badRequest: {
responseType: 'badRequest',
description: 'The provided data is invalid.',
},
},
fn: async function (req, res) {
if (!req.isSocket) {
return res.badRequest();
}
sails.sockets.join(req, 'funSockets');
sails.sockets.broadcast('funSockets', 'hello', {howdy: 'hi there!'}, req);
return res.json({
anyData: 'we want to send back'
});
}
}
The route:
'GET /sayhello': { action: 'project/api/app-socket' },

In your routes.js file you have:
'GET /sayhello': { action: 'project/api/app-socket' },
Add to this isSocket: true. So make it:
'GET /sayhello': { action: 'project/api/app-socket', isSocket: true },
How I learned this?
The convention for subscribe endpoints is to use an action prefixed with "subscribe", so when I generated an action with this command and this prefix:
sails generate action task/subscribe-to-task
Then it gave me this hint in the terminal output:
Successfully generated:
•- api/controllers/task/subscribe-to-task.js
A few reminders:
(1) For most projects, you'll need to manually configure an explicit route
in your `config/routes.js` file; e.g.
'GET /api/v1/task/subscribe-to-task': { action: 'task/subscribe-to-task', isSocket: true },
That's how I learned that we needed to add isSocket: true.

Is it possible to create a meeting request for outlook with nodeJS?

I am developing a very basic calendar with Angular and Node and I haven't found any code on this.
Workflow is the following : create an event, input the recipient's e-mail address, validate the event.
This triggers an e-mail sent to the recipient. The mail should be in the outlook meeting request format (not an attached object).
This means that when received in outlook the meeting is automatically added in the calendar.
Is this possible? If yes is it possible with only javascript on Node side?

For those still looking for an answer, here's how I managed to get the perfect solution for me.
I used iCalToolkit to create a calendar object.
It's important to make sure all the relevant fields are set up (organizer and attendees with RSVP).
Initially I was using the Postmark API service to send my emails but this solution was only working by sending an ics.file attachment.
I switched to the Postmark SMTP service where you can embed the iCal data inside the message and for that I used nodemailer.
This is what it looks like :
var icalToolkit = require('ical-toolkit');
var postmark = require('postmark');
var client = new postmark.Client('xxxxxxxKeyxxxxxxxxxxxx');
var nodemailer = require('nodemailer');
var smtpTransport = require('nodemailer-smtp-transport');
//Create a iCal object
var builder = icalToolkit.createIcsFileBuilder();
builder.method = meeting.method;
//Add the event data
var icsFileContent = builder.toString();
var smtpOptions = {
host:'smtp.postmarkapp.com',
port: 2525,
secureConnection: true,
auth:{
user:'xxxxxxxKeyxxxxxxxxxxxx',
pass:'xxxxxxxPassxxxxxxxxxxx'
}
};
var transporter = nodemailer.createTransport(smtpTransport(smtpOptions));
var mailOptions = {
from: 'message#domain.com',
to: meeting.events[0].attendees[i].email,
subject: 'Meeting to attend',
html: "Anything here",
text: "Anything here",
alternatives: [{
contentType: 'text/calendar; charset="utf-8"; method=REQUEST',
content: icsFileContent.toString()
}]
};
//send mail with defined transport object
transporter.sendMail(mailOptions, function(error, info){
if(error){
console.log(error);
}
else{
console.log('Message sent: ' + info.response);
}
});
This sends a real meeting request with the Accept, decline and Reject button.
It's really unbelievable the amount of work you need to go through for such a trivial functionality and how all of this not well documented.
Hope this helps.

If you do not want to use smtp server approach in earlier accepted solution, you have Exchange focused solution available. Whats wrong in current accepted answer? it does not create a meeting in sender's Calendar, you do not have ownership of the meeting item for further modification by the sender Outlook/OWA.
here is code snippet in javascript using npm package ews-javascript-api
var ews = require("ews-javascript-api");
var credentials = require("../credentials");
ews.EwsLogging.DebugLogEnabled = false;
var exch = new ews.ExchangeService(ews.ExchangeVersion.Exchange2013);
exch.Credentials = new ews.ExchangeCredentials(credentials.userName, credentials.password);
exch.Url = new ews.Uri("https://outlook.office365.com/Ews/Exchange.asmx");
var appointment = new ews.Appointment(exch);
appointment.Subject = "Dentist Appointment";
appointment.Body = new ews.TextBody("The appointment is with Dr. Smith.");
appointment.Start = new ews.DateTime("20170502T130000");
appointment.End = appointment.Start.Add(1, "h");
appointment.Location = "Conf Room";
appointment.RequiredAttendees.Add("user1#constoso.com");
appointment.RequiredAttendees.Add("user2#constoso.com");
appointment.OptionalAttendees.Add("user3#constoso.com");
appointment.Save(ews.SendInvitationsMode.SendToAllAndSaveCopy).then(function () {
console.log("done - check email");
}, function (error) {
console.log(error);
});

Instead of using 'ical-generator', I used 'ical-toolkit' to build a calender invite event.
Using this, the invite directly gets appended in the email instead of the attached .ics file object.
Here is a sample code:
const icalToolkit = require("ical-toolkit");
//Create a builder
var builder = icalToolkit.createIcsFileBuilder();
builder.method = "REQUEST"; // The method of the request that you want, could be REQUEST, PUBLISH, etc
//Add events
builder.events.push({
start: new Date(2020, 09, 28, 10, 30),
end: new Date(2020, 09, 28, 12, 30),
timestamp: new Date(),
summary: "My Event",
uid: uuidv4(), // a random UUID
categories: [{ name: "MEETING" }],
attendees: [
{
rsvp: true,
name: "Akarsh ****",
email: "Akarsh **** <akarsh.***#abc.com>"
},
{
rsvp: true,
name: "**** RANA",
email: "**** RANA <****.rana1#abc.com>"
}
],
organizer: {
name: "A****a N****i",
email: "A****a N****i <a****a.r.n****i#abc.com>"
}
});
//Try to build
var icsFileContent = builder.toString();
//Check if there was an error (Only required if yu configured to return error, else error will be thrown.)
if (icsFileContent instanceof Error) {
console.log("Returned Error, you can also configure to throw errors!");
//handle error
}
var mailOptions = { // Set the values you want. In the alternative section, set the calender file
from: obj.from,
to: obj.to,
cc: obj.cc,
subject: result.email.subject,
html: result.email.html,
text: result.email.text,
alternatives: [
{
contentType: 'text/calendar; charset="utf-8"; method=REQUEST',
content: icsFileContent.toString()
}
]
}
//send mail with defined transport object
transporter.sendMail(mailOptions, function(error, info){
if(error){
console.log(error);
}
else{
console.log('Message sent: ' + info.response);
}
});

It should be possible as long as you can use SOAP in Node and also if you can use NTLM authentication for Exchange with Node. I believe there are modules for each.
I found this blog very helpful when designing a similar system using PHP

Please check the following sample:
const options = {
authProvider,
};
const client = Client.init(options);
const onlineMeeting = {
startDateTime: '2019-07-12T14:30:34.2444915-07:00',
endDateTime: '2019-07-12T15:00:34.2464912-07:00',
subject: 'User Token Meeting'
};
await client.api('/me/onlineMeetings')
.post(onlineMeeting);
More Information: https://learn.microsoft.com/en-us/graph/api/application-post-onlinemeetings?view=graph-rest-1.0&tabs=http