I am currently trying to identify cost optimisation opportunity in my clients environment. Looking at their cloudwatch costs : They are primarily driven by- CW:Requests & CW:MetricMonitorUsage. Filtering it by API operation: PutMetricData is the main driver of costs. I need to identify the main drivers of these usage types and how to analyse them
I have used a javascript that lists all the metrics stored in CloudWatch and aggregates the sample count of each metric over the last 7 days. The metrics are sorted by the sample count in descending order, so the metrics with the highest sample count will be displayed at the top of the list.
The results have highlighted 90% total sample count is for ALB. Do these results relate to the CW:Requests or CW:MetricsMonitorUsage costs? How can I link it and calculate the associated costs?
Here is the script I used:
const AWS = require('aws-sdk')
// Set the region
AWS.config.update({ region: 'us-east-1' })
const DEBUG = false
// Create CloudWatch service object
const cloudwatch = new AWS.CloudWatch({ apiVersion: '2010-08-01' })
async function main () {
const metrics = await getMetrics(1)
const sum = await getSum(metrics)
const sorted = new Map([...sum.entries()].sort((a, b) => b[1] - a[1]))
for (const [key, value] of sorted.entries()) {
console.log(`${key}: ${value}`)
}
return sorted.size
}
async function getSum (metrics) {
const sum = new Map()
const promises = []
console.log('Getting Sample Counts')
for (const metric of metrics) {
promises.push(getSingleSum(metric))
}
console.log('Waiting for Sample Counts')
await Promise.all(promises).then(values => {
console.log('total metrics: ' + values.length)
for (const val of values) {
if (val.Datapoints && val.Datapoints[0]) {
if (sum.has(val.Label)) {
sum.set(val.Label, sum.get(val.Label) + val.Datapoints[0].SampleCount)
} else {
sum.set(val.Label, val.Datapoints[0].SampleCount)
}
}
}
console.log('total de-duplicated metrics: ' + sum.size)
})
return sum
}
const today = new Date()
const aWeekAgo = new Date(today.getTime() - 7 * 24 * 60 * 60 * 1000)
async function getSingleSum (metric) {
const params = { Statistics: ['SampleCount'], StartTime: aWeekAgo, EndTime: today, Period: 60 * 60 * 24 * 7, Namespace: metric.Namespace, Dimensions: metric.Dimensions, MetricName: metric.MetricName }
return cloudwatch.getMetricStatistics(params).promise()
}
async function getMetrics (count, token) {
console.log(`Iteration #${count}, ${500 * (count - 1)} metrics`)
var params = {}
if (token) {
params.NextToken = token
}
const metrics = []
const data = await cloudwatch.listMetrics(params).promise()
if (data && data.Metrics) {
metrics.push(...data.Metrics)
if (data.NextToken && count < 200) {
if (!DEBUG) {
metrics.push(...await getMetrics(count + 1, data.NextToken))
}
}
}
return metrics
}
main()
.then(data => { console.log(`total: ${data} metrics`) })
.catch(err => { console.error(`error: ${err}`) })type here
I am generating query based on user input and passing it onto LokiJS to handle. However, query doesn't seem to take effect (everything is returned as it is).
Logic to generate query :-
function prepareQueryFromOptions(options: GetAllTransactionsOptions) {
const query: Record<string, any> = {
timestamp: {
"$gte": options.dateRange.from.getTime(),
"$lte": options.dateRange.to.getTime(),
},
};
if (options.accounts) {
query.accountId = {
"$in": options.accounts,
};
}
if (options.categories) {
query.category = {
"$in": options.categories,
};
}
return query;
}
Using above function as below:
const query: Record<string, any> = prepareQueryFromOptions(options);
const transactions: Transaction[] = transactionsCollection.chain().find(query).simplesort('timestamp').data();
My Failing Testcase:
const AUG_1_2022 = new Date(2022, 7, 1);
const AUG_5_2022 = new Date(2022, 7, 5);
const AUG_12_2022 = new Date(2022, 7, 12);
it('getAllTransactions()', async () => {
let txs1 = await transactions.getAllTransactions({
dateRange: {
from: AUG_1_2022,
to: AUG_12_2022,
},
});
expect(txs1.length).toBe(2); // Successful (Both transactions are returned)
let txs = await transactions.getAllTransactions({
dateRange: {
from: AUG_1_2022,
to: AUG_5_2022,
},
});
expect(txs.length).toBe(1); // Failed since again both the transactions are returned
});
import { createRequire, runMain } from "module";
const require = createRequire(import.meta.url);
const Binance = require("node-binance-api");
const binance = new Binance().options({
APIKEY: "<API-KEY>",
APISECRET: "<API-SECRET>",
useServerTime: true,
});
async function Trade(interval, { symbol, EnterAt, Type }) {
try {
if ((interval == "1h" || interval == "4h") & (Type == "SHORT")) {
let response = await binance.futuresBalance();
let pair = response.filter((el) => el.asset == "USDT");
const leverage = 6;
let quantity = Math.round(
((parseFloat(pair[0].availableBalance) * leverage) / EnterAt).toFixed(2)
);
let takeProfit = EnterAt - (EnterAt * 0.6) / 100;
let stopMarket = EnterAt + (EnterAt * 1.1) / 100;
// Adjust leverage
let adjustLeverage = await binance.futuresLeverage(symbol, leverage);
// Futures Market Sell
let short = await binance.futuresMarketSell(symbol, quantity);
// Set TAKE PROFIT
let TP = await binance.futuresOrder("BUY", symbol, quantity, false, {
type: "TAKE_PROFIT_MARKET",
stopPrice: takeProfit.toFixed(2),
closePosition: true,
});
// Set STOP LOSS
let SL = await binance.futuresOrder("BUY", symbol, quantity, false, {
type: "STOP_MARKET",
workingType: "MARK_PRICE",
stopPrice: stopMarket.toFixed(2),
closePosition: true,
});
console.log("SHORT: ", short);
console.log("TP: ", TP);
console.log("SL: ", SL);
}
} catch (error) {
console.log(error.message);
}
}
function wrapUpTrade(interval, { symbol, EnterAt }) {
binance.useServerTime(() =>
Trade(interval, { symbol: symbol, EnterAt: EnterAt })
);
}
// wrapUpTrade("1h", { symbol: "XRPUSDT", shortAt: 0.774 });
export { wrapUpTrade };
This code is only for futures Market Sell (sell short). Why do I get an error of Insufficient Margin on placing an order of coin having a price of $4.5 and I have $2.9 in my wallet? On leveraging(6x) I will be having (6 * $2.9 = $17.4) that is greater than $4.5 still I get the error of Insufficient error. Function Trade takes some arguments:
Interval: "4h" or "1h"
symbol: "PAIR/(USDT)" // BTCUSDT, ETHUSDT
EnterAt: It is used to calculate Stoploss and TakeProfit. The type is int.
Type: "SHORT", means futuresSell
For a "flights" portfolio app, I am trying to generate tickets for each seat in a plane (so that flights are prepopulated with some people). This is part of a function that also prepopulates flights to the database to set up the data. There are calls to the database to check to see if the Flight Number, as well as Confirmation Numbers are unique, and not pre-existing. In the case of flights, if the flight number already exists, the flight just gets dropped. However, I am running into some issues with the asynchronous aspect of regenerating a confirmation number if one already exists when it is in loops. So essentially, the issues I'm having:
generateTickets returns an empty tickets array
I am not sure how to structure my nested forEach loops to return promises when generating the tickets for each seat, in each row. (In my searches on Stack Overflow, I have found multiple solutions referencing Promise.all with array.map. I've tried to implement these solutions in a variety of ways, but I'm still getting an empty array, and I think I'm just not understanding it well enough.)
I am getting: SyntaxError: Unexpected reserved word on await generateConfirmationNumber() - even though generateConfirmationNumber is an asynchronous function
I don't know what to say about what I've tried. It's a lot. I've searched through every link for the first 2-3 pages of Google, I've scoured Stack Overflow trying to find solutions, and then trying to implement them. I think I'm just having a hard time interpreting how their code works, when this has additional complexity. The solutions I have on my browser tab currently are:
Why doesn't the code after await run right away? Isn't it supposed to be non-blocking?
Pushing Elements into an Array That Is Inside a Callback Function
How to wait a Promise inside a forEach loop
Why does my async function return an empty array
Best way to wait for .forEach() to complete
Using async/await with a forEach loop
How to return values from async functions using async-await from function?
I've been googling and trying to get this to work for 3 days now, and I figured it was time to ask a question. :P So on to the code! (If you see crazy all caps comments, it just helps me notice them easily in the console.)
createFlight - Called by router, starts everything.
async function createFlight(req, res) {
create_flights().then(() => {
console.log('Done')
res.redirect('/')
})
}
create_flights - Async Function that generates a random flight.
const create_flights = async() => {
let flights_array = []
let flights_with_tickets = []
for(let i = 0; i < 1; i++) {
let airline = randomAirline();
let origin = randomAirport();
let destination = randomDestination(origin.code);
let duration = generateDuration()
let departure = generateDeparture()
let departure_date = dateString(departure)
let flight_data = {
number: generateFlightNumber(airline.code),
airline: airline.name,
plane: {
name: randomPlane()
},
origin: origin,
destination: destination,
departure: departure,
departure_date: departure_date,
duration: duration,
gate: generateGate(),
amenities: randomAmenities(duration),
tickets: []
}
const flight = new Flight(flight_data);
flights_array.push(flight)
}
console.log("FLIGHTS ARRAY")
console.log(flights_array)
for(let flight of flights_array) {
console.log(flight)
const flight_with_ticket = await returnTicketsAsync(flight)
console.log(flight_with_ticket)
if(flight_with_ticket) {
const record = await returnFlight(flight_with_ticket)
console.log(record)
if(record) {
console.log("Created Flight")
console.log(record.tickets[0])
}
}
}
}
returnTicketsAsync - Waits for the tickets to be generated. This is always console logging an empty tickets value.
async function returnTicketsAsync(flight) {
console.log("ASYNC TICKETS?!?!?!!?")
let tickets = await generateTickets(flight)
console.log(tickets)
}
returnFlight - This checks to see if the flight's confirmation number is already in the database. If it is, we just skip this flight and keep moving on. Otherwise, we save it to the database.
const returnFlight = flight => {
return new Promise((resolve, reject) => {
Flight.find({$or:[{number: flight.number}]}, function(error, records) {
if(records.length) {
console.log("Similar flight already exists")
resolve()
} else {
flight.save(function (error, result) {
if(error) {
console.log("ERROR:" + error)
resolve()
} else {
resolve(result)
console.log("Flight " + result.number + " successfully saved.")
}
})
}
})
})
}
generateTickets(flight) - The source of all my troubles. The Flight has a plane map, with sections (first class, preferred class, economy). Each section has rows with seats.
So I need to loop through first class, preferred class, and economy class, generate the tickets for all the seats. There is one function, generateConfirmationNumber() that generates the number, and checks the database for a result. If there's no result, the confirmation number is good to be used, and it returns that. I need to await for that to be done, but I'm getting a SyntaxError here.
async function generateTickets(flight) {
let tickets = []
let prices = generatePrice()
const first_class = await Promise.all(_.map(flight.plane.sections.first_class.rows, async function (row, i, rows) {
let first_class_tickets = await _.map(row, function(seat) {
if(!seat.isle) {
let passenger = generatePassenger()
let confirmation_number = await generateConfirmationNumber()
let tickets = {
confirmation_number: '',
tracking_number: generateTrackingNumber(),
data: generateDataNumber(),
options: '1ST CL',
flight_class: 'first',
seat: {
isle: seat.letter,
number: i + 1
},
boarding_zone: generateBoardingZone(),
price: prices.first,
passenger: passenger
}
if(passenger.first_name !== '' || passenger.first_name.length > 1) {
ticket.confirmation_number = confirmation_number
}
console.log(ticket)
tickets.push(ticket)
}
})
}))
const preferred_class = await Promise.all(_.map(flight.plane.sections.preferred_class.rows, async function (row, i, rows) {
let preferred_class_tickets = await _.map(row, function(seat) {
if(!seat.isle) {
let passenger = generatePassenger()
let confirmation_number = await generateConfirmationNumber()
let tickets = {
confirmation_number: '',
tracking_number: generateTrackingNumber(),
data: generateDataNumber(),
options: 'PREF PLUS',
flight_class: 'preferred',
seat: {
isle: seat.letter,
number: i + flight.plane.sections.first_class.total_rows + 1
},
boarding_zone: generateBoardingZone(),
price: prices.preferred,
passenger: passenger
}
if(passenger.first_name !== '' || passenger.first_name.length > 1) {
ticket.confirmation_number = confirmation_number
}
console.log(ticket)
tickets.push(ticket)
}
})
}))
const economy_class = await Promise.all(_.map(flight.plane.sections.economy_class.rows, async function (row, i, rows) {
let economy_class_tickets = await _.map(row, function(seat) {
if(!seat.isle) {
let passenger = generatePassenger()
let confirmation_number = await generateConfirmationNumber()
let tickets = {
confirmation_number: '',
tracking_number: generateTrackingNumber(),
data: generateDataNumber(),
options: 'PREF PLUS',
flight_class: 'preferred',
seat: {
isle: seat.letter,
number: i + flight.plane.sections.first_class.total_rows + flight.plane.sections.preferred_class.total_rows + 1
},
boarding_zone: generateBoardingZone(),
price: prices.economy,
passenger: passenger
}
if(passenger.first_name !== '' || passenger.first_name.length > 1) {
ticket.confirmation_number = confirmation_number
}
console.log(ticket)
tickets.push(ticket)
}
})
}))
console.log("Tickets")
console.log(ticekts)
}
generateConfirmationNumber - Just throwing this here, too.
async function generateConfirmationNumber() {
let letters = 'ABCDEFGHJKLMNPRSTUVWXYZ'
let number = Math.floor(Math.random() * 10000)
let confirmation = letters[Math.floor(Math.random() * letters.length)] + letters[Math.floor(Math.random() * letters.length)] + letters[Math.floor(Math.random() * letters.length)] + number
return new Promise((resolve, reject) => {
Flight.findOne({'tickets.confirmation_number': confirmation}, (err, result) => {
if (err) console.log(error)
else if (result) return resolve(generateConfirmationNumber())
else return resolve(confirmation)
})
})
}
I'm at a loss for how to get this working. The goal is simply to generate the tickets, wait for the generateConfirmationNumber() to finish, and push all the tickets into a "tickets" array for that flight. Then, return that array so it can be $set to the flight, and so then, the flight can be pushed into the database.
The main issue your codes doesn't work because you try to Promise.all() to a array of the ticket below.
Promise.all only working on promises object, but you can see the ticket itself is not a purely Promise object. Also, the map() function will not await the promise in the loop. If you want to await a promise in a loop, you need to use for loop.
let confirmation_number = await generateConfirmationNumber();
let ticket = {
confirmation_number: confirmation_number,
tracking_number: generateTrackingNumber(),
data: generateDataNumber(),
options: "1ST CL",
flight_class: "first",
seat: {
isle: seat.letter,
number: i + 1,
},
boarding_zone: generateBoardingZone(),
price: prices.first,
passenger: generatePassenger(),
};
The below codes doesn't work because ticket is not a promise object.
Promise.all([ticket1,ticket2,ticket3]);
To make it work, you have two options.
Promises in Parallel: Promise.all all the promises object and use array#map() to create a array of ticket objects.
Promises in Series: await each promise and construct the ticket one by one.
If you want to do a Promises in Parallel way, you can do like below:
const arrayOfPromises = flight.plane.sections.first_class.rows.flatMap(row =>
row.map(seat => generateConfirmationNumber())
);
const confirmationNumbers = await Promise.all(arrayOfPromises);
const tickets = confirmationNumbers.map(cfmNumber => ({
confirmation_number: cfmNumber,
tracking_number: generateTrackingNumber(),
data: generateDataNumber(),
options: "1ST CL",
flight_class: "first",
seat: {
isle: seat.letter,
number: i + 1,
},
boarding_zone: generateBoardingZone(),
price: prices.first,
passenger: generatePassenger(),
}));
The answer I use is 2nd option(Promises in Series).
I use for of loop to await each promise and create a ticket from it.
In the end, I add ticket to a array tickets.
async function generateTickets(flight) {
let tickets = [];
let prices = generatePrice();
const firstClass = [];
for (const row of flight.plane.sections.first_class.rows) {
for (const seat of row) {
if (!seat.isle) {
const passenger = generatePassenger();
const confirmation_number = await generateConfirmationNumber();
const ticket = {
confirmation_number: "",
tracking_number: generateTrackingNumber(),
data: generateDataNumber(),
options: "1ST CL",
flight_class: "first",
seat: {
isle: seat.letter,
number: flight.plane.sections.first_class.rows.indexOf(row) + 1,
},
boarding_zone: generateBoardingZone(),
price: prices.first,
passenger: passenger,
};
if (passenger.first_name !== "" || passenger.first_name.length > 1) {
ticket.confirmation_number = confirmation_number;
}
firstClass.push(ticket);
}
}
}
console.log(firstClass);
const preferredClassTickets = [];
for (const row of flight.plane.sections.preferred_class.rows) {
for (const seat of row) {
if (!seat.isle) {
let passenger = generatePassenger();
let confirmation_number = await generateConfirmationNumber();
let ticket = {
confirmation_number: "",
tracking_number: generateTrackingNumber(),
data: generateDataNumber(),
options: "PREF PLUS",
flight_class: "preferred",
seat: {
isle: seat.letter,
number: flight.plane.sections.preferred_class.rows.indexOf(row) + flight.plane.sections.first_class.total_rows + 1,
},
boarding_zone: generateBoardingZone(),
price: prices.preferred,
passenger: passenger,
};
if (passenger.first_name !== "" || passenger.first_name.length > 1) {
ticket.confirmation_number = confirmation_number;
}
preferredClassTickets.push(ticket);
}
}
}
const economyClass = [];
for (const row of flight.plane.sections.economy_class.rows) {
for (const seat of row) {
if (!seat.isle) {
let passenger = generatePassenger();
let confirmation_number = await generateConfirmationNumber();
let ticket = {
confirmation_number: "",
tracking_number: generateTrackingNumber(),
data: generateDataNumber(),
options: "PREF PLUS",
flight_class: "preferred",
seat: {
isle: seat.letter,
number:
flight.plane.sections.economy_class.rows.indexOf(row) +
flight.plane.sections.first_class.total_rows +
flight.plane.sections.preferred_class.total_rows +
1,
},
boarding_zone: generateBoardingZone(),
price: prices.economy,
passenger: passenger,
};
if (passenger.first_name !== "" || passenger.first_name.length > 1) {
ticket.confirmation_number = confirmation_number;
}
economyClass.push(ticket);
}
}
}
tickets.push(...firstClass, ...preferredClassTickets, ...economyClass);
return tickets;
}
async function generateConfirmationNumber() {
let letters = "ABCDEFGHJKLMNPRSTUVWXYZ";
let number = Math.floor(Math.random() * 10000);
let confirmation =
letters[Math.floor(Math.random() * letters.length)] +
letters[Math.floor(Math.random() * letters.length)] +
letters[Math.floor(Math.random() * letters.length)] +
number;
return new Promise((resolve, reject) => {
Flight.findOne(
{ "tickets.confirmation_number": confirmation },
(err, result) => {
if (err) reject(error);
// else if (result) return resolve(generateConfirmationNumber());
else if (result) return resolve(result);
else return resolve(confirmation);
}
);
});
}
const returnFlight = flight => {
return new Promise((resolve, reject) => {
Flight.find(
{ $or: [{ number: flight.number }] },
function (error, records) {
if (error) reject(error);
if (records.length) {
console.log("Similar flight already exists");
resolve(records);
} else {
flight.save(function (error, result) {
if (error) {
console.log("ERROR:" + error);
reject(error);
} else {
console.log("Flight " + result.number + " successfully saved.");
resolve(result);
}
});
}
}
);
});
};
const create_flights = async () => {
let flights_array = [];
// let flights_with_tickets = [];
for (let i = 0; i < 1; i++) {
let airline = randomAirline();
let origin = randomAirport();
let destination = randomDestination(origin.code);
let duration = generateDuration();
let departure = generateDeparture();
let departure_date = dateString(departure);
let flight_data = {
number: generateFlightNumber(airline.code),
airline: airline.name,
plane: {
name: randomPlane(),
},
origin: origin,
destination: destination,
departure: departure,
departure_date: departure_date,
duration: duration,
gate: generateGate(),
amenities: randomAmenities(duration),
tickets: [],
};
const flight = new Flight(flight_data);
flights_array.push(flight);
}
console.log("FLIGHTS ARRAY");
console.log(flights_array);
for (let flight of flights_array) {
const tickets = await generateTickets(flight);
if (tickets) {
flight.set({
tickets: tickets
})
const record = await returnFlight(flight);
console.log(record);
if (record) {
console.log("Created Flight");
console.log(record.tickets);
}
}
}
};
async function createFlight(req, res) {
try {
await create_flights();
console.log("Done");
res.redirect("/");
} catch (err) {
console.log(err);
}
}
So I need to take data only from the day before. Example: today is 2018/9/25, I need the data to be taken on 2018/9/24 only. But from my code below, it takes from 23 until 25. Which is more than one day, and it took also two days before from the date I need. I don't know which code that make a wrong result. Anyone can help me with this? I really appreciate it.
Api.js
const TO_DAYS = 4194304 * 1000 * 60 * 60 * 24; // this part that might be the cause
const ROOT_DATE = moment([2018, 3, 30]); // this part that might be the cause
const ROOT_DATE_ID = 440557948108800000; // this part that might be the cause
const DATE_ID = function(date) {
return ROOT_DATE_ID + date.diff(ROOT_DATE, "days") * TO_DAYS;
}; // this part that might be the cause
class discordApi {
constructor() {
this.lock = new AsyncLock();
}
get(momentDate, authorId, offset = 0) {
const url =
config.endpoint +
querystring.stringify({
min_id: DATE_ID(momentDate),
author_id: authorId
});
return fetch(url, {
headers: {
method: "GET",
Authorization: config.auth
}
}).then(res => {
// console.log(res.url);
return res.json();
});
}
async getAllData(momentDate) {
const allData = config.targets.map(author_id =>
this.get(momentDate, author_id)
);
return Promise.all(allData);
}
index.js
var yesterday = moment().subtract(1, "days"); // this part that might be the cause
async function sendEmail() {
const data = await discordApi.getAllData(yesterday);
const unfilteredMessages = data.reduce(
(prev, current) => [...prev, ...current.messages],
[]
);
const filteredMessages = unfilteredMessages.reduce((prev, current) => {
if (prev.length === 0) {
return [...prev, current];
}
const currentConversationIsDuplicated = isConversationDuplicated(
prev[prev.length - 1],
current
);
if (currentConversationIsDuplicated) {
return prev;
}
ret
urn [...prev, current];
}, []);
const convo = await discordApi.AllConvo(filteredMessages);
const mailOptions = {
from: "lala#gmail.com",
to: maillist,
subject: "Discord-Bot Daily Data",
html: convo
};
transporter.sendMail(mailOptions, function(err, info) {
if (err) console.log(err);
else console.log("Message Sent!");
});
}