I am currently trying to identify cost optimisation opportunity in my clients environment. Looking at their cloudwatch costs : They are primarily driven by- CW:Requests & CW:MetricMonitorUsage. Filtering it by API operation: PutMetricData is the main driver of costs. I need to identify the main drivers of these usage types and how to analyse them
I have used a javascript that lists all the metrics stored in CloudWatch and aggregates the sample count of each metric over the last 7 days. The metrics are sorted by the sample count in descending order, so the metrics with the highest sample count will be displayed at the top of the list.
The results have highlighted 90% total sample count is for ALB. Do these results relate to the CW:Requests or CW:MetricsMonitorUsage costs? How can I link it and calculate the associated costs?
Here is the script I used:
const AWS = require('aws-sdk')
// Set the region
AWS.config.update({ region: 'us-east-1' })
const DEBUG = false
// Create CloudWatch service object
const cloudwatch = new AWS.CloudWatch({ apiVersion: '2010-08-01' })
async function main () {
const metrics = await getMetrics(1)
const sum = await getSum(metrics)
const sorted = new Map([...sum.entries()].sort((a, b) => b[1] - a[1]))
for (const [key, value] of sorted.entries()) {
console.log(`${key}: ${value}`)
}
return sorted.size
}
async function getSum (metrics) {
const sum = new Map()
const promises = []
console.log('Getting Sample Counts')
for (const metric of metrics) {
promises.push(getSingleSum(metric))
}
console.log('Waiting for Sample Counts')
await Promise.all(promises).then(values => {
console.log('total metrics: ' + values.length)
for (const val of values) {
if (val.Datapoints && val.Datapoints[0]) {
if (sum.has(val.Label)) {
sum.set(val.Label, sum.get(val.Label) + val.Datapoints[0].SampleCount)
} else {
sum.set(val.Label, val.Datapoints[0].SampleCount)
}
}
}
console.log('total de-duplicated metrics: ' + sum.size)
})
return sum
}
const today = new Date()
const aWeekAgo = new Date(today.getTime() - 7 * 24 * 60 * 60 * 1000)
async function getSingleSum (metric) {
const params = { Statistics: ['SampleCount'], StartTime: aWeekAgo, EndTime: today, Period: 60 * 60 * 24 * 7, Namespace: metric.Namespace, Dimensions: metric.Dimensions, MetricName: metric.MetricName }
return cloudwatch.getMetricStatistics(params).promise()
}
async function getMetrics (count, token) {
console.log(`Iteration #${count}, ${500 * (count - 1)} metrics`)
var params = {}
if (token) {
params.NextToken = token
}
const metrics = []
const data = await cloudwatch.listMetrics(params).promise()
if (data && data.Metrics) {
metrics.push(...data.Metrics)
if (data.NextToken && count < 200) {
if (!DEBUG) {
metrics.push(...await getMetrics(count + 1, data.NextToken))
}
}
}
return metrics
}
main()
.then(data => { console.log(`total: ${data} metrics`) })
.catch(err => { console.error(`error: ${err}`) })type here
Related
I'm currently working with the cardano serialization library to code crypto transactions in typescript. I'm referencing a wallet connector template that has example code for transactions but is all written in Javascript. So in translating it over I've received a number of syntax errors that have generally been easy to resolve until now. I have a number of ts(2532) errors (Object is possibly 'undefined' that are solved through '?' chaining, I know that's not my best solution but for now I'm trying to get it error free first. When I utilize a ? operator chain though I then get the property does not exist on type 'never' error in the following code
getNetworkId = async () => {
try {
const networkId = await this?.API?.getNetworkId()
this.setState({ networkId })
} catch (err) {
console.log(err)
} }
Below I've included the majority of the code down until that error
export default class App extends React.Component<any, any> {
API: undefined
protocolParams: {
linearFee: { minFeeA: string; minFeeB: string }
minUtxo: string
poolDeposit: string
keyDeposit: string
maxValSize: number
maxTxSize: number
priceMem: number
priceStep: number
coinsPerUtxoWord: string
}
constructor(props: any) {
super(props)
this.state = {
selectedTabId: '1',
whichWalletSelected: undefined,
walletFound: false,
walletIsEnabled: false,
walletName: undefined,
walletIcon: undefined,
walletAPIVersion: undefined,
wallets: [],
networkId: undefined,
Utxos: undefined,
CollatUtxos: undefined,
balance: undefined,
changeAddress: undefined,
rewardAddress: undefined,
usedAddress: undefined,
txBody: undefined,
txBodyCborHex_unsigned: '',
txBodyCborHex_signed: '',
submittedTxHash: '',
addressBech32SendADA:
'addr_test1qrt7j04dtk4hfjq036r2nfewt59q8zpa69ax88utyr6es2ar72l7vd6evxct69wcje5cs25ze4qeshejy828h30zkydsu4yrmm',
lovelaceToSend: 3000000,
assetNameHex: '4c494645',
assetPolicyIdHex:
'ae02017105527c6c0c9840397a39cc5ca39fabe5b9998ba70fda5f2f',
assetAmountToSend: 5,
addressScriptBech32:
'addr_test1wpnlxv2xv9a9ucvnvzqakwepzl9ltx7jzgm53av2e9ncv4sysemm8',
datumStr: '12345678',
plutusScriptCborHex: '4e4d01000033222220051200120011',
transactionIdLocked: '',
transactionIndxLocked: 0,
lovelaceLocked: 3000000,
manualFee: 900000,
}
/**
* When the wallet is connect it returns the connector which is
* written to this API variable and all the other operations
* run using this API object
*/
this.API = undefined
/**
* Protocol parameters
* #type {{
* keyDeposit: string,
* coinsPerUtxoWord: string,
* minUtxo: string,
* poolDeposit: string,
* maxTxSize: number,
* priceMem: number,
* maxValSize: number,
* linearFee: {minFeeB: string, minFeeA: string}, priceStep: number
* }}
*/
this.protocolParams = {
linearFee: {
minFeeA: '44',
minFeeB: '155381',
},
minUtxo: '34482',
poolDeposit: '500000000',
keyDeposit: '2000000',
maxValSize: 5000,
maxTxSize: 16384,
priceMem: 0.0577,
priceStep: 0.0000721,
coinsPerUtxoWord: '34482',
}
this.pollWallets = this.pollWallets.bind(this)
}
/**
* Poll the wallets it can read from the browser.
* Sometimes the html document loads before the browser initialized browser plugins (like Nami or Flint).
* So we try to poll the wallets 3 times (with 1 second in between each try).
*
* Note: CCVault and Eternl are the same wallet, Eternl is a rebrand of CCVault
* So both of these wallets as the Eternl injects itself twice to maintain
* backward compatibility
*
* #param count The current try count.
*/
pollWallets = (count = 0) => {
const wallets = []
for (const key in window.cardano) {
if (window.cardano[key].enable && wallets.indexOf(key) === -1) {
wallets.push(key)
}
}
if (wallets.length === 0 && count < 3) {
setTimeout(() => {
this.pollWallets(count + 1)
}, 1000)
return
}
this.setState(
{
wallets,
whichWalletSelected: wallets[0],
},
() => {
this.refreshData()
}
)
}
/**
* Handles the tab selection on the user form
* #param tabId
*/
handleTabId = (tabId: any) => this.setState({ selectedTabId: tabId })
/**
* Handles the radio buttons on the form that
* let the user choose which wallet to work with
* #param obj
*/
handleWalletSelect = (obj: { target: { value: any } }) => {
const whichWalletSelected = obj.target.value
this.setState({ whichWalletSelected }, () => {
this.refreshData()
})
}
/**
* Generate address from the plutus contract cborhex
*/
generateScriptAddress = () => {
// cborhex of the alwayssucceeds.plutus
// const cborhex = "4e4d01000033222220051200120011";
// const cbor = Buffer.from(cborhex, "hex");
// const blake2bhash = blake.blake2b(cbor, 0, 28);
const script = PlutusScript.from_bytes(
Buffer.from(this.state.plutusScriptCborHex, 'hex')
)
// const blake2bhash = blake.blake2b(script.to_bytes(), 0, 28);
const blake2bhash =
'67f33146617a5e61936081db3b2117cbf59bd2123748f58ac9678656'
const scripthash = ScriptHash.from_bytes(Buffer.from(blake2bhash, 'hex'))
const cred = StakeCredential.from_scripthash(scripthash)
const networkId = NetworkInfo.testnet().network_id()
const baseAddr = EnterpriseAddress.new(networkId, cred)
const addr = baseAddr.to_address()
const addrBech32 = addr.to_bech32()
// hash of the address generated from script
console.log(Buffer.from(addr.to_bytes(), 'utf8').toString('hex'))
// hash of the address generated using cardano-cli
const ScriptAddress = Address.from_bech32(
'addr_test1wpnlxv2xv9a9ucvnvzqakwepzl9ltx7jzgm53av2e9ncv4sysemm8'
)
console.log(Buffer.from(ScriptAddress.to_bytes(), 'utf8').toString('hex'))
console.log(ScriptAddress.to_bech32())
console.log(addrBech32)
}
/**
* Checks if the wallet is running in the browser
* Does this for Nami, Eternl and Flint wallets
* #returns {boolean}
*/
checkIfWalletFound = () => {
const walletKey = this.state.whichWalletSelected
const walletFound = !!window?.cardano?.[walletKey]
this.setState({ walletFound })
return walletFound
}
/**
* Checks if a connection has been established with
* the wallet
* #returns {Promise<boolean>}
*/
checkIfWalletEnabled = async () => {
let walletIsEnabled = false
try {
const walletName = this.state.whichWalletSelected
walletIsEnabled = await window.cardano[walletName].isEnabled()
} catch (err) {
console.log(err)
}
this.setState({ walletIsEnabled })
return walletIsEnabled
}
/**
* Enables the wallet that was chosen by the user
* When this executes the user should get a window pop-up
* from the wallet asking to approve the connection
* of this app to the wallet
* #returns {Promise<boolean>}
*/
enableWallet = async () => {
const walletKey = this.state.whichWalletSelected
try {
this.API = await window.cardano[walletKey].enable()
} catch (err) {
console.log(err)
}
return this.checkIfWalletEnabled()
}
/**
* Get the API version used by the wallets
* writes the value to state
* #returns {*}
*/
getAPIVersion = () => {
const walletKey = this.state.whichWalletSelected
const walletAPIVersion = window?.cardano?.[walletKey].apiVersion
this.setState({ walletAPIVersion })
return walletAPIVersion
}
/**
* Get the name of the wallet (nami, eternl, flint)
* and store the name in the state
* #returns {*}
*/
getWalletName = () => {
const walletKey = this.state.whichWalletSelected
const walletName = window?.cardano?.[walletKey].name
this.setState({ walletName })
return walletName
}
/**
* Gets the Network ID to which the wallet is connected
* 0 = testnet
* 1 = mainnet
* Then writes either 0 or 1 to state
* #returns {Promise<void>}
*/
getNetworkId = async () => {
try {
const networkId = await this?.API?.getNetworkId()
this.setState({ networkId })
} catch (err) {
console.log(err)
}
}
/**
* Gets the UTXOs from the user's wallet and then
* stores in an object in the state
* #returns {Promise<void>}
*/
getUtxos = async () => {
let Utxos = []
try {
const rawUtxos = await this?.API?.getUtxos()
for (const rawUtxo of rawUtxos) {
const utxo = TransactionUnspentOutput.from_bytes(
Buffer.from(rawUtxo, 'hex')
)
const input = utxo.input()
const txid = Buffer.from(
input.transaction_id().to_bytes(),
'utf8'
).toString('hex')
const txindx = input.index()
const output = utxo.output()
const amount = output.amount().coin().to_str() // ADA amount in lovelace
const multiasset = output.amount().multiasset()
let multiAssetStr = ''
if (multiasset) {
const keys = multiasset.keys() // policy Ids of thee multiasset
const N = keys.len()
// console.log(`${N} Multiassets in the UTXO`)
for (let i = 0; i < N; i++) {
const policyId = keys.get(i)
const policyIdHex = Buffer.from(
policyId.to_bytes(),
'utf8'
).toString('hex')
// console.log(`policyId: ${policyIdHex}`)
const assets = multiasset.get(policyId)
const assetNames = assets.keys()
const K = assetNames.len()
// console.log(`${K} Assets in the Multiasset`)
for (let j = 0; j < K; j++) {
const assetName = assetNames.get(j)
const assetNameString = Buffer.from(
assetName.name(),
'utf8'
).toString()
const assetNameHex = Buffer.from(
assetName.name(),
'utf8'
).toString('hex')
const multiassetAmt = multiasset.get_asset(policyId, assetName)
multiAssetStr += `+ ${multiassetAmt.to_str()} + ${policyIdHex}.${assetNameHex} (${assetNameString})`
// console.log(assetNameString)
// console.log(`Asset Name: ${assetNameHex}`)
}
}
}
import { createRequire, runMain } from "module";
const require = createRequire(import.meta.url);
const Binance = require("node-binance-api");
const binance = new Binance().options({
APIKEY: "<API-KEY>",
APISECRET: "<API-SECRET>",
useServerTime: true,
});
async function Trade(interval, { symbol, EnterAt, Type }) {
try {
if ((interval == "1h" || interval == "4h") & (Type == "SHORT")) {
let response = await binance.futuresBalance();
let pair = response.filter((el) => el.asset == "USDT");
const leverage = 6;
let quantity = Math.round(
((parseFloat(pair[0].availableBalance) * leverage) / EnterAt).toFixed(2)
);
let takeProfit = EnterAt - (EnterAt * 0.6) / 100;
let stopMarket = EnterAt + (EnterAt * 1.1) / 100;
// Adjust leverage
let adjustLeverage = await binance.futuresLeverage(symbol, leverage);
// Futures Market Sell
let short = await binance.futuresMarketSell(symbol, quantity);
// Set TAKE PROFIT
let TP = await binance.futuresOrder("BUY", symbol, quantity, false, {
type: "TAKE_PROFIT_MARKET",
stopPrice: takeProfit.toFixed(2),
closePosition: true,
});
// Set STOP LOSS
let SL = await binance.futuresOrder("BUY", symbol, quantity, false, {
type: "STOP_MARKET",
workingType: "MARK_PRICE",
stopPrice: stopMarket.toFixed(2),
closePosition: true,
});
console.log("SHORT: ", short);
console.log("TP: ", TP);
console.log("SL: ", SL);
}
} catch (error) {
console.log(error.message);
}
}
function wrapUpTrade(interval, { symbol, EnterAt }) {
binance.useServerTime(() =>
Trade(interval, { symbol: symbol, EnterAt: EnterAt })
);
}
// wrapUpTrade("1h", { symbol: "XRPUSDT", shortAt: 0.774 });
export { wrapUpTrade };
This code is only for futures Market Sell (sell short). Why do I get an error of Insufficient Margin on placing an order of coin having a price of $4.5 and I have $2.9 in my wallet? On leveraging(6x) I will be having (6 * $2.9 = $17.4) that is greater than $4.5 still I get the error of Insufficient error. Function Trade takes some arguments:
Interval: "4h" or "1h"
symbol: "PAIR/(USDT)" // BTCUSDT, ETHUSDT
EnterAt: It is used to calculate Stoploss and TakeProfit. The type is int.
Type: "SHORT", means futuresSell
I'm changing the question cause I think I explain myself wrong.
I'm using currently 1 API endpoint to receive data which I need. I need to add a second end point to receive data from both end points in the same time, merge them together and store in Database.
First endpoint -
https://api.binance.com/api/v3/klines?symbol=${symbol}&interval=30m&limit=1
Second endpoint - https://api.binance.com/api/v3/ticker/24hr?symbol=${symbol}
Here is how I receive Data from first endpoint
const getBTCData = async symbol => {
let data = await fetch(`https://api.binance.com/api/v3/klines?symbol=${symbol}&interval=30m&limit=1`).then(res => res.json());
const btcusdtdata = data.map(d => {
return {
Open: parseFloat(d[1]),
High: parseFloat(d[2]),
Low: parseFloat(d[3]),
Close: parseFloat(d[4]),
Timespan: 30,
}
});
console.log(btcusdtdata);
saveToDatebase(symbol, btcusdtdata);
};
Im returning 4 parameters from this endpoint
And I need to take one parameter from second endpoint and combine it with parameters from first one.
I need this parameter from second endpoint - "quoteVolume": "15.30000000"
I founded that Promise.all can be a solution but I don't understand it properly on how I can return data from 2 api and merge them together in a single object to save in MongoDB.
FULL CODE
Small explanation - goal is to take data from both endpoints and store it in MongoDB as well as calculating the average for quoteVolume on last 200 days.
const { MongoClient } = require('mongodb');
const schedule = require('node-schedule');
const fetch = require("node-fetch");
require('dotenv').config()
"use strict"; // This is ES6 specific. Help's to run code faster(IMPORTANT FOR NOTIFICATION SYSTEM)
const nodemailer = require("nodemailer");
const symbols = ["ADABTC", "AEBTC","AIONBTC"];
//a descriptive name helps your future self and others understand code easier
const getBTCData = async symbol => {
let data = await fetch(`https://api.binance.com/api/v3/klines?symbol=${symbol}&interval=30m&limit=1`).then(res => res.json());
const btcusdtdata = data.map(d => {
return {
Open: parseFloat(d[1]),
High: parseFloat(d[2]),
Low: parseFloat(d[3]),
Close: parseFloat(d[4]),
Volume: parseFloat(d[5]),
Timespan: 30,
}
});
console.log(btcusdtdata);
saveToDatebase(symbol, btcusdtdata);
//recursive functions are complicated, we can get rid of it here
//by moving the responsibility to the caller
};
//helper function for an awaitable timeout
const sleep = ms => new Promise(res => setTimeout(res, ms));
const j = schedule.scheduleJob('* * * * * *', async() => {
//expand this function to be responsible for looping the data
for (let symbol of symbols) {
await getBTCData(symbol);
await sleep(8000);
}
});
const getDateTime = () => {
let today = new Date();
let date = today.getFullYear() + '-' + (today.getMonth() + 1) + '-' + today.getDate();
let time = today.getHours() + ":" + today.getMinutes() + ":" + today.getSeconds();
return date + ' ' + time;
};
const saveToDatebase = async(symbol, BTCdata) => {
try {
const url = 'mongodb://username:password#ip.adress.com:port/dbname?retryWrites=true&w=majority';
let dateTime = getDateTime();
let db = await MongoClient.connect(url, { useUnifiedTopology: true });
const dbo = db.db('Crypto');
const myobj = Object.assign({ Name: symbol, Date: dateTime }, BTCdata[0]);
await dbo.collection(symbol).insertOne(myobj);
const average = await dbo.collection(symbol).aggregate([{
$addFields: {
DateObj: {
$regexFindAll: { input: "$Date", regex: "\\d+" }
}
}
},
{
$set: {
DateObj: {
$dateFromParts: {
year: { $toInt: { $arrayElemAt: ["$DateObj.match", 0] } },
month: { $toInt: { $arrayElemAt: ["$DateObj.match", 1] } },
day: { $toInt: { $arrayElemAt: ["$DateObj.match", 2] } },
hour: { $toInt: { $arrayElemAt: ["$DateObj.match", 3] } },
minute: { $toInt: { $arrayElemAt: ["$DateObj.match", 4] } },
second: { $toInt: { $arrayElemAt: ["$DateObj.match", 5] } },
timezone: "Europe/London"
}
}
}
},
{
$match: {
$expr: {
$gte: ["$DateObj", { $subtract: ["$$NOW", 201 * 60 * 60 * 24 * 1000] }]
}
}
},
{
"$group": {
_id: null,
"Volume": {
"$avg": "$Volume"
}
}
}
]).toArray();
console.log('1 document inserted');
console.log(BTCdata[0].Volume);
console.log(average[0].Volume);
const RealTimeDataVolume = parseInt(BTCdata[0].Volume);
const HistoricalTimeDataVolume = parseInt(average[0].Volume); // 201 DAYS VOLUME HERE 3286033.4285714286
const DayTimesRealAverage = RealTimeDataVolume * 48; // 1 DAY REAL TIME DATA HERE 196579344
const Previous200dVolume = (HistoricalTimeDataVolume - DayTimesRealAverage) / 200;
const MultiplePrevious200dVolume = Previous200dVolume * 5;
if (MultiplePrevious200dVolume < DayTimesRealAverage) {
async function main() {
let transporter = nodemailer.createTransport({
host: "smtp.gmail.com",
port: 465,
secure: true, // true for 465, false for other ports
auth: {
user: process.env.DB_USER, // OUR ALARM EMAIL
pass: process.env.DB_PASS, // OUR ALARM PASSWORD
},
});
let info = await transporter.sendMail({
from: process.env.DB_USER, // sender address
to: process.env.DB_RECEIVER, // list of receivers
subject: symbol + 'Is UP', // Subject line
text: symbol + " IS UP", // plain text body
});
console.log("Message sent: %s", info.messageId, symbol);
}
main().catch(console.error);
} else {
console.log('false');
}
console.log(DayTimesRealAverage);
console.log(MultiplePrevious200dVolume);
} catch (e) {
console.error(e)
}
};
You can do it in the same way.
In response you are getting a son object.
You need to do res["quoteVolume"] to get the data.
I have spent a while looking around at similar problems online but I seem to be really struggling to wrap my head around this. I've reviewed a few sources online for help but was unable to apply the logic I seen to my own codebase, I'm hoping somebody out there might be able to help.
I'm running an express server with a pg Pool to handle DB logic.
I am able to successfully log out the information from within the pg.Pool logic however despite banging my head against multiple attempts I have been unable to successfully pass the data onto the clientside.
dbQueries.js
const { Pool } = require('pg');
const pool = new Pool({
user: process.env.DB_USER,
host: process.env.DB_HOST,
database:process.env.DB_NAME,
password: process.env.DB_PASSWORD,
port: process.env.DB_PORT
});
// Main function called in server.js
// step 1. fetches distinct name values from pg table
// step 2. fetches values to get full list of reviews for those distinct names
// step 3. does some data modification to make the data formatted for frontend usecase
const getFormattedReviews = async function(){
console.log('Fetching all unique therapist review names.')
const getDistinct = {
name: 'distinct-reviews',
text: 'SELECT DISTINCT therapist_name FROM reviews'
};
// step 1
const res = await pool.query(getDistinct, (err, res) => {
let data = []
if (err) {
console.log(err.stack);
} else {
// console.log(res.rows);
data = res.rows.map(
// step 2
therapist => getSpecificTherapistReviews(therapist.therapist_name)
)
}
console.log(`\n DEBUG3 - getFormattedReviews data: ${JSON.stringify(data)} \n`)
return data;
});
return res;
}
const getSpecificTherapistReviews = async function(therapist_name){
console.log(`Fetching reviews for: ${therapist_name}.`)
const getSpecificTherapistReviews = {
name: `${therapist_name}-reviews`,
text: `SELECT * FROM reviews WHERE therapist_name LIKE '%${therapist_name}%'`
};
const res = await pool.query(getSpecificTherapistReviews, (err, res) => {
let data = []
if (err) {
console.log(err.stack);
} else {
// console.log(res.rows);
// step 3
data = filteringDataForFrontend(res.rows);
}
console.log(`\n DEBUG2 - GetSpecificTherapistReviews ${JSON.stringify(data)} \n`)
return data;
});
return res;
}
const filteringDataForFrontend = function(data){
console.log(`Filtering Data for Frontend.`)
// Based on length of the reviews array, each review = 1 object
const total_reviews = data.length;
// Underlying logic assumes consistent data across all entries for these values
const therapist_name = data[0].therapist_name;
const type = data[0].type;
const image = data[0].image;
const location = data[0].location;
// Summing the rating values across multiple review entries
const ratings = data.reduce((acc, obj) => ({
rating_friendliness: acc.rating_friendliness + obj.rating_friendliness,
rating_techniques: acc.rating_techniques + obj.rating_techniques,
rating_progression: acc.rating_progression + obj.rating_progression,
rating_cost: acc.rating_progression + obj.rating_progression,
rating_listening: acc.rating_listening + obj.rating_listening,
rating_overall: acc.rating_overall + obj.rating_overall
})
)
// Placeholder as string, most likely restructure to an array of objects
const comments = data.reduce ((acc, obj) => ({
feedback_comments: acc.feedback_comments + obj.feedback_comments
})
)
// Filtered data for returning
const filteredData = {
therapist_name,
type,
image,
location,
rating_friendliness: ratings.rating_friendliness / total_reviews,
rating_techniques: ratings.rating_techniques / total_reviews,
rating_progression: ratings.rating_progression / total_reviews,
rating_cost: ratings.rating_cost / total_reviews,
rating_listening: ratings.rating_listening / total_reviews,
rating_overall: ratings.rating_overall / total_reviews,
feedback_comments: comments.feedback_comments
}
console.log(`\n DEBUG 1 - filteredData -> ${JSON.stringify(filteredData)} \n`)
return filteredData;
}
module.exports = {
getFormattedReviews,
};
An ideal setup I would like to have on the server.js side of things running express would be:
server.js
const express = require('express');
const DB = require('./dbQueries.js');
const app = express();
const port = process.env.SERVER_PORT || 8000;
app.get('/get-reviews', async (req, res) => {
const data = await DB.getFormattedReviews();
console.log(`data check ${data}`);
res.send({data});
});
Currently the endpoint is logging 'data check undefined'.
DEBUG checks 1 & 2 successfully appear to log information, however I spotted that DEBUG 3 only logs DEBUG3 - getFormattedReviews data: [{},{},{}] so perhaps I'm doing something wrong around there?
Any help/insight appreciated.
Thanks #Abraham & #Labkovsky for the suggestions -> Ill review them properly during the week.
I managed to get the basic fuctionality up and running with this code - it likely needs some refactoring but for reference:
dbQueries.js
const getFormattedReviews = async function(){
const getDistinct = {
name: 'distinct-reviews',
text: 'SELECT DISTINCT therapist_name FROM reviews'
};
const res = await new Promise(resolve => {
pool.query(getDistinct, (err, res) => {
let data = []
if (err) {
console.log(err.stack);
} else {
// console.log(res.rows);
data = res.rows.map(
async therapist => await getSpecificTherapistReviews(therapist.therapist_name)
)
// Promise.all(data).then(results => console.log(`\n DEBUG3 - getFormattedReviews data: ${JSON.stringify(results)} \n`))
}
Promise.all(data).then(results => resolve(results));
});
});
return res;
}
const getSpecificTherapistReviews = async function(therapist_name){
// console.log(`Fetching reviews for: ${therapist_name}.`)
const getSpecificTherapistReviews = {
name: `${therapist_name}-reviews`,
text: `SELECT * FROM reviews WHERE therapist_name LIKE '%${therapist_name}%'`
};
const res = await new Promise(resolve => {
pool.query(getSpecificTherapistReviews, (err, res) => {
let data = []
if (err) {
console.log(err.stack);
} else {
// console.log(res.rows);
data = filteringDataForFrontend(res.rows);
}
// console.log(`\n DEBUG2 - GetSpecificTherapistReviews ${JSON.stringify(data)} \n`)
resolve(data);
});
});
return res;
}
const filteringDataForFrontend = function(data){
// Based on length of the reviews array, each review = 1 object
const total_reviews = data.length;
// Underlying logic assumes consistent data across all entries for these values
const therapist_name = data[0].therapist_name;
const type = data[0].type;
const image = data[0].image;
const location = data[0].location;
// Summing the rating values across multiple review entries
const ratings = data.reduce((acc, obj) => ({
rating_friendliness: acc.rating_friendliness + obj.rating_friendliness,
rating_techniques: acc.rating_techniques + obj.rating_techniques,
rating_progression: acc.rating_progression + obj.rating_progression,
rating_cost: acc.rating_progression + obj.rating_progression,
rating_listening: acc.rating_listening + obj.rating_listening,
rating_overall: acc.rating_overall + obj.rating_overall
})
)
// Placeholder as string, most likely restructure to an array of objects
const comments = data.reduce ((acc, obj) => ({
feedback_comments: acc.feedback_comments + obj.feedback_comments
})
)
// Filtered data for returning
const filteredData = {
therapist_name,
type,
image,
location,
total_reviews,
rating_friendliness: ratings.rating_friendliness / total_reviews,
rating_techniques: ratings.rating_techniques / total_reviews,
rating_progression: ratings.rating_progression / total_reviews,
rating_cost: ratings.rating_cost / total_reviews,
rating_listening: ratings.rating_listening / total_reviews,
rating_overall: ratings.rating_overall / total_reviews,
feedback_comments: comments.feedback_comments
}
// console.log(`\n DEBUG 1 - filteredData -> ${JSON.stringify(filteredData)} \n`)
return filteredData;
}
module.exports = {
getFormattedReviews,
};
server.js
const express = require('express');
const DB = require('./dbQueries.js');
const app = express();
const port = process.env.SERVER_PORT || 8000;
app.get('/get-reviews', async (req, res) => {
const data = await DB.getFormattedReviews();
// data.then(data => console.log(`data2 check ${JSON.stringify(data)}`))
res.send(data);
});
Ultimately I think my limited understanding of Promises/Async/Await & Promise.all failed me a bit here, the Promise.all in the getFormattedReviews was the missing trick.
Some of the code was rewritten with some new Promise syntax which can likely be redacted however I noted when doing this with the getFormattedReviews method that it broke sending the data to the /get-reviews endpoint. Something I'll investigate later.
You are awaiting a callback. I don't think that works.
Try wrapping in a Promise. That might be the issue.
Full disclosure: I did not read your code very in depth...
const res = await new Promise(resolve => {
pool.query(getSpecificTherapistReviews, (err, res) => {
let data = []
if (err) {
console.log(err.stack);
} else {
// console.log(res.rows);
// step 3
data = filteringDataForFrontend(res.rows);
}
console.log(`\n DEBUG2 - GetSpecificTherapistReviews ${JSON.stringify(data)} \n`)
resolve(data);
});
})
So I need to take data only from the day before. Example: today is 2018/9/25, I need the data to be taken on 2018/9/24 only. But from my code below, it takes from 23 until 25. Which is more than one day, and it took also two days before from the date I need. I don't know which code that make a wrong result. Anyone can help me with this? I really appreciate it.
Api.js
const TO_DAYS = 4194304 * 1000 * 60 * 60 * 24; // this part that might be the cause
const ROOT_DATE = moment([2018, 3, 30]); // this part that might be the cause
const ROOT_DATE_ID = 440557948108800000; // this part that might be the cause
const DATE_ID = function(date) {
return ROOT_DATE_ID + date.diff(ROOT_DATE, "days") * TO_DAYS;
}; // this part that might be the cause
class discordApi {
constructor() {
this.lock = new AsyncLock();
}
get(momentDate, authorId, offset = 0) {
const url =
config.endpoint +
querystring.stringify({
min_id: DATE_ID(momentDate),
author_id: authorId
});
return fetch(url, {
headers: {
method: "GET",
Authorization: config.auth
}
}).then(res => {
// console.log(res.url);
return res.json();
});
}
async getAllData(momentDate) {
const allData = config.targets.map(author_id =>
this.get(momentDate, author_id)
);
return Promise.all(allData);
}
index.js
var yesterday = moment().subtract(1, "days"); // this part that might be the cause
async function sendEmail() {
const data = await discordApi.getAllData(yesterday);
const unfilteredMessages = data.reduce(
(prev, current) => [...prev, ...current.messages],
[]
);
const filteredMessages = unfilteredMessages.reduce((prev, current) => {
if (prev.length === 0) {
return [...prev, current];
}
const currentConversationIsDuplicated = isConversationDuplicated(
prev[prev.length - 1],
current
);
if (currentConversationIsDuplicated) {
return prev;
}
ret
urn [...prev, current];
}, []);
const convo = await discordApi.AllConvo(filteredMessages);
const mailOptions = {
from: "lala#gmail.com",
to: maillist,
subject: "Discord-Bot Daily Data",
html: convo
};
transporter.sendMail(mailOptions, function(err, info) {
if (err) console.log(err);
else console.log("Message Sent!");
});
}