I've a module which is using parsing ( a parsing functionality), other modules should query this parser values.
my question is
how should I build it (design aspects ) ?
which method should init the parser (the first method that call it
to get specific value)
This is sample code which return two object from the parser but I dont think that this is the right way to do that since maybe I'll need to provide additional properties
the is the module parse
parse = function (data) {
var ymlObj = ymlParser.parse(data);
return {
web: ymlObj.process_types.web,
con: ymlObj.con
}
};
If I understood you right you can just make simple module with getters and setter.
(parse.js)
var ymlObj = {};
function Parse() {}
Parse.prototype.setData = function (data) {
ymlObj = data;
}
Parse.prototype.getWeb = function () {
return ymlObj.process_types.web;
}
Parse.prototype.getCon = function () {
return ymlObj.con;
}
module.exports = new Parse();
(parseUser.js)
var parse = require('./parse.js');
function ParseUser() { }
ParseUser.prototype.useParse = function () {
console.log(parse.getCon());
}
module.exports = new ParseUser();
(app.js)
var parse = require('./parse.js');
var parseUser = require('parseUser.js');
parse.setData({ ... });
parseUser.useParse();
You still have to do basics like handle exceptions but hope this helps you understand the basic structure.
What comes to init it really depends when you want to initialize (fetch?) your data and where does that data come from. You can set timestamp to indicate how old your data is and make decision if you still rely on it or fetch newer data. Or you can register callbacks from your user modules to deal with new data every time its fetched.
So its up to you how you design your module. ;)
Related
We are constructing an API with Azure Functions, and the spec calls for DeepObject references in the GET request's querystring. So, the structure looks like https://example.com/api/persons?name[first]=Todd. The expectation is that some of the query keys may be DeepObject references, others will be flat.
This is a pattern that apparently Express can handle, but Azure Functions uses an ASP.NET router. The expectation is that the reference above should deserialize into req.params: { name: { first: "Todd" } }. However, instead the output looks like req.params: { "name[first]": "Todd" }.
I would really love to not have to regex/search each key, so does anyone know if:
There's a config/flag in ASP.NET to support this structure?
There's a good pattern in Javascript to deserialize this in some functional -- or at least non-idiosyncratic -- way?
Note: Anyone who suggest some use of the eval() method will not be selected. But for playing you will take home a comment with a Nineties reference, because that was the last decade the use of the that method was considered acceptable. :stuck_out_tongue_winking_eye:
For this problem, I don't think we can change some configuration to support this structure. What we can do is to implement in code by ourselves.
Here is my function code:
module.exports = async function (context, req) {
console.log("======query url is:" + req.url);
const result = queryStringToJSON(req.url);
console.log(result);
context.res = {
body: "success"
};
}
function queryStringToJSON(queryUrl) {
if(queryUrl.indexOf('?') > -1){
var queryString = queryUrl.split('?')[1];
}
var pairs = queryString.split('&');
var result = {};
pairs.forEach(function(pair) {
if (pair.indexOf('[') > -1) {
var nameObj = {};
var firstObj = {};
var nameStr = pair.substring(0, pair.indexOf('['));
var firstStr = pair.substring(pair.indexOf('[')+1, pair.indexOf(']'));
firstObj[firstStr] = pair.split('=')[1];
nameObj[nameStr] = firstObj;
Object.assign(result, nameObj);
}else {
pair = pair.split('=');
result[pair[0]] = decodeURIComponent(pair[1] || '');
}
});
return result;
}
Start the function project, I request it with http://localhost:7071/api/HttpTrigger1?name[first]=Todd&email=test#mail.com. The result shows:
After much searching, I wasn't able to find any way to natively implement this in the ASP.NET router. Though there is a great deal of suggestions on how to deserialize this structure directly in your ASP.NET controller functions, I am working in Javascript.
What was helpful was the qs library, available in NPM, which supports a number of nuances related to this query string structure.
const { query } = req;
// => { "name[first]": "Todd" };
const deserializedQuery = qs.parse(query);
// => { name: { first: "Todd" }
Equally helpful to me is that I need a way to restructure my outbound query string in this same format. Qs works with the paramsSerializer attribute in Axios.
const params = { name: { first: "Todd" };
const paramsSerializer = (params) => { return Qs.stringify(params); };
const reqOptions = { params, paramsSerializer };
axios.get("https://example.com/api/persons", reqOptions);
// => GET https://example.com/api/persons?name[first]=Todd
Thanks to #hury-shen for a completely workable solution. It just wasn't turnkey solution I was looking for.
Background: Been trying for the last 2 day to resolve this myself by looking at various examples from both this website and others and I'm still not getting it. Whenever I try adding callbacks or async/await I'm getting no where. I know this is where my problem is but I can't resolve it myself.
I'm not from a programming background :( Im sure its a quick fix for the average programmer, I am well below that level.
When I console.log(final) within the 'ready' block it works as it should, when I escape that block the output is 'undefined' if console.log(final) -or- Get req/server info, if I use console.log(ready)
const request = require('request');
const ready =
// I know 'request' is deprecated, but given my struggle with async/await (+ callbacks) in general, when I tried switching to axios I found it more confusing.
request({url: 'https://www.website.com', json: true}, function(err, res, returnedData) {
if (err) {
throw err;
}
var filter = returnedData.result.map(entry => entry.instrument_name);
var str = filter.toString();
var addToStr = str.split(",").map(function(a) { return `"trades.` + a + `.raw", `; }).join("");
var neater = addToStr.substr(0, addToStr.length-2);
var final = "[" + neater + "]";
// * * * Below works here but not outside this block* * *
// console.log(final);
});
// console.log(final);
// returns 'final is not defined'
console.log(ready);
// returns server info of GET req endpoint. This is as it is returning before actually returning the data. Not done as async.
module.exports = ready;
Below is an short example of the JSON that is returned by website.com. The actual call has 200+ 'result' objects.
What Im ultimately trying to achieve is
1) return all values of "instrument_name"
2) perform some manipulations (adding 'trades.' to the beginning of each value and '.raw' to the end of each value.
3) place these manipulations into an array.
["trades.BTC-26JUN20-8000-C.raw","trades.BTC-25SEP20-8000-C.raw"]
4) export/send this array to another file.
5) The array will be used as part of another request used in a websocket connection. The array cannot be hardcoded into this new request as the values of the array change daily.
{
"jsonrpc": "2.0",
"result": [
{
"kind": "option",
"is_active": true,
"instrument_name": "26JUN20-8000-C",
"expiration_timestamp": 1593158400000,
"creation_timestamp": 1575305837000,
"contract_size": 1,
},
{
"kind": "option",
"is_active": true,
"instrument_name": "25SEP20-8000-C",
"expiration_timestamp": 1601020800000,
"creation_timestamp": 1569484801000,
"contract_size": 1,
}
],
"usIn": 1591185090022084,
"usOut": 1591185090025382,
"usDiff": 3298,
"testnet": true
}
Looking your code we find two problems related to final and ready variables. The first one is that you're trying to console.log(final) out of its scope.
The second problem is that request doesn't immediately return the result of your API request. The reason is pretty simple, you're doing an asynchronous operation, and the result will only be returned by your callback. Your ready variable is just the reference to your request object.
I'm not sure about what is the context of your code and why you want to module.exports ready variable, but I suppose you want to export the result. If that's the case, I suggest you to return an async function which returns the response data instead of your request variable. This way you can control how to handle your response outside the module.
You can use the integrated fetch api instead of the deprecated request. I changed your code so that your component exports an asynchronous function called fetchData, which you can import somewhere and execute. It will return the result, updated with your logic:
module.exports = {
fetchData: async function fetchData() {
try {
const returnedData = await fetch({
url: "https://www.website.com/",
json: true
});
var ready = returnedData.result.map(entry => entry.instrument_name);
var str = filter.toString();
var addToStr = str
.split(",")
.map(function(a) {
return `"trades.` + a + `.raw", `;
})
.join("");
var neater = addToStr.substr(0, addToStr.length - 2);
return "[" + neater + "]";
} catch (error) {
console.error(error);
}
}
}
I hope this helps, otherwise please share more of your code. Much depends on where you want to display the fetched data. Also, how you take care of the loading and error states.
EDIT:
I can't get responses from this website, because you need an account as well as credentials for the api. Judging your code and your questions:
1) return all values of "instrument_name"
Your map function works:
var filter = returnedData.result.map(entry => entry.instrument_name);
2)perform some manipulations (adding 'trades.' to the beginning of each value and '.raw' to the end of each value.
3) place these manipulations into an array. ["trades.BTC-26JUN20-8000-C.raw","trades.BTC-25SEP20-8000-C.raw"]
This can be done using this function
const manipulatedData = filter.map(val => `trades.${val}.raw`);
You can now use manipulatedData in your next request. Being able to export this variable, depends on the component you use it in. To be honest, it sounds easier to me not to split this logic into two separate components - regarding the websocket -.
I've decided on splitting my environments keeping them in .js files in an environment folder and keep all the sensitive information in .env file (use a third-party module 'DOTENV')
That's what I've come up with but I understand that it's not the best practice and there are a lot of things which should have been implemented in a completely different way but I just lack experience and practice.
At first, I tried to use as more " for loop " as it's possible because as far as I know, it's the fastest way to loop through an object, but in some cases, it was much easier to with "map or filter".
It doesn't look nice to assign data by returning a Promise. Maybe there is a way to get data without a Promise?
I would appreciate any suggestions on how the code can be improved and good practices, your experience.
And I am not sure if I used logging right and error handling. That's a completely new thing for me at the moment, but I used "try catch" to catch them and simply logged them on the console and put into a file.
code:
import { readdirSync } from 'fs';
import path from "path";
import { logger } from '../src/utils/logging';
import { merge } from "lodash";
// FIXME: Function returns a Promise with the data.
// It's not comfortable and seem a bad practice - too much code for a simple task,
// and deal with a promise what may outcome in decreasing perfomance
// ( The simplest code, the fastest code )
export let env = getEnvironment().then(
res => { return res },
err => logger.error(err)
);
// TODO: Rewrite this function into a class Environment to keep it organized and implement ES6 standart
async function getEnvironment() {
const mode = process.env.NODE_ENV || 'development';
const rootPath = process.cwd();
const folder = 'environment';
const loadEnvironments = () => {
// Getting the list of available environments in the "environment" folder,
// at the same time excluding index.js file
const list = readdirSync(path.join(rootPath, folder)).filter(file => !/(?=^(index.js))/i.test(file));
const parameters = {};
// Loading the files found in the folder,
// merging them with the help of a "lodash" library
// just to get one common Object with all possible parameters from all found environments
const loaded = list.map(fileName => {
let name = fileName.split('.')[0];
let loadedFile = require(path.join(rootPath, folder, fileName));
const file = loadedFile[name];
merge(parameters, { ...file });
return loadedFile;
});
// Picking the currect mode out of already loaded ones
const current = { ...loaded.filter(file => file[mode]).map(file => file[mode])[0] };
// Returning an object with all parameters
return {
parameters,
current
}
};
const environments = loadEnvironments();
const environment = {} = looping(environments.parameters, environments.current);
function looping(obj, values) {
const collection = {};
for (const key in obj) {
if (obj.hasOwnProperty(key)) {
if (typeof obj[key] !== 'object') {
try {
if (values.hasOwnProperty(key)) {
// By a recursive function run through all parameters,
// transforming the keys to uppercased,
// assigning value to 'obj' (file containing all the parameters)
// from the current mode
collection[key.toUpperCase()] = values[key];
} else {
// if there is no such a key in the current mode,
// 'null' is assigned
collection[key.toUpperCase()] = null;
}
} catch (e) {
logger.error(` Missing parameter "${key.toUpperCase()}" in ${mode} mode!!!`);
}
} else {
// Recursing through the object and the nested objects
collection[key.toUpperCase()] = looping(obj[key], values[key]);
}
}
}
return collection;
}
// When parameters are ready,
// the current mode is assigned
environment["MODE"] = mode;
return environment;
}
I am trying to load a Tensorflowjs model using the model.json which is an in memory browser side object.
https://js.tensorflow.org/api/latest/#loadLayersModel
One approach may be to return the json from a dummy fetch method.
fetchFunc (Function) A function used to override the window.fetch
function.
Alternatively, it is possible to create a custom IOHandler, but there is very little documentation on this.
An tf.io.IOHandler object that loads model artifacts with its load
method.
Does anyone know how to achieve this using the tensorflow load methods?
var modelJson = "{...ModelAndWeightsConfig}";
//Do something here to load it.
var newModel = tf.loadLayersModel("/model_0/model.json", {
onProgress: onProgressCallback}).then(model =>{});
Regards,
Yes, you can write your own IOHandler to load the model. Check out the definition of an IOHandler here. You have to implement the load function that returns a Promise<ModelArtifacts>.
That means, to load a model saved by the file IOHandler, you can check out the source code and reimplement the load function yourself.
Code Sample
Here is example to get you started. The load() part is mostly copied from the loadJSONModel function from the file IOHandler. Basically, the JSON string is passed as an argument and then used when the load function is called by Tensorflow.js.
export class JSONHandler implements tfc.io.IOHandler {
constructor(jsonString) {
this.jsonString = jsonString;
}
async load() {
const modelJSON = JSON.parse(jsonString);
const modelArtifacts: tfc.io.ModelArtifacts = {
modelTopology: modelJSON.modelTopology,
format: modelJSON.format,
generatedBy: modelJSON.generatedBy,
convertedBy: modelJSON.convertedBy
};
if (modelJSON.weightsManifest != null) {
// load weights (if they exist)
}
if (modelJSON.trainingConfig != null) {
modelArtifacts.trainingConfig = modelJSON.trainingConfig;
}
if (modelJSON.userDefinedMetadata != null) {
modelArtifacts.userDefinedMetadata = modelJSON.userDefinedMetadata;
}
return modelArtifacts;
}
}
To use the model, you can then create an instance of it and pass it to the load function:
const modelJson = '{ ... }';
const handler = new JSONHandler(modelJson);
const model = await tf.loadLayersModel(handler);
Fetch the models
var fetchPromise = function(url,p1,p2,) {
return new Promise(function(resolve, reject) {
fetch(url)
.then(response => {
resolve(response);
}).catch(err =>{
reject();
});
});
};
Heroku recently posted a list of some good tips for postgres. I was most intreged by the Track the Source of Your Queries section. I was curious if this was something that's possible to use with Sequelize. I know that sequelize has hooks, but wasn't sure if hooks could be used to make actual query string adjustments.
I'm curious if it's possible to use a hook or another Sequelize method to append a comment to Sequelize query (without using .raw) to keep track of where the query was called from.
(Appending and prepending to queries would also be helpful for implementing row-level security, specifically set role / reset role)
Edit: Would it be possible to use sequelize.fn() for this?
If you want to just insert a "tag" into the SQL query you could use Sequelize.literal() to pass a literal string to the query generator. Adding this to options.attributes.include will add it, however it will also need an alias so you would have to pass some kind of value as well.
Model.findById(id, {
attributes: {
include: [
[Sequelize.literal('/* your comment */ 1'), 'an_alias'],
],
},
});
This would produce SQL along the lines of
SELECT `model`.`id`, /* your comment */ 1 as `an_alias`
FROM `model` as `model`
WHERE `model`.`id` = ???
I played around with automating this a bit and it probably goes beyond the scope of this answer, but you could modify the Sequelize.Model.prototype before you create a connection using new Sequelize() to tweak the handling of the methods. You would need to do this for all the methods you want to "tag".
// alias findById() so we can call it once we fiddle with the input
Sequelize.Model.prototype.findById_untagged = Sequelize.Model.prototype.findById;
// override the findbyId() method so we can intercept the options.
Sequelize.Model.prototype.findById = function findById(id, options) {
// get the caller somehow (I was having trouble accessing the call stack properly)
const caller = ???;
// you need to make sure it's defined and you aren't overriding settings, etc
options.attributes.include.push([Sequelize.literal('/* your comment */ 1'), 'an_alias']);
// pass it off to the aliased method to continue as normal
return this.findById_untagged(id, options);
}
// create the connection
const connection = new Sequelize(...);
Note: it may not be possible to do this automagically as Sequelize has use strict so the arguments.caller and arguments.callee properties are not accessible.
2nd Note: if you don't care about modifying the Sequelize.Model prototypes you can also abstract your calls to the Sequelize methods and tweak the options there.
function Wrapper(model) {
return {
findById(id, options) {
// do your stuff
return model.findById(id, options);
},
};
}
Wrapper(Model).findById(id, options);
3rd Note: You can also submit a pull request to add this functionality to Sequelize under a new option value, like options.comment, which is added at the end of the query.
This overrides the sequelize.query() method that's internally used by Sequelize for all queries to add a comment showing the location of the query in the code. It also adds the stack trace to errors thrown.
const excludeLineTexts = ['node_modules', 'internal/process', ' anonymous ', 'runMicrotasks', 'Promise.'];
// overwrite the query() method that Sequelize uses internally for all queries so the error shows where in the code the query is from
sequelize.query = function () {
let stack;
const getStack = () => {
if (!stack) {
const o = {};
Error.captureStackTrace(o, sequelize.query);
stack = o.stack;
}
return stack;
};
const lines = getStack().split(/\n/g).slice(1);
const line = lines.find((l) => !excludeLineTexts.some((t) => l.includes(t)));
if (line) {
const methodAndPath = line.replace(/(\s+at (async )?|[^a-z0-9.:/\\\-_ ]|:\d+\)?$)/gi, '');
if (methodAndPath) {
const comment = `/* ${methodAndPath} */`;
if (arguments[0]?.query) {
arguments[0].query = `${comment} ${arguments[0].query}`;
} else {
arguments[0] = `${comment} ${arguments[0]}`;
}
}
}
return Sequelize.prototype.query.apply(this, arguments).catch((err) => {
err.fullStack = getStack();
throw err;
});
};