I'm building a monitoring/statistic tool. The current infrastructure looks like following:
Collector-Backend: Receives queries (JSON-objects) from the frontend, fetches and stores them in a cache. Finally it notifies frontend over a message queue.
Frontend-Server: Handles subscriptions on the 'live'-views, pushes the data received from the backend to the user via WebSockets. (Also contains some user-management etc.)
Frontend itself: Some JavaScript-Spaghetti-Code which render the data using jQuery-Flot-Graphs. There are a couple of graphs per environment which are displayed on the same page. A graph is a set of measurments.
An, what I call, environment is a (set of) system(s) which contains different measurments. Therefore each environment contains following views:
a 'live'-view which displays the measurements of the last 2 hours (updated every minute).
a statistic-view on which different pre-defined graphs can be selected over an arbitrary time (same queries as the live-views, just another view)
some special reports like the yesterdays statistics.
An environment contains following variables:
{
name: 'FooEnvironment',
description: 'Foo Environment, <insert buzzwords here>',
baseTable: 'foobar', # Which table in the backend contains the corresponding data
target: 'barTarget', # The responsible target (backend-plugin)
}
The graphs are defined as a set of measurements (as a JSON-Object) and is defined like
this:
{
name: 'FooProtocol Traffic', # Graphs display name
size: 'full' # only matters for some CSS-transformations
interval: X # Which delta each tuple differs from the other
start-date: "DATE",
end-date: "DATE",
axes: {
"y1": { label: 'Foo/t', data: [['MEASURMENT1', 'AGGREGATION'], ['MEASURMENT2', 'AGGREGATION']] },
"y2": { label: 'Bar/sec', data: [['MEASURMENT_ON_Y2', 'AGGREGATION']] }
}
}
I want to be able to manage those environments on my frontend (Adding/remove environments and add/remove graphs to them).
How would you modeling this using MVC (backbone.js)?
Please report me if I have to provide some more information :-)
I'd probably try having an Environment model and a Graph model.
The Environment model would have a Graphs property, which is a collection of Graph models.
A user's subscriptions could be a collection of Environments that are being monitored.
There would be at least three views - one for holding the representation of a graph, one for holding the representation of an environment, and one for holding the collection of environments.
As you've mentioned adding/removing graphs and environments, there's a little more detail required.
For example, you'd need a list of available environments for the user to subscribe to, and under each environment subscribed to, a list of available graphs for that environment.
I know the answer is rambling and vague, but without further detail of what data is on hand, it's the best I can do.
Related
I will try to be as clear as possible but please ask if I am missing some detail.
So I have an application that I've inherited and it's running slow. It was built with Angular and Typescript on the front end, Loopback handling our CRUD operations and a Mongo DB. We isolated one of the reasons for the slowness as excess REST calls (potentially tens of thousands at times) so I am looking at reducing this. The first one I have found is a relatively small one so I hoped it'd help me to learn but I've already hit a challenge. Descriptions, table names and finer detail has been removed for simplicity.
On the first page, we present to the user a list of links to areas that they have access to. This is stored in an "Areas" table. We pull back all of the areas in a function similar to this:
this.ourApi.areas.find({
"filter": {
"include": ["generalInfo"]
/*
This table is related to most tables and
holds things like version numbers - it
is not relevant to this question
*/
}
}).$promise.then((returnedAreas) => {
/* Stuff */
});
After we have all the areas, we then have to get some configuration values for these area links (icons, mostly). Our configuration table holds configuration data for a number of items, not just Areas. So to get the relevant config information, we loop through the returnedAreas and, for each area, get the config where the relatedId matches the areaId. A bit like this:
return this.ourApi.configs.find({
"filter": {
"where": {
"relatedId": areaId
},
"include": ["generalInfo"]
}
}).$promise.then((areaConfigurations) => {
/* Stuff */
});
Ideally, I'd like to compress this down to just one call (and then mimic this pattern throughout the application). The only thing we really care about (and the only objects we actually use) are the configuration objects so we really don't need the "Areas" objects at all. The challenge I am unable to solve is how I adjust that second call above (that gets the config) so that it first gets the Areas and then uses the areaId in the query that gets the data that I care about. In SQl, this would be quite easy but I can't figure it out from the docs.
Essentially, I am looking for the Loopback equivalent of:
SELECT * FROM configs WHERE relatedId IN (SELECT areaId FROM areas)
Is this possible?
I'm, in the process of learning BookshelfJS/KnexJS (switching from SequelizeJS), and I'm running into an issue with importing data into multiple tables that were created via the migrations feature within KnexJS. There's 4 tables:
servers
operating_systems
applications
applications_servers
With the following constraints:
servers.operating_system_id references operating_systems.id
applications_servers.server_id references servers.id
applications_servers.application_id references applications.id
The tables get created just fine when I run knex migrate:latest --env development servers, it's when I import seeded data into the tables I get an error.
Originally, I organized the seed data for the 4 tables into 4 different files within the directory ./seeds/dev, which is just ${table_name}.js:
operating_systems.js
servers.js
applications.js
applications_servers.js
After a bit of debugging, I came to the realization that the error is being generated when the seed data within the file applications_servers.js, since when I take that one out, the other 3 run just fine. Then when I remove the 3 seed files and move applications_servers.js to the ./seeds/dev/ directory and execute knex seed:run, the applications_servers table gets populated just fine. However when I try to import the seeded data of all 4 files at once, I receive the following error:
# knex seed:run
Using environment: development
Error: ER_NO_REFERENCED_ROW_2: Cannot add or update a child row: a foreign key constraint fails (`bookshelf_knex_lessons`.`applications_servers`, CONSTRAINT `applications_servers_server_id_foreign` FOREIGN KEY (`server_id`) REFERENCES `servers` (`id`) ON DELETE CASCADE)
at Query.Sequence._packetToError (/Users/me/Documents/scripts/js/node/bookshelf_knex/node_modules/mysql/lib/protocol/sequences/Sequence.js:48:14)
at Query.ErrorPacket (/Users/me/Documents/scripts/js/node/bookshelf_knex/node_modules/mysql/lib/protocol/sequences/Query.js:83:18)
And not a single row gets inserted, into any table.
I thought that maybe it had something to do with the order that they were being imported. (Since they have to be imported in the order that the files were listed above). So thinking that maybe they were referenced in alpha-numeric order, I renamed them to:
1-operating_systems.js
2-servers.js
3-applications.js
4-applications_servers.js
However, nothing changed, no rows were inserted into any table, so just to be sure, I reversed the sequence of the number prefixes on the files, and again, no changes, not a single row was inserted, and same error returned.
Note: The migration script for creating the tables, as well as all of the seed data, was copy and pasted from a JS file I created which creates the same tables and imports the same data using BookshelfJS/KnexJS, but instead of using the migration features, it just does it manually when executed via node. You can view this file here
Any help would be appreciated!
Edit: When I combine all of the seed files within ./seeds/dev into one file, ./seeds/dev/servers.js, everything gets imported just fine. This makes me think that it may be caused by the knex migrations being executed asynchronously, thus the ID's inserted in the pivot table and the servers.operating_system_id may not be inserted yet into the associated tables... If this is the case, is there a way to setup dependencies in the seed files?
Knex.js's seed functionality does not provide any order of execution guarantees. Each seed should be written such that it can be executed in isolation - ie. your single file approach is correct.
If you want to break your individual seed files into submodules, then you might try the following:
// initial-data.js
var operatingSystems = require('./initial-data/operating-systems.js');
var servers = require('./initial-data/servers.js');
exports.seed = function(knex, Promise) {
return operatingSystems.seed(knex, Promise)
.then(function () {
return servers.seed(knex, Promise);
}).then(function() {
// next ordered migration...
});
}
I use the sql-fixtures module to handle FK relation dependencies in my seed file(s).
Imaginary implementation:
const dataSpec = {
applications_servers: [{
name: 'My ASP.Net thingie',
application_id: 'applications:0',
server_id: 'servers:0'
}],
servers: [{
name: 'My Windows server',
operating_system_id: 'operating_systems:0'
}],
operating_systems: [{
name: 'Windows Server 2k10'
}],
applications: [{
name: 'My fab web guestbook',
description: '...'
}]
}
but it might be superfluous if you declare your dependencies by means of the ORM.
Sometimes however, it is desirable to avoid coupling the DB seeds to your exact current Model implementation.
First of all excuse me since I don't know how it is called in computer since:
For each of my document types in my mongo app I want to define a structure, with every field defined with its constraints, validation patterns and, generally, roles that can view modify and delete this document.
For example: Book:
{
name: "Book",
viewRoles: ["Admin","User"],
createRoles: ["Admin"],
modifyRoles: ["Admin", "User"],
fields: [
{
id:"title",
name:"Book Title",
validation: "",
maxLength: 50,
minLength: 3,
required: true
},
{
id:"authorEmail",
name:"Email of the Author",
validation: "email",
maxLength: 50,
minLength: 3,
required: false
}
]
}
Then if I have this "schema" for all of my documents, I can have one view for creating modifying and showing this "entities".
I also want to have the ability to create new document types, modify their fields through admin panel of my application.
When I google "mongo dynamic schema", "mongo document meta design" I get useless information.
My question is how it is called -- when I want to have predefined schema of my documents and have the ability to modify it. Where I can get more information about how to design such systems?
Since you tagged this as having a Meteor connection, I'll point you to Simple Schema: https://github.com/aldeed/meteor-simple-schema/. I use it, along with the related collection2 package. I find it's a nice way to document and enforce schema design. When used with the autoform package, it also provides a way to create validated forms directly from your schema.
I think you are looking for how to model your data. The below link might be helpful:
http://docs.mongodb.org/manual/data-modeling/
I also want to have the ability to create new document types, modify
their fields through admin panel of my application.
For Administrative activities you may look into the options given in:
http://docs.mongodb.org/ecosystem/tools/administration-interfaces/
And once you are done, you might want to read this as a kick off:
https://blog.serverdensity.com/mongodb-schema-design-pitfalls/
In Mongo DB you don't create collections. You just start using them. So you can't define schemas before hand. The collection is created on the first insert you make to the collection. Just make sure to ensure Index on the collection before inserting documents into it:
db.collection.ensureIndex({keyField: 1})
So it all depends on maintaining the structure of the documents inserted to the collection rather than defining the collection.
Using Breeze with Entity Framework code first to return data from calls to a web service.
I have a data model that's several levels deep. In this instance I'm returning a "schedule" object, which has a number of child "DefaultItems", each one of which has a cost and a single "type" child with its own properties.
If you call the web service for one of these directly, you get something like this, which is as expected:
{
$id:"1",
$type:"Schedule_06B188AC55B213FE4B13EA5B77D9C039007E80E9DB6F6841C055777A028C5F95, EntityFrameworkDynamicProxies-Core",
DefaultItems:[
{
$id:"2",
$type:"DefaultItem, Core",
RowId:"d422af5d-d6ca-46a3-a142-1feb93348e1d",
Cost:1,
Type:{
$id:"3",
$type:"Type, Core",
RowId:"38ed6d1b-d0b7-43cb-b958-2b2424b97759",
Type:"Type1"
},
Schedule:{
$ref:"1"
}
},
//more DefaultItem objects
{},
{}
],
RowId:"627eb2f2-ec74-4646-b3d1-d6423f84a2cd",
Start:"2010-01-18T00:00:00.000",
End:"2019-01-18T00:00:00.000"
}
This then comes down to the browser, where knockout is used to bind it to data objects. The trouble is that at this point, the data only seems to be one level deep.
So I can get at Schedule.Start and Schedule.End without issue. I can also iterate through the DefaultItem objects inside my Schedule and get their Costs out. But the Type objects inside DefaultItem just aren't there.
It's not about using an incorrect name to bind them: if you pause in the browser debugger and drill down into the JSON that the browser has, there's no Type objects at all, not even empty objects where they should be.
How come they come out of the web service, but don't seem to be in the data that Breeze passes back to the browser?
Apparently in Breeze, relationships have to be defined both ways in order to propagate. So I had to ensure that my the primary key in my Type class was marked as a foreign key to the DefaultItem class.
I believe this is currently registered as a bug. It's certainly a bit annoying.
First of all, thanks to the guys of DocumentCloud for releasing those two super-useful tools.
Here goes my question(s):
I'm trying to use visulasearch.js in a backbone.js app.
In my app I have a basic index.html and a myapp.js javascript file wich contains the main application done with backbone.js
I use CouchDB as data storage, and I successfully can retrieve in a restful way all the data to be put in the collection.
I must retrieve the search query given by visualsearch.js and use it to filter a collection.
I surely need a view for the searchbox, to trigger an event when enter is hit, but..
Should I initialze the searchbox externally to myapp.js, within an additional js file or my index.html page (as suggested in the visualsearch mini.tutorial)?
Or I should initialize it within the searchbox view (myapp.js)? This latter solution seems to be too tricky (it was what I was trying to do, but even when I succeed, it's too complicated and I lost the simplicity of bacbone mvc).
Let's say I succeed in retrieving the search string as a JSON object like {name:'Fat DAvid', address:'24, slim st', phone:'0098876534287'}. Once done that, which function can I use to retrieve, in the collection, only the models whose fields match with the given string. I understand that I should do a map or a filter, but those function seems to serve natively for slightly different tasks.
a. is it really the best way to filter results? It charges the client (which must filter the results), while making a new query (a view or a filter) to CouchDB would be quite simple and, considered the small amount of data and the low access rate to the site, not so expensive. However, making all the filtering action client-side, it's much simpler than making new view(or list or filters) in CouchDB and linking it the the backbone.js view
You can initialize your VisualSearch.js search box right in your myapp.js. Just make sure you keep a reference to it so you can then extract out the facets and values later.
For example:
var visualSearch = VS.init({...})
// Returns the unstructured search query
visualSearch.searchBox.value()
// "country: "South Africa" account: 5-samuel title: "Pentagon Papers""
// Returns an array of Facet model instances
visualSearch.searchQuery.facets()
// [FacetModel<country:"South Africa">,
// FacetModel<account:5-samuel>,
// FacetModel<title:"Pentagon Papers">]
If you have these models in a Backbone collection, you can easily perform a filter:
var facets = visualSearch.searchQuery.models;
_.each(facets, function(facet) {
switch (facet.get('category')) {
case 'country':
CountriesCollection.select(function(country) {
return country.get('name') == facet.get('value');
});
break;
// etc...
}
});