I'm ending up having to do hacks to convert 'true' to just true and it's creating code smell.
Is there a library like https://github.com/thephpleague/fractal that allows me to transform my response into the types I need?
In cases like this it's almost always better to fix the API to return data in a usable format rather than trying to post-process the result on the client.
In your case there are several routes you could take:
Store the list as a JSON string directly in the database.
This means you don't have to do any processing on the server and can just return it 'as is'. However you lose the ability to do queries on the data directly and need to resort to things like LIKE and string operations.
Store the data relationally, and process it on the server to turn it into JSON
Here you retain the ability to do queries on your data, but you may need to do several queries to get all the data you need and then connect it on the server. (eg. you would do one SELECT on the user table to get a user, and then you would need to do another SELECT on the friends table where the userid matches your first user. You would then need to merge these results to create your JSON.) This is usually the best way to do it.
You can also turn the result into JSON directly inside the database engine using a user defined function. For example using https://github.com/mysqludf/lib_mysqludf_json#readme
This is somewhat similar to 2, but it ties your stored procs to the JSON format.
Related
I have a table with a thousand records in it and I want to do a google like search full-text/fuzzy search.
I read about MySQL v8's Full-Text search and let's say we don't have that functionality yet.
There is this JavaScript library called Fuse.js that do fuzzy-search which is what I need.
I can combine it by creating a API that returns the table data in JSON format and then pass it to Fuse.js to do a fuzzy-search.
Now, I think it's not recommended to load all data from table every time someone wants to search.
I read about Redis, and the first thing that came in my mind is to save all table data in Redis using JSON.stringify and just call it every time instead of querying the database. Then whenever a data is added in the table, I will also update the contents of the data in Redis.
Is there a better way to do this?
That is a very common caching pattern.
If you need a more efficient way to store and retrieve your JSON to/from Redis you might want to consider one of the available Redis Modules.
e.g.
RedisJSON allows you to efficiently store, retrieve, project (jsonpath) and update in place.
RediSearch allows you to have full text search over Redis Hash and efficiently retrieve data according to the user's query.
Last
RedisJSON2 (aka RedisDoc) combines both modules above, meaning efficient JSON store and retrieve with Full Text support
I'm working with this returned API from a third party:
(note: returns LARGE dataset)
https://www.saferproducts.gov/RestWebServices/Recall?format=json
I know I can get this giant object like so:
$.getJSON('https://www.saferproducts.gov/RestWebServices/Recall?format=json', function(json, textStatus) {
console.log(json);
});
This returns ~7000 objects. There is no way in the API call to dictate how many objects you want returned. It's all or nothing.
Here's the question...can I use getJSON (or similar) to only get the first 5 objects and stop without having to load the entire JSON file first?
I did something similar a while a go. I used PHP to fetch a webpage of an api. Then I would cache it. Via PHP logic, I stored a variable inside a text file which contained all the information from the webpage. I had another file that stored the timestamp. Then, when the page was called, php would check the timestamp to see how old it was. If it was too old, it'd recache the page and return relevant information. If it was still valid, it would just return the cached information. If you only want the last 5, the PHP logic wouldn't be too hard to write that in. Then, jQuery would query the PHP page.
They don't have anything called out in their documentation for limiting the returns. I think their intent is for you to narrow down your search so you're not fetching every single item. You could always email them and ask as what Mike McCaughan said, if they don't have a 'limit' baked in, then no, it's not possible.
It also looks like they offer weekly downloads that you can just create your own API and add a limit property:
https://www.fda.gov/%20Safety/Recalls/EnforcementReports/default.htm
Reference:
https://github.com/presidential-innovation-fellows/recalls-api/wiki/data-sources
https://www.cpsc.gov/Recalls/CPSC-Recalls-Application-Program-Interface-API-Information/
https://www.cpsc.gov/Global/info/Recall/CPSC-Recalls-Retrieval-Web-Services-Programmers-Guide_3-25-2015.pdf
If there really is no option for limiting that call, then I'd suggest caching, showing some kind of processing wheel while the call takes place or narrowing your query. They have options to for filtering that may work for you such as the following:
RecallNumber
RecallDateStart
RecallDateEnd
LastPublishDateStart
LastPublishDateEnd
RecallURL
RecallTitle
ConsumerContact
RecallDescription
ProductName
ProductDescription
ProductModel
ProductType
RecallInconjunctionCountry
ImageURL
Injury
ManufacturerCountry
UPC – see caveat below
Hazard
Manufacturer
Remedy
Retailer
This might seem like an odd questions; I know its possible to slice an array, but I was thinking, if I'm calling an array externally via a $http GET (using Angular) like so:
$http.get('/url')
is it possible for me to declare how much of the array I want to retrieve first before making the request, instead of retrieving the whole thing (and therefore saving on performance). I know its possible to do something similar using PHP but wasn't sure to the extent I had in Javascript? Or can I only slice it once the array is declared fully?
The only way to retrieve part of the array is for the server that you are requesting the data from to support that capability. You cannot do that entirely from the client unless the server has that capability.
The usual way that this would be done would be to add query parameters to the URL that specify how much of the data you want and for the server to look at those query parameters and to send only those pieces of the data.
For example:
$http.get('/url?start=0&end=20');
Arrays can only be sliced once they exist and have some data in them so I'm not sure what that part of the question was trying to ask about. Either the client or the server could slice the result array once they had built the initial array.
I have a MongoDB JavaScript function saved in db.system.js, and I want to use it to both query and produce output data.
I'm able to query the results of the function using the $where clause like so:
db.records.find(
{$where: "formatEmail(this.email.toString(), 'format type') == 'xxx'"},
{email:1})
but I'm unable to use this to return a formatted value for the projected "email" field, like so:
db.records.find({}, {"formatEmail(this.email.toString(), 'format type')": 1})
Is there any way to do this while preserving the ability to simply use a pre-built function?
UPDATE:
Thank you all for your prompt participation.
Let me explain why I need to do this in MongoDB and it's not a matter of client logic at the wrong layer.. What I am really trying to do is use the function for a shard bucketing value. Email was just one example, but in reality, what I have is a hash function that returns a mod value.
I'm aware of Mongo having the ability to shard based on a hashed value, but from what I gather, it produces a highly random value that can burden the re-balancing of shards with unnecessary load. So I want to control it like so func(_id, mod), which would return a value from 0 to say 1000 (depending on the mod value).
Also, I guess I would also like to use the output of the function in some sort of grouping scenario, and I guess Map Reduce does come to mind.. I was just hoping to avoid writing overly complex M/R for something so simple.. also, I don't really know how to do Map Reduce .. lol.
So, I gather that from your answers, there is no way to return any formatted value back from mongo (without map/reduce), is that right?
I think you are mixing your "layers" of functionality here -- the database stores and retrieves data, thats all. What you need to do is:
* get that data and store the cursor in a variable
* loop through your cursor, and for every record you go through
* format and output your record as you see fit.
This is somewhat similar to what you have described in your question, but its not part of MongoDB and you have to provide the "formatEmail" function in your "application layer"
Hope it helps
As #alernerdev has already mentioned, this is generally not done at a database layer. However, sometimes storing a pre-formatted version in your database is the way to go. Here's some instances where you may wish to store extra data:
If you need to lookup data in a particular format. For example, I have a "username" and a "usernameLowercase" fields for my primary user collection. The lowercase'd one is indexed, and is the one I use for username lookup. The mixed-case one is used for displaying the username.
If you need to report a large amount of data in a particular format. 100,000 email addresses all formatted in a particular way? Probably best to just store them in that format in the db.
If your translation from one format to another is computationally expensive. Doubly so if you're processing lots of records.
In this case, if all you're doing is looking up or retrieving an email in a specific format, I'd recommend adding a field for it and then indexing it. That way you won't need to do actual document retrieval for the lookup or the display. Super fast. Disk storage space for something the size of an email address is super cheap!
If my JSON data-file comes from a database result set and that result set is empty, how do I tell OFC2 to display an error message, instead of crashing because of a malformed JSON string?
Add tags for javascript and actionscript-3 to this question and you should get a load more views and useful responses than you currently are, with more precise details than I am giving. Post the actual JSON string that is causing you the problem and that you would like to be guarded against. That way people can suggest a regexp to catch it, treating it as a string rather than as JSON data at some point before JSON.decode() happens.
In more detail:
You can catch it in two places. One route is to switch over to using the javascript interface to OFC2 and use client side javascript to detect the bad string. This allows you to modify the JSON string client side. See http://teethgrinder.co.uk/open-flash-chart-2/tutorial-5.php for that approach. One downside is that the clients must have javascript enabled for this to work.
Alternatively, since OFC2 is LGPL, you or an actionscript developer can dive into the OFC2 source code and do the same thing there. I am not an actionscript developer so you are better off ensuring you get a reply from one.
The main thing is to add those two tags to this question. 22 Views is way too low for a question with a bounty of 500. Hope this helps.
Several solution avenues are possible, depending on your level of access to the server and your knowledge of JavaScript and/or any server-side platforms.
With access to database
Depending on the kind of data you are displaying, it might be possible to add dummy records for those queries that would otherwise have returned an empty set. If you have access to the query definition, you may check for the empty set in the DB-query. For example, if you're on MS SQL Server you could check the condition with some T-SQL statements.
With access to server
If you have access to the server side script generating the dataset, add a condition that returns some default value that OFC2 will handle correctly.
With access to another server or serverlocation
If you don't have access to the specific script, you may consider creating a new script at another location that queries the original script and replaces empty results with the default value.
Client-side only
You can add the JavaScript open_flash_chart_data function (see tutorial) to replace empty datasets. OFC2 can use that function as data source.
Hope this helps.