I have a Parse Class called Currency that provides static exchange rates using the exact format from https://openexchangerates.org/. I do not need the rates "real time" so I would like to somehow GET the info using cloud code/scheduled job once a day and then overwrite the existing entry in Currency. When users query against the rates throughout the day, it uses the Parse class and not an API call to https://openexchangerates.org. Can overwriting a class or an object in the class be done in Parse.com? If not, are there suggestions on how to go about this? Thanks in advance.
Create your job and set it to run once a day as required. In the job, make the http request. Iterate the result received to process each conversion. For that conversion, run a query to find the existing item in the data store (if one exists) by matching the conversion currencies. Update this object or create a new one if no match was found.
An alternative would be to batch delete all conversions before processing the result. This isn't great though as it leaves a finite time when no conversions will be available to users and if there is an error not even an old conversion will be available.
Related
This is more of a architectural questions. An external platform had product and price information for let's say, books. There is an API available to get this information.
What I read is that it should be possible to create a function in Javascript and connect the Javascript to a page where you want to show the data on my own website. This would mean that for each page request an API-call is made. Since the requested information only changes once a day maximum this does not sound the most efficient solution.
Can someone advise a better solution? Something into the direction of a similar php or javascript function that does the request on the background, schedule an update and import the data into mysql? If so, what language would be most common.
I need the solution for a Joomla/php/mysql environment
Here's a simple idea - fetch and store results from the API (ones you think aren't gonna change in a day), either on disk, or in the database, and later use these stored results to retrieve what you otherwise would've fetched from the API.
Since storing anything in frontend JS across page reloads isn't easy, you need to make use of PHP for that. Based on what's given, you seem to have two ways of calling the API:
via the frontend JS (no-go)
via your PHP backend (good-to-go)
Now, you need to make sure your results are synced every (say) 24 hours.
Add a snippet to your PHP code that contains a variable $lastUpdated (or something similar), and assign it the "static" value of the current time (NOT using time()). Now, add a couple of statements to update the stored results if the current time is at least 24 hours greater than $lastUpdated, followed by updating $lastUpdated to current time.
This should give you what you need with one API call per day.
PS: I'm not an expert in PHP, but you can surely figure out the datetime stuff.
It sounds like you need a cache, and you're not the first person to run into that problem - so you probably don't need to reinvent the wheel and build your own.
Look into something like Redis. There's an article on it available here as well: https://www.compose.com/articles/api-caching-with-redis-and-nodejs/
I have a table with a thousand records in it and I want to do a google like search full-text/fuzzy search.
I read about MySQL v8's Full-Text search and let's say we don't have that functionality yet.
There is this JavaScript library called Fuse.js that do fuzzy-search which is what I need.
I can combine it by creating a API that returns the table data in JSON format and then pass it to Fuse.js to do a fuzzy-search.
Now, I think it's not recommended to load all data from table every time someone wants to search.
I read about Redis, and the first thing that came in my mind is to save all table data in Redis using JSON.stringify and just call it every time instead of querying the database. Then whenever a data is added in the table, I will also update the contents of the data in Redis.
Is there a better way to do this?
That is a very common caching pattern.
If you need a more efficient way to store and retrieve your JSON to/from Redis you might want to consider one of the available Redis Modules.
e.g.
RedisJSON allows you to efficiently store, retrieve, project (jsonpath) and update in place.
RediSearch allows you to have full text search over Redis Hash and efficiently retrieve data according to the user's query.
Last
RedisJSON2 (aka RedisDoc) combines both modules above, meaning efficient JSON store and retrieve with Full Text support
I have a node.js API that is responsible for 3 things:
Registering a buyer
Getting a buyer with ID
Finding the matching buyer's offer based on some criteria
Details here
Since I'm new to Redis, I started the implementation like this:
JSON.stringify the buyer and store it with SET
Store all buyer's offers as ordered set (this is for the third endpoint, which requires the offer with the highest value) - this set contains string that represents the name of a hash
Then, that hash stores strings that represent the names of sets that have certain values and a location which the user will be redirected to after these conditions have been fulfilled (buyer1_devices, buyer1_hours, etc.)
Now, here is the problem:
I need to get GET /route working. As described on GitHub page that I have provided, I have 3 parameters: a timestamp, devices, and states. I have to browse through all the sets and get the appropriate location to redirect a user to. The location is stored in a hash, but I have to browse through all the sets. Since this is probably a bad implementation, where did it all go wrong and to go about implementing this?
Note that this is a redis problem, not a node one. I need instructions on how to go about implementing this in Redis, and then I will be ready to code it in Node.
Thank you in advance
The first rule of Redis: store the data just like you want to read it.
To answer the /route query you need "filteration" on two attributes of from the buyers' offers - state and device. There is more than one way to skin that cat, so here's one: use many Sorted Sets for the offers.
Each such offers Sorted Set key name could look like this: <device>:<state> (so the example offered in the git will be added to the key desktop:CA).
To query, use the route's arguments to compose your key's name, then proceed regularly to find the highest-scored offer and resolve the buyer's details in the Hash.
Now go get that job!
Im developing a client-server real-time program and I need my server to be up-to-date always. Unill now I have been implement a GET request from the server every X seconds that returns all the entities from the MongoDB.
Now I have big amount of entities and I need to GET only the entities which have been updated since the last GET request.
I think about running sequence in the db for each entity and check every X seconds if the sequens have been increased.
But I will prefer a better way.
Is there any way to get only the recent changes from mongo? Or any nicer architecture ?
You can have a last updated time in the collection. In the client side, you can maintain a last get time.
In the subsequent requests, get all the documents from collection where last updated time is greater than last get time. This way you will get the documents that got updated or inserted since you last get the data (I.e. delta).
Edit:
MongoDB maintains the date object in UTC format. As long as the date at client side is maintained in UTC and send the same data in the subsequent request, it should retrieve the latest updated records.
I have a MongoDB JavaScript function saved in db.system.js, and I want to use it to both query and produce output data.
I'm able to query the results of the function using the $where clause like so:
db.records.find(
{$where: "formatEmail(this.email.toString(), 'format type') == 'xxx'"},
{email:1})
but I'm unable to use this to return a formatted value for the projected "email" field, like so:
db.records.find({}, {"formatEmail(this.email.toString(), 'format type')": 1})
Is there any way to do this while preserving the ability to simply use a pre-built function?
UPDATE:
Thank you all for your prompt participation.
Let me explain why I need to do this in MongoDB and it's not a matter of client logic at the wrong layer.. What I am really trying to do is use the function for a shard bucketing value. Email was just one example, but in reality, what I have is a hash function that returns a mod value.
I'm aware of Mongo having the ability to shard based on a hashed value, but from what I gather, it produces a highly random value that can burden the re-balancing of shards with unnecessary load. So I want to control it like so func(_id, mod), which would return a value from 0 to say 1000 (depending on the mod value).
Also, I guess I would also like to use the output of the function in some sort of grouping scenario, and I guess Map Reduce does come to mind.. I was just hoping to avoid writing overly complex M/R for something so simple.. also, I don't really know how to do Map Reduce .. lol.
So, I gather that from your answers, there is no way to return any formatted value back from mongo (without map/reduce), is that right?
I think you are mixing your "layers" of functionality here -- the database stores and retrieves data, thats all. What you need to do is:
* get that data and store the cursor in a variable
* loop through your cursor, and for every record you go through
* format and output your record as you see fit.
This is somewhat similar to what you have described in your question, but its not part of MongoDB and you have to provide the "formatEmail" function in your "application layer"
Hope it helps
As #alernerdev has already mentioned, this is generally not done at a database layer. However, sometimes storing a pre-formatted version in your database is the way to go. Here's some instances where you may wish to store extra data:
If you need to lookup data in a particular format. For example, I have a "username" and a "usernameLowercase" fields for my primary user collection. The lowercase'd one is indexed, and is the one I use for username lookup. The mixed-case one is used for displaying the username.
If you need to report a large amount of data in a particular format. 100,000 email addresses all formatted in a particular way? Probably best to just store them in that format in the db.
If your translation from one format to another is computationally expensive. Doubly so if you're processing lots of records.
In this case, if all you're doing is looking up or retrieving an email in a specific format, I'd recommend adding a field for it and then indexing it. That way you won't need to do actual document retrieval for the lookup or the display. Super fast. Disk storage space for something the size of an email address is super cheap!