Currently, my website is using a Mongo.Collection to hold data submitted from another site. Strings are sent over through HTTP methods and packed into the collection afterward. However, this collection now needs to support storing larger files, but still needs to hold the data already stored, so I've been looking into converting the collection into GridFS. Is there a way to attach the data onto empty files as metadata, or is the conversion more convoluted than this?
Related
I wanna pull whole data from the firebase database and access that without change data.
firebase.database().ref(‘someBigNode’).once(‘value’, (snapshot) => {
//do something
}
But it will cost many memory.
May I ask how to pull whole data partially and save memory?
When you read data from Firebase Database with the JavaScript (or iOS or Android) SDK, it will always read complete nodes. So the only way to retrieve less data is to retrieve a node lower in the JSON tree.
If you find you need to retrieve a part of each node under someBigNode, you should split that part of each node out into a top-level node of their own importantBitsOfSomeBigNode.
It's unfortunately hard to be more helpful without a more concrete example of your data structure and the bits you're trying to retrieve (and the reason why those bits are special).
I am working with php and heatmap js to generate a heat-map.
I was thinking of going down the path of allowing the user to upload a floor-map jpg file initially and then allow him to add the sensor names to different locations in the floor-map.
Once the sensor locations are specified, I need to save that configuration to an XML file. Once I have this set of information (img_id, [sensorid1,x1,y1], [sensorid2,x2,y2],..,[sensoridn,xn,yn]), I can query my database for the latest values of sensors and then display as heat-map on the image (on the specific sensors' x and y coordinates) real-time.
I would like to know if saving the configuration as XML is the right way of doing it. Is there there a better way of temporarily storing the information using javascript/PHP?
There are likely a bunch of ways to solve this. My preference would be for JSON, as it is natively supported by Javascript and PHP. It is also MUCH easier to read and write.
When you say "saving", what do you mean? If you need it to be stored server side, then creating DB entities that the data structure can be mapped to and stored in will be far better than trying to create files server-side. Depending on how the app gets hosted, you may not have permission to do that, and if your server ever goes away you could loose that data (However, there are safe ways to create files using a service like AWS S3). Storing it in a database not only gives you a single place to worry about backups, but also lets you query the data in interesting and powerful ways (SQL etc) easily, without having to figure out how to do that for files with every new query.
I have a flat data file in the form of xml, but there isn't a real Windows viewer for the file, currently. I decided to create a simple application with Node-WebKit, just for basic viewing - the data file won't need to be written to by the application.
My problem is, I don't know the proper way to read a large file. The data file is a backup of phone SMS's and MMS's, and the MMS entries contain Base64 image strings where applicable - so, the file gets pretty big, with large amounts of images (generallly, around 250mb). I didn't create/format the original data in the file, so I can't modify it's structure.
So, the question is - assuming I already have a way to parse the XML into JavaScript objects, should I,
a) Parse the entire file when the application is first run, storing an array of objects in memory for the duration of the applications lifetime, or
b) Read through the entire file each time I want to extract a conversation (all of the messages with a specific outgoing or incoming number), and only store that data in memory, or
c) Employ some alternate, more efficient, solution that I don't know about yet.
Convert your XML data into an SQLite db. SQLite is NOT memory based by default. Query the db when you need the data, problem solved :)
I'm ending up having to do hacks to convert 'true' to just true and it's creating code smell.
Is there a library like https://github.com/thephpleague/fractal that allows me to transform my response into the types I need?
In cases like this it's almost always better to fix the API to return data in a usable format rather than trying to post-process the result on the client.
In your case there are several routes you could take:
Store the list as a JSON string directly in the database.
This means you don't have to do any processing on the server and can just return it 'as is'. However you lose the ability to do queries on the data directly and need to resort to things like LIKE and string operations.
Store the data relationally, and process it on the server to turn it into JSON
Here you retain the ability to do queries on your data, but you may need to do several queries to get all the data you need and then connect it on the server. (eg. you would do one SELECT on the user table to get a user, and then you would need to do another SELECT on the friends table where the userid matches your first user. You would then need to merge these results to create your JSON.) This is usually the best way to do it.
You can also turn the result into JSON directly inside the database engine using a user defined function. For example using https://github.com/mysqludf/lib_mysqludf_json#readme
This is somewhat similar to 2, but it ties your stored procs to the JSON format.
Getting Json from the Server and displaying it in the grid is relatively straight forward. In the application (http://pssnet.com/~devone/extjs3/loadSelection5.html) I generate a dynamic grid datastore. I need to send this to the Server for further processing.
If there is a way to convert the data store to json, I can strigyfy it and send it a param...as in jQuery.
Otherthan looping through the whole datastore to build json, there seems no methods.
Thank you.
(I can guarantee that the following applies to Ext > 2.3)
Because Ext.data.Store stores an array of Ext.data.Record objects, it has no direct access to the underlying data (Ext.data.Record encapsulates the data), so, indeed, there's no direct way to do it. But Ext.data.Record itself has a public property called "data" (an object with the field:value properties), which you can collect into an array (e.g. using the Ext.data.Store#each method) and then encode with Ext.encode() to "stringify" it.