I am creating an offline mobile web app, and am looking at using JSON to replicate some of my database tables and storing that in localStorage. (I am aware of Web SQL Database but it doesn't look particularly future-proof.)
I started with a very basic JSON output from the database, which looks a bit like this:
{
"1": {"id":"1","name":"Hello","alias":"hello","category":"8"},
"2": {"id":"2","name":"World","alias":"world","category":"3"},
...
}
However, there is a lot of data in many tables and space could be an issue with the constant repeating of field names. Storing the data like this halves the size:
{
"1": ["1","Hello","hello","8"},
"2": ["2","World","world","3"},
...
}
But now I have to reference a piece of data with a numeric index, possibly filling my code with magic numbers. I thought of storing an array like ["id","name"...] in another variable but the extra lookups seem like they would get messy.
Are there any practical ways to avoid that, but also keeping the Javascript code fairly neat? Any other useful strategies for this kind of development?
would it be possible to convert it into a format like this:
{
id:{1:1, 2:2, ...},
name:{1:hello, 2:world},
alias:{1:hello, 2:world},
category{1:8,2:3}
}
This way, you only store each column once, but you can still easily find things by their id.
JSON is not a database. JSON is a data interchange format.
Not sure if it would work across all mobile platforms, but XML would be an option.
Related
I have a MySQL database and retrieve data using php on a website where I would like to visualize the data in different ways. For this I also need to transform the data (e.g. creating sums, filtering etc.).
My question is now at which step of the data flow this kind of transformation makes most sense, especially regarding performance and flexibility. I am not a very experienced programmer, but I think these are the options I have:
1.) Prepare views in the database which are already providing the desired transformed data.
2.) Use a PHP script that SELECT's the data in the transformed way.
3.) Just have a SELECT * FROM table statement in PHP and load everything in a json, read it in js and transform the data to a desired version.
What is best practice to transform the data?
It all depends in the architectural design of your application.
At this time SOA (Service Oriented Architecture) is a very used approach. If you use It, logic use to be in the services. Database is used as a data repository, and UI manage final data in light weight format, only really needed information.
So in this case, e.g. your option number 2 is most appropriated.
I have a MongoDB JavaScript function saved in db.system.js, and I want to use it to both query and produce output data.
I'm able to query the results of the function using the $where clause like so:
db.records.find(
{$where: "formatEmail(this.email.toString(), 'format type') == 'xxx'"},
{email:1})
but I'm unable to use this to return a formatted value for the projected "email" field, like so:
db.records.find({}, {"formatEmail(this.email.toString(), 'format type')": 1})
Is there any way to do this while preserving the ability to simply use a pre-built function?
UPDATE:
Thank you all for your prompt participation.
Let me explain why I need to do this in MongoDB and it's not a matter of client logic at the wrong layer.. What I am really trying to do is use the function for a shard bucketing value. Email was just one example, but in reality, what I have is a hash function that returns a mod value.
I'm aware of Mongo having the ability to shard based on a hashed value, but from what I gather, it produces a highly random value that can burden the re-balancing of shards with unnecessary load. So I want to control it like so func(_id, mod), which would return a value from 0 to say 1000 (depending on the mod value).
Also, I guess I would also like to use the output of the function in some sort of grouping scenario, and I guess Map Reduce does come to mind.. I was just hoping to avoid writing overly complex M/R for something so simple.. also, I don't really know how to do Map Reduce .. lol.
So, I gather that from your answers, there is no way to return any formatted value back from mongo (without map/reduce), is that right?
I think you are mixing your "layers" of functionality here -- the database stores and retrieves data, thats all. What you need to do is:
* get that data and store the cursor in a variable
* loop through your cursor, and for every record you go through
* format and output your record as you see fit.
This is somewhat similar to what you have described in your question, but its not part of MongoDB and you have to provide the "formatEmail" function in your "application layer"
Hope it helps
As #alernerdev has already mentioned, this is generally not done at a database layer. However, sometimes storing a pre-formatted version in your database is the way to go. Here's some instances where you may wish to store extra data:
If you need to lookup data in a particular format. For example, I have a "username" and a "usernameLowercase" fields for my primary user collection. The lowercase'd one is indexed, and is the one I use for username lookup. The mixed-case one is used for displaying the username.
If you need to report a large amount of data in a particular format. 100,000 email addresses all formatted in a particular way? Probably best to just store them in that format in the db.
If your translation from one format to another is computationally expensive. Doubly so if you're processing lots of records.
In this case, if all you're doing is looking up or retrieving an email in a specific format, I'd recommend adding a field for it and then indexing it. That way you won't need to do actual document retrieval for the lookup or the display. Super fast. Disk storage space for something the size of an email address is super cheap!
I'm building a guestlist app for my company using PHP, Javascript/jQuery/Ajax, JSON and localstorage. It will be used on smartphones: primarily Iphone 4. I'm using localstorage as my cache since the search-part of the application has to work in offline mode.
I'm having performance issues while searching through a guestlist.
The app's flow looks like this (for this examaple I'm working with guestlist which contains 600 guests)
1. Retrieve all guests from the server with PHP encode with JSON and send back to js via AJAX. This works fine.
2. Parse the PHP responseText (called phpData) using JSON.Parse:
var parsedMysqlData = JSON.parse(phpData);
which gives us an JSON.Array containing 600 objects looking like this:
Object: {
Id: Int
EventId: int
guestInfo: string
guestlist: string
guestName: string
reference: string
total: int
used: int
}
3. Save the JSON.Array to the user's localstorage using JSON.Stringify:
localStorage.setItem(0, JSON.stringify(parsedMysqlData));
4. when the user starts searching we get his search string then retrieve our guestlist using JSON.parse in localstorage like this:
var currentGuestlist = JSON.parse(localStorage.getItem(0));
And then iterate through our objects using this for-loop trying to match his search string with our guests the array currentGuestlist:
for (i=0; i<currentGuestlist.length; i++) {
// match 'currentGuestList[i]['name']' with what the user typed in search form
}
For some reason this takes a really long time on an iPhone 4. Searching through 600 objects will freezes the iphone for about 4 seconds before returning the matches.
Before storing arrays containing JSON objects in localStorage and parsing it with JSON, I simply stored unordered strings in localStorage and it work a whole lot faster. The JSON objects ad structure to the data stored in localStorage which is crucial. So I guess the speed issue has to have something to do with the fact that I'm using JSON objects? How can i structure my data i localStorage in an organized way while still maintaining as good speed performance as before?
Lastly anykind of tips or advice on which techniques you would use to make this app as lightweight and fast as possible is greatly appreciated.
Are you fetching the list per each search from the local storage? Don't do that, instead store it in the local storage only as needed (whenever it changes), and keep it always as a data structure.
Simply having objects instead of plain strings cannot be the reason for slowness, as everything in JavaScript is an object already, and thus it should only be slowing by a constant factor.
Furthermore, if this is about autocomplete kind of behaviour, then I suggest you would slow down the search, and also consider that if the user types in the box "Ma", the list gets filtered, and the user adds "tt" for "Matt", only previously filtered matches need to be considered...
Perhaps your set up would me more suited to use the WebSQL-database instead of storing a JSON object in the local storage. WebSQL is now deprecated but it's pretty well supported in webkit-browsers. and i'm using it with good results on a couple of projects.
You can read more about it here: http://html5doctor.com/introducing-web-sql-databases/
I'm currently working on a project where I'm dealing with a fair amount of JSON data being transmitted backwards and forwards and stored by the browser as lists of javascript objects. For example:
person: {
// Primary Key
key: "id",
// The actual records
table: {
"1": {id: 1, name: "John", surname: "Smith", age: 26},
"2": {id: 2, name: "Mary", surname: "Brown", age: 19},
// etc..
},
indexes: {
// Arrays of pointers to records defined above
"name": [
{id: 89, name: "Aaron", surname: "Jones", age: 42},
// etc..
]
}
I'm finding myself coding all sorts of indexing and sorting algorithms to efficiently manipulate this data and I'm starting to think that this kind of thing must have been done before.
I have experience of using the Ext.data.Store and Ext.data.Record objects for performing this kind of data manipulation, but I think they are overly complex for junior developers and the project I'm working on is a small mobile application where we cant afford to have a 300K+ library added just for the sake of it, so I need something really minimal.
Any ideas if there is a Javascript JSON manipulation framework that has the following:
Can store,
retrieve,
sort,
and iterate through JSON data,
with a clean API,
minimal performance drag (Mobiles dont have a lot of computing power)
and a small payload that is ideally <10K?
I might be asking for too much, but hopefully someone's used something like this... The kind of thing I'm looking for the is the JSON equivalent of jQuery, maybe its not so outlandish.
Take a look on jsonQ
It fullfill all the requirement pointed on question.
Can store,
retrieve
and iterate through JSON data,
Provide traversing (like find,siblings, parent etc) and manipulation method like(value, append, prepend);
sort
Provide a direct array sort method and sort method which run on jsonQ object. (Both sort method run recursively)
with a clean API
Its a trial to have same API for JSON as jQuery DOM APIs . So if you are familiar with jquery. Its easy to catch up. More over complete documentation of apis is available.
minimal performance drag
It create a new format on initialization of jsonQ for a JSON data which is used internally to traverse data which are more efficient. (Its like having all the loops at once, so you don't have to make loop over loop to iterate each time that json).
and a small payload that is ideally <10K?
minified version is 11.7 kb.
Actually your question is not good, I suppose. From your example one could see, that you're trying to emulate SQL-like storage with JSON. Maybe, you just need to take IndexedDB?
jsonpath matches points 4-7 (and maybe 3) of your exact requirements and JSON global object allows 1 and 2 with just once call for each.
Also IMHO requirements are unreal, especially with the last one about size.
I think Lawnchair is something you're looking for. Its homepage says it is made with mobile in mind, however I've not used it before so I cannot comment on that.
It is simple key-value store, although you can define your own indexes, something like with CouchDB. I've not seen support for selectors, however there is a query plugin available, promising easy selection.
Something like jQuery relies on sizzle, which is CSS selector library, that isn't applicable in your case. XPath is in my opinion your best bet, since it's used primarily with XML, a tree structure. Since JSON object can be represented as a tree, I've found JSONSelect, a small library that supports JSON Xpath selectors in 'CSS-ish' way.
If you'd be able somehow to plug JSONSelect into Lawnchair, I think you've found a great combination :)
I am storing a JSON string in the database that represents a set of properties. In the code behind, I export it and use it for some custom logic. Essentially, I am using it only as a storage mechanism. I understand XML is better suited for this, but I read that JSON is faster and preferred.
Is it a good practice to use JSON if the intention is not to use the string on the client side?
JSON is a perfectly valid way of storing structured data and simpler and more concise than XML. I don't think it is a "bad practice" to use it for the same reason someone would use XML as long as you understand and are OK with its limitations.
Whether it's good practice I can't say, but it strikes me as odd. Having XML fields in your SQL database are at least queryable (SQL Server 2000 or later, MySQL and others) but more often than not is a last resort for metadata.
JSON is usually the carrier between JavaScript and your back end, not the storage itself unless you have a JSON back end document orientated database such as CouchDB or SOLR as JSON lends itself perfectly for storing documents.
Not to say I don't agree with using JSON as a simple (that is, not serializing references) data serializer over XML, but I won't go into a JSON vs XML rant just for the sake of it :).
If you're not using JSON for its portability between 2 languages, and you're positive you will never query the data from SQL you will be better off with the default serialization from .NET.
You're not the only one using JSON for data storage. CouchDB is a document-oriented database which uses JSON for storing information. They initially stored that info as XML, then switched to JSON. Query-ing is done via good old HTTP.
By the way, CouchDB received some attention lately and is backed by IBM and the Apache Software Foundation. It is written in Erlang and promises a lot (IMHO).
Yeah I think JSON is great. It's an open standard and can be used by any programming language. Most programming languages have libraries to parse and encode JSON as well.
JSON is just a markup language like XML, and due to its more restrictive specification is is smaller in size and faster to generate and parse. So from that aspect it's perfectly acceptable to use in the database if you need ad-hoc structured markup (and you're absolutely sure that using tables isn't possible and/or the best solution).
However, you don't say what database you're using. If it is SQL Server 2005 or later then the database itself has extensive capabilities when dealing with XML, including an XML data type, XQuery support, and the ability to optimise XQuery expressions as part of an execution plan. So if this is the case I'd be much more inclined to use XML and take advantage of the database's facilities.
Just remember that JSON has all of the same problems that XML has as a back-end data store; namely, it in no way replaces a relational database or even a fixed-size binary format. If you have millions of rows and need random access, you will run into all the same fundamental performance issues with JSON that you would with XML. I doubt you're intending to use it for this kind of data store scenario, but you never know... stranger things have been tried :)
JSON is shorter, so it will use less space in your database. I'd probably use it instead of XML or writing my own format.
On the other hand, searching for matches will suck - you're going to have to use "where json like '% somevalue %'" which will be painfully slow. This is the same limitation with any other text storage strategy (xml included).
You shouldn't use either JSON or XML for storing data in a relational database. JSON and XML are serialization formats which are useful for storing data in files or sending over the wire.
If you want to store a set of properties, just create a table for it, with each row beeing a property. That way you can query the data using ordinary SQL.