We Keep Coding

Best practice for database data rearrangement/transformation?

I have a MySQL database and retrieve data using php on a website where I would like to visualize the data in different ways. For this I also need to transform the data (e.g. creating sums, filtering etc.).
My question is now at which step of the data flow this kind of transformation makes most sense, especially regarding performance and flexibility. I am not a very experienced programmer, but I think these are the options I have:
1.) Prepare views in the database which are already providing the desired transformed data.
2.) Use a PHP script that SELECT's the data in the transformed way.
3.) Just have a SELECT * FROM table statement in PHP and load everything in a json, read it in js and transform the data to a desired version.
What is best practice to transform the data?

It all depends in the architectural design of your application.
At this time SOA (Service Oriented Architecture) is a very used approach. If you use It, logic use to be in the services. Database is used as a data repository, and UI manage final data in light weight format, only really needed information.
So in this case, e.g. your option number 2 is most appropriated.

Using math in browser using XML data

I'm trying to make pull XML or Csv data into a HTML file then I want to use math to add up the values and show the result on the page ( I'm basically trying to display invoices on a web browser)
My skill set is HTML/CSS and I understand a little JavaScript
I've managed to pull XML data into HTML using http request and style that information using xslt
Really what I'm asking is what is the best solution to my needs is it using the above method then using xquiry to add up values or would I need to learn a bit of Ajax, Json and calculate the values with JavaScript?

You really should learn AJAX in order to fetch and manipulate data instead of fetching presentation parts. That's the way everyone follows as it allows more responsive interactions with the user and a cleaner architecture in case of complex interactions.
But that doesn't mean you must abandon XML : originally AJAX was built on XML (the X in AJAX) and not JSON.
Personally I prefer JSON, and I think it will be easier to manage in the long term, but if the server side is hard to change, you can fetch the XML (look for example at jquery's ajax function), build javascript objects using it, and then change your screen using those data. If later you decide to use JSON instead of XML, you'll just have to change the "parsing" part of the client code.

"I'm trying to make pull XML or Csv data into a HTML file then I want to use math to add up the values and show the result on the page"
You can do this with either XSLT or javascript. However, with XSLT things can become pretty complicated, depending on what version you're using. XSLT 1.0 has pretty limited set of functions for aggregating results. For all XSLT, you can't reassign variables you'll have to solve many of these things with recursion. In my opinion, not really a comfortable method.
Regardless of the choice between XSLT and Javascript, I would also question the architecture that would put this kind of logic in the presentation layer in the browser. I think it would be better if the server side would perform all the calculations that are required, and limit the browser's tasks to styling the output.

Where to render Ajax search results in an object oriented approach using Coldfusion?

I'm updating a Coldfusion8/MySql site with a fairly complex search from "spaghetti to object" (= separate view, controller and process - no framework, everything handled by Jquery Mobile).
I need to run the search query through Jquery-AJAX and am now posting the search form to my searchProcess.cfc, which does the database queries.
Question:
I'm not sure where to render the results?
The results will be fairly complex (database with a few million records, rows of 40 fields) and should end up in a single result layout or a multiple result layout file.
I was thinking of constructing the files inside the cfc and handing them back via cfsavecontent, but I'm reading everywhere this is a no-no...
What are the alternatives then?
I could set up a template_single.cfm and template_multi.cfm, pass back pure search results as AJAX response and then fire another AJAX call from the success handler to call the template and then render the output of this 2nd call. This seems awfully complicated, plus I don't see where I can fit my pagination in there without passing around large datasets.
So I'm looking for some advice on how to handle search-results in an object-oriented-way?
Thanks for input!
EDIT:
After a few more hours of googling, I'm currently looking at the following option:
1.) run a single database query to return paginated results - as per here
2.) send data with 0-25 records back to AJAX in JSON
3.) trying to use a template cf/js in a loop (length 1 or length 25) - as per here
This would mean data-transfer only for 1-25 raw records in JSON. If I try to render in the success handler, I'm not having to http-request another template.
Does this approach make sense?

First off, I see nothing wrong with putting display logic in a .cfc that is specifically for display logic. I know it's not strict MVC, but depending on what you're doing, it can work just fine. What I don't like is .cfc's that do any sort of output within the function. I always hand back any data from the function.
/end opinion
For this particular problem, I second EDIT idea of setting up the view to be almost all HTML/jQuery with AJAX calls for paginated recordsets. As far as the single/multiple results, I'd go with separate jQuery functions depending on which one you needed. The nice thing about this is that you could have the multiple recordset display call the single record display to view a single record (while still retaining the full recordset in the Dom).
In both cases, I highly recommend getting a Javascript Dump function (helps so much in visualizing the DOM data.)
PS. If anybody finds any newer/better JS Dump functions that work like cfdump, please, please, please let me know!

should I ALSO keep my data in a Javascript data structure? or just in the DOM?

I am very new to Javascript and Jquery, so my apologies for this beginner's question.
In a simple ajax web app, I am creating an HTML page which is mainly a big table. Each row in this table describes an event (party, show, etc). My page not only displays this information but is meant to let the user do a bunch of things with it, in particular to dynamically search and filter the table according to a variety of criteria.
First an abstract beginner's question: in this broad kind of situation (by which I mean that you want your javascript code to run a bunch of operations on the information you retrieve from the webserver) would you use the DOM as a data structure? The ease with which one can search and manipulate it (using Jquery) makes that a possibility. (E.g., "find me table rows describing an event with date column = 2010-01-01 and event type column = 'private party'.) Or would you keep the same information in a traditional Javascript data structure, search/filter/operate on that using plain javascript code and then update the DOM accordingly to display the results to the user?
(As a newbie, I imagine the first, DOM-only approach to be slower while the latter to be take up a good deal of memory. Right? Wrong?)
Assuming the second strategy is reasonable (is it?), then a practical question: can I simply store in my Javascript objects a pointer to the corresponding Jquery object? Eg, can I do
var events = new Array();
// ....
var event3094 = new Event('party','2010-01-01' /*, ... */);
event3094.domElement = $("#correctIdOfTheEventRowInMyTable");
events.push(event3094)
Does this store just a reference (pointer?) to the Jquery object in each Event object or is it creating a new copy of the Jquery object?
I am just wondering "how the pros" do it. : )
Thank you for any advice and insight.
cheers
lara

There are so many ways to do this, but DOM manipulation will almost always be slower than JS manipulation.
To answer your question, anytime you use $(selector) a new jQuery object is created and a match to find the element is performed.
I would recommend two approaches:
FIRST OPTION
Load data in a normal HTML table
Read through the rows, and store just the data (each cell's contents) in an array similar to your code example.
Store a reference to the tr in that object.
Then you can process filter, etc, and only apply changes and searches to the DOM as needed.
SECOND OPTION
Load the page without the table
Load the data as JSON from the server, and generate a table from the data
Store reference to the tr element
Basically, you don't want to perform a $(selector) a 1000 times. The concept is something like this:
var $rows = $("table tr");
// ...
event.domElement = $rows[0]; // Stores reference to existing DOM node, not a new jQuery object.
Then when you need to use jQuery methods on the object, you could use $(yourEvent.domElement) to wrap it in a jQuery wrapper.

Depending on the number of rows you might expect to be shown for most of your users (let's assume it's no more than a few hundred), I myself would probably aim to just keep everything in the DOM table that you're already building. (If you are expecting to be dealing with thousands of rows on one page, you might want to explore a different solution rather than sending it all to the browser.)
There are a few things that you did not mention in your original post. First, how are you creating this table? I imagine using a server-side solution. How easy is that to modify? How much extra work would it be to go through and generate all of your data a second time in a different format, as XML or JSON? Does this add a bunch of complexity on the server-side, only so that you can add more complexity client-side to match? Certain platforms may make this trivial, but is something to consider.
Now, in regards to your alternatives to the DOM:
I agreed and mentioned in a comment above that I don't think JSON would be very optimal "out of the box" for what you want to do. A javascript array is no better. XML is nice in that you can use jquery to easily traverse/filter, but then you still have to deal with your DOM. Sure, you can store references to your DOM elements, but that just seems like a bunch of work up front and then some more work later when matching them up. And without necessarily guaranteeing any major performance boost.
So, to answer your question directly as it is phrased, should you ALSO keep your data in a JavaScript data structure, or just in the DOM: You did mention this was a "simple" ajax web app. My recommendation is to try and keep this simple, then! Your example of how you can so easily use jquery to find rows and cells based on search criteria should be enough to convince you to give this a try!
Best of luck!-Mike

I think you'l find that the DOM method is close to the same speed if I follow your logic right. The second method will require you to manipulate the data, and then apply the changes to the DOM, where the first method allows both operations at the same time.

If youve got alot of data i would for go making objects and just supply it as XML. that way you get most of the same features as operating on the HTML DOM but you dont have a crazy markup structure like a table to navigate through.

I think it's largely a personal preference, but I like to store the objects in JavaScript and then render them to HTML as needed. From my understanding
event3094.domElement = $("#correctIdOfTheEventRowInMyTable");
will store a reference to the jQuery wrapper of your row element (not a copy).
One benefit of storing is JS is that if you have lost of objects (eg 10,000 ) and only render a small fraction of them, the browser will perform a lot better if you're not creating 10,000 * (number of DOM elements per object) elements.
Edit: Oh, and if you can, you might want to send the data from the server to the client as JSON. It's very compact (compared to XML) and in the new browsers it can be parsed quickly and safely using JSON.parse(). For older browsers, just use this library http://www.json.org/js.html . The library will only create a global JSON namespace if the browser doesn't supply it.

One thing to consider is how you need to access your data. If all the data for an element's event is contained in the element (an event operating solely on a table cell for example), then storing in the DOM makes sense.
However, if the calculation of one element depends on data form other elements (a summation of all table cells in a particular column, for example), you may find it difficult to gather up all the data if it's scattered about in DOM elements compared to a single data structure.

Best practices for returning and displaying data from AJAX calls

I have some jquery/php interaction set up on a page. It submits some data to the server and gets back records of data which are then to be aligned on the page for comparison and possible action beyond that.
My question is what is the best practice for returning the info and then displaying it?
Return JSON object and then create
html on the fly with js and display
the data?
Return JSON object and then
put that data in already created
containers for the data on the page?
Return pure html from the server and
just put that on the page?
I was rolling these through my head last night and couldn't really figure out if one way would be better for any particular reason.
I'm not a js guru so wasn't really sure the pros/cons and caveats to these different methods.

I think it ends up depending on your app.
Pure HTML is the easiest, you just drop in in place and viola. JQuery makes it a breeze to add the proper events and what not.
However, whenever I've used AJAX it's always evolved into returning JSON and building elements on the fly. If you are populating a tree for example, it may become messy to get the proper nesting. This forces you to have to do client side code anyway at which point simply using JSON from the start is cleaner.
Also, If you plan to use the data calls from other pages then using JSON is the way to go because the HTML will bee fixed.

This completely depends on the way you have things set up in your application. I, for one, prefer to return JSON (second approach in your list), because I have an 'error code' parameter that I check in onSuccess function before updating the content on the page, and if it's not zero then I notify the user of the error, including the error message that came back from the server (server-side validation, database timeout, etc.).

I think you're missing a perfectly valid option, one which I use often. This is my typical schema and it has yet to fail me... :-)
Here's the basic jQuery template I use:
$(function() {
$.getJSON('/some/page',{foo:bar,bar:foo},function(json) {
if(json.outcome == 'success') {
$('body').prepend(json.html);
} else {
// Somehow let the user know why it didn't work
alert(json.error);
}
});
});
Here's the basic backend (PHP in my case) structure I use:
<?php // Page: '/some/page'
/* Blah Blah Blah... do whatever needs to be done... */
// If everything turns out okay (assuming '$output' is the HTML
// you want to display...
echo json_encode(array('outcome'=>'success','html'=>$output));
// If something goes wrong... just do:
echo json_encode(array('outcome'=>'error','error'=>'Uh oh... something is broken'));
Naturally, you'll want to be more specific with your error by putting them into some variable or something. But, you should get the idea. Also, of course you can add more information to the json output. You can have some pre-made HTML and also some other information like a 'success notice' or a new class name for some element, I dunno... whatever... the possibilities are endless.
Anyways, the reason I choose this route is because it's usually faster (based on my experience) to append pre-made HTML to the DOM rather than looping over JSON and inserting the stuff unless it's just, like, a bit of text to replace into an element. But, the method I've shown is, IMO, the best of both worlds. You can attach HTML as a string to one of the JSON properties.
Happy jQuerying :-)

The "possible action beyond that" part of your question makes a big difference. If you need to do other things with the data besides display it, returning as JSON is a clearly better option because you can work with the data as a native JavaScript object instead of having to traverse the HTML DOM. If all you ever intend to do is display it, I don't see any reason to go through the trouble of building that display in JavaScript; just build the HTML in your presentation layer on the server.

This came up recently and possible a dupe: The AJAX response: Data (JSON, XML) or HTML snippet?.
If you are going to be creating HTML then you may as well be returning HTML directly and inject it into the DOM. However, there are times you need to work with objects which is where JSON comes in handy.
If you return a Person object for example then you can greet Person.Name and show Person.Preferences which is really handy. It depends on your design but the general considerations should be to keep HTML out of Javascript unless you're building RIA.

I have used all three and have come to the conclusion that returning HTML is better when introducing new elements to a page.
My experiance is that when building HTML with javascript I am usually replicating work that will have already have been done for the non javascript user journey.
I still prefer parsing json for updating existing elements or creating javascript only functionality. I tell myself this for bandwidth, but I think it just because I love javascript.
As a forth option, I read a great post about how Flickr deal with vast quantities of data with string concatination. Basically just parse a big o' string down the pipe and chop it up on the client. This significantly reduces the on the server, with only a marginal increase on the client.

Returning pure HTML is the best solution. For the most part gzip should neutralize any difference in bandwidth, and rendering via javascript on the client can be slow if the client is a crappy machine. Finally, writing javascript to render HTML is hard to work with compared to using something nice like a view if you use MVC.