I have a very big object in javascript (about 10MB).
And when I stringify it, it takes a long time, so I send it to backend and parse it to an object( actually nested objects with arrays), and that takes long time too but it's not our problem in this question.
The problem:
How can I make JSON.stringify faster, any ideas or alternatives, I need a javaScript solution, libraries I can use or ideas here.
What I've tried
I googled a lot and looks there is no better performance than JSON.stringify or my googling skills got rusty!
Result
I accept any suggestion that may solve me the long saving (sending to backend) in the request (I know its big request).
Code Sample of problem (details about problem)
Request URL:http://localhost:8081/systemName/controllerA/update.html;jsessionid=FB3848B6C0F4AD9873EA12DBE61E6008
Request Method:POST
Status Code:200 OK
Am sending a POST to backend and then in JAVA
request.getParameter("BigPostParameter")
and I read it to convert to object using
public boolean fromJSON(String string) {
if (string != null && !string.isEmpty()) {
ObjectMapper json = new ObjectMapper();
DateFormat dateFormat = new SimpleDateFormat(YYYY_MM_DD_T_HH_MM_SS_SSS_Z);
dateFormat.setTimeZone(TimeZone.getDefault());
json.setDateFormat(dateFormat);
json.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);
WebObject object;
// Logger.getLogger("JSON Tracker").log(Level.SEVERE, "Start");
try {
object = json.readValue(string, this.getClass());
} catch (IOException ex) {
Logger.getLogger(JSON_ERROR).log(Level.SEVERE, "JSON Error: {0}", ex.getMessage());
return false;
}
// Logger.getLogger("JSON Tracker").log(Level.SEVERE, "END");
return this.setThis(object);
}
return false;
}
Like This
BigObject someObj = new BigObject();
someObj.fromJSON(request.getParameter("BigPostParameter"))
P.S : FYI this line object = json.readValue(string, this.getClass());
is also very very very slow.
Again to summarize
Problem in posting time (stringify) JavaScript bottle nick.
Another problem parsing that stringified into an object (using jackson), and mainly I have svg tags content in that stringified object as a style column, and other columns are strings, int mainly
As commenters said - there is no way to make parsing faster.
If the concern is that the app is blocked while it's stringifying/parsing then try to split data into separate objects, stringily them and assemble back into one object before saving on the server.
If loading time of the app is not a problem you could try to ad-hoc incremental change on top of the existing app.
... App loading
Load map data
Make full copy of the data
... End loading
... App working without changes
... When saving changes
diff copy with changed data to get JSON diff
send changes (much smaller then full data)
... On server
apply JSON diff changes on the server to the full data stored on server
save changed data
I used json-diff https://github.com/andreyvit/json-diff to calc changes, and there are few analogs.
Parsing is a slow process. If what you want is to POST a 10MB object, turn it into a file, a blob, or a buffer. Send that file/blob/buffer using formdata instead of application/json and application/x-www-form-urlencoded.
Reference
An example using express/multer
Solution
Well just as most big "repeatable" problems go, you could use async!
But wait, isn't JS still single-threaded even when it does async... yes... but you can use Service-Workers to get true async and serialize an object way faster by parallelizing the process.
General Approach
mainPage.js
//= Functions / Classes =============================================================|
// To tell JSON stringify that this is already processed, don't touch
class SerializedChunk {
constructor(data){this.data = data}
toJSON() {return this.data}
}
// Attach all events and props we need on workers to handle this use case
const mapCommonBindings = w => {
w.addEventListener('message', e => w._res(e.data), false)
w.addEventListener('error', e => w._rej(e.data), false)
w.solve = obj => {
w._state && await w._state.catch(_=>_) // Wait for any older tasks to complete if there is another queued
w._state = new Promise((_res, _rej) => {
// Give this object promise bindings that can be handled by the event bindings
// (just make sure not to fire 2 errors or 2 messages at the same time)
Object.assign(w, {_res, _rej})
})
w.postMessage(obj)
return await w._state // Return the final output, when we get the `message` event
}
}
//= Initialization ===================================================================|
// Let's make our 10 workers
const workers = Array(10).fill(0).map(_ => new Worker('worker.js'))
workers.forEach(mapCommonBindings)
// A helper function that schedules workers in a round-robin
workers.schedule = async task => {
workers._c = ((workers._c || -1) + 1) % workers.length
const worker = workers[workers._c]
return await worker.solve(task)
}
// A helper used below that takes an object key, value pair and uses a worker to solve it
const _asyncHandleValuePair = async ([key, value]) => [key, new SerializedChunk(
await workers.schedule(value)
)]
//= Final Function ===================================================================|
// The new function (You could improve the runtime by changing how this function schedules tasks)
// Note! This is async now, obviously
const jsonStringifyThreaded = async o => {
const f_pairs = await Promise.all(Object.entries(o).map(_asyncHandleValuePair))
// Take all final processed pairs, create a new object, JSON stringify top level
final = f_pairs.reduce((o, ([key, chunk]) => (
o[key] = chunk, // Add current key / chunk to object
o // Return the object to next reduce
), {}) // Seed empty object that will contain all the data
return JSON.stringify(final)
}
/* lot of other code, till the function that actually uses this code */
async function submitter() {
// other stuff
const payload = await jsonStringifyThreaded(input.value)
await server.send(payload)
console.log('Done!')
}
worker.js
self.addEventListener('message', function(e) {
const obj = e.data
self.postMessage(JSON.stringify(obj))
}, false)
Notes:
This works the following way:
Creates a list of 10 workers, and adds a few methods and props to them
We care about async .solve(Object): String which solves our tasks using promises while masking away callback hell
Use a new method: async jsonStringifyThreaded(Object): String which does the JSON.stringify asynchronously
We break the object into entries and solve each one parallelly (this can be optimized to be recursive to a certain depth, use best judgement :))
Processed chunks are cast into SerializedChunk which the JSON.stringify will use as is, and not try to process (since it has .toJSON())
Internally if the number of keys exceeds the workers, we round-robin back to the first worker and overschedule them (remember, they can handle queued tasks)
Optimizations
You may want to consider a few more things to improve performance:
Use of Transferable Objects which will decrease the overhead of passing objects to service workers significantly
Redesign jsonStringifyThreaded() to schedule more objects at deeper levels.
You can explore libraries like fast-json-stringify which use a template schema and use it while converting the json object, to boost the performance. Check the below article.
https://developpaper.com/how-to-improve-the-performance-of-json-stringify/
Related
As a corollary to Need C# JSImport async signature interop example to match async Javascript method - Blazor
I'd like to find a Span/ArraySegment solution to passing and receiving parameters via javascript interop in Blazor.
Here's my best attempt so far, with syntax I would expect to work:
The import
[JSImport("getMessage", "SampleJS")]
[return: JSMarshalAs<JSType.Promise<JSType.Any>>()]
internal static partial Task<object>
GetWelcomeMessage([JSMarshalAs<JSType.MemoryView>] ArraySegment<byte> bytes);
JS module
export async function getMessage(dataPointer) {
var maskedData = new Uint8Array(dataPointer) // this just creates a zero-filled array
console.log(maskedData)
return await new Promise((resolve) => {
setTimeout(() => {
resolve(maskedData);
}, 2000);
});
}
This just logs the data (bytes) received, waits 2s and returns the same data.
Usage
byte[] sampleData = Encoding.UTF8.GetBytes("Hello from C#");
ArraySegment<byte> sampleDataPointer = new ArraySegment<byte>(sampleData);
object? result = await GetWelcomeMessage(sampleDataPointer);
if (result is byte[] bytes)
{
message = Encoding.UTF8.GetString(bytes);
Console.WriteLine($"Got {message} from {result.GetType()}");
}
So the problem is in the javascript method. I can't turn the javascript's incoming MemoryView parameter into a Uint8Array so I can work with it. Any suggestions?
Many thanks to Mister Magoo for getting this off the ground
Here is the API of IMemoryView https://github.com/dotnet/runtime/blob/1631f312c6776c9e1d6aff0f13b3806f32bf250c/src/mono/wasm/runtime/dotnet.d.ts#L246-L264
The marshaler would GC pin it for you in case of ArraySegment but not for Span.
EDIT BELOW is from original poster so this answer gets the credit:
This answer eventually lead me to the solution. My sample code works if you modify the javascript line from
var maskedData = new Uint8Array(dataPointer)
to
var maskedData = new Uint8Array(dataPointer.slice())
Many thanks!
I'm working on a project that utilizes web workers. It seems that the workers are generating quite a bit of extra garbage that has to be collected from the message passing.
I'm sending three things to the worker via post message from the main thread. First is just a number, second is an array with 7 numbers, and 3rd is the date. The firs two are properties of an object as seen below. This is called every 16ms on RAF for about 20 objects. The GC ends up collecting 12MB every 2 seconds or so. I'm wondering if there is a way to do this without creating so much garbage? Thanks for any help!
//planet num (property of object) is just a number like: 1
//planetele looks like this (property of an object)
//[19.22942, 313.4868, 0.04441, 0.7726, 170.5310, 73.9893, 84.3234]
//date is just the date object
//posted to worker like so:
planetWorker.postMessage({
"planetnum": planet.num,
"planetele": planet.ele,
"date": datet
});
//the worker.js file uses that information to do calculations
//and sends back the planet number, with xyz coordinates. (4 numbers)
postMessage({data: {planetnum : planetnum, planetpos: planetpos}});
I tried two different avenues and ended up using a combination of them. First, before I sent some of the elements over I used JSON.stringify to convert them to strings, then JSON.parse to get them back once they were sent to the worker. For the array I ended up using transferable objects. Here is a simplified example of what I did:
var ast = [];
ast.elements = new Float64Array([0.3871, 252.2507, 0.20563, 7.005, 77.4548, 48.3305, 0.2408]);
ast.num = 1;
var astnumJ = JSON.stringify(ast.num); // Probably not needed, just example
// From main thread, post message to worker
asteroidWorker.postMessage({
"asteroidnum": astnumJ,
"asteroidele": ast.elements.buffer
},[ast.elements.buffer]);
This sends the array to the worker, it doesn't copy it, which reduces the garbage made. It is now not accessible in the main thread, so once the worker posts the message, you have to send the array back to the main thread or it wont be accessible as a property of ast anymore. In my case, because I have 20 - 30 ast objects, I need to make sure they all have their elements restored via post message before I call another update to them. I did this with a simple counter in a loop.
// In worker.js
asteroidele = new Float64Array(e.data.asteroidele); // cast to type
asteroidnum = JSON.parse(e.data.asteroidnum); // parse JSON
// Do calculations with this information in worker then return it to the main thread
// Post message from worker back to main
self.postMessage({
asteroidnum : asteroidnum,
asteroidpos : asteroidpos, // Calculated position with elements
asteroidele : asteroidele // Return the elements buffer back to main
});
// Main thread worker onmessage function
asteroidWorker.onmessage = function(e){
var data1 = e.data;
ast.ele = data1.asteroidele; // Restore elements back to ast object
}
Not sure this is the best approach yet, but it does work for sending the array to and from the worker without making a bunch of extra garbage. I think the best approach here will be to send the array to the worker and leave it there, then just return updated positions. Working on that still.
I have a file system watcher producing a Bacon.js event stream of changed file paths. I'd like to filter and debounce this stream so that each unique file path only appears in the output stream after 5 seconds of no activity for that unique value. I essentially want to write the following pseudocode:
var outputStream = inputStream.groupBy('.path',
function (groupedStream) { return groupedStream.debounce(5000); }
).merge();
I have a convoluted solution that involves creating a separate Bacon.Bus for each filtered stream, and creating a new Bus each time I encounter a new unique value. These are each debounced and plugged into an output Bus. Is there a better way? Would I be better off switching to RxJS and using its groupBy function?
It turns out Bacon.js recently added a groupBy function! I had been misled by searches that indicated it didn't exist. So this works for me:
var outputStream = inputStream.groupBy(function (item) { return item.path; })
.flatMap(function (groupedStream) { return groupedStream.debounce(5000); });
Edit: here's a simplified version based on OlliM's comment (kiitos!):
var outputStream = inputStream.groupBy('.path')
.flatMap(function (groupedStream) { return groupedStream.debounce(5000); });
I have a list of request to make to Parse.com using the swif API for tasks acumulated once the application was offine. Some tests show that if I dowload it all at once the overall time is slow that is I use multiple requests. However I couldn't figureout how can I request many "random" objectId's from Pase.com (I have a list of course, by random I mean out of order and not a fix number)
At the moment I am using a loop and calling many:
let pred = NSPredicate(format: "newDataID = %#, dataID[i])
query.findObjectsInBackgroundWithBlock { (result:[AnyObject]?, error:NSError?)
I was thinking in auto generate the string for the predicate but it can get very long what I image would make the query very slow.
Any ideas?
Under any circumstances, initiating many requests in a tight loop is ill-advised. Instead, send the dataID array to a cloud function. Also, if its really an array of object ids, then find is the wrong method, use get() instead...
var _ = require('underscore'); // underscore includes many handy functions, including map and toArray
Parse.Cloud.define("getManyObjectsById", function(request, response) {
var dataID = request.params.dataID;
var promises = _.map(dataID, function(anID) {
var query = new Parse.Query("MyCustomClassName");
return query.get(anID);
});
Parse.Promise.when(promises).then(function() {
response.success(_.toArray(arguments));
}, function(error) {
response.error(error);
});
});
Call it...
PFCloud.callFunctionInBackground("getManyObjectsById", withParameters: dataID) {
(objects: [AnyObject]?, error: NSError?) -> Void in
// objects should be an array of objects corresponding to the ids
}
This question already has answers here:
form serialize javascript (no framework)
(25 answers)
Closed 8 years ago.
For a lot of reasons (first of all: learning javascript), I need to serialize a form without jQuery, and send the resulting serialized data-structure to a php page with ajax.
The serialized data must be in JSON format.
How can I do that?
--EDIT--
this is how my form looks like: http://jsfiddle.net/XGD4X/
I am working on a similar problem, and I agree that it is worthwhile to learn how to program first without using a framework. I am using a data object (BP.reading) to hold the information, in my case a blood pressure reading. Then the JSON.stringify(dataObj) dose the work for you.
Here is the handler for the 'save' button click, which is a method on the dataObj. Note I am using a form instead of a table to input data, but the same idea should apply.
update: function () {
var arr = document.getElementById("BP_input_form").firstChild.elements,
request = JDK.makeAjaxPost(); // simple cross-browser httpxmlrequest with post headings preset
// gather the data and store in this data obj
this.name = arr[0].value.trim();
...
this.systolic = arr[3].value;
this.diastolic = arr[4].value;
// still testing so just put server message on page
request.callback = function (text) {
msgDiv.innerHTML += 'server said ' + text;
};
//
request.call("BP_update_server.php", JSON.stringify(this));
}
I hope this is helpful
* edit to show generic version *
In my program, I am using objects to send, receive, display, and input the same kind of data, so I already have objects ready. For a quicker solution you can just use a empty object and add the data to it. If the data is a set of the same type of data then just use an array. However, with a object you have useful names on the server side. Here is a more generic version untested, but passed jslint.
function postUsingJSON() {
// collect elements that hold data on the page, here I have an array
var elms = document.getElementById('parent_id').elements,
// create a post request object
// JDK is a namespace I use for helper function I intend to use in other
// programs or that i use over and over
// makeAjaxPost returns a request object with post header prefilled
req = JDK.makeAjaxPost(),
// create object to hold the data, or use one you have already
dataObj = {}, // empty object or use array dataArray = []
n = elms.length - 1; // last field in form
// next add the data to the object, trim whitespace
// use meaningful names here to make it easy on the server side
dataObj.dataFromField0 = elms[0].value.trim(); // dataArray[0] =
// ....
dataObj.dataFromFieldn = elms[n].value;
// define a callback method on post to use the server response
req.callback = function (text) {
// ...
};
// JDK.makeAjaxPost.call(ULR, data)
req.call('handle_post_on_server.php', JSON.stringify(dataObj));
}
Good Luck.
CoffeeScript implementation returning a GET query string:
serialize = (form) ->
enabled = [].filter.call form.elements, (node) -> not node.disabled
pairs = [].map.call enabled, (node) ->
encoded = [node.name, node.value].map(encodeURIComponent)
encoded.join '='
pairs.join '&'
Or if you rather prefer a key-value map:
serialize = (form) ->
data = {}
for node in form.elements when not node.disabled and node.name
data[node.name] = node.value
data
I haven't looked at jQuery's implementation, so no 100% compatibility guaranteed.