I'm trying to save a 2gb variable into a file.This variable comes from webtorrent. I tried to use the blob url , but it crashes as it is bigger than max blob size. I also tried StreamSaver.js and got 2 different problems. By writting periodically while I download the variable, streamSaver fails at some point (no idea why, it writes like 500 mb then crashes). By writting it all at the end it obviously crashes the tab. I even tried to write it at the end but chunks by chunks but nothing seems to get this working. So I was wondering if you guys knew how I could save this variable to a file (any API, tricks, I'll take anything)
Related
The R rgl package exports an HTML widget with the rglwidget() function, built using the htmlwidgets package. Often the data for this widget is quite large, and Pandoc and webshot2 choke on it.
I would like to try compressing the data when the HTML page is created, and uncompressing it in Javascript before display. I can see that there's a Javascript package pako that appears to do what I want, and it can be "browserified", but I can't see how to make it available to rglwidget(). Can anyone describe what's necessary?
Edited to add some more detail in response to a comment: This needs to happen on the server (or actually, even before the file gets to the server). htmlwidgets produces output in a Markdown document that is converted to HTML by Pandoc, and that step is failing because Pandoc chokes on the large JSON datasets. I'm not sure if it's just the size, or the complexity (I think Pandoc parses the JSON, and appears to blow up to a huge memory footprint before it crashes). I'm hoping that by using a base64 blob Pandoc will handle it better. webshot2 also has problems that may be the same.
2nd Edit: I've got some evidence that it's the size that matters, not the complexity. I used base64 encoding on the JSON to make it simpler (just one long string), but 33% bigger, and things were worse. So I'm back to thinking that compression would help.
I want to get the screenshots from PageSpeed Insights. Using the API, I used a code that i founded here : https://embed.plnkr.co/plunk/c7fAFx, but doesn't work.
please help me! I am learning to code.
Why doesn't the linked code work?
Well because it is ancient and attempting to use the version 1 Page Speed Insights API.
It is currently on version 5 so that is why it does not work, v1 no longer exists as a public API.
How to recreate the functionality of this App?
As you are learning to code I will lay out the steps for you and then you can research how to do each step and use that to learn.
I will warn you as a beginner there is a lot to learn here. However on the flip side if you manage to work out how to do the below you will have a good first project that has covered multiple areas of JS development.
As you have marked this "JavaScript" I have assumed you want to do this in the browser.
This is fine up until the point where you want to save the images as you will have to work out how to ZIP them which is probably the most difficult part.
I have highlighted the steps you need to learn / implement in bold
1. First call the API:
The current URL for Page Speed Insights API is:
https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=https://yoursite.com
Just change url=https://yoursite.com to any site you want to gather the images from.
For a small amount of requests a day you do not need to worry about an API key.
However if you do already have an API key just add &key=yourAPIKey to the end of the URL (replacing the yourAPIKey part obviously :-P).
You want to make an AJAX call to the API URL first.
2. Parse the response
Then when you get a response you are going to get a large JSON response.
You need to parse the JSON response and turn it into a JavaScript Object or Array you can work with.
3. Find the relevant parts
So once you have a JavaScript Object you can work with you are looking for "final-screenshot" and "screenshot-thumbnails".
These are located under "audits".
So for example if you parsed to an array called lighthouseResults you would be looking for lighthouseResults['audits']['final-screenshot'] or lighthouseResults['audits']['screenshot-thumbnails']
"final-screenshot" contains how the site looked after it was loaded, so if you just want that you want this element.
This contains an image that is base64 encoded (lighthouseResults['audits']['final-screenshot']['details']['data']).
"screenshot-thumbnails" is the part you want if you want the "filmstrip" of how the site loads over time. This contains a list of the thumbnails base64 encoded.
To access each of these you need to loop over each of the items located at lighthouseResults['audits']['screenshot-thumbnails']['details']['items'] and return the ['data'] part for each ['item']
Find the parts that you want and store them to a variable
4a. Decode the image(s)
Once you have the image(s) in a variable, you will have them as a base64 encoded string at the moment. You need to convert these into usable jpg images.
To do this you need to base64 decode each image.
For now I would just display them in the browser once they are decoded.
learn how to decode a base64 encoded image
4b. Alternative to decoding the image
As the images are base64 encoded they can be displayed directly in a browser without decoding first.
You can just add an image where the src your base64 image string you gathered in step 3.
If you just want to display the images this is much easier.
Add images to the screen and set the src to the base64 image string you have from step 3
Saving the images
Now you said in a comment you want to save the images. Although this can be done via JavaScript it is probably a little advanced for starting out.
If you want to save the images you really want to be doing that server side.
However if you do want to download the images (filmstrip) in the browser then you want to look into a zip utility such as jszip.js.
The beauty of this is they normally want you to convert the images to base64 first before zipping them, so it may not actually be that difficult!
$(function(){
$("#testdiv").load("sometext.txt",function(){
alert("Succes");
});
});
I get the alert but I am not able to see anything in the "#testdiv".The file,sometext.txt is in the same folder.I tried giving it absolute path but i get 404 ,url not found on this server.
What am I doing wrong here?
Windows has several "features" that are built to make their operating system more usable for most people. Hiding the file extension is one of those "features" that often do more harm than how much they help.
Though, as you might know, a file doesn't need a file extension, the first few bytes of the file are used to tell the OS what file type it is, the extension just makes it easy.
When using windows to develop anything, you probably want to see the actual name of your files, so make sure to check out one of the guides on how to enable displaying the file extension, like this one.
If you can't GET a file with JS always double, or better even tripple check, that the file can be accessed through the browser first.
I have a pretty run-of-the-mill node.js server with expressjs installed locally for development purposes; I store various files and request them via a basic HTTP call that returns the file through express' res.download feature. Most of the time, this works without a hitch. For a very small subset of files, however, the end-user receives a file that is much larger than expected (almost 2x) and is unreadable by any conventional viewer. Out of maybe a hundred files, this has only happened twice, and both were JPG files, but the sample is too small to draw any conclusion. What I know :
The issue is reproductible : if it happens with a file, it always happens;
The issue is not related to the way files are stored : if I swap the problematic file with another one but keep everything else the same (name, location, etc.), it works fine;
Right before the res.download happens, the file is okay : checking its size with fs.stats returns the correct value
The HTTP response encounters no visible problem : no error, 200 response code...
The source file seems to have normal metadata and JPG markers
UPDATE I did some tests, and the issue seems to be somehow related to encoding : the mangled response file is, for reasons unknown, encoded in UTF-8 ; the size discrepancy comes from all non-UTF-8 characters being replaced by EF BF BD (the unknown character symbol) ! I still can't understand why it happens, what makes these few files different from others, and if it can be detected and/or corrected upstream.
UPDATE 2 After some additional tests, I still can't quite pinpoint the cause, but I can add the following info :
Systems-wise, the issue happens during the data streaming in fs.js
The root cause is located somewhere in the EXIF data of the image
For those interested, the source image (source.JPG) and download result (response.JPG) can be found here : http://www.sycomor.fr/test/ ; I also added a similar image that isn't affected by the download and comes out clean. For what it's worth, both pictures were taken minutes apart, with the same camera at the same settings, so I strongly doubt the issue is caused by some external source.
Thanks !
Your issue comes from 'connect-livereload' in your express configuration.
It corrupts binary stream while injecting the reloading script.
Refer to https://github.com/intesso/connect-livereload/issues/39 for details.
I am making a game using only HTML5 and Javascript. I want to take an MP3 file the user selects, and split into few second long chunks. Is it possible to do this in Javascript, and if so, how?
If you need a client side solution, but are not bound to Javascript, you may want to give Flash a try. There is many excellent solutions for editing in Flex/Actionscript. Here is one tutorial I came across after a quick google: http://scriptplayground.com/tutorials/as/Play-Sound-Samples-From-One-File-Using-ActionScript-3/
However, if you are dead set on using Javascript, try following these steps:
Check out this fiddle: http://jsfiddle.net/andrewodri/PKPD6/. It is using the HTML5 audio element to play a sound, but the interesting thing is that this data is encoded as a data URI. Whatever encoded data we throw at the source, it will be able to play.
Read up on the W3C File API specification, and especially the FileReader interface. The is support for this in Chrome and Firefox at least, although I am not too sure how comprehensive it is. You will notice that you are able to read the binary data into a data URI, which you can 1) manipulate as a string, and 2) send to the src attribute of the audio element.
This is going to be the hardest part... You can start by getting familiar with the structure of an MP3 file; ie. understanding how big the ID3 tag is and how to identify it, how big frames are and where you can chop them off, and how the file is closed. The will help you extract relevant information, but also figure out where you can start chopping. I put some references at the bottom. A lot of similar code has been created for this in Actionscript which may be a good reference.
If you have enough patience, you can do the following:
Retrieve the binary data of the uploaded file, and convert that into a data URI
Determine how big the ID3 tag is when encoded in base64 (base64 represents 6 bits with 1 character), and save that code to variable
Determine how big each audio frame is, and build an array of all the audio frames
Determine how the file is closed, and save that code into a variable
Stitch it all together however you like, with the ID3 at the front, your selected frames in the middle, and file closure code
Send it to the src attribute of the audio element
I would conclude by saying that it would be far, far easier to do this in Flash, or even to send it to the server as mentioned in other comments. But if you have the justification for doing it all in Javascript, this is definitely possible in most modern browsers.
Additional References:
http://www.html5rocks.com/en/tutorials/file/dndfiles/
http://www.mpgedit.org/mpgedit/mpeg_format/MP3Format.html
http://www.mpgedit.org/mpgedit/mpeg_format/mpeghdr.htm
http://www.aardwulf.com/tutor/base64/base64.pdf
http://en.wikibooks.org/wiki/Algorithm_Implementation/Miscellaneous/Base64