Can't seem to find anything related to gzip compressing a string. Only find broken sites or suggestions for compressions that won't work as gzip. Also lots of talk of server side implementation. However I wish to send the encoded data from the client.
To give clarification all my clients use greasemonkey or scriptish and all my clients are generally on some version of a recent Firefox or one of it's derivatives, so content encoding for everyone is not an issue.
What I do need is a pure javascript or some sort of library loadable by javascript to gzip compress a string.
Just achieved this using https://github.com/dankogai/js-deflate However the postdata for whatever reason will strip the + signs and replace them with spaces.
To send the data via javascript:
params.mapdata= btoa(RawDeflate.deflate(JSON.stringify(mapdata)));
To receive the data via php:
$value = gzinflate(base64_decode(preg_replace('/\s/', '+',$value)));
Related
I want to get text from a site using Python.
But the site uses JavaScript and the requests package to receive only JavaScript code.
Is there a way to get text without using Selenium?
import requests as r
a=r.get('https://aparat.com/').text
If the site loads content using javascript then the javascript has to be run in order to get the content. I ran into this issue a while back when I did some web scraping, and ended up using Selenium. Yes its slower than BeautifulSoup but it's the easiest solution.
If you know how the server works you could send a request and it should return with content of some kind (whether that be html, json, etc)
Edit: Load the developer tools, go to network tab and refresh the page. Look for an XHR request and the URL it uses. You may be able to use this data for your needs.
For example I found these URLs:
https://www.aparat.com/api/fa/v1/etc/page/config/mode/full
https://www.aparat.com/api/fa/v1/video/video/list/tagid/1?next=1
If you navigate to these in your browser you will notice JSON content, you might be able to use this. I think some of the text is encoded in Unicode e.g \u062e\u0644\u0627\u0635\u0647 \u0628\u0627\u0632\u06cc -> خلاصه بازی
I don't know the specific python implementation you might use. Look for libs that support making http requests and recieving data. That way you can avoid selenium. But you must know the URL's beforehand. Like shown above.
For example this is what I would do:
Make a http request to the URL you find in developer tools
With JSON content, use a JSON parser to get a table/array/dictionary natively. You can then traverse this in the native programming language.
Use a unicode decoder to get the text in normal text format, there might be a lib to do this, but for example on this website using the "Decode/Unescape Unicode Entities" I was able to get the text.
I hope this helps.
Sample code:
import requests;
req = requests.get('https://www.aparat.com/api/fa/v1/video/video/show/videohash/IueKs?pr=1&mf=1&referer=direct')
res = req.json()
#do stuff with res
print(res)
When i am sending binary data with UTF-8 encoded from PHP client to Node.JS server, Node.JS internally encodes this data to URL Encode(percentage encoding). I guess Node.JS doesn't have support for UTF-8 as i already checked with base64 and it was working fine. I googled a lot and i found that everyone is facing the same issue. I manually checked the string but I get "URIError: URI malformed" error.
I also use the deprecated functions unescape, escape, encodeURI and decodeURI.
Little background of my work:
console.log(decodeURIComponent('%B1');
As you can search from the URLEncoding list that %B1 is for "±" sign. But i am getting the above mentioned error. This function can't decode other many special characters too. I don't know why Node.JS doesn't have support for the standard decoding style like UTF-8.
Please help me guys.
I have pretty standard javascript/XHR drag-and-drop file upload code, and just came across an unfortunate real-world snag. I have a file on my (Win7) desktop called "TEST-é-TEST.txt". In Chrome (30.0.1599.69), it arrives at the server with filename in UTF-8, which works out fine. In Firefox (24.0), the filename seems mangled when it arrives at the server.
I didn't trust what Firebug/Chrome might be telling me about the encoding, so I examined the hex of the request packet. Everything else is the same except the non-ASCII character is indeed being encoded differently in the two browsers:
Chrome: C3 A9 (this is the expected UTF-8 for that character)
Firefox: EF BF BD (UTF-8 "replacement character"?!)
Is this a Firefox bug? I tried renaming the file, replacing the é with ó, and the Firefox hex was the same... so such a mangle really seems like a browser bug. (If Firefox were confusedly sending along ISO-8859-1, for example, without touching it, I'd see an E9 byte, and I could handle that on the server side, but it shouldn't mangle it!)
Regardless of the reason, is there something I can do on either the client or server sides to correct for this? If a replacement character is indeed being sent to the server, then it would seem unrecoverable there, so I almost certainly need to do it on the client side.
And yes, the page on which this code exists has charset=utf-8, and Firefox confirms that it perceives the page as UTF-8 under View>Character Encoding.
Furthermore, if I dump the filename to console.log, it appears fine there--I guess it's just getting mangled in/after setRequestHeader("X-File-Name",file.name).
Finally, it would seem that the value passed to setRequestHeader() should be able to have code points up to U+00FF, so U+00E9 (é) and U+00F3 (ó) shouldn't cause a problem, though higher codes could trigger a SyntaxError: http://www.w3.org/TR/XMLHttpRequest2/#the-setrequestheader-method
Thanks so much for Boris's help. Here's a summary of what I discovered through our interactions in comments:
1) The core issue is that HTTP Request headers are supposed to be ISO-8859-1. Prior versions of Chrome and Firefox both passed along UTF-8 strings unchanged in setRequestHeader() calls. This changed in FF24.0 (and apparently will be changing in Chrome soon too), such that FF drops high bytes and passes along only the low byte for each character. In the example I gave in the question, this was recoverable, but characters with higher codes could be mangled irretrievably.
2) One workaround would be to encode on the client side, e.g.:
setRequestHeader('X-File-Name',encodeURIComponent(filename))
and then decode on the server side, e.g. in PHP:
$filename=rawurldecode($_SERVER['HTTP_X_FILE_NAME'])
3) Note that this is only problematic because my ajax file upload approach is to send the raw file data in the request body, so I need to send the filename via a custom request header (as shown in many tutorials online). If I used FormData instead, I wouldn't have to worry about this. I believe if you want solid, standards-based unicode filename support, you should use FormData and not the request header approach.
I am integrating yahoo open ID for my site. My site is running for different languages( en, jp, cn etc.). When I am trying to logged in from english site with yahoo opend ID then its fine but when the same this I tried from japanese or chinese site then it's not redirecting me to yahoo open id login.
Each time I am getting the below error from javascript
Error: The character encoding of the plain text document was not
declared. The document will render with garbled text in some browser
configurations if the document contains characters from outside the
US-ASCII range. The character encoding of the file needs to be
declared in the transfer protocol or file needs to use a byte order
mark as an encoding signature. Source File:
http://uatstorefrontjpcr.mobi-book.com/ReturnFromSocial/LogOnYahoo
Line: 0
Can anyone suggest what to do.
I have used SocialAuth-net.dll for this purpose. I have set all required wrapper in web.config. Same coding is okay with Google and facebook open ID.
Your web server is probably sending back a response with a Content-Type of text/plain. When a web browser receives a response with that content type, it doesn't know what encoding should be used to decode it; since you haven't told it how to decode it, different browsers might choose different ways.
The solution is to provide an explicit encoding. For example, if you know that the text is UTF-8 encoded, then you could provide it in a header like so:
Content-Type: text/plain; charset=UTF-8
According to an informational page from the W3C, you can get ASP.Net to include that bit in the header using Response.ContentEncoding. Again using UTF-8 as an example, you can set it like so:
Response.ContentEncoding = Encoding.UTF8;
I have a simple Android App using simple js, jquery mobile and phonegap that has form data that I would like to write to CSV on /sdcard. I can not figure out the best way to do this. It needs to write locally to CSV since the device will not have internet access. Each time the form is filed out, each CSV file name must be unique, containing 'CompanyName' field. I have searched everywhere and cannot figure this out. Help Please...
I have implemented similar things (read/write data via mobile web-app with no internet connection).
You might be able to utilize the same approach that I took which is using custom ContentProvider in Android to save your data. Provided the data is not large, this approach should work. If that is the case then you could do the following:
Create custom content URI scheme such as content://myuri. This is a neat feature of Android since I could call this URI from the built-in browser including even using links in your HTML page. It basically treat this URI as any other http:// uri
In the implementation for writing you could devise the URL as content://myuri/writeData/input1=value1/input2=value2/input3=value3 and so on. I am not sure if there is a limitation in ContentProvider for the length of URI accepted by the ContentProvider which is why I put the caveat above. I don't have a large form when I tried this approach last time
In the implementation of your extension of ContentProvider, you need to override the function public ParcelFileDescriptor openFile (Uri uri, String mode) and have them interpret the URI that is given in step #2. The implementation should be parsing the URI and then write the data to the CSV as you intended
You can get this data back using different URI scheme such as content://myuri/getData/[theFileID] which now in the implementation of #3 above, you will do the reverse and return the CSV data to your mobile web-app
Let me know if you have any question.