The issue:
I have written a ton of code (to automate some pretty laborious tasks online), and have used the mechanize library for Python to handle network requests. It is working very well, except now I have encountered a page which I need javascript functionality... mechanize does not handle javascript.
Proposed Solution:
I am using PyQt to write the GUI for this app, and it comes packaged with QtWebKit, which DOES handle javascript. I want to use QtWebKit to evaluate the javascript on the page that I am stuck on, and the easiest way of doing this would be to transfer my web session from mechanize over to QtWebKit.
I DO NOT want to use PhantomJS, Selenium, or QtWebKit for the entirety of my web requests; I 100% want to keep mechanize for this purpose. I'm wondering how I might be able to transfer my logged in session from mechanize to QtWebKit.
Would this work?
Transfer all cookies from mechanize to QtWebView
Transfer the values of all state variables (like _VIEWSTATE, etc.) from mechanize to QWebView (the page is an ASP.net page...)
Change the User-Agent header of QWebView to be identical to mechanize...
I don't really see how I could make the two "browsers" appear more identical to the server... would this work? Thanks!
Since nobody answered, I will post my work-around.
Basically, wanted to "transfer" my session from Mechanize (the python module) to the QtWebKits QWebView (PyQt4 module) because the vast majority of my project was automated headless, but I had encountered a road block where I had no choice but to have the user manually enter data into a possible resulting page (as the form was different each time depending on circumstances).
Instead of transferring sessions, I met this requirement by utilizing QWebViews javascript functionality. My method went like this:
Load page in Mechanize, and save the downloaded HTML to a local temporary file.
Load this local file in QWebView.
The user can now enter required data into the local copy of this page.
Locate the form fields on this page, and pull the data the user entered using javascript. You can do this by getting the main frame object for the page (QWebView->Page()->MainFrame()), and then evaluating javascript code to accomplish the above task (use evaluateJavaScript()).
Take the data you have extracted from the form fields, and use it to submit the form with the connection you still have open with mechanize.
That's it! A bit of a work-around, but it works none-the-less :\
Related
I have an html page that has a form and is communicating to a server via JSON requests, when loading (e.g. in order to remember user's previous answers) and when submit button is pressed (e.g. in order to save his new answers). Every user has a specific user_id that is located as an attribute in the url of the website. I have only HTML pages, CSS and Javascript that makes some simple functions, as well as received and sends the requests.
The problem is that the server needs an api-key for the request to happen, that I need to be kept hidden, but instead is easily discovered when the user sees the source code. I want this specific line to be hidden, but I guess this is not possible without any backend in the game.
I would like to know the easiest and fastest way to get from this state (just frontend, where every piece of information in the source code is totally insecure) to another where the api-key (at least) is not on the open.
I guess the solution is to use a server for that part but I need suggestion on the easiest transition from my code to another. I tried to use GWT, as I am a bit more familiar with JAVA backend application (but not with GWT), but seems like a big trouble, where I need to change my HTML, my Javascript and also the CSS that I have may not be useful, as well as I face a lot of problems when trying to read my parameters.
I know that it is not the best way but I do not have a lot of time to make it work, so sorry if it seems lazy (I am new to this type of programming), but I haven't found anything helpful and I cannot study for 2 weeks in order to be able to begin implementing it.
Is node.js (which is Javascript and I already have implemented the request sending/receiving in this language) easier than GWT for that matter? Will my sensitive data be secure in that way? I would be grateful if there was a similar sample, that I could start using for my implementation, as I haven't find anything that is specifically helpful for my situation.
Thanks in advance!
NodeJs is not javascript, NodeJs is specific javascript "interpreter" whose is purpose is mainly to be executed server-side. If you have an HTML page, it is likely to be loaded in a web browser (client-side), so not in a NodeJs environnement.
Everything that is hard-coded in the javascript of you web page is visible from the client, there is no way around that. So yes, you need some server-ish thing somewhere.
If you are not to manage a server by yourself or via PaaS, you can go for a serverless architecture. For instance, If you consider AWS which I know the most, you can easilly add some user management to your web page using Aws Cognito User Pool. Once a user is connected and have the good permission, he can be granted access to some other resources via a JWT token that you send along with your request.
I've managed to scrape websites that require no login using js only and a little help from websites that allow me to pass the CORS issues(like allorigins), but I just couldn't manage to get pass through the login problem.
I've seen many posts discussing of doing it using node.js and python beautifulsoup, but none on how to do it with javascript.
So how do I go about it?
Is it even possible doing it purely on client-side?
I'm willing to do all the learning and searching needed, but I need some direction in this vast subject.
Assuming you meant using in-browser JavaScript, how did you get around CORS? And if you did, then once the page refreshed after a successful login you code would stop running anyway unless you were a browser extension.
If you mean on your computer, then Node is what you're looking for, but unless you use a project like Headless Chrome then you'll run into the issue of saving the cookies between requests which is what keeps track of your session and actually keeps you logged in.
Login requires a direct interaction with your browser, like saving a cookie, returning a security token etc.
If you use JavaScript from a html page, it would theoretically require to visit the other page, at least inside a iFrame. There is a limit of how much you can do with javascript inside a iFrame.
With other words you try to imitate something like Selenium. Give it a try. It works with Java. You can control you browser, telling what to do, like a real user, and fetch the results, make even screenshots.
I am working on a Web Application developed & hosted on Domino R9 Server (Non Xpages Web Application). Each form has WebQueryOpen and WebQuerySave events implemented for the server side business logic. Problem is that on every page, when the page refreshes from the browser, application's home page/form is loaded. I want to maintain the state of a page in this application.
Moreover, is there any possibility of implementing an Ajax based session handling (backend) in case the page was idle for sometime and session expired. For example if a user was writing something in the Web Editor (implemented in CKEditor latest version) and if his session is automatically dropped he should be able to start with least hassle and his already written stuff may not be lost.
My suggestion is to not use WebQueryOpen and WebQuerySave. Instead write the application to use modern web technologies (Ajax, JSON, REST API), then you don't need to save and reopen the page all the time. Doing that will give you everything you want.
I would build the page using standard HTML and Javascript (or even jQuery), then make Ajax calls to agents on the server to read/write data.
I have given several presentations on exactly this, with plenty of code samples and even finished code to download:
http://blog.texasswede.com/mwlug-2015/
http://blog.texasswede.com/my-mwlug-presentation-2/
http://blog.texasswede.com/my-connect-2017-demo-code/
You can save the contents from all fields and CKEditor in a JSON string and also the UNID (or any "primary key") to the current entity (=page).
Afterwards, you can send this string to the server using AJAX, or as an alternative...
... have you tried using HTML5 local storage? See this page:
https://www.w3schools.com/html/html5_webstorage.asp
Tell us how do you continue!!!
I am at the stage where I am thinking of integrating a social login method into my site. Of course my first one will be the facebook login before moving onwards.
Basically my question is : Which language is best for this type of OAuth connection, and which is going to let me do everything I want?
This will encompass all OAuth connections in general eventually, but specifically Facebook for now.
From what I have read of the documentation, the JavaScript SDK allows you to login, and connect to the open graph API - which in turn will allow me to post / upload etc etc. This is also available in PHP.
From experience which is the more durable route to go?
PHP or JAVASCRIPT
The best and recommend way to authorize users is the JavaScript SDK (FB.login). No redirect needed (better usability), very easy to handle, no PHP needed (the new PHP SDK needs PHP 5.4+). Use PHP only for stuff that involves usage of the App Secret or Extended Access Tokens. And for cron jobs, obviously. You can even just forget about the PHP SDK and use your own CURL calls.
Btw, security is no problem, you should just activate "appsecret_proof" in the App Settings.
More information about appsecret_proof:
https://developers.facebook.com/docs/graph-api/securing-requests
http://www.devils-heaven.com/facebook-php-sdk-4-0-tutorial/
One more reason (and one of the most important ones) is the possibility to refresh Access Tokens (=User Sessions) easily without page refresh by using FB.getLoginStatus.
And another reason is that you need to upgrade to new PHP SDK versions on your own. The JavaScript SDK does not need any upgrades, in the lase few years you only had to change one or two parameters in some cases, the SDK gets downloaded from the Facebook servers.
Also, if you add Social Plugins, you need the JavaScript SDK anyway.
TL;DR PHP/both
I'd really recommend PHP. You'll want to store the login in database. If you do it via JS, you'll need to make an Ajax call to the server, which is not really that secure.
Having said that, they are targeted for different uses. JS is for frontend more, while PHP is for backend (db storage, checks, actual site login, etc). Using JS will let you generate the buttons on the fly, while using PHP you'll need to do some more coding.
Somehting else, the php library will get updated from time to time and you'll need to keep up after testing. JS also, but it's easier, since the code usually works.
If you only want one, use PHP. You can control what the code does and JS will not break your site since it's written by you. However, I'd recommend using both since you will probably want more than just simple login
Edit:
As facebook states, use PHP SDK: Usually this means you're developing with PHP for a Facebook Canvas app, building your own website, or adding server-side functionality to an app that already uses the Facebook SDK for JavaScript.
https://developers.facebook.com/docs/reference/php/4.0.0
The best route would be to use both, together. Some users might have javascript disabled or you might do something within your javascript code that will not work on some browsers. So as a fallback method you can use php api.
Going with only php would be solid and will work regardless of what the users client is, but you can make the user experience better with javascript.
Still, in most cases you'll end up having to use both.
I have just spent the afternoon playing with the JavaScript SDK for facebook and I have to admit I think this is going to be the best option for what I need.
If anyone else is reading this, it may not be perfect for you - but with the way my application has been built I think it is a perfect fit. Here is why :
My standard login system uses JavaScript to grab my form data, then validate, which then passes the validated data via AJAX to a PHP validation script. Which in turn returns a JSON response to the original AJAX call. If my call comes back with "ok" : true then we are good to go basically.
The way the facebook JavaScript SDK works is almost a perfect little jigsaw puzzle to bolt onto the system I am using. All I require is a little bit of profile data, to then keep a record of this user on my system. THIS is provided by facebook, then validated by myself.
However, other social network logins may not be as nice and simple to use as the facebook API, so I could end up changing my mind on the overall system. For now just using facebook, I think the JavaScript SDK is absolutely spot on, as it just gets the information for you to run through your own validation on site. As I said this is a perfect fit for my system however it may not be for yours.
I'm working on a website that is going to be offline. All the html files will be in a folder stored on the hard-disc. I've managed to do 90% of the work and the last part I have no idea of. Here is what it is:
I have stored a list of products in the localStorage as various strings under the keys - like buying objects and it goes to the cart, the cart objects are in localStorage. I created a page that showed the list of all the products in the localStorage. It can be cleared if the user clears them. Now I need to create a page where all the objects that was selected before, regardless of the localStorage being cleared, show as list in this page. You can take it as the page that lists products that have been ordered in the past, i.e even after the cart is cleared the products will show in the past-orders page.
I do not know any server side codes, I did everything using JavaScript as it was supposed to be a simple project, but I'm stuck at this part. So I cannot use PHP or anything to generate files or use a database to store stuff.
Here's what I thought but I don't think it works but wanted to confirm if it does or not:
Generate an XML file or a .txt file and store it in the drive and then clear the localStorage. But I don't think it is possible. If its possible just using JavaScript please point me in the right direction and I'll research and come up with something.
P.S. the website will be entirely offline what I mean is the users will never connect to the internet for this to work. Also there won't be a server or localhost.
Thank you!
The site is completely offline, but functionality is similar to an eCommerce site. You click a button and some content from the website stores in the localStorage and I have to call it in multiple pages, when a user clicks another button, localStorage clears but whatever was selected before must be available without localStorage. Hmmmm.. Consider a quiz site where you answer everything and when you take a new quiz, old scores will be stored somewhere else and it won't change when you take a new test.
Is it possible to attain this functionality without a server side script? It seems the final-targeted-users won't know how to install any softwares that can provide a localhost or a server or something like that.
Client-side, browser's JavaScript runtimes don't have local file system access excepting a very limited one when uploading files to a remote server, and anyway, browers won't give you direct access to the file.
You'll need some kind of server-side logic behind the scenes if you want full I/O in the local file system.
Perhaps you can take a look at NodeJS, a server-side JavaScript runtime that can work as lighty Web server on either small and large projects, and it has built-in server-side JavaScript I/O functions.
Otherwise, you're absolutely stuck in the client-side Web browser's sandbox limitations.
U can refer documents of knockoutjs and NodeJS.. That would probablky help... A far as my knowledger is concerned NodeJS does contain a way to handle your problem.