External API access with Ruby blocked by CloudFlare - javascript

I am trying to access the HaveIBeenPwned web API for breached sites and emails, but I am being blocked by cloudflare's anti-DDoS protection. I've found that there are ways to get around this with Python and JavaScript, but I haven't been successful with my React/Rails app.
This post has the following quote: "Currently, they check if the client supports JavaScript, which can be spoofed." I haven't been able to find any other documentation of this behavior.
I need this information in the backend, so calling HIBP from the frontend is not ideal. Any idea how to hit the API from Rails?

If they want it to be used on frontend only, using it on backend can be tricky. You will need to create your own wrapper with, for example, puppeteer. And from Rails side execute command that will do some work in background. But keep in mind, it doesn't work very fast (can take up to 5 seconds per request), and it will block your Rails process.
I would start with single node.js app that will accept command-line parameters. Ruby isn't very good at advanced web scraping, so there are no any gentle solutions. Also, keep in mind that you don't have any guarantees. One day it can just stop working.

Related

How to Communicate Between Web App and C Sharp

I'm attempting to make a Web app that needs to communicate to a program written in C Sharp. But I can't find a good form of communication. What I need is if a user clicks something on the Web app, it will notify the C Sharp program. Also, if an event happens on the C Sharp program, it needs to alert the Web app. Both of these are going to be running on the same machine.
Right now I'm mainly focusing on the C Sharp program just periodically "asking" what the status of the Web app is.
I've tried using POST requests to the Web app and I've had a bit of success with this but I don't know how to essentially store and update a "status" on the Web App. For example, C Sharp program sends a POST/GET request asking for the status, the Web app responds with "nothing has changed" or some sort of status code. I don't know how to keep track of that status.
I've attempted using Web Sockets but I don't think it is going to be possible on the C Sharp side. However, I'm definitely open to suggestions on how this might work.
I've looked into using the ReST architectural style but I'm having a hard time understanding how I would implement it. I'm using mainly AJAX on an apache server and most of the ReST examples I saw used IIS.
One way I've been successful with this is a horrible workaround. I use 3 files to store contents, status of Web app, and status of C Sharp program. But this requires me constantly fetching files, reading them, writing a new one, and uploading it.
Sorry if this question is poorly formed, I'm obviously new to a lot of this and this is my first SO post. Also, I'd include example code but I'm posting this from my tablet so it's not accessible right now.
If they are on the same machine, you can use 'pipes' (Unix), local sockets or file handlers.
These are all types of IO objects both applications can 'listen' to without exposing themselves to the network and without blocking while they are 'waiting' for data..
... But this will limit your scalability.
Another option is to use a Pub/Sub service such as Redis. This is a better option than websockets, because you can have multiple C# apps listening to multiple web apps on a centralized data source.
It uses the same IO concept (using sockets) behind an abstraction layer coupled with a database - it's very effective.
Good luck!
I implemented something similar to this. I needed a desktop application to listen for api calls from the browser. I ultimately decided to implement a "web connector" which can either be created as part of the application OR installed as a service.
Here is a good example: https://msdn.microsoft.com/en-us/library/system.net.sockets.tcplistener(v=vs.110).aspx

Access nodejs with web javascript but hitting XSS?

I have a nodejs app that runs a web server and works as a web API. Simple GETs to interact with it. My goal is to be as accessible as possible by any language or programmatic scenario. My biggest problem is that javascript run in the browser can't hit it because the browser (specifically chrome) prevent cross site scripting. I'm open to any ideas that allow this. I want any site to be able to make requests against the url, sort of like how twitter has a javascript API.
I've tried using jQuery's ajax with JSONP but I was having all sorts of problems. Either it wouldn't go through or if it did go through I wouldn't get the response.
If there's a pure javascript way, I'd prefer that because of it having fewer dependencies.

OAuth, javascript and many URI's

I'm trying to make a program that can be hosted by many peoples, like an app.
The app use a REST API, so I must authenticate with Oauth,
and because anyone should be able to host the program, the redirect URI cannot be static.
Further, I don't want to use any server-side processing, which means only javascript for me.
Is it even possible to make a secure and working solution with non-static redirect URI,
and only using javascript, to work in a normal webbrowser?
So you use the information provided in the request to your app to indicate the URL for your app. For instance, if the request came to http://example.com/path/to/app and you knew in your app that /to/app was part of your routing infrastructure, then the path to your app is http://example.com/path/.
That is how I would determine it, using a serverside language.
Using a javascript library, which would be loaded from the server, I would either determine it like the above, or I would just hard code it on the generation of the javascript file (when you tell people where to download the javascript, it can use a form that requires their web address first).

Securing AJAX API

I have an API (1) on which I have build an web application with its own AJAX API (2). The reason for this is not to expose the source API.
However, the web application uses AJAX (through JQuery) go get new data from its AJAX API, the data retrieved is currently in XML.
Lately I have secured the main API (1) with an authorization algorithm. However, I would like to secure the web application as well so it cannot be parsed. Currently it is being parsed to get the hash used to call the AJAX API, which returns XML.
My question: How can I improve the security and decrease the possibility of others able to parse my web application.
The only ideas I have are: stop sending XML, but send HTML instead. Use flash (yet, this is not an option).
I understand that since the site is public, and no login can be implemented, it can be hard to refuse access to bots (non legitimate users). Also, Flash is not an option... it never is ;)
edit
The Web Application I am referring to: https://bikemap.appified.net/
This is somewhat of an odd request; you wish to lock down a system that your own web application depends on to work. This is almost always a recipe for disaster.
Web applications should always expect to be sidelined, so the real security must come from the server; tarpitting, session tokens, throttling, etc.
If that's already in place, I don't see any reason why should jump through hoops in your own web application to give robots a tougher time ... unless you really want to separate humans from robots ;-)
One way to reduce the refactoring pain on your side is to wrap the $.ajax function in a piece of code that could sign the outgoing requests (or somehow add fields to it) ... then minify / obscurify that code and hope it won't get decoded so fast.

Interacting with RESTful API's via Javascript?

to start off, I know C++, C#, Python, some Ruby, and basic Javascript. Anyway, my question revolves around how to interact with RESTful API's via Javascript. I haven't been able to find any good examples on various websites, and so I've come here.
So my basic question is: How do I interact with RESTful API's via JS? And where can I find out how to implement OAuth in JS? I know how to get my keys and such, just not how to actually code them in.
Below is an example of a twitter API status update run from my MAC terminal with curl:
curl -u username:password
-d "my tweet"
http://api.twitter.com/1/statuses/update.json
How can I implement this in Javascript (preferably with OAuth authentication)? This would at least start me going in the right direction.
Thanks so much!!
The problem is that you will need to use AJAX to query the remote REST API, and AJAX is only allowed to query resources on the same domain as the page. So, a request to api.twitter.com will fail because it is on a different domain than your server.
To correct this you will need to code your server to make the request to twitter. You can however create your own AJAX stubs that will accept data directly from your page, and then build / send requests to twitter server-side using data supplied by your client.
Generally Justin's approach is the correct one, however if you must have your client script interact with the REST service then you can do it with JsonP. that's JSON data wrapped in a function call.
see this page how to do it
http://www.ibm.com/developerworks/library/wa-aj-jsonp1/
OAuth version 1.0 in JavaScript is a bad idea because you need to expose your application's secret key, by doing so you may be allowing anyone else to impersonate your application. OAuth 1.0 was intended for use with a server under your control. So your users can send their tokens to your server and then you fire off the request to twitter on their behalf.
OAuth 2.0 solves this though twitter does not support it yet.
If you really want OAuth 1.0 you use my plugin: https://github.com/jpillora/jquery.rest and also make the change specified in this GitHub issue

Categories