We Keep Coding

Browser-based client-side scraping

I wonder if its possible to scrape an external (cross-domain) page through the user's IP?
For a shopping comparison site, I need to scrape pages of an e-com site but several requests from the server would get me banned, so I'm looking for ways to do client-side scraping — that is, request pages from the user's IP and send to server for processing.

No, you won't be able to use the browser of your clients to scrape content from other websites using JavaScript because of a security measure called Same-origin policy.
There should be no way to circumvent this policy and that's for a good reason. Imagine you could instruct the browser of your visitors to do anything on any website. That's not something you want to happen automatically.
However, you could create a browser extension to do that. JavaScript browser extensions can be equipped with more privileges than regular JavaScript.
Adobe Flash has similar security features but I guess you could use Java (not JavaScript) to create a web-scraper that uses your user's IP address. Then again, you probably don't want to do that as Java plugins are considered insecure (and slow to load!) and not all users will even have it installed.
So now back to your problem:
I need to scrape pages of an e-com site but several requests from the server would get me banned.
If the owner of that website doesn't want you to use his service in that way, you probably shouldn't do it. Otherwise you would risk legal implications (look here for details).
If you are on the "dark side of the law" and don't care if that's illegal or not, you could use something like http://luminati.io/ to use IP adresses of real people.

Basically browsers are made to avoid doing this…
The solution everyone thinks about first:
jQuery/JavaScript: accessing contents of an iframe
But it will not work in most cases with "recent" browsers (<10 years old)
Alternatives are:
Using the official apis of the server (if any)
Try finding if the server is providing a JSONP service (good luck)
Being on the same domain, try a cross site scripting (if possible, not very ethical)
Using a trusted relay or proxy (but this will still use your own ip)
Pretends you are a google web crawler (why not, but not very reliable and no warranties about it)
Use a hack to setup the relay / proxy on the client itself I can think about java or possibly flash. (will not work on most mobile devices, slow, and flash does have its own cross site limitations too)
Ask google or another search engine for getting the content (you might have then a problem with the search engine if you abuse of it…)
Just do this job by yourself and cache the answer, this in order to unload their server and decrease the risk of being banned.
Index the site by yourself (your own web crawler), then use your own indexed website. (depends on the source changes frequency)
http://www.quora.com/How-can-I-build-a-web-crawler-from-scratch
[EDIT]
One more solution I can think about is using going through a YQL service, in this manner it is a bit like using a search engine / a public proxy as a bridge to retrieve the informations for you.
Here is a simple example to do so, In short, you get cross domain GET requests

Have a look at http://import.io, they provide a couple of crawlers, connectors and extractors. I'm not pretty sure how they get around bans but they do somehow (we are using their system over a year now with no problems).

You could build an browser extension with artoo.
http://medialab.github.io/artoo/chrome/
That would allow you to get around the same orgin policy restrictions. It is all javascript and on the client side.

HTML5/javascript X11 server?

We have an application which consumes a large amount of data. Currently a desktop app, but we would like to deliver it via the browser.
It doesn't make sense to me to create a web app where we need to transfer a ll the data used for the visualizations.
We're looking at RDP and some products out there that provide RDP access via a fully javascript client. They seem to work well with our app, but I've been thinking about what it would take to move off Windows.
Switching the front end so that it could run under Linux would not be trivial, but not impossible, so the main stumbling block would be delivery.
I was wondering if there are any X11 javascript servers out there, but have not found any leads.

Use xpra's builtin html5 client, it supports any application you can run on an X11 desktop.

You can use an HTML5 VNC viewer like https://github.com/kanaka/noVNC coupled with a VNC server like RealVNC

AFAIK, recent GTK has been ported to HTML5+Javascript in Gtk Broadway
And you could make your application a web application, for instance by using Wt, or by making it an HTTP server thru specialized HTTP server libraries like libonion, libmicrohttpd etc.
By using AJAX techniques (e.g. thru jquery) your application won't transmit all the display data to the browser at once (but only incrementally and only the actually shown data).
You might also consider fastcgi as a way to connect your application to some web server.

I know two, both at very infancy:
https://github.com/GothAck/javascript-x-server
and
https://github.com/ttaubert/x-server-js
Both need simple tcp-to-websockets proxy in front, but all X11 logic happen inside web page and all x11 objects exist and interact within browser (so it's not just remote framebuffer but real server)

You can ever run full Linux distribution in Web Browser, but that's require to run x86/ARM emulator and GNU/Linux inside it. It provides X server with possible web connection too.

For very simple applications you can use libgreattao toolkit and tao-network-client to connect to it. I'm the author of both project. The API isn't yet frozen, but it rather behaves stable. You can read about it here:
https://nintyfan.wordpress.com/2015/04/30/server-buildin-into-libgreattao-and-tao-network-client/
It can provide some problems with applications with a lot of data, because all elements must be send to client, when it were created, but instead we don't send full graphics(only icons is send) and user interface could be changed quickly. It also don't support mouse enter/leave/move events.
I must tell: do not download tarbar, but download version from svn.

Sounds like the easiest approach for you is to get xrdp, which is an RDP-server for X. Then you would use your RDP client to connect to it. I think Nomachine NX supports html directly now, but I'm not sure. There was talk of an html X2go-client, but I don't know anything about that either.

direct client-to-client message

I have been working with node.js for a while, now when I'm looking deeper into it, for a chat aplication instead of sending message as client - server - client, there must be some possible ways for direct client to client message sending?

Browsers tend to communicate with servers via HTTP. Some implement other protocols, like websockets & SPDY, but again, these are mostly client-server protocols.
Some plug-ins (like Flash & Java) can open ports and communicate client-client. (AFAIK, haven't actually used them.)
Chrome is the only browser I'm aware of that can (soon) open TCP and UDP sockets from Javascript and do direct client-client communication. At the moment normal web apps can't do this, your app needs to be run as a "Chrome Packaged App", with a special manifest file.
Here are the docs, a blog post describing the feature and a browserify module that can behave like the net node.js module in the browser.
EDIT: This should probably not be tagged as [node.js] since you're trying to run in browsers (not in your node vm), this is a Javascript / Browser question.

This is maybe out of date question, but take a look on PeerJS.
It requires server only as a connection manager (broker). But all communication is done between clients directly.

This does not have anything with server. If you need something like that and if clients are flash you can use RTMFP . For JS i google this library which is js bridge for RTMFP, I dont know how it works. At the end you can write you own library to chat beetween clients but this is much harder(IP addresses are behind NAT, etc...)

I think answer for your question is here
PS Also exist open-source in-browser server which written using JS, but I didn't google it quickly. If you find it, please notify me.

If you just don't want to write your own server you can use:
https://httprelay.io
Use AJAX calls to communicate between peers.

Chrome Extension - Communicating with external program

I have an external application that is automating some tasks on a website. My goal is to implement a system which allows for the program and Chrome to synchronize cookies. While it is possible to query Chrome's cookie DB to read cookies, it is not possible to update the DB since Chrome maintains an I/O lock on the file, thus preventing easy synchronization.
The next logical step to me was to attempt to create an extension which will update cookies as necessary (through Chrome's cookie API). However, after about two days of research I have been unable to find an effective means to communicating cookie data between the browser and my application (which is written in Python.)
Sockets are out because it's for desktop based applications only. Websockets are out because as far as I can see it's impossible to setup a Websocket server using the HTML5 API (which is what I need since the browser needs to be the server and the program would be a connecting client). I'm really not sure what I am left with at this point. Is there something really obvious that I'm missing here? Any help is appreciated, cheers.

This feels like a very weird way to do whatever you're trying to do. Why are you doing this again?
Anyway, the most obvious solution is this:
You obviously have to secure communication between the app/plugin and the server. Again, this feels like a very weird way of doing stuff. But the solution will work. In this case both the app and the plugin are WS clients and your server is the arbiter.

Instant Message on browser

Introduction: I want to develop the chat client that user can chat on the browser and I use the protocol call xmpp. Because of HTML5 web socket not yet available I try flash xmlsocket instead.
Problem: I cannot connect to the server via browser. I'm not sure why, but I think that it is the problem of the server configuration.
Question: Which is the best jabber server suitable for this job?

Most probably this is a permission issue. Either that or you've just configured the wrong host/port to connect to. Flash is not allowed to connect to other hosts than the one it was gotten from itself. You have to explicitely allow flash connections on the receiving side of the request (so on the chat server that you are connecting to). Google for crossdomain.xml to get more info.

ejabberd sounds like a suitable option for you. ejabberd is xmpp server written in erlang and is used quite widely. Many of the well known web based im services like www.meebo.com etc are known to be running on ejabberd. It allows you to install other transports which would let you enable talking to users of other protocol like yahoo, msn, icq etc.
You can also have a look at the xiff action script library by ignite realtime. It is an xmpp client library in action script. If you use this library you would just need to implement the ui components.

Hum... flash is ok, but you'd be better of using something like BOSH, which is basically an HTTP layer over XMPP.
Ejabberd would be a good server, as it supports BOSH, I don't know about OpenFire or Tigase (but I'd say they do). Other servers should be looked at carefully because they don't seem to have a "dynamic" community.
You will need a lot of Javascript, and for that, I can recommend StropheJS, which is probably the very best library out there today.
We have created a MUC (Multi-user chat room) client called Aristochat that works in the browser recently at Superfeedr. You can find the code on Github and an example here.