I have a python code (server side) which doesn't interact with client side. However, I need to represent some items when it (server code) will has done. The only idea I came up with is the JS function which represents an item, calling from Python. Could you advise me either packages or another idea to implement this.
Some Details (I do not aware is it necessary, but might be it's helpful)
async def start_delete_delay(app, delay):
"""
The very function which thrust a delay for each front token.
Key arguments:
app -- our application.
delay -- a delay in seconds
"""
async with app['db'].acquire() as conn:
# First of all we need to check for database emptiness
query = text("SELECT True FROM tokens LIMIT(1)")
if await conn.fetch(query):
# If database is not empty then we are processing a waiting delay.
# First, fetching an id & related token from the first position (due to it queue) from database.
query = select([db.tokens.c.id, db.tokens.c.token]).order_by(asc(db.tokens.c.id)).limit(1)
query_result = await conn.fetchrow(query)
# Retrieving an id and token
id_before_sleep, token = query_result['id'], query_result['token']
# Setting a delay
try:
await asyncio.sleep(delay)
# Some information related with cancellation error
# https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.cancel
except asyncio.CancelledError:
pass
# Check whether a token at the first place as same as it was before
finally:
# If it possible but all of members picked their tokens over 60 seconds.
if await conn.fetch(text("SELECT True FROM tokens LIMIT(1)")):
query_result = await conn.fetchrow(query)
id_after_sleep = query_result['id']
# If they are same then we delete that token and starting delay again.
if id_before_sleep == id_after_sleep:
query = delete(db.tokens).where(db.tokens.c.id == id_before_sleep)
# preparing a token for reuse.
app['new_token'].prepare_used_token(token)
# Deleting a token
await conn.fetchrow(query)
# I'd like to call a JS function (which I already have) here
# Starting a delay for adjacent token, over and over and over
task = make_task(start_delete_delay, app, delay)
asyncio.gather(task)
I found two solutions, so if someone faced with such problems try to use them:
First solution
The clue is WebSockets. I used aiohttp and asyncio.
In JavaScript file I added up a listening socket:
var socket = new WebSocket('/link-to-websocket')
In server-side I added a websocket_handler, in my case it sending the message after deleting a toke from database
async def websocket_handler(request):
ws = web.WebSocketResponse()
await ws.prepare(request)
async for msg in ws:
if msg.type == aiohttp.WSMsgType.TEXT:
if app['updt_flag']:
await ws.send_str("signal")
else:
await ws.close()
return ws
And adding it to routes
app.add_routes([web.get('/link-to-websocket', websocket_handler)])
1) How JavaScript works: Deep dive into WebSockets and HTTP/2 with SSE + how to pick the right path
2) Python aiohttp websockets
However this method isn't the best, we don't use entire websocket's functionality therefore let's go ahead to another method: Server-Sent Events (SSE). This method is more suitable for the my problem because we always receiving response from server without request to it (whereas websockets doesn't incorporate such option):
Second solution
As I said above I will use SSE and for these purposes it required a sse-package
pip install aiohttp_sse
import asyncio
from aiohttp_sse import sse_response
async def SSE_request(request):
loop = request.app.loop
async with sse_response(request) as resp:
while True:
if request.app['updt_flag']:
await resp.send("signal")
request.app['updt_flag'] = False
await asyncio.sleep(1, loop=loop)
return resp
Adding route
web.get('/update', SSE_request)
Adding listening sse to JS:
const evtSource = new EventSource("/update");
evtSource.onmessage = function(e) {
display_queue_remove();
}
Thats all:)
Related
I have this code, whenever the user goes to this certain endpoint, it is supposed to emit a message to a python client, which then gets some data and then returns it back as a callback so I can show the users the data.
This is the server-side code (NodeJS):
app.get('/hueapi/lights', verifyToken, (req,res) => {
const bridgeIDFromApp = req.header('bridgeID');
const socketID = socketRefDic[bridgeIDFromApp]['socketID'];
io.to(socketID).emit('getAllLights', 'getAllLights', function(data){
res.send(data); // The callback function that shows the data given by the python client
});
});
It just sends a simple 'getAllLights' message to the python client in question and then runs the function which provides the data.
This is the client-side code (python):
def getAllLights(data):
lightData = requests.get('http://localhost:4000/lights/')
return lightData
Am I doing the call back wrong or? I just want to send the data straight back to the user after retrieving it.
EDIT:
I am now using io.to(...).emit(...) instead of io.send(...).emit(...) yet I am still getting the error saying I'm broadcasting, yet I'm not, am I?
I don't think that the ack method will work for you unless it is implemented on the python side as well. The reason that you are still getting the broadcasting error is because io.to does not return a socket it returns a room which does broadcast.
Probably easier to just have a separate endpoint on the client side. Which your python code doesn't even attempt from what I see. The python code should still be able to write to the socket.
So to implement your own ack function you would simply write your ack message to the socket. If you need it to be statefully namespaced then you would have to include an address for the python code to reference with your getAllLights message.
Node:
app.get('/hueapi/lights', verifyToken, (req,res) => {
const bridgeIDFromApp = req.header('bridgeID');
const socketID = socketRefDic[bridgeIDFromApp]['socketID'];
const uniqAck = "some unique endpoint path";
const socket = getSocketByID(socketID);
socket.on(uniqAck, (data) => res.send);
socket.emit('getAllLights', 'getAllLights:'+uniqAck);
});
Python:
def getAllLights(data):
lightData = requests.get('http://localhost:4000/lights/');
return (lightData, split(data, ":")[1]); // split if its not already done by this point.
// capture 'value' from getAllLights when it is called...
socket.emit(value[1], value[0]);
I'm trying to build a short Python program that extracts Pewdiepie's number of subscribers which is updated every second on socialblade to show it in the terminal. I want this data like every 30 seconds.
I've tried using PyQt but it's slow, i've turned to dryscrape, slightly faster but doesn't work either as I want it to. I've just found Invader and written some short code that still has the same problem : the number returned is the one before the Javascript on the page is executed :
from invader import Invader
url = 'https://socialblade.com/youtube/user/pewdiepie/realtime'
invader = Invader(url, js=True)
subscribers = invader.take(['#rawCount', 'text'])
print(subscribers.text)
I know that this data is accessible via the site's API but it's not always working, sometimes it just redirect to this.
Is there a way to get this number after the Javascript on the page modified the counter and not before ? And which method seems the best to you ? Extract it :
from the original page which always returns the same number for hours ?
from the API's page which bugs when not using cookies in the code and after a certain amount of time ?
Thanks for your advices !
If you want scrape a web page that has parts of it loaded in by javascript you pretty much need to use a real browser.
In python this can be achieved with pyppeteer:
import asyncio
from pyppeteer import launch
async def main():
browser = await launch(headless=False)
page = await browser.newPage()
await page.goto('https://socialblade.com/youtube/user/pewdiepie/realtime',{
'waitUntil': 'networkidle0'
})
count = int(await page.Jeval('#rawCount', 'e => e.innerText'))
print(count)
asyncio.get_event_loop().run_until_complete(main())
Note: It does not seems like the website you mentioned above is updating the subscriber count frequently any more (even with JavaScript). See: https://socialblade.com/blog/abbreviated-subscriber-counts-on-youtube/
For best success and reliability you will probably need to set the user agent(page.setUserAgent in pyppeteer) and keep it up to date and use proxies (so your ip does not get banned). This can be a lot of work.
It might be easier and cheaper (in time and than buying a large pool of proxies) to use a service that will handle this for you like Scraper's Proxy. It supports will use a real browser and return the resulting html after the JavaScript has run and route all of our requests through a large network of proxies, so you can sent a lot of requests without getting you ip banned.
Here is an example using the Scraper's Proxy API getting the count directly from YouTube:
import requests
from pyquery import PyQuery
# Send request to API
url = "https://scrapers-proxy2.p.rapidapi.com/javascript"
params = {
"click_selector": '#subscriber-count', # (Wait for selector work-around)
"wait_ajax": 'true',
"url":"https://www.youtube.com/user/PewDiePie"
}
headers = {
'x-rapidapi-host': "scrapers-proxy2.p.rapidapi.com",
'x-rapidapi-key': "<INSERT YOUR KEY HERE>" # TODO
}
response = requests.request("GET", url, headers=headers, params=params)
# Query html
pq = PyQuery(response.text)
count_text = pq('#subscriber-count').text()
# Extract count from text
clean_count_text = count_text.split(' ')[0]
clean_count_text = clean_count_text.replace('K','000')
clean_count_text = clean_count_text.replace('M','000000')
count = int(clean_count_text)
print(count)
I know this is a bit late, but I hope this helps
I have created a Redis cluster with 30 instances (15 masters/ 15 nodes). With python code i connected to these instances, i found the masters and then i wanted to add some keys to them.
def settomasters(port, host):
r = redis.Redis( host=host, port=port )
r.set("key"+port,"value")
Error:
redis.exceptions.ResponseError: MOVED 12539 127.0.0.1:30012
If i try to set key from redis-cli -c -p portofmyinstance sometimes i get a redirection message that tells where the keys stored.
I know that in case of get requests for example, a smart client is needed in order to redirect the requests to the correct node (the node that holds the key) otherwise a moved error occurs. Is it the same situation? I need to catch the redis.exceptions.ResponseError and try to set again?
while True:
try:
r.set("key","value")
break
except:
print "error"
pass
My first try was above code but without solution. The set operation never succeeds.
On the other hand below code in javascript does not throw an error and i cannot figure the reason:
var redis = require('redis-stream'),
client = new redis(30001, '127.0.0.1');
// Open stream
var stream = client.stream();
// Example of setting 200 records
for(var record = 0; record <200; record++) {
var command = ['set', 'qwerty' + record, 'QWERTYUIOP'];
stream.redis.write( redis.parse(command) );
}
stream.on('close', function () {
console.log('Completed!');
});
// Close the stream after batch insert
stream.end();
Any help will be appreciated, thanks.
with a redis cluster you can use the normal redis client only if you "find for the certain key the slot that belongs and then the slots that each master serves. With this information i can set keys to the correct node without moved redirection errors." as #Antonis said. Otherwise you need http://redis-py-cluster.readthedocs.io/en/master/
so i am using faye pub-sub in my application, publish is happening from different application, in my faye.js i have written ajax post method for rails. now if 5 pages of my application is opened in browser, faye.js is loaded 5 times and post method is called 5 times. if not a single page is opened, post method wont work even once. however i am receiving published data in faye server. so is there a way of calling rails post method in faye.ru file when i use callback method.
This is my faye.ru
require 'faye'
require File.expand_path('../config/initializers/faye_token.rb', __FILE__)
Faye::WebSocket.load_adapter('thin')
Faye.logger = Logger.new(STDOUT)
class ServerAuth
def incoming(message, callback)
if message['channel'] !~ %r{^/meta/}
if message['ext']['auth_token'] != ENV['FAYE_TOKEN']
message['error'] = 'Invalid authentication token'
end
end
callback.call(message)
end
# IMPORTANT: clear out the auth token so it is not leaked to the client
def outgoing(message, callback)
if message['ext'] && message['ext']['auth_token']
message['ext'] = {}
end
callback.call(message)
end
end
$bayeux = Faye::RackAdapter.new(:mount => '/faye', :timeout => 25)
$bayeux.add_extension(ServerAuth.new)
run $bayeux
and my faye.js
$(function() {
var faye = new Faye.Client('http://localhost:9292/faye');
faye.subscribe("/functional", function(data) {
ajax_call(data);
});
});
You can use uri and http in your faye.ru
require "uri"
require "net/http"
params = {"param1":"param2"}
x = Net::HTTP.post_form(URI.parse('http://your-post url'), params)
puts x.body
I'm using Meteor 0.6.3.1 and have improvised my own user system (not really a user system but I thought I might as well make use of the userId variable since nobody else laid claim to it.)
The problem is, the variable isn't persisting.
I have this code
Meteor.methods({
'initCart': function () {
console.log(this.userId);
if(!this.userId) {
var id = Carts.insert({products: []});
this.setUserId(id);
console.log("cart id " + id + " assigned");
}
return this.userId;
}
});
The point being, you should be able to switch pages but still use the same shopping cart.
I can't use Sessions since they're client-side and could lead to information leaking between users..
How should I go about doing this? Is there anything like Amplify for server-side Meteor?
From Meteor docs:
setUserId is not retroactive. It affects the current method call and
any future method calls on the connection. Any previous method calls
on this connection will still see the value of userId that was in
effect when they started.
When you refresh you create a new connection. On that connection you log-in using the cookie stored by the user system on the client side.
You can store the cart id in a cookie...
This works for me:
# server/methods/app.coffee
#---------------------------------------------------
# Info found in Meteor documentation (24 Feb. 2015)
#
# > Currently when a client reconnects to the server
# (such as after temporarily losing its Internet connection),
# it will get a new connection each time. The onConnection callbacks will be called again,
# and the new connection will have a new connection id.
# > In the future, when client reconnection is fully implemented,
# reconnecting from the client will reconnect to the same connection on the server:
# the onConnection callback won't be called for that connection again,
# and the connection will still have the same connection id.
#
# In order to avoid removing data from persistent collections (ex. cartitems) associated with client sessionId (conn.id), at the moment we'll implement the next logic:
#
# 1. Add the client IP (conn.clientAddress) to the cartitems document (also keep the conn.id)
# 2. When a new connection is created, we find the cartitems associated with this conn.clientAddress and we'll update (multi: true) the conn.id on the all cartitems.
# 3. Only remove the cartitems associated with the conn.id after 10 seconds (if in this time the customer reconnect, the conn.id will have been updated at the point #2. we avoid this way removing after refresh the page for example.
# 4. After 10 seconds (ex. the user close the window) we'll remove the cartitems associated with the conn.id that it has been closed.
Meteor.onConnection (conn) ->
CartItems.update({clientAddress: conn.clientAddress}, {$set: {sessionId: conn.id}}, {multi: true})
conn.onClose ->
Meteor.setTimeout ->
CartItems.remove {sessionId: conn.id}
, 10000
Meteor.methods
getSessionData: ->
conn = this.connection
{sessionId: conn.id, clientAddress: conn.clientAddress}