We Keep Coding

Scraping with Python returns NONE

I'm trying to scrape info from a site that is using java script overlayed on google maps. Using inspector I can clearly see the data I want to recover, but I have been unsuccessful thus far.
I have tried the following; requests, Beautiful soup with PyQt (Sentdex example) - this works for his example but I can't get it to work for mijn. See code... and I've tried many others. Any ideas? Thanks.
import bs4 as bs
import sys
import urllib.request
from PyQt5.QtWebEngineWidgets import QWebEnginePage
from PyQt5.QtWidgets import QApplication
from PyQt5.QtCore import QUrl
class Page(QWebEnginePage):
def __init__(self, url):
self.app = QApplication(sys.argv)
QWebEnginePage.__init__(self)
self.html = ''
self.loadFinished.connect(self._on_load_finished)
self.load(QUrl(url))
self.app.exec_()
def _on_load_finished(self):
self.html = self.toHtml(self.Callable)
print('Load finished')
def Callable(self, html_str):
self.html = html_str
self.app.quit()
def main():
page = Page('https://inzameling.spaarnelanden.nl/')
soup = bs.BeautifulSoup(page.html, 'lxml')
js_test = soup.find('div', class_='infoBoxValue')
print(js_test)
if __name__ == '__main__':
main()

431 (Request Header Fields Too Large) (ReactJS/ Flask)

I'm getting a 431 error on a simple fetch post, specifically:
net::ERR_ABORTED 431 (Request Header Fields Too Large)
Proxy set to: "proxy": "http://0.0.0.0:3000" (on reactjs)
fetch("/process_image", {
method:"POST",
cache: "no-store",
headers:{
"content_type":"application/json",
},
body:JSON.stringify("testing")
}
).then(response => {
//return response.json()
})
.then(json => {
//this.setState({playerName: json[0]})
});
I have tried to clear the cache and cookies on my browser based on the solutions similar to my problem but nothing is helping, and I wouldn't think this would cause an issue since all I'm passing is a small text.
I have also tried to replicate my solution using .net core, and I am able to post data with no issues, but using .net core does not fit in my scope. (I'm learning how to use flask)
I'm passing the data to a .py file:
from flask import Flask, request, jsonify
from flask_cors import CORS
import Image, io, base64
import numpy as np
from keras import models
app = Flask(__name__)
CORS(app)
#app.route("/")
def home():
return "Hello, Flask!"
#app.route("/process_image", methods=['GET', 'POST'])
def process_image():
data_uri = request.json()
print("hello: ", data_uri)
Any ideas what could be causing the problem?

How to Un-register ( remove ) service worker from a Django login and admin pages?

I had developed an application using Django and implemented service worker on it. but when I redirect to the login and admin page, I want to remove the service worker.
Service worker
Service worker registration process:
url.py
url(r'^sw(.*.js)$' , 'project.views.sw_js', name='sw_js'),
view.py
def sw_js(request, js):
template = get_template('sw.js')
html = template.render()
return HttpResponse(html, content_type="application/x-javascript")

Use django decorators
# decorators.py
from django.core.urlresolvers import reverse
from django.http import Http404
def service_worker(function):
def wrap(request, *args, **kwargs):
if request.path.startswith(reverse('admin:index')) or request.path.startswith(reverse('django.contrib.auth.views.login')):
raise Http404
else:
return function(request, *args, **kwargs)
wrap.__doc__=function.__doc__
wrap.__name__=function.__name__
return wrap
url(r'^sw(.*.js)$' , 'project.views.sw_js', name='sw_js'),
#service_worker
def sw_js(request, js):
template = get_template('sw.js')
html = template.render()
return HttpResponse(html, content_type="application/x-javascript")

Send data from javascript to python in google app engine

i want to create an application in google app engine using python. I have to send a list of json string to python script.I have done the following code but ain't worked.
$.post("/javascriptdata",{v:s},function(data,status) {});
In python script i have a class named javascriptdata to where data has to be send to
import wsgiref.handlers
import json
import sys
import cgi
from google.appengine.ext import webapp
from google.appengine.ext.webapp import Request
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.ext.webapp import template
class mainh(webapp.RequestHandler):
def get(self):
self.response.out.write(template.render("paint.html",{}))
class javascriptdata(webapp.RequestHandler):
def post(self):
self.response.headers['content-Type'] = 'html'
data1=self.request.get('v')
self.response.out.write("""<html><body>""")
self.response.out.write(data1)
self.response.out.write("""</body></html>""")
def main():
app = webapp.WSGIApplication([
('/',mainh),("/save",javascriptdata)], debug=True)
wsgiref.handlers.CGIHandler().run(app)
if __name__ == "__main__":
main()
The javascriptdata is associated with the url "/save". I have created a submit button named "save" that would redirect to /save but iam not getting any output. I know it may be a silly mistake but Iam struggling to sort it out. Please suggest me how to post and read the data for this code.

This seems suspicious:
$.post("/javascriptdata",{v:s},function(data,status) {});
since you don't have a /javascriptdata URL mapped in the python code. Perhaps you meant
$.post('/save', ...?
Alternatively, you could change the WSGIApplication init to be:
...("/javascriptdata",javascriptdata)...

Flask, html and javascript desktop app

I want to build a desktop application using python, html and javascript. So far i have followed the tuts on flask and have a hello world working example. What should i do now to make it working? how do the html files "talk" to the python scripts below them?
here is my code so far :
from flask import Flask, url_for, render_template, redirect
app = Flask(__name__)
#app.route('/hello/')
#app.route('/hello/<name>')
def hello(name=None):
return render_template('hello.html', name=name)
#app.route('/')
def index():
return redirect(url_for('init'))
#app.route('/init/')
def init():
css = url_for('static', filename='zaab.css')
return render_template('init.html', csse=css)
if __name__ == '__main__':
app.run()

You can use HTML forms just as you normally would in your Jinja templates - then in your handler you use the following:
from flask import Flask, url_for, render_template, redirect
from flask import request # <-- add this
# ... snip setup code ...
# We need to specify the methods that we accept
#app.route("/test-post", methods=["GET","POST"])
def test_post():
# method tells us if the user submitted the form
if request.method == "POST":
name = request.form.name
email = request.form.email
return render_template("form_page.html", name=name, email=email)
If you wanted to use GET instaed of POST to submit the form you would just check request.args rather than request.form (See flask.Request's documentation for more information). If you are going to be doing much with forms though, I recommend checking out the excellent WTForms project and the Flask-WTForms extension.