I am currently building a tool using flask that does various actions using ssh. One of those actions is using DD to copy from X to Y.
I currently have the following javascript set up on my page
<script>
var source == new EventSource("/dd");
source.onmessage = function(event){
$('.determinate'.css('width', event.data+'%')
if(event.data == 100){
source.close()
}
}
Which calls the following flask generator which parse's the stdout of DD to return a % value for the current progress.
#app.route('/dd')
def progress():
def generate():
ssh.SSHClient('127.0.0.1', username, password)
chan = ssh.get_transport().open_session()
chan.settimeout(10800)
try:
ssh.do('dd if=/dev/sda of=/test.img')
while data:
data = chan.recv_stderr(1024)
try:
yield "data: " + str(data) + "\n\n\"
return Response(generate(), mimetype='text/event-stream')
The above is pseudo code, but the things i want to be able to change are the DD command (dd if=/dev/sda of=/test/img) from variables i get from the form that triggers this page, as well as the hostname from the ssh.connect function with request.remote_addr.
When i try to replace '127.0.0.1' with request.remote_addr i get an out of context error.
Is there anyway to pass flask/jinja2 variables such as {{ image.path }} to my generator view?. The pseudo code i want would be this, where hostname, and dd is dynamic ( changes are in curly brackets )
#app.route('/dd')
def progress():
def generate():
ssh.SSHClient({{ request.remote_addr }}, username, password)
chan = ssh.get_transport().open_session()
chan.settimeout(10800)
try:
ssh.do('dd if={{ device }} of={{ image.path }}')
while data:
data = chan.recv_stderr(1024)
try:
yield "data: " + str(data) + "\n\n\"
return Response(generate(), mimetype='text/event-stream')
Just figured it out, didn't read the last paragraph of the docs.
If You want to keep the context of the previous request in the generator just change
return Response(generate(), mimetype='text/event-stream')
to
return Response(stream_with_context(generatE()), mimetype='text/event-stream')
For passing data i just used the "session" object to pass data and then remove it when im done using it.
Related
I'm trying to scrape the following site
I'm able to receive a response but i don't know how can i access the inner data of the below items in order to scrape it:
I noticed that accessing the items is actually handled by JavaScript and also the pagination.
What should i do in such case?
Below is my code:
import scrapy
from scrapy_splash import SplashRequest
class NmpaSpider(scrapy.Spider):
name = 'nmpa'
http_user = 'hidden' # as am using Cloud Splash
allowed_domains = ['nmpa.gov.cn']
def start_requests(self):
yield SplashRequest('http://app1.nmpa.gov.cn/data_nmpa/face3/base.jsp?tableId=27&tableName=TABLE27&title=%E8%BF%9B%E5%8F%A3%E5%8C%BB%E7%96%97%E5%99%A8%E6%A2%B0%E4%BA%A7%E5%93%81%EF%BC%88%E6%B3%A8%E5%86%8C&bcId=152904442584853439006654836900', args={
'wait': 5}
)
def parse(self, response):
goal = response.xpath("//*[#id='content']//a/#href").getall()
print(goal)
If you use some breakpoints you'll see its a frustrating job that I explain what I understood from my research.
when you are working with this kind of situation you have two ways:
1 - [Easy Way] use selenium and open a browser and click on each link and get the returned contents easily, you can run multiple browsers and get link contents simultaneously.
2 - [Hard Way] simulate what website does (by making similar functions inside python) and do exactly what website does in JS but in the end instead of showing the results, just save it in a variable and use it the way you want.
Now if you choose the HARD WAY this is what I found:
the link JS is like this:
commitForECMA(callbackC,'content.jsp?tableId=26&tableName=TABLE26&tableView=国产医疗器械产品（注册&Id=138150',null)
it calls a function named commitForECMA and get what this function returns and pass it to callBackC function.
well this was obvious, but its important to know what these functions do and how to replicate it.
commitForECMA:
this is the function:
function commitForECMA($_8, $_10, $_12) {
request = createXMLHttp();
request.onreadystatechange = $_8;
if ($_12 == null) {
_$du(request, _$Fe('uM6r2MG'), _$Fe("jp0YV"), $_10);
request.setRequestHeader(_$Fe("XACeXwDYXwcTV8Ur2"), _$Fe("YwDYgwceLwDT7iCYX3Ce9FKyvHKwPFa"));
} else {
var $_9 = "";
var $_19 = $_12.elements;
var $_0 = $_19.length;
for (var $_18 = 0; $_18 < $_0; $_18++) {
var $_14 = _$3P($_19, $_18);
if ($_14.type != _$Fe("yQ6YPMK20") && _$3P($_14, _$Fe('uwbm7wKV')) != "") {
if ($_9.length > 0) {
$_9 += "&" + $_14.name + "=" + _$3P($_14, _$Fe('kwbm7wKV'));
} else {
$_9 += $_14.name + "=" + _$3P($_14, _$Fe('Ewbm7wKV'));
}
$_9 += _$Fe("Jx2J03Up2Hsl");
}
}
_$du(request, _$Fe('uM6r2MG'), _$Fe("HVlesYq"), $_10);
$_9 = encodeURI($_9);
$_9 = encodeURI($_9);
request.setRequestHeader(_$Fe("d3CmOFDVz3CeXwoxBMq"), _$Fe("FMbZz3CmOFDV"));
request.setRequestHeader(_$Fe("yACeXwDYXwcTV8Ur2"), _$Fe("g3UraMD2O3UpNMCgB8cT6w6QzRbenM1TTQbS2MbJBRDY9"));
}
request.send($_9);
if ($_12 != null) {
$_12.reset();
}
}
yes as you can see it just creates a XMLHTTP request which (for the links in question) Posts the $_10 content to the server and get the results in callBackC function which is now in $_8. but the trick here is the $_10 contents goes through ~13000 lines of code to create links like this:
http://app1.nmpa.gov.cn/data_nmpa/face3/content.jsp?6SQk6G2z=GBK-56.it.xmhx8IaDT25ZyaSxljrwULe8AkNw8QjmeNqdT0YqZYbMZ2P6Jgn3ZUIgh3ibPI81bjA6xUCKJmzy1LD.4AZnk4g4G_iMO4tdiebiVDoPPtdVDIkDWw0OnDHek.d_2r.PfBtuIoxDvrbGDL.Lv2AuD6lxiObz_lldDHq6HnEw_irAP1hCH.Dr3KdW33DN2w0X1R75N3f8GXdHinmxXLtYbZNYZEE9K7lk9AGmBWgcTds.XgGVW3gDS5OEwoRat44Ecke8k7ZXoY_2revEbUrD8UpOrGprlPEwVYuAvLoTSZX8WJEWQ_QT2CDjNw0FOwAECzsFJa4hGgUtjCPzG&c1SoYK0a=GBK-4aeKAo74EouxLY.stFwdwvXQQG_hXMGG8gB0Hhe6V2Il9k9c8yiTLqduIXpv2RNt.H.weYXeF5XhV0CR2lATieRmk.cs8.fPhNpfGx7JkG1uacp75kDcmXsNtuKgbzRUHZh8vkj4UEYbPcwIYIOw5gFG_cMi9n1GYq0AXXK9UQn9IsmjCBuI7AOFw.pk91OgjvkJCcg2y0y3yDkGwZPcg5EktfAXi.PjmfaecWg8hodU87q6B3ZuPxhel9K9I3EDBxzCHtZqt_0YFlkJCcK4hLq
the problem is with obfuscation and also the nested variables and functions that can keep you out of track for hours if you try to debug it line by line (which I did) and the code makes the characters after content.jsp? part one by one and that explains why its about 13000 lines!!
this part request.send($_9); should have a body for request because its a POST request and $_9 was always null! it seems there are more protection levels to it as it seems.
callBackC:
well the callbackC is apparently a simple function to get responseText and show it to user:
function callbackC() {
if (request.readyState == 1) {
_$c2(document.getElementById(_$Fe("Y3CeXwDYXwq")), '=', _$Fe('vFKyXRUxEYlTW'), _$Fe("kHDxnHOaB3vE5HDxnHOSNMKQGQ6xOHK2z3Kw2Qne7MCm9FKyvhbwNROg"));
}
if (request.readyState == 4) {
if (request.status == 200) {
oldContent[oldContent.length] = request.responseText;
_$c2(document.getElementById(_$Fe("b3CeXwDYXwq")), '=', _$Fe('EFKyXRUxEYlTW'), request.responseText);
request = null;
} else {
_$c2(document.getElementById(_$Fe("u3CeXwDYXwq")), '=', _$Fe('BFKyXRUxEYlTW'), "<br><br><br><span style=font-size:x-large;color:#215add>服务器未返回数据</span>");
}
}
}
I didn't quiet get what those _$Xx functions do (because it goes so deep that its out of my patient!) but it seems they simply replaced the document.getElementById("someThing").innerText="Contents"; with multi layered functions so we can't understand the code easily, and the request.responseText is what you need which is HTML code for the table of results.
there is also a 3rd way which I don't know if you can implement it in your code, but since these functions are in a public scope you can simply override them by redefining these two functions (or replace the functions in the link with your own functions and run them). I tried to get the URL for the request which gave me the link I used in middle of this post, but it didn't worked (I just override the callBackC function and get request.responseURL) and the link gave me 404 error.
I don't think I said all I got from my observations but I think it's enough for you to know what you are up against if you are not already aware, and I hope I was helpful.
Reference:
XMLHttpRequest: Living Standard — Last Updated 16 August 2021
When pressing a button i call a javascript function in my html file which takes two strings as parameters (from input fields). When the function is called i want to pass these parameters to my flask file and call another function there. How would i accomplish this?
The javascript:
<script>
function ToPython(FreeSearch,LimitContent)
{
alert(FreeSearch);
alert(LimitContent);
}
</script>
The flask function that i want to call:
#app.route('/list')
def alist(FreeSearch,LimitContent):
new = FreeSearch+LimitContent;
return render_template('list.html', title="Projects - " + page_name, new = new)
I want to do something like "filename.py".alist(FreeSearch,LimitContent) in the javascript but its not possible...
From JS code, call (using GET method) the URL to your flask route, passing parameters as query args:
/list?freesearch=value1&limit_content=value2
Then in your function definition:
#app.route('/list')
def alist():
freesearch = request.args.get('freesearch')
limitcontent = request.args.get('limit_content')
new = freesearch + limitcontent
return render_template('list.html', title="Projects - "+page_name, new=new)
Alternatively, you could use path variables:
/list/value1/value2
and
#app.route('/list/<freesearch>/<limit_content>')
def alist():
new = free_search + limit_content
return render_template('list.html', title="Projects - "+page_name, new=new)
I've been trying to modify the sample dashboard widget at this location
https://learn.microsoft.com/en-us/vsts/extend/develop/add-dashboard-widget?view=vsts#part-2-hello-world-with-vsts-rest-api
However, reluctantly have to admit I simply can't understand the structure required to extend it
Near the end, it uses "load: function" and returns the outputs of a REST API call, which I can consume however I want
However, I need to make more than one different REST call, and I simply cannot figure out how to get that info usable in my function
I modified the code so it starts like this:
VSS.require(["TFS/Dashboards/WidgetHelpers", "TFS/Work/RestClient","VSS/Service", "TFS/WorkItemTracking/RestClient" ],
I then created a handle for the other call I want to make like this:
var queryClient = VSS_Service.getCollectionClient(TFS_Wit_QueryAPI.WorkItemTrackingHttpClient);
var queryResults = queryClient.getQuery(projectId, "Shared Queries/My Bugs");
However, I cannot consume the contents of queryResults - I know it's working up to a point as if I put in an invalid URL it will error as it knows it can't access anything there. If the URL is correct, no matter what I've tried - even stringify just to see what comes back - I get 'undefined' or something similar (it's definitely a valid JavaScript object)
The key seems to be right at the end when you have "load: function" except that only allows one thing to be returned? The reason I know this is if I change the function that it returns to be the one I've written rather than the one from the sample, it works fine - but the problem remains the same in that I can only process the results of one API call.
You can call more than one APIs, the code in that article is just the simple sample.
For Widget extension, you just need to return the status (e.g. Success()) in load function, so you can return status at the end of the function. For example:
var getQueryInfo = function (widgetSettings) {
// Get a WIT client to make REST calls to VSTS
return TFS_Wit_WebApi.getClient().getQuery(projectId, "Shared Queries/Feedback")
.then(function (query) {
// Create a list with query details
var $list = $('<ul>');
$list.append($('<li>').text("Query ID: " + query.id));
$list.append($('<li>').text("Query Name: " + query.name));
$list.append($('<li>').text("Created By: " + (query.createdBy ? query.createdBy.displayName: "<unknown>") ));
// Append the list to the query-info-container
var $container = $('#query-info-container');
$container.empty();
$container.append($list);
// Use the widget helper and return success as Widget Status
return true;
}, function (error) {
// Use the widget helper and return failure as Widget Status
console.log(error);
return false;
});
}
var getAnOhterQueryInfo = function (widgetSettings) {
// Get a WIT client to make REST calls to VSTS
return TFS_Wit_WebApi.getClient().getQuery(projectId, "Shared Queries/Bug")
.then(function (query) {
// Create a list with query details
var $list = $('<ul>');
$list.append($('<li>').text("Query ID: " + query.id));
$list.append($('<li>').text("Query Name: " + query.name));
$list.append($('<li>').text("Created By: " + (query.createdBy ? query.createdBy.displayName: "<unknown>") ));
// Append the list to the query-info-container
var $container = $('#query-info-container');
$container.empty();
$container.append($list);
// Use the widget helper and return success as Widget Status
return true;
}, function (error) {
// Use the widget helper and return failure as Widget Status
console.log(error);
return false;
});
}
return {
load: function (widgetSettings) {
// Set your title
var $title = $('h2.title');
$title.text('Hello World');
var r1= getQueryInfo(widgetSettings);
var r2=getAnOhterQueryInfo(widgetSettings);
if(r1==true && r2==true){
return WidgetHelpers.WidgetStatusHelper.Success();
}else{
return WidgetHelpers.WidgetStatusHelper.Failure("failed, check error in console");
}
}
I am newbie in AWS Arena. This is my 2nd question regarding AWS Lambda function and AWS LEX. I want to write a lambda function to trigger 2 different intents based on the value of something without any user Utterance. For example
if a >= 90.....Intent-1 will work and say "Messi is the best Footballer" and
if a < 90......Intent-2 will work and say "Ronaldo is the best Footballer"
It is not supposed to work like that, intents are triggered based on what user types. For example, you can make an intent BestFootballer and it will be triggered on utterance who is the best footballer.
Now, once the intent is triggered you can apply some logic to dynamically create a response.
def build_response(message):
return {
"dialogAction":{
"type":"Close",
"fulfillmentState":"Fulfilled",
"message":{
"contentType":"PlainText",
"content":message
}
}
}
def perform_action(intent_request):
source = intent_request['invocationSource']
output_session_attributes = intent_request['sessionAttributes'] if intent_request['sessionAttributes'] is not None else {}
if source == 'FulfillmentCodeHook':
a = 100
if a < 90:
return build_response('Ronaldo is the best Footballer')
else:
return build_response('Messi is the best Footballer')
def dispatch(intent_request):
intent_name = intent_request['currentIntent']['name']
if intent_name == 'BestFootballer':
return perform_action(intent_request)
raise Exception('Intent with name ' + intent_name + ' not supported')
def lambda_handler(event, context):
return dispatch(event)
Hope it helps.
I'm using jQuery autocomplete plugin http://www.devbridge.com/projects/autocomplete/jquery/ to provide search suggestions in my web application where I want to send the response in json format.
Django views.py for sending the suggestions response:
def keywords_suggestions(request):
if request.is_ajax():
suggestions = []
q = request.POST.get('q')
try:
g = KeywordsModel.objects.filter(keyword__startswith=q).order_by('count')
except KeywordsModel.DoesNotExist:
return HttpResponse("")
else:
for i in range(0,len(g)):
global suggestions
suggestions.append(g[i].keyword)
to_json = {
"query": q,
"suggestions": suggestions
}
return HttpResponse(simplejson.dumps(to_json), mimetype='application/json')
Django models.py:
class KeywordsModel(models.Model):
keyword = models.CharField(max_length=40, blank=False)
count = models.IntegerField(max_length=20)
def __unicode__(self):
return self.keyword
jQuery code:
$("#add-keywords").keyup(function() {
$('#add-keywords').autocomplete({
serviceUrl:'/keywords_suggestions',
minChars:3,
maxHeight:220,
width:280,
zIndex: 9999,
params: { q: $('#add-keywords').val() },
onSelect: function(value, data){ $('#add-keywords').val(value); },
});
});
I'm getting this error when I type on the #add-keywords text box.
Request URL:http://127.0.0.1:8000/keywords_suggestions/?q=web&query=web
Request Method:GET
Status Code:500 INTERNAL SERVER ERROR
UPDATE
ValueError at /keywords_suggestions/
The view information.views.keywords_suggestions didn't return an HttpResponse object.
UPDATE-2
I'm having doubt in the suggestions variable, maybe global suggestions will have the problem. Am I doing it right?
Could anyone guide me to make it work?
UPDATE-3
<input type="text" id="add-keywords" name="add-keywords" title="e.g. Web developer, Javascript, Musician, Jazz" />
How to get the value of #add-keywords text box in the Django views.py. Does this work q = request.POST.get('add-keywords')?
Thanks!
the judgement request.is_ajax() returns False
The condition branches
try:
g = KeywordsModel.objects.filter(keyword__startswith=q).order_by('count')
except KeywordsModel.DoesNotExist:
return HttpResponse("")
else:
...
also could fail as ValueError if, for example, request.POST.get('q') results None
Plus, try '/keywords_suggestions/', note the suffix slash, instead of '/keywords_suggestions' in the serviceUrl:'/keywords_suggestions', line