We Keep Coding

Scrapy: POST request returning JSON response (200 OK) but with incomplete data

MySpider is trying to depict the load-more action click which results in loading of more items on web-page dyanamically. And this continues until nothing more is left to be loaded.
yield FormRequest(url,headers=header,formdata={'entity_id': '70431','profile_action': 'review-top','page':str(p), 'limit': '5'},callback=self.parse_review)
header = {#'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0',
'X-Requested-With': 'XMLHttpRequest',
'Host': 'www.zomato.com',
'Accept': '*/*',
'Referer': 'https://www.zomato.com',
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'dont_filter':'True' }
url = 'https://www.zomato.com/php/social_load_more.php'
The response received is the json response.
jsonresponse = json.load(response)
And i do see -
('data==', {u'status': u'success', u'left_count': 0, u'html': u"<script type='text/javascript'>if (typeof initiateLaziness == 'function') initiateLaziness() </script>", u'page': u'1', u'more': 0})
U see i get response for status, left_count, page, more.
However i am interested in 'html'. Unfortunately, its the in-correct value which i do receive if done through browser(inspected the network calls and verified)
Expected 'html' is ----
<div><a> very long html stuff...............................................<div><script type='text/javascript'>if (typeof initiateLaziness == 'function') initiateLaziness() </script>
I am receiving only later part
<script>...................................</script>.
Real html stuff is missing.
Thing to note is that i do receive response but incomplete one for 'html' only.All good for rest. I believe it might be something related to dynamically generated html. But i am getting any clue on it.
No content-length is added by scrapy middleware. And not allowing me to add one as well. Respons fails with 400 when adding it to header.
Request Header being actually sent to server:
{'Accept-Language': ['en'], 'Accept-Encoding': ['gzip, deflate,br'], 'Dont_Filter': ['True'], 'Connection': ['keep-alive'], 'Accept': ['*/*'], 'User-Agent': ['Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0'], 'Host': ['www.zomato.com'], 'X-Requested-With': ['XMLHttpRequest'], 'Cookie': ['zl=en; fbtrack=9be27330646d24088c56c2531ea2fbf5; fbcity=7; PHPSESSID=2338004ce3fd540477242c3eaee685168163bd05'], 'Referer': ['https://www.zomato.com'], 'Content-Type': ['application/x-www-form-urlencoded; charset=UTF-8']})
Can any one please help me if i am missing anything here?
Or someway i can sent the send the content-length/or make middleware sent it for me?
Many Thanks.

You won't get the html content in response because of not using cookies. In the actual request header that you have mentioned, there is a cookie attribute. But in the ajax request you are sending through your code, there is no cookie field.
First a cookie is set in the response to the request made from zomato's restaurant page with the url: https://www.zomato.com/city/restaurant/reviews. Now, when the load more button is clicked, a request is sent with the cookie field containing the cookie set by the server in the previous response to the url 'https://www.zomato.com/php/social_load_more.php'. So, everytime an ajax request is made, the cookie that was set in the previous response should be sent in the request header and a new cookie will be set in the response of the present request.
So, in order to manage these cookies, I used session object of requests package. The script can be written without using scrapy also. As you wrote your code in scrapy, see if there are any session objects available to manage the cookies for scrapy.
My code :
import requests
url : 'https://www.zomato.com/city/restaurant/reviews'
s = requests.Session()
resp = s.get(url, headers=header)
The above code is to send requests to the url of the restaurant reviews. This is essential because the first cookie is set in the response to this request.
params={
'entity_id':res_id,
'profile_action':'reviews-dd',
'page':'1',
'limit':'5'
}
header = {"origin":"https://www.zomato.com","Referer":"https://www.zomato.com/","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0", "x-requested-with":"XMLHttpRequest", 'Accept-Encoding': 'gzip, deflate, br'}
loadreviews_text = s.post("https://www.zomato.com/php/social_load_more.php", data=params, headers=header)
loadreviews = loadreviews_text.json()
Now a request is made to the social_load_more.php. The object 's' manages the cookies. The variable loadreviews will now have the html data in json format.

ngResource PUT returns bad request due to $promise and $resolved in the JSON object

I've been struggling with this problem for the last few hours, and every tutorial points toward the solution that I have implemented but it doesn't work.
Basically my PUT request returns an error:
PUT http://localhost:8083/stockapi/rest/stocks/5485cba248673a0dd82bb86f 400 (Bad Request)
When I intercept the request, I see that it contains a $promise and $resolved data element:
> {"id":"5485cba248673a0dd82bb86f","name":"iShares ESTOCK DivXXX","ticker":"AMS:IDVY","url":"https://www.google.com/finance?q=AMS%3AIDVY&ei=F5BxVLiCB8GlwQPJ1YD4DQ","currency":"EUR","currentPrice":19.81,"currentPriceInEuro":19.81,"lastModified":1418054562234,"historyStockPrices":[{"timestamp":1418054562234,"price":19.81}],"$promise":{},"$resolved":true}
This makes sense since I'm using the ngResource object -- but every tutorial shows that the following code should be able to handle it, but it doesn't.
Note/edit: if i PUT the JSON object without the "$promise" and "$resolved" elements through an external program (such as Postman REST client) then it works fine.
Factory:
.factory('Stock',function($resource){
return $resource('http://localhost:8083/stockapi/rest/stocks/:id',
{ id: '#id' },{
update: { method: 'PUT' },
show: { method: 'GET' }
}); });
Controller (note: doing 4 updates but none of them work, 4 times the same Bad Request):
.controller('StockEditController',function($scope,$log,$http,$state,$stateParams,Stock){
$scope.stock = Stock.get({id:$stateParams.id});
$scope.updateStock=function(stock) {
Stock.update(stock);
stock.$update();
Stock.update($scope.stock);
$scope.stock.$update();
$state.go('stocks');
};
});
I'm really clueless right now how to use the ngResource object in the correct way so that I can use it to put/post to my webservice. Any ideas?
Thanks!
EDIT:
Chrome network output:
Response header
Remote Address:[::1]:8080
Request URL:http://localhost:8080/stockapi/rest/stocks/5485cba248673a0dd82bb86f
Request Method:PUT
Status Code:400 Bad Request
Request Headersview parsed
PUT /stockapi/rest/stocks/5485cba248673a0dd82bb86f HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 355
Pragma: no-cache
Cache-Control: no-cache
Accept: application/json, text/plain, */*
Origin: http://localhost:8080
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36
Content-Type: application/json;charset=UTF-8
Referer: http://localhost:8080/stockapi/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Request Payloadview parsed
{"id":"5485cba248673a0dd82bb86f","name":"iShares ESTOCK DivXXXYYY","ticker":"AMS:IDVY","url":"https://www.google.com/finance?q=AMS%3AIDVY&ei=F5BxVLiCB8GlwQPJ1YD4DQ","currency":"EUR","currentPrice":19.81,"currentPriceInEuro":19.81,"lastModified":1418054562234,"historyStockPrices":[{"timestamp":1418054562234,"price":19.81}],"$promise":{},"$resolved":true}
Response Headersview parsed
HTTP/1.1 400 Bad Request
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 968
Date: Tue, 09 Dec 2014 06:36:24 GMT
Connection: close

According to the docs you are not quite using the update action correctly.
So really your updateStock method should be:
$scope.updateStock=function(stock) {
Stock.update({id: stock.id}, stock);
Stock.update({id: $scope.stock.id}, $scope.stock); //not sure why you have 2 calls here
$state.go('stocks');
};

Just looking at the factory definition and how it differs from mine, you should change that second parameter of factory definition into array to something like this:
.factory('Stock',[$resource, function($resource){
return $resource('http://localhost:8083/stockapi/rest/stocks/:id',
{ id: '#id' },{
update: { method: 'PUT' },
show: { method: 'GET' }
});
}]);
Not sure if this is the issue but that is at least significantly different to angular docs' definition on dependency injection here: https://docs.angularjs.org/guide/di
If not, could you also print out the contents of $scope.stock after the GET returns the data there?

I know this question is super old, but I stumbled upon the same issue and found this SO question as well as this question:
http://www.scriptscoop2.com/t/fddc3f0a1f6f/angularjs-using-ngresource-for-crud-adds-extra-key-value-when-using-save.html
There was 1 answer which explained that the issue is coming due to CORS (cross origins). Which means that your server side needs to allow it. If you are using Spring MVC it would be enough to add the org.springframework.web.bind.annotation.CrossOrigin annotation at the controllers request mapping:
#CrossOrigin
#RequestMapping(value = "/product/{id}", method = RequestMethod.POST)
#ResponseBody
public void editProduct(#PathVariable("id") final long id, #RequestBody final ProductDto productDto) {
// your code
}

how to configure the http head in jQuery post function with spring mvc

I have create a page and submit a post request to server, the jquery code like this
jquery code：
$("#button").click(function() {
var book = {
name: "book123",
isbn: "123456"
};
$.post("book/add", book, function(data) {
console.debug(data);
});
});
Spring controller code:
#RequestMapping(value = "/add",method = RequestMethod.POST)
public #ResponseBody String addBook(
#RequestBody Book book, HttpServletResponse response) throws Exception {
book.setRanking(DEFAULT_RANKING);
Book returnbook = bookService.createBook(book);
return mapper.writerWithDefaultPrettyPrinter().writeValueAsString(returnbook);
}
But it has error, in firebug, I get this error code:
NetworkError: 415 Unsupported Media Type - /book/add
the requst header information is this:
Accept / Accept-Encoding gzip, deflate
Accept-Language en;q=0.8,en-us;q=0.5
Content-Length 24
Content-Type application/x-www-form-urlencoded; charset=UTF-8
DNT 1
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101
Firefox/24.0
X-Requested-With XMLHttpRequest
I think the content-type is not correct as in bold. How to resolve this issue? Thanks!

You could use jQuery's ajax function instead of post, where you can specify the type.
To see how to use it, check out this SO post
Also it would be good to set the consumes variable on #RequestMapping to "application/json"

Why do I get the error 'not allowed by Access-Control-Allow-Origin' after I use setRequestHeader?

After searching for a while, I cannot find the answer yet.
My problem is when I call a web service function setRequestHeader, I got the error "not allowed by Access-Control-Allow-Origin".
Here is my javaScript code:
var loginController = new sap.ltst.login.loginController({controllerName: "sap.ltst.login.loginController"});
var session = loginController.login("I051486", "123456789");
var config = {};
$.ajax({
beforeSend: function(req) {
req.setRequestHeader('Authentication', 'Authentication-Token ' + session.session_token);
},
url : "http://localhost:8081/com.sap.st.gtpapi/program/"
+ this.program + "/configs",
dataType : 'json',
type : 'GET',
async : false,
success : function(data) {
config = data;
}
});
return config;
In web service side, I have a function that I can enable or disable the authentication. I tried to set the auth as false (not check the auth) then remove setRequestHeader, I got no error and the web service returns me some data.
In another way I tried to put it back, I got the error.
XMLHttpRequest cannot load http://localhost:8081/gtpapi/program/Business%20Intelligence%20platform%204.1%20(BI%20Aurora%204.1)/configs. Origin http://localhost:8080 is not allowed by Access-Control-Allow-Origin.
So I don't think that it's the problem of the auth because in the web service side, I disable the auth verification.
Let's move to the web service side, this is the interface:
public static final String HEADER_AUTH_TOKEN = "Authentication-Token";
#GET
#Consumes({MediaType.APPLICATION_JSON})
#Produces({MediaType.APPLICATION_JSON})
#Path("/guid_{planId}/packages/{packId}/{results}")
public Response setPackageResult(#PathParam("planId") final String planGuid, #PathParam("packId") final String packGuid, #PathParam("results") final String results, #HeaderParam(WebServiceBase.HEADER_AUTH_TOKEN) String token);
This is the header response et request on Chrome:
Request URL:http://localhost:8081/com.sap.st.gtpapi/program/SBOP%20EXPLORER%204.1/configs
Request Method:OPTIONS
Status Code:200 OK
Request Headersview source
Accept:*/*
Accept-Charset:UTF-8,*;q=0.5
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8,fr;q=0.6
Access-Control-Request-Headers:accept, authentication, origin
Access-Control-Request-Method:GET
Cache-Control:no-cache
Connection:keep-alive
Host:localhost:8081
Origin:http://localhost:8080
Pragma:no-cache
Referer:http://localhost:8080/LTST_Frontend/index.html
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
Response Headersview source
Allow:GET,OPTIONS,HEAD
Content-Length:0
Content-Type:text/xml
Date:Fri, 02 Aug 2013 15:44:10 GMT
Server:Apache-Coyote/1.1
I'm not sure I did some mistake whether the problem comes from javaScript or web service. Any ideas?

Put any header you want to send in the safe list:
header("Access-Control-Allow-Headers: Authentication, X-Custom-Header, .. etc");
This should be part of the CORS headers on the receiving domain.

How to get the data via ajax in servlet?

I would like to send some data on the page to servlet
so I have written following jquery to do this
I use all data to build a json string, and directly send it to servlet
but I don't know how to get the whole data from the ajax in servlet
$("#save").click
(
function()
{
$.ajax
(
{
url:'/WebApplication1/Controller',
data:'{"name":"abc","address":"cde"}',
type:'post',
cache:false,
success:function(data){alert(data);},
error:function(){alert('error');}
}
);
}
);
if see the the Form Data segment of request headers from chrome
you will see the whole json string is the key.
Request URL:http://192.168.0.13/WebApplication1/Controller
Request Method:POST
Status Code:404 Not Found
Request Headersview source
Accept:*/*
Accept-Charset:Big5,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:zh-TW,zh;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:112
Content-Type:application/x-www-form-urlencoded
Host:192.168.0.13
Origin:http://192.168.0.13
Referer:http://192.168.0.13/system_admin/building.html
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.910.0 Safari/535.7
X-Requested-With:XMLHttpRequest
Form Dataview URL encoded
{"name":"abc","address":"cde"}:
Response Headersview source
Accept-Ranges:bytes
Connection:Keep-Alive
Content-Language:en
Content-Type:text/html; charset=iso-8859-1
Date:Wed, 15 Feb 2012 12:37:24 GMT
Keep-Alive:timeout=5, max=100
Server:Apache/2.2.14 (Win32) DAV/2 mod_ssl/2.2.14 OpenSSL/0.9.8l mod_autoindex_color PHP/5.3.1
Transfer-Encoding:chunked
Vary:accept-language,accept-charset

Look here,
data:'{"name":"abc","address":"cde"}',
Your data attribtue is wrong. It should not be a string, but a real JSON object. Remove those singlequotes.
data:{"name":"abc","address":"cde"},
This way it's available in the servlet the usual way
String name = request.getParameter("name"); // abc
String address = request.getParameter("address"); // cde
If it still doesn't work, head to How should I use servlets and Ajax? for complete kickoff examples, just in order to exclude that your actual problem is caused elsewhere.
See also:
jQuery.ajax() documentation

you can send data by ajax without any form, you can do it via GET or POST, but you have to invoke in some way the function that does the ajax petition (http request).
For example you can call your function from a link, this way:
the html:
<a href="" onClick="javascript: yourAjaxCallFunction(parameters)" >Link Text</a>
the javascript:
function yourAjaxCallFunction(parameters)
{
//this function sends the request via jquery ($().ajax)
sendHttpRequest(parameters);
//this one handles the response (process data)
processResult();
}
this link is maybe what you're looking for
ajax function in jQuery documentation