I have a simple program that is scraping a web site for some items. I am using Angular $http service to call the below C# method to get the markup from the page and then handling everything else with JS. Everything is working perfectly fine with the exception of a minor annoyance: a bunch of 404 errors.
The 404 errors are being displayed in the developer tools once the http get call completes. It's almost like the javascript is trying to interpret the HTML and then fails on all the get requests for the images in the browser:
What I'm trying to figure out is how to get the 404 errors to go away or fail silently (not display in the console). I'm not finding anything in my research but am assuming there is some way to handle this whether it be on the server or client side
C#
public static string GetPageSource()
{
JObject result = new JObject();
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://awebpage.html");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
result["data"] = reader.ReadToEnd();
result["success"] = true;
reader.Close();
response.Close();
}
catch (Exception ex)
{
result["data"] = ex.Message;
result["success"] = false;
}
return JsonConvert.SerializeObject(result);
}
JS
$scope.getPageSource = function () {
var ajaxProcessor = Utils.ajaxMessage('Scraping Beer Menu From Source');
ajaxProcessor.start();
$http({
method: 'POST',
url: 'AJAX/MenuHandler.aspx/GetPageSource',
contentType: 'application/json; charset=utf-8',
dataType: 'json',
data: {}
}).then(function (response) {
ajaxProcessor.stop();
var result = $.parseJSON(response.data.d);
if (result.success === false) {
Utils.showMessage('error', result.data);
} else {
var beerMenu = new BeerMenu(result.data, $scope.loggedInUser, function (beerMenu) {
$scope.buildDisplayMenu(beerMenu);
});
}
}, function (err) {
ajaxProcessor.stop();
console.log(err);
Utils.showMessage('error', err.data.Message);
});
};
UPDATE
Thanks to #dandavis, my issue is narrowed down to calling $.parseHTML within the buildDisplayMenu function (which calls buildCurrentMenu). Is there anyway to make it ignore the images or any get request?
buildCurrentMenu: function () {
var html = $.parseHTML(this.pageSource);
var menuDiv = $(html).find('.TabbedPanelsContent')[0];
var categories = $(menuDiv).find('h2');
var categegoryItems = [];
var beerArray = [];
for (var i = 0; i < categories.length; i++) {
...
}
return beerArray;
}
The resolution is to remove any img tags (or any other tag that should be ignored) from the page source before calling $.parseHTML
this.pageSource = this.pageSource.replace(/<img[^>]*>/g, "");
Related
I'm trying to make a method call in client, but I get
Exception in template helper: Error: Syntax error, unrecognized expression: https://www.example.com
I'm using a Meteor method to parse a URL in XML to get some data, so my variable is a string URL
Method
rank: function(postAttributes) {
check(postAttributes, {
url: String
});
var alexa = require('alexarank');
var alexaSync = Meteor.wrapAsync(alexa);
var result = alexaSync({url: postAttributes.url});
return result.rank;
}
Client Call
function() {
var a = document.createElement('a');
a.href = this.url;
var post = {
url: $(a.href)
};
Meteor.call('rank', post, function(error, result) {
// display the error to the user and abort
if (error)
return throwError(error.reason);
// show this result but route anyway
return result;
});
}
Any idea?
Perhaps $(a.href) is not returning a String and the template executes the url as if it was javascript code.
Try using:
url: '' + $(a.href)
Whilst debugging, using Unit Tests, the function returns the expected data, however when the same function is called from JavaScript, the function is hit but then doesn't return any data.
This function that I'm calling that's in the dll is hanging, but only when it is called by a function that has been called by a JS request, why would this be?
EDIT:
As in comments, my best guess is that it is something to do with a thread being in use, but I don't know, as the function itself is working, just not when called from a C# function called by AJAX.
AJAX call :
function getOnHoldTickets() {
$.ajax({
type: "GET",
url: "/cloud/getTicketCount/",
dataType: "json",
success: function (data) {
onHoldHandler(data);
},
failure: function () {
alert("getOnHoldTickets failled");
}
});
}
Controller :
// api gets hit from the JS call
[Route("cloud/getTicketCount")]
public List<UberTicket> getTicketCount()
{
var tickets = Dashboard.getTODTickets("On Hold");
return tickets;
}
[TestMethod] // calls the same method as JS
public void supportTicketTesting()
{
var openTickets = Dashboard.getTODTickets("On Hold");
var check = openTickets != null;
}
// method calling the dll
public static List<UberTicket> getTODTickets(string type)
{
var tickets = UberAPI.getTODTickets(type);
return tickets;
}
DLL Method:
// the method within the dll that's hanging when called by a function invoked by JS
public static async Task<RootObjectClass<T>> genericGet<T>(string function, string parameters)
{
try
{
// create credentials to pass to httpClient
var httpClientCredentials = new HttpClientHandler()
{
Credentials = new NetworkCredential(uberAPIUser, uberAPIPass)
};
using (var client = new HttpClient(httpClientCredentials))
{
// unsure if the headers are being passed in correctly - getting good response though
client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "application/x-www-form-urlencoded");
var response = await client.GetAsync(ubersmithURL + function + parameters);
var result = await response.Content.ReadAsStringAsync();
// remove nulls from json
result = Regex.Replace(result, ":null,", ":\"\",");
var ms = new MemoryStream(Encoding.UTF8.GetBytes(result));
var success = JsonConvert.DeserializeObject<RootObjectClass<T>>(result);
return success;
}
}
catch(Exception ex)
{
// log error
}
return new RootObjectClass<T>();
}
While this might not be addressing the issue fully, you should not be returning C# Data types as JavaScript won't be able to parse them and will result in an error similar to this in the worst case
System.Collections.Generic.List`1[...]
Like I said in my comment, you should return a JsonResult from your controller to retrieve the data in JS.
[Route("cloud/getTicketCount")]
public JsonResult getTicketCount()
{
var tickets = Dashboard.getTODTickets("On Hold");
return Json(tickets ,JsonRequestBehavior.AllowGet);
}
Understanding JsonRequestBehavior.AllowGet
and your Ajax call
$.ajax({
type: "GET",
url: "/cloud/getTicketCount/",
dataType: "json"
})
.done(function(data){
console.log(data);
})
.fail(function(xhr){
console.log(xhr.responseText);
});
Edit:
I believe this is a deadlock issue you have. Perfect answer elaborating the issue is here.
I use jquery (ajax) to connect to a web service which returns string , it is not working with me. it always go to error function. here is my web service :
[HttpGet]
[ActionName("GetImage")]
public string GetImage(string base64String, string imgName,string reqTitle , string reqSubject, string reqStatus,string Creator , DateTime creationdate )
{
try
{
using (PhMobAppEntities context = new PhMobAppEntities())
{
ClaimsApproval _ca = new ClaimsApproval();
_ca.imageBasestrg = base64String;
_ca.imageName = imgName;
_ca.Creator = Creator;
_ca.CreationTime = creationdate;
_ca.ReqStatus = reqStatus;
_ca.ReqTitle = reqTitle;
_ca.ReqSubject = reqSubject;
context.ClaimsApprovals.Add(_ca);
context.SaveChanges();
return "Success";
}
}
catch (DbEntityValidationException ex)
{
var errorMessages = ex.EntityValidationErrors
.SelectMany(x => x.ValidationErrors)
.Select(x => x.ErrorMessage);
var fullErrorMessage = string.Join("; ", errorMessages);
var exceptionMessage = string.Concat(ex.Message, " The validation errors are: ", fullErrorMessage);
throw new DbEntityValidationException(exceptionMessage, ex.EntityValidationErrors);
}
}
and here is my js code :
$("#sendphoto").click(function () {
var url = "http://41.128.183.109:1212/api/Data/GetImage";
var data = {
imgName: "test"
};
$.ajax({
url: url,
type: 'Get',
data: data,
success: function (data) {
alert("Success");
},
error: function (data) {
alert("Please Check Your Internet Connection");
}
});
});
It is running ok when i tested my web service in advanced rest client ,please advice .
I tried connecting to your web service and I get the following response:
{"$id":"1","Message":"No HTTP resource was found that matches the request URI 'http://41.128.183.109:1212/api/Data/GetImage'."}
I think what you have is an internal problem with your c# code, probably with your routing. Your javascript call is probably working fine, but you are passing only one parameter, "test" while you have many more in your declaration.
What http response code are you getting?
I have a JavaScript client that works in Chrome and Firefox, but fails in IE. Looking at the network trace in the IE debugger it shows that multiple of the AJAX calls have been aborted.
I've been able to get around it by setting the timeout to 0. I'd like to know if this is the correct way to handle my requests being aborted? Basically what could go wrong?
My initial thought was that I should capture and resend on error, and if multiple resubmits do not result in a completed request, finally alert the user. I'd still like to know how to do this even if the setTimeout is the proper way to address my immediate issue.
Also the application will process an excel workbook of addresses, call a web service to add some data to them and then allow the user to download the enhanced file.
This is what I have so far, first in the app.js
var requestWithFeedback = function (args) {
$(".loader").removeClass('hidden');
var oldConfig = args.config || function () { };
args.config = function (xhr) {
xhr.setRequestHeader("Authorization", "Bearer " + localStorage.token);
oldConfig(xhr);
extract: extract;
};
var deferred = m.deferred();
setTimeout(function () { // <== This solved in IE, but is this the way to handle this?
m.request(args).then(deferred.resolve, function(err){
if (err === "Invalid token!"){
m.route('/');
}
})}, 0);
$(".loader").addClass('hidden');
return deferred.promise;
}
From the model.js
app.MarkedAddresses.ProcessAddressBatch = function () {
var requestData = {
Addresses: app.MarkedAddresses.vm.addresses
}
return requestWithFeedback({
method: "POST"
, url: "API/server.ashx"
, data: requestData
, deserialize: function (value) { return value; }
})
.then(function (value) {
var responseJSON = $.parseJSON(value);
$.merge(app.MarkedAddresses.vm.results, responseJSON)
app.MarkedAddresses.vm.currentRecord(app.MarkedAddresses.vm.results.length);
app.MarkedAddresses.vm.progress(Math.max(app.MarkedAddresses.vm.progress(), ~~(app.MarkedAddresses.vm.currentRecord() / app.MarkedAddresses.vm.totalRecords() * 100)));
m.redraw(); //Force redraw for progress bar
return value;
},
function (error) { console.log(error) } // <== I thought error would show up here, but I never hit a breakpoint here.
);
}
Added loops
function process_wb(wb) {
app.MarkedAddresses.vm.results.length = 0;
$('.descending').removeClass("descending");
$('.ascending').removeClass("ascending");
app.MarkedAddresses.vm.progress(.1);
m.redraw();
var header = mapHeader(wb);
var addressJSON = to_json(wb, header);
app.MarkedAddresses.vm.totalRecords(addressJSON.length);
for (var i = 0; (i < addressJSON.length + 1) ; i += 1000) {
app.MarkedAddresses.vm.addresses = addressJSON.slice(i, Math.min(((i) + 1000), addressJSON.length));
app.MarkedAddresses.vm.response(new app.MarkedAddresses.vm.processAddressBatch());
}
}
Why isn't the error triggered in the section of the code?
It seems like I should add a deferred section here, but anything I've tried has been a syntax error.
Good day everyone,
I am facing a slight problem.
Up until last week our mobile application connected to an in-house Web Api, which in turn connected to web services run and maintained by our partners.
Problem is, we would like the remove our Web Api as the middle man, and connect directly from our Cordova app (Javascript) to the Restful service.
Below is the C# code I am trying to emulate, any and all help would be appreciated:
(At this point I'm sure everything is right, except the authentication but I might be wrong)
QUESTION:
How can I achieve this in Javascript (If it is at all possible)
public static AuthenticateResult CheckLogin(LoginModel login)
{
var serviceClient = new WebClient();
var proxy = serviceClient.Proxy;
proxy.Credentials = new NetworkCredential("username_goes_here", "password_goes_here");
serviceClient.Headers["Content-type"] = "application/json";
try
{
var requestHeader = new UnauthenticatedRequestHeader
{
Code = ConstantModel.PartnerCode,
Partner = ConstantModel.PartnerName
};
var authenticateRequest = new AuthenticateRequest
{
Username = login.Username,
Password = login.Password,
Handset = "iPhone Emulator"
};
var serviceRequest = new
{
header = requestHeader,
request = authenticateRequest
};
var jsonizedServiceRequest = JsonConvert.SerializeObject(serviceRequest);
var requestBytes = Encoding.UTF8.GetBytes(jsonizedServiceRequest);
var requestStream = new MemoryStream(requestBytes);
var ms = requestStream.ToArray();
var responseBytes = serviceClient.UploadData("Url_goes_here", "POST", ms);
var jsonizedServiceResponse = Encoding.UTF8.GetString(responseBytes);
var authResult = JsonConvert.DeserializeObject<AuthenticateResponse>(jsonizedServiceResponse);
return authResult.AuthenticateResult;
}
catch (Exception ex)
{
return null;
}
}
What I have so far in Javascript is (this returns a Bad Request error):
btnTestClick: function () {
var header = {
Code: 'guid_goes_here',
Partner: 'partnerid_goes_here'
};
var request = {
Username: 'username_goes_here',
Password: 'password_goes_here',
Handset: 'iPhone Emulator'
};
var myrequest = {
header: header,
request: request
};
var string = JSON.stringify(myrequest);
var data = tobytearray(string);
$.ajax({
type: "POST",
url: "url_goes_here",
crossDomain: true,
data: data,
contentType: 'application/octet-stream;',
dataType: "json",
username: 'auth_username_goes_here',
password: 'auth_password_goes_here',
processData: false,
success: function (result) {
debugger;
},
error: function (jqXHR, textStatus, errorThrown) {
alert("error");
},
beforeSend: function (xhr) {
},
});
function tobytearray(str) {
var bytes = [];
for (var i = 0; i < str.length; ++i) {
bytes.push(str.charCodeAt(i));
}
return bytes;
}
}
What I see from your examples is the difference in the dataType used in the 2 examples.
In the C# example you use serviceClient.Headers["Content-type"] = "application/json"; and in the javascript implementation you use contentType: 'application/octet-stream;'.
octet-stream should be used for attachments and not JSON objects.
The webservice could be validating the request on the content-Type and if the test result is that your request is invalid it can return (and should return) a HTTP 400 - Bad Request.
I don't know the webservice where you shoot your requests at and what validations it does but try changing that contenttype in the javascript implementation to contentType: 'application/json;'.