This is my situation:
I have a third part that uses a software called microstrategy which is able to generate documents and allow to export them as PDF or Excel files. They provide me only web api of this product, and I haven't any web service to work with.
The url is like:
http://<third_part_domain>/microstrategy/asp/Main.aspx?Server=<third_part_domain>&Project=<project_name>&evt=3069&src=Main.aspx.3069&executionMode=3&promptAnswerMode=1&documentID=<doc_id>&uid=<username>&pwd=<password>&<other_parameters_for_request>
I have try to obtain the file (that I must save on server side) by java code, but the response of the link that we use is an HTML page with some code Javascript that does more than one redirect, so I can not interpreted correctly the response and I should use a browser to obtain the PDF.
So I have thought to put the page into a iframe and after a while (usually the server takes 20 second) take the PDF object by javascript code and send to my server. But obviously the third part have another domain and the CORS policies block everything. To make matters worse, I can not use the final url to obtain the file because the microstrategy respond me with an internal page of the administration console.
So, that's my question:
Is there a way (that is not on the microstrategy server side) to obtain directly the PDF from microstrategy?
Or exists a way from client side to bypass the problem of origin control? I have evaluated to implement a proxy for solution but it's too expensive.
Thanks to all!
You need two things in order to download a PDF from MicroStrategy using a URL:
In the document property set that default visualization as PDF. This is pretty trivial and I think any of your MicroStrategy savvy colleague can help you with this.
Disable the waiting page, this is more complicated. When MicroStrategy generates a documents, usually it needs some time, meanwhile the server is working it will show you a waiting page. Useful if the request comes from a human (the human can go on StackOverflow), not that much if the call arrives from API.
The instruction to disable the waiting page are here: TN34124: How to Disable the Wait Page in MicroStrategy Web using the MicroStrategy Web SDK 9.x.
But I read from your question that you have no control on the third party MicroStrategy application. In that case there is little you can do. You can try to ask them to implement the customization to remove the waiting page or allow you to use taskproc API, but that's a story for another day.
Some options:
Ask the third party to schedule the PDF generation on their side and send it via mail to you. Or place it on a shared folder that is shared between you.
Ask for a different URL Tuareg from the file-share menu options. This will give a URL with 'subscriptionid' in it.
Related
I have shared folder between in my server which will allow other server to send XML file to me and I want my script read this file auto without opening any page.
I know how to open and read the file.
But the issue how to auto load in the backhand.
you have to create a one page which will read the provided file and do the required actions , then share this URL and format with the team who will going to provide you the xml file.
It is very much like API Endpoint, Where you have to write the code which will handled request and in this scenario your Endpoint will treat as a server and XML file provider will treat as clients.
I hope this answer helps u.
Thanks
Traditionally, you need your server to periodically execute the script which reads the XML. That PHP will need to parse the XML and handle the changes.
Alternatively, the source of the API can use push notification to avoid polling with your server. The XML will be received whenever a change occurred on the server without the creation of a lot of useless requests, but the XML will be parsed as in the previous approach.
Last, but not least, you can use WebSockets for this purpose, or if both computers are in the same network, you can use sockets. Off course, a lot depends on the data source, whether you have access there, how modern is its technology and what does it allow you to do.
I am looking at express.js for the back end and JS on the client side.
My app is a single page Web App.
The server will only serve JSON messages and my question is about "routing" for express.
Is one supposed to use routing to connect the UI and the server side business logic?
How will that work with my single page app?
so lets say, the client makes an Ajax call to the server looking for a value in the database and there is server side script that provides the JSON back to the UI. How is this UI and node script relationship setup?
Can someone shed some light on this?
Single page apps are those that live on a single HTML document. This means that if you want to display some different content to the user, depending on the state of the application, you will need to do some DOM manipulation (cutting out and replacing certain elements of the current document with different HTML) in order to update the 'view' that the user sees. Excuse me if this is obvious to you, please don't take offense. I figured I'd start from here. Hang with me and I'll explain how your routing situation is going to play out (more or less).
URLs are composed of a few different parts, each of which informs the browser of a particular bit of information that is required in order to download the resource that the user is attempting to access. Typically the resources that you are looking for are off on a server somewhere and the browser knows this because of pieces in the URL like 'protocol' ('http:') and 'host' ('www.mydomain.com'), so it goes off to that server to find what you're requesting. There are also 'query' parameters in URLs which provide some additional information to the server regarding a particular action, like the search terms of a search query. After the query parameters, comes the 'hash'. The hash is where the magic of single page apps happens... eh, well, kind of.....
First a bit about the hash. When you add a '#' to a URL, the browser then interprets the information that comes after it to be some location (element) within the currently displayed document. That means, if you have an element with an 'id' of 'main' and you add '#main' to the end of the URL, like so: 'http: //www.example.com#main', the browser will 'scroll' (typically 'jump') to the beginning of that element, so that the you can see it. Be aware, though, that if you type 'http://www.example.com/#main' (with the hash separated from the URL by a slash) then you will force a complete page reload and the browser will attempt to find a file by the name '#main' on the server (I bet it doesn't find it).
The takeaway here is that the browser will not attempt to navigate away from the current document if there is a hash in the URL, the exception being of course the case mention above, and this is great because single-page apps don't want to navigate away from the page or request a new document from the server. (See how routing is different for single-page apps?)
Now, this whole thing about the hash isn't vital to single-page apps, as you could make one without dealing with it all. A bunch of click handlers and DOM manipulation is all you'd need really... But, that would mean that users will have no way of sharing links to particular views in your app. The URL would never change, and we would never be able to navigate to any particular view directly. We'd always be starting from the starting position of your app, which could easily be a very annoying situation.
If your single-page app is going to have different views, and you want users to be able to navigate directly to particular ones via bookmarks or links, then you will need to implement a form of routing on the front-end in addition to the routing that you'll need to implement on the backend (routing for data API, etc.), which means that you will need to make use of the hash.
I don't want to get into how different frameworks accomplish routing on the front-end, but it's basically a matter of updating the browser's address field when the user clicks a link, and watching the address bar to determine what the current URL is and loading the HTML that is associated with that URL into the DOM in the designated location in the document tree.
So, within a single-page app, you have one route on the server that deals with rendering the app HTML document (index.html), and you have routes that are responsible for dealing with the data of your app (creating new instances in the database, logging in and out, editing or destroying instances in the DB, and fetching data...) which are called via AJAX requests.
This is actually a fairly complicated situation in that HTML5 allows us to be able to forgo the hash (with the help of some link rewriting on the server) and also be able to use the 'back' and 'forward' buttons as if we've actually navigated away from the original document (which we haven't because we have only pointed the browser to the exact same URL, only with modified hash values, so no new page loads have occurred). Traditional site navigation and linking can be achieved by utilizing the browser's History API, which is available for IE beginning with version 10 (I believe), the rest of the big browser vendors were already on to it quite a bit earlier, so frameworks that leverage this technology will allow your users to navigate your app without the hash in the URL. Explaining this is a diversion and not necessary for understanding routing in single-page apps, but it is interesting and you'll have to learn it eventually anyway, probably..
AJAX should be used to request JSON from the server. AJAX requests will always hit your server because you don't include the hash symbol in AJAX requests (it would be ridiculous to do so because the hash is meant only for in-document browsing), so server-side routes must be responsible for exposing your data API (consider a RESTful one). While this is not their sole purpose in single-page apps, it is perhaps their most important one.
Soooo, to wrap it up, you will have two sets of routes. One on the client (as part of a client-side framework like AngularJS or EmberJS, the list goes on... I prefer Angular, but there is a fairly steep learning curve for that one.), and one on the server. When you think about 'server routes' think data API. When you think of 'page routing', remember that this all gets handled on the client, by the javascript that you delivered with the initial server response (this is the one and only necessary server-side route involved with rendering HTML to the browser, loading your 'index.html' and all of the necessary scripts and stylesheets, etc). You will use express.static middleware to serve static files, so you don't have to worry about assigning routes for that stuff.
EDIT A quick mention of AJAX implementation.
On the server, you will have routes similar those that Alex has provided as examples and you will make calls to those URLs from the client using whatever XMLHttpRequest (XHR) object is exposed by your framework or library of choice. It is now considered more or less standard and best practice for frameworks/libraries to implement these requests as Promises http://wiki.commonjs.org/wiki/Promises/A. You should read a bit about it on your own, but I might be able to summarize it by saying that it is an asynchronous operation analogous to 'try, catch, throw' in synchronous operations. You will instantiate a promise object and through it you will attempt to load data from the server, for instance, via GET request. Make sure that you have assigned functions to handle requests made to the URL that you made the request to (server-side route)! This object that you instantiate and subsequently make the request to the server through, promises to return the result of the request to you once it comes back from the server (no matter whether it was successful or not) If it is successful, it will call a function that you have written and will supply it with the data from the server. If it fails, it will call a different function, also written by you, and will supply it with the error object (or 'reason' for failure), so you can handle the error appropriately.
Hope that helped answer your question.
You only have to route requests you serve dynamically. Your HTML, CSS, JS are all static assets. So all you need to handling routing for is your data.
It sounds like you want a Restful API, which basically means that you have URLs for specific resources, and HTTP verbs for manipulating them.
Something like:
GET /books.json - Get all books
POST /books.json - Create a new book with properties passed in the body of the request
GET /books/123.json - Get book with id of 123
PUT /books/123.json - Update an existing book with properties passed in the body of the request
This blog post seems to show how to set this up in Express.
Once you have a sane API delivering JSON, you just make your AJAX calls use it based on what objects you want to fetch.
I'm looking for a method to scrape a website from server side (which uses javascript) and save the output after analyzing data into a mysql database. I need to navigate from page to page by clicking links and submitting data from the database,without session expiring . Is this possible using phpquery web browser plugin? . I've started doing this using casperjs. I would like to know the pros and cons of both methods. I'm a beginner in the coding space. Please help.
I would recommend that you use PhantomJS or CasperJS and parse the DOM with JavaScript selectors to get the parts of the pages you want back. Don't use phpQuery as it's based on PHP and would require a separate step in your processing versus using just JavaScript DOM parsing. Also, you won't be able to perform click events using PHP. Anything client side would need to be run in PhantomJS or CasperJS.
It might even be possible to write a full scraping engine using just PHP if that's your server side language of choice. You would need to reverse engineer the login process and maintain a cookie jar with your cURL requests to keep your login valid with each request. Once you've established a session with the the website, you can then setup your navigation path with an array of links that you would like to crawl. The idea behind web crawling is that you load a page from some link and process the page and then move to the next link. You continue this process until all pages have been processed and then your crawl is complete.
I would check out Google's guide Making AJAX Applications Crawlable the website you're trying to scrap might have adopted the scheme (making their site's content crawlable).
You want to look for #! in the URL's hash fragment, this indicates to the crawler that the site supports the AJAX crawling scheme.
To put it simply, when you come across a URL like this.
www.example.com/ajax.html#!key=value you would modify it to www.example.com/ajax.html?_escaped_fragment_=key=value. The server should respond with a HTML snapshot of that page.
Here is the Full Specification
I want to fetch particular HTML contents from remote websites url.
The website URL is as follow,
http://www.realtor.com/realestateandhomes-detail/10216-Montwood-Drive_El-Paso_TX_79925_M78337-06548
I want to fetch some specific information from above website url.
Here I attached image it highlight the specific area I want to all highlighted portion from there is a title,image, and descriptions.
How can I fetch the contents using JQuery or Javascript or Json call?
Is any other way to get these?
You might be interested in checking out pjscrape (disclaimer: this is my project). It's a command-line tool using PhantomJS to allow scraping using JavaScript and jQuery in a full browser context.
Scrapers can be written in straight Javascript, executed in the context of the site you're scraping, with a very simple, jQuery-friendly syntax.
It can scrape a single page, an array of pages, or you can define a function to look for more URLs to spider on each page.
It supports JSON and CSV output, either to file or to STDOUT
If the site is static and the structure is uniform, it should be very fast to scrape all the content you need into a structured data format.
This will help you out:
http://papermashup.com/use-jquery-and-php-to-scrape-page-content/
When scraping content, it is vital to consider the following:
Is the content static html or will part of it's content be rendered by ajax-calls?
In the first case, simple http-get-routines like the one used in JNDPNT's comment's Link will be sufficient.
In the second case, you may want to look at automating Selenium via it's Webdriver.
In any case it might be better to ask your colleague if he can provide you with an interface to the raw data, e.g. over a webservice.
If I'm getting you right, you want The user's Browser to scrape The content of another Domain on The Fly, right?
That will Not Be Possible without proxying The Request through some Script on The Same Domain (or via a jsonp Request to a Service that returns The HTML to you) due to The Same Origin Policy.
Sorry to disappoint.
Use the Yahoo Pipes (http://pipes.yahoo.com/pipes/ )service.
This can be used to grab and manipulate the page HTML, extracting the bits you want. Data can then be posted server side using the Web Service module or sent directly to the clients browser using an ordinary javascript callback.
There is this 3rd party webservice. One of the public webmethods available is a GetDocument() method. This method returns a Document object. The Document object has properties for File(byte[]), ContentType(string) ect.
My Question : Can I subscribe to this service using javascript(mootools) + ajax + JSON, return the document object, in this case an excel document, and force the file download?
It is true that typically you cannot initiate a download from JavaScript, but there is a flash component, Downloadify that does enable client side file generation.
So you can serve files for download from HTML/JavaScript.
With that problem solved, you still have the problem of how to get the data that you wish to serve from the source web service.
3rd party implies XSS (cross site scripting) which is a no-no using XmlHttpRequest (Ajax).
A possible solution to this problem could be to use a common hidden IFrame technique to get the data.
Simply have an appropriate (hidden?) form that correctly posts to the web service and point it's action to an hidden IFrame element upon which you are trapping the Load event and parse the data returned.
But current browsers have different levels of security measures that limit your ability to access IFrames with an external source so you are actually stuck here. Sorry to get your hopes up.
The only practical robust way to accomplish what you would like to do is to have a local server side script that can act as a proxy between your HTML/JavaScript and the external web service.
Using such a proxy, you can simply go back to using Ajax to get your data to serve up with Downloadify.
But then, since you are using a server script to get the data, why not just serve the data from the script for download?
These are just my observations on the problem domain you present.