How do I correctly test a web scraping library - javascript

I'm currently writing a library in typescript that retrieves videos from certain pages. I'm using jest to write the test but I'm facing the problem of testing the scraper since some content is dynamic and the library makes some post requests to store some cookies mocks are not feasible, what would be the correct way to test such a library?

Related

ReactJS vs NodeJS - Why do I need to create both?

I understand that React is frontend, and NodeJS is the backend that allows Javascript code to function outside of a browser. What I don't understand (and this is after following tutorials online on setting up a React project and a NodeJS project) is why I have to create an instance of each.
For example, in my React project, I managed to create a website. But because I needed a backend, I decided to use NodeJS. But I'm doing NodeJS tutorials, and I can create a website using NodeJS too. I'm confused because right now, it's appearing to be that React and NodeJS do the SAME THING.
I have never worked with NodeJS before so I am a bit confused. I was under the impression that I would just use NodeJS to host the backend, but after seeing that I'm literally having to create an entire website with NodeJS, I don't understand how I'm supposed to use React and NodeJS together.
How do the two, React and NodeJS, integrate together to create a fully-functioning web app? I have yet to see something online that clearly breaks down how the two interact.
React is front-end library. It provides great tooling for creatiing user interfaces. And it creates a single page application. Which means when you open a react app. It does not reload and its really fast.
While you could also use nodejs and something like handlebars to create a website. But that website would be rendered on server and then served to the user. But its alot more than that. There are a lot of things that you want to do on the server. Like authentication. You want that part to be secure. So you keep it on the server.
Now the answer to the final part of your question.
For a fully functional app. You would use react to create user interfaces. And nodejs for creating an API which your react app will call.
NodeJS is not just regular javascript, it is a javascript runtime that sits on top of a C++ engine called V8, provided by Google. Node executes outside the browser, whereas React / Vue / Angular / etc are in-browser javascript frameworks.
React is a whole separate animal; it is a framework that renders its own DOM in the browser. It is a javascript engine that is configured to optimize DOM manipulation.
While the development pattern of frontend and backend appear similar, they are doing different things. React is handling component lifecycles, applying dynamic style rules, processing in-browser data, and making API calls. Node is handling requests from the browser, coordinating access to the server's file system, managing network I-O, performing cryptographic evaluation, etc. Because of these different responsibilities, Node makes use of different dependencies (read: node modules) than a frontend framework.
Ultimately, Node and React communicate through HTTP calls (or, less frequently, through a WebSocket or SOAP protocol).
It would behoove you to read about how node works under the hood.
NodeJS is just a runtime that allows you to run javascript code outside of the browser.
In order to compile and transpile the react JS app, they use webpack and other tools which runs over NodeJS.
NodeJS will serve as your backend, whereas ReactJS will create the interface/UI where you can actually manipulate your server (nodeJS). So first you'll write your NodeJS server or API. You don't need to use ReactJS to create a frontend that would interact with your node server, like you said you can use NodeJS to create your views as well through a different library. ReactJS is just one choice of many for the front end of your NodeJS app.
The point is that react and any other SPA library is working on a client-side (browser).
React fetch and consume the data from the server API.
You don't need to use Node.js for building API. You can use various frameworks based on the technology you prefer.
If you are not familiar with the Back End, you can use https://www.npmjs.com/package/http-server to have a fake API service and can build the Front End part with it.
NodeJS is a javascript framework that allows you to create a server to serve up websites using Express or the built in libraries. It also is capable of building a website with just NodeJS.
You can take advantage of the ability to do server side rendering with a NodeJS server.
https://reactjs.org/docs/react-dom-server.html
There is a ReactJS framework called NextJS tha has server side rendering of ReactJS component.
https://nextjs.org/#features
You could potentially have some areas of your website that are built solely with NodeJS and other pages that use ReactJS and a NodeJS backend. But it is cleaner to use ReactJS for the front-end and NodeJS for the backend.

Integrate nodsjs as embedded website in wordpress

I need some help.
I'm kind of new to javascript, but I built a flight search single page application using nodejs and angularjs and the Skyscanner API.
Now I would like to integrate this application as an embedded website in a wordpress in order to enjoy the advantages of wordpress. Does anybody have experience with this? Do the two system compete? Whats the best way to do this?
As wordpress is based on php, I guess there should be some troubles, maybe you know a turnaround.
Thanks in advance!
WordPress itself is written in PHP but whatever frontend you build with it can access your API written in Node, either directly from Angular or indirectly via PHP.
If you can get your Angular app embedded in WordPress then you shouldn't have problems accessing your Node API from it (keep in mind that you may need to configure CORS correctly).
But whether you can get your Angular app embedded in WordPress may be a big "if".
You can either use an iframe in order to embed a page that you're running in your Node.js environment or just provide some APIs from your Node.js server and call them from WordPress the way you prefer (jQuery, Angular.js or whatever). I don't see any particular issue about running an Angular app in WordPress.

Run java code inside javascript

I want to build a site that calls a pi and displays the results. The problem is the API is a jar file that i normally import into a java project and go from there. However i want to build a web app but not sure i how can interact with my jar file.
I see some folks executing the jar in a command line but im not sure that the same, i want to interact with all the methods exposed to me via this java API by the jar file.
any ideas if there is a popular way of doing this or a framework i could use?
You can use spring MVC, you can build web apps there, consume the API jar file and do whatever you want in the views(frontend) with javascript and so on.
I recommend you use thymeleaf, which is pretty simple and easy.
If you want to build a fast app I also recommend you to use spring boot
You will build an app within minutes
https://spring.io/guides/gs/spring-boot/

How to test my selenium/jquery/AJAX project?

I am working on a tiny webapp project using ruby/sinatra. It is displaying a dashboard which is loading its data periodically using AJAX call to the REST API part of the app. (Basically the json results are transformed to some HTML on the page.)
I was driving the development of the REST API and the HTML using RSPec tests, but when I started with AJAX calls I couldn't use TDD.
So, how can I test the result of my javascript/AJAX calls? I would like to examine the resulting HTML.
I would like to avoid to use Selenium with browser to keep the project as lightweight as possible.
Cheers
Alex
The MockJax library (for example) will allow you to use TDD for your javascript and jQuery code. It will enable you to mock your AJAX calls, hence allowing you to test your client-side components in isolation.
As for testing the result of the AJAX calls, you could use the Capybara test framework; as well as supporting Selenium (which you want to avoid), it also supports a headless webdriver Capybara-webkit which should be "lighter" than Selenium as it does not load the entire browser.

Unit testing a modular Javascript web-app

I am building a web app using BackboneJS and RequireJS and need to implement some form of unit testing for UI interaction and data retrieval via AJAX. I have come across QUnit and Jasmine but don't really know how I can integrate this into my app.
If I am testing things such as:
Is the user logged in alright?
Has the data been received from the server ok?
Does clicking a button trigger the expected response?
Do click events work on dynamically loaded html content?
Does the app respond correctly to changes in hash/push-state urls?
I would imagine the testing has to be directly integrated into my app so as to have access to specific JS objects, work with session specific data and respond to changes in push state URLs.
How can I integrate QUnit or Jasmine (or other suggestions) into my modular app to unit test such features?
Unit testing is really simple.
You make a test HTML page. You include QUnit/NodeUnit/Jasmine/TestLibraryOfChoice
You then use requireJS and load one of your javascript modules,
and you simply test the exported object or function. That means testing the valid inputs of your module and asserting the outputs are correct.
You may have to mock out ajax and write HTML mocks
Dojo Objective Harness (DOH) is a very good unit test framework, which is browser agnostic and supports testing asynchronous functions, see here for a walkthrough guide.
However, from your test cases it looks like you want something more like an integration test?
If so Selenium is a good browser automation tool.
Crucially, neither of these tools will require you to modify your code (unless you find bugs :))
If you want to see an example where requireJS based modules are unit tested with QUnit, download the javascript reference architecture at http://boilerplatejs.org.
Disclaimer: I'm the main author of it.

Categories