I would like to write a script to access data on a website, such as:
1) automatically searching a youtuber's profile for a new posting, and printing the title of it to stdout.
2) automatically posting a new video, question, or comment to a website at a specified time. For a lot of sites, there is a required login, so that is something that would need to be automated as well.
I would like to able to do all this stuff from the command line.
What set of tools should I use for this? I was intending to use Bash, mostly because I am in the process of learning it, but if there are other options, like Python or Javascript, please let me know.
In a more general sense, it would be nice to know how to read and directly interact with a website's JS; I've tried looking at the browser console, but I can't make much sense of it.
Python or Node (JS) will probably be a lot easier for this task than Bash, primarily because you're going to have to do OAuth to get access to the social network.
Or, if you're willing to get a bit "hacky", you could issue scripts to PhantomJS, and automate the interaction with the sites in question...
Related
A few days ago I decided to make my own "interface" to make it easier to organize (and work with) some of my personal files. You know when a lot of related data, pictures and links are right in front of you and you can change them in a couple of clicks, this is very convenient.
I started by studying HTML, CSS and JS, because I thought that the changes made to the local page would be saved somewhere on my PC so I can just run Index.html and do whatever I want. But they didn't. Refreshing the page erased all changes.
Using browser localstorage does not suit me, because if I change the browser, the data will be lost. I wanted it to just open with Index.html and work fine even if I change my browser or move the site folder to another computer.
Then I decided to learn more about server-side languages (such as PHP or Node.js) because they are directly related to databases, so I was hoping to save changes through them. But these languages required me to really open the server, with ip and port tracking. And I just wanted to open a local page through one file, without any ports or connections via the console. So this method scared me off quickly.
So is there an easy way to make a local page like this? Maybe I have not studied well one of the above methods and it has this opportunity?
Or the best I can hope for is a simple application that will use that local page as an interface to interact with data? I accidentally heard about this possibility a long time ago. Then I will ask you to give at least a hint as to which language to choose for this.
I don't understand well everything that lies outside of vanilla HTML, CSS and JS, so a complete study of a complex language like Java or Python will be too difficult for me, and the goal is not worth such a lot of effort.
I hope I understand correcly what you are trying to do.
If your goal is to make an application to manage your files, I think the simplest solution will be, as you said, to look into NodeJS and the File system api which will let you interact with your files through javascript code.
Your program will have to be in two part that will have to interact:
the "front" html page
the "back" nodejs script
The downside is that you'll have to go deeper into your study of the language to learn how to create the interactions you want between your html file and your NodeJS application.
However, there is no need to open your server to the web to make it work. The NodeJS application can be set to listen to requests from only the computer that runs it (localhost).
I obviously can't get too much into details without knowing precisely what you want to do but you'll probably have to learn to make a local server with node (search "nodejs http" or "nodejs express"), then make requests to it via the html page's scripts (search "ajax request").
What you need to look into are (web based) content management systems. like strapi or "grand old dame" WordPress.
I am creating software that helps me and my colleagues to work on tickets at work, as our current software is really bad. At this moment I am using program that I wrote in Tampermonkey, it is giving information about queue and all other fancy stuff. But I reached limit of that, as website is a limitation, so I started to create website that could do much more, yet I am not sure how to connect it as when I am using Tampermonkey to create overlay with all needed data. My idea was to: 1) Gather all data by tampermonkey and send it to nodejs server. Node would keep array (that's all I need) locally and then I would be able to access it.
Is this possible, is this a correct approach? Unfortunately I am limited by options that I can use by company security policy.
I have created a website for a third party, who have no experience in editing HTML. However, the third party wishes to be able to edit the content on the website without having to open the files and edit it this way, they wish to do it somewhat WYSIWYG (For example, hit an "edit" button for the content they wish to edit). Is this possible to achieve? It is not an internal website, it has user tracking (this should obviously only be available to admin users).
Is there a way of making contents of a div editable, then saving the change directly to the server, so the content gets updated publicly?
I am currently researching the topic, and although I have found some indications that the solution may be a PHP script, I have yet to find any definitive solutions or examples of similar functionality.
Yes you will need a backend language or framework to archive this. Where Javascript is used to interact with the page, the actual storage of information requires a database or similar technology.
Unfortunately which backend language or framework to choose really is the million dollar question. It largely depends on exactly what you are trying to accomplish, what your client or user is comfortable with, and how much experience you have programming.
PHP is fast and time tested backend language. Node is the new kid on the block, and it very popular also. Java and dotNet are on the way out. You can dig up a bunch more including Go, Python, Haskel, Etc.
You can use a languge listed above and start scripting away, but this can be time consuming and error prone. Most people use a framework to get started, and program using that framework's tools. The most popular PHP framework is WordPress, but it is designed for blogs and might not fit your use case. I use the framework Craft CMS which is very customizable. But the way you are phrasing the question a framework might be overkill. This is really up to you to decide after doing research into the available options and comparing them to what you wish to accomplish.
For the WYSIWYG, you might want to look into the following tools for the client to edit content:
https://imperavi.com/redactor/
https://ckeditor.com/
Hopefully this provides some direction, happy coding!
Let me preface by saying I have no idea of what I'm doing. I've inherited a system from a contractor that we hired to do a job. I'm not significantly familiar with web development, and I have no idea how the magic voodoo was configured or really works. If you're going to reply, be patient, and assume I don't know jack about what you're telling me - please don't leave anything "for the reader to figure out." I'm embedded by trade and would rather bang bits than develop back end code for a website.
Server is running on an embedded Linux platform (basis is OpenWRT). The core is Lua/Luci, but there's a plethora of HTM files that utilize both HTML and JavaScript.
What I want to do seems really, really straight forward, but I can't seem to make it work: There is a file in /etc that I want to be able to download from the server to the local machine. Needs to work with IE, Firefox, and Chrome.
I would have loved something like:
download
But it doesn't work for files outside the subdirectory area that lua/luci knows about (i.e. I can't "../../../etc/file")
I've tried several different things, but the biggest issue is I can't seem to get the lua/luci stuff to recognize anything new in the same directory that contains some of the htm files, nor anything from the server's root directory (e.g. /etc/file). Usually what I do goes back to the home page or displays:
No page is registered at '/admin/talon/file'.
If this url belongs to an extension, make sure it is properly installed.
If the extension was recently installed, try removing the /tmp/luci-indexcache file.
(And yes, I clear the cache before I reload the page).
I'm OK with creating a symlink to the /etc/, but that hasn't been fruitful, either - mainly because I really don't know what kind of magic the lua service is doing to find the existing files.
I'd prefer for the solution to be in just HTML and JavaScript.
Yes, I've looked around for a basic solution, but either the questions want to do more than just download, or there's not enough information for me to figure out what is supposed to be done.
Please post a full solution, not just snippets.
I was able to figure it out based on some other code within that same source. It worked on one page, but not another. Not sure why - just more sorcery. Had to work within the LUA scripting language to get to the file I wanted; HTML was straight forward. If I knew what the magic thing was to make it work, I'd post the actual solution, but I think the solution is somewhat unique to how the original developer put things together, so it wouldn't be useful to anyone else.
I'm working right now on a project that could allow me to generate movies based on the user input. User will upload some samples (photos, movies) to the web app and web server should generate movie based on that input and some predefined movie compositions.
I know that there are plenty of libraries for ffmpeg that could let me connect movies, photos programmatically (for example https://github.com/schaermu/node-fluent-ffmpeg for node.js) , but I was wondering if it's possible to use Aftereffects for that purpose since I have some knowledge in that software. I imagine that there should be set of scripts in Aftereffects that could import user uploaded data, fire the movie renderer and save output to the given location.
Do you think this is achievable using Aftereffects? Or maybe someone had similar problem and solved that differently ?
Cheers!
I have done the exactly same thing.
I DO NOT suggest you use script to do it. I have made the same mistake. Script is fine for a small job, but when you try to use it on a web server and run constantly for days and days it's very unstable. You will be facing a lot of crashing.
I would suggest you use sdk to do the job. It's much more difficult to use sdk than using script, but is more stable and much faster! When you try to create a web service app, you want it to be stable and fast, don't you?
Yes, this is definitely possible. There are existing scripts for rendering and uploading via FTP, and the possibilities are pretty much endless. The part that jumps out at me as needing clarification is "scripts in Aftereffects that could import user uploaded data". This suggests a need for a back-end that "looks for" or "waits for" elements to "appear" in a directory to kick off the ExtendScript script in AE. This is where it gets slightly dicey in that you have to devise a way to do this with a "daemon" in your preferred operating system using any number of languages -- python, Java, AppleScript, shell, batch, etc. The rest of it ("import user uploaded data, fire the movie renderer and save output to the given location") could be done in ExtendScript.