MySQL Stream and piping data - javascript

I am using MySQL to get urls I have for a web scraper. Currently I get the data using csv-stringify as shown below:
conn.query('SELECT id, URL FROM company_table')
.stream()
.pipe(stringifier).pipe(process.stdout);
I see the data in the process.stdout but how can I use the data itself as I need to pass the url and id to another function for processing. I am unsure what I need to put in place of process.stdout for my needs.
The data basically looks like 38391,ysu.edu
I have another scraper where I stream in a csv file like
let fname = '/Users/tom/Scrappers/companies1k.csv';
fs.createReadStream(fname).pipe(csv()).on('data',
(row) => cluster.queue({url: `http://www.${row.domain}`}));
and that works fine but I just can't see how I pipe the db stream the same way to puppeteer-cluster.
Any ideas?

Related

Download Azure DevOps work item attachments using node js and azure-devops-node-api

Got stuck while trying to download an attachment of a work item (Azure DevOps).
I'm Using node.js 'azure-devops-node-api' client (https://www.npmjs.com/package/azure-devops-node-api) to interact with ADO API. I get a certain workItem using WorkItemTracking client (wit):
let workItem = await witApi.getWorkItem(1234, undefined, undefined, WorkItemExpand.All);
let attachment = await witApi.getAttachmentContent(attachmentId, fileName, projectId, true);
Documentation states that getAttachmentContent method downloads an attachment (https://github.com/microsoft/azure-devops-node-api/blob/ff820b2dd0c9a09cf09e64e94d3f95818a77249d/api/WorkItemTrackingApi.ts#L392), as a return value I'm getting a ReadableStream which i tried to write to a file using standard fs module:
fs.writeFile('WordDoc.docx', attachment, function (err) {if (err) return console.log(err);});
File is created but is empty. While debugging i also see that attachment variable has type ReadableStream but inside has lot's of properties and values, among them there is a buffer which i actually would like to extract and pass to fs.writeFile but can't reach them
What am i doing wrong ?
I believe you should write using WritableStream. As the getAttachmentContent is returning a ReadableStream. Below is the pseudo code. It might work
let readableStream = await witApi.getAttachmentContent(attachmentId, fileName, projectId, true);
let writableStream = fs.createWriteStream('./WordDoc.docx');
readableStream.pipe(writableStream);

How to get data from back end side, to use it in the browser side?

I am new to programming, and I heard that some guys on this website are quite angry, but please don't be. I am creating one web app, that has a web page and also makes som ecalculations and works with database (NeDB). I have an index.js
const selects = document.getElementsByClassName("sel");
const arr = ["Yura", "Nairi", "Mher", "Hayko"];
for (let el in selects) {
for (let key in arr) {
selects[el].innerHTML += `<option>${arr[key]}</option>`;
}
}
I have a function which fills the select elements with data from an array.
In other file named: getData.js:
var Datastore = require("nedb");
var users = new Datastore({ filename: "players" });
users.loadDatabase();
const names = [];
users.find({}, function (err, doc) {
for (let key in doc) {
names.push(doc[key].name);
}
});
I have some code that gets data from db and puts it in array. And I need that data to use in the index.js mentioned above, but the problem is that I don't know how to tranfer the data from getData.js to index.js. I have tried module.exports but it is not working, the browser console says that it can't recognize require keyword, I also can't get data directly in index.js because the browse can't recognize the code related to database.
You need to provide a server, which is connected to the Database.
Browser -> Server -> DB
Browser -> Server: Server provides endpoints where the Browser(Client) can fetch data from. https://expressjs.com/en/starter/hello-world.html
Server -> DB: gets the Data out of the Database and can do whatever it want with it. In your case the Data should get provided to the Client.
TODOs
Step 1: set up a server. For example with express.js (google it)
Step 2: learn how to fetch Data from the Browser(Client) AJAX GET are the keywords to google.
Step 3: setup a Database connection from you Server and get your data
Step 4: Do whatever you want with your data.
At first I thought it is a simple method, but them I researched a little bit and realized that I didn't have enough information about how it really works. Now I solved the problem, using promises and templete engine ejs. Thank you all for your time. I appreciate your help)

Express with JSON Data Control

I use lowDB dependency to control the JSON Data with Express and actually it works. But there is a bug and I cannot find how to solve it.
I create /create page to add information in JSON file and it contains 4 form and submit button.
And In express I code like this. each forms data will save it in variable and push with lowdb module.
router.post('/post', function (req, res) {
let pjName = req.body.projectName;
let pjURL = req.body.projectURL;
let pjtExplanation = req.body.projectExplanation;
let pjImgURL = req.body.projectImgURL;
console.log(pjName);
db.get('project').push({
name: pjName,
url: pjURL,
explanation: pjtExplanation,
imgurl: pjImgURL
}).write();
console.log(db.get('project'));
console.log(db.get('project').value());
res.redirect('/');
})
And it works well. But when I modify the JSON file myself (ex. reset the JSON file) and execute again. It shows the data that I reset before. I think in this app somewhere saves the all data and show save it in array again.
And When I shutdown the app in CMD and execute again, the Array is initialized.
As you may already know the lowdb persist the data into your secondary memory (hdd), and may return a promise depending on your environment when you call write method.As mentioned in the doc
Persists database using adapter.write (depending on the adapter, may return a promise).
So the data may be still getting write when you read them, so the old data is queried. Try this,
db.get('project').push({
name: pjName,
url: pjURL,
explanation: pjtExplanation,
imgurl: pjImgURL
}).write().then(() => {
console.log(db.get('project'));
console.log(db.get('project').value());
});

Streaming binary saved image from mongo to the response object

I have schema that take two fields - name and file.
file is Binary type and name is a String
I use this schema to save images as documents in my db
While the image is getting saved without issues - I am not sure how can I read it later. The main propose is to be able to search a file by its name using findOne and then to stream it to the response using pipe or any other solution that might work - to show the image to the client inside <img/> tag. I tried to convert the file to base64 and then to send something like res.send('data:image/png;base64,${file.file}') without much luck .
* Note that because I have inside the document two fields (name and file) I need to stream only the file for obvious reasons
This is my GET request to fetch the file :
router.get('/:url',(req, res) => {
const url = req.params.url;
console.log(url)
console.log('streaming file')
File.
findOne({ name:url}, 'file').
cursor().
on('data', (doc) => {
console.log('doc', doc.file);
res.send(doc.file)
}).
on('end', () => { console.log('Done!'); });
})
this not helps because its for streaming file with path which I dont have. My file stored in a db
Image of the saved file inside the db:

How to store values in MONGO DB using JAVA?

I want to make a blog post in my web application. Initially I was using mysql as DB. In which i will get the post entered in the text area of blog as an object in JS and send that object to java server side. There I will write mysql query and get the object in resultset and save in database. But now i want to use mongoDB for the same. I am able to understand the basic stuff through many tutorials which I learnt. But I am unable to implement that in my application. I want to know how the object that comes from the JS will be sent inside the loop and how I should query to save the object also similarly if I need to send a object from server side to JS. How should i do.?
My Server side code:
public DB MongoConnection(Blog blog) throws UnknownHostException, MongoException{
Mongo m = new Mongo("localhost" , 27017); //mongo object
DB db = m.getDB("myblog");
System.out.println("Connected");
//making a collection object which is table when compared to sql
DBCollection items = db.getCollection("items");
System.out.println("items got");
//to work with document we need basicDbObject
BasicDBObject query = new BasicDBObject();
System.out.println("Created mongoObject");
//Cursor, which is like rs in sql
DBCursor cursor = items.find();
System.out.println("items got");
while(cursor.hasNext()){
System.out.println(cursor.next());
}
In the above code i understood everything such as how a mongo connection, documents, collections and cursor's work. Now how should i save the value coming as an object from JS and save in mongoDB. Any Suggestions Please?
Use method save of DBCollection class something like that:
while(cursor.hasNext()){
DBObject doc = cursor.next();
doc.put("name", "Leo-vin");
items.save(doc);
}
method cursor.next() returns object of type DBObject. It is your BSONObject.
Update:
to modify document (BSON) use method put of class BSONObject

Categories