Hi there: I'm wondering if anyone knows a method for listing all files inside of an S3 bucket with node.js and Awssum, starting with the most recent file first.
By default, my code takes the first created file.
function getFileList(callback){
s3.ListObjects(coldBucket, function(err, data) {
var fileList = getFilenames(data, coldBucketPath);
callback(fileList);
});
};
Any help is much appreciated! Thanks.
this is Andy, creator of AwsSum.
That's correct, there is no way of asking S3 for the filenames in order of inserted time. However, you can get it using the LastModified.
If you'd like to see an example to get all the keys, see this example here:
https://github.com/appsattic/node-awssum-scripts/blob/master/bin/amazon-s3-list.js
You could then store each item in an array, then sort it using LastModified. The underscore library contains a function that would be useful for this:
http://documentcloud.github.com/underscore/#sortBy
I looking at http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGET.html, I don't believe s3 allows you to specify the sort order of the response.
However, you may be able to work around that by changing your file structure to include folders named after the date they were created, then you could do a couple of ListObjects requests with prefixes for recent days/months/etc.
For example, you could have year-month folder names so your file structure looks like this:
bucket_name/
- 2012-3/
- file2.jpg
- file3.jpg
- 2012-2/
- file1.jpg
and then before you make your request, do this
var now = new Date();
coldBucket.Prefix = now.getFullYear() + "-" + now.getMonth();
And then if you wanted older stuff you'd have to do a followup request for previous months.
(If you have too many files even for that, you could try year-month-day)
Related
I have the following UseCase:
A creates a Chat and invites B and C - On the Server A creates a
File. A, B and C writes messages into this file. A, B and C read this
file.
I want a to create a file on server and observe this file if anybody else writes something into this file send the new content back with websockets.
So, any change of this file should be observed by my node.js application.
How can I observe files-changes? Is this possible with node js without locking the files?
If not possible with files, would it be possible with database object (NoSQL)
Good news is that you can observe filechanges with Node's API.
This however doesn't give you access to the contents that has been written into the file.
You can maybe use the fs.appendFile(); function so that when something is being written into the file you emit an event to something else that "logs" your new data that is being written.
fs.watch(): Directly pasted from the docs
fs.watch('somedir', function (event, filename) {
console.log('event is: ' + event);
if (filename) {
console.log('filename provided: ' + filename);
} else {
console.log('filename not provided');
}
});
Read here about the fs.watch(); function
EDIT: You can use the function
fs.watchFile();
Read here about the fs.watchFile(); function
This will allow you to watch a file for changes. Ie. whenever it is accessed by some other processes of any kind.
Also you could use node-watch. Here's an easy example:
const watch = require('node-watch')
watch('README.md', function(event, filename) {
console.log(filename, ' changed.')
})
I do not think you need to have observe file changes or use a NoSQL database for this (if you do not want to). My advice would be to look at events(Observer pattern). There are more than enough tutorials on this topic available online (Google). For example Felix's article about Using EventEmitters
This publish/subcribe semantic can also be achieved with NoSQL. In Redis for example, I think you should have a look at pubsub.
In MongoDB I think tailable cursors is what you are looking for. On their blog they have a post explaining pub/sub.
I have some .bqy (Brio query file) files I need to loop through them and read the metadata which is the OCE (database connection strings) and write them to an excel file.
Unfortunately, a bqy doesn't know the information contained in the oce. It pretty much only knows what's in the Connection Manager tool (thing you get when you press F11), even if that information is wrong and won't connect.
The official Oracle documentation says the data you're seeking might be in the bqmeta0.ini file, if it's provided by a third party vendor:
https://docs.oracle.com/cd/E17236_01/epm.1112/ir_user/ch36s09s01.html
There are a lot of read only Properties under the DataModel object, which might be useful:
https://docs.oracle.com/cd/E17236_01/epm.1112/ir_user/ch30s18.html
https://docs.oracle.com/cd/E17236_01/epm.1112/ir_user/ch30s17.html
Desktop Hyperion - Create a BQY and use For loops to open each bqy then loop thru the object model. You can write output to excel or access. Something like this
// Request Loop
for (var z = 1; z <= ActiveDocument.Sections[j].Requests.Count; z++) {
// get_data
}
Apologies if this seems basic to some, but I'm new to JS/node.js/JSON and still finding my way. I've searched this forum for an hour but cannot find a specific solution.
I have a basic website setup running of a local Node.js server along with 2x JSON data files with information about 32x local suburbs.
An example of an API GET request URL on the site would be:
.../api/b?field=HECTARES
The structure of the JSON files are like:
JSON Structure
In the JSON file there are 32x Features (suburbs), each with it's own list of Properties as shown above. What I am trying to do is use the API 'field' query to push all the HECTARES values each of the 32x Features into a single output variable. The code below is an example of how far I have got:
var fieldStats = [];
var fieldQ = req.query['field'];
for (i in suburbs.features) {
x = suburbs.features[i].properties.HECTARES;
fieldStats.push(x);
}
As you can see in the above "HECTARES" is hard-coded - I need to be able to pass the 'fieldQ' variable to this code but have no idea how to.
Advice appreciated!
Exactly the same syntax you are using just above:
suburbs.features[i].properties[fieldQ];
for (var i = 3848450; i > 3848400; i--) {
var query = {
url: 'http://classifieds.rennug.com/classifieds/viewad.cgi?adindex=' + i,
type: 'html',
selector: 'tr',
extract: 'text'
}
,
uriQuery = encodeURIComponent(JSON.stringify(query)),
request = 'http://127.0.0.1:8888//?q=' +
uriQuery + '&callback=?';
jQuery.getJSON(request, function (data) {
var datastring = data[0].results;
var datasplit = datastring.toString().split('Sign');
$('#inner-content').append(datasplit[0]);
});
}
I want to listen for new URLs of ads that are posted without writing some kind of arbitrary code that takes up a lot of memory looping through new URLs, etc. I can do that but it seems redundant and such as my code listed above. Im using noodle.js to get the info from the pages. Now I would like a way to listen for new urls instead of looping through every possible url from a to z. Since I don't know z it's a safe bet I'll be using an if statement but how would one go about incorporating this nth URL without ending up with undefined iterations. Im still learning and find this place has many helpful people. This is simply a fun project I'm doing to learn something new.
If I understand you correcly, you want an external thing to inform your javascript when there's new a URL or JSON data
Unfortunately the web is not built for servers to contact clients, with one exception to my knowleadge: WebSockets
You already seem to have a local server so you meet the requirements plus node ships with them ready for use (also available on the browser). To use noodlejs with websockets you'd have to require the package and set up the WebSocket to send data to your client
Other than pointing you towards that direction, I don't think I can do better than the internet at giving you a tutorial. Hope this helps, Have fun! Also thanks for telling me about noodle, that thing is awesome!
I have been using Deployd for a week or so, and was curious if I could expose the contents of the Collections Event Script itself, from the API. (the contents of the /my-project/resources/my-collection/get.js file itself)
This could be useful to automatically produce documentation of the scripts being applied to Get, Post and other requests.
Thanks for the help,
Jacob
This is what I have so far: If I start at localhost:2404/dashboard , I can run the following code in the Chrome Console to retrieve the string content of the GET Event on the collection Tshirts:
dpd('__resources').get(Context.resourceId + '/' + 'get.js', function(res, err)
{
_events['get'] = res && res.value;
console.log(res.value);
});
Context.resourceId simplifies to the collection ID which is just "tshirts".
This successfully outputs the data I am trying to access, but I wonder if it is possible to retrieve from the API. I imagine I need to dig into Node.JS in general to wrap my head around this.
Thanks again,
Jacob