I have a database table full of logs, and I would like to upload that into Splunk by executing a query to get an array of logs, then iterating over it and running console.log() on each log. This almost works fine, however the timestamp that Splunk picks up for each log is based on whatever time I run the upload script.
Is there any way I can transform my log object so that Splunk will use the timestamps I provide? For example, one of my logs might look like:
{
id: 2910432221,
log_timestamp: 2019-08-07T19:04:03.000Z,
userId: 2331,
actionId: 45
}
Ideally, I would be able to run something like this, and have the value of splunk_timestamp appear in the Time column in the Splunk dashboard.
console.log({
id: 2910432221,
splunk_timestamp: 2019-08-07T19:04:03.000Z,
userId: 2331,
actionId: 45
})
Is anything like that possible?
Splunk will try to automatically identify the _time value by picking the first field it encounters matching TIME_FORMAT. You can override this in props.conf.
I believe your problem is in the json object: what you are passing to console.log is not a valid json object. Field names and values should be properly quoted.
{
"id": 2910432221,
"splunk_timestamp": "2019-08-07T19:04:03.000Z",
"userId": 2331,
"actionId": 45
}
Related
I am using Cloud Firestore as my database
This is my form codes on my webpage that creates a new document into my Cloud Firestore collection called "esequiz". So how do I code it in such a way that it always plus 1 to the number of documents there are in the database? And also set a limit to having the amount of documents inside the database
form.addEventListener('submit', (e) => {
e.preventDefault();
db.collection('esequiz').add({
question: form.question.value,
right: form.right.value,
wrong: form.wrong.value
});
form.question.value = '';
form.right.value = '';
form.wrong.value = '';
});
It currently works but it will show up as an auto generated ID. How do I make it carry on from the numbers, like as my current documents? When i save I would like it to read the current last document id, OR simply count the number of documents, then just + 1
Insight from Andrei Cusnir, counting documents in Cloud Firestore is not supported.
Now I am trying Andrei's approach 2, to query documents in descending order, then using .limit to retrieve the first one only.
UPDATED
form.addEventListener('submit', (e) => {
e.preventDefault();
let query = db.collection('esequiz');
let getvalue = query.orderBy('id', 'desc').limit(1).get();
let newvalue = getvalue + 1;
db.collection('esequiz').doc(newvalue).set({
question: form.question.value,
right: form.right.value,
wrong: form.wrong.value
});
form.question.value = '';
form.right.value = '';
form.wrong.value = '';
});
No more error, but instead, the code below returns [object Promise]
let getvalue = query.orderBy('id', 'desc').limit(1).get();
So when my form saves, it saves as [object Promise]1, which I don't know why it is like this. Can someone advise me on how to return the document id value instead of [object Promise]
I think it is because I did specify to pull the document id as the value, how do I do so?
UPDATED: FINAL SOLUTION
Played around with the codes from Andrei, and here are the final codes that works. Much thanks to Andrei!
let query = db.collection('esequiz');
//let getvalue = query.orderBy('id', 'desc').limit(1).get();
//let newvalue = getvalue + 1;
query.orderBy('id', 'desc').limit(1).get().then(querySnapshot => {
querySnapshot.forEach(documentSnapshot => {
var newID = documentSnapshot.id;
console.log(`Found document at ${documentSnapshot.ref.path}`);
console.log(`Document's ID: ${documentSnapshot.id}`);
var newvalue = parseInt(newID, 10) + 1;
var ToString = ""+ newvalue;
db.collection('esequiz').doc(ToString).set({
id: newvalue,
question: form.question.value,
right: form.right.value,
wrong: form.wrong.value
});
});
});
If I understood correctly you are adding data to the Cloud Firestore and each new document will have as name an incremental number.
If you query all the documents and then count how many are of them, then you are going to end up with many document reads as the database increases. Don't forget that Cloud Firestore is charging per document Read and Write, therefore if you have 100 documents and you want to add new document with ID: 101, then with the approach of first reading all of them and then counting them will cost you 100 Reads and then 1 Write. The next time it will cost you 101 Reads and 1 Write. And it will go on as your database increases.
The way I see is from two different approaches:
Approach 1:
You can have a single document that will hold all the information of the database and what the next name should be.
e.g.
The structure of the database:
esequiz:
0:
last_document: 2
1:
question: "What is 3+3?
right: "6"
wrong: "0"
2:
question: "What is 2+3?
right: "5"
wrong: "0"
So the process will go as follows:
Read document "/esequiz/0" Counts as 1 READ
Create new document with ID: last_document + 1 Counts as 1 WRITE
Update the document that holds the information: last_document = 3; Counts as 1 WRITE
This approach cost you 1 READ and 2 WRITES to the database.
Approach 2:
You can load only the last document from the database and get it's ID.
e.g.
The structure of the database (Same as before, but without the additional doc):
esequiz:
1:
question: "What is 3+3?
right: "6"
wrong: "0"
2:
question: "What is 2+3?
right: "5"
wrong: "0"
So the process will go as follows:
Read the last document using the approach described in Order and limit data with Cloud Firestore documentation. So you can use direction=firestore.Query.DESCENDING with combination of limit(1) which will give you the last document. Counts as 1 READ
Now you know the ID of the loaded document so you can create new document with ID: that will use the loaded value and increase it by 1. Counts as 1 WRITE
This approach cost you 1 READ and 1 WRITE in total to the database.
I hope that this information was helpful and it resolves your issue. Currently counting documents in Cloud Firestore is not supported.
UPDATE
In order for the sorting to work, you will also have to include the id as a filed of the document that so you can be able to order based on it. I have tested the following example and it is working for me:
Structure of database:
esequiz:
1:
id: 1
question: "What is 3+3?
right: "6"
wrong: "0"
2:
id:2
question: "What is 2+3?
right: "5"
wrong: "0"
As you can see the ID is set the same as the document's ID.
Now you can query all the documents and order based on that filed. At the same time you can only retrieve the last document from the query:
const {Firestore} = require('#google-cloud/firestore');
const firestore = new Firestore();
async function getLastDocument(){
let query = firestore.collection('esequiz');
query.orderBy('id', 'desc').limit(1).get().then(querySnapshot => {
querySnapshot.forEach(documentSnapshot => {
console.log(`Found document at ${documentSnapshot.ref.path}`);
console.log(`Document's ID: ${documentSnapshot.id}`);
});
});
}
OUTPUT:
Found document at esequiz/2
Document's ID: 2
Then you can take the ID and increase it by 1 to generate the name for your new document!
UPDATE 2
So, the initial question is about "How to store data in the Cloud Firestore with documents having incremental ID", at the moment you are facing issues of setting up Firestore with you project. Unfortunately, the new raised questions should be discussed in another Stackoverflow post as they have nothing to do with the logic of having incremental IDs for the document and it is better to keep one issue per question, to give better community support for members that are looking for a solution about particular issues. Therefore, I will try to help you, in this post, to execute a simple Node.js script and resolve the initial issue, which is storing to Cloud Firestore documents with incremental IDs. Everything else, on how to setup this in your project and how to have this function in your page, should be addressed in additional question, where you also will need to provide as much information as possible about the Framework you are using, the project setup etc.
So, lets make a simple app.js work with the logic described above:
Since you have Cloud Firestore already working, this means that you already have Google Cloud Platform project (where the Firestore relies) and the proper APIs already enabled. Otherwise it wouldn't be working.
Your guide in this tutorial is the Cloud Firestore: Node.js Client documentation. It will help you to understand all the methods you can use with the Firestore Node.js API. You can find helpful links for adding, reading, querying documents and many more operations. (I will post entire working code later in this steps. I just shared the link so you know where to look for additional features)
Go to Google Cloud Console Dashboard page. You should login with your Google account where your project with the Firestore database is setup.
On top right corner you should see 4 buttons and your profile picture. The first button is the Activate Cloud Shell. This will open a terminal on the bottom of the page with linux OS and Google Cloud SDK already install. There you can interact with your resources within GCP projects and test your code locally before using it in your projects.
After clicking that button, you will notice that the terminal will open in the bottom of your page.
To make sure that you are properly authenticated we will set up the project and authenticate the account again, even if it is already done by default. So first execute $ gcloud auth login
On the prompted question type Y and hit enter
Click on the generated link and authenticate your account on the prompted window
Copy the generated string back to the terminal and hit enter. Now you should be properly authenticated.
Then setup the project that contains Cloud Firestore database with the following command: $ gcloud config set project PROJECT_ID. Now you are ready to build a simple app.js script and execute it.
Create a new app.js file: nano app.js
Inside paste my code example that can be found in this GitHub link. It contains fully working example and many comments explaining each part therefore it is better that it is shared through GitHub link and not pasted here. Without doing any modifications, this code will execute exactly what you are trying to do. I have tested it my self and it is working.
Execute the script as: node app.js
This will give you the following error:
Error: Cannot find module '#google-cloud/firestore'
Since we are importing the library #google-cloud/firestore but haven't installed it yet.
Install #google-cloud/firestore library as follows: $ npm i #google-cloud/firestore. Described in DOC.
Execute the script again: $ node app.js.
You should see e.g. Document with ID: 3 is written.
If you execute again, you should see e.g. Document with ID: 4 is written.
All those changes should appear in your Cloud Firestore database as well. As you can see it is loading the ID of the last document, it is creating a new ID and then it creates a new document with the given arguments, while using the new generated ID as document name. This is exactly what the initial issue was about.
So I have shared with you the full code that works and does exactly what you are trying to do. Unfortunately, the other newly raised issues, should be addressed in another Stackoverflow post, as they have nothing to do with the initial issue, which is "How to create documents with incremental ID". I recommend you to follow the steps and have a working example and then try to implement the logic to your project. However, if you are still facing any issues with how to setup Firestore in your project then you can ask another question. After that you can combine both solutions and you will have working app!
Good luck!
I don't think the way you are trying to get the length of the collection is right and I am entirely not sure what is the best way to get that either. Because the method you are trying to implement will cost you a lot more as you are trying to read all the records of the collection.
But there can be alternatives to get the number you require.
Start storing the ID in the record and make the query with limit 1 and a descending sort on ID.
Store the latest number in another collection and increment that every time you create a new record, And fetch the same whenever needed.
These methods might fail if concurrent requests are being made without transactions.
I use the .once( function in Firebase and am successfully retrieving the snapshot.key NAME of the directory. But this key has also a value.
When requesting snapshot.val() also gives out null despise it having a value.
The database looks like this:
stages {
stage1AVERYLONGHASH : "values i want to get"
stage2AVERYLONGHASH : "values i want to get"
}
My code
var ref = "stage1AVERYLONGHASH";
var branch = firebase.database().ref(ref).once('value').then(function(s){
console.log(s.key); // GIVES THE CORRECT ANSWER
console.log(s.val()); //GIVES null ... doesn't give me the "values i want.."
Am i doing it completely wrong?
I'm new to Firebase.
Please know that the database is structured like this intentionally.
I restricted read permission on "stages".
The game works as just requesting a hash from the database as a reference, to get contents.
I think you suppose to get the reference of the stage and then get the stage1 reference from him like :
.ref("stage").child("stage1")
I'm not sure about the actual syntax but I know that this is how I get my firebase data in android studio
In my user collection, I have an object that contains an array of contacts.
The object definition is below.
How can this entire object, with the full array of contacts, be written to the user database in Meteor from the server, ideally in a single command?
I have spent considerable time reading the mongo docs and meteor docs, but can't get this to work.
I have also tried a large number of different commands and approaches using both the whole object and iterating through the component parts to try to achieve this, unsuccessfully. Here is an (unsuccessful) example that attempts to write the entire contacts object using $set:
Meteor.users.update({ _id: this.userId }, {$set: { 'Contacts': contacts}});
Thank you.
Object definition (this is a field within the user collection):
"Contacts" : {
"contactInfo" : [
{
"phoneMobile" : "1234567890",
"lastName" : "Johnny"
"firstName" : "Appleseed"
}
]
}
This update should absolutely work. What I suspect is happening is that you're not publishing the Contacts data back to the client because Meteor doesn't publish every key in the current user document automatically. So your update is working and saving data to mongo but you're not seeing it back on the client. You can check this by doing meteor mongo on the command line then inspecting the user document in question.
Try:
server:
Meteor.publish('me',function(){
if (this.userId) return Meteor.users.find(this.userId, { fields: { profile: 1, Contacts: 1 }});
this.ready();
});
client:
Meteor.subscribe('me');
The command above is correct. The issue is schema verification. Simple Schema was defeating the ability to write to the database while running 'in the background'. It doesn't produce an error, it just fails to produce the expected outcome.
I'm creating a mock application with JSON server as the backend and I'm wondering if it is possible to get the total number of records contained at an end point without loading all the records themselves? Assuming the db.json file looks like the JSON snippet below, how would I find out that the end point only has one record without fetching the record itself, provided it's possible?
{
"books": [{
"title": "The Da Vinci Code",
"rating": "0"}]
}
You can simply retrieve the X-Total-Count header
This is a screen-shot of a response headers returned by JSON Server when enabling pagination i.e using the _page parameter (e.g. localhost:3000/contacts?_page=1)
Whenever you fetch the data, json-server actually returns the total count by default (it has an x-total-count property:
Example:
axios
.get("http://localhost:3001/users", {
params: {
_page: 1,
_limit: 10
}
})
.then(res => {
console.log(res.data); // access your data which is limited to "10" per page
console.log(res.headers["x-total-count"]); // length of your data without page limit
});
You've three options. I'd recommend the 3rd one to you:
Return all the records and count them. This could be slow and send a lot of data over the wire but probably is the smallest code change for you. It also opens you up to attacks where people can hammer your server by requesting many records repeatedly.
Add a new endpoint. You could add a new endpoint that simply returns the count. It's simple but slightly annoying having a 2nd endpointime to document and maintain.
Modify the existing endpoint. Return something like
{
count: 157,
rows: [...data]
}
The benefit of 3 is its all in one endpoint. It also nears you toward a point where you can add a skip and take parameter in future to allow pagination of the resultant data.
You will write another end point that returns number of records. Usually also you may want end point for limit and offset to be used with pagination.
let response = await fetch("http://localhost:3001/books?_page=1");
let total = response.headers.get('X-Total-Count');
I'm having trouble finding good documentation about how to use javascript in order to find out if a like button is clicked or not. I can't use an on-click event because the button might already be checked when a user comes onto my page but that seems to be all I can find regarding tracking likes externally (not counting other methods that are no longer supported). Anyone have any experience with such issues? Thanks.
It depends on the Like button. If it's for a Facebook object that has an ID, you can query the like table (but you'll need user_likes permission)
If it's for a URL, it's not really possible: facebook graph api determine if user likes url [stackoverflow]
It it's for a built-in like you could use a cookie to remember the click, or save the click in your own database.
Just to expand on Gil's answer. For built-in likes you can use a batch request consisting of two request:
1) Try to like the object in question
2) If there was no error, delete the like connection
Example batch:
[{"method":"POST", "relative_url":"me/og.likes", "body":"object=<SOME_URL>", "name":"like-attempt", "omit_response_on_success": false},
{"method":"DELETE", "relative_url":"{result=like-attempt:$.id}"}]
If an object was already liked, batch response would be:
[
{
"code": 400,
"headers": [
...
{
"name": "WWW-Authenticate",
"value": "OAuth \"Facebook Platform\" \"invalid_request\" \"(#3501) User is already associated to the object object on a unique action type Like. Original Action ID: 143539809123515\""
}
],
"body": "{\n \"error\": {\n \"message\": \"(#3501) User is already associated to the object object on a unique action type Like. Original Action ID: 143539809123515\",\n \"type\": \"OAuthException\",\n \"code\": 3501\n }\n}"
},
null
]
So to check if object was liked, JSON.parse the first response body and look for an error with error code 3501. Keep in mind that the actual error code is not documented anywhere so it could change, although I don't think it is likely.