etherscan-api does not output pending transactions - javascript

I am using the etherscan-api here: (https://sebs.github.io/etherscan-api/#txlist) to get the list of transactions for a user (given public key), and I am getting the list of cleared transactions only, even though on etherscan.io I can see all pending transactions as well, so the information is "there". Digging into the source https://github.com/sebs/etherscan-api/blob/master/lib/account.js#L122, I find no indication of where I can look to find pending transactions. Code below:
const etherscan = require('etherscan-api').init(ETHERSCAN_TOKEN, 'ropsten', '3000');
etherscan.account
.txlist(public_key, 1, 'latest', 1, 100, 'asc')
.then(res => {
console.log("tx_list: ", res.result)
})
// get num of transactions
etherscan.proxy
.eth_getTransactionCount(public_key, 'latest')
.then(res => {
let count = res.result
console.log("eth_getTransactionCount: ", parseInt(count))
})

Etherscan doesn't provide pending transactions in their API.
It's probably because of the nature of how pending transactions are collected and displayed. All pending transactions are collected together in the Ethereum mempool. My guess is that Etherscan adds each entry from mempool to some kind of quickly searchable database such as ElasticSearch, and removes the DB entry when the transaction is taken off the mempool. The address detail page then performs a simple search in ES, filtering only transactions from/to this particular address.
If your web3 provider allows websocket connections and subscriptions, you can subscribe to the pendingTransactions event.
It fires an event everytime a new transaction is added to the mempool. The event contains only the transaction hash (and no additional data such as the sender and receiver).
Because the event contains only tx hash, it's not very effecient to always perform a new query on the tx details, but it's useful in some other cases such as when you know the tx hash beforehand and you only want to know when it has been successfully broadcasted.
There's also BlockCypher REST API that has a limited free plan (and somewhat less limited paid plans) that returns all pending transactions including the tx details so that you can perform your own search on the result. See their docs if this fits your use case.

Related

I need get a all transactions like mint , burn in web3

I need to get all transactions from aa address on bsc. I tried bscscan api but it doesn't give everything like burning minting actions, they just give transfers , how can I get all transactions with web3 js or web3 py. I would appreciate if someone can help me
You need scan all block and found that you need. This is very bad way.
Other way -- you can use API
check etherscan.io api you can get a list of all transactions easily as an alternative
https://etherscan.io/apis
http://api.etherscan.io/api?module=account&action=txlist&address=0xde0b295669a9fd93d5f28d9ec85e40f4cb697bae&startblock=0&endblock=99999999&sort=asc&apikey=YourApiKeyToken
and work with this answer.
This is easy way, but no easy way to get all transaction via web3js or web3py.
But if you need only specified events -- that was much easy.
you need function getPastEvents
const START_BLOCK = 7700000;
const END_BLOCK = 7701000;
contract.getPastEvents("allEvents",
{
fromBlock: START_BLOCK,
toBlock: END_BLOCK // You can also specify 'latest'
})
.then(events => console.log(events))
.catch((err) => console.error(err));
You can set mint or burn events only.
https://web3js.readthedocs.io/en/v1.2.11/web3-eth-contract.html#getpastevents

Is it possible to make a bot in discord.js search for previous messages without a json,or having to store it?

Basically like the search function in discord.
For example if I want to make a
!search [user] [query]
and make it respond
"50 messages fit the query"
a word counter bot without the db or local storage basically.The bot would return 87 on this query
You can't sadly. Here's why...
Explanation
Message query is limited to user accounts.
There's a lot of ways to do the workaround but the best way and most used is to fetch a large amount of messages in the specific channel.
Fetching all messages?
All endpoints that are related to fetching always have a limit of 100 things per request. Fetching messages? 100 messages per request. etc. etc.
There's no way to bypass this, and the only solution is to fetch and re-fetch and re-fetch until you reached certain amount.
Demonstration
// Fetch a 100 messages from a Channel
// You can always fetch more and concat() the collections together
// <Channel> is a placeholder for guild's text-based channels
const messages = await <Channel>.messages.fetch({ limit: 100 });
// Make a filter
// In this case, we're finding messages with "hello". Case insensitive
const query = (msg) => msg.content.toLowerCase().includes("hello");
// Filter out non-matching messages
const matching = messages.filter(query);
console.log(matching);
// `matching` is Collection<Snowflake, Message>

Firebase firestore: how to query relevant documents and then update each of them

I am developing a web app on Firebase/firestore, in which users can sign in and write their own posts. The data are stored in the following way:
-User information are stored as under collection('user').doc('uid').
-Information about posts the user has written are stored in collection('post').doc('postid'), and the doc has 'userinfo' and 'uid' fields. The 'userinfo' field contains exact copy of what is stored in 'uid' doc, just in object format.
Here are the operations that I want to do:
When the user changes the data, the changes are reflected in the document.
Look for the all the posts that the user has written based on 'uid' data, and then update userinfo in those data.
The last part is tricky for me. The Firebase documentations cover situations where the references are pretty much static, i.e. you know the exact path to write/update. What I am trying to do is look for a set of documents that is not necessarily static, and then update each of them.
Here is the code I wrote for this effort. The first part works without any problem. Of course, the second part doesn't work. :) What would be the code to do the do the second part?
const update = () => {
//This part is for updating user information. This works without any problem.
firebase.firestore().collection('user').doc(user.uid).update({
username: username1,
nickname: nickname1,
intro: intro1
})
.then(()=>{
//This part is for updating all of the document that the user has written based on 'uid' value. This doesn't work.
//Below code is probably way off, but it shows where I am going and what I am trying to do.
firebase.firestore().collection('post').where('uid','==',user.uid).get()
.then((querysnapshot)=>{
querysnapshot.forEach((doc)=>{
let ref=firebase.firestore().collection('post').doc(doc.id);
ref.update({
userinfo: {nickname:nickname1,username:username1,intro:intro1}
})
})
})
}).then(()=>{
alert("Successfully updated!");
window.location.href='/'+username1;
}).catch((error)=>{
alert("Error!");
})
}
Thanks a lot in advance!
What's the error that you get running this code? It seems on the right track for me.
But despite that, here are some suggestions to deal with this kind of update:
Don't do the second part on the client side, do it on the server side with a Firestore Trigger (create a onUpdate trigger in the user collection in your case): https://firebase.google.com/docs/functions/firestore-events.
The problem of doing in the client side, is because if the user closes the page/browser or the site goes offline in the middle of the update, you will have inconsistent data.
You don't need to recreate the DocumentReference after getting the query result, the docs returned already have a .ref that you can call .ref.update() directly.
EDIT: If you want to keep your original code (updating on client side), the problem of the navigation occurring before all the updates to conclude is because ref.update() returns a promise.
So the update queue is asynchronous being performed on database when the client navigates away.
To solve this, I would use a Promise.all() to wait all updates being completed.
firebase.firestore().collection('post').where('uid','==',user.uid).get()
.then((querysnapshot)=>{
const promises = [];
querysnapshot.forEach((doc)=>{
promises.push(doc.ref.update({
userinfo: {nickname:nickname1,username:username1,intro:intro1}
});
});
Promise.all(promises).then(()=>{window.location.href='/'+username1;});
});
Or using the await syntax (I think it's easier to maintain and understand):
const querysnapshot = await firebase.firestore().collection('post').where('uid','==',user.uid).get();
const promises = [];
querysnapshot.forEach((doc)=>{
promises.push(doc.ref.update({
userinfo: {nickname:nickname1,username:username1,intro:intro1}
});
});
await Promise.all(promises);
window.location.href='/'+username1;

BigQuery similarity to "signed urls"

I have the following use case in BigQuery:
A non-trusted user will be querying a BigQuery table. Let's say the query is SELECT * FROM [bigquery.table123].
The query will return a large amount of data, let's say 200MB, which will then be displayed in the user's browser.
Our goal is to provide the most efficient way to get the 200MB data into the user's browser (and the worst way seems to do two trips instead of one -- from BQ to our server and then (compressed) to the client). I think the solution for this would probably be to enable the end (non-trusted) user to get something like a "signed-url" to perform the query directly from their browser to BigQuery. The flow would then be like this:
User issues query to our backend.
Authentication is done and a signed url is generated and passed back into javascript.
The client then sends the signed url and the data is loaded directly into the browser.
Only that exact query that has been authorized may be performed, and no other queries could be done (for example, if the client copied any tokens from the javascript)
I would never, ever want the end user to know the ProjectId or Table Name(s) that they are querying.
Is something like this possible to do in BigQuery? Here is an example of a similar need in Cloud Storage. Here is an example of an authenticated/trusted user doing this in browser: https://github.com/googleapis/nodejs-bigquery/blob/master/samples/browseRows.js or . https://stackoverflow.com/a/11509425/651174, but is there a way to do this in-browser for a non-trusted user?
Below is an option that involves two levels of authorized views. This allows to shield not only underlying data from end user - but also hides what exactly data is being used
Let's assume data is in DatasetA. Below steps explain the logic
Create InternalView in DatasetB - this one will target real data from DatasetA.
Make InternalView as Authorized View for DatasetA
Create PublicView in DatasetC - this one will target InternalView
Make PublicView as Authorized View for DatasetB
Give users read access to DatasetC
Users will be ale to run PublicView which will actually be running PrivateView against readl data.
Meantime, users will not be able to see the definition of PrivateView thus will never know ProjectId or Table Name(s) that they are querying, etc.
Note: this does not address how we'd prevent users from being able to issue queries that we haven't pre-authorized? part of your question but I am adding my answer as you asked me to do
Meantime - at least theoretically - you can embed some logic into your PrivateView, which will be querying some internal metatable with info which user and when allowed to get result. Assuming that such meta-table will be managed by your backend based on authentication/token or whatever else you have in mind
Below is simplified and brief outline of that approach
#standardSQL
WITH `projectA.datasetA.table` AS (
SELECT 'data1' col UNION ALL
SELECT 'data2' UNION ALL
SELECT 'data3'
), `projectA.datasetA.applicationPermissions` AS (
SELECT 'user1#gmail.com' user UNION ALL
SELECT 'user2#gmail.com'
), `projectA.datasetB.privateView` AS (
SELECT d.*
FROM `projectA.datasetA.table` d
CROSS JOIN `projectA.datasetA.applicationPermissions` p
WHERE LOWER(user) = LOWER(SESSION_USER())
), `projectA.datasetC.publicView` AS (
SELECT *
FROM `projectA.datasetB.privateView`
)
SELECT *
FROM `projectA.datasetC.publicView`
If user1#gmail.com or user2#gmail.com will run below query
SELECT *
FROM `projectA.datasetC.publicView`
they will get below result
Row col
1 data1
2 data2
3 data3
while if user3#gmail.com will run same very query - result will be
Row col
Query returned zero records.
Obviously, you can extend your meta-table (applicationPermissions) with for example timeframe during which user will be allowed to get result (respective lines to check time conditions will need to be added to projectA.datasetB.privateView )

Webhooks in Bitcoin Transactions - How to Apply Business Logics?

I want to start accepting Bitcoin on my website.
In order to do that, I wrote the following piece of code, but I truly struggle to understand how I can implement proper business logic after that the transaction is completed.
Here is the code:
<html>
<head>
<title>Pay with Bitcoin</title>
<script>
//Gets the URL of the Webpage and gets the price value of this transaction in USD.
//For simplicity Here the Value is passed in the URL.
//However in production you wanna use POST instead of GET.
const myUrl = window.location.href;
const url = new URL(myUrl);
const usdPrice = url.searchParams.get("price");
//This is the function where all the magin happens
const showQR = () => {
//URL of the api which will provide us with current BTC exchange rate
const apiUrl = "https://blockchain.info/ticker";
const hr = new XMLHttpRequest();
hr.open('GET', apiUrl, true);
hr.onreadystatechange = function(){
//Make sure the API sent a valid response
if(hr.readyState == 4){
let ticker = JSON.parse(hr.responseText);
//Get last BTC/USD exchange value from the API , then convert Price from USD to BTC
let BTCprice = ticker.USD.last;
let btcToPay = usdPrice / BTCprice;
//Make sure you have just 8 decimal points in your BTC price!!
btcToPay = btcToPay.toFixed(8);
//Use google API (or other...) to create the QR code. Pass on your btc public address and
//the amount (btc price) dynamically created. Message and label parameters can be dynamic too.
let qrurl = "https://chart.googleapis.com/chart?chs=250x250&cht=qr&chl=bitcoin:1BAnkZn1qW42uRTyG2sCRN9F5kgtfb5Bci?amount="+btcToPay+"%26label=CarRental%26message=BookingID123456";
//Populate the 'btc' DIV with QR code and other info...
document.getElementById('btc').innerHTML = "<img src=" +qrurl+"><br> <span class = 'greenMoney'>" + usdPrice + " usd / " + btcToPay + " BTC </span>";
}
}
hr.send();
};
</script>
</head>
<body onload = "showQR()">
<h1>Pay with BitCoin</h1>
<div id = "btc">
</div>
</body>
</html>
This code does the following:
Gets current USD/BTC exchange rate using the blockchain API.
takes the price in USD for the URL and converts it into BTC
generates a QR code using google API.
Embeds the price, label and message into the QR code
Renders the QR code in a DIV
I ve also set up a web hook service which will be listening to new transactions happening in the specified wallet address. Then a callback to my server is made, by mean of a POST request.
The problem is: the label and message parameters passed to the QR code will not be written in the blockchain.
They are just a handy reference for the customer to remind him what that specific transaction paid for.
As a result the callback to my server is practically useless.
In fact, the callback doesn't return any Booking Id or any other piece of information which could help me to understand who paid for what. Needless to say, in this scenario no business logic is possible: I can't update the order status on my DB, I can't send a confirmation email to the right customer.
How can I embed relevant information (e.g. Booking ID) into the BTC payment, ideally through the QR code?
If this is possible, how can I retrieve this information later on when my server receives the callback informing me that a new payment was made to my BTC wallet?
In short, you can't.
When accepting payments, you are supposed to give each invoice a new BTC address. This way, when you receive notification of an incoming transaction, you can check the receiving address to see which invoice is being paid, and compare the received amount against the expected amount.
Note
Technically, you could embed stuff like a order ID into an OP_RETURN. However, most wallets don't support transactions like that, and any users who want to pay you from an exchange account would be unable to comply.
#Raghav Sood thank you for your input which routed me to the right direction.
Using NodeJS/Express/MongoDB in the backend, I managed to implement a solution which I would like to share here.
Before starting, I wanna make a big disclaimer: this solution is not the only one, it is not the best one, it is not the fastest and probably it is not the most elegant.
Anyway, this solution has the advantage of not relying on packaged third parties solutions. This is in line with the spirit of the whole "no intermediation" philosophy of the bitcoin community. Most imortantly, your XPub always stay in your server and is NOT shared with any external service, which is probably the wisest approach.
Having said that, here is how one can show dynamic unique BTC addresses to customers:
First of all , I put in place a counter which keeps track of how many btc addresses were created for customers from a my HD wallet.
This is important to make sure than you never present the same address twice to customers, which is good for privacy of all parties and also for the sake of implementing business logic in your app.
In order to do this, I store a "counter value" into my DB. Everytime someone visits the BTC payment page, this value is retrived from mongo using a "dealCount" function and is assigned to a "serialPay" variable, which is equal to the value gotten from Mongo + 1. In the backend, the code would be something like this:
`function dealCount(){`
return new Promise(function(resolve, reject){
Deal.find({_id: "ID_OF_OBJ_WHERE_YOU_STORE_COUNTER"}, function(err, data){
if(err){
console.log(err);
}
resolve(data[0].serialDeal + 1);
})
})
};
The new value obtained (which later on will be saved again into Mongo in order to keep track of addresses created) is used to generate the new BTC public address for the customer at hand. If you keep reading you will see how.
To create new public addresses dynamically, one needs the xPub Key of his or her HD Wallet. If one is coding in NodeJS there are a couple of libraries (which can be imported into the server) that will enable this operation rather easily: bitcoinjs-lib and/or bitcore-lib. Personally I opted for Bitcore-lib, because there are less dependencies to deal with and I found the supporting material easier to digest.
Codewise, address generation goes as follows:
const bitcore = require('bitcore-lib');
app.post("/pay.html", urlencodedParser, function(req, res){
let serialPay = dealCount();
serialPay.then(function(serialPay){
const pub = new bitcore.HDPublicKey('INSERT_HERE_YOUR_XPUB_KEY');
let derivedHdPk = pub.derive('m/0/'+serialPay);
let derivedPk = derivedHdPk.publicKey;
let myDynAddress = new bitcore.Address(derivedPk);
res.render('pay', {myDynAddress: myDynAddress});
});
});
Then, using EJS as a templating engine, I could easily make the receiving bitcoin address dynamic in the front-end (/pay.ejs):
let myDynAddress = "<%=myDynAddress%>";
let qrurl = "https://chart.googleapis.com/chart?chs=250x250&cht=qr&chl=bitcoin:"+myDynAddress+"?amount="+btcToPay+"%26label=CarRental";
This will generate the QR Code Dynamically. In the original question, one can see how to render that into the webpage. In the meantime one should also put in place a function to store the updated "serialPay" counter back to the DB.
At this point one should only start monitoring incoming (non-confirmed) payments to the dynamic BTC address generated. A simple way to do it, is using the blockchain.info websocket API. When the payment arrives, things go forward as suggested by #Raghav Sood: one checks the incoming transaction making sure the customer paid the right amount to the right address.
Now you know who paid for what and all sorts of business logics can be triggered.

Categories