We Keep Coding

How can I add an Algolia index to Gatsby GraphQL?

I would like to obtain all (or a subset) of my records from an Algolia index and access them via GraphQL.
I know there is a Gatsby plugin that allows you to do the opposite i.e., add data from a GraphQL query to Algolia, but not the other way around.
I have been able to get the tutorial for adding GraphQL data to work, but I have not had any success when trying to go beyond hardcoded arrays (this is in the gatsby-node.js file):
const algoliasearch = require("algoliasearch/lite");
const searchClient = algoliasearch(
process.env.GATSBY_ALGOLIA_APP_ID,
process.env.GATSBY_ALGOLIA_SEARCH_KEY
)
const searchIndex = searchClient.initIndex(process.env.GATSBY_ALGOLIA_INDEX_NAME)
exports.sourceNodes = ({ actions, createNodeId, createContentDigest }) => {
searchIndex.search("", {
attributesToRetrieve: ["name", "url"]
}).then(({ hits }) => {
hits.forEach(hit => {
const node = {
name: hit.name,
url: hit.url,
id: createNodeId(`hit-${hit.name}`),
internal: {
type: "hit",
contentDigest: createContentDigest(hit),
},
}
actions.createNode(hit)
})
});
}
While the console successfully logs the array of nodes, and the verbose Gatsby deploy output includes the "hit" node as a node type, they do not appear in the GraphQL explorer.
Any help is greatly appreciated, thank you!

How to use AWS CDK to look up existing ApiGateway

I am using AWS CDK to build my lambda, and I would like to register endpoints from the lambda's CDK stack.
I found I can get an existing ApiGateway construct using fromRestApiId(scope, id, restApiId)
(documentation here)
So currently this works well:
//TODO how to look up by ARN instead of restApiId and rootResourceId??
const lambdaApi = apiGateway.LambdaRestApi
.fromRestApiAttributes(this, generateConstructName("api-gateway"), {
restApiId: <API_GATEWAY_ID>,
rootResourceId: <API_GATEWAY_ROOT_RESOURCE_ID>,
});
const lambdaApiIntegration = new apiGateway.LambdaIntegration(lambdaFunction,{
proxy: true,
allowTestInvoke: true,
})
const root = lambdaApi.root;
root.resourceForPath("/v1/meeting/health")
.addMethod("GET", lambdaApiIntegration);
But I would like to deploy to many AWS accounts, and many regions. I don't want to have to hardcode the API_GATEWAY_ID or API_GATEWAY_ROOT_RESOURCE_ID for each account-region pair.
Is there a more generic way to get the existing ApiGateway construct, (e.g. by name or ARN)?
Thank you in advance.

Lets take a simple Api with one resource
const restApi = new apigw.RestApi(this, "my-api", {
restApiName: `my-api`,
});
const mockIntegration = new apigw.MockIntegration();
const someResource = new apigw.Resource(this, "new-resource", {
parent: restApi.root,
pathPart: "somePath",
defaultIntegration: mockIntegration,
});
someResource.addMethod("GET", mockIntegration);
Lets assume we want use this api and resource in another stack, we first need to export
new cdk.CfnOutput(this, `my-api-export`, {
exportName: `my-api-id`,
value: restApi.restApiId,
});
new cdk.CfnOutput(this, `my-api-somepath-export`, {
exportName: `my-api-somepath-resource-id`,
value: someResource.resourceId,
});
Now we need to import in new stack
const restApi = apigw.RestApi.fromRestApiAttributes(this, "my-api", {
restApiId: cdk.Fn.importValue(`my-api-id`),
rootResourceId: cdk.Fn.importValue(`my-api-somepath-resource-id`),
});
and simply add additional resources and methods.
const mockIntegration = new apigw.MockIntegration();
new apigw.Resource(this, "new-resource", {
parent: restApi.root,
pathPart: "new",
defaultIntegration: mockIntegration,
});

How can I get dynamic data from within gatsby-config.js?

Consider the following code within gatsby-config.js:
module.exports = {
plugins: [
{
resolve: `gatsby-source-fetch`,
options: {
name: `brands`,
type: `brands`,
url: `${dynamicURL}`, // This is the part I need to be dynamic at run/build time.
method: `get`,
axiosConfig: {
headers: { Accept: "text/csv" },
},
saveTo: `${__dirname}/src/data/brands-summary.csv`,
createNodes: false,
},
},
],
}
As you can see above, the URL for the source plugin is something that I need to be dynamic. The reason for this is that the file URL will change every time it's updated in the CMS. I need to query the CMS for that field and get its CDN URL before passing to the plugin.
I tried adding the following to the top of gatsby-config.js but I'm getting errors.
const axios = require("axios")
let dynamicURL = ""
const getBrands = async () => {
return await axios({
method: "get",
url: "https://some-proxy-url-that-returns-json-with-the-csv-file-url",
})
}
;(async () => {
const brands = await getBrands()
dynamicURL = brands.data.summary.url
})()
I'm assuming this doesn't work because the config is not waiting for the request above to resolve and therefore, all we get is a blank URL.
Is there any better way to do this? I can't simply supply the source plugin with a fixed/known URL ahead of time.
Any help greatly appreciated. I'm normally a Vue.js guy but having to work with React/Gatsby and so I'm not entirely familiar with it.

I had similar requirement where I need to set siteId of gatsby-plugin-matomo dynamically by fetching data from async api. After searching a lot of documentation of gatsby build lifecycle, I found a solution.
Here is my approach -
gatsby-config.js
module.exports = {
siteMetadata: {
...
},
plugins: {
{
resolve: 'gatsby-plugin-matomo',
options: {
siteId: '',
matomoUrl: 'MATOMO_URL',
siteUrl: 'GATSBY_SITE_URL',
dev: true
}
}
}
};
Here siteId is blank because I need to put it dynamically.
gatsby-node.js
exports.onPreInit = async ({ actions, store }) => {
const { setPluginStatus } = actions;
const state = store.getState();
const plugin = state.flattenedPlugins.find(plugin => plugin.name === "gatsby-plugin-matomo");
if (plugin) {
const matomo_site_id = await fetchMatomoSiteId('API_ENDPOINT_URL');
plugin.pluginOptions = {...plugin.pluginOptions, ...{ siteId: matomo_site_id }};
setPluginStatus({ pluginOptions: plugin.pluginOptions }, plugin);
}
};
exports.createPages = async function createPages({ actions, graphql }) {
/* Create page code */
};
onPreInit is a gatsby lifecycle method which is executing just after plugin loaded from config. onPreInit lifecycle hook has some built in methods.
store is the redux store where gatsby is storing all required information for build process.
setPluginStatus is a redux action by which plugin data can be modified in redux store of gatsby.
Here the important thing is onPreInit lifecycle hook has to be called in async way.
Hope this helps someone in future.

Another approach that may work for you is using environment variables as you said, the URL is known so, you can add them in a .env file rather than a CSV.
By default, Gatsby uses .env.development for gatsby develop and a .env.production for gatsby build command. So you will need to create two files in the root of your project.
In your .env (both and .env.development and .env.production) just add:
DYNAMIC_URL: https://yourUrl.com
Since your gatsby-config.js is rendered in your Node server, you don't need to prefix them by GATSBY_ as the ones rendered in the client-side needs. So, in your gatsby-config.js:
module.exports = {
plugins: [
{
resolve: `gatsby-source-fetch`,
options: {
name: `brands`,
type: `brands`,
url: process.env.DYNAMIC_URL, // This is the part I need to be dynamic at run/build time.
method: `get`,
axiosConfig: {
headers: { Accept: "text/csv" },
},
saveTo: `${__dirname}/src/data/brands-summary.csv`,
createNodes: false,
},
},
],
It's important to avoid tracking those files in your Git repository since you don't want to expose this type of data.

Unable to implement React-table sever data loading

I am testing the React-table server side data to render a huge amount of data fetched from an web api without crashing the browser. With the base react-table settings the browser is unable to handle such amount of records (500000) and crash (it gets stuck in the pending state of the request).
So I find the server side data that maybe can help me.
I followed the instructions from the documentation but typescript is complaining about the data that I am trying to use when I update the state.
This is what I have until now:
The method that fetch the data from web api:
private fetchSales() {
fetch(`http://localhost:50335/api/RK`)
.then(response => response.json())
.then(data =>
this.setState({
sales: data // here I get 500000 items
})
)
}
This fetchSales gets called in the componentDidMount().
Then I have the ReactTable component inside the render():
render() {
const {
sales,
pages
} = this.state;
return (
<div className = "App" >
<ReactTable
data = {sales}
manual
pages = {pages}
defaultPageSize = {10}
onFetchData = {this._fetchData}
columns = {
[{
Header: "Region",
accessor: "Region"
},
{
Header: "Country",
accessor: "Country"
}]
}
/>
</div>
);
}
In the ReactTable there is call to a function called _fetchData and that function looks like this:
private _fetchData(state: any) {
requestData(
state.sales,
state.pageSize,
state.page
).
then(res => {
this.setState({
sales: res.rows, // here typescript complain: "res is of type 'unknown'"
pages: res.pages // here typescript complain: "res is of type 'unknown'"
});
})
}
Inside the setState the res object is type 'unknown' and typescript doesn't like it.
requestData is a function that lives outside the class and get the sales, pageSize and page states:
const requestData = (sales: any, page: number, pageSize: number) => {
return new Promise((resolve, reject) => {
const res = {
rows: sales.slice(pageSize * page, pageSize * page * pageSize),
pages: Math.ceil(sales.length / pageSize)
}
resolve(res);
})
}
The function is almost identical the in the documentation I only removed the filtering and sorting because I don't need them. I only need the res object that return the function.
And I almost forget it, inside the constructor I am attaching the this to the _fetchData method: this._fetchData = this._fetchData.bind(this);
Why is typescript complaining about the res object that I am trying to use to set the state?
Best regards!
Americo

EDIT
I noticed a few mistakes.
private _fetchData(state: any) {
requestData(
state.sales, // that should be state.data (state is the state of ReactTable)
state.pageSize,
state.page
...
Next, I've noticed that you do pagination after fetching the data. But using manual with onFetchData is for handling pagination on the server. For example, with page and pageSize you could pass these parameters to your API. That's the whole point of pagination!
An example from the documentation:
onFetchData={(state, instance) => { //onFetchData is called on load and also when you click on 'next', when you change page, etc.
// show the loading overlay
this.setState({loading: true})
// fetch your data
Axios.post('mysite.com/data', {
//These are all properties provided by ReactTable
page: state.page,
pageSize: state.pageSize,
sorted: state.sorted,
filtered: state.filtered
})
Since you're fetching all of them at once (why though? can't your API on the server handle pagination? this way, you won't have to wait ages before the results are returned), then I suggest you let ReactTable do the work.
That is, you just do:
<ReactTable
columns={columns}
data={data}
/>
ReactTable will take care of pagination. And now, you may use your query to the API in ComponentDidMount.
Could you please instead try to put all the logic in the onFetchData. I could be wrong but it seems to me you misunderstood the instructions in the documentation. OnFetchData is called at ComponentDidMount, it's not telling you that you have to put your function there.
private _fetchData(state, instance) {
const { page, pageSize } = state
fetch(`http://localhost:50335/api/RK`)
.then(response => response.json())
.then(data =>
let sales = data
this.setState({
sales: sales.slice(pageSize * page, pageSize * page * pageSize),
pages: Math.ceil(sales.length / pageSize)
});
)
}
As for Typescript, from what I gather, Typescript doesn't have enough information to infer the type of what your Promise returns.
So you have to explicitly annotate Promises generic type parameter:
return new Promise<{ sales: object; pages: number; }>((resolve, reject) => { ... }

Use existing variables when using refetchQueries in react-apollo

I am using postsConnection query for infinite scroll. It contains variables like after.
After doing an upvote mutation, I want to refetchQueries... like this 👇
const upvote = await client.mutate({
mutation: UPVOTE_MUTATION,
variables: {
postId: this.props.post.id
},
refetchQueries: [
{ query: POST_AUTHOR_QUERY }
]
})
Above code gives error because POST_AUTHOR_QUERY accepts few variables. Here's that query 👇
export const POST_AUTHOR_QUERY = gql`
query POST_AUTHOR_QUERY($authorUsername: String! $orderBy: PostOrderByInput $after: String){
postsAuthorConnection(authorUsername: $authorUsername orderBy: $orderBy after: $after) {
....
}
}
I do not want to add variables manually. Variables are already stored in the cache. How do I reuse them while using refetchQueries???
Here are a few resources I have read about this issue 👇
https://github.com/apollographql/react-apollo/issues/817
https://github.com/apollographql/apollo-client/issues/1900

As mentioned in the issue you linked, you should be able to do the following:
import { getOperationName } from 'apollo-link'
const upvote = await client.mutate({
// other options
refetchQueries={[getOperationName(POST_AUTHOR_QUERY)]}
})
From the docs:
Please note that if you call refetchQueries with an array of strings, then Apollo Client will look for any previously called queries that have the same names as the provided strings. It will then refetch those queries with their current variables.
getOperationName simply parses the document you pass it and extracts the operation name from it. You can, of course, provide the operation name yourself as a string instead, but this way avoids issues if the operation name changes in the future or you fat finger it.

If you don't want to pull in apollo-link, you can also get this via the base graphql package (note that I use optional chaining for convenience:
import { getOperationAST } from 'graphql';
const operationName = getOperationAST(POST_AUTHOR_QUERY)?.name?.value;
// Note that this could technically return `undefined`
const upvote = await client.mutate({
mutation: UPVOTE_MUTATION,
variables: {
postId: this.props.post.id
},
refetchQueries: [operationName]
})