Using context.drawImage() with a MediaStream - javascript

I'm new to React and I am building an app that takes screen grabs from a MediaStream. Based on what I've seen, the best way to do that is to draw it onto a canvas element using the context.drawImage() method, passing in the HTMLVideoElement as an argument. Here's what my action creator looks like:
const RecordImage = function(video, canvas, encoder) {
if(!encoder) {
encoder = new GifReadWrite.Encoder()
encoder.setRepeat(0)
encoder.setDelay(100)
encoder.start()
}
const context = canvas.getContext('2d')
context.drawImage(video, 0, 0)
encoder.addFrame(context)
return {
type: RECORD_IMAGE,
payload: encoder
}
}
This worked in the past because the RecordImage action was being called from the same component that housed the <video /> and <canvas /> element, and I could pass them in like so:
takePic(event) {
event.preventDefault()
this.props.RecordImage(this.video, this.canvas, this.props.encoder)
}
...
render() {
return (
<div>
<video
ref = { video => this.video = video }
width = { this.props.constraints.video.width }
height = { this.props.constraints.video.height }
autoPlay = "true"
/>
<canvas
ref = { canvas => this.canvas = canvas }
width = { this.props.constraints.video.width }
height = { this.props.constraints.video.height }
/>
<button onClick = { this.takePic }>Take Picture</button>
</div>
)
}
However, I would like to house the "Take Picture" button in a different component. This is a problem because now I don't know how to access the <video /> and <canvas /> elements from a sibling component. Normally I would store the arguments I need as part of the state, but the drawImage() method needs to use the HTML elements themselves. I've heard it's a bad idea to store DOM elements in the state, so what would be the best way to go about this?

In React it is possible to directly call a function on one of your components.
You could add a 'getPicture' function in your video component which returns the video element?
Your video component:
getPicture() {
return this.video
}
...
render() {
return (
<div>
<video
ref = { video => this.video = video }
width = { this.props.constraints.video.width }
height = { this.props.constraints.video.height }
autoPlay = "true"
/>
</div>
)
}
The picture button component could look something like this:
takePic() {
const element = this.props.getPic
const encoder = new GifReadWrite.Encoder()
encoder.setRepeat(0)
encoder.setDelay(100)
encoder.start()
const canvas = document.createElement("canvas")
canvas.width = element.offsetWidth
canvas.height = element.offsetHeight
canvas.drawImage(video, 0, 0)
encoder.addFrame(context)
return {
type: RECORD_IMAGE,
payload: encoder
}
}
...
render() {
return (
<button onClick = { () => this.takePic() }>Take Picture</button>
)
}
And then a parent component to bind the button and the video component:
getPicture() {
return this.video.getPicture()
}
...
render() {
return (
<div>
<VideoElement ref="video => this.video = video"/>
<TakePictureButton getPic="() => this.getPicture()" />
</div>
)
}
Disclaimer: I'm not sure how the draw function works, but you get the idea

Many thanks to #stefan-van-de-vooren for setting me on the right path! My situation was complicated by the child components using Redux connect, so getting them to work required some additional setup.
First, set up the parent component to call one child from the other:
setMedia(pic) {
return this.control.getWrappedInstance().setMedia(pic)
}
getPicture() {
return this.display.getWrappedInstance().getPicture()
}
componentDidMount() {
this.setMedia( this.getPicture() )
}
render() {
return (
<div>
<DisplayContainer ref= { display => this.display = display } />
<ControlContainer ref= { control => this.control = control } />
</div>
)
}
Because Redux connect returns a higher order component, we need to use getWrappedInstance() to gain access to the child functions. Next, we need to enable getWrappedInstance() in our child components by telling connect to use refs. Also, we will set up our child functions.
DisplayContainer:
getPicture() {
return this.video
}
...
render() {
return ( <video ref= { video => this.video = video } /> )
}
export default connect(mapStateToProps, null, null, { withRef: true })(Display)
ControlContainer:
setMedia(pic) {
this.setState({ media: pic })
}
takePic(event) {
event.preventDefault()
this.props.RecordImage(this.state.media, this.canvas, this.props.encoder)
}
...
render() {
return(
<button onClick= { this.takePic }>Take Picture</button>
<canvas ref= { canvas => this.canvas = canvas } />
)
}
export default connect(mapStateToProps, mapDispatchToProps, null, { withRef: true })(Control)
The action creator remains the same:
const RecordImage = function(video, canvas, encoder) {
if(!encoder) {
encoder = new GifReadWrite.Encoder()
encoder.setRepeat(0)
encoder.setDelay(100)
encoder.start()
}
const context = canvas.getContext('2d')
context.drawImage(video, 0, 0)
encoder.addFrame(context)
return {
type: RECORD_IMAGE,
payload: encoder
}
}
I should note, we are using refs quite a bit here. It is necessary in this case because the context.drawImage() needs the actual <video /> element in order to work. However refs are sometimes considered an anti-pattern, and it is worth considering if there is a better approach. Read this article for more information: https://reactjs.org/docs/refs-and-the-dom.html#dont-overuse-refs
A better solution would be to grab the image directly from the MediaStream. There is an experimental grabFrame() method that does just that, but as of Feb 2018, there is limited browser support. https://prod.mdn.moz.works/en-US/docs/Web/API/ImageCapture/grabFrame

Related

Error when using setState inside a ReactJS API

I'm using an API called 'react-svg', it allows me to inject SVG file into the DOM and also have a copy of the SVG element so I can use it later.
This API has a functionality called 'afterInjection', it's basically a function called after injecting the SVG file, inside that function I tried to update the component state and I got an error :
Error: Maximum update depth exceeded. This can happen when a component repeatedly calls setState inside componentWillUpdate or componentDidUpdate. React limits the number of nested updates to prevent infinite loops.
import { ReactSVG } from 'react-svg';
import Sketch from '../sketch/Sketch';
import './ImageUploader.scss';
import Complex from 'complex.js';
class ImageUploader extends Component {
constructor( props ) {
super( props );
this.state = {
pointsArray : [],
fileURL : null,
file: null
}
this.handleChange = this.handleChange.bind(this);
}
handleChange = (event) => {
this.setState({
file : event.target.files[0],
fileURL: URL.createObjectURL(event.target.files[0])
})
}
add_svg = () => {
if (this.state.fileURL !== null) {
return (
<ReactSVG
src={this.state.file.name}
afterInjection={(error, svg) => {
if (error) {
console.error(error)
return
}
this.convert_path( svg.children[0].children[0] );
}}
/>
);
}
return null;
}
convert_path = path => {
let points = [];
const length = path.getTotalLength();
const step = length / 100;
for (let i = length - 1; i >= 0; i -= step) {
// console.log(path.getPointAtLength(i));
points.push( new Complex( path.getPointAtLength(i).x, path.getPointAtLength(i).y) );
}
this.setState({ pointsArray : points });
}
render() {
return (
<div >
<div className="upload_sketch_container">
<div className="input_container">
<input type="file" id="file" onChange={this.handleChange} />
<label htmlFor="file">Upload SVG file</label>
</div>
<div className="image_sketch_container">
<div className="image_container" >
{this.add_svg()}
</div>
<div className="sketch_container">
<Sketch data={this.state.pointsArray} />
</div>
</div>
</div>
</div>
);
}
}
export default ImageUploader;
The error message you posted is exactly correct. You've created an infinite loop in your component. You render your component, which calls add_svg, which calls convert_path, which updates state, which triggers a rerender, which calls add_svg, etc. If you coment out the call to this.convert_path in add_svg, your infinite loop problem will vanish.

HTMLImageElement not valid as a React Child

I'm trying to load an image asynchronously and only when it's been loaded, display it in a React app.
componentDidMount() {
const img = new Image();
img.onload = () => {
this.setState({
originalImage: img,
});
}
img.src = './images/testImage.jpg'
}
render() {
return (
<main>
{
this.state.originalImage
}
</main>
);
}
I'm getting below error:
Objects are not valid as a React child (found: [object HTMLImageElement])
I would like to know why this error is happening. Of course if I just add an <img> tag, it works fine.
React cannot directly display HTML Element. If you want to create elements programmatically you have to use functions provided by React.
componentDidMount() {
const url = 'https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRYd74A26KhImI1w9ZBB-KvWUcasVnQRe_UOrdZPqI4GOIN0mC3EA';
let img = React.createElement(
"img",
{
src: url,
},
)
this.setState({
originalImage: img,
})
let divExample = React.createElement(
'div',
null,
`Hello World`
);
this.setState({
divExample: divExample
})
}
render() {
return (
<div>
<div>Hi</div>
{
this.state.divExample
}
<main>
{
this.state.originalImage
}
</main>
</div>
);
}
React parses jsx element like div, img, HelloWorld (custom) etc. and create React Elements out of it.
As the error says, this.state.originalImage is an object. You probably are looking for it's src prop, which you can use like so:
render() {
return (
<main>
<img src={this.state.originalImage.src}/>
</main>
);
}
another idea is dangerouslysetInnerHTML - something like this ?
<div dangerouslySetInnerHTML={{ __html: this.state.originalImage.outerHTML }}/>}

react-virtualized grid retrieves image from service but does not render until scroll action

I am building a component that displays data retrieved from a server in a react-virtualized grid. Everything works fine with the exception of one issue: When the component first renders, the images that are retrieved are not displayed. They will, however, become available once a scroll action takes place.
I think it has something to do with the way my Promises work but I can't seem to find a solution.
The image data is successfully retrieved and the state is successfully updated with the data, however it is still not displayed.
The way I understand this to work is that when any cells in the grid are rendered using the "rowRenderer" callback, the "whenSectionRendered" callback should fire causing the following sequence:
a call to "loadNewImagesBetweenIndices"
a call to "loadHistImage" for each row between the indices provided
"loadHistImage" returns a promise from the imageService.retrieveImage function that receives the base64 image, then places the image data as "image_decision" on my object and calls "updateSpotHistoryData" with the new object.
"updateSpotHistoryData" updates the state, which replaces the "spotHistoryData" state with the new data and triggers a re-render.
I expect the re-render to make the image available to display when the "HistoryImageButton" is clicked, but it is not displayed until a scroll event occurs. This is my issue.
I have tried the following things:
Calling "loadNewImagesBetweenIndices" in "componentDidMount" and/or "componentDidUpdate".
Calling "loadHistImage" directly in the "rowRenderer" callback.
Forcing a re-render of the object by using setState again.
Removing the "clearImagesBetweenIndicesNotInView" call from "whenSectionRendered" function.
class HistoryDisplay extends PureComponent{
constructor(props) {
super(props);
this.state = {
width: 1000,
previousIdxStart: -1,
previousIdxStop: -1,
spotHistoryData: ''
}
this.cellRenderer = this.cellRenderer.bind(this);
this.whenSectionRendered = this.whenSectionRendered.bind(this);
this.loadHistImage = this.loadHistImage.bind(this);
this.loadHistory = this.loadHistory.bind(this);
this.removeHistImages = this.removeHistImages.bind(this);
}
componentDidMount() {
this.resetRowWidthIfRequired();
this.loadHistory(this.props.spotId);
}
componentDidUpdate(prevProps) {
this.resetPrevIdxIfRequired(prevProps);
if(prevProps.spotId !== this.props.spotId){
this.loadHistory(this.props.spotId);
}
}
...
async loadHistImage(imageId) {
return imageService.retrieveImage(imageId)
.then(data => {
const newHistData = this.state.spotHistoryData.map(x => {
if (x.id_image_decision === imageId){
x.image_decision = data;
}
return x;
});
this.updateSpotHistoryData(newHistData);
});
}
updateSpotHistoryData(newHistoryData) {
this.setState({
spotHistoryData: newHistoryData
});
}
whenSectionRendered({rowOverscanStartIndex, rowOverscanStopIndex}) {
const { previousIdxStart, previousIdxStop } = this.state;
this.loadNewImagesBetweenIndices(
this.state.spotHistoryData,
rowOverscanStartIndex,
rowOverscanStopIndex);
this.clearImagesBetweenIndicesNotInView(
rowOverscanStartIndex,
rowOverscanStopIndex,
previousIdxStart,
previousIdxStop);
}
loadNewImagesBetweenIndices(array, startIdx, stopIdx) {
for(let rowIndex=startIdx; rowIndex<stopIdx; rowIndex++) {
this.loadHistImage(array[rowIndex].id_image_decision)
}
this.setState({
previousIdxStart: startIdx,
previousIdxStop: stopIdx
});
}
cellRenderer({columnIndex, key, rowIndex, style}) {
const { spotHistoryData } = this.state;
return(
<div key={key} >
{this.createCell(spotHistoryData, columnIndex, rowIndex, style)}
</div>
);
}
createCell(items, columnIndex, rowIndex, style){
const formattedTimestamp = (new Date(items[rowIndex].time_stamp_decision)).toLocaleString();
const btnColor = 'white';
switch(columnIndex){
case 0:
return this.renderHistoryImageButton(
this.createCellId(columnIndex, rowIndex),
btnColor,
style,
formattedTimestamp,
items[rowIndex].image_decision
);
case 1:
return this.renderHistoryDataButton(
this.createCellId(columnIndex, rowIndex),
btnColor,
style,
'Timestamp',
formattedTimestamp
);
...
}
}
render() {
const { height, rowHeight } = this.props;
const { width, spotHistoryData } = this.state;
return(
<Col>
<Grid
width={width}
height={height}
rowHeight={rowHeight}
cellRenderer={this.cellRenderer}
rowCount={spotHistoryData.length}
columnCount={7}
columnWidth={this.getCellWidth}
estimatedColumnSize={100}
overscanRowCount={3}
overscanColumnCount={3}
onSectionRendered={this.whenSectionRendered}
noContentRenderer={this.renderNoContent}
/>
</Col>
);
}
}
My expectations are that once the promise returns that provides the image data and updates the state of the "spotHistoryData" object, the images attached to the objects would be displayed without any scroll action being required.

Switch to next video from Youtube API (React)

I am attempting to have the next video in a channel play right after the other. Currently, the website has the videos showing one after the other, but my goal is to show one video and the second one plays right after the other is done. I have the function set up for the video ending, but right now it just causes an alert. I am using the Youtube Data API to pull in the videos and their information.
Here is a snippet of the code I am using:
constructor() {
super();
this.state = {
videos: [],
};
}
componentDidMount() {
fetch('https://www.googleapis.com/youtube/v3/search?key='APIKey'&channelId=UCXIJgqnII2ZOINSWNOGFThA&part=snippet,id&order=date&maxResults=2')
.then(results => {
return results.json();
}).then(data => {
let videos = data.items.map((videos) => {
return(
<div key={videos.items}>
<YouTube
className="player"
id="video"
videoId={videos.id.videoId}
opts={VIDEO_OPTS}
onEnd={this.playNextVideo}
/>
<h2>{videos.snippet.title}</h2>
<p className="channel">Video by: {videos.snippet.channelTitle}</p>
</div>
);
});
this.setState({videos: videos});
console.log("state", this.state.videos);
})
}
playNextVideo = () => {
alert('The video is done!');
}
I suggest you to do few things a little bit different.
First save the results.json(); to your videos variable in the state and not the whole youtube component, that's bad practice.
Second save another variable in your state that indicates the current playing video id (playingVideoId). Initialize it in the componentDidMount and change it in your playNextVideo function like this:
constructor() {
super();
this.index=0;
}
componentDidMount() {
fetch('https://www.googleapis.com/youtube/v3/search?key='APIKey'&channelId=UCXIJgqnII2ZOINSWNOGFThA&part=snippet,id&order=date&maxResults=2').then(results => {
this.setState({videos: results.json()});
this.setState({playingVideoId: this.state.videos[this.index]});
})}
playNextVideo = () => {
this.setState({playingVideoId: this.state.videos[++this.index]});
}
Now use the render function to render the component
render() {
return(
<YouTube
className="player"
id="video"
videoId={this.state.playingVideoId}
opts={VIDEO_OPTS}
onEnd={this.playNextVideo}
/>
);
}

Why a child component's state keeps clearing?

I have multiple layers of React components for getting an embed from a music service API, including a higher-order component that hits the API to populate the embed. My problem is that my lowest-level child component won't change state. I basically want the populated embed (lowest level component) to display an album cover, which disappears after clicking it (revealing an iframe), and whose state remains stable barring any change in props higher up (by the time this component is revealed, there should be no other state changes aside from focus higher up). Here's the code:
Parent:
return (
/*...*/
<Embed
embed={this.props.attributes.embed}
cb={updateEmbed}
/>
/*...*/
First child ( above):
render() {
const {embed, className, cb} = this.props;
const {error, errorType} = this.state;
const WithAPIEmbed = withAPI( Embed );
/*...*/
return <WithAPIEmbed
embed={embed[0]}
className={className}
cb={cb}
/>;
/*...*/
withAPI:
/*...*/
componentWillMount() {
this.setState( {fetching: true} );
}
componentDidMount() {
const {embed} = this.props;
if ( ! embed.loaded ) {
this.fetchData();
} else {
this.setState( {
fetching: false,
error: false,
} );
}
}
fetchData() {
/*... some API stuff, which calls the callback in the top level parent (cb()) setting the embed prop when the promise resolves -- this works just fine ...*/
}
render() {
const {embed, className} = this.props;
const {fetching, error, errorType} = this.state;
if ( fetching ) {
/* Return some spinner/placeholder stuff */
}
if ( error ) {
/* Return some error stuff */
}
return (
<WrappedComponent
{...this.props}
embed={embed}
/>
)
}
And finally the last child I'm interested in:
constructor() {
super( ...arguments );
this.state = {
showCover: true,
};
}
render() {
const {embed, setFocus, className} = this.props;
const {showCover} = this.state;
if ( showCover ) {
return [
<div key="cover-image" className={classnames( className )}>
<figure className='cover-art'>
<img src={embed.coverArt} alt={__( 'Embed cover image' )}/>
<i onClick={() => {
this.setState( {showCover: false,} );
}}>{icon}</i> // <-- Play icon referenced below.
</figure>
</div>,
]
}
return [
<div key="embed" className={className}>
<EmbedSandbox
html={iframeHtml}
type={embed.embedType}
onFocus={() => setFocus()}
/>
</div>,
];
}
My issue is that clicking the play icon should clear the album cover and reveal the iframe embed, but even though the click is registering, the state never changes (or does and then changes back). I believe it's because a higher-level component is mounting/unmounting and reinstantiating this component with its default state. I could move this state up the tree or use something like Flux, but I really feel I shouldn't need to do that, and that there's something fundamental I'm missing here.
The problem is that const WithAPIEmbed = withAPI( Embed ); is inside the render method. This creates a fresh WithAPIEmbed object on each render, which will be remounted, clearing any state below. Lifting it out of the class definition makes it stable and fixes the problem.

Categories