Record system audio with chrome extension - javascript

I wanted to record system audio for 5 seconds (Audio is not just from browser tabs, can be from anywhere - system), is it possible to do with chrome extension?
Our intension is to search our servers for matching records(audio search) and present relevant information to users.
Example: If some audio is played by itunes or any software, we just wanted to record audio output directly from anything that is playing on the System.
Any help!

Related

How to access a device webcam and auto save the frames into the device without clicking any single button for capturing and saving?

I am working on a project which takes video input from user and puts the prediction and sends the video frames to the users. As it is easy on my device as it does not need any permission to access. But when it comes to the client side it asks for capturing and saving the frames which is a irritating task. I want a solution that it has to take the device camera permission and autosave the video frames into the device.
I tried opening the device camera using python OpenCV. It works fine with my system, but when it comes to the deployment stage python is unable to open the user device camera. I am searching for a solution which can open device camera using Javascript and autosave the video frames into the device without any single click for capturing and saving the frames.

Web Audio Api integration with Web Speech Api - stream speaker/soundcard output to voice recognition api

Problem:
Ideally I would acquire the streaming output from the soundcard (generated by an mp4 file being played) and send it to both the microphone and speakers. I know I can use "getUserMedia" and "createChannelSplitter" (in the Web Audio Api) to acquire & split (based on Audacity analysis the original signal is in stereo) the user media into 2 outputs which leaves me with 2 problems.
getUserMedia can only get streaming input from the microphone
not from the soundcard (from what I have read)
streaming output can only be recorded/sent to a buffer and not sent
to the microphone directly (from what I have read)
Is this correct?
Possible workaround - stalled:
The user will most likely have a headset microphone on but one workaround I have thought of is to switch to the inbuilt microphone on the device and capture what comes out of the speakers and then switch back to the headset for user input. However, I haven't found a way to switch between the inbuilt microphone and the headset microphone without asking the user every time.
Is there a way to do this that I haven't found?
What other solutions would you suggest?
Project Explanation:
I am creating a Spanish language practice program/website written in html & javascript. An mp4 will play and the speech recognition api will display what it says on the screen (as it is spoken in Spanish) and it will be translated into english so the user hears, sees, and understands what is being said by the person speaking in the mp4. Then the user will use the headset microphone to answer the mp4 person (often the inbuilt microphone doesn't give good enough quality for voice recognition - depending on the device - thus the use of the headset).
flow chart of my workaround using inbuilt microphone
mp4->soundcard-> Web Audio Api -> channel 1 -> user's ears
channel 2 -> microphone input-> Web Speech Api-> html->text onscreen
flow chart of ideal situation skipping microphone input
mp4->soundcard-> Web Audio Api -> channel 1 -> user's ears
channel 2-> Web Speech Api-> html->text onscreen -> user's eyes
Another potential work around:
I would like to avoid having to manually strip an mp3 from each mp4 and then have to try and sync them so the voice recognition happens as the mp4 person speaks. I have read that I can run an mp3 through the voice recognition api.
The short answer is that there is not currently (12/19) a way to accomplish this on this platform with the tools and budget I have. I have opted for the laborious way to do this which is setting up individual divs with text blocks to be revealed as the person is speaking on a timer. I will still use the speech api to capture what the user says so the program can run the correct video in response.
Switching between speaker and user headset is a definite no go.
Speech recognition software usually requires clean and well captured audio. So, if the sound is coming from speakers, the users microphone is not likely to pick it up very well. And if the user is using headphones, then there is no way for the microphone to capture the audio at all.
As far as I know, you cannot send audio files Web Speech Api directly (I may be wrong here)
Web Speech Api Is not supported by all browsers so that is a downside to consider too: https://caniuse.com/#feat=speech-recognition
What I would recommend is checking out Google's Speech to text API: https://cloud.google.com/speech-to-text/
With this service you can send them directly the audio file and they will send back the transcription.
It does support streaming so you could have the audio transcribed at the same time it is playing. The timing wouldn't be perfect though.

Reading output audio data from Spotify Web Playback stream

I am currently playing around with audio visualization and I am trying to work with Spotify's Web Playback SDK to stream and analyze songs directly on my site.
However, I am unsure what the limitations are when it comes to actually reading the streamed data. I've noticed that an iframe is generated for the Spotify player, and I've read that spotify uses the encrypted media extensions to stream the audio on chrome.
Is it even possible to read the music data from the Spotify api? Maybe, I can read the output audio from the browser?
According to the web API documentation, you aren't able to play back full songs and get the audio data like you desire (for obvious reasons). Although, 30 second "song previews" are allowed through URL streaming, as well as full song playback on desktop browsers (excluding safari at the time of this post), with the Web Playback SDK.
However on the mobile API it is now possible to get the raw PCM data (Android or iOS). This will require you registering for a developers account and setting up the access tokens if you haven't already done so.
For quick reference, on Android it involves using the AudioController class.
EDIT : Thanks to #Leme for the Web Playback SDK link.

Ability to record web page within browser without any extension

I'm working on a POC to record webpage screen and send the recording back to server without any extensions or application software support. Tried various approaches but those are not meeting my requirement.
Requirement in detail: Need to record the web page without adding any extensions or application software.
Application details: Single page application developed with KnockoutJS
Below are the approaches tried:
RecordRTC ( https://www.webrtc-experiment.com/RecordRTC/Canvas-Recording/ )
But as we have duration of around 15 minutes for each recording, after 2 or 3 recordings browser is hanging or not reponding.
Screen recorder software's ( Ex: Apowersoft )
To use these tools ( though tried with the API ) user needs to install application software. As we don't want user to do any installation I'm unable to use any such screen recorder
Using session recording/replay feature of Anaytics tools such as ( mouseflow, inpectlet ). If we observe -> link here this is exctly meeting my requirement. As a user we no need to install anything infact user is not aware of the recording happening. But the problem with these tools such as Mouseflow are not providing the download of recording as video file, Recording's are saved in their server and we can play only with their embed script.
I feel approach 3 is closer to my requirement but as i'm unable to download that recorded session as video file I'm unable to move further. can some one has idea of any tool which can provide recording as video file.
Please provide your thought/inputs how i can meet my requirements.

HTML5 audio tag does not work in Android - Chrome when created in JS?

I'm using a .mp3 file, the .mp3 file plays okay when viewed directly and also when embeded using the HTML5 audio tag, however when creating the HTML5 audio tag in JS it does not play! (very strange)
I do not have this issue in any other browser/device, for example Desktop - Chrome works perfectly.
sound = document.createElement('audio');
sound.setAttribute('src', 'sound.mp3');
sound.play();
I've tested sound.canPlayType('audio/mpeg') and this produces true (so it is supported).
Perhaps there's a bug in Android - Chrome? (it is the latest version)
Looks like this is intended feature that spans more then just the Chrome browser. User interaction is required to get media elements to play.
Blink and WebKit have a setting for requiring a “user gesture” to play or pause an audio or video element, which is enabled in Opera for Android, Chrome for Android, the default Android browser, Safari for iOS and probably other browsers. This makes some sense, since mobile devices are used in public and in bed, where unsolicited sound from random Web sites could be a nuisance. Also, autoplaying video ads would waste bandwidth. Block Quote from 'blog.foolip.org'
Duplicate Threads from Other Users
Autoplay audio on mobile safari
How can I autoplay media in ios 4.2.1
Autoplay audio with ios 5 workaround?
Current Status
Developers have requested the deletion of 'mediaPlaybackRequiresUserGesture' which was reviewed and denied (for now). "We're going to gather some data about how users react to autoplaying videos in order to decide whether to keep this restriction."
Upon further inspection i found this...
"I misunderstood the outcome of the discussion (removing mediaPlaybackRequiresUserGesture) surrounding this topic. We need to keep this code in order to not break google.com while gathering data about this feature."
Google.com relies on the feature being disabled, otherwise it breaks (they didn't say what it breaks).
Original Bug Report
Try appending it to the document body.
document.body.appendChild(sound);
Though it is possible that mobile devices will not automatically play the audio or videos. If you are targeting mobile devices, autoplaying is considered bad practice since it can consume bandwidth. So it may be worth considering adding controls.
sound.setAttribute('controls', 'true');
OK, well, now that we know it won't work with audio, the only path left to you is to switch to the Web Audio API. You'll need to load the mp3 into an ArrayBuffer (e.g. using an XHR), then pass that to the decodeAudioData method, which gets you an Audio buffer that you can play back at will from an AudioBufferSourceNode.
Not every browser on every platform can play the mp3 audio format. Generally, as I would recommend, you should provide two <source> elements within your audio element, one providing the mp3 format, and another one providing the ogg vorbis format.
You can read more here: https://developer.mozilla.org/en-US/docs/Web/HTML/Supported_media_formats

Categories