Building realtime interfaces is very different from building static screens. In a traditional screen, the frontend usually responds to controlled inputs: forms, buttons, validations, HTTP calls, and relatively predictable states. In a realtime interface, the browser, operating system, permissions, hardware, network, an external SDK, and other users all participate directly in the experience.
That changes how frontend systems need to be designed.
When an application depends on audio, video, microphone permissions, device selection, live events, and multiuser sessions, the interface stops being only a visual layer. It becomes a coordination layer between multiple sources of truth: local React state, Browser APIs, remote session state, SDK events, and the user's actual environment.
This article is based on a concrete experience building a call/video interface with React, TypeScript, and a realtime SDK. During development, I customized the ParticipantView component from @stream-io/video-react-sdk to gain greater visual control over participants, placeholders, audio indicators, reactions, and the experience when video was disabled.
That apparently visual change had important technical consequences. By overriding the SDK's internal UI, some default behavior was also lost, including reaction animations. That meant explicitly rebuilding parts of the interface, handling realtime events, normalizing state, detecting audio with Web Audio, and validating edge cases through manual QA.
The goal of this post is not to explain React or the MediaDevices API from scratch. The goal is to document engineering decisions for building reliable realtime interfaces where users interact with audio, browser permissions, multimedia devices, session state, visual feedback, and real-world environmental failures.
1. The problem: realtime interfaces are not ordinary forms
A realtime interface differs fundamentally from a traditional screen because state does not depend only on the local user or on an HTTP request.
In a typical form, the flow is relatively controlled:
- The user types.
- React updates state.
- The input is validated.
- A request is sent.
- The API responds.
- Success or an error is displayed.
That flow can still be complex, but the frontend usually has a high level of control over what happens.
In an audio/video realtime interface, the situation changes. The user is not only interacting with buttons. They are also interacting with the browser, microphone, permissions, connected devices, the network, and other participants.
A call screen can fail for reasons that do not come directly from UI code:
- The user denies microphone permission.
- The browser blocks permission.
- No microphone is connected.
- A microphone exists but is in use by another application.
- The user changes devices during the session.
- The SDK emits events while the UI is still initializing.
- Another participant joins, leaves, or reconnects.
- The network degrades.
- The user refreshes the page during an active session.
- Browser behavior differs across Chrome, Safari, Firefox, desktop, or mobile.
That means realtime interfaces must be designed for uncertainty. Modeling only the happy path is not enough. Intermediate states, recoverable errors, and scenarios where the application cannot guarantee that everything will be available must be explicit.
In an audio interface, for example, a boolean such as isMuted is not enough. The UI needs to answer several different questions:
- Did the user grant microphone permission?
- Is an input device available?
- Is the microphone active?
- Is the user muted?
- Is there actual audio signal?
- Is the stream still alive?
- Does the state belong to the local user or a remote participant?
- Is the UI showing the current state or a stale one?
A poor design can show “microphone active” even when nobody can hear the user. It can display a speaking animation after the user stops speaking. It can also stop rendering reactions because a default SDK behavior was accidentally removed when the UI was customized.
Reliability in realtime interfaces depends on making these states explicit.
2. Browser APIs and the cost of depending on the user's environment
Working with Browser APIs means accepting that the frontend does not completely control its environment.
APIs such as MediaDevices, getUserMedia, and Web Audio connect an application to browser and operating-system capabilities. That enables audio workflows, video calls, device selection, and collaborative experiences, but it also introduces real variability.
Browser behavior may depend on:
- Operating system.
- Device type.
- Browser.
- Browser version.
- Permissions granted or blocked.
- Secure or insecure context.
- Connected devices.
- Autoplay policies.
- Previous user configuration.
Requesting microphone access may appear as simple as this:
async function requestMicrophoneAccess() {
try {
const stream = await navigator.mediaDevices.getUserMedia({
audio: true,
})
return {
ok: true,
stream,
}
} catch (error) {
return {
ok: false,
error,
}
}
}This is a useful starting point, but it is not enough for a production product. Getting or not getting a MediaStream is only one part of the flow.
More important questions appear immediately afterward:
- What state do we display while the browser is asking for permission?
- What message do we show when the user rejects permission?
- How do we distinguish denied permission from a missing device?
- What happens when a stream starts successfully and later stops?
- How does the UI react if headphones are connected during the session?
- How do we recover if the selected device disappears?
- How do we test these scenarios in QA?
The browser gives us access to media capabilities, but the user experience depends on how the state layer around those APIs is designed.
In advanced frontend work, especially realtime UI, the question is not only “how do I call this API?” It is “which product states are derived from this API, and how do I represent them reliably?”
3. MediaDevices API: permissions, devices, and errors
The MediaDevices API should not be viewed only as a way to request access to a microphone or camera. In a real application, it represents a boundary between the UI and the user's environment.
A getUserMedia call can produce several outcomes:
- Permission is granted.
- Permission is denied.
- The browser blocks access.
- No device exists.
- The device is busy.
- Requested constraints cannot be satisfied.
- An unexpected error occurs.
It is better to model explicit states instead of relying on ambiguous booleans.
type AudioPermissionState =
| 'idle'
| 'requesting'
| 'granted'
| 'denied'
| 'device-not-found'
| 'device-in-use'
| 'error'This model better represents the actual flow.
A boolean such as hasPermission can hide too much information. false may mean the user has not decided yet, explicitly denied permission, has no microphone, or that the browser blocked the request. Those cases should not look identical in a sensitive UI.
A more useful function can normalize the result:
type RequestMicrophoneResult =
| {
ok: true
stream: MediaStream
}
| {
ok: false
reason: AudioPermissionState
error?: unknown
}
export async function requestMicrophoneAccess(): Promise<RequestMicrophoneResult> {
try {
const stream = await navigator.mediaDevices.getUserMedia({
audio: true,
})
return {
ok: true,
stream,
}
} catch (error) {
if (error instanceof DOMException) {
if (error.name === 'NotAllowedError') {
return { ok: false, reason: 'denied', error }
}
if (error.name === 'NotFoundError') {
return { ok: false, reason: 'device-not-found', error }
}
if (error.name === 'NotReadableError') {
return { ok: false, reason: 'device-in-use', error }
}
}
return { ok: false, reason: 'error', error }
}
}A component can then focus on rendering the normalized product state:
type AudioStatusProps = {
state: AudioPermissionState
}
export function AudioStatus({ state }: AudioStatusProps) {
if (state === 'requesting') return <span>Requesting microphone access</span>
if (state === 'granted') return <span>Audio ready</span>
if (state === 'denied') return <span>Microphone permission denied</span>
if (state === 'device-not-found') return <span>No microphone detected</span>
if (state === 'device-in-use') {
return <span>Microphone is being used by another application</span>
}
if (state === 'error') return <span>Audio could not be initialized</span>
return <span>Waiting for audio setup</span>
}The advantage is not only technical. A reliable interface does not hide failure; it translates failure into understandable feedback or a recoverable action.
4. Device selection: selecting devices without breaking the experience
Device selection feels like a secondary feature until the product is used by real users.
In local development, we often use one microphone and one browser. In production, a user may have:
- Built-in microphone.
- Bluetooth headset.
- External audio interface.
- Camera with integrated microphone.
- Virtual devices.
- A microphone locked by another application.
- Devices that connect or disconnect during the session.
The foundation for listing devices is usually enumerateDevices:
export async function getAudioInputDevices() {
const devices = await navigator.mediaDevices.enumerateDevices()
return devices.filter((device) => device.kind === 'audioinput')
}But listing devices does not solve the whole experience. Some browsers do not expose device labels until the user grants permission, so a list can initially contain empty or generic labels.
A mapping layer protects the UI from those details:
type AudioDevice = {
deviceId: string
label: string
}
export function mapMediaDevicesToAudioInputs(
devices: MediaDeviceInfo[],
): AudioDevice[] {
return devices
.filter((device) => device.kind === 'audioinput')
.map((device, index) => ({
deviceId: device.deviceId,
label: device.label || `Microphone ${index + 1}`,
}))
}It is also useful to model selection states:
type DeviceSelectionState =
| 'loading'
| 'ready'
| 'permission-required'
| 'empty'
| 'selected-device-missing'
| 'error'Those states make it possible to handle cases where the list is loading, permission is required, no devices exist, the selected device disappears, or the browser API fails.
In a Material UI implementation, components such as Select, MenuItem, and FormControl can improve the presentation, but the important architectural decision is recognizing that device selection is part of the reliability flow.
An unavailable microphone should not leave the screen broken. It should become a clear state with a possible next action.
5. Audio workflows: state, feedback, and recovery
A reliable audio workflow needs more than a mute button.
During a realtime session, a user needs to understand:
- Whether the microphone is connected.
- Whether permission was granted.
- Whether they are muted.
- Whether the browser receives actual signal.
- Whether other users can hear them.
- Whether an error occurred.
- What they can do to recover.
A useful decision is to separate “audio enabled” from “audio detected.” A participant can have audio enabled without currently producing any audible signal.
The SDK may expose states such as hasAudio(participant) and participant.isSpeaking, but those values do not always answer the same question. In my case, participant.isSpeaking was available, but it reacted better when the user spoke loudly than when they spoke at a normal volume. That led me to build a custom hook using participant.audioStream and Web Audio.
6. React as the state and interaction layer
React works well for realtime interfaces when it is used as a coordination layer rather than as a place where every responsibility is mixed together.
A common anti-pattern is putting everything in the main component:
- Requesting permissions.
- Listing devices.
- Rendering participants.
- Listening for SDK events.
- Detecting audio.
- Managing reactions.
- Normalizing errors.
- Managing layout.
A better approach is to isolate responsibilities into hooks, utilities, and view components.
A possible structure:
/realtime
/components
ParticipantTile.tsx
AudioStatus.tsx
AudioDeviceSelect.tsx
ReactionOverlay.tsx
/hooks
useAudioDevices.ts
useMicrophonePermission.ts
useParticipantSpeaking.ts
useRealtimeReactions.ts
/utils
getEmojiFromCode.ts
mapMediaDevicesToAudioInputs.ts
normalizeMediaError.ts
/types
audio.ts
reactions.ts
participants.tsThis separation makes the flow easier to test, document, and modify.
7. Material UI and UI systems for a consistent experience
Material UI, NextUI, or any design system can help create a consistent interface, but they do not automatically solve the UX problem.
In realtime flows, a visual library can help with:
- Alert for important states.
- Dialog for permissions or blockers.
- Chip for compact status indicators.
- Tooltips for controls.
- Standardized loading and disabled states.
The key is visual hierarchy. The interface needs to communicate which state is local, which is remote, which is temporary, and which needs user action.
A realtime interface can become visually noisy very quickly if every event is displayed with the same importance.
8. Customizing a realtime SDK: visual control vs default behavior
One of the most important lessons from this case was that customizing an SDK's UI can have side effects.
Originally, the interface used Stream's ParticipantView directly:
<ParticipantView participant={participant} className="!max-h-[467px]" mirror />That component handled multiple responsibilities internally. Replacing the default UI gave us greater visual control, but it also meant recreating behavior that the SDK had previously provided automatically.
The custom implementation eventually had to explicitly handle:
- Participant identity.
- Video placeholder.
- Microphone state.
- Reaction overlay.
- emoji_code mapping.
- Custom audio detection.
This was not only a visual change. It was a frontend architecture decision.
9. Video placeholders: when there is no camera, there is still an experience
A user without active video should not feel like an incomplete case of the interface. They still need a clear representation.
A video placeholder serves several functions:
- Preserves the call layout.
- Identifies the participant.
- Provides an alternative when the camera is disabled.
- Can show initials or avatar information.
- Integrates speaking state.
- Supports light/dark mode.
- Keeps dimensions consistent with video tiles.
The placeholder is not a minor visual fallback. In a real call, it can be the primary state for many participants.
10. Realtime reactions and ephemeral events
When ParticipantViewUI was customized, one feature that disappeared was the SDK's default reaction rendering. To recover it, we needed to listen for SDK events and build our own overlay.
The relevant event was:
call.on('call.reaction_new', handleReactionCreated)A reaction is an ephemeral UI event. It does not necessarily need to live in global application state forever. Instead, the UI can normalize it, display it temporarily, and remove it after a timeout.
For example:
const reaction = {
id: crypto.randomUUID(),
emoji: getEmojiFromCode(event.reaction.emoji_code),
x: Math.floor(Math.random() * 80) + 10,
}This small normalization step represents an important pattern: adapt external data before it reaches presentation components.
The UI should not need to know whether a reaction arrives as thumbs_up, like, 👍, or another variant. A normalization layer keeps rendering logic simpler and reduces visual bugs.
This pattern also applies to:
- Connection states.
- Browser errors.
- Audio states.
- Multimedia devices.
- Participants.
- SDK messages.
- Session events.
A good frontend architecture does not pass raw external data everywhere. It creates view models and small utilities that convert external data into product state.
11. Normalizing external state before rendering
Realtime SDKs and Browser APIs expose technical structures that are useful at the integration layer but not always ideal for UI components.
Instead of spreading SDK-specific types throughout the application, I prefer adapting them near the boundary and exposing simpler product-oriented types to the UI.
That approach reduces coupling and makes it easier to replace or upgrade an SDK later.
12. Detecting whether a user is actually speaking
One of the most interesting details in the workflow was distinguishing between “audio is active” and “the user is speaking.”
hasAudio(participant) can tell us whether the participant has audio available or enabled. It does not measure signal.
The SDK exposed participant.isSpeaking, but in this case it was not sensitive enough for normal speech. It changed to true more reliably when the user spoke loudly.
The solution was to analyze participant.audioStream using Web Audio.
A simplified approach looks like this:
export function useParticipantSpeaking(stream?: MediaStream) {
const [isSpeaking, setIsSpeaking] = React.useState(false)
React.useEffect(() => {
if (!stream) return
const audioContext = new AudioContext()
const source = audioContext.createMediaStreamSource(stream)
const analyser = audioContext.createAnalyser()
const data = new Uint8Array(analyser.frequencyBinCount)
source.connect(analyser)
analyser.fftSize = 256
let animationFrameId = 0
const detect = () => {
analyser.getByteFrequencyData(data)
const average = data.reduce((sum, value) => sum + value, 0) / data.length
setIsSpeaking(average > 8)
animationFrameId = requestAnimationFrame(detect)
}
detect()
return () => {
cancelAnimationFrame(animationFrameId)
source.disconnect()
analyser.disconnect()
void audioContext.close()
}
}, [stream])
return isSpeaking
}This kind of hook is a good example of advanced frontend work because it combines React, Web Audio, streams, performance, cleanup, and UX.
13. Visual feedback: an animation also communicates state
After detecting whether a user was speaking, the next challenge was communicating that state visually.
An initial solution used an animated border. The problem was that a long animation could visually lag behind the actual signal. The user might stop speaking while the animation continued briefly.
For realtime state, a short transition often communicates truth more accurately than a long decorative animation.
This reflects a practical rule: visual feedback in realtime interfaces should follow actual state as closely as possible.
Not everything needs a long animation. Sometimes a stable border, glow, or highlight that appears and disappears quickly is more useful.
14. Multiuser interfaces: synchronization, presence, and shared state
A multiuser interface introduces another layer of complexity because the experience no longer depends only on the local user.
During a call, the user sees both their own state and the states of other participants. Each participant can have audio, video, reactions, connection, and presence state.
A participant model can separate concerns such as presence, audio, and video instead of treating the SDK object as the UI model.
The goal is not to completely hide the SDK. It is to prevent the entire interface from depending on its internal implementation details.
15. Error handling and edge cases
In realtime interfaces, edge cases are not rare exceptions. They are normal parts of the product.
Common media errors include:
- NotAllowedError: the user denied permission.
- NotFoundError: no device is available.
- NotReadableError: the device exists but cannot be used.
- OverconstrainedError: requested constraints cannot be satisfied.
- AbortError: the operation was interrupted.
- SecurityError: the context does not allow access.
- SDK-specific errors.
- Unnormalized browser failures.
A normalization function prevents every component from interpreting technical exceptions directly:
type NormalizedMediaError =
| 'permission-denied'
| 'device-not-found'
| 'device-in-use'
| 'constraints-not-supported'
| 'security-error'
| 'unknown-error'
export function normalizeMediaError(error: unknown): NormalizedMediaError {
if (!(error instanceof DOMException)) {
return 'unknown-error'
}
switch (error.name) {
case 'NotAllowedError':
return 'permission-denied'
case 'NotFoundError':
return 'device-not-found'
case 'NotReadableError':
return 'device-in-use'
case 'OverconstrainedError':
return 'constraints-not-supported'
case 'SecurityError':
return 'security-error'
default:
return 'unknown-error'
}
}The purpose of normalization is not to hide complexity. It is to convert technical failures into product states that can be rendered, tested, and documented.
16. QA cycles: testing beyond the happy path
QA is especially important in realtime interfaces because many bugs do not appear in local development.
Locally, we normally test with:
- One browser.
- One user.
- One microphone.
- Stable permissions.
- A familiar network.
Real users have very different environments.
Useful QA scenarios include:
- Permission granted.
- Permission denied.
- Permission blocked at browser level.
- No microphone connected.
- Device disconnected during a session.
- Switching devices during a session.
- Refreshing during an active call.
- Joining from two users.
- Reconnecting after network loss.
- Running in different browsers.
- Reactions while participants join or leave.
- Speaking softly vs loudly.
A useful pattern is turning every QA-discovered case into an explicit state. For example:
type AudioSetupState =
| 'idle'
| 'requesting'
| 'ready'
| 'permission-denied'
| 'device-not-found'
| 'device-lost'
| 'failed'QA also improves copy. “Audio error” does not help the user. “No microphone was detected. Connect a device and try again” communicates both the problem and a possible action.
17. Accessibility and visual clarity
Accessibility should not be treated as a later improvement in realtime interfaces. It is part of reliability.
Important practices include:
- Do not rely on color alone to indicate state.
- Use clear labels for microphone and camera controls.
- Add aria-label to icon-only buttons.
- Ensure focus behavior works in dialogs and controls.
- Maintain sufficient contrast for state indicators.
- Make critical errors readable by assistive technologies.
Visual clarity also applies to states such as “speaking.” A border or highlight can be useful, but it should remain stable. If it flickers constantly, the UI feels broken even when it is technically responding to signal.
18. Performance and stability in long sessions
Realtime sessions can last a long time. That changes how performance needs to be considered.
It is not enough for the screen to work during the first few seconds. It needs to stay stable with:
- Multiple participants.
- Frequent events.
- Reactions.
- requestAnimationFrame loops.
- Re-renders.
- Tab changes.
- Reconnections.
Clean up listeners
Whenever realtime events are subscribed to, cleanup must also exist:
useEffect(() => {
if (!call) return
call.on('call.reaction_new', handleReaction)
return () => {
call.off('call.reaction_new', handleReaction)
}
}, [call])Clean up streams and tracks
When a stream is no longer needed:
export function stopMediaStream(stream: MediaStream) {
stream.getTracks().forEach((track) => {
track.stop()
})
}Cancel animation frames
When using Web Audio or JS-controlled animation loops:
useEffect(() => {
let animationFrameId = requestAnimationFrame(detect)
return () => {
cancelAnimationFrame(animationFrameId)
}
}, [])Avoid unnecessary re-renders
In an interface with many participants, every unnecessary re-render can multiply quickly.
export const ParticipantTile = React.memo(function ParticipantTile(
props: ParticipantTileProps,
) {
// ...
})Ephemeral events should also be kept local when possible. Not every realtime event needs to live in a global store.
19. Practices that have worked well for me
Model states before designing components
Before building the UI, list the possible states. This reduces improvised decisions later.
type AudioSetupState =
| 'idle'
| 'checking-permissions'
| 'ready'
| 'permission-denied'
| 'device-not-found'
| 'device-lost'
| 'failed'Separate browser logic from presentation
MediaDevices, Web Audio, and permissions should not be directly mixed into presentation components.
Do not depend on one boolean
isAudioReady, hasAudio, or isMuted never explain the whole audio lifecycle by themselves.
Design recovery paths
Every error should answer one question: what can the user do now?
Test permissions from a clean state
It is important to test with fresh permissions, not only with permissions already granted in the development browser.
Document edge cases
When QA discovers a scenario, documenting it prevents that knowledge from remaining informal.
Use consistent components
Material UI or a design system helps maintain visual patterns, but the state model must still be correct at the architecture level.
Keep feedback visible without making it intrusive
Audio, connection, and presence indicators should help the user without overwhelming the screen.
Validate in multiple browsers
This is especially important when using MediaDevices, getUserMedia, or Web Audio.
Review what is lost when customizing an SDK
When replacing default UI, identify which behaviors came for free and which ones must now be rebuilt explicitly.
20. Common mistakes when building this type of interface
Treating permissions as a boolean
Permission is not simply true or false. It may be pending, blocked, denied, granted, or constrained by browser behavior.
Confusing hasAudio with “is speaking”
A participant can have audio active without producing any signal. They can also speak softly enough that a default SDK detector does not react reliably.
Ignoring the absence of devices
Development machines usually have a microphone, but real users may have none available or may have a device locked by another application.
Failing to clean up tracks or listeners
This can cause memory leaks, duplicate audio, or events firing multiple times.
Mixing realtime logic into visual components
When a presentation component also handles permissions, streams, events, and errors, it becomes difficult to maintain.
Designing only for the local user
In multiuser interfaces, remote state matters just as much as local state.
Ignoring manual QA
Many permission and hardware edge cases appear only when testing real scenarios manually.
Using animations that do not follow real state
If an animation continues after the user stops speaking, the UI communicates incorrect information.
Overriding an SDK without understanding its internal responsibilities
Custom UI provides control, but it can also remove default behaviors that need to be rebuilt afterward.
21. Lessons learned
The main lesson is that realtime interfaces must be designed for uncertainty.
When an application depends on audio, permissions, devices, SDK events, and multiuser sessions, the frontend becomes a coordination layer. React does more than render components; it organizes state, events, visual feedback, recovery, and resource cleanup.
MediaDevices and getUserMedia are powerful tools, but they expose the complexity of the user's environment. The browser may behave differently depending on permissions, operating system, device, or configuration. The architecture therefore needs explicit states and flexible recovery paths.
Customizing SDK components also requires care. Replacing a default UI may be necessary to meet product requirements, but it is important to understand which behavior is being removed. In my case, customizing ParticipantViewUI provided better control over participant presentation but also required rebuilding reactions, overlays, and indicators.
I also learned that detecting audio does not always mean reading one SDK property. Sometimes it is necessary to move one layer deeper and analyze the stream directly with Web Audio, tune sensitivity, and control visual feedback more precisely.
QA is not a final phase separate from development. In this kind of interface, QA discovers states that do not appear in local development. Each bug can become a better-defined state, clearer message, or documented test scenario.
Building a reliable realtime interface does not mean creating a perfect system. It means accepting that the environment can fail and designing an experience that responds clearly.
Signals for recruiters
- Built realtime interfaces with React, TypeScript, and Material UI.
- Integrated MediaDevices API, getUserMedia, and Web Audio for audio workflows.
- Customized realtime SDK components such as @stream-io/video-react-sdk.
- Designed robust state models for permissions, devices, streams, and browser errors.
- Experience building realtime UI, multiuser interfaces, and shared-state coordination.
- Implemented custom audio-signal detection with Web Audio.
- Rebuilt reaction overlays and other default SDK behavior after UI customization.
- Applied QA-driven state modeling for browser, device, and permission edge cases.
- Managed cleanup of listeners, streams, animation frames, and long-running realtime sessions.
- Designed recoverable and accessible feedback for multimedia workflows.
