This system is closer to a B2B institutional device-fleet scenario than a consumer calling app. In healthcare, eldercare, and similar facilities, many shared devices may need to stay online for long periods and keep calls usable in noisy rooms where pickup distance and speech quality both matter.
Server-side audio noise reduction for WebRTC calls should start with a narrow validation target: can a Go media service receive an Opus track with Pion, decode it to PCM, run FFmpeg’s RNN noise reduction filter, and produce an output that is worth evaluating? This prototype does not try to replace WebRTC’s built-in audio processing.
The source experiment is based on a public sample project: snowlyg/webrtc_denoise_use_ffmpge. It is a prototype, not production-ready RTC infrastructure.
Background
WebRTC already includes audio processing blocks such as echo cancellation, noise suppression, and automatic gain control. In a controlled device environment, using the client-side WebRTC audio stack is usually the first and best option.
In field deployments, the device side is not always controlled:
- microphones and speakers may vary across batches.
- hardware acoustic design may change faster than the software release cycle.
- vendor firmware may expose inconsistent audio behavior.
- tuning each device family deeply can create a high maintenance cost.
That makes a server-side experiment useful. If a service can receive audio, apply a known filter, and compare the result offline, it becomes easier to decide whether server-side processing is a viable supplement for specific environments.
The first safe milestone is not real-time forwarding. It is file-based validation.
Processing Boundary
The server-side path is:
- Browser or client sends a WebRTC audio track.
- Pion receives the remote track in
OnTrack. - The service reads RTP packets with
track.ReadRTP(). - Opus payload is decoded to PCM.
- PCM is written to an FFmpeg process through stdin.
- FFmpeg applies
arnndnwith an RNN model. - The filtered output is written to a file for comparison.
This boundary matters because RTP, Opus, PCM, and FFmpeg raw audio input are different formats. Mixing them up can produce a file, but not necessarily valid audio.
Minimal Experiment
The prototype starts from Pion’s save-to-disk idea: connect a browser page to a Go process, receive audio/video, and write media out for inspection.
The public dependencies are:
The prototype uses Pion v4 and an Opus decoder from gopkg.in/hraban/opus.v2.
Decoding Opus to PCM
The audio track is usually Opus at 48 kHz. The example code keeps the sample rate explicit and decodes the RTP payload into an int16 PCM buffer.
var sampleRate = 48000
var channels = 2
var frameSizeMs = 60
frameSize := channels * frameSizeMs * sampleRate / 1000
pcm := make([]int16, frameSize)
dec, err := opus.NewDecoder(sampleRate, channels)
if err != nil {
return err
}
for {
rtpPacket, _, err := track.ReadRTP()
if err != nil {
return err
}
if rtpPacket == nil || len(rtpPacket.Payload) == 0 {
continue
}
n, err := dec.Decode(rtpPacket.Payload, pcm)
if err != nil {
continue
}
decoded := pcm[:n*channels]
buf := new(bytes.Buffer)
if err := binary.Write(buf, binary.LittleEndian, decoded); err != nil {
return err
}
if _, err := pipeWriter.Write(buf.Bytes()); err != nil {
return err
}
}Two details should not be hidden:
- The
channelsvalue must match the decoded stream assumptions. Do not hard-code stereo if the negotiated track is mono. - The FFmpeg input format must match the PCM buffer.
int16PCM maps tos16le; usings32lewithint16bytes is a bug to review before treating results as trustworthy.
Running FFmpeg arnndn
FFmpeg’s arnndn filter applies an RNN noise reduction model. For a validation file, the command shape is:
cmd := exec.Command(
"ffmpeg",
"-v", "warning",
"-f", "s16le",
"-ac", "2",
"-ar", "48000",
"-i", "pipe:0",
"-af", "arnndn=m=models/cb.rnnn",
"-c:a", "libopus",
"-b:a", "64k",
"output_rnn.opus",
)The prototype repository currently shows the experimental shape, including the pipe to FFmpeg. Before using this in a production path, the raw audio format, channel count, frame duration, process lifecycle, and error handling all need to be made explicit.
Validation
For the first pass, I would validate with files:
- Capture the unprocessed Opus output.
- Decode or open the file in an audio tool such as Audacity.
- Generate
output_rnn.opusfrom the FFmpeg path. - Compare the perceived noise, waveform, and frequency content.
- Check whether speech quality was damaged while noise was reduced.
This comparison should not be reduced to “the waveform looks cleaner”. Noise suppression can also remove weak speech details, create artifacts, or change the perceived naturalness of a call. Listening tests are still required.
Production Boundaries
The file-based prototype is not the same as a real-time WebRTC media server.
For real-time use, the design has to answer:
- How much buffering is introduced by Opus decoding, FFmpeg stdin, filtering, and re-encoding?
- Is FFmpeg started once per call, per participant, or as a managed worker pool?
- What happens when FFmpeg exits, blocks, or falls behind?
- How are CPU and memory isolated from signaling services?
- Does the system return processed audio to WebRTC, or only record a cleaned stream?
- How are packet loss, jitter, RTP timestamps, and audio/video sync preserved?
My practical default is to keep the first implementation as an offline validation path. Only after measuring latency and CPU cost should it move toward a real-time forwarding path.
Takeaway
Server-side noise reduction can be useful when device-side audio behavior is hard to control, but it is not free. The clean boundary is:
- Pion receives the WebRTC track.
- Opus is decoded into correctly described PCM.
- FFmpeg
arnndnapplies an RNN model. - The result is validated first as a file.
- Real-time forwarding is a separate engineering decision.
This prototype isolates the audio processing boundary. The next decision can then be based on measured latency, resource cost, and listening results before turning it into production infrastructure.