Table of Contents

Receiving Voice

This section will cover how to receive audio from Discord voice channels using NetCord. You'll learn how to capture audio from users in a voice channel, decode the Opus-encoded audio data, and process it for various applications such as recording.

Receiving Audio

Receiving audio from VoiceClient is done through the VoiceReceive event. This event is triggered whenever a new Opus-encoded audio packet is received from Discord. The event handler receives a VoiceReceiveEventArgs object, which contains:

  • Frame: The Opus-encoded audio data received.
  • Ssrc: The SSRC of the user who sent the audio data.
  • Timestamp: The timestamp of the received audio packet. This is useful for jitter buffering and synchronization.
  • SequenceNumber: The sequence number of the received audio packet.

The VoiceReceiveEventArgs does not contain any user information, only the SSRC. You can use the SSRC to identify which user sent the audio data by looking it up in the VoiceClient's cache. See the example below for how to do that.

voiceClient.VoiceReceive += args =>
{
    if (voiceClient.Cache.SsrcUsers.TryGetValue(args.Ssrc, out var userId))
        Console.WriteLine($"Received audio from user {userId}");
    else
        Console.WriteLine($"Received audio from unknown user with SSRC {args.Ssrc}");

    return default;
};

You may notice that the VoiceReceiveEventArgs is a ref struct. This is because it uses an internal buffer used by NetCord to receive audio data from Discord. You don't need to worry about managing this buffer carefully, because the compiler ensures that you don't accidentally store a reference to it or use it after the event handler returns. This also comes with some limitations. Handlers of this event cannot be async. If you need to perform asynchronous operations with the received audio data, you can copy the Opus frame data to a separate buffer and then process it asynchronously.

voiceClient.VoiceReceive += args =>
{
    // You may consider renting a buffer
    // from 'ArrayPool<T>' in a real application
    var ownedFrame = args.Frame.ToArray();

    return HandleAsync(ownedFrame, args.Ssrc);

    static async ValueTask HandleAsync(byte[] frame, uint ssrc)
    {
        await Task.Delay(100); // Simulate async processing

        Console.WriteLine($"Processed audio frame from SSRC {ssrc}");
    }
};

Also keep in mind that the VoiceReceive event is raised for every received audio packet, which can be quite frequent. If you perform heavy (synchronous) processing in the event handler, it can cause your bot to stop keeping up with the incoming audio data. Consider using Channel to buffer the received audio data and process it in a separate task to avoid this issue.

// Configure the channel to drop frames when the buffer is full;
// 1000 frames means from 2.5 to 120 seconds of
// audio depending on the Opus frame duration;
// Dropping frames should generally never happen, but it is a good
// idea to handle it just in case to avoid running out of memory
var channel = Channel.CreateBounded<(byte[] Frame, uint? Timestamp)>(new BoundedChannelOptions(1000)
{
    FullMode = BoundedChannelFullMode.DropWrite,
});

var writer = channel.Writer;

voiceClient.VoiceReceive += args =>
{
    // You may consider renting a buffer from 'ArrayPool<T>' in a real application
    var frame = args.Frame.ToArray();

    return writer.WriteAsync((frame, args.Timestamp));
};

await foreach (var (frame, timestamp) in channel.Reader.ReadAllAsync())
    Console.WriteLine($"Received audio frame of size {frame.Length} with timestamp {timestamp}");

Handling Packet Loss and Sequencing

Because Discord sends voice data over UDP, audio packets can sometimes arrive out of order, get delayed, or drop completely. To handle disorganized packets, you might want to implement an audio sequencer or a jitter buffer. For packets that are lost entirely, you can take advantage of Opus features like Forward Error Correction (FEC) and Packet Loss Concealment (PLC).

While building these mechanisms is beyond the scope of this guide, NetCord provides the data you need to do so. You can use SequenceNumber to sort packets into their correct order and detect missing ones, and Timestamp to determine precise audio timing.

Opus Decoding

Discord sends audio in Opus format, which you may need to decode to PCM for processing. NetCord provides OpusDecodeStream, which decodes Opus-encoded audio data that gets written into PCM format and writes the PCM data to the stream passed into its constructor. It uses OpusDecoder internally, which can be used directly if you need more control over the decoding process.

When creating the OpusDecodeStream you need to specify the PCM format and the number of channels in addition to the stream to write the decoded audio data to. The supported PCM formats are 16-bit signed integer (Short) and 32-bit floating point (Float). Float is generally preferred for high-quality audio. The number of channels can be either 1 (Mono) or 2 (Stereo).

OpusDecodeStream opusDecodeStream = new(stream,
                                        PcmFormat.Float,
                                        VoiceChannels.Stereo);

Updating Our Bot

Now it's time to finally add a /record command to our bot! It records what a user is saying in a voice channel and sends it to the channel where the command was used, either when a max file size is triggered or when the bot leaves the voice channel.

host.AddSlashCommand("record", "Records audio", async (ApplicationCommandContext context, User? user = null) =>
{
    if (context.Guild is not { } guild)
    {
        await context.Interaction.SendResponseAsync(InteractionCallback.Message(new InteractionMessageProperties()
            .WithContent("The guild is not available. Try again later.")
            .WithFlags(MessageFlags.Ephemeral)));
        return;
    }

    var guildId = guild.Id;

    if (!voiceInstances.TryGetValue(guildId, out var voiceInstance) || voiceInstance is null)
    {
        await context.Interaction.SendResponseAsync(InteractionCallback.Message(new InteractionMessageProperties()
            .WithContent("Not connected to a voice channel in this guild.")
            .WithFlags(MessageFlags.Ephemeral)));
        return;
    }

    using var job = voiceInstance.TryEnterJob(VoiceJobType.Recording);
    if (job is not { CancellationToken: var cancellationToken })
    {
        await context.Interaction.SendResponseAsync(InteractionCallback.Message(new InteractionMessageProperties()
            .WithContent("Already recording audio in this guild.")
            .WithFlags(MessageFlags.Ephemeral)));
        return;
    }

    await context.Interaction.SendResponseAsync(InteractionCallback.Message($"Recording..."));

    var voiceClient = voiceInstance.Client;

    user ??= context.User;

    using var ffmpeg = Process.Start(new ProcessStartInfo
    {
        FileName = "ffmpeg",
        ArgumentList =
        {
            "-f", BitConverter.IsLittleEndian ? "f32le" : "f32be",
            "-ar", "48000",
            "-ac", "1",
            "-i", "pipe:0",
            "-f", "ogg",
            "-c:a", "libopus",
            "pipe:1",
        },
        RedirectStandardInput = true,
        RedirectStandardOutput = true,
    })!;

    using var ffmpegInput = ffmpeg.StandardInput.BaseStream;
    using OpusDecodeStream opusDecodeStream = new(ffmpegInput, PcmFormat.Float, VoiceChannels.Mono);

    Func<VoiceReceiveEventArgs, ValueTask> voiceReceive = args =>
    {
        if (voiceClient.Cache.SsrcUsers.TryGetValue(args.Ssrc, out var userId) && userId == user.Id)
            opusDecodeStream.Write(args.Frame);

        return default;
    };

    voiceClient.VoiceReceive += voiceReceive;

    using var ffmpegOutput = ffmpeg.StandardOutput.BaseStream;

    using MemoryStream outputStream = new();

    string? recordingStopReason;

    var copyBuffer = ArrayPool<byte>.Shared.Rent(4096);

    var ffmpegRunning = true;

    const int MaxFileSize = 10 * 1024 * 1024; // 10 MiB

    try
    {
        int count;
        while (true)
        {
            if ((count = await ffmpegOutput.ReadAsync(copyBuffer, cancellationToken)) is 0)
            {
                // Should never happen since Ffmpeg should be keept alive
                // until we kill it, but we'll handle it just in case

                ffmpegRunning = false;
                recordingStopReason = "End of stream";
                break;
            }

            outputStream.Write(copyBuffer.AsSpan(0, count));

            if (outputStream.Length >= MaxFileSize)
            {
                recordingStopReason = "Maximum file size exceeded";
                break;
            }
        }
    }
    catch (Exception ex)
    {
        if (ex is not OperationCanceledException and not AggregateException { InnerException: OperationCanceledException })
            throw;

        recordingStopReason = "Disconnected";
    }
    finally
    {
        voiceClient.VoiceReceive -= voiceReceive;

        if (ffmpegRunning)
        {
            // Flush the stream to ensure all data is written to Ffmpeg
            // This way we don't cut off the end of the recording
            await opusDecodeStream.FlushAsync();
            await opusDecodeStream.DisposeAsync();

            await ffmpegOutput.CopyToAsync(outputStream);

            ffmpeg.Kill();
        }

        ArrayPool<byte>.Shared.Return(copyBuffer);
    }

    // Ensure the output stream is not larger than the maximum file size
    if (outputStream.Length > MaxFileSize)
        outputStream.SetLength(MaxFileSize);

    outputStream.Position = 0;

    await context.Channel.SendMessageAsync(new MessageProperties()
        .WithContent($"Finished recording. Stop reason: {recordingStopReason}")
        .AddAttachments(new AttachmentProperties("recording.ogg", outputStream)));
}).AddContexts(InteractionContextType.Guild);

Now, our bot can finally record voice! You can test it out by using the /record command. Please note that you need to use the /join command to have the bot join a voice channel before you can use the /record command.

See Also

Edit this page