Thanks for choosing this awesome audio codelab. You're going to be making a game that makes sound, so please head to the codelab reception to pick up some headphones.

Once you have your headphones, please connect them to the Pixel phone which runs the game.

Also, please note that this is a C++ codelab. You should have some basic C++ experience and know how to use header and implementation files. You should also be aware that you'll be using the Android NDK APIs rather than the Java APIs.

In this codelab you'll build a simple musical game using the Oboe library, a C++ library which uses the high-performance audio APIs in the Android NDK. The objective of the game is to copy the clapping pattern you hear by tapping on the screen.

What you'll learn

What you'll need

The game plays a funky four-beat backing track that continually loops. When the game starts, it also plays a clapping sound on the first three beats of the bar.

The user must try to repeat the three claps with the same timing by tapping on the screen when the second bar begins.

Each time the user taps, the game plays a clap sound. If the tap happens at the right time, the screen flashes green. If the tap is too early or too late, the screen flashes orange or purple respectively.

Clone project

Clone the Oboe repository on github and switch to the game-codelab branch.

git clone https://github.com/google/oboe 
cd oboe
git checkout game-codelab

Open the project in Android Studio

Load Android Studio and open the codelab project:

Note that this project contains all the code samples for the Oboe library. For this codelab, you are only working with the RhythmGame sample.

Run the project

Choose the RhythmGame run configuration.

Then press CTRL+R to build and run the template app - it should compile and run but it doesn't do anything except turn the screen yellow. You will add functionality to the game during this codelab.

Open the RhythmGame module

The files you'll work on for this codelab are stored in the RhythmGame module. Expand this module in the Project window, making sure that the Android view is selected.

Now expand the cpp/native-lib folder. During this codelab you will edit Game.h and Game.cpp.

Comparing with the final version

During the codelab it can be useful to refer to the final version of the code which is stored in the game-codelab-final branch. Android Studio makes it easy to compare changes across branches.

First, enable version control integration.

Now you can compare your code with code in the game-codelab-final branch.

This opens a new window. Choose the "Files" tab. A list of files with differences appears.

Click on any file to view the differences.

Here's the game architecture:

UI

The left side of the diagram shows objects associated with the UI.

The OpenGL Surface calls tick each time the screen needs to be updated, typically 60 times a second. Game then instructs any UI rendering objects to render pixels to the OpenGL surface and the screen is updated.

The UI for the game is very simple: the single method SetGLScreenColor updates the color of the screen. The following colours are used to show what's happening in the game:

Tap events

Each time the user taps on the screen the tap method is called, passing the time the event occurred.

Audio

The right side of the diagram shows objects associated with audio. Oboe provides the AudioStream class and associated objects to allow Game to send audio data to the audio output (a speaker or headphones).

Each time the AudioStream needs more data it calls AudioDataCallback::onAudioReady. This passes an array named audioData to Game which must then fill the array with numFrames of audio frames.

Let's start by making a sound! We're going to load an MP3 file into memory and play it whenever the user taps on the screen.

Load the sound file

The project includes a file in the assets folder named CLAP.mp3 which contains MP3 audio data. We're going to decode that MP3 file and store it as audio data in memory.

Open Game.h and declare a std::unique_ptr<Player> called mClap. Also declare a method called bool setupAudioSources. 

private:
    // ...existing code... 
    std::unique_ptr<Player> mClap;
   bool setupAudioSources();

Open Game.cpp and add the following code:

bool Game::setupAudioSources() {

   // Create a data source and player for the clap sound
   std::shared_ptr<AAssetDataSource> mClapSource {
           AAssetDataSource::newFromCompressedAsset(mAssetManager, "CLAP.mp3")
   };
   if (mClapSource == nullptr){
       LOGE("Could not load source data for clap sound");
       return false;
   }
   mClap = std::make_unique<Player>(mClapSource);
   return true;
}

This decodes CLAP.mp3 into PCM data and stores it in the Player object.

Build an AudioStream

An AudioStream allows us to communicate with an audio device like speakers or headphones. To create one we use an AudioStreamBuilder. This allows us to specify the properties which we would like our stream to have once we open it.

Create a new private method called bool openStream in the Game class with the following code. Remember to put the method declaration in the Game.h header file as well.

bool Game::openStream() {
   AudioStreamBuilder builder;
   builder.setFormat(AudioFormat::Float);
   builder.setPerformanceMode(PerformanceMode::LowLatency);
   builder.setSharingMode(SharingMode::Exclusive);
   builder.setSampleRate(48000);
   builder.setSampleRateConversionQuality(
      SampleRateConversionQuality::Medium);
   builder.setChannelCount(2);
}

There's quite a bit going on here so let's break it down.

First, we create the stream builder and request the following properties:

Open the stream

Now that the stream has been set up using the builder we can go ahead and open it. There's two methods we can use to do this. Each method works by accepting an AudioStream object as its parameter, the builder then takes care of construction. The methods are:

We will use openManagedStream because it's less work for us and follows the RAII idiom.

Declare a member variable of type ManagedStream inside Game.h.

private:
    ...
    ManagedStream mAudioStream { nullptr };
}

Now add the following to the end of openStream back in Game.cpp. 

bool Game::openStream() {

    [...]
    Result result = builder.openManagedStream(mAudioStream);
    if (result != Result::OK){
        LOGE("Failed to open stream. Error: %s", convertToText(result));
        return false;
    }
    return true;
}

This code attempts to open the stream and returns false if there was an error.

Setting up the callback

So far so good, we have methods for opening an audio stream and loading our MP3 file into memory. Now we need to get the audio data from memory into the audio stream.

To do this we'll use an AudioDataCallback because this approach provides the best performance. Let's update our Game class to implement the AudioDataCallback interface.

Open Game.h and locate the following line: 

class Game {

Change it to:

class Game : public AudioStreamCallback {

Now override the AudioStreamCallback::onAudioReady method:

public:
    // ...existing code... 
     
    // Inherited from oboe::AudioStreamCallback
    DataCallbackResult
    onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) override;

The audioData parameter for onAudioReady is an array into which you render the audio data using mClap->renderAudio.

Add the implementation of onAudioReady to Game.cpp. 

// ...existing code... 

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {
    mClap->renderAudio(static_cast<float *>(audioData), numFrames);
    return DataCallbackResult::Continue;
}

The return value DataCallbackResult::Continue tells the stream that you intend to keep sending audio data so callbacks should continue. If we return DataCallbackResult::Stop the callbacks would stop and no more audio would be played through the stream.

To complete the callback setup we must tell the audio stream builder where to find the callback object using setCallback in openStream. Do this before the stream is opened.

bool Game::openStream() {
    ...
    builder.setCallback(this);
    Result result = builder.openManagedStream(mAudioStream);

Loading...

Before the game can be played there's a couple of things that must happen:

These operations are blocking, and depending on the size of the MP3 files and the speed of the decoder they might take several seconds to complete. We should avoid performing these operations on the main thread otherwise we might get a dreaded ANR.

Another thing which must happen before the game can be played is starting the audio stream. It makes sense to do this after the other loading operations have completed.

Add the following code to the existing load method. We will call this method on a separate thread.

void Game::load() {

   if (!openStream()) {
       mGameState = GameState::FailedToLoad;
       return;
   }

   if (!setupAudioSources()) {
       mGameState = GameState::FailedToLoad;
       return;
   }

   Result result = mAudioStream->requestStart();
   if (result != Result::OK){
       LOGE("Failed to start stream. Error: %s", convertToText(result));
       mGameState = GameState::FailedToLoad;
       return;
   }

   mGameState = GameState::Playing;
}

Here's what's going on. We're using a member variable mGameState to keep track of the game state. This is initially Loading, changing to FailedToLoad or Playing. We'll update the tick method shortly to check mGameState and update the screen background color accordingly.

We call openStream to open our audio stream, then setupAudioSources to load the MP3 file from memory.

Lastly, we start our audio stream by calling requestStart. This is a non-blocking method which will start the audio callbacks as soon as possible.

Starting asynchronously

All we need to do now is call our load method asynchronously. For this we can use the C++ async function which will call a function on a separate thread asynchronously. Let's update our Game's start method:

void Game::start() {
   mLoadingResult = std::async(&Game::load, this);
}

This simply calls our load method asynchronously and stores the result in mLoadingResult.

Update the background color

This step is simple. Depending on the game state we'll update the background color. Update the tick method. 

void Game::tick(){
   switch (mGameState){
       case GameState::Playing:
           SetGLScreenColor(kPlayingColor);
           break;
       case GameState::Loading:
           SetGLScreenColor(kLoadingColor);
           break;
       case GameState::FailedToLoad:
           SetGLScreenColor(kLoadingFailedColor);
           break;
   }
}

Handle the tap event

We're almost there, just one more thing to do. The tap method is called each time the user taps the screen. Start the clap sound by calling setPlaying. Add the following to tap:

void Game::tap(int64_t eventTimeAsUptime) {
    mClap->setPlaying(true);
}

Build and run the app. You should hear a clap sound when you tap the screen. Give yourself a round of applause!

Playing a single clap sound is going to get boring pretty quickly. It would be nice to also play a backing track with a beat you can clap along to.

Up until now, the game places only clap sounds into the audio stream.

Using a mixer

To play multiple sounds simultaneously we must mix them together. Conventienly a Mixer object has been provided which does this as part of this codelab.

Create the backing track and mixer

Open Game.h and declare another std::unique_ptr<Player> for the backing track and a Mixer: 

private:
    ..
    std::unique_ptr<Player> mBackingTrack; 
    Mixer mMixer;

Now in Game.cpp add the following code after the clap sound has been loaded in setupAudioSources.

bool Game::setupAudioSources() {
   ...
   // Create a data source and player for our backing track
   std::shared_ptr<AAssetDataSource> backingTrackSource {
           AAssetDataSource::newFromCompressedAsset(mAssetManager, "FUNKY_HOUSE.mp3")
   };
   if (backingTrackSource == nullptr){
       LOGE("Could not load source data for backing track");
       return false;
   }
   mBackingTrack = std::make_unique<Player>(backingTrackSource);
   mBackingTrack->setPlaying(true);
   mBackingTrack->setLooping(true);

   // Add both players to a mixer
   mMixer.addTrack(mClap.get());
   mMixer.addTrack(mBackingTrack.get());
   mMixer.setChannelCount(mAudioStream->getChannelCount());
   return true;
}

This loads the contents of the FUNKY_HOUSE.mp3 asset (which contains MP3 data in the same format as the clap sound asset) into a Player object. Playback starts when the game starts and loops indefinitely.

Both the clap sound and backing track are added to the mixer, and the mixer's channel count is set to match that of our audio stream.

Updating the audio callback

You now need to tell the audio callback to use the mixer rather than the clap sound for rendering. Update onAudioReady to the following: 

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {

    mMixer.renderAudio(static_cast<float*>(audioData), numFrames);
    return DataCallbackResult::Continue;
}

Build and run the game. You should hear the backing track and the clap sound when you tap the screen. Feel free to jam for a minute!

Now things start to get interesting. We're going to start adding the gameplay mechanics. The game plays a series of claps at specific times. This is called the clap pattern.

For this simple game, the clap pattern is just three claps that start on the first beat of the first bar of the backing track. The user must repeat the same pattern starting on the first beat of the second bar.

When should the game play the claps?

The backing track has a tempo of 120 beats per minute, or 1 beat every 0.5 seconds. So the game must play a clap sound at the following times in the backing track:

Beat

Time (milliseconds)

1

0

2

500

3

1000

Synchronizing clap events with the backing track

Each time onAudioReady is called audio frames from the backing track (via the mixer) are rendered into the audio stream - this is what the user actually hears. By counting the number of frames which have been written we know the exact playback time, and therefore when to play a clap.

With this in mind, here's how we can play the clap events at exactly the right time:

Key point: By counting the number of frames which are written inside onAudioReady you know the exact playback position and can ensure perfect synchronization with the backing track.

Cross thread communication

The game has three threads: an OpenGL thread, a UI thread (main thread) and a real-time audio thread.

Clap events are pushed onto the scheduling queue from the UI thread and popped off the queue from the audio thread.

The queue is accessed from multiple threads so it must be thread-safe. It must also be lock-free so it does not block the audio thread. This requirement is true for any object shared with the audio thread. Why? Because blocking the audio thread can cause audio glitches, and no-one wants to hear that!

Add the code

The game already includes a LockFreeQueue class template which is thread-safe when used with a single reader thread (in this case the audio thread) and a single writer thread (the UI thread).

To declare a LockFreeQueue you must supply two template parameters:

Open Game.h and add the following declarations: 

private:
    // ...existing code...  
    Mixer mMixer;
    
    LockFreeQueue<int64_t, 4> mClapEvents;
    std::atomic<int64_t> mCurrentFrame { 0 };
    std::atomic<int64_t> mSongPositionMs { 0 };

Note that mCurrentFrame and mSongPositionMs are std::atomic because they are accessed from the UI thread.

Now in Game.cpp create a new method called scheduleSongEvents. Inside this method enqueue the clap events.

void Game::scheduleSongEvents() {

   // schedule the claps
   mClapEvents.push(0);
   mClapEvents.push(500);
   mClapEvents.push(1000);
}

We need to call scheduleSongEvents from our load method before the stream is started so that all events are enqueued before the backing track starts playing. 

void Game::load() {
   ...
   scheduleSongEvents();
   Result result = mAudioStream->requestStart();
   ...
}

Now update onAudioReady to the following:

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {

   float *outputBuffer = static_cast<float *>(audioData);
   int64_t nextClapEventMs;

   for (int i = 0; i < numFrames; ++i) {

       mSongPositionMs = convertFramesToMillis(
               mCurrentFrame,
               mAudioStream->getSampleRate());

       if (mClapEvents.peek(nextClapEventMs) && mSongPositionMs >= nextClapEventMs){
           mClap->setPlaying(true);
           mClapEvents.pop(nextClapEventMs);
       }mMixer.renderAudio(outputBuffer+(oboeStream->getChannelCount()*i), 1);
       mCurrentFrame++;
   }

   return DataCallbackResult::Continue;
}

The for loop iterates for numFrames. On each iteration it does the following:

Build and run the game. Three claps should be played exactly on the beat when the game starts. Tapping on the screen still plays the clap sounds.

Feel free to experiment with different clap patterns by changing the frame values for the clap events. You can also add more clap events, remember to increase the capacity of the mClapEvents queue.

The game plays a clap pattern and expects the user to imitate the pattern. Finally, complete the game by scoring the user's taps.

Did the user tap at the right time?

After the game claps three times in the first bar, the user should tap three times starting on the first beat of the second bar.

You shouldn't expect the user to tap at exactly the right time - that would be virtually impossible! Instead, we'll allow some tolerance before and after the expected time. This defines a time range which we'll call the tap window.

If the user taps during the tap window, make the screen flash green, too early: orange and too late: purple.

Storing the tap windows is easy: store the song position at the center of each window in a queue, the same way we did for clap events. We can then pop each window off the queue when the user taps on the screen.

The song positions at the center of the tap window are as follows:

Beat

Time (milliseconds)

5

2000

6

2500

7

3000

Declare a new member variable to store these clap windows. 

private: 
    LockFreeQueue<int64_t, kMaxQueueItems> mClapWindows;

Now add the song positions for the clap windows.

void Game::scheduleSongEvents() {

    ...
    // schedule the clap windows
    mClapWindows.push(2000);
    mClapWindows.push(2500);
    mClapWindows.push(3000);
}

Comparing tap events with the tap window

When the user taps on the screen we need to know whether the tap fell within the current tap window. The tap event is delivered as system uptime (milliseconds since boot), so we need to convert this to a position within the song.

Luckily this is simple. Store the uptime at the current song position each time onAudioReady is called. Declare a member variable in the header to store the song position: 

private: 
    int64_t mLastUpdateTime { 0 };

Now add the following code to the end of onAudioReady. 

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {
   ...
   mLastUpdateTime = nowUptimeMillis();

   return DataCallbackResult::Continue;
}

Now update the tap method to the following. 

void Game::tap(int64_t eventTimeAsUptime) {

   if (mGameState != GameState::Playing){
       LOGW("Game not in playing state, ignoring tap event");
   } else {
       mClap->setPlaying(true);

       int64_t nextClapWindowTimeMs;
       if (mClapWindows.pop(nextClapWindowTimeMs)){

           // Convert the tap time to a song position
           int64_t tapTimeInSongMs = mSongPositionMs + (eventTimeAsUptime - mLastUpdateTime);
           TapResult result = getTapResult(tapTimeInSongMs, nextClapWindowTimeMs);
           mUiEvents.push(result);
       }
   }
}

We use the provided getTapResult method to determine the result of the user's tap. We then push the result onto a queue of UI events. This is explained in the next section.

Updating the screen

Once we know the accuracy of a user's tap (early, late, right-on), we need to update the screen to provide visual feedback.

To do this we'll use another instance of LockFreeQueue class with TapResult objects to create a queue of UI events. Declare a new member variable to store these. 

private: 
    LockFreeQueue<TapResult, kMaxQueueItems> mUiEvents;

Then in the tick method we'll pop any pending UI events and update the screen color accordingly. Update the code for the Playing state in tick. 

case GameState::Playing:
   TapResult r;
   if (mUiEvents.pop(r)) {
       renderEvent(r);
   } else {
       SetGLScreenColor(kPlayingColor);
   }
   break;

That's it! Build and run the game.

You should hear three claps when the game starts. If you tap exactly on the beat three times in the second bar you should see the screen flash green on each tap. If you're early it'll flash orange, if you're late it'll flash purple. Good luck!

Additional resources

OLD CONTENT - DELETE BEFORE PUBLICATION

Did we get what we asked for?

Just because you ask for something doesn't mean you're going to get it!

The stream builder will do its best to give you a stream which matches your requested properties, but sometimes it can't give you everything you asked for. Perhaps the current audio device doesn't allow low latency streams, or maybe it doesn't support exclusive sharing mode.

In our case, it is essential that the stream's sample format is float because that's the format we'll get from our audio assets. Let's add a check for that. Add the following at the end of openStream.

bool Game::openStream() {

    [...]
    if (mAudioStream->getFormat() != AudioFormat::Float){
        LOGE("The codelab version of this app only supports floating point output."
             "Please see the master branch for a version which includes sample format conversion");
    }

    return true;
}

If the check passes we can return true to indicate that the stream opened successfully.