ARCore is a platform for building augmented reality apps on Android. Augmented Images gives you the ability to create AR apps that are able to recognize pre-registered images and anchor virtual content on them.
This codelab guides you through modifying an existing ARCore sample app to incorporate Augmented Images that are moving or fixed in place.
In this codelab, you're going to build upon a pre-existing ARCore sample app. By the end of the codelab, your app will:
The codelab computer should have everything you need:
Now that you've got everything ready, let's start!
We'll start by downloading the ARCore Java SDK from GitHub arcore-android-sdk-1.18.1.zip. Unzip it to your preferred location. The extract folder will be called arcore-android-sdk-1.18.1.
Launch Android Studio, and click Open an existing Android Studio project.
Navigate to this unzipped folder:
arcore-android-sdk-1.18.1/samples/augmented_image_java
Click Open.
Wait for Android studio to finish syncing the project. If your Android Studio doesn't have the required components, it may fail with the message "Install missing platform and sync project". Follow the instructions to fix the problem.
Now that you have a working ARCore app project, let's give it a test run.
Connect your ARCore device to the development machine, and use menu Run > Run ‘app' to run the debug version on the device. In the dialog prompting you to choose which device to run from,
choose the connected device, and click OK.
This sample project uses targetSdkVersion 28
. If you have a build error such as Failed to find Build Tools revision 28.0.3
, follow the instructions described in Android Studio to download and install the required Android Build Tools version.
If everything is successful, the sample app launches on the device and prompts you for permission to allow Augmented Image to take pictures and videos. Tap ALLOW to grant permission.
Let's give our sample app an image to look at.
Back in Android Studio, in the Project window, navigate to app > assets, and double-click the file default.jpg
to open it.
Point your device camera at the image of the Earth on screen, and follow the instructions to fit the image you're scanning into the crosshairs.
An image frame will overlay on top of the image, like this:
Next, we'll make small improvements to the sample app.
As we mentioned at the beginning of this codelab, we're going to have a little maze game on the image. First, let's find a maze model on poly.google.com, which contains many 3D models under the CC-BY license for free use.
For this codelab we're going to use "Circle Maze - Green," by Evol, and licensed under CC-BY 3.0.
Follow these steps to download the model and get it into Android Studio:
This downloads a file called green-maze.zip
.
green-maze.zip
and copy the content to this location:arcore-android-sdk-1.18.1/samples/augmented_image_java/app/src/main/assets/models/green-maze
There should be two files in this folder: GreenMaze.obj
and GreenMaze.mtl
.
Next, we'll load this OBJ file and display it over the detected image.
Now that we have the 3D model, GreenMaze.obj
, let's display it on top of our image.
AugmentedImageRenderer.java
, add a member variable called mazeRenderer
to render the maze model. Because the maze should attach to the image, it makes sense to put the mazeRenderer
inside the AugmentedImageRenderer
class.createOnGlThread
function, load the GreenMaze.ob
j. For simplicity, we'll use the same frame texture as its texture.draw
function, adjust the size of maze to the size of the detected image, and draw it.In AugmentedImageRenderer.java
, make these changes.
// Add a member variable to hold the maze model.
private final ObjectRenderer mazeRenderer = new ObjectRenderer();
// Replace the definition of the createOnGlThread function with the
// following code, which loads GreenMaze.obj.
public void createOnGlThread(Context context) throws IOException {
mazeRenderer.createOnGlThread(
context, "models/green-maze/GreenMaze.obj", "models/frame_base.png");
mazeRenderer.setMaterialProperties(0.0f, 3.5f, 1.0f, 6.0f);
}
// Replace the definition of the draw function with the
// following code
public void draw(
float[] viewMatrix,
float[] projectionMatrix,
AugmentedImage augmentedImage,
Anchor centerAnchor,
float[] colorCorrectionRgba) {
float[] tintColor =
convertHexToColor(TINT_COLORS_HEX[augmentedImage.getIndex() % TINT_COLORS_HEX.length]);
final float maze_edge_size = 492.65f; // Magic number of maze size
final float max_image_edge = Math.max(augmentedImage.getExtentX(), augmentedImage.getExtentZ()); // Get largest detected image edge size
Pose anchorPose = centerAnchor.getPose();
float mazsScaleFactor = max_image_edge / maze_edge_size; // scale to set Maze to image size
float[] modelMatrix = new float[16];
// OpenGL Matrix operation is in the order: Scale, rotation and Translation
// So the manual adjustment is after scale
// The 251.3f and 129.0f is magic number from the maze obj file
// We need to do this adjustment because the maze obj file
// is not centered around origin. Normally when you
// work with your own model, you don't have this problem.
Pose mozeModelLocalOffset = Pose.makeTranslation(
-251.3f * mazsScaleFactor,
0.0f,
129.0f * mazsScaleFactor);
anchorPose.compose(mozeModelLocalOffset).toMatrix(modelMatrix, 0);
mazeRenderer.updateModelMatrix(modelMatrix, mazsScaleFactor, mazsScaleFactor/10.0f, mazsScaleFactor);
mazeRenderer.draw(viewMatrix, projectionMatrix, colorCorrectionRgba, tintColor);
}
Ok, it seems we just had enough code change to display the maze on top of our default.jpg
picture of the Earth.
There are some magic numbers that I used in the above code. Don't be scared, they are there simply because we don't have full control over this 3D model. So I manually parsed the obj file, to understand the center (x, y, z) position of the model and its size. We are not going to do that in this codelab, I'll simply give out the value here. The dimension of the maze model is 492.65 x 120 x 492.65, with the center at (251.3, 60, -129.0). The X, Y, Z coordinates value range of its vertices are [5.02, 497.67], [0, 120], [-375.17, 117.25] respectively. So we need to set the scale of maze mode by image_size / 492.65
. As you may already noticed, the maze 3D model is not centered around origin (0, 0, 0), that's why we need to introduce an offset mozeModelLocalOffset
manually.
Also, because the maze wall is still a bit too high for our codelab, let's scale it additional 0.1 times. This lowers the wall so that the gaps are more visible. To do so, we need to introduce a helper function that allows us scale X, Y, Z coordinates unevenly.
In augmentedimage/rendering/ObjectRenderer.java
, make these changes.
public void updateModelMatrix(float[] modelMatrix, float scaleFactorX, float scaleFactorY, float scaleFactorZ) {
float[] scaleMatrix = new float[16];
Matrix.setIdentityM(scaleMatrix, 0);
scaleMatrix[0] = scaleFactorX;
scaleMatrix[5] = scaleFactorY;
scaleMatrix[10] = scaleFactorZ;
Matrix.multiplyMM(this.modelMatrix, 0, modelMatrix, 0, scaleMatrix, 0);
}
Ok, let's try running on your ARCore-supported device. Now the maze size should be the same as the image size.
Now let's add an object that moves around inside the maze. In this codelab, we'll keep it simple and just use the Android figurine andy.obj file that's included in ARCore Android SDK. And use the image frame texture as its texture, because it looks different from the green maze that we render on top of the image.
Add this code in AugmentedImageNode.java
.
// Add a private member to render andy
private final ObjectRenderer andyRenderer = new ObjectRenderer();
public void createOnGlThread(Context context) throws IOException {
// Add initialization for andyRenderer at the end of the createOnGlThread function.
andyRenderer.createOnGlThread(
context, "models/andy.obj", "models/andy.png");
andyRenderer.setMaterialProperties(0.0f, 3.5f, 1.0f, 6.0f);
}
public void draw(
float[] viewMatrix,
float[] projectionMatrix,
AugmentedImage augmentedImage,
Anchor centerAnchor,
float[] colorCorrectionRgba) {
// In draw() function, at the end add code to display the Andy, standing on top of the maze
Pose andyModelLocalOffset = Pose.makeTranslation(
0.0f,
0.1f,
0.0f);
anchorPose.compose(andyModelLocalOffset).toMatrix(modelMatrix, 0);
andyRenderer.updateModelMatrix(modelMatrix, 0.05f); // 0.05f is a Magic number to scale
andyRenderer.draw(viewMatrix, projectionMatrix, colorCorrectionRgba, tintColor);
}
Then let's try running it on device. We should see something like this.
To recognize an image, ARCore relies on visual features in the image. Not all images have the same quality and can be recognized easily.
The arcoreimg tool in the ARCore Android SDK lets you verify the quality of a target image. We can run this command line tool to determine how recognizable an image will be to ARCore. This tool outputs a number between 0 to 100, with 100 being the easiest to recognize. Here's an example:
arcore-android-sdk-1.18.1/tools/arcoreimg/macos$ $ ./arcoreimg eval-img --input_image_path=/Users/username/maze.jpg 100
The last section is not really relevant to ARCore, but it's an additional part that makes this example app fun. It is totally fine if you skip this part.
We'll use an open source Physics engine, jBullet, to handle physics simulation.
Herew's what we're going to do:
Download the PhysicsController.java code and add it to your project in this directory arcore-android-sdk-1.18.1/samples/augmented_image_java/app/src/main/java/com/google/ar/core/examples/java/augmentedimage/
Then make those changes in existing java code. As below,
In Android Studio, copy GreenMaze.obj
from
app > assets > models > green-maze
to:
app > assets
In app/build.gradle
, add this code.
// Add these dependencies.
implementation 'cz.advel.jbullet:jbullet:20101010-1'
// Obj - a simple Wavefront OBJ file loader
// https://github.com/javagl/Obj
implementation 'de.javagl:obj:0.2.1'
In AugmentedImageRenderer.java
, add this code.
// Add this line at the top with the rest of the imports.
private Pose andyPose = Pose.IDENTITY;
public void draw(
float[] viewMatrix,
float[] projectionMatrix,
AugmentedImage augmentedImage,
Anchor centerAnchor,
float[] colorCorrectionRgba) {
// Use these code to replace previous code for rendering the Andy object
// Adjust andy's rendering position
// Andy's pose is at Maze's vertex's coordinate
Pose andyPoseInImageSpace = Pose.makeTranslation(
andyPose.tx() * mazsScaleFactor,
andyPose.ty() * mazsScaleFactor,
andyPose.tz() * mazsScaleFactor);
anchorPose.compose(andyPoseInImageSpace).toMatrix(modelMatrix, 0);
andyRenderer.updateModelMatrix(modelMatrix, 0.05f);
andyRenderer.draw(viewMatrix, projectionMatrix, colorCorrectionRgba, tintColor);
}
// Add a new utility function to receive Andy pose updates
public void updateAndyPose(Pose pose) {
andyPose = pose;
}
In AugmentedImageActivity.java
, add this code.
import com.google.ar.core.Pose;
// Declare the PhysicsController class.
private PhysicsController physicsController;
// Update the case clause for TRACKING as below
private void drawAugmentedImages(
...
case TRACKING:
// Have to switch to UI Thread to update View.
this.runOnUiThread(
new Runnable() {
@Override
public void run() {
fitToScanView.setVisibility(View.GONE);
}
});
// Create a new anchor for newly found images.
if (!augmentedImageMap.containsKey(augmentedImage.getIndex())) {
Anchor centerPoseAnchor = augmentedImage.createAnchor(augmentedImage.getCenterPose());
augmentedImageMap.put(
augmentedImage.getIndex(), Pair.create(augmentedImage, centerPoseAnchor));
physicsController = new PhysicsController(this);
} else {
Pose ballPose = physicsController.getBallPose();
augmentedImageRenderer.updateAndyPose(ballPose);
// Use real world gravity, (0, -10, 0) as gravity
// Convert to Physics world coordinate (because Maze mesh has to be static)
// Use it as a force to move the ball
Pose worldGravityPose = Pose.makeTranslation(0, -10f, 0);
Pose mazeGravityPose = augmentedImage.getCenterPose().inverse().compose(worldGravityPose);
float mazeGravity[] = mazeGravityPose.getTranslation();
physicsController.applyGravityToBall(mazeGravity);
physicsController.updatePhysics();
}
break;
Then we can get it moving like this.
Have fun!
Congratulations, you have reached the end of this codelab. Let's look back at what we have achieved in this codelab.
If you would like to refer to the complete code, you can download it here.