What you'll learn

What you'll build

In this codelab, you start with a sample app that already displays a list of GitHub repositories. Whenever the user scrolls to the end of the displayed list, a new network request is triggered and its result is displayed on the screen.

You will add code through a series of steps, to achieve the following:

Here's what your app will look like in the end:

What you'll need

In this step, you will download the code for the entire codelab and then run a simple example app.

To get you started as quickly as possible, we have prepared a starter project for you to build on.

If you have git installed, you can simply run the command below. (You can check by typing git --version in the terminal / command line and verify it executes correctly.)

 git clone https://github.com/googlecodelabs/android-paging

The initial state is on the master branch. For certain steps you can see the solution as follows:

If you do not have git, you can click the following button to download all the code for this codelab:

Download source code

  1. Unzip the code, and then open the project Android Studio version 3.6.1 or newer.
  2. Run the app run configuration on a device or emulator.

The app runs and displays a list of GitHub repositories similar to this one:

The app allows you to search GitHub for repositories whose name or description contains a specific word. The list of repositories is displayed in descending order based on the number of stars, then alphabetically by name.

The app follows the architecture recommended in the "Guide to app architecture". Here's what you will find in each package:

The GithubRepository class retrieves the list of repository names from the network every time the user scrolls towards the end of the list, or when the user searches for a new repository. The list of results for a query is kept in memory in the GithubRepository in a ConflatedBroadcastChannel and exposed as a Flow.

SearchRepositoriesViewModel requests the data from GithubRepository and exposes it to the SearchRepositoriesActivity. Because we want to ensure that we're not requesting the data multiple times on configuration change (e.g. rotation), we're converting the Flow to LiveData in the ViewModel using the liveData() builder method. That way, the LiveData caches the latest list of results in memory, and when the SearchRepositoriesActivity gets recreated, the content of the LiveData will be displayed on the screen.

From a usability perspective, we have the following issues:

From an implementation perspective, we have the following issues:

Let's find out how the Paging library can help us with these issues and what components it includes.

The Paging library makes it easier for you to load data incrementally and gracefully within your app's UI. The Paging API provides support for many of the functionalities that you would otherwise need to implement manually when you need to load data in pages:

The Guide to app architecture proposes an architecture with the following main components:

The Paging library works with all of these components and coordinates the interactions between them, so that it can load "pages" of content from a data source and display that content in the UI.

This codelab introduces you to the Paging library and its main components:

In this codelab, you will implement examples of each of the components described above.

The PagingSource implementation defines the source of data and how to retrieve data from that source. The PagingData object queries data from the PagingSource in response to loading hints that are generated as the user scrolls in a RecyclerView.

Currently, the GithubRepository has a lot of the responsibilities of a data source that the Paging library will handle once we're done adding it:

To build the PagingSource you need to define the following:

So, in the data package, let's create a PagingSource implementation called GithubPagingSource: 

class GithubPagingSource(
        private val service: GithubService,
        private val query: String
) : PagingSource<Int, Repo>() { 
    override suspend fun load(params: LoadParams<Int>): LoadResult<Int, Repo> {
        TODO("Not yet implemented")
    }
}

We'll see that PagingSource requires us to implement a load() function—this will be called to trigger the async load. The LoadParams object keeps information related to the load operation, including the following:

The load function returns a LoadResult. This will replace the usage of RepoSearchResult in our app, as LoadResult can take one of the following types:

When constructing the LoadResult.Page, pass null for nextKey or prevKey if the list can't be loaded in the corresponding direction.. For example, in our case, we could consider that if the network response was successful but the list was empty, we don't have any data left to be loaded; so the nextKey can be null.

Based on all of this information, we should be able to implement the load() function!

The GithubPagingSource implementation looks like this:

// GitHub page API is 1 based: https://developer.github.com/v3/#pagination
private const val GITHUB_STARTING_PAGE_INDEX = 1

class GithubPagingSource(
        private val service: GithubService,
        private val query: String
) : PagingSource<Int, Repo>() {

    override suspend fun load(params: LoadParams<Int>): LoadResult<Int, Repo> {
        val position = params.key ?: GITHUB_STARTING_PAGE_INDEX
        val apiQuery = query + IN_QUALIFIER
        return try {
            val response = service.searchRepos(apiQuery, position, params.loadSize)
            val repos = response.items 
            LoadResult.Page(
                    data = repos,
                    prevKey = if (position == GITHUB_STARTING_PAGE_INDEX) null else position - 1,
                    nextKey = if (repos.isEmpty()) null else position + 1
            )
        } catch (exception: IOException) {
            return LoadResult.Error(exception)
        } catch (exception: HttpException) {
            return LoadResult.Error(exception)
        }
    }
}

In our current implementation, we use a Flow<RepoSearchResult> in the GitHubRepository to get the data from the network and pass it to the ViewModel. The ViewModel then transforms it into a LiveData and exposes it to the UI. Whenever we get to the end of the displayed list and more data is loaded from the network, the Flow<RepoSearchResult> will contain the entire list of previously retrieved data for that query in addition to the latest data.

RepoSearchResult encapsulates both the success and error cases. The success case holds the repository data. The error case contains the Exception reason. With Paging 3.0 we don't need the RepoSearchResult anymore, as the library models both the success and error cases with LoadResult. Feel free to delete RepoSearchResult as in the next few steps we'll replace it.

To construct the PagingData, we first need to decide what API we want to use to pass the PagingData to other layers of our app:

As we're already using Flow in our app, we'll continue with this approach; but instead of using Flow<RepoSearchResult>, we'll use Flow<PagingData<Repo>>.

No matter which PagingData builder you use, you'll have to pass the following parameters:

Let's modify our GithubRepository!

Update GithubRepository.getSearchResultStream:

fun getSearchResultStream(query: String): Flow<PagingData<Repo>> {
    return Pager(
          config = PagingConfig(
            pageSize = NETWORK_PAGE_SIZE,
            enablePlaceholders = false
         ),
          pagingSourceFactory = { GithubPagingSource(service, query) }
    ).flow
}

Cleanup GithubRepository

Paging 3.0 does a lot of things for us:

This means that everything else in our GithubRepository can be removed, except getSearchResultStream and the companion object where we defined the NETWORK_PAGE_SIZE. Your GithubRepository should now look like this:

class GithubRepository(private val service: GithubService) {

    fun getSearchResultStream(query: String): Flow<PagingData<Repo>> {
        return Pager(
                config = PagingConfig(
                pageSize = NETWORK_PAGE_SIZE,
                enablePlaceholders = false
             ),
                pagingSourceFactory = { GithubPagingSource(service, query) }
        ).flow
    }

    companion object {
        private const val NETWORK_PAGE_SIZE = 50
    }
}

You should now have compile errors in the SearchRepositoriesViewModel. Let's see what changes need to be made there!

From our SearchRepositoriesViewModel we expose a repoResult: LiveData<RepoSearchResult>. The role of repoResult is to be an in-memory cache for result searches that survives configuration changes. With Paging 3.0 we don't need to convert our Flow to LiveData anymore. Instead, SearchRepositoriesViewModel will have a private Flow<PagingData<Repo>> member that serves the same purpose that repoResult did..

Instead of using a LiveData object for each new query, we can just use a String. This will help us ensure that whenever we get a new search query that is the same as the current query, we will just return the existing Flow. We only need to call repository.getSearchResultStream() if the new search query is different.

Flow<PagingData> has a handy cachedIn() method that allows us to cache the content of a Flow<PagingData> in a CoroutineScope. Since we're in a ViewModel, we will use the androidx.lifecycle.viewModelScope.

We'll rewrite most of SearchRepositoriesViewModel to leverage the built-in functionality from Paging 3.0. Your SearchRepositoriesViewModel will look like this: 

class SearchRepositoriesViewModel(private val repository: GithubRepository) : ViewModel() {

    private var currentQueryValue: String? = null

    private var currentSearchResult: Flow<PagingData<Repo>>? = null

    fun searchRepo(queryString: String): Flow<PagingData<Repo>> {
        val lastResult = currentSearchResult
        if (queryString == currentQueryValue && lastResult != null) {
            return lastResult
        }
        currentQueryValue = queryString
        val newResult: Flow<PagingData<Repo>> = repository.getSearchResultStream(queryString)
                .cachedIn(viewModelScope)
        currentSearchResult = newResult
        return newResult
    }
}

Now, let's see what changes we made to SearchRepositoriesViewModel:

To bind a PagingData to a RecyclerView, use a PagingDataAdapter. The PagingDataAdapter gets notified whenever the PagingData content is loaded and then it signals the RecyclerView to update.

Update the ui.ReposAdapter to work with a PagingData stream:

class ReposAdapter : PagingDataAdapter<Repo, RecyclerView.ViewHolder>(REPO_COMPARATOR) {
// body is unchanged

We've been making a lot of changes so far, but now we're just one step away from being able to run the app—we just need to connect the UI!

Let's update SearchRepositoriesActivity to work with Paging 3.0. To be able to work with Flow<PagingData>, we need to launch a new coroutine. We will do that in the lifecycleScope, which is responsible for canceling the request when the activity is recreated.

We also want to ensure that whenever the user searches for a new query, the previous query is cancelled. To do this, our SearchRepositoriesActivity can hold a reference to a new Job that will be cancelled every time we search for a new query.

Let's create a new search function that gets a query as a parameter. The function should do the following:

private var searchJob: Job? = null

private fun search(query: String) {
   // Make sure we cancel the previous job before creating a new one
   searchJob?.cancel()
   searchJob = lifecycleScope.launch {
       viewModel.searchRepo(query).collectLatest {
           adapter.submitData(it)
       }
   }
}

The search function should be called in SearchRepositoriesActivity in the onCreate() method, In updateRepoListFromInput(), replace the viewModel and adapter calls with searchRepo:

private fun updateRepoListFromInput() {
    binding.searchRepo.text.trim().let {
        if (it.isNotEmpty()) {
            binding.list.scrollToPosition(0)
            search(it.toString())
        }
    }
}

Because we wanted to make sure that the scroll position is reset for each new search, we had: binding.list.scrollToPosition(0). But, instead of resetting the position on new search, we should reset the position when the list adapter is updated with the result of a new search. To achieve this, we can use the PagingDataAdapter.loadStateFlow API. This Flow emits us every time there's a change in the load state via a CombinedLoadStates object.

CombinedLoadStates allows us to get the load state for the 3 different types of load operations:

In our case, we want to reset the scroll position only when the refresh has completed i.e LoadState is refresh, NotLoading.

Let's collect from this flow when we initialize the search, in the initSearch method and at every new emission of the flow, let's scroll to position 0. 

private fun initSearch(query: String) {
    ...
    // First part of the method is unchanged

        // Scroll to top when the list is refreshed from network.
        lifecycleScope.launch {
            adapter.loadStateFlow
                   // Only emit when REFRESH LoadState changes.
                   .distinctUntilChangedBy { it.refresh }
                   // Only react to cases where REFRESH completes i.e., NotLoading.
                   .filter { it.refresh is LoadState.NotLoading }
                   .collect { binding.list.scrollToPosition(0) }
        }
}

We can now remove binding.list.scrollToPosition(0) from updateRepoListFromInput().

Currently, we use an OnScrollListener attached to the RecyclerView to know when to trigger more data. We can let the Paging library handle list scrolling for us. Remove the setupScrollListener() method and all references to it.

Let's also remove repoResult usage. This is what your activity should look like:

@ExperimentalCoroutinesApi
class SearchRepositoriesActivity : AppCompatActivity() {

    private lateinit var binding: ActivitySearchRepositoriesBinding
    private lateinit var viewModel: SearchRepositoriesViewModel
    private val adapter = ReposAdapter()

    private var searchJob: Job? = null

    private fun search(query: String) {
        // Make sure we cancel the previous job before creating a new one
        searchJob?.cancel()
        searchJob = lifecycleScope.launch {
            viewModel.searchRepo(query).collect {
                adapter.submitData(it)
            }
        }
    }

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivitySearchRepositoriesBinding.inflate(layoutInflater)
        val view = binding.root
        setContentView(view)

        // get the view model
        viewModel = ViewModelProvider(this, Injection.provideViewModelFactory())
                .get(SearchRepositoriesViewModel::class.java)

        // add dividers between RecyclerView's row items
        val decoration = DividerItemDecoration(this, DividerItemDecoration.VERTICAL)
        binding.list.addItemDecoration(decoration)

        initAdapter()
        val query = savedInstanceState?.getString(LAST_SEARCH_QUERY) ?: DEFAULT_QUERY
        search(query)
        initSearch(query)
    }

    override fun onSaveInstanceState(outState: Bundle) {
        super.onSaveInstanceState(outState)
        outState.putString(LAST_SEARCH_QUERY, binding.searchRepo.text.trim().toString())
    }

    private fun initAdapter() {
        binding.list.adapter = adapter
    }

    private fun initSearch(query: String) {
        binding.searchRepo.setText(query)

        binding.searchRepo.setOnEditorActionListener { _, actionId, _ ->
            if (actionId == EditorInfo.IME_ACTION_GO) {
                updateRepoListFromInput()
                true
            } else {
                false
            }
        }
        binding.searchRepo.setOnKeyListener { _, keyCode, event ->
            if (event.action == KeyEvent.ACTION_DOWN && keyCode == KeyEvent.KEYCODE_ENTER) {
                updateRepoListFromInput()
                true
            } else {
                false
            }
        }

        // Scroll to top when the list is refreshed from network.
        lifecycleScope.launch {
            adapter.loadStateFlow
                    // Only emit when REFRESH LoadState for RemoteMediator changes.
                    .distinctUntilChangedBy { it.refresh }
                    // Only react to cases where Remote REFRESH completes i.e., NotLoading.
                    .filter { it.refresh is LoadState.NotLoading }
                    .collect { binding.list.scrollToPosition(0) }
        }
    }

    private fun updateRepoListFromInput() {
        binding.searchRepo.text.trim().let {
            if (it.isNotEmpty()) {
                binding.list.scrollToPosition(0)
                search(it.toString())
            }
        }
    }

    private fun showEmptyList(show: Boolean) {
        if (show) {
            binding.emptyList.visibility = View.VISIBLE
            binding.list.visibility = View.GONE
        } else {
            binding.emptyList.visibility = View.GONE
            binding.list.visibility = View.VISIBLE
        }
    }

    companion object {
        private const val LAST_SEARCH_QUERY: String = "last_search_query"
        private const val DEFAULT_QUERY = "Android"
    }
}

Our app should compile and run, but without the loading state footer and the Toast that displays on error. In the next step, we will see how to display the loading state footer.

You can find the full code for the steps done so far in branch step5-9_paging_3.0.

In our app, we want to be able to display a footer based on the load status: when the list is loading, we want to show a progress spinner. In case of an error, we want to show the error and a retry button.

The header/footer that we need to build follows the idea of a list that needs to be appended either at the beginning (as a header) or at the end (as a footer) of the actual list of items we're displaying. The header/footer is a list with only one element: a view that displays a progress bar or an error with a retry button, based on the Paging LoadState.

As displaying a header/footer based on the loading state and implementing a retry mechanism are common tasks, the Paging 3.0 API helps us with both of these.

For header/footer implementation we'll use a LoadStateAdapter. This implementation of RecyclerView.Adapter is automatically notified of changes in load state. It makes sure that only Loading and Error states lead to items being displayed and notifies the RecyclerView when an item is removed, inserted, or changed, depending on the LoadState.

For the retry mechanism we use adapter.retry(). Under the hood, this method ends up calling your PagingSource implementation for the right page. The response will be automatically propagated via Flow<PagingData>.

Let's see what our header/footer implementation looks like!

Like with any list, we have 3 files to create:

Create the view layout

Create the repos_load_state_footer_view_item layout for our repo load state. It should have a ProgressBar, a TextView (to display the error), and a retry Button. The necessary strings and dimensions are already declared in the project.

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    xmlns:tools="http://schemas.android.com/tools"
    android:orientation="vertical"
    android:padding="8dp">
    <TextView
        android:id="@+id/error_msg"
        android:textColor="?android:textColorPrimary"
        android:textSize="@dimen/error_text_size"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_gravity="center"
        android:textAlignment="center"
        tools:text="Timeout"/>
    <ProgressBar
        android:id="@+id/progress_bar"
        style="?android:attr/progressBarStyle"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_gravity="center"/>
    <Button
        android:id="@+id/retry_button"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_gravity="center"
        android:text="@string/retry"/>
</LinearLayout>

Create the ViewHolder

Create a new ViewHolder called ReposLoadStateViewHolder in the ui folder. It should receive a retry function as a parameter, to be called when the retry button is pressed. Create a bind() function that receives the LoadState as a parameter and sets the visibility of each view depending on the LoadState. An implementation of ReposLoadStateViewHolder using ViewBinding looks like this:

class ReposLoadStateViewHolder(
        private val binding: ReposLoadStateFooterViewItemBinding,
        retry: () -> Unit
) : RecyclerView.ViewHolder(binding.root) {

    init {
        binding.retryButton.setOnClickListener { retry.invoke() }
    }

    fun bind(loadState: LoadState) {
        if (loadState is LoadState.Error) {
            binding.errorMsg.text = loadState.error.localizedMessage
        }
        binding.progressBar.isVisible = loadState is LoadState.Loading
        binding.retryButton.isVisible = loadState !is LoadState.Loading
        binding.errorMsg.isVisible = loadState !is LoadState.Loading
    }

    companion object {
        fun create(parent: ViewGroup, retry: () -> Unit): ReposLoadStateViewHolder {
            val view = LayoutInflater.from(parent.context)
                    .inflate(R.layout.repos_load_state_footer_view_item, parent, false)
            val binding = ReposLoadStateFooterViewItemBinding.bind(view)
            return ReposLoadStateViewHolder(binding, retry)
        }
    }
}

Create the LoadStateAdapter

Create a ReposLoadStateAdapter that extends LoadStateAdapter in the ui folder as well. The adapter should receive the retry function as a parameter, since the retry function will be passed to the ViewHolder when constructed.

As with any Adapter, we need to implement the onBind() and onCreate() methods. LoadStateAdapter makes it easier as it passes the LoadState in both of these functions. In onBindViewHolder(), bind your ViewHolder. In onCreateViewHolder(), define how to create the ReposLoadStateViewHolder based on the parent ViewGroup and the retry function:

class ReposLoadStateAdapter(private val retry: () -> Unit) : LoadStateAdapter<ReposLoadStateViewHolder>() {
    override fun onBindViewHolder(holder: ReposLoadStateViewHolder, loadState: LoadState) {
        holder.bind(loadState)
    }

    override fun onCreateViewHolder(parent: ViewGroup, loadState: LoadState): ReposLoadStateViewHolder {
        return ReposLoadStateViewHolder.create(parent, retry)
    }
}

Bind the footer adapter with the list

Now that we have all of the elements of our footer, let's bind them to our list. To do this, the PagingDataAdapter has 3 useful methods:

Update the SearchRepositoriesActivity.initAdapter() method and call withLoadStateHeaderAndFooter() on the adapter. As a retry function, we can call adapter.retry().

private fun initAdapter() {
    binding.list.adapter = adapter.withLoadStateHeaderAndFooter(
            header = ReposLoadStateAdapter { adapter.retry() },
            footer = ReposLoadStateAdapter { adapter.retry() }
    )
}

Since we have an infinite scrolling list, one easy way to get to see the footer is by putting your phone or emulator in airplane mode and scrolling until the end of the list.

Let's run the app!

You can find the full code for the steps done so far in branch step10_loading_state_footer.

You might have noticed that whenever you search for a new query, the current query result stays on screen until we get a network response. That's bad user experience! Instead we should display a progress bar or a retry button.

Let's update our activity_search_repositories.xml to include these UI elements, instead of the emptyList TextView:

<androidx.constraintlayout.widget.ConstraintLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".ui.SearchRepositoriesActivity">
    <com.google.android.material.textfield.TextInputLayout
        android:id="@+id/input_layout"
        android:layout_width="0dp"
        android:layout_height="wrap_content"
        android:layout_marginEnd="8dp"
        android:layout_marginStart="8dp"
        android:layout_marginTop="8dp"
        android:layout_marginLeft="8dp"
        android:layout_marginRight="8dp"
        style="@style/Widget.MaterialComponents.TextInputLayout.OutlinedBox"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toTopOf="parent">
        <EditText
            android:id="@+id/search_repo"
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            android:hint="@string/search_hint"
            android:imeOptions="actionSearch"
            android:inputType="textNoSuggestions"
            android:selectAllOnFocus="true"
            tools:text="Android"/>
    </com.google.android.material.textfield.TextInputLayout>

    <androidx.recyclerview.widget.RecyclerView
        android:id="@+id/list"
        android:layout_width="0dp"
        android:layout_height="0dp"
        android:paddingVertical="@dimen/row_item_margin_vertical"
        android:scrollbars="vertical"
        app:layoutManager="LinearLayoutManager"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toBottomOf="@+id/input_layout"
        tools:ignore="UnusedAttribute"/>

    <ProgressBar
        android:id="@+id/progress_bar"
        style="?android:attr/progressBarStyle"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_gravity="center"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        app:layout_constraintTop_toTopOf="parent"
        app:layout_constraintBottom_toBottomOf="parent"/>

    <Button
        android:id="@+id/retry_button"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="@string/retry"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        app:layout_constraintTop_toTopOf="parent"
        app:layout_constraintBottom_toBottomOf="parent"/>

</androidx.constraintlayout.widget.ConstraintLayout>

Our retry button should trigger a reload of the PagingData. To do this, we call adapter.retry() in the onClickListener implementation, like we did for the header/footer:

// SearchRepositoriesActivity.kt

override fun onCreate(savedInstanceState: Bundle?) {
    ...
    binding.retryButton.setOnClickListener { adapter.retry() }
}

To fix our functionality, we need to react to load state changes in our SearchRepositoriesActivity. For this, we will use the PagingDataAdapter.addLoadStateListener() method. This callback notifies us every time there's a change in the load state via a CombinedLoadStates object.

CombinedLoadStates gives us the load state for the PageSource we defined or for the RemoteMediator needed for network and database case (more about this later).

Since we only want our progress bar to be displayed when we have a new query, we need to rely on load from our paging source, specifically CombinedLoadStates.source.refresh and on the LoadState: Loading or Error rely. Also, one piece of functionality we commented out in a previous step was displaying a Toast when we got an error, so let's make sure we bring that in as well. To display the error message we will have to check whether CombinedLoadStates.prepend or CombinedLoadStates.append is an instance of LoadState.Error and retrieve the error message from the error.

Let's update our SearchRepositoriesActivity.initAdapter method to have this functionality:

private fun initAdapter() {
    binding.list.adapter = adapter.withLoadStateHeaderAndFooter(
            header = ReposLoadStateAdapter { adapter.retry() },
            footer = ReposLoadStateAdapter { adapter.retry() }
    )
    adapter.addLoadStateListener { loadState ->
        // Only show the list if refresh succeeds.
        binding.list.isVisible = loadState.source.refresh is LoadState.NotLoading
        // Show loading spinner during initial load or refresh.
        binding.progressBar.isVisible = loadState.source.refresh is LoadState.Loading
        // Show the retry state if initial load or refresh fails.
        binding.retryButton.isVisible = loadState.source.refresh is LoadState.Error

        // Toast on any error, regardless of whether it came from RemoteMediator or PagingSource
        val errorState = loadState.source.append as? LoadState.Error
                ?: loadState.source.prepend as? LoadState.Error
                ?: loadState.append as? LoadState.Error
                ?: loadState.prepend as? LoadState.Error
        errorState?.let {
            Toast.makeText(
                    this,
                    "\uD83D\uDE28 Wooops ${it.error}",
                    Toast.LENGTH_LONG
            ).show()
        }
    }
}

Make sure you remove the showEmptyList() method, as we no longer need it.

Now let's run the app and check out how it works!

That's it! With the current setup, the Paging library components are the ones triggering the API requests at the right time, handling the in-memory cache, and displaying the data. Run the app and try searching for repositories.

You can find the full code for the steps done so far in branch step11_loading_state.

One way to improve your list's readability is to add separators. For example, in our app, since the repositories are ordered by number of stars descending, we could have separators every 10k stars. To help implement this, the Paging 3.0 API allows inserting separators into PagingData.

Adding separators in PagingData will lead to the modification of the list we display on our screen. We no longer display just Repo objects but also separator objects. Therefore, we have to change the UI model we're exposing from the ViewModel from Repo to another type that can encapsulate both types: RepoItem and SeparatorItem. Next, we'll have to update our UI to support separators:

Let's take this step by step and see what the implementation looks like.

Change the UI model

Currently SearchRepositoriesViewModel.searchRepo() returns Flow<PagingData<Repo>>. To support both repositories and separators, we'll create a UiModel sealed class in the same file with SearchRepositoriesViewModel. We can have 2 types of UiModel objects: RepoItem and SeparatorItem.

sealed class UiModel {
    data class RepoItem(val repo: Repo) : UiModel()
    data class SeparatorItem(val description: String) : UiModel()
}

Because we want to separate repositories based on 10k stars, let's create an extension property on RepoItem that rounds up the number of stars for us:

private val UiModel.RepoItem.roundedStarCount: Int
    get() = this.repo.stars / 10_000

Insert separators

SearchRepositoriesViewModel.searchRepo() should now return Flow<PagingData<UiModel>>. Make currentSearchResult the same type.

class SearchRepositoriesViewModel(private val repository: GithubRepository) : ViewModel() {

    private var currentQueryValue: String? = null

    private var currentSearchResult: Flow<PagingData<UiModel>>? = null

    fun searchRepo(queryString: String): Flow<PagingData<UiModel>> {
        ... 
    }
}

Let's see how the implementation changes! Currently, repository.getSearchResultStream(queryString) returns a Flow<PagingData<Repo>>, so the first operation we need to add is to transform each Repo into a UiModel.RepoItem. To do this, we can use the Flow.map operator and then map each PagingData to build a new UiModel.Repo from the current Repo item, resulting in a Flow<PagingData<UiModel.RepoItem>>:

...
val newResult: Flow<PagingData<UiModel>> = repository.getSearchResultStream(queryString)
                .map { pagingData -> pagingData.map { UiModel.RepoItem(it) } }
...

Now we can insert the separators! For each emission of the Flow, we'll call PagingData.insertSeparators(). This method returns a PagingData containing each original element, with an optional separator that you will generate, given the elements before and after. In boundary conditions (at the beginning or end of the list) the respective before or after elements will be null. If a separator doesn't need to be created, return null.

Because we're changing the type of PagingData elements from UiModel.Repo to UiModel, make sure you explicitly set the type arguments of the insertSeparators() method.

Here's what the searchRepo() method should look like:

fun searchRepo(queryString: String): Flow<PagingData<UiModel>> {
    val lastResult = currentSearchResult
    if (queryString == currentQueryValue && lastResult != null) {
        return lastResult
    }
    currentQueryValue = queryString
    val newResult: Flow<PagingData<UiModel>> = repository.getSearchResultStream(queryString)
            .map { pagingData -> pagingData.map { UiModel.RepoItem(it) } }
            .map {
                it.insertSeparators<UiModel.RepoItem, UiModel> { before, after ->
                    if (after == null) {
                        // we're at the end of the list
                        return@insertSeparators null
                    }

                    if (before == null) {
                        // we're at the beginning of the list
                        return@insertSeparators UiModel.SeparatorItem("${after.roundedStarCount}0.000+ stars")
                    }
                    // check between 2 items
                    if (before.roundedStarCount > after.roundedStarCount) {
                        if (after.roundedStarCount >= 1) {
                            UiModel.SeparatorItem("${after.roundedStarCount}0.000+ stars")
                        } else {
                            UiModel.SeparatorItem("< 10.000+ stars")
                        }
                    } else {
                        // no separator
                        null
                    }
                }
            }
            .cachedIn(viewModelScope)
    currentSearchResult = newResult
    return newResult
}

Support multiple view types

SeparatorItem objects need to be displayed in our RecyclerView. We're only displaying a string here, so let's create a separator_view_item layout with a TextView in the res/layout folder:

<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout 
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    android:background="@color/separatorBackground">

    <TextView
        android:id="@+id/separator_description"
        android:layout_width="0dp"
        android:layout_height="wrap_content"
        android:padding="@dimen/row_item_margin_horizontal"
        android:textColor="@color/separatorText"
        android:textSize="@dimen/repo_name_size"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toTopOf="parent"
        tools:text="10000+ stars" />
</androidx.constraintlayout.widget.ConstraintLayout>

Let's create a SeparatorViewHolder in the ui folder, where we just bind a string to the TextView:

class SeparatorViewHolder(view: View) : RecyclerView.ViewHolder(view) {
    private val description: TextView = view.findViewById(R.id.separator_description)

    fun bind(separatorText: String) {
        description.text = separatorText
    }

    companion object {
        fun create(parent: ViewGroup): SeparatorViewHolder {
            val view = LayoutInflater.from(parent.context)
                    .inflate(R.layout.separator_view_item, parent, false)
            return SeparatorViewHolder(view)
        }
    }
}

Update ReposAdapter to support a UiModel instead of a Repo:

Here's what your final ReposAdapter will look like: 

class ReposAdapter : PagingDataAdapter<UiModel, ViewHolder>(UIMODEL_COMPARATOR) {

    override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): ViewHolder {
        return if (viewType == R.layout.repo_view_item) {
            RepoViewHolder.create(parent)
        } else {
            SeparatorViewHolder.create(parent)
        }
    }

    override fun getItemViewType(position: Int): Int {
        return when (getItem(position)) {
            is UiModel.RepoItem -> R.layout.repo_view_item
            is UiModel.SeparatorItem -> R.layout.separator_view_item
            null -> throw UnsupportedOperationException("Unknown view")
        }
    }

    override fun onBindViewHolder(holder: ViewHolder, position: Int) {
        val uiModel = getItem(position)
        uiModel.let {
            when (uiModel) {
                is UiModel.RepoItem -> (holder as RepoViewHolder).bind(uiModel.repo)
                is UiModel.SeparatorItem -> (holder as SeparatorViewHolder).bind(uiModel.description)
            }
        }
    }

    companion object {
        private val UIMODEL_COMPARATOR = object : DiffUtil.ItemCallback<UiModel>() {
            override fun areItemsTheSame(oldItem: UiModel, newItem: UiModel): Boolean {
                return (oldItem is UiModel.RepoItem && newItem is UiModel.RepoItem &&
                        oldItem.repo.fullName == newItem.repo.fullName) ||
                        (oldItem is UiModel.SeparatorItem && newItem is UiModel.SeparatorItem &&
                                oldItem.description == newItem.description)
            }

            override fun areContentsTheSame(oldItem: UiModel, newItem: UiModel): Boolean =
                    oldItem == newItem
        }
    }
}

That's it! When running the app you should be able to see the separators!

You can find the full code for the steps done so far in branch step12_separators.

Let's add offline support to our app by saving the data in a local database. That way, the database will be the source of truth for our app and we will always load data from there. Whenever we don't have any more data, we request more from the network and then save it in the database. Because the database is the source of truth, the UI will be automatically updated when more data is saved.

Here's what we need to do to add offline support:

  1. Create a Room database, a table to save the Repo objects in, and a DAO that we'll use to work with the Repo objects.
  2. Define how to load data from the network when we reach the end of the data in the database by implementing a RemoteMediator.
  3. Build a Pager based on the Repos table as a data source and the RemoteMediator for loading and saving data.

Let's take each of these steps!

Our Repo objects need to be saved in the database, so let's start by making the Repo class an entity, with tableName = "repos", where the Repo.id is the primary key. To do this, annotate the Repo class with @Entity(tableName = "repos") and add the @PrimaryKey annotation to id. This is what your Repo class should look like now:

@Entity(tableName = "repos")
data class Repo(
    @PrimaryKey @field:SerializedName("id") val id: Long,
    @field:SerializedName("name") val name: String,
    @field:SerializedName("full_name") val fullName: String,
    @field:SerializedName("description") val description: String?,
    @field:SerializedName("html_url") val url: String,
    @field:SerializedName("stargazers_count") val stars: Int,
    @field:SerializedName("forks_count") val forks: Int,
    @field:SerializedName("language") val language: String?
)

Create a new db package. This is where we will implement the class that accesses data in the database and the class that defines the database.

Implement the data access object (DAO) to access the repos table by creating a RepoDao interface, annotated with @Dao. We need the following actions on Repo:

 @Insert(onConflict = OnConflictStrategy.REPLACE)
 suspend fun insertAll(repos: List<Repo>)
@Query("SELECT * FROM repos WHERE " +
  "name LIKE :queryString OR description LIKE :queryString " +
  "ORDER BY stars DESC, name ASC")
fun reposByName(queryString: String): PagingSource<Int, Repo>
@Query("DELETE FROM repos")
suspend fun clearRepos()

This is what your RepoDao should look like: 

@Dao
interface RepoDao {

    @Insert(onConflict = OnConflictStrategy.REPLACE)
    suspend fun insertAll(repos: List<Repo>)

    @Query("SELECT * FROM repos WHERE " +
   "name LIKE :queryString OR description LIKE :queryString " +
   "ORDER BY stars DESC, name ASC")
    fun reposByName(queryString: String): PagingSource<Int, Repo>

    @Query("DELETE FROM repos")
    suspend fun clearRepos()
}

Implement the Repo database:

Here's what your RepoDatabase looks like:

@Database(
    entities = [Repo::class],
    version = 1,
    exportSchema = false
)
abstract class RepoDatabase : RoomDatabase() {

    abstract fun reposDao(): RepoDao

    companion object {

        @Volatile
        private var INSTANCE: RepoDatabase? = null

        fun getInstance(context: Context): RepoDatabase =
                INSTANCE ?: synchronized(this) {
                    INSTANCE
                            ?: buildDatabase(context).also { INSTANCE = it }
                }

        private fun buildDatabase(context: Context) =
                Room.databaseBuilder(context.applicationContext,
                        RepoDatabase::class.java, "Github.db")
                        .build()
    }
}

Now that we've set up our database, let's see how we request data from the network and save it in the database.

The Paging library uses the database as a source of truth for the data that needs to be displayed in the UI. Whenever we don't have any more data in the database, we need to request more from the network. To help with this, Paging 3.0 defines the RemoteMediator abstract class, with one method that needs to be implemented: load(). This method will be called whenever we need to load more data from the network. This class returns a MediatorResult object, that can either be:

In the data package, let's create a new class called GithubRemoteMediator that extends RemoteMediator. This class will be recreated for every new query, so it will receive the following as parameters: 

@OptIn(ExperimentalPagingApi::class)
class GithubRemoteMediator(
    private val query: String,
    private val service: GithubService,
    private val repoDatabase: RepoDatabase
) : RemoteMediator<Int, Repo>() {

    override suspend fun load(loadType: LoadType, state: PagingState<Int, Repo>): MediatorResult {

   }
}

To be able to build the network request, the load method has 2 parameters that should give us all the information we need:

For example, if the load type is LoadType.APPEND then we retrieve the last item that was loaded from the PagingState. Based on that we should be able to find out how to load the next batch of Repo objects, by computing the next page to be loaded.

In the next section you'll find out how to compute keys for the next and previous pages to be loaded.

For the purposes of the Github API, the page key that we use to request pages of repos is just a page index that is incremented when getting the next page. This means that given a Repo object, the next batch of Repo objects can be requested based on page index + 1. The previous batch of Repo objects can be requested based on page index - 1. All Repo objects received on a certain page response will have the same next and previous keys.

When we get the last item loaded from the PagingState, there's no way to know the index of the page it belonged to. To solve this problem, we can add another table that stores the next and previous page keys for each Repo; we can call it remote_keys. While this can be done in the Repo table, creating a new table for the next and previous remote keys associated with a Repo allows us to have a better separation of concerns.

In the db package, let's create a new data class called RemoteKeys, annotate it with @Entity, and add 3 properties: the repo id (which is also the primary key), and the previous and next keys (which can be null when we can't append or prepend data). 

@Entity(tableName = "remote_keys")
data class RemoteKeys(
    @PrimaryKey
    val repoId: Long,
    val prevKey: Int?,
    val nextKey: Int?
)

Let's create a RemoteKeysDao interface. We will need the following capabilities:

@Dao
interface RemoteKeysDao {

    @Insert(onConflict = OnConflictStrategy.REPLACE)
    suspend fun insertAll(remoteKey: List<RemoteKeys>)

    @Query("SELECT * FROM remote_keys WHERE repoId = :repoId")
    suspend fun remoteKeysRepoId(repoId: Long): RemoteKeys?

    @Query("DELETE FROM remote_keys")
    suspend fun clearRemoteKeys()
}

Let's add the RemoteKeys table to our database and provide access to the RemoteKeysDao. To do this, update the RepoDatabase as follows:

@Database(
        entities = [Repo::class, RemoteKeys::class],
        version = 1,
        exportSchema = false
)
abstract class RepoDatabase : RoomDatabase() {

    abstract fun reposDao(): RepoDao
    abstract fun remoteKeysDao(): RemoteKeysDao

    ... 
    // rest of the class doesn't change
}

Now that we saved the remote keys, let's get back to GithubRemoteMediator and see how to use them. This class will replace our GithubPagingSource. Let's copy the GITHUB_STARTING_PAGE_INDEX declaration from GithubPagingSource in our GithubRemoteMediator and delete the GithubPagingSource class.

Let's see how we can implement the GithubRemoteMediator.load() method:

  1. Find out what page we need to load from the network, based on the LoadType.
  2. Trigger the network request.
  3. Once the network request completes, if the received list of repositories is not empty, then do the following:
  1. We compute the RemoteKeys for every Repo.
  2. If this a new query (loadType = REFRESH) then we clear the database.
  3. Save the RemoteKeys and Repos in the database.
  4. Return MediatorResult.Success(endOfPaginationReached = false).
  1. If the list of repos was empty then we return MediatorResult.Success(endOfPaginationReached = true). If we get an error requesting data we return MediatorResult.Error.

Here's how the code looks like overall. We'll replace the TODOs later on. 

override suspend fun load(loadType: LoadType, state: PagingState<Int, Repo>): MediatorResult {
    val page = when (loadType) {
        LoadType.REFRESH -> {
         // TODO
        }
        LoadType.PREPEND -> {
        // TODO
        }
        LoadType.APPEND -> {
        // TODO
        }
    }
    val apiQuery = query + IN_QUALIFIER

    try {
        val apiResponse = service.searchRepos(apiQuery, page, state.config.pageSize)

        val repos = apiResponse.items
        val endOfPaginationReached = repos.isEmpty()
        repoDatabase.withTransaction {
            // clear all tables in the database
            if (loadType == LoadType.REFRESH) {
                repoDatabase.remoteKeysDao().clearRemoteKeys()
                repoDatabase.reposDao().clearRepos()
            }
            val prevKey = if (page == GITHUB_STARTING_PAGE_INDEX) null else page - 1
            val nextKey = if (endOfPaginationReached) null else page + 1
            val keys = repos.map {
                RemoteKeys(repoId = it.id, prevKey = prevKey, nextKey = nextKey)
            }
            repoDatabase.remoteKeysDao().insertAll(keys)
            repoDatabase.reposDao().insertAll(repos)
        }
        return MediatorResult.Success(endOfPaginationReached = endOfPaginationReached)
    } catch (exception: IOException) {
        return MediatorResult.Error(exception)
    } catch (exception: HttpException) {
        return MediatorResult.Error(exception)
    }
}

Let's see how we find the page to load based on the LoadType.

Now that we know what happens in the GithubRemoteMediator.load() method once we have the page key, let's see how we compute it. This will depend on the LoadType.

LoadType.APPEND

When we need to load data at the end of the currently loaded data set, the load parameter is LoadType.APPEND. So now, based on the last item in the database we need to compute the network page key.

  1. We need to get the remote key of the last Repo item loaded from the database—let's separate this in a function:
    private suspend fun getRemoteKeyForLastItem(state: PagingState<Int, Repo>): RemoteKeys? {
        // Get the last page that was retrieved, that contained items.
        // From that last page, get the last item
        return state.pages.lastOrNull() { it.data.isNotEmpty() }?.data?.lastOrNull()
                ?.let { repo ->
                    // Get the remote keys of the last item retrieved
repoDatabase.remoteKeysDao().remoteKeysRepoId(repo.id)
                }
    }
  1. RemoteKey should never be null. If we need to append data, it means that we got Repo data in the database; therefore, we should have data in the RemoteKeys table as well. If RemoteKey is null, let's throw an error.
  2. In the previous step, when we loaded the previous data from the network and we didn't have any more data to load, we set RemoteKey.nextKey to null and returned MediatorResult.Success(endOfPaginationReached = true). If RemoteKey.nextKey is null we should also throw an error because we shouldn't have gotten into this situation.
val page = when (loadType) {
    LoadType.APPEND -> {
        val remoteKeys = getRemoteKeyForLastItem(state)
        if (remoteKeys == null || remoteKeys.nextKey == null) {
            throw InvalidObjectException("Remote key should not be null for $loadType")
        }
        remoteKeys.nextKey
    }
      ...
  }

LoadType.PREPEND

When we need to load data at the beginning of the currently loaded data set, the load parameter is LoadType.PREPEND. Based on the first item in the database we need to compute the network page key.

  1. We need to get the remote key of the first Repo item loaded from the database—let's separate this in a function:
private suspend fun getRemoteKeyForFirstItem(state: PagingState<Int, Repo>): RemoteKeys? {
    // Get the first page that was retrieved, that contained items.
    // From that first page, get the first item
    return state.pages.firstOrNull { it.data.isNotEmpty() }?.data?.firstOrNull()
            ?.let { repo ->
                // Get the remote keys of the first items retrieved
                repoDatabase.remoteKeysDao().remoteKeysRepoId(repo.id)
            }
}
  1. RemoteKey should never be null. If we need to prepend data, it means that we got Repo data in the database; therefore, we should have data in RemoteKeys table as well. If RemoteKey is null, let's throw an error.
  2. If the prevKey == null, it means that we can't request more data so we can return MediatorResult.Success(endOfPaginationReached = true)
val page = when (loadType) {
    LoadType.PREPEND -> {
        val remoteKeys = getRemoteKeyForFirstItem(state)
        if (remoteKeys == null) {
            // The LoadType is PREPEND so some data was loaded before,
            // so we should have been able to get remote keys
            // If the remoteKeys are null, then we're an invalid state and we have a bug
            throw InvalidObjectException("Remote key and the prevKey should not be null")
        }
        // If the previous key is null, then we can't request more data
        val prevKey = remoteKeys.prevKey
        if (prevKey == null) {
            return MediatorResult.Success(endOfPaginationReached = true)
        }
        remoteKeys.prevKey
    }

      ...
  }

LoadType.REFRESH

LoadType.REFRESH gets called when it's the first time we're loading data, or when PagingDataAdapter.refresh() is called; so now the point of reference for loading our data is the state.anchorPosition. If this is the first load, then the anchorPosition is null. When PagingDataAdapter.refresh() is called, the anchorPosition is the first visible position in the displayed list, so we will need to load the page that contains that specific item.

  1. Based on the anchorPosition from the state, we can get the closest Repo item to that position by calling state.closestItemToPosition().
  2. Based on the Repo item, we can get the RemoteKeys from the database. 
private suspend fun getRemoteKeyClosestToCurrentPosition(
        state: PagingState<Int, Repo>
): RemoteKeys? {
    // The paging library is trying to load data after the anchor position
    // Get the item closest to the anchor position
    return state.anchorPosition?.let { position ->
        state.closestItemToPosition(position)?.id?.let { repoId -> 
   repoDatabase.remoteKeysDao().remoteKeysRepoId(repoId)
        }
    }
}
  1. If remoteKey is not null, then we can get the nextKey from it. In the Github API the page keys are incremented sequentially. So to get the page that contains the current item, we just subtract 1 from remoteKey.nextKey.
  2. If RemoteKey is null (because the anchorPosition was null), then the page we need to load is the initial one: GITHUB_STARTING_PAGE_INDEX

Now, the full page computation looks like this:

val page = when (loadType) {
    LoadType.REFRESH -> {
        val remoteKeys = getRemoteKeyClosestToCurrentPosition(state)
        remoteKeys?.nextKey?.minus(1) ?: GITHUB_STARTING_PAGE_INDEX
    }
    LoadType.PREPEND -> {
        val remoteKeys = getRemoteKeyForFirstItem(state)
        if (remoteKeys == null) {
            throw InvalidObjectException("Remote key and the prevKey should not be null")
        }
        val prevKey = remoteKeys.prevKey
        if (prevKey == null) {
            return MediatorResult.Success(endOfPaginationReached = false)
        }
        remoteKeys.prevKey
    }
    LoadType.APPEND -> {
        val remoteKeys = getRemoteKeyForLastItem(state)
        if (remoteKeys?.nextKey == null) {
            throw InvalidObjectException("Remote key should not be null for $loadType")
        }
        remoteKeys.nextKey
    }
}

Now that we have the GithubRemoteMediator and the PagingSource in our ReposDao implemented, we need to update GithubRepository.getSearchResultStream to use them.

In order to do this, GithubRepository needs access to the database. Let's pass the database as a parameter in the constructor. Also, since this class will use GithubRemoteMediator: 

class GithubRepository(
        private val service: GithubService,
        private val database: RepoDatabase
) { ... }

Update the Injection file: 

object Injection {

    private fun provideGithubRepository(context: Context): GithubRepository {
        return GithubRepository(GithubService.create(), RepoDatabase.getInstance(context))
    }

    fun provideViewModelFactory(context: Context): ViewModelProvider.Factory {
        return ViewModelFactory(provideGithubRepository(context))
    }
}

Update the SearchRepositoriesActivity.onCreate() method and pass the context to Injection.provideViewModelFactory(): 

// get the view model
viewModel = ViewModelProvider(this, Injection.provideViewModelFactory(this))
        .get(SearchRepositoriesViewModel::class.java)

Let's get back to GithubRepository. First, to be able to search for repos by name, we'll have to add % to the beginning and end of the query string. Then, when calling the reposDao.reposByName, we get a PagingSource. Because the PagingSource is invalidated every time we make a change in the database, we need to tell Paging how to get a new instance of the PagingSource. To do this, we just create a function that calls the database query: 

// appending '%' so we can allow other characters to be before and after the query string
val dbQuery = "%${query.replace(' ', '%')}%"
val pagingSourceFactory =  { database.reposDao().reposByName(dbQuery)}

Now we can change the Pager builder, to use a GithubRemoteMediator and the pagingSourceFactory.

return Pager(
        config = PagingConfig(
            pageSize = NETWORK_PAGE_SIZE,
            enablePlaceholders = false
         ),
        remoteMediator = GithubRemoteMediator(
                query,
                service,
                database
        ),
        pagingSourceFactory = pagingSourceFactory
).flow

That's it! Let's run the app!

You can find the full code for the steps done so far in branch step13-19_network_and_database.

Now that we have added all the components, let's recap what we've learned!