fix: bake the NDK into the image so debugSymbolLevel works

ndk { debugSymbolLevel = "SYMBOL_TABLE" } needs the NDK's llvm-objcopy
to extract native symbol tables into the .aab's BUNDLE-METADATA. The
image shipped no NDK, so AGP silently skipped the step and Play warned
"no debug symbols" on every bundle with native code (confirmed by
inspecting Dade's 0.0.2 .aab: onnxruntime .so present, debugsymbols
metadata dir absent).

Install ndk;27.0.12077973 (AGP 8.13's default, so app repos need no
android.ndkVersion), export ANDROID_NDK_ROOT/HOME, and add llvm-objcopy
to the image sanity check so an arch/version mismatch fails the image
build instead of silently dropping symbols downstream.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.gitea/workflows/builder-image.yml

@@ -15,7 +15,9 @@ env:
 
 jobs:
   build-image:
-    runs-on: ubuntu-24.04-arm64
+    # amd64: Google ships the Linux build-tools (aapt2, d8) for x86_64 only,
+    # so the toolchain must be built for and run on amd64.
+    runs-on: ubuntu-24.04
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -43,12 +45,12 @@ jobs:
       - name: Set up Buildx
         uses: docker/setup-buildx-action@v3
 
-      - name: Build and push (linux/arm64)
+      - name: Build and push (linux/amd64)
         uses: docker/build-push-action@v6
         with:
           context: .
           file: ./Dockerfile
-          platforms: linux/arm64
+          platforms: linux/amd64
           push: true
           tags: ${{ steps.tags.outputs.tags }}
           # Bump these here if you want to override the Dockerfile defaults

Dockerfile

@@ -1,7 +1,9 @@
-# Native linux/arm64 Android build toolchain for CI.
+# Native linux/amd64 Android build toolchain for CI.
 # Builds release artifacts only — no emulator, no test/system images.
-# eclipse-temurin publishes a native linux/arm64 manifest, so this runs
-# without emulation on an OCI Ampere (aarch64) runner.
+# Must be amd64: Google publishes the Linux build-tools (aapt2, d8, …) for
+# x86_64 only — there is no arm64 Linux build-tools package — so packaging
+# cannot run natively on an aarch64 runner. The APK output is architecture-
+# neutral regardless of the build host.
 FROM eclipse-temurin:21-jdk-noble
 
 # --- Version pins (bump deliberately; this is what your app repos trust) ---
@@ -10,6 +12,12 @@ FROM eclipse-temurin:21-jdk-noble
 ARG CMDLINE_TOOLS_VERSION=13114758
 ARG BUILD_TOOLS_VERSION=36.0.0
 ARG PLATFORM_VERSION=android-36
+# NDK: required for `ndk { debugSymbolLevel }` — AGP shells out to the NDK's
+# llvm-objcopy to extract native symbol tables into the .aab's BUNDLE-METADATA.
+# Without it AGP silently skips the step and Play warns "no debug symbols". Pin
+# to AGP's default NDK (r27 for AGP 8.13) so app repos need no `ndkVersion`.
+# Check the default at https://developer.android.com/build/releases/gradle-plugin
+ARG NDK_VERSION=27.0.12077973
 
 ENV ANDROID_SDK_ROOT=/opt/android-sdk \
     ANDROID_HOME=/opt/android-sdk \
@@ -20,6 +28,15 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
         curl unzip git ca-certificates && \
     rm -rf /var/lib/apt/lists/*
 
+# Node is required for Gitea Actions JS actions (actions/checkout,
+# upload-artifact, ...) to run when this image is used as a job container —
+# the runner exec's `node` inside the container. Noble's apt nodejs is v18;
+# pull the current Node 24 from NodeSource instead.
+ARG NODE_MAJOR=24
+RUN curl -fsSL "https://deb.nodesource.com/setup_${NODE_MAJOR}.x" | bash - && \
+    apt-get install -y --no-install-recommends nodejs && \
+    rm -rf /var/lib/apt/lists/*
+
 # Install the command-line tools into the canonical "latest" location.
 RUN mkdir -p ${ANDROID_SDK_ROOT}/cmdline-tools && \
     curl -fsSL -o /tmp/tools.zip \
@@ -31,18 +48,31 @@ RUN mkdir -p ${ANDROID_SDK_ROOT}/cmdline-tools && \
 ENV PATH=${PATH}:${ANDROID_SDK_ROOT}/cmdline-tools/latest/bin:${ANDROID_SDK_ROOT}/platform-tools
 
 # Accept licenses and bake the SDK packages into the image so prod builds
-# don't depend on Google's endpoint at job time.
-# aapt2 and the build-tools binaries ship native arm64 for 35.x, so APK/AAB
-# packaging runs natively on aarch64.
+# don't depend on Google's endpoint at job time. The build-tools binaries
+# (aapt2, d8) are x86_64 ELF — this is why the image must be amd64.
 RUN yes | sdkmanager --licenses >/dev/null && \
     sdkmanager --install \
       "platform-tools" \
       "platforms;${PLATFORM_VERSION}" \
-      "build-tools;${BUILD_TOOLS_VERSION}" && \
+      "build-tools;${BUILD_TOOLS_VERSION}" \
+      "ndk;${NDK_VERSION}" && \
     rm -rf ${ANDROID_SDK_ROOT}/.android
 
+# Point AGP at the baked-in NDK. AGP auto-discovers ndk;<ver> under the SDK
+# root, but exporting these makes it explicit and lets the debugSymbolLevel
+# step resolve llvm-objcopy without an app-side android.ndkVersion.
+ENV ANDROID_NDK_ROOT=${ANDROID_SDK_ROOT}/ndk/${NDK_VERSION} \
+    ANDROID_NDK_HOME=${ANDROID_SDK_ROOT}/ndk/${NDK_VERSION}
+
 # Sanity: fail the image build if the toolchain isn't actually usable.
-RUN java -version && sdkmanager --version && \
-    test -d "${ANDROID_SDK_ROOT}/platforms/${PLATFORM_VERSION}"
+# Execute aapt2 (not just --version a tool) so an arch mismatch in the
+# build-tools binaries fails the image build here, not in every app build.
+# Execute the NDK's llvm-objcopy too — it is the exact binary the bundle's
+# debug-symbol extraction shells out to, so an arch/version mismatch fails
+# here rather than silently dropping symbols from every app's .aab.
+RUN java -version && sdkmanager --version && node --version && \
+    "${ANDROID_SDK_ROOT}/build-tools/${BUILD_TOOLS_VERSION}/aapt2" version && \
+    test -d "${ANDROID_SDK_ROOT}/platforms/${PLATFORM_VERSION}" && \
+    "${ANDROID_NDK_ROOT}/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-objcopy" --version
 
 WORKDIR /workspace

README.md

@@ -1,21 +1,35 @@
 # android-builder
 
-Native `linux/arm64` Android build toolchain for CI, published to the Gitea
+Native `linux/amd64` Android build toolchain for CI, published to the Gitea
 container registry at `git.helu.ca/r/android`.
 
 App repos consume this image to build signed release artifacts. Instrumented
 tests are **not** part of this toolchain — by design, CI builds are promotions
 of code already tested in Dev (on Apple silicon, where the emulator runs
-natively). The OCI Ampere (aarch64) runner has no `/dev/kvm` (the guest VM
-boots at EL1, so KVM can't access HYP/EL2), so there's no accelerated emulator
-here — and we don't need one.
+natively).
+
+**Publishing to Google Play?** See [`docs/google-play-publishing.md`](docs/google-play-publishing.md)
+— the reusable Heluca playbook (publisher setup, upload keystore, per-app
+checklist, internal testing, promotion to production), learned from shipping Dade.
+
+**Why amd64:** Google publishes the Linux Android build-tools (`aapt2`, `d8`,
+…) for **x86_64 only** — there is no arm64 Linux build-tools package. Running
+the toolchain on an aarch64 runner fails at resource packaging
+(`aapt2: Syntax error: newline unexpected`, an x86_64 binary the kernel can't
+exec). The produced APK/AAB is architecture-neutral regardless of build host,
+so building on amd64 costs nothing on the output side.
 
 ## What's in the image
 
-- Eclipse Temurin JDK 21 (native arm64)
-- Android cmdline-tools, platform, and build-tools — **baked in**, so prod
-  builds don't depend on Google's download endpoint at job time
+- Eclipse Temurin JDK 21 (native amd64)
+- Android cmdline-tools, platform, build-tools, and **NDK** — **baked in**, so
+  prod builds don't depend on Google's download endpoint at job time. The NDK is
+  what makes `ndk { debugSymbolLevel }` work: AGP shells out to its
+  `llvm-objcopy` to embed native symbol tables in the `.aab`, so Play can
+  symbolicate native crashes. Without it AGP skips that step silently.
 - `git`, `curl`, `unzip`
+- Node.js 24 — so Gitea Actions JS actions (`actions/checkout`,
+  `upload-artifact`, …) run when this image is used as a job `container`
 
 Pinned versions live as `ARG`s at the top of the `Dockerfile`:
 
@@ -24,9 +38,10 @@ Pinned versions live as `ARG`s at the top of the `Dockerfile`:
 | `CMDLINE_TOOLS_VERSION` | `13114758`  | https://developer.android.com/studio#command-line-tools-only |
 | `BUILD_TOOLS_VERSION`   | `36.0.0`    | SDK Manager / release notes |
 | `PLATFORM_VERSION`      | `android-36`| your app's `compileSdk` |
+| `NDK_VERSION`           | `27.0.12077973` | AGP's default NDK: https://developer.android.com/build/releases/gradle-plugin |
 
-`aapt2` and the build-tools binaries ship native arm64 for 36.x, so packaging
-runs without emulation.
+The build-tools binaries (`aapt2`, `d8`) are x86_64 ELF — this is why the
+image is amd64 (see "Why amd64" above).
 
 ## Tagging model
 
@@ -61,7 +76,7 @@ app and pin the toolchain:
 ```yaml
 jobs:
   build:
-    runs-on: ubuntu-24.04-arm64
+    runs-on: ubuntu-24.04
     container:
       image: git.helu.ca/r/android:2026.06
       credentials:

docs/google-play-publishing.md

@@ -0,0 +1,240 @@
+# Publishing a Heluca Android app to Google Play
+
+The reusable playbook for taking a Heluca app from repo to Google Play, learned
+from shipping **Dade** (`r/dade`) to internal testing. It covers the whole
+lifecycle: the one-time publisher setup, the per-app checklist, and promotion to
+production — with Heluca's conventions (this `r/android` toolchain, Gitea CI,
+Play App Signing) baked in.
+
+Authority order for anything ambiguous: **live app config > this guide > memory**.
+When an app diverges from the convention here, that's allowed — but note it in the
+app's own README so the next person knows it's deliberate. Dade's divergences are
+called out throughout (see *Where Dade diverges*).
+
+---
+
+## 0. Concepts you need once
+
+- **Two keys, one you hold.** Under **Play App Signing** (use it — it's the
+  default and lets Google reset a lost upload key), you generate an **upload key**
+  and sign your bundle with it; Google re-signs with the **app signing key** it
+  holds. You never see the app key.
+- **The upload key is publisher-wide, not per-app.** One keystore signs every
+  Heluca app — Play keys an app by `applicationId`, not by signing key. Generate it
+  once (below), reuse it forever. Don't mint a new one per app.
+- **Play wants an `.aab`, not an APK.** App Bundles only. That means
+  `bundleRelease` (or a flavored `bundle<Flavor>Release`), never `assembleRelease`,
+  for anything going to Play. (`assembleRelease` APKs remain useful for direct
+  `adb install` sideloading — Dade keeps a separate workflow for exactly that.)
+- **`versionCode` is a bare integer Play compares; `versionName` is the human
+  label.** Every upload needs a strictly-greater `versionCode`. Heluca convention:
+  **encode the semver** as `MAJOR*10000 + MINOR*100 + PATCH` so the two track each
+  other (`0.0.1` → `1`, `1.2.3` → `10203`). Bump both together per release.
+- **One listing per `applicationId`.** If an app ships per-environment flavors with
+  distinct ids (Dade: `ca.helu.daedalus` prod, `ca.helu.daedalus.dev` dev), each is
+  a **separate Play app** with its own listing, track, and opt-in link.
+
+---
+
+## 1. Publisher account — one-time for all of Heluca
+
+Done once; every future app reuses it.
+
+1. **Register** at https://play.google.com/console with a long-lived Google
+   account (not a throwaway). **$25 one-time** fee.
+2. **Account type:** Personal is adequate — internal testing isn't gated either
+   way. Organization needs a D-U-N-S number and unlocks skipping the
+   new-personal-account production testing gate; only worth it if you'll push to
+   production broadly.
+3. **Identity verification** (2026 rules): government ID, possibly a selfie and an
+   address document. Can take a few hours to a couple of days.
+4. **Device verification** (personal accounts): install the **Play Console** app
+   on a real Android phone and confirm it before you can publish.
+
+## 2. Generate the upload keystore — one-time for all of Heluca
+
+Lives in `~/.android` (alongside the Android debug keystore) — out of your home
+root, easy to include in backups. **Back it up and store the password in your
+vault.** JDK 17+ `keytool` prompts for a single password and reuses it for both
+the store and the key, so the two password secrets later hold the same value.
+
+```bash
+keytool -genkeypair -v -keystore ~/.android/heluca-upload.jks -alias heluca-upload \
+  -keyalg RSA -keysize 4096 -validity 10000 \
+  -dname "CN=Heluca, O=helu.ca, C=CA"
+```
+
+`-dname` is cosmetic under Play App Signing (Google re-signs), so `CN=Heluca` is
+fine and consistent across apps.
+
+Encode it once for CI secrets:
+
+```bash
+base64 -i ~/.android/heluca-upload.jks | pbcopy    # macOS
+# base64 -w0 ~/.android/heluca-upload.jks           # Linux (one line, no wrap)
+```
+
+---
+
+## 3. Per-app checklist — repeat for each new app
+
+Assumes the account (§1) and upload keystore (§2) already exist.
+
+### 3a. Gradle — signing + AAB + versionCode
+
+The **Heluca convention** is the `r/android` template's approach: **inject signing
+at job time via `-Pandroid.injected.signing.*`, no `signingConfig` in
+`build.gradle.kts` at all.** This keeps the build file clean and the keystore
+purely a CI concern. See `templates/build.yml` in this repo.
+
+Set the versionCode encoding in `defaultConfig`:
+
+```kotlin
+defaultConfig {
+    // MAJOR*10000 + MINOR*100 + PATCH — versionName 0.0.1 -> versionCode 1
+    versionCode = 1
+    versionName = "0.0.1"
+}
+```
+
+Nothing else is required for signing if you use `-Pandroid.injected.signing.*`.
+
+### 3b. Native debug symbols (only if the app bundles `.so` libraries)
+
+If the app ships native libraries (JNI, ONNX, a VAD lib, etc.), Play warns that
+the `.aab` has no native debug symbols and native crashes show raw addresses. Fix
+it in the release build type:
+
+```kotlin
+buildTypes {
+    release {
+        ndk { debugSymbolLevel = "SYMBOL_TABLE" }   // function names; FULL adds line numbers
+    }
+}
+```
+
+AGP embeds the symbols in the `.aab`; Play consumes them on upload. Use
+`SYMBOL_TABLE` for prebuilt deps you don't build from source; `FULL` only if you
+need line numbers. Pure-Kotlin/Java apps can skip this.
+
+### 3c. Minification & the mapping file
+
+If `isMinifyEnabled = true`, upload the R8 mapping file so Play deobfuscates
+crash traces (AGP attaches it to the bundle automatically). If minification is
+**off**, Play still prints a generic "no deobfuscation file" warning — **ignore
+it**, there's nothing to upload.
+
+### 3d. CI secrets (per app repo, or org-level)
+
+Following the `r/android` convention, add these Gitea Action Secrets to the app
+repo (**Settings → Actions → Secrets**). Org-level secrets avoid re-adding them
+per app.
+
+| Secret | Value |
+|--------|-------|
+| `KEYSTORE_BASE64` | `base64 -w0 ~/.android/heluca-upload.jks` (the whole `.jks` as text) |
+| `KEYSTORE_PASSWORD` | the keystore password |
+| `KEY_ALIAS` | `heluca-upload` |
+| `KEY_PASSWORD` | same as the keystore password (JDK 17 unifies them) |
+| `PACKAGE_TOKEN` | PAT with `read:package` (to pull the toolchain image) |
+
+### 3e. CI workflow
+
+Drop `templates/build.yml` from this repo into the app at
+`.gitea/workflows/build.yml`. It runs inside the pinned toolchain image, decodes
+the keystore, and builds the signed artifact. For a Play app, run it with the
+`bundleRelease` task (the template exposes a `workflow_dispatch` choice; flavored
+apps need `bundle<Flavor>Release` — adjust the task/matrix accordingly).
+
+**Pin the toolchain to an immutable tag** (`git.helu.ca/r/android:2026.06`), never
+`:latest`, for anything whose output you'll ship. See this repo's README →
+*Tagging model*.
+
+### 3f. Create the Play app(s) & upload
+
+Per `applicationId` (one app, or one per flavor):
+
+1. Play Console → **Create app** → name, language, app/game, free/paid, accept
+   declarations.
+2. Complete the **App content** declarations (privacy policy URL, data safety, ads,
+   content rating, target audience). Internal testing tolerates a minimal honest
+   pass.
+3. **Testing → Internal testing → Create new release** → upload the `.aab`
+   (download the CI artifact; it's zipped by Gitea's transport — unzip to get the
+   raw `app-*-release.aab`). Enable **Play App Signing** when prompted.
+4. Add release notes ("What's new," ≤500 chars — describe the app for a first-time
+   tester, not the CI plumbing). **Save → Review release → Start rollout.**
+
+### 3g. Add testers & install
+
+1. Internal testing → **Testers** tab → create an email list → add the Google
+   account(s) on your phones.
+2. **Copy link** (the opt-in URL) → open it **on the phone**, signed into that same
+   account → accept → the Play listing's **Install** appears.
+3. Repeat the opt-in per app (each `applicationId` has its own link). Flavored apps
+   install side by side.
+
+---
+
+## 4. Gotchas (all learned the hard way on Dade)
+
+- **"Item isn't available" right after opting in is normal propagation lag** —
+  10–30 min typical, up to a couple of hours on a brand-new app's first release,
+  even when the console says "Available to internal testers." Not a
+  misconfiguration. Confirm the release status, confirm the **exact tester account**
+  is active in the phone's Play Store, then wait.
+- **"Not reviewed" / "(unreviewed)" temporary app name does NOT block internal
+  testing** — that track is review-exempt. It clears on its own.
+- **No install notification.** Google doesn't ping you that a testing app/build
+  exists. First install is always via the opt-in link. Updates auto-arrive (if
+  auto-update is on) or via Play Store → Manage apps & device → Update — no reliable
+  per-app notification for testing tracks.
+- **Each upload needs a higher `versionCode`** or Play rejects the duplicate. The
+  encoding in §0 makes this mechanical.
+- **Same Google account** on the device as in the tester list, or the opt-in link
+  404s / "not available."
+- **The AAB is already a zip; Gitea's artifact download re-zips it** — the outer
+  `.zip` is ~the same size because you can't re-compress compressed data. Unzip once
+  and upload the inner `.aab`. Harmless.
+
+---
+
+## 5. Promoting to production
+
+Internal testing (≤100 testers, no review gate) is where Heluca apps live for
+personal use. To go public:
+
+1. **Closed/open testing first if required.** New *personal* accounts must run a
+   closed test with **≥12 testers for ≥14 days** before production unlocks.
+   Organization accounts skip this. Internal testing does **not** count toward it.
+2. **Full store listing + all App content declarations** must be complete (they're
+   optional-ish for internal, mandatory for production).
+3. **Production track → Create release** → promote the same `.aab` (or a newer one)
+   → it goes through Google **review** (hours to days) before rollout.
+4. Consider a **staged rollout** (start at a small % of users).
+
+For a personal, off-public-store app, you may never leave internal testing — that's
+a valid end state, not a stepping stone.
+
+---
+
+## Where Dade diverges from this convention
+
+Dade shipped before this guide was written and differs in a few deliberate ways.
+Documented here so the next app can decide to follow the template (recommended) or
+knowingly match Dade.
+
+| Aspect | Convention (this guide / `templates/build.yml`) | Dade |
+|--------|--------------------------------------------------|------|
+| Signing injection | `-Pandroid.injected.signing.*` at job time, no `signingConfig` in Gradle | Custom `signingConfig` reading `DADE_KEYSTORE*` env vars |
+| Secret names | `KEYSTORE_BASE64` / `KEYSTORE_PASSWORD` / `KEY_ALIAS` / `KEY_PASSWORD` | `UPLOAD_KEYSTORE_BASE64` / `UPLOAD_KEYSTORE_PASSWORD` / `UPLOAD_KEY_PASSWORD` |
+| Toolchain pin | Immutable `:2026.06` | Moving `:latest` (README flags this to pin later) |
+| Workflow | One `templates/build.yml`, task selectable | Two: `build-apks.yml` (unsigned APKs, sideload) + `build-bundles.yml` (signed AABs, Play) |
+
+Dade's split (separate unsigned-APK and signed-AAB workflows) is worth keeping for
+an app you also sideload during dev. A **new app that only targets Play** should
+follow the single-template convention and standard secret names — less to maintain,
+and org-level secrets work across repos when the names match.
+
+**New app? Prefer the convention in §3.** Reach for Dade's pattern only when you
+have Dade's reason (per-env flavors + a parallel sideload path).

templates/build.yml

@@ -24,7 +24,9 @@ env:
 
 jobs:
   build:
-    runs-on: ubuntu-24.04-arm64
+    # amd64: the toolchain image carries x86_64 Android build-tools (Google
+    # ships no arm64 Linux build-tools), so run it on an amd64 runner.
+    runs-on: ubuntu-24.04
     container:
       image: git.helu.ca/r/android:2026.06
       credentials: