manage_mobile_deployment_vault permission). The
vault holds the secrets, signing files, and CI tokens consumed by the
.github/workflows/mobile-deploy-stores.yaml workflow that builds and publishes
the Flutter mobile app to the Google Play and Apple App stores.
Each field on the settings page has an inline help tooltip with a short summary
and the console URL; this page is the full reference with clickable links.
How The Vault Is Organized
The Infrastructure deployment must have the existingENCRYPTION_MASTER_KEY
that encrypted the vault’s data keys. Link the team’s shared variable to the
Infrastructure project and redeploy for environment changes to take effect.
Do not generate a replacement key for an existing vault: it would make stored
versions unreadable. A server-render error with a missing master-key message in
the deployment logs indicates this configuration is absent.
Vault administration uses the signed-in Infrastructure app session and checks
the root workspace’s manage_mobile_deployment_vault permission. CI reads use
the separately scoped CI token and GitHub OIDC identity.
- Secrets — short scalar values (passwords, IDs, names) and preset environment variables baked into the build.
- Files — signing certificates, keystores, and Firebase/Play config files.
- CI tokens — bearer tokens the GitHub Actions workflow uses to read the vault. Each token value is shown only once, at issue time.
- Overview — readiness checks plus draft activation and rollback.
replay_detected. If the macOS fetch returns HTTP 401 after a
successful preflight, check the vault replay policy before rerunning the job.
Android Signing
Google Play
Apple Signing
App Store Connect API
Firebase
Apple validation builds
The iOS simulator and macOS development CI workflows copy the checked-inGoogleService-Info-development.plist into Xcode’s generic resource path.
They do not generate dummy configuration or retrieve production vault assets.
After building, CI verifies the bundled plist matches the development source and
its bundle ID matches the app’s Info.plist; missing, malformed, dummy, or
wrong-flavor resources fail before artifact upload.
Production TestFlight builds continue to retrieve ios_google_service_info_plist
from Infrastructure using the protected deployment token and GitHub OIDC.
That policy remains restricted to the production store workflow and environment.
Do not grant PR validation workflows access to the production signing bundle.
Run the fixture controls without building an app:
python3 scripts/mobile-deployment/test_firebase_service_config.py.
The two development workflows run these controls before preparing configuration.
Production mobile diagnostics
The production Flutter flavor uses the existing Firebase configuration for Crashlytics and Google Analytics. Development and staging flavors disable collection. Crashlytics captures native crashes, uncaught Flutter and platform errors, and throttled nonfatal Bloc and Meet media failures. Analytics records automatic app usage plusmobile_error,
meet_media_published, and meet_media_received events. Meet failure sources
identify capture, publish, send, receive, or connection stages without recording
room IDs, user IDs, SDP, ICE credentials, message contents, or media.
For a failed call, filter Crashlytics nonfatal issues by meet_media_*,
meet_camera_toggle, or meet_microphone_toggle, then compare the same build’s
Analytics events. A successful publisher event means its WebRTC sender reported
outgoing packets, while a received event means the remote track reached Flutter’s
rendering path. Neither event proves that a person saw or heard the media.
Confirm reception with a physical two-device call. Crashlytics nonfatal reports
may appear on a later app launch; a missing immediate issue is not proof that
the failure was never reported.
Before relying on an iOS crash stack, check the store workflow’s
Crashlytics symbol upload step. Store CI checks the archived Runner binary against its
dSYM UUID and uploads the archive’s dSYMs before sending the IPA to TestFlight.
The ios-production-dsyms workflow artifact is retained for 14 days for manual
recovery if Firebase later reports missing symbols. Historical builds that
skipped Xcode’s optional FlutterFire upload may still need manual symbol recovery.
Use Firebase DebugView on an internal
production-flavor build to verify event delivery without injecting a crash into
the public release.
App Environment Variables
Root platform administrators withmanage_mobile_deployment_vault can export
the active production Dart define file from Infrastructure → Mobile
Deployment → Overview. The export includes build-time environment values only;
it omits signing files, scalar passwords, and CI tokens. The request uses the
same-origin action guard, returns no-store, and records an env.exported
audit event. Save the download outside the repository and pass it to a local
production-flavor simulator build:
CI Tokens
Activating Changes
Edits land in a draft version. Use Verify to re-run readiness checks, Activate draft to promote it to the active bundle the workflow reads, and Roll back to return to the previous active version. Fix any readiness issues listed on the Overview tab before activating.Automatic beta rollout
Mobile Settings → What’s new reads bundled release history when the user opens that screen. Release Please ownsapps/mobile/CHANGELOG.md for tagged
releases. A short shared preparation job looks up earlier TestFlight patch
versions and their source workflow commits, then bundles the mobile commit
summaries for each mapped patch version into assets/release_history.json.
Older manual builds without a matching workflow remain listed with localized
“details unavailable” copy instead of blocking the next beta. CI logs their
version and build number for follow-up. The current build is included before
upload. The iOS and Android jobs download the same generated asset and start
in parallel after preparation. A failure in either store does not prevent the
other platform from publishing; both use the same prepared version and build
number. The preparation job fetches App Store metadata credentials from the
protected vault and removes them before its runner exits. Both apps read the
bundled result offline without a changelog API or extra database table.
The installed version and build number remain visible. Release summaries omit
pure Git merge and branch-sync bookkeeping, including remote-tracking merges;
legitimate product changes such as merging duplicate contacts remain visible.
Mobile history traverses product commits within the exact shipped range, so a
merge does not hide its underlying work. Installed What’s New and the public
GitHub release feed filter old bookkeeping at display time without rewriting
published changelogs or version metadata. TestFlight notes use only the exact
uploaded build’s original history artifact; a metadata retry removes known
bookkeeping bullets from existing English notes while preserving genuine text
and other locales. A history containing only bookkeeping is unavailable test
evidence, rather than a fabricated product summary. Existing installed binaries
and Apple metadata change only after an authorized app update or metadata retry.
Regression coverage: release-note-policy.test.js, release-history.test.mjs,
build-test-notes.test.mjs, release_notes_test.dart, and release-query.test.ts.
The workflow runs after relevant pushes to production. A manual dispatch on
production with mode=release (the default) also prepares and uploads a new
Android internal and iOS TestFlight beta. Use mode=retry_ios_review to retry an
existing iOS build without uploading a release. Both modes require enabled CI;
publication additionally requires successful credential preflight. Other refs
are excluded before credential jobs start. Vault OIDC validation continues to
require the production branch for both push and manual events.
Configure the mobile-store-beta GitHub environment to allow only the
production branch, and store the issued vault token as its
MOBILE_DEPLOYMENT_CI_TOKEN secret. A missing token fails the run instead of
silently skipping both publishing jobs.
Both jobs fetch the active encrypted vault bundle from
https://infrastructure.tuturuuu.com/api/v1/mobile-deployment/bundle, with the
CI bearer token and GitHub OIDC attestation. Signing files, private keys, and
build environment files are hydrated only into temporary runner paths and
cleaned up afterward. Do not copy signing values into repository secrets or
commit them to source. The App Store private key uses API_PRIVATE_KEYS_DIR.
If a bundle fetch returns 401, first confirm that the request uses
environment=production, a supported platform, and a bearer authorization
header. Invalid inputs return before the route logs a denial. For valid inputs,
check the infrastructure production request log for
Mobile deployment bundle request denied. Its stage and code identify where
validation stopped without recording credentials or the OIDC token. If that log
is absent, inspect the proxy or deployment edge. Vault bundle-fetch rows appear
only after OIDC validation and admin-client setup have succeeded.
Each workflow run assigns both platforms the build number
100000 + GITHUB_RUN_NUMBER * 1000 + GITHUB_RUN_ATTEMPT. Retrying a job receives
a fresh number. Release Please continues to own the marketing version. Retry
the newest release workflow rather than an older source revision after a more
recent build has already been published.
Android uploads a signed AAB to the internal track with completed status.
The verification step reads that track using a temporary Play API edit and
requires the exact version code. It deletes the edit without committing it.
iOS validates the profile’s expiration, exact team and bundle ID, App Store
distribution type, and production push entitlement. CI uses a temporary
keychain and an ignored ios/Flutter/StoreSigning.xcconfig for manual
distribution signing. Export uses the profile UUID and App Store Connect export
method. Local development keeps its existing signing settings when that file
does not exist.
Create an internal TestFlight group with automatic distribution enabled. After
upload, CI waits up to 30 minutes for the exact build to become valid and
available for internal testing. It then assigns the build to all existing
TestFlight groups, including external groups, and reads the assignments back.
An accepted upload alone does not pass verification. New groups are discovered
on each run, so future groups are included by default.
The mobile-store-beta GitHub environment accepts two optional variables:
External groups require TestFlight beta review before testers can install a new
build. If an external group is selected, CI creates a missing English (U.S.)
What to Test localization, enables automatic tester notification, and submits
the build for beta review. A workflow success with
WAITING_FOR_REVIEW or
IN_REVIEW confirms submission, not external tester availability. Confirm
Apple’s approval and external testing status in App Store Connect. Keep the
app’s beta review contact and app description current there. Missing export
compliance, processing failures, and internal distribution must also be
resolved in App Store Connect.
Reviewer sign-in and QA accounts
Apple requires an active demo account or a fully featured demo mode for an app whose features require sign-in. In App Store Connect, go to Apps → Tuturuuu → TestFlight → Additional → Test Information. Set Demo Account Required and put the working email and password in the separate Demo Account Name and Demo Account Password fields. Put sign-in steps and feature configuration in Review Notes, without credentials. Apple documents these as separate fields, and says review notes should not contain the demo account details. Keep the backend and credentials active throughout review. App Store release review has its own App Review Information fields; update those separately when submitting the app itself. In the production Infrastructure app, open the root workspace’s Account Security → Review and QA accounts panel. A platform administrator withmanage_internal_accounts can create a reviewer account at a new,
organization-controlled, routed @tutur3u.com address outside staff domains.
Infrastructure confirms its
email and generates a strong password, then displays credentials once and
credential-free review steps separately. Copy them to the corresponding private
App Store Connect fields; never place credentials in source, an issue, CI
output, review notes, or a public TestFlight description. The account has
ordinary user permissions and a
personal workspace from the standard signup triggers. Creating an account does
not seed data or grant workspace administrator permissions.
Before resubmitting an external beta, sign out of a test device, use Email and
password on the production mobile build with the reviewer credentials, and
follow the current login path: enter the email and tap Continue with email;
enter the password directly if shown, or tap Use password instead if a code
screen appears. Complete the security check if prompted, then tap Sign in.
Verify Home, Apps, Profile, Tasks, Calendar, Finance, and Notes in the account’s
personal workspace. Add representative non-sensitive sample data through the
app if the review needs it. Record the tested build number and any non-obvious
navigation steps in the private review notes. Keep the account enabled and the
password stable until Apple finishes reviewing that build. If the password is
lost, rotate it in the same panel, repeat the device sign-in check, and update
App Store Connect before resubmitting.
For a rejected TestFlight build such as 0.16.1 (272001), first correct the
Test Information and verify the reviewer login on the production app. Then use
the next eligible uploaded build through the existing production TestFlight
workflow. scripts/mobile-deployment/verify-store.mjs ios-pending intentionally
does not resubmit a latest build whose beta review state is REJECTED; it waits
for a newer build. If the existing binary should be reconsidered without a new
upload, use Apple’s TestFlight App Review appeal/contact path rather than
assuming the retry workflow will resubmit it. Confirm the newly submitted
build reaches WAITING_FOR_REVIEW, IN_REVIEW, or APPROVED in App Store
Connect; an upload or metadata save alone does not reopen a rejected review.
The store workflow now checks Apple’s private demoAccountRequired,
demoAccountName, demoAccountPassword, and credential-free notes fields
before a new external beta submission. The reviewer address must match the
provisioning domain rule (@tutur3u.com); staff-domain and other addresses defer
external submission. Provisioning separately rejects addresses reserved in
platform_email_roles; the store metadata check does not query that table.
It reports only whether access is configured, never credential values. The production push uploads a new IPA
before this check. A missing field defers external review, logs that it was
not submitted, and leaves the store workflow green once internal distribution
is verified. A green store workflow or beta group assignment does not mean
external testers can install: verify the build is Testing for the external
group after Apple approval. This metadata safeguard does not verify current
account enabled status, working login, permissions, or representative demo
content. Those remain unverified until normal read-only QA in the reviewer
session on the relevant production mobile build; record the tested build and
outcomes without credentials. It does not guarantee Apple approval.
Per-build What to Test
What to Test isbetaBuildLocalizations.whatsNew, separate from app-level
reviewer credentials and Review Notes. The upload uses the prepared
mobile-beta-release-history artifact for the exact version, build number, and
source SHA. It creates missing English text or fills blank text, preserves
nonempty manual text and other locales, and reads the result back before a new
external review. Notes preparation runs before reviewer-metadata or pending-review
deferral; internal group distribution remains available when notes cannot be
verified. This does not mean external testing is approved.
Retries resolve the selected build number to its original production upload run
and the specific attempt encoded in the build number, then verify its history
artifact. A later rerun of another job does not invalidate that original attempt.
Retries never use the current checkout’s newer changes. A missing, expired,
ambiguous, or mismatched source/artifact requires manually prepared version-specific notes; the workflow does not invent a changelog or
submit a new external review with unverified notes. A legacy history artifact
is usable only when its original upload run is verified and the exact version
entry has changes. An empty .0 entry must use manually reviewed notes from that
source’s matching release changelog. The former generic
TESTFLIGHT_BETA_WHATS_NEW fallback is no longer used. Automatic writes only fill
absent or blank text; correcting nonempty text remains an explicit manual action.
After Infrastructure deploys, create and sign in to the reviewer
account. In Infrastructure → Internal Accounts → Account Security → Review and
QA accounts, use Publish to Apple after QA while the one-time credential
dialog is open. This action requires both internal account and mobile vault
permissions, and writes Apple’s dedicated demo fields and credential-free notes
using the active vault API key. If the dialog was closed, rotate the reviewer
password, verify sign-in again, and publish from the new one-time dialog. Then
dispatch Mobile Store Deployment on the production branch (gh workflow run mobile-deploy-stores.yaml --ref production -f mode=retry_ios_review). This
explicit review mode runs its Retry latest eligible TestFlight review job
against the newest processed
build; it does not upload another IPA. Confirm the new build reaches
WAITING_FOR_REVIEW, IN_REVIEW, or APPROVED in App Store Connect.
The same panel can send an External invite to an address outside the
organization. The recipient must accept the emailed invitation and verify the
address. Pending invitations do not work as Apple reviewer credentials. The
panel lists managed accounts and can disable or re-enable them; disabling a
review account during an active review will prevent Apple from reaching the app.
Apple allows at most six TestFlight beta review submissions in a 24-hour
period (Apple’s TestFlight guidance).
If Apple returns ENTITY_UNPROCESSABLE.SUBMISSION_LIMIT_REACHED, CI
keeps the verified build available to internal testers, records external review
as deferred, and lets the scheduled review queue retry the newest eligible
build after the limit clears. A green workflow in this case does not mean the
build is available to external testers; confirm approval in App Store Connect.
When App Store Connect rejects a review submission, the workflow logs the HTTP
status, API error code, and JSON field pointer. It omits Apple error titles,
details, and request bodies because they may contain account data. Use the code
and field pointer to inspect the rejected build’s TestFlight information in
App Store Connect before retrying the production review workflow.
Apple permits one external beta build per app version in review at a time. The
review queue checks every five minutes and explicitly dispatches production
with mode=retry_ios_review, so it never starts a new release. It leaves existing
reviews in place and submits the newest eligible
build, even while an older app version is under review. It waits rather than
falling back to an older build when the newest build is still processing. Once
an approved build exists, the queue expires superseded builds older than 14
days, retaining the two newest uploads and two newest approved builds, any
build in review, and builds attached to an App Store release. GitHub’s scheduled
runs can start late, so this is a best-effort interval rather than a guaranteed
five-minute submission time.
Per-build notes defer to a manual handoff when exact upload history is unavailable
or the complete history exceeds Apple’s 4,000-character limit. The verifier never
silently drops later changes. It re-reads English notes immediately before writing
to preserve manual text added while history loads. App Store Connect API failures
and failed write readback fail verification rather than masquerading as missing
history; they require a retry or investigation.
Run focused release checks with node --experimental-strip-types --test scripts/mobile-deployment/*.test.mjs scripts/ci/mobile-store-workflow.test.js. Use ttr resources run -- bun check
for repository validation and ttr resources run -- bun check:mobile for the
Flutter checks. Normally run the applicable mobile tests and native builds in
CI on the exact branch commit, even when local checks pass. If the CI queue is
demonstrably stuck, record the pending jobs and local evidence, then finish the
exact-commit CI checks before treating the release as ready. Store availability
still requires an authenticated release run with the active production vault
and real signing assets.
Device authenticator smoke checks
Open Settings → Session → Device authenticator. Setup opens above the shell in a dismissible phone sheet or tablet dialog; test both orientations, keyboard visibility, Close and system Back. Enroll only on a device with local identity verification configured. Verify registration, a generated code, and a second device approval separately from successful store upload. The devices API uses explicit JWT assurance checks for stateless mobile Bearer requests. A cookie-session-only assurance lookup returns no level for those clients and can reject confirmation after the provider activates the factor. An interrupted setup resumes from its account-scoped secure credential; never clear secure storage as a troubleshooting shortcut. Retry repeats the failed operation. Error messages distinguish network, session, rate-limit, service unavailability, and cancelled local verification without exposing provider payloads or credentials. Simulator biometric support varies by runtime; passing widget tests does not establish real-device biometric approval.Live audio regression checks
The mobile app overridesrecord_ios with the licensed local copy in
apps/mobile/vendor/record_ios. The patch rejects zero-channel or zero-rate
input formats before installing an audio tap, returning a recoverable platform
error instead of an uncaught Core Audio exception. Keep this guard when updating
the recording dependency and test unavailable inputs on the simulator, plus
real microphone and headset routes on a physical device.
Live chat history is read server-side only after verifying that the caller owns
the chat. Test denied ownership and failed ownership lookup before any privileged
message access. The seed includes at most the latest 200 messages in stable
chronological order. Deploy the API fix before treating a mobile-only build as
proof of a working Live session.
Updating one credential and repairing an older draft
The production sync workflow applies20260923120000_mobile_vault_draft_inheritance.sql. Verify that migration and
the updated application deployment have completed before repairing or activating
a draft; do not run a separate direct production SQL push. New drafts
inherit the active version’s credentials under a fresh envelope key before edits
are accepted. An interrupted initialization can be retried; it cannot be used as
an editable partial draft.
For a draft created by the previous implementation, open Infrastructure → Mobile
Deployment → Overview and choose Inherit missing credentials. This is an
explicit administrator mutation using the same permission and CSRF checks as
other vault changes. It retains existing draft values, including an updated
provisioning profile, and preserves recorded deletions and .env replacements.
Legacy deletions are reconstructed from the old audit history; if that history
is incomplete, review the missing-resource list before repair. Repair is a
one-time operation per active source, not a fallback on every read.
Inheritance decrypts only inside the server, checks plaintext and ciphertext
hashes, and re-encrypts under the draft key. Files receive separate immutable
attempt paths, and ordinary credential uploads also use unique encrypted paths
so concurrent uploads cannot overwrite one another’s file bytes. Database
metadata is committed atomically with a draft revision check; concurrent edits
cause a conflict rather than replacement of new values.
After a conflict, refresh and retry. Verify readiness, then activate the draft
before triggering the store workflow.
Preparation failures clean up their own attempted uploads. An ambiguous network
failure during the final database commit can leave unreferenced encrypted attempt
files. Do not delete these merely because the HTTP request failed: first confirm
whether metadata committed and ensure no inheritance request is still running.
Only unreferenced inherit-* files in that draft’s reserved vault directory may
be reclaimed. Never remove active-version or currently referenced artifacts.
Live screen-sharing extension
Live screen sharing uses Android MediaProjection and an iOS ReplayKit Broadcast Upload Extension, rather than camera capture or an app-only screenshot. The operating system asks the user to choose and authorize the capture session. Live shows a stop control; Android also exposes a persistent stop notification and iOS retains its system broadcast indicator. Android adds the microphone foreground service type only while the existing, user-authorized Live microphone is active, and removes that type when muted; screen sharing itself can continue muted. On iOS, an active Live microphone session is required before screen sharing. The real voice session keeps the host’s audio transport running while the user opens other apps; muting it stops screen sharing. No silent background-audio workaround is used. The extension stores only the latest bounded JPEG in a protected App Group directory, with a short host heartbeat and random capture-session ID. No account token, Live token, history, or capture archive crosses into the group. Frames are removed after delivery and capture stops if the host stops responding. Full iOS broadcast signing requires App Groupgroup.com.tuturuuu.app.mobile.live on both the Runner identifier
com.tuturuuu.app.mobile and extension identifier
com.tuturuuu.app.mobile.LiveScreenBroadcast. Regenerate the Runner App Store
profile in the encrypted Infrastructure vault with that entitlement, and put the
separate extension App Store profile into the mobile-store-beta environment’s
APPLE_BROADCAST_PROVISIONING_PROFILE_B64 encrypted secret. Keep that profile
available for a later build with the extension. Never weaken mobile OIDC to
export a signing key or reuse a Runner profile for the extension.
App Store Connect rejected build 207001 with error 90362 even though the signed
IPA contained NSExtension.NSExtensionAttributes.RPBroadcastProcessMode set to
RPBroadcastProcessModeSampleBuffer. While Apple rejects the extension, the
production TestFlight workflow removes Runner’s extension dependency and embed
phase only in its CI workspace with
scripts/mobile-deployment/omit-broadcast-extension.rb. It builds with
IOS_LIVE_SCREEN_BROADCAST_ENABLED=false so iOS Live does not offer an unusable
screen-sharing control. The workflow verifies the exported IPA has no broadcast
extension before uploading it. Android sharing and local iOS builds retain
their source configuration. To restore iOS TestFlight broadcast, restore its
profile mapping, extension signature checks, and Dart build flag together, then
verify that Apple accepts the signed IPA and that a physical iPhone can stream.
The project membership helper uses Swift and XcodeProj:
Local Cloudflare channel parity
Runbun apps/meet-realtime/tests/mobile-local-check.ts through the shared
resource queue to exercise the real local Worker with Web and Dart clients.
The fixture delays native startup beyond five seconds; its Web ticket lasts
60 seconds to support the normal 45-second renewal cadence. Native tickets
still expire after five seconds and renew every second, with account-bound
presence and multiple-ticket assertions. A five-second reply deadline remains
in place. Failure diagnostics report only statuses and counters, never tickets
or signing secrets. The harness closes both clients, its loopback fixture and
the Worker; it does not validate hosted SFU or physical-device screen capture.