Build Music-Aware Apps With Music Understanding
Music apps have always wanted richer metadata than “artist, title, duration.”
A DJ app wants tracks that mix well together. A video editor wants cuts that land on beats. A practice app wants the chorus, the bridge, the tempo, and the loud parts. A journaling app might want to remember which recordings were piano-heavy, vocal-heavy, unusually loud, or unusually quiet without asking the user to tag every file by hand.
That is the shape of Apple’s new Music Understanding framework from WWDC26. It gives Swift apps a first-party way to ask musical questions about audio: key, rhythm, structure, pace, instruments, and loudness.
One beta note before the code: the examples below follow the WWDC26 session shape. Treat MusicUnderstandingSession, SessionResult, analyze(), analyze(for:), and the result structs as the useful mental model, but let Xcode autocomplete and the current documentation be the source of truth for exact initializer labels and availability checks.
What Music Understanding is for
Music Understanding is not an audio player, a music catalog, a Shazam replacement, or a magic “understand every sound” API. Think of it as an analysis layer you run against audio your app can access.
The useful output is not one giant answer. It is a set of musical features:
- key: the tonal center or key estimate for a track or a time range
- rhythm: beat and rhythmic information you can line UI up with
- structure: musical sections such as repeated parts or major transitions
- pace: tempo or perceived speed
- instrument: instrument presence over the track
- loudness: overall and time-varying energy
That makes the framework interesting well beyond traditional music apps.
For a DJ or library organization app, you can sort by tempo, group by compatible keys, and identify tracks that build energy in similar ways. You might precompute analysis after import, store a compact summary in SwiftData, and use the result to power smart crates like “124-128 BPM, minor key, vocal-light, high-energy outro.”
For a video editor, rhythm and loudness are editing primitives. A timeline can snap cuts to beats, suggest where a montage should accelerate, or place a title just after a downbeat. Structure is useful when a user drops a full song under a 30-second clip: the app can look for a chorus, a drop, or a quieter section instead of trimming from the beginning every time.
For a podcast, lesson, or music practice app, analysis can make navigation better. A guitarist might loop the same eight bars and gradually slow the pace. A drummer might practice along with beat markers. A podcast editor who uses music beds can spot loudness spikes where speech will get buried.
For ambient recording and audio journaling, the product should be gentler. You are not trying to label a user’s life with false precision. But you can help them find “the piano practice recording from Tuesday” or “the unusually quiet note” by combining instrument and loudness summaries with the user’s own title, date, and location choices.
The mental model
The basic flow is:
- Start with a local audio file URL or an
AVAsset. - Create a
MusicUnderstandingSessionfor that input. - Call
analyze()to get aSessionResult, or callanalyze(for:)when you only need selected result types. - Map the optional result fields into your app’s own small model.
- Store or display that model, not raw framework types, across the rest of your app.
That last point matters during a beta. Keep the Apple API access close to the edge of your feature. Your SwiftUI views, sort descriptors, export logic, and persistence model should not all depend on every provisional result type.
Here is the shape:
import AVFoundation
import MusicUnderstanding
@available(iOS 27.0, *)
func inspectTrack(at fileURL: URL) async throws {
let asset = AVURLAsset(
url: fileURL,
options: [AVURLAssetPreferPreciseDurationAndTimingKey: true]
)
let session = try await MusicUnderstandingSession(asset: asset)
let results = try await session.analyze()
print(
results.key as Any,
results.rhythm as Any,
results.structure as Any,
results.pace as Any,
results.instrumentActivity as Any,
results.loudness as Any
)
}The SessionResult fields are optional. When you use the general analyze() call, the framework analyzes every supported result type. When you use targeted analysis, such as analyze(for: [.loudness]), results you did not request are nil.
Think in timelines, not just labels
Some analysis results are naturally track-level. “This track is around 126 BPM” or “the predominant key is A minor” is useful for sorting a library.
But music changes over time. A song can modulate key. A live recording can speed up. A verse might be sparse, the chorus loud, and the bridge guitar-heavy. If you flatten everything into one string and one number, you throw away the part that makes the framework exciting.
Most app models should separate three ideas:
- summary values for search, sort, and display
- timed values for moments such as beats or tempo changes
- ranged values for sections, instrument presence, and loudness spans
I like starting with tiny app-owned types like these:
import Foundation
struct TrackAnalysisSummary: Sendable, Equatable {
var duration: TimeInterval
var displayKey: String?
var averageBPM: Double?
var sections: [TimelineRange<String>]
var instruments: [TimelineRange<String>]
var loudness: [TimelinePoint<Double>]
}
struct TimelinePoint<Value: Sendable & Equatable>: Sendable, Equatable {
var time: TimeInterval
var value: Value
}
struct TimelineRange<Value: Sendable & Equatable>: Sendable, Equatable, Identifiable {
var id = UUID()
var start: TimeInterval
var end: TimeInterval
var value: Value
var duration: TimeInterval {
max(0, end - start)
}
}These are intentionally boring. They are easy to render in SwiftUI, easy to save, easy to test, and easy to change when the beta APIs settle.
A practical analyzer service
The service below analyzes a local file and maps the results into TrackAnalysisSummary. The important boundary is the adapter: your app asks Music Understanding for a SessionResult, then immediately converts the pieces it needs into your own model.
The useful architecture is the important part: ask for the expensive analysis once, turn it into your own model, and keep the rest of the app insulated.
import AVFoundation
import Foundation
import MusicUnderstanding
@available(iOS 27.0, *)
struct LocalMusicAnalysisService {
func analyze(fileURL: URL) async throws -> TrackAnalysisSummary {
let asset = AVURLAsset(
url: fileURL,
options: [AVURLAssetPreferPreciseDurationAndTimingKey: true]
)
let duration = try await asset.load(.duration).seconds
let session = try await MusicUnderstandingSession(asset: asset)
let results = try await session.analyze()
return TrackAnalysisSummary(
duration: duration,
displayKey: displayKey(from: results.key),
averageBPM: averageBPM(from: results.rhythm),
sections: sections(from: results.structure),
instruments: instrumentRanges(from: results.instrumentActivity),
loudness: loudnessPoints(from: results.loudness)
)
}
private func displayKey(from result: KeyResult?) -> String? {
guard let signature = result?.ranges.first?.value else {
return nil
}
return String(describing: signature)
}
private func averageBPM(from result: RhythmResult?) -> Double? {
result?.beatsPerMinute
}
private func sections(from result: StructureResult?) -> [TimelineRange<String>] {
result?.sections.map { range in
TimelineRange(
start: range.start.seconds,
end: range.end.seconds,
value: "Section"
)
} ?? []
}
private func instrumentRanges(
from result: InstrumentActivityResult?
) -> [TimelineRange<String>] {
result?.ranges.flatMap { instrument, ranges in
ranges.map { range in
TimelineRange(
start: range.start.seconds,
end: range.end.seconds,
value: String(describing: instrument)
)
}
} ?? []
}
private func loudnessPoints(from result: LoudnessResult?) -> [TimelinePoint<Double>] {
result?.momentary.map { value in
TimelinePoint(
time: value.time.seconds,
value: Double(value.value)
)
} ?? []
}
}This adapter intentionally chooses a simple display shape. KeyResult exposes key ranges, RhythmResult includes beats, bars, and optional beatsPerMinute, StructureResult includes sections, segments, and phrases, PaceResult provides ranged pace values, InstrumentActivityResult exposes both ranges and activity, and LoudnessResult includes integrated, momentary, short-term, and peak loudness values. You can map more of those fields as the UI needs them.
Show analysis in SwiftUI
Here is a small view model that analyzes one local file URL and exposes simple loading state to SwiftUI.
import Foundation
import Observation
@MainActor
@Observable
final class TrackAnalysisViewModel {
enum State: Equatable {
case idle
case analyzing
case loaded(TrackAnalysisSummary)
case failed(String)
}
private let fileURL: URL
var state: State = .idle
init(fileURL: URL) {
self.fileURL = fileURL
}
func analyze() {
if case .analyzing = state {
return
}
state = .analyzing
Task {
await runAnalysis()
}
}
private func runAnalysis() async {
do {
if #available(iOS 27.0, *) {
let service = LocalMusicAnalysisService()
let summary = try await service.analyze(fileURL: fileURL)
state = .loaded(summary)
} else {
state = .failed("Music analysis requires iOS 27 or later.")
}
} catch {
state = .failed(error.localizedDescription)
}
}
}And a compact view:
import Foundation
import SwiftUI
struct TrackAnalysisView: View {
@State private var viewModel: TrackAnalysisViewModel
init(fileURL: URL) {
_viewModel = State(initialValue: TrackAnalysisViewModel(fileURL: fileURL))
}
var body: some View {
List {
switch viewModel.state {
case .idle:
Button("Analyze Track") {
viewModel.analyze()
}
case .analyzing:
ProgressView("Analyzing")
case .loaded(let summary):
Section("Summary") {
LabeledContent("Key", value: summary.displayKey ?? "Unknown")
if let bpm = summary.averageBPM {
LabeledContent("Tempo", value: "\(bpm.formatted(.number.precision(.fractionLength(0)))) BPM")
}
}
Section("Structure") {
ForEach(summary.sections) { section in
TimelineRangeRow(range: section)
}
}
Section("Instruments") {
ForEach(summary.instruments) { instrument in
TimelineRangeRow(range: instrument)
}
}
case .failed(let message):
ContentUnavailableView(
"Analysis Failed",
systemImage: "waveform.badge.exclamationmark",
description: Text(message)
)
}
}
.task {
viewModel.analyze()
}
}
}
private struct TimelineRangeRow: View {
var range: TimelineRange<String>
var body: some View {
LabeledContent {
Text("\(range.start.formattedTime)-\(range.end.formattedTime)")
.font(.caption)
.foregroundStyle(.secondary)
} label: {
Text(range.value)
}
}
}
private extension TimeInterval {
var formattedTime: String {
let totalSeconds = Int(self.rounded())
return String(format: "%d:%02d", totalSeconds / 60, totalSeconds % 60)
}
}This is enough for a first useful UI, but a production app should avoid reanalyzing the same file on every appearance. Add a cache key based on the file URL, file size, modification date, media persistent ID, or your own import identifier. Store the app-owned summary and rerun analysis only when the source changes or the user asks for a refresh.
Use loudness as a stream when the UI is live
For library organization, a batch loudness result is fine. For playback UI, recording UI, or a live editor preview, loudness is more useful as a stream of values.
Music Understanding also exposes a loudness stream. The WWDC26 session shows MusicUnderstandingSession.loudnessResults as an AsyncSequence of LoudnessResult values while a loudness analysis is running:
import Foundation
import MusicUnderstanding
@MainActor
@Observable
final class LoudnessMeterModel {
var currentLUFS: Double?
var recentValues: [TimelinePoint<Double>] = []
private var task: Task<Void, Never>?
func startMonitoring(session: MusicUnderstandingSession) {
task?.cancel()
task = Task {
do {
let analysisTask = Task {
try await session.analyze(for: [.loudness])
}
for try await result in await session.loudnessResults {
let newPoints = result.momentary.map { value in
TimelinePoint(
time: value.time.seconds,
value: Double(value.value)
)
}
currentLUFS = newPoints.last?.value
recentValues.append(contentsOf: newPoints)
recentValues = Array(recentValues.suffix(120))
}
_ = try await analysisTask.value
} catch is CancellationError {
return
} catch {
currentLUFS = nil
}
}
}
func stopMonitoring() {
task?.cancel()
task = nil
}
}Do not update the UI for every tiny analysis tick if the stream is high frequency. Throttle for visualization, store lower-resolution points for timelines, and reserve full-resolution data for export or offline processing when users actually need it.
Product patterns that work well
The best Music Understanding features treat analysis as assistive metadata, not as the whole product.
In a DJ app, use key, rhythm, and pace to propose compatible next tracks, but let the DJ override the suggestion. Analysis should rank and suggest; it should not hide tracks from search or treat an estimate as an uneditable truth.
In a video editor, use rhythm and loudness to make the timeline feel magnetic. Snapping to every beat can get annoying, so expose it as a mode: snap to beats, snap to bars, or snap only to major structure changes. Structure can also make auto-trimming less crude. Instead of taking the first 15 seconds of a song, look for a high-energy range that fits the edit.
In a practice app, combine structure with playback controls. Let the user tap the chorus, loop four bars, slow the pace, and keep the beat overlay visible. If the analysis says the bridge starts at 1:42, treat that as a great default, not an uneditable truth.
In audio journaling, be careful with tone and privacy. “This recording includes piano and low loudness” is more respectful than “You were relaxed.” Store local summaries when possible, explain what is being analyzed, and make deletion obvious. Music Understanding should help users find their own recordings, not turn private files into opaque behavioral profiles.
Edge cases to design for
Real audio is messy.
Some files will not have a stable key. Some tracks intentionally drift tempo. Speech-heavy recordings may not produce useful musical structure. Loudness can vary dramatically between a mastered song, a phone voice memo, and a rehearsal room recording. Instrument labels can be uncertain, especially in dense mixes.
Plan for these cases in the model and UI:
- Make every analysis field optional.
- Let users correct or hide analysis metadata.
- Avoid destructive edits based only on analysis output.
- Cache results, because analysis can be expensive.
- Do not block playback while metadata is still being computed.
For import pipelines, I would make analysis a background task. Add the track immediately, show the metadata you already have, then fill in key, BPM, sections, instruments, and loudness when analysis finishes. Users should never feel like the app is broken because a long file is still being understood.
A small storage shape
If you use SwiftData or SQLite, store your app model rather than framework result objects. A simple persisted shape might look like:
struct PersistedTrackAnalysis: Codable, Sendable {
var analyzedAt: Date
var sourceFingerprint: String
var duration: TimeInterval
var displayKey: String?
var averageBPM: Double?
var sections: [PersistedRange]
var instruments: [PersistedRange]
var loudness: [PersistedPoint]
}
struct PersistedRange: Codable, Sendable {
var start: TimeInterval
var end: TimeInterval
var label: String
}
struct PersistedPoint: Codable, Sendable {
var time: TimeInterval
var value: Double
}That gives you room to change the UI without rerunning analysis every time, and room to change the beta-facing adapter without migrating your whole app around framework type names.
Source note
This post is based on Apple’s WWDC26 Music Understanding materials, especially the iOS guide for Music Understanding and the session “Meet the Music Understanding framework.” Because the framework is in the WWDC26 beta cycle, check the current Apple documentation and your installed SDK before treating exact type names as final.
Official sources:
