Analyze Images With Foundation Models In SwiftUI

June 29, 2026 |
Foundation Models SwiftUI Swift AI iOS WWDC26

The most useful Foundation Models image feature is not “describe this picture.”

It is taking a messy photo from a real workflow and turning it into app data the user can correct. A field-notes app can extract a plant observation. A receipts app can draft merchant, total, and tax fields. An inventory app can turn a shelf photo into a few candidate items with confidence notes.

WWDC26 session 241 says Foundation Models on iOS 27 adds Vision-backed image understanding. Prompt builders can include image attachments next to text, and those attachments can be created from UIImage, NSImage, CGImage, Core Image types, CoreVideo pixel buffers, and file URLs.

The code below uses the Xcode 27 beta-era API shape Apple showed at WWDC26. Treat exact type and initializer names as seed-sensitive, especially around image attachment construction and the built-in tools. The durable pattern is what matters: prepare a small-enough image, ask for a typed result, surface failures clearly, and only escalate to Private Cloud Compute when the local model is the wrong fit.

Start With The Result Type

Structured output is the difference between a demo and a feature. If the user is adding a receipt, do not ask for a paragraph and parse it later. Ask for a receipt draft.

Here is a result type that works for field notes, receipts, and inventory-style capture. The app can render this as editable fields immediately after generation:

import FoundationModels

@Generable
struct ImageCaptureDraft: Equatable {
    @Guide(description: "A short title for the captured item or scene")
    var title: String

    @Guide(description: "The most likely category, such as receipt, inventory, field note, document, or unknown")
    var category: CaptureCategory

    @Guide(description: "Important facts visible in the image. Use short, user-editable phrases.")
    var facts: [CaptureFact]

    @Guide(description: "Text that appears important enough to save, such as totals, labels, serial numbers, or notes")
    var extractedText: [String]

    @Guide(description: "Barcodes, QR codes, SKU values, or other machine-readable identifiers found in the image")
    var identifiers: [CapturedIdentifier]

    @Guide(description: "Questions the user should answer before saving")
    var needsReview: [String]

    @Guide(description: "A confidence score from 0.0 to 1.0")
    var confidence: Double
}

@Generable
enum CaptureCategory: String, Codable, Equatable {
    case receipt
    case inventory
    case fieldNote
    case document
    case unknown
}

@Generable
struct CaptureFact: Equatable {
    var label: String
    var value: String
}

@Generable
struct CapturedIdentifier: Equatable {
    var kind: String
    var value: String
}

The guides should sound like your UI. “Important facts visible in the image” is more useful than “all information.” It keeps the output compact and reminds the model that the result is a draft, not a forensic report.

Attach The Image Beside The Prompt

The WWDC26 change is that the prompt can carry image attachments directly. You no longer need to run your own vision model first just to get a textual description.

In current beta-shaped code, the flow looks like this:

import FoundationModels
import UIKit

@available(iOS 27.0, *)
struct ImageCaptureAnalyzer {
    func analyze(
        image: UIImage,
        userContext: String,
        mode: CaptureCategory?
    ) async throws -> ImageCaptureDraft {
        let preparedImage = image.preparedForModel(maxLongEdge: 1600)

        let prompt = Prompt {
            """
            Analyze this image for a data-entry workflow.
            Return only information that is visible in the image or strongly implied by visible text.
            Prefer concise editable fields over long prose.
            If a value is uncertain, add a needsReview item instead of guessing.
            """

            if let mode {
                "The user is capturing a \(mode.rawValue)."
            }

            if !userContext.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty {
                "User context: \(userContext)"
            }

            // Exact attachment initializer names may move during the iOS 27 beta.
            // WWDC26 shows prompt builders accepting image attachments from UIImage
            // and other platform image types.
            Attachment(preparedImage)
        }

        let session = LanguageModelSession(
            instructions: """
            You extract structured drafts from user-provided images.
            Do not invent values that are not visible.
            Keep the result suitable for an edit-before-save screen.
            """
        )

        let response = try await session.respond(
            to: prompt,
            generating: ImageCaptureDraft.self,
            options: GenerationOptions(
                temperature: 0.1,
                maximumResponseTokens: 600
            )
        )

        return response.content
    }
}

There are two details worth keeping:

  • The prompt tells the model not to invent values. Image understanding is useful, but the UI still needs to make uncertainty visible.
  • The attachment is part of the prompt next to text. That means your text context, selected capture mode, and image all compete for the same context window.

Foundation Models supports arbitrary image sizes and aspect ratios, but that does not make every original camera photo a good prompt attachment. Larger images consume more tokens and add latency. Resize before you count, count before you submit, and keep headroom for the output.

Downsize Before Counting

Most capture UIs do not need the full 48 MP original. A receipt total, shelf label, or field-note subject often survives a long edge around 1200 to 1800 pixels. Tiny text may need more, so make the limit a product choice, not a magic number hidden in the model call.

import UIKit

extension UIImage {
    func preparedForModel(maxLongEdge: CGFloat) -> UIImage {
        let longestEdge = max(size.width, size.height)
        guard longestEdge > maxLongEdge else {
            return self
        }

        let scale = maxLongEdge / longestEdge
        let newSize = CGSize(
            width: size.width * scale,
            height: size.height * scale
        )

        let renderer = UIGraphicsImageRenderer(size: newSize)
        return renderer.image { _ in
            draw(in: CGRect(origin: .zero, size: newSize))
        }
    }
}

If the user zoomed in on a barcode or a single receipt line, preserve that crop. If they picked a whole shelf, resize the whole image and ask for a small number of likely rows. The model call should match the screen the user expects to review.

The on-device model exposes context size and token counting APIs. Use them. A rough character estimate is worse for images because the image attachment itself has a token cost.

import FoundationModels
import UIKit

@available(iOS 27.0, *)
struct CaptureTokenBudget {
    var inputTokens: Int
    var contextSize: Int
    var reservedOutputTokens: Int

    var fitsOnDevice: Bool {
        inputTokens + reservedOutputTokens <= contextSize
    }
}

@available(iOS 27.0, *)
func budgetForCapturePrompt(
    image: UIImage,
    userContext: String,
    reservedOutputTokens: Int = 700
) async throws -> CaptureTokenBudget {
    let model = SystemLanguageModel.default
    let preparedImage = image.preparedForModel(maxLongEdge: 1600)

    let prompt = Prompt {
        "Analyze this capture image and return a structured draft."
        "User context: \(userContext)"
        Attachment(preparedImage)
    }

    return CaptureTokenBudget(
        inputTokens: try await model.tokenCount(for: prompt),
        contextSize: model.contextSize,
        reservedOutputTokens: reservedOutputTokens
    )
}

The exact overload may differ in your SDK seed, but the policy should not: count the prompt you are actually going to send. Include the image, instructions if your token-counting API accounts for them separately, tool definitions, and enough reserved space for the structured result.

Use OCR And Barcode Tools When They Matter

Image understanding can see text-like content, but receipts and inventory labels deserve specialized help. WWDC26 adds built-in OCRTool and BarcodeReaderTool backed by Vision. Use them when exact text, totals, serial numbers, SKUs, QR codes, or UPCs matter.

Keep those tools optional. A field-note photo of a plant does not need barcode reading. A receipt capture probably does.

import FoundationModels
import Vision

@available(iOS 27.0, *)
enum CaptureToolSet {
    case visualOnly
    case textAndCodes

    var tools: [any Tool] {
        switch self {
        case .visualOnly:
            []
        case .textAndCodes:
            [
                OCRTool(),
                BarcodeReaderTool()
            ]
        }
    }
}

Then make the mode explicit at the call site:

@available(iOS 27.0, *)
func makeCaptureSession(toolSet: CaptureToolSet) -> LanguageModelSession {
    LanguageModelSession(
        tools: toolSet.tools,
        instructions: """
        Extract a structured draft from the user's image.
        Use OCR when exact printed text matters.
        Use barcode reading when a machine-readable code may identify the item.
        Mention uncertain values in needsReview instead of fabricating them.
        """
    )
}

This is not an argument for turning every image prompt into a tool-heavy agent. It is an argument for letting Vision do the exact-reading work when your app is going to save exact fields.

For a receipts app, I would usually enable both tools. For a field-notes app, I would enable them only when the user turns on “read labels” or when the selected template includes text fields.

Put It Behind A SwiftUI Model

The view should not know about token windows, tool sets, or model availability. It should know that the user picked an image, tapped Analyze, and got either a draft or a reviewable error.

import FoundationModels
import Observation
import SwiftUI
import UIKit

@available(iOS 27.0, *)
@MainActor
@Observable
final class ImageCaptureModel {
    var image: UIImage?
    var context = ""
    var selectedCategory: CaptureCategory?
    var draft: ImageCaptureDraft?
    var usageSummary: String?
    var errorMessage: String?
    var isAnalyzing = false

    private let languageModel = SystemLanguageModel.default
    private var task: Task<Void, Never>?

    var canAnalyze: Bool {
        image != nil && !isAnalyzing && isModelAvailable
    }

    var isModelAvailable: Bool {
        if case .available = languageModel.availability {
            return true
        }

        return false
    }

    func analyze() {
        guard let image else { return }

        task?.cancel()
        task = Task {
            await runAnalysis(image: image)
        }
    }

    func cancel() {
        task?.cancel()
        task = nil
        isAnalyzing = false
    }

    private func runAnalysis(image: UIImage) async {
        guard checkAvailability() else { return }

        isAnalyzing = true
        draft = nil
        usageSummary = nil
        errorMessage = nil

        do {
            let preparedImage = image.preparedForModel(maxLongEdge: 1600)
            let prompt = makePrompt(image: preparedImage)
            let inputTokens = try await languageModel.tokenCount(for: prompt)
            let budget = CaptureTokenBudget(
                inputTokens: inputTokens,
                contextSize: languageModel.contextSize,
                reservedOutputTokens: 700
            )

            guard budget.fitsOnDevice else {
                errorMessage = "This image and prompt are too large for on-device analysis. Try cropping the image or use the deeper analysis option if your app supports it."
                isAnalyzing = false
                return
            }

            let toolSet: CaptureToolSet = selectedCategory == .receipt || selectedCategory == .inventory
                ? .textAndCodes
                : .visualOnly

            let session = makeCaptureSession(toolSet: toolSet)
            let response = try await session.respond(
                to: prompt,
                generating: ImageCaptureDraft.self,
                options: GenerationOptions(
                    temperature: 0.1,
                    maximumResponseTokens: budget.reservedOutputTokens
                )
            )

            draft = response.content
            usageSummary = response.usage.captureSummary
        } catch LanguageModelSession.GenerationError.unsupportedLanguageOrLocale(_) {
            errorMessage = "The model cannot analyze this language or locale."
        } catch LanguageModelSession.GenerationError.exceededContextWindowSize(_) {
            errorMessage = "This capture is too large. Crop the image or reduce the extra context."
        } catch is CancellationError {
            errorMessage = nil
        } catch {
            errorMessage = error.localizedDescription
        }

        isAnalyzing = false
    }

    private func checkAvailability() -> Bool {
        switch languageModel.availability {
        case .available:
            return true
        case .unavailable(let reason):
            errorMessage = unavailableMessage(for: reason)
            return false
        }
    }

    private func makePrompt(image: UIImage) -> Prompt {
        Prompt {
            """
            Analyze this image and produce an editable capture draft.
            Use short values.
            Do not infer hidden fields.
            """

            if let selectedCategory {
                "Expected capture type: \(selectedCategory.rawValue)"
            }

            if !context.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty {
                "User note: \(context)"
            }

            Attachment(image)
        }
    }

    private func unavailableMessage(
        for reason: SystemLanguageModel.Availability.UnavailableReason
    ) -> String {
        switch reason {
        case .deviceNotEligible:
            "This device does not support Apple Intelligence."
        case .appleIntelligenceNotEnabled:
            "Apple Intelligence is not enabled."
        case .modelNotReady:
            "The language model is not ready yet."
        @unknown default:
            "Foundation Models is not available right now."
        }
    }
}

@available(iOS 27.0, *)
private extension LanguageModelSession.Usage {
    var captureSummary: String {
        """
        Input: \(input.totalTokenCount) tokens (\(input.cachedTokenCount) cached)
        Output: \(output.totalTokenCount) tokens
        Reasoning: \(output.reasoningTokenCount) tokens
        """
    }
}

That last extension is intentionally about observability, not billing. WWDC26 shows responses and sessions exposing usage information including total and cached input tokens, output tokens, and reasoning tokens. The exact property labels may move, but you should record those numbers during development. They tell you when an innocent image upload became a 6,000-token prompt.

There is also a small bug in many first versions of this feature: they forget that the chosen image can change while the request is running. The model above captures the current UIImage into the task and cancels the previous task before starting a new one. If your real app stores capture records, also attach a local capture ID and discard stale responses whose ID no longer matches the selected draft.

Keep The SwiftUI View Ordinary

The UI can stay small. Let users choose the type when they know it, add a sentence of context, and edit the result after generation.

import PhotosUI
import SwiftUI

@available(iOS 27.0, *)
struct ImageCaptureView: View {
    @State private var model = ImageCaptureModel()
    @State private var selectedPhoto: PhotosPickerItem?

    var body: some View {
        Form {
            Section {
                PhotosPicker(selection: $selectedPhoto, matching: .images) {
                    Label("Choose Image", systemImage: "photo")
                }

                Picker("Type", selection: $model.selectedCategory) {
                    Text("Auto").tag(CaptureCategory?.none)
                    Text("Receipt").tag(CaptureCategory?.some(.receipt))
                    Text("Inventory").tag(CaptureCategory?.some(.inventory))
                    Text("Field Note").tag(CaptureCategory?.some(.fieldNote))
                    Text("Document").tag(CaptureCategory?.some(.document))
                }

                TextField("Context", text: $model.context, axis: .vertical)
                    .lineLimit(2...4)

                Button {
                    model.analyze()
                } label: {
                    Label("Analyze", systemImage: "sparkles")
                }
                .disabled(!model.canAnalyze)
            }

            if model.isAnalyzing {
                ProgressView("Analyzing image")
            }

            if let errorMessage = model.errorMessage {
                Text(errorMessage)
                    .foregroundStyle(.secondary)
            }

            if let draft = model.draft {
                CaptureDraftEditor(draft: draft)
            }

            if let usageSummary = model.usageSummary {
                Section("Usage") {
                    Text(usageSummary)
                        .font(.footnote.monospaced())
                        .foregroundStyle(.secondary)
                }
            }
        }
        .navigationTitle("Capture")
        .task(id: selectedPhoto) {
            guard let data = try? await selectedPhoto?.loadTransferable(type: Data.self),
                  let image = UIImage(data: data)
            else {
                return
            }

            model.image = image
        }
    }
}

struct CaptureDraftEditor: View {
    var draft: ImageCaptureDraft

    var body: some View {
        Section("Draft") {
            LabeledContent("Title", value: draft.title)
            LabeledContent("Category", value: draft.category.rawValue)

            ForEach(Array(draft.facts.enumerated()), id: \.offset) { _, fact in
                LabeledContent(fact.label, value: fact.value)
            }

            if !draft.extractedText.isEmpty {
                Text(draft.extractedText.joined(separator: "\n"))
                    .font(.footnote)
            }

            ForEach(Array(draft.identifiers.enumerated()), id: \.offset) { _, identifier in
                LabeledContent(identifier.kind, value: identifier.value)
            }

            ForEach(draft.needsReview, id: \.self) { question in
                Label(question, systemImage: "exclamationmark.circle")
            }
        }
    }
}

In a production app, CaptureDraftEditor should bind into your local draft model instead of taking an immutable value. The important product shape is edit-before-save. The model gives the user a head start, not a silent database write.

Put PCC Behind A Boundary

Private Cloud Compute is not the default path for this feature. The local model should be the first attempt when the prompt fits, the device is eligible, and the task is a normal capture draft.

PCC becomes useful when the app needs more context or more reasoning. Examples:

  • A user attaches several images and asks for cross-image comparison.
  • The app includes a long existing inventory history in the prompt.
  • A damaged receipt needs a deeper reconstruction pass.
  • A field-notes app asks for a careful narrative summary after the structured facts are extracted.

WWDC26 says PrivateCloudComputeLanguageModel has a 32,000 token context window and supports reasoning levels through ContextOptions(reasoningLevel:). It also has entitlement and eligibility details that you must verify against Apple’s current developer documentation before you make it a product dependency.

I would model the boundary as an explicit user-visible choice:

import FoundationModels

@available(iOS 27.0, *)
enum CaptureAnalysisRoute {
    case onDevice
    case privateCloud(reasoning: ContextOptions.ReasoningLevel)
}

@available(iOS 27.0, *)
struct CaptureAnalysisRouter {
    var localContextSize: Int
    var cloudContextSize = 32_000
    var isPrivateCloudAvailable: Bool
    var cloudQuotaAllowsRequest: Bool
    var isNetworkReachable: Bool

    func route(
        inputTokens: Int,
        reservedOutputTokens: Int,
        wantsDeepReview: Bool
    ) -> CaptureAnalysisRoute? {
        let requiredTokens = inputTokens + reservedOutputTokens

        if !wantsDeepReview, requiredTokens <= localContextSize {
            return .onDevice
        }

        guard isNetworkReachable,
              isPrivateCloudAvailable,
              cloudQuotaAllowsRequest
        else {
            return requiredTokens <= localContextSize ? .onDevice : nil
        }

        guard requiredTokens <= cloudContextSize else {
            return nil
        }

        return .privateCloud(reasoning: wantsDeepReview ? .moderate : .light)
    }
}

In a real implementation, populate isPrivateCloudAvailable from PrivateCloudComputeLanguageModel availability and derive cloudQuotaAllowsRequest from the model’s quota state before you show or run the deeper analysis path. The exact quota labels can move during the beta, so keep that mapping in one small adapter instead of scattering it through views.

Then keep the call site narrow:

import FoundationModels
import Vision

@available(iOS 27.0, *)
func analyzeWithRoute(
    route: CaptureAnalysisRoute,
    prompt: Prompt
) async throws -> ImageCaptureDraft {
    switch route {
    case .onDevice:
        let session = LanguageModelSession(
            model: SystemLanguageModel.default,
            tools: [OCRTool(), BarcodeReaderTool()],
            instructions: "Extract an editable capture draft from the image."
        )

        let response = try await session.respond(
            to: prompt,
            generating: ImageCaptureDraft.self
        )
        return response.content

    case .privateCloud(let reasoning):
        let session = LanguageModelSession(
            model: PrivateCloudComputeLanguageModel(),
            tools: [OCRTool(), BarcodeReaderTool()],
            instructions: "Extract an editable capture draft from the image."
        )

        let response = try await session.respond(
            to: prompt,
            generating: ImageCaptureDraft.self,
            contextOptions: ContextOptions(reasoningLevel: reasoning)
        )
        return response.content
    }
}

This keeps the default capture interaction local and fast. It also keeps your privacy copy honest: the deeper path is a distinct mode with network, quota, availability, and entitlement constraints.

What To Ship Carefully

The rough edges in an image-aware Foundation Models feature are mostly product edges:

  • Show the generated result as a draft, not truth.
  • Resize or crop intentionally before sending the image.
  • Use contextSize and tokenCount(for:) instead of guessing.
  • Enable OCRTool and BarcodeReaderTool when exact text or codes affect saved data.
  • Record usage during development so token and latency regressions are visible.
  • Treat PCC as a fallback for bigger context or reasoning, and verify entitlement and eligibility before relying on it.
  • Gate the whole feature with iOS 27 availability and the model availability state.

That gives you a feature that feels native to a SwiftUI app. The user picks an image, gets an editable draft, fixes the parts the model was unsure about, and saves structured data. The AI part is useful because it is bounded.

Beta Notes

This post is based on Apple’s WWDC26 Foundation Models material, especially session 241, and the iOS 27 beta-era API shape described on June 29, 2026. Verify exact symbol names, availability annotations, built-in tool construction, image attachment initializers, PCC entitlement requirements, and response usage property labels against the Xcode 27 seed you build with.