Make Visual Intelligence Understand Your App's Content

June 26, 2026 |

Visual Intelligence is easy to underestimate if you think of it as a camera feature.

The product opportunity is bigger than “identify the thing in front of me.” People also use it on screenshots, messages, social posts, product pages, maps, tickets, menus, and whatever else happens to be on screen. If your app owns useful content related to that visual context, Visual Intelligence can become a search doorway into your app instead of a dead end outside it.

Apple’s WWDC26 session on integrating Visual Intelligence frames the app side around Image Search: model the content you can return, accept a SemanticContentDescriptor, rank results, and route the selected entity back into the app. The Visual Intelligence documentation describes the same goal from the docs side: include app content in the results Visual Intelligence provides.

The snippets below follow that WWDC26 shape. This is beta-season API territory, so verify exact names against the SDK you are building with before pasting. In particular, check the spellings and availability of VisualIntelligence, SemanticContentDescriptor, CVReadOnlyPixelBuffer, IntentValueQuery, GenerateImageFeaturePrintRequest, @UnionValue, and the semanticContentSearch schema in your installed Xcode seed.

Start with the app result, not the camera

Imagine an app called Framebox. It keeps a private catalog of saved artwork, posters, prints, and collections. A user sees a concert poster in Messages, a print on a gallery website, or a photo of a wall display. They invoke Visual Intelligence and ask what apps can do with it.

Framebox should be able to answer:

“This looks like a saved print.”
“This belongs to your Modern Posters collection.”
“Here are related notes or buying details.”
“Open the exact item in the app.”
“Show me more results if the small Visual Intelligence sheet is not enough.”

That starts with App Intents modeling. Visual Intelligence does not want a random dictionary from your database. It wants app content represented as system-facing values.

Model searchable content as AppEntity values

Use an AppEntity for each kind of result Visual Intelligence can display. Keep the entity small and stable. The entity should identify your app content, provide a good display representation, and resolve by identifier later.

import AppIntents
import Foundation

struct ArtworkEntity: AppEntity, Identifiable, Sendable {
    typealias ID = String

    let id: String

    @Property
    var title: String

    @Property
    var artistName: String?

    @Property
    var collectionName: String?

    var thumbnailData: Data

    static let defaultQuery = ArtworkEntityQuery()

    static var typeDisplayRepresentation: TypeDisplayRepresentation {
        "Artwork"
    }

    var displayRepresentation: DisplayRepresentation {
        DisplayRepresentation(
            title: "\(title)",
            subtitle: "\(artistName ?? collectionName ?? "Saved artwork")",
            image: .init(data: thumbnailData)
        )
    }
}

struct ArtworkEntityQuery: EntityQuery {
    @Dependency
    var store: ArtworkStore

    func entities(for identifiers: [ArtworkEntity.ID]) async throws -> [ArtworkEntity] {
        try await store.entities(for: identifiers)
    }
}

The DisplayRepresentation matters more here than it does in many Shortcuts-only integrations. Apple’s session calls out that Visual Intelligence results have limited space for text plus a thumbnail. Do not use a vague title like “Saved item.” Use the identifying text a person needs to trust the result.

Thumbnails are part of performance too. If you provide image data or a URL for display, serve thumbnail-sized assets for normal multi-result layouts. Full-resolution artwork is expensive, and in a two-column result sheet it usually does not make the result more useful. If your app often returns one hero result, test that case separately because the image may be displayed much larger.

Use IntentValueQuery for captured semantic content

For Visual Intelligence Image Search, the query is the interesting layer. Instead of a string, the system gives your app a SemanticContentDescriptor. In the WWDC26 sample, that descriptor includes a pixelBuffer for the captured image.

import AppIntents
import VisualIntelligence

struct FrameboxVisualSearchQuery: IntentValueQuery {
    @Dependency
    var visualIndex: ArtworkVisualIndex

    func values(
        for input: SemanticContentDescriptor
    ) async throws -> [VisualSearchResult] {
        guard let pixelBuffer = input.pixelBuffer else {
            return []
        }

        let matches = try await visualIndex.search(
            matching: pixelBuffer,
            limit: 8
        )

        return matches.map { VisualSearchResult.artwork($0) }
    }
}

Treat this query like a search provider, not a full app launch. It should return quickly, return ranked results, and return an empty array when it does not have a confident answer. The system can display the empty state. Your app does not need to fake a weak match.

Also notice the boundary: SemanticContentDescriptor is not your app’s model. It is the captured context. Convert it into your own search input, search your own index, and return your own entities.

Add image similarity with Vision feature prints

If your content has a visual identity, Vision feature prints are a good first search strategy. Apple’s session uses GenerateImageFeaturePrintRequest to create compact image representations, then compares distances between the captured image and precomputed catalog entries.

The important production move is precomputation. Generate feature prints when content is imported, downloaded, or reindexed. Do not generate every catalog feature print while Visual Intelligence is waiting for results.

import CoreGraphics
import Vision
import VideoToolbox

actor ArtworkVisualIndex {
    struct Entry: Sendable {
        var artwork: ArtworkEntity
        var featurePrint: FeaturePrintObservation
    }

    private var entries: [Entry] = []

    func replaceCatalog(with assets: [ArtworkAsset]) async throws {
        var rebuilt: [Entry] = []
        rebuilt.reserveCapacity(assets.count)

        for asset in assets {
            let searchImage = try await asset.loadSearchImage(maxDimension: 1_024)
            let featurePrint = try await generateFeaturePrint(for: searchImage)

            rebuilt.append(
                Entry(
                    artwork: asset.entity,
                    featurePrint: featurePrint
                )
            )
        }

        entries = rebuilt
    }

    func search(
        matching pixelBuffer: CVReadOnlyPixelBuffer,
        limit: Int = 8,
        maxDistance: Double = 0.85
    ) async throws -> [ArtworkEntity] {
        guard let cgImage = makeCGImage(from: pixelBuffer) else {
            return []
        }

        let queryImage = cgImage.resizedForVisualSearch(maxDimension: 1_024)
        let queryPrint = try await generateFeaturePrint(for: queryImage)

        return try entries.compactMap { entry in
            let distance = try queryPrint.distance(to: entry.featurePrint)

            guard distance <= maxDistance else {
                return nil
            }

            return (artwork: entry.artwork, distance: distance)
        }
        .sorted { $0.distance < $1.distance }
        .prefix(limit)
        .map(\.artwork)
    }

    private func generateFeaturePrint(
        for image: CGImage
    ) async throws -> FeaturePrintObservation {
        let request = GenerateImageFeaturePrintRequest()
        return try await request.perform(on: image)
    }

    private func makeCGImage(
        from pixelBuffer: CVReadOnlyPixelBuffer
    ) -> CGImage? {
        var image: CGImage?

        _ = pixelBuffer.withUnsafeBuffer { buffer in
            VTCreateCGImageFromCVPixelBuffer(
                buffer,
                options: nil,
                imageOut: &image
            )
        }

        return image
    }
}

There are a few policies hiding in that code:

Downsample the query image before producing a feature print, especially for screenshots from large displays.
Pick a result limit that keeps the sheet useful. Five confident results are usually better than twenty weak ones.
Use a maximum distance threshold so bad matches become no matches.
Sort by similarity before mapping to entities.
Make indexing an explicit app operation so you can rebuild after downloads, account changes, or content deletion.

Feature prints are not the only option. For a document app, OCR might matter more. For a retail app, barcode recognition may be the fastest path. For a photo app, face or object classification might be one signal among several. The useful pattern is the same: Visual Intelligence gives you captured context, then your app chooses the search strategy that fits its content.

Return more than one result type when it helps

The first version can return artwork only. Real apps often have better follow-up content.

In Framebox, a captured poster might match an ArtworkEntity, but the same result can also lead to a CollectionEntity. Apple’s session shows @UnionValue for returning more than one entity type from the same IntentValueQuery.

import AppIntents

@UnionValue
enum VisualSearchResult {
    case artwork(ArtworkEntity)
    case collection(CollectionEntity)
}

Then the query can rank primary matches and derive related results:

struct FrameboxVisualSearchQuery: IntentValueQuery {
    @Dependency
    var visualIndex: ArtworkVisualIndex

    @Dependency
    var collections: CollectionStore

    func values(
        for input: SemanticContentDescriptor
    ) async throws -> [VisualSearchResult] {
        guard let pixelBuffer = input.pixelBuffer else {
            return []
        }

        let artwork = try await visualIndex.search(
            matching: pixelBuffer,
            limit: 6
        )

        let relatedCollections = try await collections.collections(
            containingArtworkIDs: artwork.map(\.id),
            limit: 2
        )

        return artwork.map { .artwork($0) }
            + relatedCollections.map { .collection($0) }
    }
}

Use this carefully. A union result should make the experience richer, not noisy. If the captured image is a strong artwork match, show the artwork first. Related collections, events, notes, or saved searches can follow when they are useful and cheap to compute.

Also verify the final @UnionValue shape in your SDK. Other WWDC26 App Intents APIs have had generated case-provider details during the seed cycle, so let Xcode’s interface and docs win over any blog snippet.

Route selected results with OpenIntent

When someone taps a Visual Intelligence result, the app should open directly to that content. Use the same OpenIntent you would use for Spotlight, Siri, Shortcuts, or another App Intents surface.

import AppIntents

struct OpenArtworkIntent: OpenIntent {
    static let title: LocalizedStringResource = "Open Artwork"

    @Parameter(title: "Artwork")
    var target: ArtworkEntity

    @Dependency
    var router: AppRouter

    func perform() async throws -> some IntentResult {
        await router.open(.artwork(id: target.id))
        return .result()
    }
}

struct OpenCollectionIntent: OpenIntent {
    static let title: LocalizedStringResource = "Open Collection"

    @Parameter(title: "Collection")
    var target: CollectionEntity

    @Dependency
    var router: AppRouter

    func perform() async throws -> some IntentResult {
        await router.open(.collection(id: target.id))
        return .result()
    }
}

Keep this lightweight. Apple’s session calls out that perform() runs while the app is coming to the foreground. This is a routing method, not a good place to load every high-resolution image, warm every cache, or rebuild a search index. Move the app to the right route, then let the destination screen hydrate normally.

Add a “more results” path into your app

The Visual Intelligence sheet is intentionally compact. If your app has filters, categories, sorting, or a deeper result set, provide a continuation path into the full app.

Apple’s sample uses the semanticContentSearch schema for this. In that shape, the system provides the same semantic content to an intent that opens your app’s own search UI.

import AppIntents
import VisualIntelligence

@AppIntent(schema: .visualIntelligence.semanticContentSearch)
struct SearchCapturedContentIntent: AppIntent {
    static let title: LocalizedStringResource = "Search in Framebox"
    static let openAppWhenRun = true

    var semanticContent: SemanticContentDescriptor

    @Dependency
    var visualIndex: ArtworkVisualIndex

    @Dependency
    var router: AppRouter

    func perform() async throws -> some IntentResult {
        guard let pixelBuffer = semanticContent.pixelBuffer else {
            await router.open(.visualSearch(results: []))
            return .result()
        }

        let results = try await visualIndex.search(
            matching: pixelBuffer,
            limit: 24
        )

        await router.open(.visualSearch(results: results.map(\.id)))
        return .result()
    }
}

Check the foreground-launch spelling in your SDK. Some App Intents surfaces have been moving toward supportedModes, while Apple’s Visual Intelligence sample still shows openAppWhenRun. The product behavior is the important part: “More results” should land in a real in-app search experience that is already populated from the captured context.

Design for iPhone, iPad, and Mac inputs

The same entity, query, and open-intent architecture can carry across iOS, iPadOS, and macOS, but the input patterns are different.

On iPhone, many Visual Intelligence flows start from the camera. The captured content may be a physical object under uneven lighting, a display photographed at an angle, or a poster in the real world.

On iPad and Mac, screenshots become much more important. A screenshot can be huge, perfectly flat, full of UI chrome, and dense with text. On Mac especially, the pixel buffer can be much larger than the images you tested on iPhone.

That changes the engineering checklist:

Resize before feature-print generation if the pixel buffer is larger than your model needs.
Test screenshots from Retina and non-Retina displays.
Include screen content in fixtures, not only clean catalog images.
Handle letterboxing, browser chrome, dark mode, and app UI around the target object.
Consider OCR, barcode, or text signals when screenshots include useful text next to the image.
Keep the same result ranking rules across platforms so a Mac screenshot and an iPhone camera capture do not feel like two different products.

The goal is not to special-case every device. It is to make the search input boring before your ranking code sees it.

Keep performance visible

Visual Intelligence result quality is partly speed. A great match that arrives too late is not a great integration.

I would put these limits in the first production version:

Precompute catalog feature prints instead of doing catalog-wide image processing in the query.
Cap query image size before Vision work.
Return a small ranked set to Visual Intelligence, then use the in-app continuation for deeper browsing.
Use thumbnail data or thumbnail URLs in display representations.
Return [] for low-confidence matches.
Add signposts around descriptor handling, feature-print generation, ranking, and thumbnail loading.

Also remember that your app appears alongside other available Image Search providers. The system controls provider ordering. You control whether your results are fast, clear, and worth tapping.

Treat captured content as private context

The descriptor can represent something the user just photographed or captured from the screen. Handle it like sensitive user context.

A conservative privacy posture looks like this:

Prefer on-device search when a local index can answer the query.
Do not log raw pixel buffers, screenshots, or derived thumbnails.
If server search is necessary, make that clear in your privacy disclosures and send only what the server needs.
Do not retain captured images unless the user explicitly saves them into your app.
Remove deleted or signed-out content from your visual index.
Keep account boundaries strict. A shared device should not return another account’s private catalog entries.

The system-store side of Visual Intelligence has a different privacy model. Apple’s session also describes Visual Intelligence writing data to stores your app may already read, such as EventKit, Contacts, and HealthKit. If your app observes those stores, respect the framework’s authorization model and present the feature as normal store data, not as secret Visual Intelligence telemetry.

Test below the system sheet

Do not make Visual Intelligence itself your first test harness. Build tests around the layers your app owns, then run end-to-end manual checks on real devices and Mac.

For the visual index, keep a small fixture set:

import Testing

@Test
func visualIndexRanksKnownArtworkFirst() async throws {
    let assets = try await ArtworkAsset.fixtures([
        "night-market-poster",
        "botanical-print",
        "blue-grid-study"
    ])

    let index = ArtworkVisualIndex()
    try await index.replaceCatalog(with: assets)

    let queryImage = try await FixtureImages.requiredPixelBuffer(
        named: "night-market-screenshot"
    )

    let matches = try await index.search(
        matching: queryImage,
        limit: 3
    )

    #expect(matches.first?.id == "night-market-poster")
}

For App Intents, test ordinary pieces with AppIntentsTesting where it fits:

ArtworkEntityQuery.entities(for:) resolves stable identifiers.
DisplayRepresentation has useful titles, subtitles, and thumbnails.
OpenArtworkIntent routes through the same app router as a normal tap.
The “more results” intent opens the full search route.

For the full Visual Intelligence integration, use a small manual matrix:

iPhone camera capture of real-world content.
iPhone screenshot of the same content.
iPad screenshot with split view or Stage Manager chrome.
Mac screenshot at a large display resolution.
No-match input.
Signed-out or permission-restricted state.
Deleted content after the visual index updates.

The most useful failures are ranking failures. Keep the captured image, expected entity, actual top results, distances, image dimensions, and platform in a local test log. Do not store private user captures from production, but do keep your own fixture failures. They will tell you whether the problem is image preprocessing, catalog coverage, threshold tuning, or routing.

The useful mental model

A good Visual Intelligence integration is not an AI trick bolted onto the side of an app. It is a search contract:

AppEntity values define what your app can return.
DisplayRepresentation makes those values understandable in a tiny result view.
IntentValueQuery converts captured semantic content into ranked app entities.
Vision, OCR, barcode, server search, or another strategy finds the matches.
OpenIntent lands the user on the exact content.
The semantic-content search intent lets the user continue into your full app.

That is why this belongs next to your App Intents and search architecture, not in a one-off camera utility. Visual Intelligence can only understand your app’s content if your app has already made that content understandable.