OnDevice Kit documentation

Requirements

• Xcode 26 and Swift 6. Apps target iOS 17+ or macOS 14+.
• Local models run on iPhone and iPad, in the iOS Simulator (on the CPU) and on the Mac; see Simulator.
• A local model needs memory: a 4-bit 2–3B model (about 1.6 GB) runs on iPhones with 6 GB of RAM or more, which means iPhone 13 Pro and Pro Max, iPhone 14 and newer.
• Apple Intelligence is used automatically on iOS 26 and macOS 26 where the device supports it and the user has turned it on.

Install

Unzip the package next to your project. In Xcode choose File › Add Package Dependencies › Add Local… and select the OnDeviceKit folder. Or add it to your own Package.swift:

dependencies: [
    .package(path: "../OnDeviceKit")
],
targets: [
    .target(name: "MyApp", dependencies: [
        .product(name: "OnDeviceKit", package: "OnDeviceKit")
    ])
]

Products: OnDeviceKit brings everything in with one import OnDeviceKit. OnDeviceLLM, OnDeviceModels and OnDeviceStore can be used on their own. llama.cpp (b11887, MIT) is pulled in by Swift Package Manager as an unchanged build that adds the iOS Simulator to the official release, pinned by URL and checksum.

The fastest way to see it working is the example app in Example/OnDeviceDemo: run it on an iPhone, tap Download, paste any text and tap Make a note. It shows which tier answered and how long it took.

Schema

Schema describes the JSON object you want, key by key, in the order the model writes it. One schema drives every tier: the kit compiles it to a GBNF grammar for llama.cpp and to a DynamicGenerationSchema for Apple’s Foundation Models. Your Codable type decodes the result either way.

struct Letter: Codable, Sendable {
    var sender: String
    var action: String
    var urgency: Int
    var deadline: String
    var steps: [String]

    static let schema = Schema("Letter") {
        Property("sender", .string, "Who sent the letter")
        Property("action", .choice(["none", "pay", "reply"]))
        Property("urgency", .integer(0...3), "0 only information, 3 deadline soon")
        Property("deadline", .date, "Only if the letter names one")
        Property("steps", .list(.string, max: 4))
    }
}

The third argument of Property is a description. It goes into the prompt (the grammar enforces the shape, the description explains the meaning) and into the Apple schema.

Field types

What it compiles to

schema.gbnf is a llama.cpp grammar that allows only JSON of exactly this shape. Because sampling can only pick tokens the grammar allows, the answer always parses: no retries, no “please answer in JSON”. schema.promptDescription is a short plain-text list of the keys for the system prompt. On iOS 26 and macOS 26, the same schema becomes a DynamicGenerationSchema with choices as anyOf, integer ranges as guides, dates as a pattern and lists with maximumElements.

root ::= "{" ws "\"sender\":" ws str "," ws "\"action\":" ws root-action-choice "," ws "\"urgency\":" ws root-urgency-int "," ws "\"deadline\":" ws date "," ws "\"steps\":" ws root-steps-list ws "}"
root-action-choice ::= "\"none\"" | "\"pay\"" | "\"reply\""
root-urgency-int ::= [0-3]
root-steps-list ::= "[" ws "]" | "[" ws str ( ws "," ws str ){0,3} ws "]"

More on how this works, with code you can use on its own: Force valid JSON from a local LLM with a GBNF grammar.

LocalLLM

LocalLLM is an actor that loads a GGUF file once and keeps it in memory. Generation runs off the main thread.

let llm = try LocalLLM(model: url)                                   // loads the GGUF once
let text = try await llm.generate([.system("Be brief."), .user("Hi!")])
for try await piece in llm.stream([.user("Tell me a story")]) { print(piece, terminator: "") }
let letter = try await llm.generate(Letter.self, schema: schema, instructions: "...", input: text)

• Chat templates. Messages are formatted with the chat template stored in the model file, so LFM2, Qwen, Gemma, Llama and Phi all work without prompt fiddling. ChatML is the fallback.
• No n_batch crash. The context is created with n_batch = n_ctx (and n_ubatch at most 512), so long prompts don’t hit llama.cpp’s GGML_ASSERT(n_tokens_all <= cparams.n_batch) abort. Why that happens.
• promptTooLong. If prompt plus maxTokens doesn’t fit the context, LocalLLMError.promptTooLong(tokens:limit:) is thrown before anything runs. tokenCount(_:) lets you check first.
• Clean streaming. stream yields whole characters only: a token that ends in the middle of “ä” or “漢” is held back until it’s complete. Cancel the task to stop early.
• Sampling. generate is greedy by default (temperature: 0), which is what you want for extraction. stream defaults to 0.7 with top-p 0.95.

Options

var options = LocalLLM.Options()
options.contextLength = 4096   // prompt + answer; 4096 fits a page of text comfortably
options.gpuLayers = 99         // layers on the GPU (Metal), 99 = all
options.threads = 4            // default: active cores minus two, at least two
let llm = try LocalLLM(model: url, options: options)

Errors

LocalLLMError: modelNotFound, loadFailed (not enough memory, or not a GGUF file), promptTooLong, decodeFailed, invalidOutput (the answer didn’t decode into your type; carries the raw text). All have readable errorDescriptions.

ModelCache

Loading a 1.6 GB file takes seconds; generating takes less. ModelCache.shared.llm(for: url) keeps one model loaded across calls and frees the old one before loading another, because two rarely fit into an iPhone’s memory. Call await ModelCache.shared.unload() on a memory warning or before deleting the file.

Assistant

Assistant tries the tiers in order and returns the first answer that decodes, together with the tier that produced it.

let assistant = Assistant<Letter>(
    schema: Letter.schema,
    instructions: "Read the letter. Use only facts from it.",
    examples: [Example(input: sampleLetter, output: .sample)],
    order: [.apple, .local, .rules],      // the default
    maxTokens: 1024,
    model: store.readyFile,               // nil until the model is downloaded
    rules: { text in Letter.guess(from: text) }
)

let (letter, tier) = try await assistant.run(scannedText)

• .apple is used when AppleModel.isAvailable. AppleModel.unavailableReason gives a sentence you can show, such as “Apple Intelligence is turned off in Settings.”
• .local is used when the model closure returns a file URL. store.readyFile returns the file only once it is downloaded and verified.
• .rules is your closure: keywords and regular expressions. Return nil to give up.
• If a tier throws (say, the output didn’t decode), the next one is tried. If none answers, the last error is thrown, or AssistantError.noTierAvailable.
• await assistant.currentTier() tells you which tier would answer right now, for example to offer the download.

For the Apple tier on its own and how to detect availability, see Apple Foundation Models with a fallback.

ModelStore

ModelStore downloads a model once, in the background, and keeps it. It is @Observable and lives on the main actor, so SwiftUI views can switch over its state.

@State private var store = ModelStore(.lfm2_2_6B.mirrored(first: URL(string: "https://your.cdn/lfm2.gguf")!))

switch store.state {
case .notDownloaded:        Button("Download (1.6 GB)") { store.download() }
case .downloading(let p):   ProgressView(value: p)
case .verifying:            Text("Checking …")
case .ready:                Text("Ready")
case .failed(let reason):   Text(reason)
}

• Uses a background URLSession: the download continues while the app is in the background or closed, and reattaches on the next launch.
• Tries each URL in order: your mirror first, Hugging Face second. HTTP errors (a 404 also “finishes” a download) and files with the wrong checksum move on to the next URL.
• Verifies SHA-256 off the main thread in 8 MB chunks, and excludes the file from iCloud backup.
• cancel(), await delete() and onDelete, which you can set to { await ModelCache.shared.unload() }.

Forward background events from your app delegate so iOS can finish the download while the app isn’t running:

final class AppDelegate: NSObject, UIApplicationDelegate {
    func application(_ application: UIApplication,
                     handleEventsForBackgroundURLSession identifier: String,
                     completionHandler: @escaping () -> Void) {
        if !ModelStore.handleBackgroundEvents(identifier: identifier, completion: completionHandler) {
            completionHandler()
        }
    }
}

Your own model

// This is how the built-in .lfm2_2_6B spec is defined, with your server added first
let lfm2 = ModelSpec(
    id: "lfm2-2.6b-q4_k_m",
    urls: [
        URL(string: "https://your.cdn/models/lfm2-2.6b-q4_k_m.gguf")!,
        URL(string: "https://huggingface.co/LiquidAI/LFM2-2.6B-GGUF/resolve/main/LFM2-2.6B-Q4_K_M.gguf")!,
    ],
    fileName: "lfm2-2.6b-q4_k_m.gguf",
    bytes: 1_563_668_704,     // ls -l
    sha256: "384bc877b6c37064982f96885bef69e4475919f5969218ed4e3b9399ae0340df",   // shasum -a 256
    license: URL(string: "https://huggingface.co/LiquidAI/LFM2-2.6B/blob/main/LICENSE")
)

Get the exact size with ls -l and the checksum with shasum -a 256. bytes drives the progress bar when a server sends no length. The guide Download a 2 GB model in the background walks through every part of this.

Store

Three small types for the business side of an on-device AI app.

let quota = FreeQuota("letters", free: 2)            // Keychain: survives reinstalling
let daily = DailyPass("forest")                      // once a day for free
@State var unlock = Unlock("com.you.app.unlimited")  // StoreKit 2 non-consumable

if quota.isAvailable || unlock.isUnlocked { analyze(); quota.use() } else { showPaywall = true }
Button("Unlock for \(unlock.price ?? "…")") { Task { try await unlock.buy() } }
Button("Restore purchase") { Task { try? await unlock.restore() } }

• FreeQuota counts uses in the Keychain. Unlike UserDefaults, Keychain items stay when the app is deleted and reinstalled, so the free quota can’t be reset that way. remaining, isAvailable, use(_:), and reset() for testing.
• DailyPass remembers the last day it was used: isAvailableToday, use().
• Unlock wraps one non-consumable: price, buy() (returns false for Ask to Buy and cancel), restore(), isPurchasing. It listens to Transaction.updates, re-reads currentEntitlements (refunds and Family Sharing changes included) and caches the last state so your UI doesn’t flash a paywall at launch.
• StoreError: productNotFound, unverified, nothingToRestore, each with a readable message.

Background and pitfalls: A free trial that survives reinstalling.

Screenshots

Tools/screenshots produces App Store screenshots in every language from the Simulator.

PROJECT=MyApp.xcodeproj SCHEME=MyApp BUNDLE=com.you.myapp ./capture.sh en de ja ar
python3 compose.py            # all locales in captions.json
python3 compose.py de ja ar   # or some of them

• shots.txt: one line per screenshot with the launch arguments that open that screen, for example -demoData -open 0. Your app reads them in DEBUG.
• capture.sh builds the app, boots the Simulator (SIM, default iPhone 17 Pro Max), sets a 9:41 status bar with full battery, reinstalls per language and launches with -AppleLanguages and -AppleLocale. Raw shots land in raw/<locale>/.
• captions.json: title and subtitle per language, plus background and text colours.
• compose.py (needs Pillow) puts the caption on a soft gradient and the screen in a device frame, in 6.9″ 1320×2868 and 6.3″ 1206×2622.
• Captions are rendered by macOS’s own text engine through a tiny Swift tool, so Arabic, Hebrew, Japanese, Korean and Chinese come out right. Long lines shrink until they fit.

{
  "style": { "background": ["#F4F6F5", "#E2EDEE"], "title": "#346270", "subtitle": "#687074" },
  "en": [
    {"title": "Your first line,\nshort and clear", "sub": "One benefit per screenshot"},
    {"title": "All on your iPhone", "sub": "No account, no cloud"}
  ]
}

Choosing a model

Measured on an M-series Mac: LFM2 2.6B fills a six-field letter schema in about 6.5 seconds. The first load takes 20–30 seconds; after that the model stays cached. ModelSpec.lfm2_2_6B is ready to use with size, SHA-256 and license link. Check each license yourself before shipping: you are the one distributing the model.

Simulator

llama.cpp’s official xcframework contains iOS (device) and macOS only. OnDevice Kit links an unchanged build of the same release (b11887) that adds the iOS Simulator, made with llama.cpp’s own build-xcframework.sh and published at theyplay-games/llama-xcframework.

In the Simulator, LocalLLM runs on the CPU by default (gpuLayers = 0), because the Simulator’s Metal lacks most of llama.cpp’s kernels. A 2–3B model takes a minute or two per answer there instead of seconds. To skip the download while developing, copy a GGUF into the app container:

C=$(xcrun simctl get_app_container booted com.you.app data)
mkdir -p "$C/Library/Application Support/Models"
cp lfm2-2.6b-q4_k_m.gguf "$C/Library/Application Support/Models/"

Apple Intelligence in the Simulator needs Apple Intelligence turned on on the Mac.

Integration tests run on the Mac against a real model:

ONDEVICE_MODEL=~/Models/lfm2-2.6b-q4_k_m.gguf swift test

App Review

• Mention in your review notes that the app downloads a language model on request and runs it on the device.
• Keep the download optional or explain it during onboarding, and show its size before downloading.
• Link the model license in your settings screen (store.spec.license).
• Non-consumable purchases need a visible “Restore purchase” button; Unlock.restore() is made for it.

License

The kit’s code is under the OnDevice Kit License included in the package: use it in any number of your own apps, including client work; don’t resell or publish the source. llama.cpp is MIT (THIRD_PARTY.md). Models have their own licenses. Questions: info@theyplay.games.