Force valid JSON from a local LLM with a GBNF grammar in Swift

Small models are good at reading and bad at punctuation. Ask LFM2 2.6B or Qwen3 4B to "answer only with JSON" and you get valid JSON most of the time, plus the occasional missing brace, a trailing comment, a key in the wrong language or a friendly sentence before the object. Retry loops and regex repair hide the problem and cost seconds on a phone.

llama.cpp has a better tool: GBNF grammars. A grammar describes which strings are allowed. At every step, the sampler removes all tokens that could not continue a valid string, and the model picks from what is left. The answer can only take your shape.

The target: a letter, as JSON

The running example reads a paper letter and returns what to do about it:

struct Letter: Codable {
    var sender: String
    var action: String      // "none", "pay", "reply" or "sign"
    var urgency: Int        // 0...3
    var deadline: String    // "YYYY-MM-DD" or ""
    var steps: [String]     // at most 4
}

And a valid answer looks like this:

{"sender": "City Energy", "action": "pay", "urgency": 2, "deadline": "2026-10-30",
 "steps": ["Open your banking app", "Transfer 84.20 EUR", "Use reference 4711"]}

The grammar, rule by rule

root ::= "{" ws "\"sender\":" ws str "," ws "\"action\":" ws action "," ws "\"urgency\":" ws urgency "," ws "\"deadline\":" ws date "," ws "\"steps\":" ws steps ws "}"
action ::= "\"none\"" | "\"pay\"" | "\"reply\"" | "\"sign\""
urgency ::= [0-3]
steps ::= "[" ws "]" | "[" ws str ( ws "," ws str ){0,3} ws "]"
str ::= "\"" chr* "\""
chr ::= [^"\\\x00-\x1f] | "\\" ["\\/bfnrt] | "\\u" [0-9a-fA-F]{4}
date ::= "\"\"" | "\"" [0-9]{4} "-" [0-9]{2} "-" [0-9]{2} "\""
ws ::= [ \t\n]{0,8}

• root is where llama.cpp starts. It spells out every key, in order. The model cannot skip a key, invent one or reorder them. Keys are literals, so the model does not even spend its choices on them: once { is written, the only legal continuation is "sender":.
• action is an enum. Four alternatives, nothing else. This is where grammars pay off most: no "Pay", no "payment", no "pay the bill".
• urgency is a single digit from 0 to 3. A character class is the simplest way to bound a small integer.
• steps is a list with at most four items: one item, then {0,3} more. Bounding repetition matters on a phone: an unbounded list lets a model ramble until maxTokens.
• str and chr are a JSON string: any character except a quote, backslash or control character, or a valid escape. Newlines inside strings are excluded, which keeps values on one line.
• date is either an empty string or "YYYY-MM-DD". The empty alternative gives the model an honest way to say "there is no deadline" instead of making one up.
• ws is whitespace, at most eight characters. An unbounded [ \t\n]* lets some models emit newlines forever; the bound stops that.

Generate it from Swift

Hand-written grammars drift away from the Codable type. Describe the shape once and build the grammar from it:

indirect enum Field {
    case string
    case choice([String])
    case digit(ClosedRange<Int>)       // a single digit, e.g. 0...3
    case date
    case list(Field, max: Int)
}

struct Key {
    let name: String                   // letters and digits only: it doubles as a rule name
    let field: Field
    let hint: String                   // goes into the prompt, not the grammar
}

func gbnf(_ keys: [Key]) -> String {
    var rules: [String] = []

    func rule(_ field: Field, _ name: String) -> String {
        switch field {
        case .string: return "str"
        case .date: return "date"
        case .digit(let r):
            rules.append("\(name) ::= [\(r.lowerBound)-\(r.upperBound)]")
            return name
        case .choice(let values):
            let alternatives = values.map { #""\"\#($0)\"""# }.joined(separator: " | ")
            rules.append("\(name) ::= \(alternatives)")
            return name
        case .list(let item, let max):
            let i = rule(item, name + "-item")
            rules.append(#"\#(name) ::= "[" ws "]" | "[" ws \#(i) ( ws "," ws \#(i) ){0,\#(max - 1)} ws "]""#)
            return name
        }
    }

    let fields = keys.map { k in #""\"\#(k.name)\":" ws "# + rule(k.field, k.name) }
    let root = #"root ::= "{" ws "# + fields.joined(separator: #" "," ws "#) + #" ws "}""#
    let shared = [
        #"str ::= "\"" chr* "\"""#,
        #"chr ::= [^"\\\x00-\x1f] | "\\" ["\\/bfnrt] | "\\u" [0-9a-fA-F]{4}"#,
        #"date ::= "\"\"" | "\"" [0-9]{4} "-" [0-9]{2} "-" [0-9]{2} "\"""#,
        #"ws ::= [ \t\n]{0,8}"#,
    ]
    return ([root] + rules + shared).joined(separator: "\n")
}

let letterKeys = [
    Key(name: "sender", field: .string, hint: "Who sent the letter, short"),
    Key(name: "action", field: .choice(["none", "pay", "reply", "sign"]), hint: "What the reader has to do"),
    Key(name: "urgency", field: .digit(0...3), hint: "0 only information, 3 deadline soon"),
    Key(name: "deadline", field: .date, hint: "Only if the letter names one"),
    Key(name: "steps", field: .list(.string, max: 4), hint: "Tiny steps, each starting with a verb"),
]

Swift's raw strings (#"..."#) keep the backslashes readable: inside them \" is a literal backslash and quote, exactly what GBNF wants, and \#(...) interpolates. gbnf(letterKeys) produces the grammar above. Rule names in GBNF may contain letters, digits and dashes, so keep key names simple or map them to safe names. This sketch assumes choice values without quotes or backslashes; escape them if yours can contain either.

Tell the model what the keys mean

The grammar enforces the shape; it says nothing about meaning. A model that has never heard of urgency will still fill it with a legal digit, just not a useful one. Put a short description of each key into the system prompt:

func describe(_ keys: [Key]) -> String {
    keys.map { "- \($0.name): \($0.hint)" }.joined(separator: "\n")
}

let system = """
    You help people deal with paper mail. Use only facts from the letter.
    Answer only with JSON with these keys:
    \(describe(letterKeys))
    """

One example input with its ideal answer, given as a previous user and assistant turn, teaches a 2B model more than a page of rules. Keep it short, because every token in the prompt counts against the context (see the n_batch guide for what happens when prompts get long).

Put the grammar in the sampler chain

In llama.cpp the grammar is a sampler. Add it first, then the sampler that picks the token. The grammar sets every illegal token's probability to zero; greedy picks the most likely of the rest.

guard let sampler = llama_sampler_chain_init(llama_sampler_chain_default_params()) else { throw LLMError.decodeFailed }
defer { llama_sampler_free(sampler) }

guard let grammar = llama_sampler_init_grammar(vocab, gbnf(letterKeys), "root") else {
    throw LLMError.decodeFailed            // the grammar text does not parse
}
llama_sampler_chain_add(sampler, grammar)
llama_sampler_chain_add(sampler, llama_sampler_init_greedy())

The generation loop stays the same as in the iPhone setup guide. llama_sampler_sample also tells every sampler in the chain which token was chosen, so the grammar advances its state without extra calls. When the closing brace is written, the grammar allows only the end-of-generation token and the loop ends by itself.

If you started from that guide's LLM actor, give generate a grammar: String? parameter and replace its sampler setup with the block above when a grammar is passed. Then decode as usual:

let raw = try await llm.generate(messages, grammar: gbnf(letterKeys), maxTokens: 1024)
let letter = try JSONDecoder().decode(Letter.self, from: Data(raw.utf8))

For extraction, use temperature 0. A grammar combined with random sampling still gives valid JSON, but values that change between runs are rarely what you want from a form.

Caveats

• Shape, not truth. A grammar guarantees that deadline looks like a date. It does not guarantee it is the right date. Validate values that matter and show them to the user for confirmation.
• maxTokens can still cut the answer. If the model writes a very long string and hits the token limit, the JSON is unfinished. Bound lists, keep maxTokens generous for the schema, and treat a decoding error as "try the next tier" rather than a crash.
• Order matters. The model writes keys in the order of the grammar. Put the easy, grounding fields (sender, action) first and derived ones (steps) last; later fields can build on earlier ones.
• Keep the Codable type in sync. Generating the grammar from one description avoids drift. Going one step further, the same description can produce Apple's DynamicGenerationSchema, as shown in the Foundation Models fallback guide.

How fast is it? In our measurements LFM2 2.6B Q4_K_M filled a six-field letter schema in about 6.5 seconds on an M-series Mac. The grammar check adds some work per token, but no retries means it is usually the faster path overall.