llama_decode crash with long prompts: GGML_ASSERT n_tokens_all <= n_batch, explained and fixed

The symptom is consistent: everything works in testing, then a user pastes a long document or scans a two-page letter and the app is gone. No Swift error, no exception to catch. The device log shows something like this:

GGML_ASSERT(n_tokens_all <= cparams.n_batch) failed

An assertion in ggml is an abort(). It takes the whole process down, so do/catch does not help. Depending on the llama.cpp version, an oversized batch either triggers this abort or makes llama_decode return an error. Either way the prompt is not processed, and the fix is the same.

The cause: two different limits

A llama.cpp context has several size parameters that are easy to mix up:

llama.cpp caps n_batch at n_ctx, so with a small context you never notice. The trouble starts when you raise n_ctx to 4096 or 8192 so a page of text fits, and keep the default n_batch. Now a prompt of 3000 tokens fits the context, but not the batch.

The usual integration then hands the whole prompt to llama.cpp in one call:

var tokens = tokenize(prompt)        // 3000 tokens
tokens.withUnsafeMutableBufferPointer {
    _ = llama_decode(ctx, llama_batch_get_one($0.baseAddress, Int32($0.count)))   // 3000 > n_batch (2048)
}

llama_batch_get_one wraps all tokens in a single batch. llama_decode checks the batch against n_batch, and 3000 is more than 2048. Short prompts never reach that line, which is why the bug survives testing.

Fix 1: set n_batch to n_ctx

If your prompt has to fit in the context anyway, let the batch be as large as the context:

var cp = llama_context_default_params()
cp.n_ctx = 4096
// The whole prompt is decoded in one call. It has to fit into one batch,
// or llama.cpp aborts with GGML_ASSERT(n_tokens_all <= cparams.n_batch).
cp.n_batch = cp.n_ctx
cp.n_ubatch = min(cp.n_ctx, 512)
guard let ctx = llama_init_from_model(model, cp) else { throw LLMError.loadFailed }

Keep n_ubatch at 512 or lower. It decides the size of the compute buffers, so on a phone a large micro-batch costs real memory, while a large logical batch does not: llama.cpp still processes the prompt 512 tokens at a time, it just accepts all of them in one call. n_ubatch must never be larger than n_batch.

This is the fix to use in almost every app. One line, no change to the generation code.

Fix 2: decode in chunks

If you cannot change the context parameters, for example because the context is created by code you do not own, feed the prompt in pieces no larger than n_batch. With your own batch you are responsible for positions, sequence ids and which token gets logits:

/// Decodes `tokens` in chunks of at most `llama_n_batch(ctx)` tokens.
/// Only the very last token requests logits, which is all sampling needs.
func decodeInChunks(_ ctx: OpaquePointer, _ tokens: [llama_token]) -> Bool {
    let size = Int(llama_n_batch(ctx))
    var batch = llama_batch_init(Int32(size), 0, 1)
    defer { llama_batch_free(batch) }

    var start = 0
    while start < tokens.count {
        let end = min(start + size, tokens.count)
        batch.n_tokens = 0
        for i in start..<end {
            let k = Int(batch.n_tokens)
            batch.token[k] = tokens[i]
            batch.pos[k] = llama_pos(i)              // absolute position in the sequence
            batch.n_seq_id[k] = 1
            batch.seq_id[k]![0] = 0                  // everything belongs to sequence 0
            batch.logits[k] = i == tokens.count - 1 ? 1 : 0
            batch.n_tokens += 1
        }
        if llama_decode(ctx, batch) != 0 { return false }
        start = end
    }
    return true
}

Points to get right:

• Positions are absolute. Chunk two starts at position size, not at 0. Restarting positions corrupts attention without any error message.
• Clear the memory first with llama_memory_clear(llama_get_memory(ctx), true) when you start a new prompt; otherwise positions collide with the previous conversation.
• Logits only for the last token. Requesting logits for every token wastes memory and time. llama_sampler_sample(sampler, ctx, -1) reads the last logits.
• Free the batch. llama_batch_init allocates; pair it with llama_batch_free.

After the prompt, the generation loop can keep using llama_batch_get_one for single tokens. A batch without explicit positions continues after the last position in the context's memory.

Fix 3: check the length before you decode

Neither fix helps when the prompt is longer than the context itself. Tokens beyond n_ctx have nowhere to go, and you also need room for the answer. Count tokens first and fail with a clear error:

enum LLMError: Error {
    case promptTooLong(tokens: Int, limit: Int)
}

let tokens = tokenize(prompt)
let limit = Int(contextLength) - maxTokens        // leave room for the answer
guard tokens.count <= limit else {
    throw LLMError.promptTooLong(tokens: tokens.count, limit: limit)
}

Now a long input is an ordinary Swift error. Your UI can tell the user that the text is too long, trim it, or summarise it in parts. Tokenizing is fast compared to decoding, so the check costs nothing noticeable. The tokenizer function is in the iPhone setup guide.

How long is too long?

As a rough guide, one token is three to four characters of English text and fewer in languages like German or Japanese. A page of a letter is often 400 to 800 tokens, plus your instructions, plus examples. Chat templates and few-shot examples add up quickly, which is another reason to measure instead of guessing. A grammar for structured output, as in the GBNF guide, does not take context space, but the description of the keys in the system prompt does.

Raising n_ctx is not free either: the KV cache grows with the context. On a phone with a 1.6 GB model, 4096 tokens is a reasonable default that fits a page of text comfortably.

Checklist

1. Set n_batch = n_ctx and n_ubatch to 512 or less when you create the context.
2. Clear the memory before each new prompt.
3. Count tokens and throw promptTooLong when tokens + maxTokens exceeds n_ctx.
4. Test with a long input on purpose, on a real device: the official xcframework has no Simulator slice (see Simulator notes).