{
  "_published": {
    "source": "docs/measurements/mistral-small-24b-speed-ctx4k-2026-11-15.json",
    "published": "2026-11-15",
    "publisher": "J4SGON S.L., vorluxai.com",
    "index": "https://vorluxai.com/data/",
    "cleaned": "Copied from our internal repository by scripts/publish_measurements.py. Every number is unchanged. Redacted: private IP addresses -> <prime> / <second-box> / <private-ip> (2). Hostnames (spark-43d5, spark-7180) and the PRIME / spark2 labels are kept: the posts use them.",
    "redactions": {
      "private_ip": 2
    }
  },
  "measured_at": "2026-11-15T08:23:51+00:00",
  "finished_at": "2026-11-15T08:24:39+00:00",
  "host": "spark-7180 via http://<second-box>:11434 from spark-43d5",
  "machine": "NVIDIA DGX Spark, GB10, 128 GB unified memory (spark2, spark-7180)",
  "kernel": "7.0.0-1019-nvidia",
  "gpu_driver": "NVIDIA GB10, 580.178.04",
  "ollama_version": "0.34.0",
  "command": "python3 scripts/bench/local_model_benchmark.py --models mistral-small:24b --runs 3 --max-tokens 200 --think false --ollama http://<second-box>:11434 --ssh spark2 --num-ctx 4096 --out docs/measurements/mistral-small-24b-speed-ctx4k-2026-11-15.json",
  "method": "same prompt for every model, temperature 0, seed fixed, num_predict = max tokens; model unloaded before run 1 (cold, discarded), run 2 reported; a per-run first line defeats Ollama's prompt cache; think=false for thinking-capable models; generation and prompt tok/s from Ollama's eval_count/eval_duration and prompt_eval_count/prompt_eval_duration; time to first token and wall time measured client-side on the stream; loaded size from /api/ps; one model resident at a time, keep_alive 0 after each",
  "memory_rule": "MemAvailable >= 40 GB and PSI full avg10 < 2 before each load; no model file over 20 GB",
  "prompt": "Read the following note from a small accounting firm and answer in English. First list the key entities (people, organisations, dates, amounts). Then summarise the main decision in one sentence. Finally, write a short paragraph with the three risks you see and what the firm should check before the end of the quarter.\n\nNote from the Monday team meeting, 5 October. Present: Ana (director), Javier (accounts), Lucia (payroll) and Pablo (client service). Ana opened by reminding everyone that the third-quarter close ends on 20 October and that this year 48 clients file a quarterly VAT return, six more than last year. Javier said that 11 clients have still not sent their September invoices and proposed calling them this week instead of waiting for the last day. The team agreed that Pablo will make those calls on Tuesday and Wednesday. Lucia reported that two client companies hired new staff in October and that their social security registrations must be filed before the new employees start work; one is done and the other is waiting for the employee's identity document. The team also discussed the AI assistant the firm has been testing since September to draft routine emails. Pablo said it saves time on simple replies but gets dates and amounts wrong, so the team agreed that no email containing an amount goes out without a human check. Ana asked everyone to note in a shared sheet which tasks they use the assistant for, to prepare the register of AI uses that the European rules require. The next meeting was set for Monday 19 October at 9:00, and Javier will bring a list of the clients who still owe documents.",
  "prompt_words": 278,
  "prompt_note": "each run prepends 'Request <n> of <runs>.' so the prompt cache cannot serve run 2",
  "max_tokens": 200,
  "num_ctx": 4096,
  "results": [
    {
      "model": "mistral-small:24b",
      "digest": "cbc05938bb7b197f129393e57058b6f8fc1b9e3a540c8a2a5f96561ca7af92ed",
      "details": {
        "parent_model": "mistral-small:24b",
        "format": "gguf",
        "family": "llama",
        "families": [
          "llama"
        ],
        "parameter_size": "23.6B",
        "quantization_level": "Q4_K_M"
      },
      "capabilities": [
        "completion",
        "tools"
      ],
      "think": null,
      "file_size_gb": 14.33,
      "loaded_size_gb": 19.79,
      "loaded_size_vram_gb": 19.79,
      "context_length": 4096,
      "num_ctx_requested": 4096,
      "memory_before_load": {
        "mem_total_gb": 121.7,
        "mem_free_gb": 42.6,
        "mem_available_gb": 65.4,
        "psi_full_avg10": 0.0
      },
      "memory_while_loaded": {
        "mem_total_gb": 121.7,
        "mem_free_gb": 22.8,
        "mem_available_gb": 45.6,
        "psi_full_avg10": 0.0,
        "other_models_resident": [
          {
            "name": "gemma4-32k:latest",
            "size_gb": 22.7
          }
        ]
      },
      "memory_after_runs": {
        "mem_total_gb": 121.7,
        "mem_free_gb": 42.6,
        "mem_available_gb": 65.4,
        "psi_full_avg10": 0.0
      },
      "reported_run": 2,
      "reported": {
        "generation_tok_s": 14.4,
        "prompt_tok_s": 1509.7,
        "time_to_first_token_s": 0.36,
        "wall_s": 14.286,
        "eval_count": 200,
        "prompt_eval_count": 500
      },
      "runs": [
        {
          "time_to_first_token_s": 3.424,
          "wall_s": 17.287,
          "eval_count": 200,
          "eval_duration_s": 13.861,
          "generation_tok_s": 14.4,
          "prompt_eval_count": 500,
          "prompt_eval_duration_s": 0.482,
          "prompt_tok_s": 1037.1,
          "load_duration_s": 2.934,
          "total_duration_s": 17.284,
          "done_reason": "length",
          "thinking": "",
          "output": "### Key Entities:\n- **People:**\n  - Ana (director)\n  - Javier (accounts)\n  - Lucia (payroll)\n  - Pablo (client service)\n\n- **Organisations:**\n  - The small accounting firm\n\n- **Dates:**\n  - 5 October (meeting date)\n  - 20 October (third-quarter close)\n  - September (start of AI assistant testing)\n  - 19 October (next meeting date)\n  - Tuesday and Wednesday (days for Pablo to make calls)\n\n- **Amounts:**\n  - 48 clients file a quarterly VAT return\n  - 6 more clients than last year\n  - 11 clients have not sent their September invoices\n\n### Main Decision:\nThe team decided that Pablo will call the 11 clients who have not sent their September invoices on Tuesday and Wednesday.\n\n### Risks and Checks:\n1. **Risk of Incomplete Client Data:** There is a risk that some",
          "run": 1,
          "psi_full_avg10_after": 0.0
        },
        {
          "time_to_first_token_s": 0.36,
          "wall_s": 14.286,
          "eval_count": 200,
          "eval_duration_s": 13.923,
          "generation_tok_s": 14.4,
          "prompt_eval_count": 500,
          "prompt_eval_duration_s": 0.331,
          "prompt_tok_s": 1509.7,
          "load_duration_s": 0.001,
          "total_duration_s": 14.281,
          "done_reason": "length",
          "thinking": "",
          "output": "### Key Entities:\n- **People:**\n  - Ana (director)\n  - Javier (accounts)\n  - Lucia (payroll)\n  - Pablo (client service)\n\n- **Organisations:**\n  - The small accounting firm\n\n- **Dates:**\n  - 5 October (meeting date)\n  - 20 October (third-quarter close)\n  - September (start of AI assistant testing)\n  - 19 October (next meeting date)\n  - Tuesday and Wednesday (days for Pablo to make calls)\n\n- **Amounts:**\n  - 48 clients file a quarterly VAT return\n  - 6 more clients than last year\n  - 11 clients have not sent their September invoices\n\n### Main Decision:\nThe team decided that Pablo will call the 11 clients who have not sent their September invoices on Tuesday and Wednesday.\n\n### Risks and Checks:\n1. **Risk of Incomplete Client Data:** There is a risk that some",
          "run": 2,
          "psi_full_avg10_after": 0.0
        },
        {
          "time_to_first_token_s": 0.363,
          "wall_s": 14.353,
          "eval_count": 200,
          "eval_duration_s": 13.987,
          "generation_tok_s": 14.3,
          "prompt_eval_count": 500,
          "prompt_eval_duration_s": 0.333,
          "prompt_tok_s": 1499.3,
          "load_duration_s": 0.001,
          "total_duration_s": 14.348,
          "done_reason": "length",
          "thinking": "",
          "output": "### Key Entities:\n- **People:**\n  - Ana (director)\n  - Javier (accounts)\n  - Lucia (payroll)\n  - Pablo (client service)\n\n- **Organisations:**\n  - The small accounting firm\n  - 48 clients (filing quarterly VAT return)\n  - 11 clients (have not sent September invoices)\n  - Two client companies (hired new staff in October)\n\n- **Dates:**\n  - 5 October (date of the meeting)\n  - 20 October (third-quarter close)\n  - September (start of AI assistant testing)\n  - Tuesday and Wednesday (days Pablo will make calls)\n  - 19 October (next meeting date)\n\n- **Amounts:**\n  - 48 clients (filing quarterly VAT return)\n  - 6 more clients than last year\n  - 11 clients (have not sent September invoices)\n\n### Main Decision:\nThe team decided that Pablo",
          "run": 3,
          "psi_full_avg10_after": 0.0
        }
      ]
    }
  ],
  "conditions_note": "Added by hand after the run, nothing else in this file was edited. Run from PRIME (spark-43d5) against the Ollama 0.34.0 on spark2 (spark-7180); memory, kernel and driver were read on spark2 over ssh. The tag mistral-small:24b was pulled through /api/pull on 2026-11-15 just before the run: /api/tags lists it with digest 8039dd90c1138d772437a0779a33b7349efd5d9cca71edcd26e4dd463f90439d, 14,333,921,662 bytes, GGUF Q4_K_M, 23.6B parameters, base model Mistral Small 2501 (Instruct), native context 32768, general.license apache-2.0 (all from /api/show); About 90 seconds after the pull and before the first run, spark2's own automation rewrote the tag with PARAMETER num_ctx 16384 baked in (same weights blob sha256-102a747c1376...; the tag then lists digest cbc05938bb7b197f129393e57058b6f8fc1b9e3a540c8a2a5f96561ca7af92ed, 14,333,921,268 bytes, modified 09:23:14 local). Every run here used that rewritten tag, which is the `digest` in the results, and sent its own num_ctx, which overrides the baked value. spark2 serves 8 parallel slots (OLLAMA_NUM_PARALLEL=8), so the loaded size covers 8 x num_ctx tokens of context. Also on spark2 during the run: gemma4-32k resident in the same Ollama (22.7 GB per /api/ps), a vLLM engine holding about 19 GB of GPU memory and an idle ComfyUI (nvidia-smi, read at 09:20 local time before the run). Nobody else's model was unloaded. We did not log GPU utilisation during the run, so we cannot prove the other services were idle; the 15 timed generations in the three files of this run stayed between 14.1 and 14.4 tok/s. The file had just been downloaded, so run 1 loaded from the page cache and is not a cold load from disk. 16k and 32k contexts were not loaded: with 8 slots they would have taken memory the shared box needs for its other models."
}
