{
  "_published": {
    "source": "docs/measurements/quantization-speed-2026-11-05.json",
    "published": "2026-11-05",
    "publisher": "J4SGON S.L., vorluxai.com",
    "index": "https://vorluxai.com/data/",
    "cleaned": "Copied from our internal repository by scripts/publish_measurements.py. Every number is unchanged. Redacted: nothing. Hostnames (spark-43d5, spark-7180) and the PRIME / spark2 labels are kept: the posts use them.",
    "redactions": {}
  },
  "schema": "vorlux-bench-run/1",
  "slug": "quantization-speed",
  "topic_publish": null,
  "date": "2026-11-05",
  "started_at": "2026-11-05T10:32:46+00:00",
  "finished_at": "2026-11-05T10:34:25+00:00",
  "duration_s": 98.9,
  "machine": {
    "alias": "prime",
    "hostname": "spark-43d5",
    "kernel": "6.17.0-1032-nvidia",
    "arch": "aarch64",
    "gpu": "NVIDIA GB10, 580.173.02",
    "mem_total_gb": 121.7
  },
  "command": "python3 scripts/bench/local_model_benchmark.py --models qwen2.5:7b-instruct-q4_K_M,qwen2.5:7b-instruct-q8_0,qwen2.5:7b-instruct-fp16 --runs 3 --out <tmp>  (via scripts/bench/run_for_topic.py, host prime)",
  "versions": {
    "python": "Python 3.12.3",
    "ollama": "ollama version is 0.30.10",
    "gpu_driver": "NVIDIA GB10, 580.173.02",
    "bench_script": "scripts/bench/local_model_benchmark.py",
    "bench_script_sha256": "be67a3b9dff690a9c9acf3c7902adbe56042e6219a0ce2ffe7a54b6d62f8b011",
    "repo_commit": "d18491c61cb89d17f9b304da8f3015ea721cf84f",
    "bench_dir_dirty": true
  },
  "preflight": {
    "host": "prime",
    "mem_available_gb": 69.5,
    "psi_full_avg10": 0.0,
    "models": {
      "qwen2.5:7b-instruct-q4_K_M": {
        "size_gb": 4.7,
        "loads_on": "prime"
      },
      "qwen2.5:7b-instruct-q8_0": {
        "size_gb": 8.1,
        "loads_on": "prime"
      },
      "qwen2.5:7b-instruct-fp16": {
        "size_gb": 15.2,
        "loads_on": "prime"
      }
    }
  },
  "exit_code": 0,
  "stdout_tail": "{\"run\": 1, \"generation_tok_s\": 43.1, \"prompt_tok_s\": 2014.9, \"time_to_first_token_s\": 6.208, \"wall_s\": 10.855, \"eval_count\": 200}  qwen2.5:7b-instruct-q4_K_M\n{\"run\": 2, \"generation_tok_s\": 42.7, \"prompt_tok_s\": 3207.9, \"time_to_first_token_s\": 0.284, \"wall_s\": 4.967, \"eval_count\": 200}  qwen2.5:7b-instruct-q4_K_M\n{\"run\": 3, \"generation_tok_s\": 42.9, \"prompt_tok_s\": 3233.5, \"time_to_first_token_s\": 0.301, \"wall_s\": 4.961, \"eval_count\": 200}  qwen2.5:7b-instruct-q4_K_M\n{\"run\": 1, \"generation_tok_s\": 28.3, \"prompt_tok_s\": 2071.7, \"time_to_first_token_s\": 5.703, \"wall_s\": 12.78, \"eval_count\": 200}  qwen2.5:7b-instruct-q8_0\n{\"run\": 2, \"generation_tok_s\": 28.2, \"prompt_tok_s\": 2732.1, \"time_to_first_token_s\": 0.322, \"wall_s\": 7.414, \"eval_count\": 200}  qwen2.5:7b-instruct-q8_0\n{\"run\": 3, \"generation_tok_s\": 28.3, \"prompt_tok_s\": 2776.7, \"time_to_first_token_s\": 0.346, \"wall_s\": 7.427, \"eval_count\": 200}  qwen2.5:7b-instruct-q8_0\n{\"run\": 1, \"generation_tok_s\": 16.5, \"prompt_tok_s\": 1577.1, \"time_to_first_token_s\": 9.777, \"wall_s\": 21.91, \"eval_count\": 200}  qwen2.5:7b-instruct-fp16\n{\"run\": 2, \"generation_tok_s\": 16.5, \"prompt_tok_s\": 2981.8, \"time_to_first_token_s\": 0.331, \"wall_s\": 12.488, \"eval_count\": 200}  qwen2.5:7b-instruct-fp16\n{\"run\": 3, \"generation_tok_s\": 16.0, \"prompt_tok_s\": 2997.2, \"time_to_first_token_s\": 0.311, \"wall_s\": 12.849, \"eval_count\": 200}  qwen2.5:7b-instruct-fp16\nwrote /tmp/vorlux-bench-quantization-speed-20261105103246.json\n",
  "results": {
    "measured_at": "2026-11-05T10:32:46+00:00",
    "finished_at": "2026-11-05T10:34:25+00:00",
    "host": "spark-43d5",
    "machine": "NVIDIA DGX Spark, GB10, 128 GB unified memory (PRIME, spark-43d5)",
    "kernel": "6.17.0-1032-nvidia",
    "gpu_driver": "NVIDIA GB10, 580.173.02",
    "ollama_version": "0.30.10",
    "command": "python3 scripts/bench/local_model_benchmark.py --models qwen2.5:7b-instruct-q4_K_M,qwen2.5:7b-instruct-q8_0,qwen2.5:7b-instruct-fp16 --runs 3 --max-tokens 200 --think false --out /tmp/vorlux-bench-quantization-speed-20261105103246.json",
    "method": "same prompt for every model, temperature 0, seed fixed, num_predict = max tokens; model unloaded before run 1 (cold, discarded), run 2 reported; a per-run first line defeats Ollama's prompt cache; think=false for thinking-capable models; generation and prompt tok/s from Ollama's eval_count/eval_duration and prompt_eval_count/prompt_eval_duration; time to first token and wall time measured client-side on the stream; loaded size from /api/ps; one model resident at a time, keep_alive 0 after each",
    "memory_rule": "MemAvailable >= 40 GB and PSI full avg10 < 2 before each load; no model file over 20 GB",
    "prompt": "Read the following note from a small accounting firm and answer in English. First list the key entities (people, organisations, dates, amounts). Then summarise the main decision in one sentence. Finally, write a short paragraph with the three risks you see and what the firm should check before the end of the quarter.\n\nNote from the Monday team meeting, 5 October. Present: Ana (director), Javier (accounts), Lucia (payroll) and Pablo (client service). Ana opened by reminding everyone that the third-quarter close ends on 20 October and that this year 48 clients file a quarterly VAT return, six more than last year. Javier said that 11 clients have still not sent their September invoices and proposed calling them this week instead of waiting for the last day. The team agreed that Pablo will make those calls on Tuesday and Wednesday. Lucia reported that two client companies hired new staff in October and that their social security registrations must be filed before the new employees start work; one is done and the other is waiting for the employee's identity document. The team also discussed the AI assistant the firm has been testing since September to draft routine emails. Pablo said it saves time on simple replies but gets dates and amounts wrong, so the team agreed that no email containing an amount goes out without a human check. Ana asked everyone to note in a shared sheet which tasks they use the assistant for, to prepare the register of AI uses that the European rules require. The next meeting was set for Monday 19 October at 9:00, and Javier will bring a list of the clients who still owe documents.",
    "prompt_words": 278,
    "prompt_note": "each run prepends 'Request <n> of <runs>.' so the prompt cache cannot serve run 2",
    "max_tokens": 200,
    "results": [
      {
        "model": "qwen2.5:7b-instruct-q4_K_M",
        "digest": "845dbda0ea48ed749caafd9e6037047aa19acfcfd82e704d7ca97d631a0b697e",
        "details": {
          "parent_model": "",
          "format": "gguf",
          "family": "qwen2",
          "families": [
            "qwen2"
          ],
          "parameter_size": "7.6B",
          "quantization_level": "Q4_K_M"
        },
        "capabilities": [
          "completion",
          "tools"
        ],
        "think": null,
        "file_size_gb": 4.68,
        "loaded_size_gb": 6.59,
        "loaded_size_vram_gb": 6.59,
        "context_length": 32768,
        "memory_before_load": {
          "mem_total_gb": 121.7,
          "mem_free_gb": 31.3,
          "mem_available_gb": 69.5,
          "psi_full_avg10": 0.0
        },
        "memory_after_runs": {
          "mem_total_gb": 121.7,
          "mem_free_gb": 30.2,
          "mem_available_gb": 75.1,
          "psi_full_avg10": 0.36
        },
        "reported_run": 2,
        "reported": {
          "generation_tok_s": 42.7,
          "prompt_tok_s": 3207.9,
          "time_to_first_token_s": 0.284,
          "wall_s": 4.967,
          "eval_count": 200,
          "prompt_eval_count": 364
        },
        "runs": [
          {
            "time_to_first_token_s": 6.208,
            "wall_s": 10.855,
            "eval_count": 200,
            "eval_duration_s": 4.645,
            "generation_tok_s": 43.1,
            "prompt_eval_count": 364,
            "prompt_eval_duration_s": 0.181,
            "prompt_tok_s": 2014.9,
            "load_duration_s": 6.021,
            "total_duration_s": 10.852,
            "done_reason": "length",
            "thinking": "",
            "output": "### Key Entities:\n- **People:** Ana (director), Javier (accounts), Lucia (payroll), Pablo (client service)\n- **Organisations:** Small accounting firm\n- **Dates:** 5 October (Monday team meeting), 20 October (end of third-quarter close), 19 October (next meeting)\n- **Amounts:** Not explicitly mentioned in the note, but it mentions that emails containing amounts must be checked by a human.\n\n### Main Decision:\nThe main decision is to have Pablo make calls to clients who have not sent their September invoices this week instead of waiting until the last day before the third-quarter close on 20 October.\n\n### Risks and Precautions:\n1. **Client Communication Risk:** There's a risk that some clients might misunderstand or misinterpret the urgency of the call, potentially causing delays in sending necessary documents. The firm should ensure clear communication about the importance of timely submission.\n   \n2. **Accuracy Risk with AI Assistant:** While the",
            "run": 1,
            "psi_full_avg10_after": 0.0
          },
          {
            "time_to_first_token_s": 0.284,
            "wall_s": 4.967,
            "eval_count": 200,
            "eval_duration_s": 4.681,
            "generation_tok_s": 42.7,
            "prompt_eval_count": 364,
            "prompt_eval_duration_s": 0.113,
            "prompt_tok_s": 3207.9,
            "load_duration_s": 0.153,
            "total_duration_s": 4.964,
            "done_reason": "length",
            "thinking": "",
            "output": "### Key Entities:\n- **People:** Ana (director), Javier (accounts), Lucia (payroll), Pablo (client service)\n- **Organisations:** Small accounting firm\n- **Dates:** 5 October (Monday team meeting), 20 October (end of third-quarter close), 19 October (next meeting)\n- **Amounts:** Not explicitly mentioned in the note, but it mentions that emails containing amounts need to be checked by a human.\n\n### Main Decision:\nThe main decision is to call clients who have not sent their September invoices this week instead of waiting until the last day before the third-quarter close on 20 October.\n\n### Risks and Precautions:\n1. **Client Communication Risk:** There's a risk that some clients might be unavailable or unresponsive during the calls, leading to delays in receiving necessary documents. The firm should ensure they have up-to-date contact information for their clients and consider sending reminders via email before making phone calls.\n\n2. **Accuracy",
            "run": 2,
            "psi_full_avg10_after": 0.0
          },
          {
            "time_to_first_token_s": 0.301,
            "wall_s": 4.961,
            "eval_count": 200,
            "eval_duration_s": 4.659,
            "generation_tok_s": 42.9,
            "prompt_eval_count": 364,
            "prompt_eval_duration_s": 0.113,
            "prompt_tok_s": 3233.5,
            "load_duration_s": 0.172,
            "total_duration_s": 4.958,
            "done_reason": "length",
            "thinking": "",
            "output": "### Key Entities:\n- **People:** Ana (director), Javier (accounts), Lucia (payroll), Pablo (client service)\n- **Organisations:** Small accounting firm\n- **Dates:** 5 October (Monday team meeting), 20 October (end of third-quarter close), 19 October (next team meeting)\n- **Amounts:** Not explicitly mentioned in the note, but it mentions that emails containing amounts must be checked by a human.\n\n### Main Decision:\nThe main decision is to call clients who have not sent their September invoices this week instead of waiting until the last day before the third-quarter close on 20 October.\n\n### Risks and Precautions:\n1. **Client Communication Risk:** There's a risk that some clients might be unresponsive or unavailable during the calls, leading to delays in receiving necessary documents. The firm should ensure they have up-to-date contact information for their clients and consider sending reminders via email before making phone calls.\n\n2. **Accuracy",
            "run": 3,
            "psi_full_avg10_after": 0.0
          }
        ]
      },
      {
        "model": "qwen2.5:7b-instruct-q8_0",
        "digest": "2d9500c948416525a044cc51a47a1ef93deb045aebf2f61e8d6b3ac3a3cf2290",
        "details": {
          "parent_model": "",
          "format": "gguf",
          "family": "qwen2",
          "families": [
            "qwen2"
          ],
          "parameter_size": "7.6B",
          "quantization_level": "Q8_0"
        },
        "capabilities": [
          "completion",
          "tools"
        ],
        "think": null,
        "file_size_gb": 8.1,
        "loaded_size_gb": 9.74,
        "loaded_size_vram_gb": 9.74,
        "context_length": 32768,
        "memory_before_load": {
          "mem_total_gb": 121.7,
          "mem_free_gb": 30.2,
          "mem_available_gb": 75.1,
          "psi_full_avg10": 0.36
        },
        "memory_after_runs": {
          "mem_total_gb": 121.7,
          "mem_free_gb": 30.0,
          "mem_available_gb": 74.8,
          "psi_full_avg10": 0.02
        },
        "reported_run": 2,
        "reported": {
          "generation_tok_s": 28.2,
          "prompt_tok_s": 2732.1,
          "time_to_first_token_s": 0.322,
          "wall_s": 7.414,
          "eval_count": 200,
          "prompt_eval_count": 364
        },
        "runs": [
          {
            "time_to_first_token_s": 5.703,
            "wall_s": 12.78,
            "eval_count": 200,
            "eval_duration_s": 7.074,
            "generation_tok_s": 28.3,
            "prompt_eval_count": 364,
            "prompt_eval_duration_s": 0.176,
            "prompt_tok_s": 2071.7,
            "load_duration_s": 5.522,
            "total_duration_s": 12.777,
            "done_reason": "length",
            "thinking": "",
            "output": "### Key Entities:\n- **People:** Ana (director), Javier (accounts), Lucia (payroll), Pablo (client service)\n- **Organisations:** Small accounting firm\n- **Dates:** 5 October (Monday team meeting), 20 October (end of third-quarter close), 19 October (next meeting)\n- **Amounts:** Not explicitly mentioned in the note, but it mentions that emails containing amounts need to be checked by a human.\n\n### Main Decision:\nThe team decided to call clients who have not sent their September invoices this week instead of waiting until the last day before the third-quarter close on 20 October.\n\n### Risks and Checks Before the End of the Quarter:\n\n1. **Client Communication Risk:** There is a risk that some clients might still fail to provide necessary documents or payments by the end of the quarter, which could delay the firm's ability to complete the quarterly VAT returns and other financial tasks on time. The firm should ensure that all calls are",
            "run": 1,
            "psi_full_avg10_after": 0.1
          },
          {
            "time_to_first_token_s": 0.322,
            "wall_s": 7.414,
            "eval_count": 200,
            "eval_duration_s": 7.091,
            "generation_tok_s": 28.2,
            "prompt_eval_count": 364,
            "prompt_eval_duration_s": 0.133,
            "prompt_tok_s": 2732.1,
            "load_duration_s": 0.171,
            "total_duration_s": 7.411,
            "done_reason": "length",
            "thinking": "",
            "output": "### Key Entities:\n- **People:** Ana (director), Javier (accounts), Lucia (payroll), Pablo (client service)\n- **Organisations:** Small accounting firm\n- **Dates:** 5 October (Monday team meeting), 20 October (end of third-quarter close), 19 October (next meeting)\n- **Amounts:** Not explicitly mentioned in the note, but it mentions that emails containing amounts need to be checked by a human.\n\n### Main Decision:\nThe main decision is to have Pablo make calls to clients who have not sent their September invoices this week instead of waiting until the last day before the third-quarter close on 20 October.\n\n### Risks and Checks Before the End of the Quarter:\n\n1. **Client Communication Risk:** There is a risk that some clients might still fail to provide necessary documents or payments by the end of the quarter, which could delay the firm's ability to complete their financial reports accurately. The firm should ensure that all outstanding invoices are",
            "run": 2,
            "psi_full_avg10_after": 0.04
          },
          {
            "time_to_first_token_s": 0.346,
            "wall_s": 7.427,
            "eval_count": 200,
            "eval_duration_s": 7.079,
            "generation_tok_s": 28.3,
            "prompt_eval_count": 364,
            "prompt_eval_duration_s": 0.131,
            "prompt_tok_s": 2776.7,
            "load_duration_s": 0.198,
            "total_duration_s": 7.424,
            "done_reason": "length",
            "thinking": "",
            "output": "### Key Entities:\n- **People:** Ana (director), Javier (accounts), Lucia (payroll), Pablo (client service)\n- **Organisations:** Small accounting firm\n- **Dates:** 5 October (Monday team meeting), 20 October (end of third-quarter close), 19 October (next meeting)\n- **Amounts:** Not explicitly mentioned in the note, but it mentions that emails containing amounts need to be checked by a human.\n\n### Main Decision:\nThe team decided to call clients who have not sent their September invoices this week instead of waiting until the last day before the third-quarter close on 20 October.\n\n### Risks and Checks Before Quarter End:\n\n1. **Client Communication Risk:** There is a risk that some clients might still fail to provide necessary documents even after being called, which could delay the firm's ability to complete the quarterly VAT returns by the deadline. The firm should ensure they have contingency plans in place for such scenarios, such as sending follow",
            "run": 3,
            "psi_full_avg10_after": 0.02
          }
        ]
      },
      {
        "model": "qwen2.5:7b-instruct-fp16",
        "digest": "59805ce4a4046be2d8f63231a78daacd2e66f5dccf1a64d0d138ebeeb26ff16c",
        "details": {
          "parent_model": "/Users/ollama/.ollama/models/blobs/sha256-678e1e0883636c4234928f23fd208337f790456121124a1986cbf1b806baeadb",
          "format": "gguf",
          "family": "qwen2",
          "families": [
            "qwen2"
          ],
          "parameter_size": "7.6B",
          "quantization_level": "F16"
        },
        "capabilities": [
          "completion",
          "tools"
        ],
        "think": null,
        "file_size_gb": 15.24,
        "loaded_size_gb": 16.36,
        "loaded_size_vram_gb": 16.36,
        "context_length": 32768,
        "memory_before_load": {
          "mem_total_gb": 121.7,
          "mem_free_gb": 30.0,
          "mem_available_gb": 74.8,
          "psi_full_avg10": 0.02
        },
        "memory_after_runs": {
          "mem_total_gb": 121.7,
          "mem_free_gb": 34.4,
          "mem_available_gb": 75.9,
          "psi_full_avg10": 7.94
        },
        "reported_run": 2,
        "reported": {
          "generation_tok_s": 16.5,
          "prompt_tok_s": 2981.8,
          "time_to_first_token_s": 0.331,
          "wall_s": 12.488,
          "eval_count": 200,
          "prompt_eval_count": 364
        },
        "runs": [
          {
            "time_to_first_token_s": 9.777,
            "wall_s": 21.91,
            "eval_count": 200,
            "eval_duration_s": 12.131,
            "generation_tok_s": 16.5,
            "prompt_eval_count": 364,
            "prompt_eval_duration_s": 0.231,
            "prompt_tok_s": 1577.1,
            "load_duration_s": 9.541,
            "total_duration_s": 21.907,
            "done_reason": "length",
            "thinking": "",
            "output": "### Key Entities:\n- **People:** Ana (director), Javier (accounts), Lucia (payroll), Pablo (client service)\n- **Organisations:** Small accounting firm\n- **Dates:** 5 October (Monday team meeting), 20 October (end of third-quarter close), 19 October (next meeting)\n- **Amounts:** Not explicitly mentioned in the note, but it mentions that emails containing amounts need to be checked by a human.\n\n### Main Decision:\nThe team decided to call clients who have not sent their September invoices this week instead of waiting until the last day before the third-quarter close on 20 October.\n\n### Risks and Checks Before End of Quarter:\n\n1. **Client Communication Risk:** There is a risk that some clients might still fail to provide necessary documents even after being called, which could delay the firm's ability to complete the quarterly VAT returns by the deadline. The firm should ensure they have contingency plans in place for such scenarios, such as sending",
            "run": 1,
            "psi_full_avg10_after": 0.0
          },
          {
            "time_to_first_token_s": 0.331,
            "wall_s": 12.488,
            "eval_count": 200,
            "eval_duration_s": 12.156,
            "generation_tok_s": 16.5,
            "prompt_eval_count": 364,
            "prompt_eval_duration_s": 0.122,
            "prompt_tok_s": 2981.8,
            "load_duration_s": 0.19,
            "total_duration_s": 12.485,
            "done_reason": "length",
            "thinking": "",
            "output": "### Key Entities:\n- **People:** Ana (director), Javier (accounts), Lucia (payroll), Pablo (client service)\n- **Organisations:** Small accounting firm\n- **Dates:** 5 October (Monday team meeting), 20 October (end of third-quarter close), 19 October (next meeting)\n- **Amounts:** Not explicitly mentioned in the note, but it mentions that emails containing amounts need to be checked by a human.\n\n### Main Decision:\nThe main decision is to have Pablo make calls to clients who have not sent their September invoices this week instead of waiting until the last day before the third-quarter close on 20 October.\n\n### Risks and Checks Before the End of the Quarter:\n\n1. **Client Communication Risk:** There is a risk that some clients might still fail to provide necessary documents or payments by the end of the quarter, which could delay the firm's ability to complete their financial reports accurately. The firm should ensure that all outstanding invoices are",
            "run": 2,
            "psi_full_avg10_after": 0.0
          },
          {
            "time_to_first_token_s": 0.311,
            "wall_s": 12.849,
            "eval_count": 200,
            "eval_duration_s": 12.536,
            "generation_tok_s": 16.0,
            "prompt_eval_count": 364,
            "prompt_eval_duration_s": 0.121,
            "prompt_tok_s": 2997.2,
            "load_duration_s": 0.173,
            "total_duration_s": 12.846,
            "done_reason": "length",
            "thinking": "",
            "output": "### Key Entities:\n- **People:** Ana (director), Javier (accounts), Lucia (payroll), Pablo (client service)\n- **Organisations:** Small accounting firm\n- **Dates:** 5 October (Monday team meeting), 20 October (end of third-quarter close), 19 October (next meeting)\n- **Amounts:** Not explicitly mentioned in the note, but it mentions that emails containing amounts need to be checked by a human.\n\n### Main Decision:\nThe team decided to call clients who have not sent their September invoices this week instead of waiting until the last day before the third-quarter close on 20 October.\n\n### Risks and Checks Before the End of the Quarter:\n\n1. **Client Communication Risk:** There is a risk that some clients might still fail to provide necessary documents even after being called, which could delay the firm's ability to complete the quarterly VAT returns by the deadline. The firm should ensure they have contingency plans in place for such scenarios, such",
            "run": 3,
            "psi_full_avg10_after": 7.94
          }
        ]
      }
    ]
  }
}
