{
  "_published": {
    "source": "docs/measurements/local-model-benchmark-2026-10-17.json",
    "published": "2026-11-05",
    "publisher": "J4SGON S.L., vorluxai.com",
    "index": "https://vorluxai.com/data/",
    "cleaned": "Copied from our internal repository by scripts/publish_measurements.py. Every number is unchanged. Redacted: nothing. Hostnames (spark-43d5, spark-7180) and the PRIME / spark2 labels are kept: the posts use them.",
    "redactions": {}
  },
  "measured_at": "2026-10-17T21:00:31+00:00",
  "finished_at": "2026-10-17T21:01:41+00:00",
  "host": "spark-43d5",
  "machine": "NVIDIA DGX Spark, GB10, 128 GB unified memory (PRIME, spark-43d5)",
  "kernel": "6.17.0-1032-nvidia",
  "gpu_driver": "NVIDIA GB10, 580.173.02",
  "ollama_version": "0.30.10",
  "command": "python3 scripts/bench/local_model_benchmark.py --models llama3.1:8b,qwen2.5-coder:7b,gemma4:26b --runs 3 --max-tokens 200 --think false --out docs/measurements/local-model-benchmark-2026-10-17.json",
  "method": "same prompt for every model, temperature 0, seed fixed, num_predict = max tokens; model unloaded before run 1 (cold, discarded), run 2 reported; a per-run first line defeats Ollama's prompt cache; think=false for thinking-capable models; generation and prompt tok/s from Ollama's eval_count/eval_duration and prompt_eval_count/prompt_eval_duration; time to first token and wall time measured client-side on the stream; loaded size from /api/ps; one model resident at a time, keep_alive 0 after each",
  "memory_rule": "MemAvailable >= 40 GB and PSI full avg10 < 2 before each load; no model file over 20 GB",
  "prompt": "Read the following note from a small accounting firm and answer in English. First list the key entities (people, organisations, dates, amounts). Then summarise the main decision in one sentence. Finally, write a short paragraph with the three risks you see and what the firm should check before the end of the quarter.\n\nNote from the Monday team meeting, 5 October. Present: Ana (director), Javier (accounts), Lucia (payroll) and Pablo (client service). Ana opened by reminding everyone that the third-quarter close ends on 20 October and that this year 48 clients file a quarterly VAT return, six more than last year. Javier said that 11 clients have still not sent their September invoices and proposed calling them this week instead of waiting for the last day. The team agreed that Pablo will make those calls on Tuesday and Wednesday. Lucia reported that two client companies hired new staff in October and that their social security registrations must be filed before the new employees start work; one is done and the other is waiting for the employee's identity document. The team also discussed the AI assistant the firm has been testing since September to draft routine emails. Pablo said it saves time on simple replies but gets dates and amounts wrong, so the team agreed that no email containing an amount goes out without a human check. Ana asked everyone to note in a shared sheet which tasks they use the assistant for, to prepare the register of AI uses that the European rules require. The next meeting was set for Monday 19 October at 9:00, and Javier will bring a list of the clients who still owe documents.",
  "prompt_words": 278,
  "prompt_note": "each run prepends 'Request <n> of <runs>.' so the prompt cache cannot serve run 2",
  "max_tokens": 200,
  "results": [
    {
      "model": "llama3.1:8b",
      "digest": "46e0c10c039e019119339687c3c1757cc81b9da49709a3b3924863ba87ca666e",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "llama",
        "families": [
          "llama"
        ],
        "parameter_size": "8.0B",
        "quantization_level": "Q4_K_M"
      },
      "capabilities": [
        "completion",
        "tools"
      ],
      "think": null,
      "file_size_gb": 4.92,
      "loaded_size_gb": 9.21,
      "loaded_size_vram_gb": 9.21,
      "context_length": 32768,
      "memory_before_load": {
        "mem_total_gb": 121.7,
        "mem_free_gb": 31.4,
        "mem_available_gb": 82.9,
        "psi_full_avg10": 0.58
      },
      "memory_after_runs": {
        "mem_total_gb": 121.7,
        "mem_free_gb": 32.2,
        "mem_available_gb": 83.2,
        "psi_full_avg10": 0.47
      },
      "reported_run": 2,
      "reported": {
        "generation_tok_s": 40.2,
        "prompt_tok_s": 3342.4,
        "time_to_first_token_s": 0.365,
        "wall_s": 5.346,
        "eval_count": 200,
        "prompt_eval_count": 340
      },
      "runs": [
        {
          "time_to_first_token_s": 6.422,
          "wall_s": 11.412,
          "eval_count": 200,
          "eval_duration_s": 4.989,
          "generation_tok_s": 40.1,
          "prompt_eval_count": 340,
          "prompt_eval_duration_s": 0.149,
          "prompt_tok_s": 2280.5,
          "load_duration_s": 6.269,
          "total_duration_s": 11.41,
          "done_reason": "length",
          "thinking": "",
          "output": "**Key entities:**\n\n* People:\n\t+ Ana (director)\n\t+ Javier (accounts)\n\t+ Lucia (payroll)\n\t+ Pablo (client service)\n* Organisations:\n\t+ The small accounting firm\n\t+ 48 client companies\n* Dates:\n\t+ Monday, 5 October\n\t+ Tuesday and Wednesday\n\t+ Monday, 19 October\n\t+ 20 October (end of third-quarter close)\n\t+ September\n* Amounts: None mentioned\n\n**Main decision:** The team agreed that Pablo will make calls to 11 clients who have not sent their September invoices this week instead of waiting for the last day.\n\n**Risks and recommendations:**\n\nThe firm should be aware of the following risks:\n\n1. **Inaccurate invoicing**: If the AI assistant continues to get dates and amounts wrong, it may lead to incorrect invoices being sent to clients, which could damage the firm's reputation and relationships with clients.\n2. **Late social security",
          "run": 1,
          "psi_full_avg10_after": 1.59
        },
        {
          "time_to_first_token_s": 0.365,
          "wall_s": 5.346,
          "eval_count": 200,
          "eval_duration_s": 4.98,
          "generation_tok_s": 40.2,
          "prompt_eval_count": 340,
          "prompt_eval_duration_s": 0.102,
          "prompt_tok_s": 3342.4,
          "load_duration_s": 0.232,
          "total_duration_s": 5.344,
          "done_reason": "length",
          "thinking": "",
          "output": "**Key entities:**\n\n* People:\n\t+ Ana (director)\n\t+ Javier (accounts)\n\t+ Lucia (payroll)\n\t+ Pablo (client service)\n* Organisations:\n\t+ The small accounting firm\n\t+ 48 client companies\n* Dates:\n\t+ Monday, 5 October\n\t+ Tuesday and Wednesday\n\t+ Monday, 19 October\n\t+ 20 October (end of third-quarter close)\n\t+ September\n* Amounts: None mentioned\n\n**Main decision:** The team agreed that Pablo will make calls to 11 clients who have not sent their September invoices this week instead of waiting for the last day.\n\n**Risks and checks needed before the end of the quarter:**\n\nThe firm should be aware of three potential risks:\n\n1. **Inaccurate VAT returns**: With six more clients filing a quarterly VAT return than last year, there is a risk that some errors may occur in the calculations or submissions.\n2. **Late social security",
          "run": 2,
          "psi_full_avg10_after": 0.87
        },
        {
          "time_to_first_token_s": 0.342,
          "wall_s": 5.334,
          "eval_count": 200,
          "eval_duration_s": 4.991,
          "generation_tok_s": 40.1,
          "prompt_eval_count": 340,
          "prompt_eval_duration_s": 0.101,
          "prompt_tok_s": 3371.7,
          "load_duration_s": 0.21,
          "total_duration_s": 5.331,
          "done_reason": "length",
          "thinking": "",
          "output": "**Key entities:**\n\n* People:\n\t+ Ana (director)\n\t+ Javier (accounts)\n\t+ Lucia (payroll)\n\t+ Pablo (client service)\n* Organisations:\n\t+ The small accounting firm\n\t+ 48 client companies\n* Dates:\n\t+ Monday, 5 October\n\t+ Tuesday and Wednesday\n\t+ Monday, 19 October\n\t+ 20 October (end of third-quarter close)\n\t+ September\n* Amounts: None mentioned\n\n**Main decision:** The team agreed that Pablo will make calls to 11 clients who have not sent their September invoices this week instead of waiting for the last day.\n\n**Risks and recommendations:**\n\nThe firm should be aware of three potential risks:\n\n1. **Inaccurate invoicing**: If the AI assistant continues to get dates and amounts wrong, it may lead to incorrect invoices being sent to clients, which could damage the firm's reputation and relationships with clients.\n2. **Late social security",
          "run": 3,
          "psi_full_avg10_after": 0.47
        }
      ]
    },
    {
      "model": "qwen2.5-coder:7b",
      "digest": "dae161e27b0e90dd1856c8bb3209201fd6736d8eb66298e75ed87571486f4364",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "qwen2",
        "families": [
          "qwen2"
        ],
        "parameter_size": "7.6B",
        "quantization_level": "Q4_K_M"
      },
      "capabilities": [
        "completion",
        "tools",
        "insert"
      ],
      "think": null,
      "file_size_gb": 4.68,
      "loaded_size_gb": 6.59,
      "loaded_size_vram_gb": 6.59,
      "context_length": 32768,
      "memory_before_load": {
        "mem_total_gb": 121.7,
        "mem_free_gb": 32.2,
        "mem_available_gb": 83.2,
        "psi_full_avg10": 0.47
      },
      "memory_after_runs": {
        "mem_total_gb": 121.7,
        "mem_free_gb": 32.4,
        "mem_available_gb": 83.0,
        "psi_full_avg10": 1.68
      },
      "reported_run": 2,
      "reported": {
        "generation_tok_s": 42.1,
        "prompt_tok_s": 3162.0,
        "time_to_first_token_s": 0.328,
        "wall_s": 5.079,
        "eval_count": 200,
        "prompt_eval_count": 364
      },
      "runs": [
        {
          "time_to_first_token_s": 5.932,
          "wall_s": 10.8,
          "eval_count": 200,
          "eval_duration_s": 4.866,
          "generation_tok_s": 41.1,
          "prompt_eval_count": 364,
          "prompt_eval_duration_s": 0.161,
          "prompt_tok_s": 2263.1,
          "load_duration_s": 5.765,
          "total_duration_s": 10.797,
          "done_reason": "length",
          "thinking": "",
          "output": "### Key Entities:\n- Ana (director)\n- Javier (accounts)\n- Lucia (payroll)\n- Pablo (client service)\n- Third-quarter close date: 20 October\n- Number of clients filing quarterly VAT returns this year: 48 (6 more than last year)\n- Clients who have not sent September invoices: 11\n- New staff hired by client companies in October: 2\n- AI assistant for drafting routine emails\n\n### Main Decision:\nThe team decided to call the 11 clients who have not sent their September invoices this week instead of waiting until the last day.\n\n### Risks and Checks Before End of Quarter:\n1. **Client Invoices**: The firm should ensure that all outstanding invoices are collected by the end of the quarter to avoid penalties or delays in VAT returns.\n2. **Social Security Registrations**: They need to verify that both client companies have completed their social security registrations for the new employees before they start work, as non-compliance",
          "run": 1,
          "psi_full_avg10_after": 4.57
        },
        {
          "time_to_first_token_s": 0.328,
          "wall_s": 5.079,
          "eval_count": 200,
          "eval_duration_s": 4.749,
          "generation_tok_s": 42.1,
          "prompt_eval_count": 364,
          "prompt_eval_duration_s": 0.115,
          "prompt_tok_s": 3162.0,
          "load_duration_s": 0.195,
          "total_duration_s": 5.076,
          "done_reason": "length",
          "thinking": "",
          "output": "### Key Entities:\n- Ana (director)\n- Javier (accounts)\n- Lucia (payroll)\n- Pablo (client service)\n- Third-quarter close date: 20 October\n- Number of clients filing VAT returns this year: 48 (6 more than last year)\n- Clients who have not sent September invoices: 11\n- Client companies hired new staff in October\n- Social security registrations for new employees\n- AI assistant for drafting routine emails\n\n### Main Decision:\nThe team decided to call the clients who have not sent their September invoices this week instead of waiting until the last day.\n\n### Risks and Checks Before End of Quarter:\n1. **Client Invoices**: The firm should ensure that all outstanding invoices are collected by the end of the quarter to avoid penalties or delays in VAT returns.\n2. **Social Security Registrations**: They need to verify that all social security registrations for new employees are completed before their start dates, as non-compliance can lead to fines",
          "run": 2,
          "psi_full_avg10_after": 3.06
        },
        {
          "time_to_first_token_s": 0.337,
          "wall_s": 5.131,
          "eval_count": 200,
          "eval_duration_s": 4.793,
          "generation_tok_s": 41.7,
          "prompt_eval_count": 364,
          "prompt_eval_duration_s": 0.114,
          "prompt_tok_s": 3186.3,
          "load_duration_s": 0.204,
          "total_duration_s": 5.127,
          "done_reason": "length",
          "thinking": "",
          "output": "### Key Entities:\n- People: Ana (director), Javier (accounts), Lucia (payroll), Pablo (client service)\n- Organizations: Small accounting firm\n- Dates: 5 October (meeting date), 20 October (third-quarter close), 19 October (next meeting date)\n- Amounts: Not specified in the note\n\n### Main Decision:\nThe team decided to call clients who have not sent their September invoices by the end of the week instead of waiting until the last day.\n\n### Risks and Checks Before End of Quarter:\n1. **Client Invoices**: The firm should ensure that all outstanding invoices are collected before the third-quarter close on 20 October. This includes following up with clients who have not sent their September invoices.\n   \n2. **Social Security Registrations**: The firm needs to verify that social security registrations for new employees hired by two client companies are completed and submitted before they start work. This is crucial to avoid any delays in payroll processing.\n\n3",
          "run": 3,
          "psi_full_avg10_after": 1.68
        }
      ]
    },
    {
      "model": "gemma4:26b",
      "digest": "5571076f3d70050487b26b341705799e0ab29b808164f90d20d4cf84f699d251",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "gemma4",
        "families": [
          "gemma4"
        ],
        "parameter_size": "25.8B",
        "quantization_level": "Q4_K_M"
      },
      "capabilities": [
        "completion",
        "vision",
        "tools",
        "thinking"
      ],
      "think": false,
      "file_size_gb": 17.99,
      "loaded_size_gb": 17.47,
      "loaded_size_vram_gb": 17.47,
      "context_length": 32768,
      "memory_before_load": {
        "mem_total_gb": 121.7,
        "mem_free_gb": 32.4,
        "mem_available_gb": 82.9,
        "psi_full_avg10": 1.68
      },
      "memory_after_runs": {
        "mem_total_gb": 121.7,
        "mem_free_gb": 32.8,
        "mem_available_gb": 76.6,
        "psi_full_avg10": 0.87
      },
      "reported_run": 2,
      "reported": {
        "generation_tok_s": 62.0,
        "prompt_tok_s": 1864.3,
        "time_to_first_token_s": 0.687,
        "wall_s": 3.916,
        "eval_count": 200,
        "prompt_eval_count": 350
      },
      "runs": [
        {
          "time_to_first_token_s": 12.649,
          "wall_s": 15.907,
          "eval_count": 200,
          "eval_duration_s": 3.255,
          "generation_tok_s": 61.4,
          "prompt_eval_count": 350,
          "prompt_eval_duration_s": 0.446,
          "prompt_tok_s": 784.8,
          "load_duration_s": 12.196,
          "total_duration_s": 15.903,
          "done_reason": "length",
          "thinking": "",
          "output": "**Key Entities**\n\n*   **People:** Ana (Director), Javier (Accounts), Lucia (Payroll), Pablo (Client Service).\n*   **Organisations:** The accounting firm (unnamed), unnamed clients (48 total for VAT returns, 11 with missing invoices, 2 with new staff).\n*   **Dates:** 5 October (meeting date), 20 October (Q3 close deadline), Tuesday and Wednesday (scheduled calls), October (new hires), September (start of AI testing), Monday 19 October at 9:00 (next meeting).\n*   **Amounts:** 48 clients (VAT returns), six more than last year, 11 clients (missing invoices), two client companies (hiring staff).\n\n**Main Decision**\nThe team decided that Pablo will proactively call clients with missing September invoices this week and that all AI-generated emails containing financial figures must undergo a manual human review.\n\n**Risk Assessment**",
          "run": 1,
          "psi_full_avg10_after": 2.39
        },
        {
          "time_to_first_token_s": 0.687,
          "wall_s": 3.916,
          "eval_count": 200,
          "eval_duration_s": 3.227,
          "generation_tok_s": 62.0,
          "prompt_eval_count": 350,
          "prompt_eval_duration_s": 0.188,
          "prompt_tok_s": 1864.3,
          "load_duration_s": 0.43,
          "total_duration_s": 3.912,
          "done_reason": "length",
          "thinking": "",
          "output": "**Key Entities**\n\n*   **People:** Ana (Director), Javier (Accounts), Lucia (Payroll), Pablo (Client Service).\n*   **Organisations:** The accounting firm (unnamed), unnamed clients (48 total for VAT returns, 2 with new staff).\n*   **Dates:** 5 October (meeting date), 20 October (Q3 close deadline), September (start of AI testing), Tuesday and Wednesday (scheduled calls), October (new staff hiring month), 19 October at 9:00 (next meeting).\n*   **Amounts:** 48 clients (VAT returns), 6 more clients than last year, 11 clients (missing invoices).\n\n**Main Decision**\n\nThe team decided that Pablo will proactively call the eleven clients with missing September invoices this Tuesday and Wednesday, and all AI-generated emails containing financial figures must undergo a manual human review.\n\n**Risks and Recommendations**\n\nThe firm faces three primary",
          "run": 2,
          "psi_full_avg10_after": 1.6
        },
        {
          "time_to_first_token_s": 0.698,
          "wall_s": 3.927,
          "eval_count": 200,
          "eval_duration_s": 3.226,
          "generation_tok_s": 62.0,
          "prompt_eval_count": 350,
          "prompt_eval_duration_s": 0.185,
          "prompt_tok_s": 1895.0,
          "load_duration_s": 0.449,
          "total_duration_s": 3.923,
          "done_reason": "length",
          "thinking": "",
          "output": "**Key Entities**\n\n*   **People:** Ana (Director), Javier (Accounts), Lucia (Payroll), Pablo (Client Service).\n*   **Organisations:** The accounting firm (unnamed), unnamed clients (48 total for VAT, 11 with missing invoices, 2 with new staff).\n*   **Dates:** 5 October (meeting date), 2/3 October (Tuesday/Wednesday for calls), 20 October (Q3 close deadline), September (start of AI testing), October (new staff hiring), 19 October at 9:00 (next meeting).\n*   **Amounts:** 48 clients (VAT returns), 6 more clients than last year, 11 missing invoices.\n\n**Main Decision**\n\nThe team decided that Pablo will proactively call clients with missing September invoices this week and that all AI-generated emails containing financial figures must undergo a manual human review to ensure accuracy.\n\n**Risk Assessment**\n\n",
          "run": 3,
          "psi_full_avg10_after": 1.07
        }
      ]
    }
  ]
}
