Best Open-Weight LLMs for Private and Offline AI in 2026

Many organisations want AI that reads their documents without sending them to someone else's servers. The way to do that is to run an open-weight model on infrastructure you control. This guide is for buyers in the GCC, the US, the UK, Europe and Australia who must choose that model.

We compare the models a mid-size organisation can realistically run. Then we cover accuracy on your documents, Arabic quality, licence traps, hardware prices, serving software and the honest cost picture.

We are Webify.AI, and we build private AI systems, so we have an interest here. We have not benchmarked these models ourselves. Every published figure links to its source, and we checked each one on 12 October 2026.

What do offline, on-premise, air-gapped and in-country cloud mean?

They describe where the model runs and what it connects to. Offline means no calls to an outside AI service. On-premise means servers you control. Air-gapped means no network link to the outside. In-country cloud means a provider's data centre inside your jurisdiction. They differ in how much control you keep and how much work you take on.

• Offline: The model runs without calling an external AI API. The server may still reach the internet for updates.
• On-premise: The model runs on servers you own or lease, in your building or your data centre. Your team or a contractor operates it.
• Air-gapped: The system has no live link to outside networks. NIST describes an air gap as two systems with no physical connection, where data moves only manually, "under human control" (NIST).
• In-country cloud: The model runs in a cloud region or sovereign cloud inside your country. Data stays in the jurisdiction, but not in your building.

What do regulators in each region say?

Rules differ by region, and many can be met without a server in your own building.

• GCC: Saudi Arabia's current framework "does not impose a general requirement" to store personal data in the Kingdom. A draft amendment would require it, and consultation closes on 5 November 2026 (Clyde & Co).
• UK: The government's AI Playbook calls running models locally "not recommended for most production services". It also says a private cloud keeps data in "an environment that you own" (GOV.UK).
• Australia: The privacy regulator says deploying "locally or on premises is likely to be more privacy-preserving" (OAIC).
• US and Europe: The trigger is often a sector rule, a client contract or classified work. Check those with your compliance team before you pick a location.

The engineering is the same in all four settings: an open-weight model, private retrieval, guardrails and testing. Only the location and the operating model change.

How should you choose an open-weight model?

Filter in this order: the task, the languages, the hardware you can buy, the licence, and measured quality on your own documents. Many buyers start with general leaderboards. That is backwards, because a model that ranks well overall can still invent facts when it summarises your files.

• Task: Document questions, extraction and summaries need a model that sticks to the source. Coding and multi-step reasoning need a stronger general model.
• Language: English-only models rule themselves out for Arabic work. Phi-4's model card says it "is not intended to support multilingual use" (Microsoft).
• Hardware: Memory decides what fits. IBM's rule of thumb is billions of parameters times 3 GB, plus about 1 GB per 100,000 tokens of context (IBM).
• Licence: Apache 2.0 and MIT are the simplest. Custom licences can add naming rules, revenue limits or banned uses.
• Quality: Published scores narrow the list. A test set built from your own documents makes the final decision.

How far behind the best closed models are open models?

Epoch AI found that since January 2026, the most capable open-weight models trail the best closed models by about four months (Epoch AI). Epoch adds that its estimate "may tend to understate the true gap".

Models small enough for one consumer graphics card trail by six to twelve months (Epoch AI). In our view, the gap matters most for hard reasoning and long agent tasks. It matters least when the model answers from documents you supply, as the accuracy section shows.

Which open-weight models can a mid-size organisation realistically run?

Ten families cover most needs on one to four GPUs. From the US and Europe come IBM Granite 4, OpenAI's gpt-oss, Google's Gemma 4, Mistral, Microsoft's Phi-4 and Meta's Llama 3.3 70B. Alibaba's Qwen comes from China, and Jais 2, ALLaM and Fanar from the Gulf. The table shows licence, size, hardware and best use.

Sizes, context windows and licences come from each model card. See IBM, OpenAI, Google, Qwen, Qwen3.5, Mistral Small 4, Ministral 3, Microsoft, Jais 2, ALLaM and Fanar.

Hardware comes from the model card or from IBM's tested configurations where stated (IBM). Elsewhere we applied IBM's sizing rule at 16-bit precision, so treat those cells as estimates. Compressed (quantised) versions need less memory.

Several of these are "mixture of experts" models, which use only part of their parameters for each word. Google says its 26B Gemma 4 model "runs much faster than its 26B total might suggest" for this reason (Google). You still need memory for all the parameters.

We left out frontier-scale open models on purpose. Mistral Large 3, for example, has 675B parameters, and its card suggests a full node of eight H200s (Mistral AI).

How accurate are open models when they answer from your documents?

Several open models are as faithful as leading closed models. On Vectara's leaderboard, Phi-4 hallucinates in 3.7% of summaries and Llama-3.3-70B in 4.1%. By contrast, gpt-oss-120b does so in 14.2%. Bigger or newer does not mean more faithful, so test before you choose.

Vectara gives each model over 7,700 articles and asks for a summary "using only the facts presented in the document". A detection model, HHEM-2.3, then checks each summary against its source. The board was last updated on 22 September 2026 (Vectara).

Three lessons follow. First, a mid-size open model can beat closed models on this task. Second, larger or newer is not automatically more faithful. Qwen3-8B (4.8%) beats the newer, larger Qwen3.5-27B (12.1%) here. Third, no model wins everywhere, because Qwen3.5 does far better than Qwen3 in Arabic, as the next section shows.

The test has limits. It measures faithfulness, not whether a summary is useful. Vectara's detection model works only in English for now (Vectara). No Gulf-built Arabic model appears on the board, so Arabic grounding needs your own test.

Which open models handle Arabic best?

On QIMMA, a 2026 Arabic leaderboard, Jais-2-70B leads at 65.81, level with Qwen2.5-72B at 65.75. Near 8B parameters, Jais-2-8B, Fanar and ALLaM score 56 to 58. Qwen3-8B scores 39.38 and gpt-oss-20b 32.10. Stanford's HELM Arabic, which tested different models, gives another view.

QIMMA comes from the Technology Innovation Institute in Abu Dhabi and was published on 3 April 2026. It quality-checks over 52,000 test items, mostly native Arabic, before scoring models (QIMMA paper).

Averages hide detail. Fanar scored 0.02 on the poetry task, which pulls its average down. On legal questions it scored 65.47, close to Jais-2-8B at 65.54.

The model generation matters too. Qwen3.5-9B scores 56.28, against 39.38 for the older Qwen3-8B. The authors add that gpt-oss-20b's reasoning mode "cannot compensate for limited Arabic language capability".

What does Stanford's HELM Arabic show?

Stanford's HELM Arabic found that the best open-weight model was Qwen3-235B, with a mean score of 0.786. Models built specifically for Arabic, including ALLaM and JAIS, "underperformed" (Stanford CRFM).

Stanford suggests age explains this, because the newest Arabic-specific family it tested dated from October 2024. Jais 2 and Fanar were not tested. The post's results also refer to version 1.0.0, which had a bug that was later fixed.

So the two boards test different model sets. Neither replaces a test on your own Arabic documents.

Why do short context windows matter?

ALLaM-7B and Fanar-1-9B accept 4,096 tokens, and Jais 2 accepts 8,192 (ALLaM; Fanar; Jais 2). Qwen3.5-27B accepts 262,144 tokens and Granite 4 H-Small 128K (Qwen; IBM).

A short window limits how much retrieved text fits in one prompt. Retrieval must pick fewer, better passages, and long reports need splitting into sections.

Which licence traps should you check before deploying?

Check four things: naming and attribution rules, terms inherited from a base model, revenue limits, and gated downloads. Llama, Fanar, MiniMax and Mistral Medium 3.5 each show at least one. Apache 2.0 and MIT models avoid most of them, but read every model card before you sign.

Llama: the name and the badge

Meta's Llama 4 licence says you must "prominently display 'Built with Llama'" if you distribute the model or a product containing it (Meta). Any model built with Llama or its outputs, then distributed or made available, must start its name with "Llama".

Companies with over 700 million monthly active users must ask Meta for a separate licence. The Llama 3.3 licence carries the same badge and naming rules (Meta on GitHub).

Fanar: an Apache label on a Gemma base

Fanar-1-9B-Instruct is labelled Apache 2.0, but its card says it was continually pretrained from Google's gemma-2-9b (QCRI). Google's Gemma Terms of Use cover Gemma 2 and define "Model Derivatives" to include "works based on Gemma" (Google).

Under those terms, anyone who redistributes a derivative must pass on Google's use restrictions and include a notice. Ask your lawyer whether they apply to Fanar. Gemma 4 is different, because Google releases it under Apache 2.0 (Google).

MiniMax and Mistral Medium 3.5: revenue limits

MiniMax M3 uses the MiniMax Community License, whose main grant covers "non-commercial purposes". Commercial users must display "Built with MiniMax M3" and notify MiniMax (MiniMax). Products earning over $20 million a year need prior written authorisation, and the licence bans "any military purpose".

Mistral Medium 3.5 uses a "Modified MIT License". It grants no rights to a company whose monthly revenue exceeded $20 million in the previous month (Mistral AI). Mistral Small 4 and Ministral 3, by contrast, are Apache 2.0.

Gated downloads

Jais 2 is Apache 2.0, but you must agree to share your contact information before downloading it (Inception). For an air-gapped site, do this on a connected machine, then move checked files across. Keep a licence register for every model, base model and version you run.

What hardware do you need, and what does it cost?

Most mid-size deployments fit one of four tiers. These are a workstation for testing, a two-GPU server, a four-GPU server and an eight-GPU server. Lenovo prices those servers from about $68,000 for two GPUs to $398,000 for eight H200s. Workstation GPU prices rose sharply in 2026.

• Workstation: A 96GB RTX PRO 6000 workstation card was listed at $16,000 on NVIDIA's US store in August 2026, and was out of stock. In June it was $13,250 (Let’s Data Science).
• Compact test box: In October 2025, LMSYS measured NVIDIA's 128GB DGX Spark generating Llama 3.1 70B (FP8) at 2.7 tokens per second. It ran gpt-oss-20b at 49.7 tokens per second, against 215 on an RTX PRO 6000 (LMSYS). LMSYS says large models on Spark are "best suited for prototyping and experimentation".
• Departmental server: Lenovo prices a server with two RTX PRO 6000 cards at $68,010.96 (Lenovo). Its 192GB of GPU memory suits a department running Granite 4 H-Small, gpt-oss-120b or a 70B model.
• Four-GPU server: Lenovo prices a server with four 48GB L40S cards at $113,186.50. That is the same 192GB in total as the two-card server, at a higher price. IBM lists Llama 3.3 70B on four L40S cards.
• Largest models: An eight-H200 server costs $397,801.60, and an eight-B200 server costs $550,475.10. These suit frontier-scale open models or many users at once.

Lenovo's figures are its usual sale prices as of 15 June 2026, in a paper last updated on 24 July 2026. Treat all GPU prices as fast-moving and get a current quote.

Vendor software can add cost. NVIDIA AI Enterprise lists at $4,500 per GPU for one year (NVIDIA). A five-year term is included with H100 and H200 NVL cards. Open-source serving avoids the fee, but you then carry the support work.

Which serving software should you use?

Use vLLM for servers that many people share. Use llama.cpp or Ollama on a laptop or a single small machine. Avoid starting new projects on Hugging Face's TGI, which is now in maintenance mode. All three recommended tools are open source, with vLLM under Apache 2.0 and the other two under MIT.

• vLLM: Apache 2.0. It batches many users' requests together and manages memory with a method called PagedAttention. It offers an OpenAI-compatible API and runs on NVIDIA and AMD GPUs and on CPUs (vLLM).
• llama.cpp: MIT. It runs compressed models from 1.5-bit to 8-bit. It can split a model between CPU and GPU when graphics memory is short (llama.cpp).
• Ollama: MIT. It installs on macOS, Windows, Linux and Docker, which suits pilots and desktops (Ollama).
• TGI: Hugging Face says it "is now in maintenance mode" and will accept only minor fixes (Hugging Face).

For an air-gapped site, download and check models on a connected machine, then carry them in. Setting HF_HUB_OFFLINE=1 stops Hugging Face tools from calling the Hub (Hugging Face). IBM signs its Granite 4 checkpoints, so you can verify where they came from (IBM).

Should you use RAG or fine-tuning?

Start with retrieval-augmented generation (RAG) for knowledge, such as policies, contracts and changing facts. Use fine-tuning for behaviour, such as a fixed output format, a house style or a narrow extraction task. Research finds RAG beats unsupervised fine-tuning for adding facts, and RAG keeps documents easy to update.

A study by Ovadia and colleagues found that "RAG consistently outperforms" unsupervised fine-tuning. That held for knowledge the model had seen in training and for entirely new knowledge. The authors add that models "struggle to learn new factual information" this way (arXiv).

Fine-tuning hardware is modest. Unsloth lists minimum memory for QLoRA tuning of 6GB for an 8B model and 41GB for a 70B model (Unsloth). OpenAI says gpt-oss-120b can be fine-tuned on a single H100 node (OpenAI).

The real cost is data. You need clean examples and a test set that proves the tuned model is better. Check the licence too, because a distributed Llama fine-tune must carry the Llama name.

Is running your own model cheaper than using an API?

Usually not. Hosted gpt-oss-120b costs $0.04 to $0.39 per million tokens (blended) across 17 providers. Lenovo's best case for its own servers is $0.13 to $0.16 per million output tokens, at full load and with no staff costs. Offline AI sells on control and compliance, not savings.

Artificial Analysis lists gpt-oss-120b on 17 API providers, from $0.04 per million tokens at CoreWeave to $0.39 at Cerebras (Artificial Analysis). These blended prices use a 7:2:1 mix of cached input, fresh input and output tokens. The cheapest output tokens alone cost $0.17 per million.

Lenovo's paper puts on-premise cost at $0.159 per million output tokens for a 70B Llama model on eight H200s (Lenovo). It reaches $0.13 for DeepSeek R1 on eight B300s. Both figures assume peak benchmark speed, a five-year life and zero resale value.

They include maintenance, power and colocation, but not staff. Real servers rarely run flat out around the clock, so real cost per token is higher.

An academic break-even study reaches a similar view. It finds self-hosting viable mainly at 50 million tokens a month or more, or under "strict data residency mandates" (arXiv). It also leaves out staffing.

Self-hosting can still pay in three cases: steady high volume, a small model doing one narrow job, or replacing rented cloud GPUs. Lenovo estimates an eight-H200 server breaks even in about 5.2 months against on-demand Azure rental, used around the clock. Otherwise, buy offline AI for control, residency and a predictable bill.

How do you test models on your own documents?

Build a test set from your real documents and run three or four shortlisted models on it. Score faithfulness, correctness, refusals and speed on the hardware you plan to buy. Published leaderboards narrow the list. Only your own test shows which model works on your files, in your languages.

• Collect questions: Gather a few hundred real questions from staff, each with its answer and the source passage.
• Add traps: Include questions your documents cannot answer. The right response is a refusal, not a guess.
• Mirror your languages: If staff write in Arabic and English, test both, including dialect and mixed-language questions.
• Score faithfulness: Vectara's HHEM-2.1-Open is a small Apache 2.0 model that checks whether an answer matches its source. It works in English only, so Arabic answers need bilingual reviewers (Hugging Face).
• Measure speed: Test with a realistic number of users at once, on the target hardware. Single-user speed is misleading.
• Repeat on refresh: Open models improve every few months, so rerun the same test before you swap models.

Keep the test set private and stable. If you change it every time, you cannot compare one model with the next.

How can Webify.AI help?

Webify.AI is an AI and software company headquartered in Ahmedabad, India, and an IBM Partner. We build AI agents and private AI deployed air-gapped, on-premise or in an in-country cloud. We offer fixed-price pilots, so you can test a model on your own documents before committing further.

Our private AI page explains the three deployment options. Our offline LLM development and RAG development services cover the build itself.

To see the idea working, the offline demo loads a small model into your browser. You can then switch off Wi-Fi and watch it keep answering.

Two honest notes. We do not hold security certifications. We have not benchmarked the models in this guide ourselves, so every figure above comes from the linked sources.

Sources

• NIST: Glossary entry for air gap, checked 12 October 2026.
• Clyde & Co: SDAIA publishes draft amendments to the PDPL, 8 October 2026, checked 12 October 2026.
• GOV.UK: AI Playbook for the UK Government, checked 12 October 2026.
• OAIC: Guidance on privacy and the use of commercially available AI products, checked 12 October 2026.
• Epoch AI: Open models lag state-of-the-art closed models by 4 months, 29 May 2026, checked 12 October 2026.
• Epoch AI: Frontier AI performance becomes accessible on consumer hardware within a year, 15 August 2025, checked 12 October 2026.
• Vectara: Hallucination leaderboard, updated 22 September 2026, checked 12 October 2026.
• Vectara: HHEM-2.1-Open model card, checked 12 October 2026.
• Technology Innovation Institute: QIMMA paper, arXiv 2604.03395, 3 April 2026, checked 12 October 2026.
• Stanford CRFM: HELM Arabic, 18 December 2025, errata 3 June 2026, checked 12 October 2026.
• IBM: Granite 4.0 announcement, 2 October 2025, checked 12 October 2026.
• IBM: Granite-4.0-H-Small model card, checked 12 October 2026.
• IBM: watsonx.ai foundation models and GPU requirements, checked 12 October 2026.
• OpenAI: gpt-oss-120b model card, checked 12 October 2026.
• Google: Gemma 4 31B model card, checked 12 October 2026.
• Google: Gemma 4 licence, checked 12 October 2026.
• Google: Gemma Terms of Use, last modified 1 April 2026, checked 12 October 2026.
• Qwen: Qwen3 release blog, checked 12 October 2026.
• Qwen: Qwen3.5-27B model card, checked 12 October 2026.
• Mistral AI: Mistral Small 4 model card, checked 12 October 2026.
• Mistral AI: Ministral 3 14B model card, checked 12 October 2026.
• Mistral AI: Mistral Medium 3.5 model card and licence, checked 12 October 2026.
• Microsoft: Phi-4 model card, checked 12 October 2026.
• Inception: Jais-2-70B-Chat model card, checked 12 October 2026.
• Inception: Jais-2-8B-Chat model card, checked 12 October 2026.
• HUMAIN: ALLaM-7B-Instruct-preview model card, checked 12 October 2026.
• QCRI: Fanar-1-9B-Instruct model card, checked 12 October 2026.
• Meta: Llama 4 Community License, checked 12 October 2026.
• Meta: Llama 3.3 Community License, checked 12 October 2026.
• MiniMax: MiniMax M3 licence, checked 12 October 2026.
• Lenovo Press: On-premise vs cloud generative AI total cost of ownership, 2026 edition, updated 24 July 2026, checked 12 October 2026.
• Let’s Data Science: Nvidia lists RTX PRO 6000 Blackwell at $16,000, 12 August 2026, checked 12 October 2026.
• LMSYS: NVIDIA DGX Spark in-depth review, 13 October 2025, checked 12 October 2026.
• NVIDIA: AI Enterprise pricing, updated 2 September 2026, checked 12 October 2026.
• vLLM: Documentation, checked 12 October 2026.
• ggml-org: llama.cpp on GitHub, checked 12 October 2026.
• Ollama: Ollama on GitHub, checked 12 October 2026.
• Hugging Face: Text Generation Inference documentation, checked 12 October 2026.
• Hugging Face: Hub environment variables, checked 12 October 2026.
• Ovadia et al.: Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, v3, 30 January 2024, checked 12 October 2026.
• Unsloth: Fine-tuning VRAM requirements, checked 12 October 2026.
• Artificial Analysis: gpt-oss-120b API providers, checked 12 October 2026.
• Pan et al.: A Cost-Benefit Analysis of On-Premise Large Language Model Deployment, arXiv 2509.18101 v3, first submitted 30 August 2025, checked 12 October 2026.
• Mistral AI: Mistral Large 3 model card, checked 12 October 2026.