Model Merging

Training a new model from scratch for every capability you want is slow and expensive; fine-tuning a single model on task after task risks catastrophic forgetting of what it already knew. Model merging offers a third path: take two or more models that already exist, most often variants fine-tuned from the same base, and combine their weights directly, arithmetically, with no gradient step involved. The result is a single set of weights that behaves as though it inherited abilities from every model that went into it, at exactly the inference cost of one model rather than an ensemble of several.

This sounds implausible on its face. Neural network loss landscapes are famously non-convex, so averaging two arbitrary sets of weights should, in principle, land somewhere useless between them. The reason merging often works anyway, and why it fails when it does, is one of the more interesting empirical puzzles in modern deep learning, and it’s the thread this entry follows through linear averaging, task arithmetic, conflict-aware merging, and the “frankenmerges” that stitch entire layer blocks from different models together.

Why Merge Instead of Ensemble or Retrain

An ensemble of five fine-tuned models improves accuracy but costs five times the inference compute and memory. Retraining a single model on the union of five tasks’ data requires access to that data (often unavailable across organizations for privacy or licensing reasons) and risks each new task degrading performance on the others. Model merging sidesteps both problems: it needs no training data at merge time, only the weight files themselves, and it produces one deployable model with one model’s worth of latency and memory footprint.

This is why merging has become the default way the open-weight community builds new models: take a strong base (Llama, Mistral, Qwen), collect a handful of specialists fine-tuned from it (a coding model, a roleplay model, a reasoning model), and merge them into a generalist that is competitive with, and sometimes exceeds, purpose-trained models of the same size. Most of the top entries on community leaderboards for mid-size open models over the last two years have been merges, not from-scratch fine-tunes.

Linear Interpolation and Model Soups

The simplest merge is a weighted average of two models’ parameters, position by position:

w_merged = (1 - t) · w_A + t · w_B

Wortsman et al.’s model soups paper showed this works surprisingly well when w_A and w_B are fine-tunes of the same pretrained checkpoint on the same task with different hyperparameters: averaging dozens of independently fine-tuned CLIP and ViT checkpoints beat the best individual model in the soup, with zero added inference cost, because averaging flat, wide minima that different training runs converge near tends to land in an even flatter, better-generalizing region.

# Scenario: 6 runs of the same fine-tuning job with different seeds/LR
# schedules, all producing decent-but-different checkpoints. Average them.
import torch

checkpoints = [torch.load(f"run_{i}.pt") for i in range(6)]
soup = {
    k: sum(ckpt[k] for ckpt in checkpoints) / len(checkpoints)
    for k in checkpoints[0]
}
torch.save(soup, "model_soup.pt")

The catch is that this only reliably works when the models share initialization and are close together in weight space. Average two models that diverged too far, or that were trained independently from different initializations, and the averaged weights can land in a high-loss region between two separate basins, a phenomenon documented well before merging became popular: two independently trained networks are rarely linearly connected by a low-loss path, because their hidden units are permuted relative to each other in essentially arbitrary ways. Git Re-Basin (Ainsworth et al.) showed that if you first find the permutation of one network’s neurons that best aligns it with the other, most pairs of independently trained networks are connected by a low-loss path after all, a property the paper calls approximate single-basin structure. This is why “just average the weights” works for close fine-tunes of one base model but needs permutation alignment or more careful methods for models trained from separate initializations. The interactive widget below makes this concrete.

Task Arithmetic

Ilharco et al.’s task arithmetic reframes fine-tuning itself as a vector operation. A task vector is the difference between a fine-tuned model’s weights and the base it was fine-tuned from:

τ = θ_finetuned − θ_base

This vector, in weight space, points in the direction that improves performance on the fine-tuning task. Because it’s just a subtraction, task vectors from different fine-tunes of the same base can be added together and applied back to the base with a scaling coefficient λ:

θ_merged = θ_base + Σᵢ λᵢ · τᵢ

The paper demonstrates three arithmetic operations that behave the way the names suggest: negating a task vector removes that capability (subtracting a toxicity task vector reduces toxic generations), adding task vectors combines capabilities (a code task vector plus a math task vector yields a model competent at both), and combining task vectors via analogies (A - B + C) can transfer a relationship learned on one domain to another, echoing the classic word2vec king - man + woman analogy but in the space of entire fine-tuned model updates. Crucially, this is all linear algebra on cached weight diffs; no forward or backward pass through data is needed at merge time.

Resolving Conflicts: TIES-Merging and DARE

Naive summation of task vectors runs into two failure modes once you merge more than two specialists: redundancy (many parameters barely changed during fine-tuning and just add noise when summed) and sign conflict (task A’s fine-tuning increases a weight while task B’s decreases the same weight, and a plain sum partially cancels both signals).

TIES-Merging (Yadav et al.) addresses this with three explicit steps, giving the technique its name (Trim, Elect sign, Disjoint merge… the paper calls it TrIm, Elect sign, mErge):

1. Trim: zero out the smallest-magnitude changes in each task vector, keeping only the parameters that moved the most during fine-tuning.
2. Elect sign: for each parameter, take a majority vote across task vectors on whether it should end up positive or negative overall.
3. Disjoint merge: average only the task vectors that agree with the elected sign at that parameter, ignoring the ones that don’t.

# Scenario: merging 4 task vectors that disagree on sign for many
# parameters (task A wants a weight to increase, task C wants it to
# decrease). TIES resolves this instead of letting them cancel out.
import torch

def ties_merge(task_vectors: list[dict], trim_frac: float = 0.8) -> dict:
    merged = {}
    for key in task_vectors[0]:
        stacked = torch.stack([tv[key] for tv in task_vectors])

        # Trim: zero out the smallest-magnitude entries per task vector
        k = int(stacked[0].numel() * trim_frac)
        flat = stacked.abs().view(stacked.shape[0], -1)
        threshold = flat.kthvalue(k, dim=1, keepdim=True).values
        mask = flat >= threshold
        trimmed = (stacked.view(stacked.shape[0], -1) * mask).view(stacked.shape)

        # Elect sign: majority vote of the trimmed, sign-bearing values
        sign = torch.sign(trimmed.sum(dim=0))

        # Disjoint merge: average only vectors agreeing with the elected sign
        agree = (torch.sign(trimmed) == sign) & (trimmed != 0)
        count = agree.sum(dim=0).clamp(min=1)
        merged[key] = (trimmed * agree).sum(dim=0) / count
    return merged

DARE (Drop And REscale, Yu et al., published as “Language Models are Super Mario”) tackles redundancy from a different angle: it randomly drops a large fraction p of each task vector’s parameters (the paper shows 90%, and even 99%, can be dropped) and rescales the survivors by 1/(1-p) to keep the vector’s expected magnitude unchanged. This sparsifies each task vector before it’s merged, which reduces interference between task vectors and, combined with TIES-style conflict resolution, lets more models be merged together before quality degrades. DARE’s own reported result was a merged 7B model that briefly topped the Open LLM Leaderboard for its size class using only weight arithmetic on existing checkpoints.

Spherical Interpolation (SLERP)

Linear interpolation treats weight vectors as points in Euclidean space, but high-dimensional weight vectors behave more like directions than points; spherical linear interpolation (SLERP), borrowed from quaternion animation, interpolates along the arc of the hypersphere connecting two vectors rather than the straight line between them:

slerp(w_A, w_B, t) = [sin((1-t)·Ω) / sin(Ω)] · w_A + [sin(t·Ω) / sin(Ω)] · w_B

where Ω is the angle between w_A and w_B. In practice, SLERP tends to preserve more of each model’s distinctive character than linear averaging when merging exactly two models, because it doesn’t shrink the resulting vector’s norm the way linear averaging of two divergent directions can. It is the default merge method in mergekit, the open-source toolkit from Arcee AI that formalized most of these methods into a single reusable pipeline:

# Scenario: merging a general chat model with a coding specialist
# using mergekit's SLERP method, layer by layer.
merge_method: slerp
base_model: meta-llama/Llama-3-8B
models:
  - model: meta-llama/Llama-3-8B-chat
  - model: codellama/CodeLlama-8B
parameters:
  t: 0.5          # 0 = pure chat model, 1 = pure coding model
dtype: bfloat16

Frankenmerging: Combining Architecture, Not Just Weights

Every method above assumes the merged models share an identical architecture, layer for layer. Passthrough merging, nicknamed frankenmerging, drops that assumption: it builds a new model by concatenating layers from different source models (or duplicating layers from one model) rather than averaging weights at matching positions. This is how models like Goliath-120B were built by interleaving layers from two 70B Llama-2 fine-tunes to produce a deeper model than either source, without any additional pretraining. The result is architecturally unusual (layer count no longer matches any single source model) but functional, because transformer layers are compositionally closer to independent processing stages than the weight-averaging methods assume.

graph TD
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    A[Model Merging]:::output
    B[Weight-Space Merging]:::process
    C[Architecture-Space Merging]:::process

    A --> B
    A --> C

    B --> B1[Linear averaging /\nmodel soups]:::data
    B --> B2[Task arithmetic]:::data
    B --> B3[TIES-Merging\nconflict resolution]:::data
    B --> B4[DARE\nsparsify then merge]:::data
    B --> B5[SLERP\nspherical interpolation]:::data

    C --> C1[Passthrough / frankenmerge\nlayer concatenation]:::data
    C --> C2[MoE construction\nfrom dense experts]:::data
    C --> C3[Branch-Train-Merge\nparallel domain experts]:::data

A related architecture-space technique builds a mixture-of-experts model out of several dense fine-tunes: instead of interleaving layers sequentially, each fine-tuned model becomes one expert in an MoE layer, with a newly initialized router learning to dispatch tokens to whichever expert (source model) handles them best. Unlike a standard Mixture of Experts trained end-to-end from scratch, this only requires training the small router, since the experts themselves arrive pre-trained via the merge. Meta’s earlier Branch-Train-Merge work anticipated this pattern at the pretraining stage: train several domain-expert language models independently and in parallel (no cross-node gradient synchronization needed), then merge or ensemble them post hoc depending on whether inference-time efficiency or maximum accuracy matters more for the deployment.

Interactive: The Loss Barrier Between Two Fine-Tunes

The reason merging isn’t always free lunch comes down to whether the straight-line path between two models’ weights stays in a low-loss region the whole way, or dips through a high-loss “barrier” partway through. Two fine-tunes of the same base model, starting from the same initialization, usually stay in the same basin and interpolate smoothly. Two models trained from different initializations usually don’t, unless their hidden units are first permutation-aligned (Git Re-Basin’s contribution). Drag the slider below to see both cases:

The indigo curve (permutation-aligned) barely dips, since aligning units first keeps both models in the same basin; the violet curve (naive average of differently-initialized models) drops sharply near t = 0.5, the loss barrier that motivated Git Re-Basin in the first place. This is why merging fine-tunes of the same base model with plain linear interpolation “just works,” while merging models trained from scratch with different initializations needs alignment first.

Model Merging vs. Related Techniques

Merging is unique on this list for requiring neither training data nor gradient computation, which is exactly why it has become the fastest, cheapest way for the open-weight community to iterate: a new merge can be produced and evaluated in minutes on a laptop, compared to hours or days for even a lightweight fine-tune.

What’s New (2025-2026)

Evolutionary merge search. Sakana AI’s evolutionary model merging (Akiba et al.) replaced hand-tuned merge recipes with an evolutionary search that optimizes both the merge coefficients and, in a second axis, which layers from which source models to interleave, jointly in parameter space and in what the paper calls data-flow space. Their EvoLLM-JP model, produced this way from existing open Japanese and math/reasoning models, beat some 70B models on Japanese math reasoning despite being far smaller, without any additional pretraining, purely through a better-discovered merge recipe.

Merging as a continual learning strategy. A growing line of work, including recent work on adaptive iterative merging using training trajectories, treats periodic merging of sequential fine-tuning checkpoints as a lightweight alternative to formal continual learning regularizers: instead of an explicit penalty like Elastic Weight Consolidation, you fine-tune on the new task, then merge the resulting checkpoint back toward the previous one, trading off some new-task performance for retained old-task performance. See Continual Learning for the fuller comparison of these approaches.

Merge-based model families as a release strategy. Multiple frontier open-weight releases in 2025 shipped several fine-tuned variants (chat, code, long-context, multilingual) from the same base specifically so the community could merge them, rather than the vendor picking one blend; mergekit’s adoption numbers (many thousands of community-produced merges hosted on Hugging Face) reflect this becoming a standard downstream step rather than a niche trick.

Safety erosion through merging. Because merging operates purely on weights with no data-driven check on what capability is being reintroduced, several 2025 analyses documented that merging a safety-aligned model with an unaligned or lightly-aligned fine-tune can partially undo the safety tuning, since the merge has no mechanism to distinguish “helpful capability” task vectors from “guardrail-weakening” ones. This has pushed some merge tooling toward re-running a safety/alignment pass after merging rather than treating the merged weights as production-ready by default.

Applications

Community model families. Nearly every popular mid-size open-weight “best of” leaderboard entry over the past two years has been a merge of existing fine-tunes rather than a fresh training run, because merging is orders of magnitude cheaper than training and can be iterated in a tight loop.

Multi-domain enterprise deployment. A company with separate fine-tunes for legal document review, customer support, and internal code assistance can merge them into one deployed model instead of running three separate inference services, cutting GPU footprint roughly in proportion to how many specialists get folded into one.

Cross-lingual and cross-domain transfer. Task arithmetic’s analogy operation (A − B + C) has been used to transfer a capability learned in one language to a model that otherwise only saw it in another, by combining task vectors captured in each language.

Federated and privacy-constrained settings. Because merging needs only the weight files, not the underlying training data, organizations that cannot share data across teams or jurisdictions (health systems, banks) can still combine model capabilities by exchanging checkpoints and merging locally.

Rapid checkpoint recovery. Averaging the last several checkpoints of a single training run (a special case of model soups) is a cheap way to reduce the variance/noise of the very last training step without any additional compute, and is used as a routine post-training step in some large-scale pretraining pipelines.

import torch
from safetensors.torch import load_file

# Scenario: you have one base model plus two independently
# fine-tuned checkpoints (a coding specialist and a math specialist)
# and want a single model that is decent at both, with no retraining.

base_sd  = load_file("base-model.safetensors")
code_sd  = load_file("code-finetune.safetensors")
math_sd  = load_file("math-finetune.safetensors")

def task_vector(finetuned_sd: dict, base_sd: dict) -> dict:
    # A task vector is just "what fine-tuning changed": ft - base.
    return {k: finetuned_sd[k] - base_sd[k] for k in base_sd}

tv_code = task_vector(code_sd, base_sd)
tv_math = task_vector(math_sd, base_sd)

def apply_task_arithmetic(
    base_sd: dict,
    task_vectors: list[dict],
    scaling_coeffs: list[float],
) -> dict:
    merged = {k: v.clone() for k, v in base_sd.items()}
    for tv, lam in zip(task_vectors, scaling_coeffs):
        for k in merged:
            merged[k] += lam * tv[k]
    return merged

# lam controls how much of each specialist survives in the merge;
# 0.3-0.5 per task vector is a common starting point for two-task merges.
merged_sd = apply_task_arithmetic(
    base_sd,
    task_vectors=[tv_code, tv_math],
    scaling_coeffs=[0.4, 0.4],
)

torch.save(merged_sd, "merged-code-math-model.pt")