LLM Foundry: a modular framework for orchestration, memory, compression, tools, and evaluation

Let a task be represented by a prompt x, a context bundle c, and a model backend M.

We write the effective model interface as:

y = M(x, c; θ)

where θ denotes backend-specific parameters or API state. The surrounding system transforms raw user intent into a structured context bundle:

c = (c_sys, c_user, c_mem, c_tools, c_trace)

with system instructions, user prompt, retrieved memory, tool affordances, and trace history. The job of LLM Foundry is to construct c in a way that improves utility without forcing the user to think about every layer manually.

Let S be a summary candidate and T be the original transcript or working context.

U(S; T) = α · Rel(S, T) - β · Redundancy(S) + γ · Retrieval(S) - δ · Cost(S)

where Rel measures relevance to the task, Redundancy penalises repeated content, Retrieval rewards future usefulness, and Cost accounts for token count and later rehydration overhead. The coefficients are not sacred; they are a tuning problem.

A simple extractive compressor can be written as a ranking problem over sentences or spans. If σ_i is a sentence score, then we keep the top-scoring spans subject to a budget constraint:

maximize   Σ_i σ_i z_i
subject to Σ_i ℓ_i z_i ≤ B,   z_i ∈ {0, 1}

where ℓ_i is span length and B is the token budget. The relaxation z_i ∈ [0,1] gives a continuous form that is easier to optimise and easier to explain to a human.

Let an agent trajectory be τ = (o_1, a_1, o_2, a_2, …, o_n, a_n), where observations and actions alternate.

The runtime consumes a task, selects tools, executes calls, and records a trace:

trace = {
  task,
  steps: [
    { observation, action, tool_result },
    ...
  ],
  final_answer
}

That trace can be exported into supervised examples. A simple export objective is to maximise the likelihood of correct tool sequences while preserving enough metadata to reproduce the run later.

The repo documents multi-endpoint configuration for failover or round-robin routing. If each endpoint i has latency ℓ_i, cost c_i, and reliability r_i, then a routing score can be written as:

score(i) = w_ℓ · ℓ_i + w_c · c_i - w_r · r_i

The chosen backend is then argmin_i score(i), optionally subject to hard constraints such as provider availability, token limits, or policy filters. This is not fancy, but it is robust, which is the only reason it matters.

A practical metric suite can include exact-match, pass@k, tool success rate, retrieval hit rate, and end-to-end task completion.

task_score = λ1·exact_match + λ2·pass@k + λ3·tool_success + λ4·retrieval_hit_rate

Algorithm 1 — context build

Input: user prompt x, memory store V, backend M
1. retrieve top-k notes from V using x
2. build system prompt c_sys
3. compose c = (c_sys, x, retrieved notes, tool schema, trace summary)
4. call backend M(x, c)
5. record output and metadata

Algorithm 2 — trace export

Input: run trace τ
1. normalise tool names and arguments
2. split trajectory into observation/action pairs
3. write structured record to disk
4. attach task, final answer, and outcome flags

Algorithm 3 — routing

Input: endpoint set E, task t
1. score each endpoint by latency, cost, and reliability
2. filter unavailable endpoints
3. choose argmin score
4. if the call fails, retry on the next best endpoint

A trace can be turned into supervised data by pairing the observed state with the chosen action. If a policy π produced the trace, then standard sequence learning minimises:

L_sft = -Σ_t log π(a_t | o_tt}, c_t)

Additional penalties can be added for invalid tool use, incoherent reasoning, or failed retrieval. The general objective is:

L = L_sft + λ_trace L_trace + λ_mem L_mem + λ_safe L_safe

The point is not to worship one loss function. It is to make the right behaviour easy to measure and harder to fake.

• This is a control-plane framework, not a frontier base model.
• Compression and memory quality depend on the backend and the task.
• Multi-endpoint routing is only as good as the provider metadata you feed it.
• Trace export is useful only if the traces are actually valid.
• Evaluation must be repeated on real workloads, not just demo prompts.

python -m llm_foundry demo --provider qwen --model Qwen/Qwen2.5-0.5B-Instruct --prompt "Hello"

python -m llm_foundry proof --provider qwen --model Qwen/Qwen2.5-0.5B-Instruct

python -m llm_foundry super-suit --api-endpoints-file endpoints.json --api-strategy failover --task "Do the whole workflow"

LLM Foundry is a control plane around an LLM, not a replacement for one. Its value comes from wrapping the model with memory, compression, tools, traces, evaluation, and routing so the whole thing behaves like a system instead of a toy.

The paper draft is intentionally practical. It says what the repo already does, formalises the moving parts, and leaves enough mathematical structure for a proper final manuscript. That is better than bluff.