Routing Methodology
Understanding the engineering under the hood of the TriForge Hybrid routing framework
Tier 1: Intelligent Semantic Intent Classification
When a query arrives, it is processed by our dual semantic classifier in semantic_classifier.py. Queries that match high-complexity profiles (Coding, Mathematics, Reasoning) are routed **directly** to the remote Fireworks model, as smaller local models (3B) lack the multi-step reasoning depth for these domains. Short, factual, conversational, or translation prompts are routed to the local model.
Tier 2: Local Self-Consistency double Sampling
If routed locally, we sample the local model **twice** at temperature 0.7. We calculate their string agreement similarity ratio. If the two samples are highly similar (exceeding the consistency threshold, typically 0.8), it is statistically highly probable that the local model is stable and correct. If the similarity is low, it indicates uncertainty, triggering immediate escalation.
Tier 3: Hedging & Uncertainty Audits
Before local responses are delivered, they are scanned for hedging keywords (e.g. *"not sure if"*, *"as an AI"*, *"unable to answer"*). Even if double samples are highly similar (e.g. they both say *"I apologize, I cannot answer this"*), hedging triggers an escalation to prevent delivering unhelpful empty responses.
Tier 4: Optimal Escalation via Verify-Draft
When escalated, we do not discard the local model's draft. Instead, we submit the task alongside the local model's draft and instruct the remote model to act as a **verifier**. The remote model corrects errors and verifies details instead of composing the answer from scratch. This significantly decreases completion tokens compared to direct remote prompting, yielding substantial token savings.