Research Methodology

What This Document Covers

The Research Agenda defines what Fullstacko studies and why. This document defines how.

It covers the end-to-end research process: how operational systems are instrumented to capture signal, how observations are documented, how patterns are synthesized into findings, how findings are validated, and what standards each output type must meet before publication.

This document is the operating manual for Fullstacko’s research practice. It changes when the practice changes.

---

The Core Method: Production Observation

Fullstacko is not a research lab. Every research artifact we produce is grounded in direct observation of real systems — middleware we built, operate, and maintain under real-world constraints.

This is not a limitation. It is the method.

Most research on AI agents and middleware is conducted in controlled environments with synthetic tasks, curated inputs, and clean state. Fullstacko studies the same systems in production: connected to live tools, under real user load, with incomplete data, unexpected inputs, and the full range of conditions that controlled research excludes by design.

The method has one source and one direction:

Instrumented production systems
         ↓
  Raw observation logs
         ↓
  Pattern identification
         ↓
  Hypothesis formation
         ↓
  Targeted investigation
         ↓
  Findings and validation
         ↓
  Publication-ready artifact

Each stage is deliberate. Observations do not become findings automatically. Findings do not become publications until they meet the validation standards defined in this document.

---

Positionality

Fullstacko builds and operates the systems we study. That is a genuine advantage in terms of access and depth of observation. It is also a source of bias that must be managed explicitly.

We acknowledge this directly:

• We are not neutral observers. We have commercial incentives tied to the systems we study.
• Findings that make our products look better are not more likely to be true.
• We publish findings that reflect what we actually observe — including failure modes and limitations in our own systems.

Every research artifact published by Fullstacko states explicitly that findings are derived from Fullstacko-operated systems and may not generalize without independent replication. We do not paper over this. It is part of our methodology.

---

Instrumentation Standards

Research signal must be captured at the time of operation. Reconstructed observations are less reliable than real-time logs. The instrumentation layer is part of the product — not an afterthought.

What Gets Instrumented

Every Fullstacko MCP server and skill runs with structured logging active from its first deployment. The minimum instrumentation set covers:

Tool calls

• Timestamp, duration, tool identifier, input schema hash
• Outcome: success, failure type, partial success with classification
• Retry attempts and final resolution

Agent sessions

• Session identifier, model, task type classification
• Total tool calls, unique tools used, escalation events
• Task completion status: completed, abandoned, degraded, error

Context events

• Context length at key checkpoints in long-running tasks
• Truncation events, with note of what was dropped
• Model switches within a session (where applicable)

Cost events

• Input and output token counts per call
• Model tier per call
• Cumulative cost per session and per workflow type

Escalation events

• Trigger type: agent-initiated, rule-triggered, user-forced
• Task state at escalation point
• Post-intervention outcome

Instrumentation Principles

Capture at the boundary. Log at the middleware layer, not inside the model. We cannot observe what happens inside the model; we can observe exactly what enters and exits the middleware boundary. That is where our signal lives.

Structured over unstructured. Every log entry follows a defined schema. Free-text logs are useful for debugging; they are not useful for research synthesis. Research logs are structured from day one.

No PII in research logs. User-facing content is never stored in research logs. Task metadata, tool identifiers, timing, and cost data are captured. Content that could identify a user is excluded.

Versioned schemas. Log schemas are versioned. When a schema changes, the version boundary is recorded. Research findings are always scoped to the schema version range they were derived from.

Instrument first, analyze later. We do not wait until we know what we want to study before instrumenting. The instrumentation layer captures a broad signal set continuously. Research questions are formed after observation, not before instrumentation.

---

Observation Documentation

Raw logs are not research observations. An observation is a logged event plus context: what was running, what conditions were present, and why this event is notable.

The Observation Record

Every observation that may become research input is documented in a structured record with these fields:

Date:         ISO 8601
System:       MCP server or skill identifier + version
Area:         Agent Reliability | AI Economics | Human-AI Collaboration | Middleware Systems
Event type:   [see per-area classification below]
Log ref:      Pointer to raw log entry or session
Description:  What happened, in plain language
Context:      What was running, what inputs were present, relevant conditions
Hypothesis:   If applicable — what this might indicate
Linked obs:   Any prior observations this connects to
Status:       Raw | Pattern candidate | Under investigation | Resolved

Observations are logged continuously as systems operate. The synthesis cycle (described later) determines which raw observations become pattern candidates and which are resolved without further investigation.

Event Classification by Research Area

Each research area has a defined event taxonomy. Classification is applied at the time of documentation, not during synthesis.

Agent Reliability

Event type	Description
tool_call_failure	Tool call did not return a valid response
silent_failure	Task completed without error but output is incorrect or incomplete
graceful_recovery	Agent successfully recovered from a failure without human input
ungraceful_recovery	Agent attempted recovery but produced a worse outcome
coordination_breakdown	Multi-agent task failed at the handoff between agents
long_horizon_degradation	Task quality declined measurably over extended execution
context_loss_event	Agent behavior changed in a way attributable to context truncation
loop_detection	Agent repeated the same tool call or reasoning step without progress

AI Economics

Event type	Description
cost_spike	Session cost exceeded 2x the workflow type baseline
latency_outlier	End-to-end task latency exceeded 2x the workflow type baseline
routing_opportunity	Frontier model used for a task where a smaller model would have been sufficient
context_cost_waste	Significant portion of context was unused but incurred token cost
roi_negative	Estimated cost of agent completion exceeded the estimated value of the task
model_tier_comparison	Same task was completed with different model tiers, enabling direct comparison

Human-AI Collaboration

Event type	Description
escalation_initiated	Agent stopped and requested human input
escalation_missed	Agent proceeded autonomously when it should have escalated
escalation_false_positive	Agent escalated when autonomous completion would have been appropriate
intervention_quality_high	Human intervention improved downstream task quality measurably
intervention_quality_low	Human intervention did not improve or worsened downstream outcome
trust_signal_change	User behavior changed in a way suggesting a shift in trust toward or away from the agent
ambiguity_handling	Agent encountered ambiguous task requirements and resolved or escalated

Middleware Systems

Event type	Description
protocol_edge_case	MCP protocol interaction produced unexpected behavior
orchestration_pattern	A novel or notable pattern of tool composition was observed
tool_integration_failure	External tool integration failed in a way attributable to the middleware layer
context_management_strategy	Agent used a notable strategy to manage context length or prioritization
schema_mismatch	Tool call failed due to schema incompatibility between agent and tool
transport_failure	Communication failure at the protocol transport layer
capability_discovery	Agent located and used a capability it was not explicitly directed to use

---

Methods by Research Area

Each research area uses the shared instrumentation and observation documentation standards above. Each also has specific investigative methods applied when pattern candidates warrant deeper investigation.

Agent Reliability

Primary method: Longitudinal failure analysis

When a pattern candidate is identified from the observation log — for example, a cluster of tool_call_failure events in a specific tool category — the investigation method is:

1. Extract all events of the same type within the observation window
2. Identify shared conditions: model, tool, task type, context length, session age
3. Attempt to reproduce the failure in a controlled environment
4. Classify: is this a deterministic failure, a probabilistic failure, or a configuration-dependent failure?
5. Document the reproduction case as part of the benchmark suite

Reproducibility requirement: A failure mode is not documented as a finding until it can be reproduced independently or confirmed across at least three independent production observations. Single-occurrence anomalies are logged but not published until corroborated.

Recovery strategy testing: When a graceful_recovery event is observed, the recovery strategy is extracted and tested explicitly: does it generalize to similar failures? Does it hold across model variants? This is how benchmark test cases are generated.

Long-horizon methodology: Long-horizon degradation requires time-series analysis. Sessions are segmented into equal time or token-count windows. Quality indicators (task progress rate, tool call success rate, context relevance) are measured per window. Degradation is defined as a statistically significant decline across at least three consecutive windows.

AI Economics

Primary method: Workflow-level cost-performance analysis

The unit of analysis is the workflow type, not the individual model call. A workflow type is a class of tasks with similar input characteristics, tool use patterns, and completion criteria.

Methodology:

1. Define workflow types from observed task patterns — not a priori categories
2. Establish cost and latency baselines per workflow type from the first 30 completed sessions
3. Track cost and latency distributions per workflow type continuously
4. Flag outliers for investigation using the event taxonomy above
5. For routing opportunities, conduct controlled comparisons: same task type, different model tiers, measured by completion quality and cost

Quality measurement: Cost data alone is not sufficient. Every cost comparison includes a quality measurement. Quality is assessed using task-specific criteria defined per workflow type. Where possible, ground truth comparison is used. Where not possible, structured rubric evaluation is applied and the rubric is published alongside the findings.

ROI framework: ROI is estimated at the workflow level. The value side of the equation uses a conservative estimate: the cost of the equivalent manual operation, not the aspirational business value. We do not publish ROI figures that depend on assumed value multipliers we cannot verify.

Human-AI Collaboration

Primary method: Escalation event analysis with outcome tracking

The escalation boundary is the most observable point in human-AI collaboration. Every escalation event is logged with its trigger and followed through to outcome.

Methodology:

1. Log all escalation events with the task state at the point of escalation
2. Track post-intervention outcomes: did the task complete? Was the output quality higher or lower than pre-escalation baseline?
3. Classify escalation quality: appropriate, false positive, missed
4. Identify predictive signals for appropriate escalation by comparing event features across classifications

Trust calibration measurement: Trust is not directly observable. We use behavioral proxies: rate of user override of agent decisions, rate of user-initiated escalations (separate from agent-initiated), and session abandonment rate. Changes in these proxies over time are treated as trust calibration signals. We are explicit about what these are — proxies — and do not overclaim.

Intervention analysis: For each intervention_quality_high or intervention_quality_low event, the investigation asks: what information was present in the intervention that was absent from the agent’s context? This question drives recommendations for escalation interface design.

Middleware Systems

Primary method: Protocol behavior documentation and pattern cataloguing

Middleware Systems research is more descriptive and taxonomic than the other areas in its early stages. The goal is to build a rigorous catalogue of how orchestration patterns and protocol interactions behave in production before moving to predictive or prescriptive findings.

Methodology:

1. Document each orchestration_pattern event with a structured description: tools involved, sequencing, dependency structure, outcome
2. Classify patterns into families: sequential, parallel, conditional, iterative, recursive
3. Track success rates and latency profiles per pattern family
4. For protocol_edge_case events: attempt to reproduce in isolation, determine whether the behavior is spec-compliant or a deviation, and file with the protocol maintainer if it represents a spec gap

Comparative methodology: Where the same orchestration goal is achieved through different patterns across sessions, document and compare. The comparison unit is: task success rate, total tool calls, total latency, and total cost per pattern variant. This is the primary input to orchestration design recommendations.

Schema and compatibility tracking: Every schema_mismatch event is documented with the full schema pair. Over time, these form a compatibility matrix — which tool integration patterns are robust and which are fragile. This matrix is published as part of the evaluation framework for this area.

---

Synthesis Process

Observations accumulate continuously. Synthesis is periodic — quarterly by default, triggered earlier if a significant pattern cluster emerges.

Stage 1: Pattern Review

At each synthesis cycle, all observations with status Raw since the previous cycle are reviewed. The question for each observation is:

• Does this stand alone, or does it connect to prior observations?
• Does it match a known pattern, extend a known pattern, or represent something new?
• Is it a candidate for the benchmark suite, a practitioner brief, or a whitepaper finding?

Observations are reclassified as Pattern candidate, Under investigation, or Resolved (i.e., explained by an existing finding or too isolated to pursue).

Stage 2: Pattern Investigation

Pattern candidates with at least three corroborating observations move to active investigation. Investigation means:

• Attempt to reproduce the pattern in a controlled environment
• Determine whether the pattern is deterministic, probabilistic, or context-dependent
• Identify the minimum conditions under which the pattern occurs
• Test candidate explanations by manipulating individual conditions

An investigation that cannot produce a reproducible finding after two synthesis cycles is closed. The observations are retained but the investigation is marked Inconclusive with a note on what was attempted.

Stage 3: Finding Formation

A finding is a pattern that has been reproduced, has a plausible explanation, and has been tested against at least one alternative explanation.

Every finding is written as a statement with explicit scope conditions:

“In [system configuration], when [conditions], [agent/model/middleware] [exhibits behavior]. This was observed [N] times across [time range]. It was not observed when [alternative conditions].”

Findings do not include recommendations at this stage. Recommendations are added at the output stage, not during synthesis.

Stage 4: Output Preparation

Findings are grouped by research area and assessed for output type:

If the finding is…	Then the output is…
Immediately actionable for practitioners	Practitioner brief
Reproducible with a standard test case	Benchmark suite addition
Supported by enough depth for rigorous argument	Whitepaper section
All three	All three, in that production order

Practitioner briefs are produced first. They validate the direction, generate community feedback, and establish the publication record before the more resource-intensive whitepaper is written.

---

Validation Standards

A finding is not ready to publish until it meets these standards.

Minimum Evidence Bar

Output type	Minimum evidence
Practitioner brief	3+ independent observations, 1 reproduction attempt
Benchmark test case	Deterministic reproduction in controlled environment
Whitepaper finding	5+ independent observations, 2+ reproduction attempts, 1 alternative explanation tested and rejected

Reproducibility

Every finding published in a whitepaper or benchmark suite must include enough procedural detail that another team with access to similar systems could attempt to reproduce it. We do not publish findings that depend on proprietary system state we cannot describe.

Where exact reproduction is not possible due to proprietary tooling, the methodology for obtaining equivalent observations in other systems is documented.

Uncertainty Disclosure

Every finding states:

• The observation count it is based on
• The system versions and configuration it was observed in
• Known conditions under which the finding may not hold
• What would change or invalidate the finding

Findings are not promoted to stronger claims than the evidence supports. Patterns observed in Fullstacko’s systems are not assumed to generalize to all production middleware without qualification.

Conflict of Interest Statement

Every published research artifact includes a statement noting that Fullstacko builds and operates the systems studied. This is not buried in a footnote — it is in the opening section.

---

Output Standards

Practitioner Briefs

Length: 1,500–3,500 words.

Structure:

1. The finding — stated directly in the opening paragraph. No preamble.
2. What we observed — description of the observation pattern with enough detail to evaluate.
3. What it means for builders — concrete, actionable implications.
4. What we don’t yet know — explicitly stated uncertainty.
5. Methodology note — brief description of how the observation was made.

Voice: Written for builders who will act on the findings. No academic hedging. Direct about what was observed, what it means, and what is still uncertain.

Publication channel: Studio research section, then syndicated to ecosystem channels. GitHub-linked where a benchmark or code artifact is involved.

Benchmark Suites

What a benchmark suite entry requires:

• A reproducible test case with defined inputs, environment specification, and expected outputs
• A classification of the failure mode or behavior pattern it tests
• A pass/fail criterion that does not require human judgment (or, where human judgment is unavoidable, a scoring rubric)
• A reference to the production observation(s) that motivated the test case

What it does not require:

• A full research paper
• A novel algorithmic contribution
• Coverage of every edge case

Benchmark suites are published as open-source repositories with a defined schema for contributions. The value is the standard, not the completeness of initial coverage.

Technical Whitepapers

Length: 4,000–8,000 words.

Structure:

1. Abstract — findings stated plainly, before methodology.
2. Introduction — problem framing, prior work, gap being addressed.
3. Methodology — full description sufficient for replication attempt.
4. Findings — each finding with evidence and scope conditions.
5. Discussion — what this means, alternative interpretations, limitations.
6. Implications for practitioners — same as practitioner brief but elaborated.
7. Future directions — what would strengthen or extend these findings.
8. Conflict of interest and data disclosure.

Review standard: Each whitepaper is reviewed against the validation standards above before submission. A checklist of these standards is completed and retained internally.

Evaluation Frameworks

What an evaluation framework provides:

• A set of dimensions along which a system behavior can be assessed
• Measurement criteria for each dimension
• A scoring method that is explicit and reproducible
• Reference implementations where applicable

Publication: Evaluation frameworks are published as open-source repositories. The companion documentation explains the framework’s scope and what it does not measure.

---

Research ↔ Product Feedback Loop

Research that does not change what we build is a cost without a return.

Before any finding is published, we document the product implication — specifically, what it means for how we design or operate the systems we build. This connection is made at the finding formation stage, not after publication.

Findings with no articulable product implication are still publishable — the research value is real. But the absence of a product implication is noted explicitly and treated as a prompt to look harder for one. The filter is the same as for everything else we build: does this compound over time?

---

Publication Cadence

Research is produced sequentially, not in parallel. Practitioner briefs come first — lower bar to publish, faster feedback loop, immediate value for the builder community. Benchmark suites and whitepapers follow once the direction is validated by published work.

We do not hold findings waiting for a complete picture. We publish what is ready, clearly scoped, and openly acknowledge what remains uncertain or uninvestigated.