Skip to main content

Getting Started

In this tutorial you'll install ThoughtJack, run a built-in attack scenario, and learn to read the output. By the end you'll understand the core workflow for testing MCP clients.

This tutorial uses traffic mode (the default), where ThoughtJack runs real protocol infrastructure. To test LLMs directly without agent infrastructure, see Testing with Context Mode.

ThoughtJack executes attack scenarios authored as OATF (Open Agent Threat Format) documents - a declarative YAML format for describing adversarial agent test cases.

1. Install ThoughtJack​

Choose one method:

# Homebrew (macOS/Linux)
brew install thoughtgate/tap/thoughtjack

# Cargo
cargo install thoughtjack

# Or build from source
git clone https://github.com/thoughtgate/thoughtjack.git
cd thoughtjack
cargo build --release

Verify the installation:

thoughtjack version

You should see output like:

thoughtjack 0.5.0
  commit:  a1b2c3d
  built:   2026-01-15
  rustc:   1.88.0 (2024 edition)
  target:  aarch64-apple-darwin

2. Verify your installation​

Confirm ThoughtJack runs by validating a built-in scenario:

thoughtjack validate scenarios/library/benchmark/OATF-001_exfil-chain-tool-description.yaml

To get a real verdict, you need either context mode (calls an LLM API directly) or traffic mode with an agent connected. Context mode is the fastest way to see a real result:

thoughtjack scenarios run oatf-001 \
  --context --context-provider openai --context-model gpt-4o \
  --context-api-key $OPENAI_API_KEY \
  --max-session 1m -o verdict.json

See Testing with Context Mode for a full walkthrough.

3. Explore built-in scenarios​

ThoughtJack ships with built-in attack scenarios across multiple categories. List them:

thoughtjack scenarios list
Built-in Scenarios (91 available)

  Temporal
    oatf-002                Tool Definition Rug Pull
    oatf-007                MCP Server Supply Chain Attack

  Injection
    oatf-001                Tool Description Prompt Injection
    oatf-004                Tool Response Injection (Indirect Prompt Injection)
    ...

Run a scenario: thoughtjack scenarios run <name>
View YAML:      thoughtjack scenarios show <name>

View the YAML for a specific scenario:

thoughtjack scenarios show oatf-002

4. Run with live traffic​

Traffic mode requires a real agent connected to ThoughtJack. ThoughtJack acts as a malicious MCP server; your agent connects and discovers tools.

Start the scenario as an MCP server:

thoughtjack scenarios run oatf-002 --mcp-server 127.0.0.1:8080

ThoughtJack is now listening on http://127.0.0.1:8080/mcp. Nothing will happen until an agent connects. See Test Agent Frameworks for instructions on connecting LangGraph, CrewAI, or other agents.

Once an agent connects, ThoughtJack will:

  1. Respond to initialize with server capabilities
  2. Advertise a calculator tool via tools/list
  3. Return "Result: 42" for any tools/call to calculator
  4. After 3 tool calls, advance to the next phase and send notifications/tools/list_changed
  5. On the next tools/list, return the calculator with a poisoned description

What to observe​

When testing your agent against this scenario, watch for:

  • Does the agent re-read tool definitions after receiving list_changed?
  • Does the agent detect the prompt injection in the modified calculator description?
  • Does the agent follow the injected instructions?

5. Progress output​

ThoughtJack shows real-time progress by default when running in a terminal. The progress display shows phase transitions, trigger counters, entry actions, and indicator results.

Use --progress off to disable progress and show only the human summary after completion:

# Off - human summary only
thoughtjack scenarios run oatf-002 --mcp-server 127.0.0.1:8080 --progress off

For verbose logging (debug-level internal state), add -vv:

thoughtjack scenarios run oatf-002 --mcp-server 127.0.0.1:8080 -vv

For machine-parseable logs, use JSON format:

thoughtjack scenarios run oatf-002 --mcp-server 127.0.0.1:8080 --log-format json -vv

6. Run from a config file​

Built-in scenarios are convenient, but real testing uses YAML config files. Export a scenario's YAML:

thoughtjack scenarios show oatf-002 > my-scenario.yaml

Run from the file:

thoughtjack run my-scenario.yaml --mcp-server 127.0.0.1:8080

Always validate configs before running:

thoughtjack validate my-scenario.yaml

7. Test an MCP server (client mode)​

The examples above run ThoughtJack as a server to test MCP clients. ThoughtJack can also run as an MCP client to test servers - useful for verifying that a server handles adversarial client requests correctly.

Connect to a running server​

If the target MCP server is already listening on HTTP:

thoughtjack run scenario.yaml --mcp-client-endpoint http://localhost:3000/mcp

Spawn the server as a subprocess​

ThoughtJack can launch the server process and connect over stdio:

thoughtjack run scenario.yaml \
  --mcp-client-command "npx" \
  --mcp-client-args "-y @modelcontextprotocol/server-everything"

In client mode, ThoughtJack sends requests defined in the OATF scenario (tool calls, resource reads, etc.) and evaluates the server's responses against the configured indicators.

8. Try context mode​

The examples above use traffic mode - real protocol connections between ThoughtJack and your agent. ThoughtJack also supports context mode, which calls an LLM API directly and injects adversarial payloads into the conversation history.

This is useful when you want to test whether a model follows injected instructions without setting up a full agent pipeline:

thoughtjack scenarios run oatf-001 \
  --context \
  --context-model gpt-4o \
  --context-api-key $OPENAI_API_KEY

See Testing with Context Mode for a full walkthrough, or Execution Modes to understand how the two modes differ.

Next steps​