SOP for disabling a chain

sop-pause-a-chain.md

SOP: Temporarily Disable a Chain

Standard Operating Procedure for Node Operators to temporarily halt a specific chain integration.

Overview

If a Node Operator notices an issue with a specific chain (e.g., unusual behavior, potential exploit, sync issues, or external chain problems), it is recommended to pause the chain integration. This brings attention to the issue, prevents potential fund loss, and allows time for investigation.

{% hint style="warning" %} Even during chain halts, Node Operators should refrain from doxxing themselves. Staying pseudo-anonymous is critical to ensuring the network is impartial, neutral and resistant to capture. {% endhint %}

When to Use This Procedure

• Suspected exploit or vulnerability affecting a specific chain
• Chain sync issues or Bifrost anomalies
• External chain hardfork or network instability
• Solvency discrepancies detected on a chain
• Unusual transaction patterns observed
• Double spend alert

Halt Types

There are three levels of chain-specific halts. Understanding the differences is critical for choosing the appropriate response.

Halt Type	Mimir Key Format	Example
Trading Halt	HALT<CHAIN>TRADING	HALTETHTRADING
Signing Halt	HALTSIGNING<CHAIN>	HALTSIGNINGETH
Chain Halt	HALT<CHAIN>CHAIN	HALTETHCHAIN

Trading Halt (HALT<CHAIN>TRADING)

• Disables quoting for the chain
• Halts currently streaming or queued swaps
• Does NOT prevent outbounds from being sent
• Does NOT prevent assets from leaving the network
• Does NOT pause inbound observations (newly observed inbounds remain in the inbound queue)

Signing Halt (HALTSIGNING<CHAIN>)

• Prevents all outbounds from being sent for the chain
• Affects all outbound types: swaps, trade asset withdrawals, secured asset withdrawals, LP withdrawals
• Does NOT disable quoting (integrations may still send funds that won't be processed)

Chain Halt (HALT<CHAIN>CHAIN)

• Halts all chain operations: swapping, signing, outbounds, LP actions, trade assets, and secured assets
• Transactions are no longer observed; Bifrost goes offline for the chain
• Resumption requires a majority of nodes to resync to the chain tip

{% hint style="danger" %} Important: When a problem is detected, you should halt both trading and signing. Halting only signing keeps quoting active, which means integrations may continue sending funds that won't be processed—resulting in a poor user experience. {% endhint %}

Common Situations

Situation	Recommended Action
Node daemons are erroring or falling behind	Halt signing and halt trading
Issue resolved, but many nodes still behind	Keep signing and trading halted
Issue resolved, supermajority of nodes at tip	Re-enable signing first, verify orderly processing of outbounds, then re-enable trading
Double spend alert detected	Halt the entire chain (HALT<CHAIN>CHAIN) and wait for dev instructions

Procedure

{% hint style="success" %} Halt often, halt early. Node Operators are encouraged to halt when they suspect something is wrong. It is better to halt proactively and investigate than to wait and risk fund loss. Most halts should include both signing and trading. {% endhint %}

Step 1: Halt the Chain

When you notice an issue, halt first. For most issues, halt both trading and signing:

Halt Trading:

make mimir
=> Enter THORNode Mimir key: HALT<CHAIN>TRADING
=> Enter THORNode Mimir value: 1

Halt Signing:

make mimir
=> Enter THORNode Mimir key: HALTSIGNING<CHAIN>
=> Enter THORNode Mimir value: 1

For severe issues (double spend, critical exploit):

make mimir
=> Enter THORNode Mimir key: HALT<CHAIN>CHAIN
=> Enter THORNode Mimir value: 1

Step 2: Notify the Community

After initiating the halt, notify other Node Operators:

1. Use make relay to communicate anonymously via the THORChain Dev Discord #mainnet channel
2. Alternatively, use a burner Discord account to report the issue
3. Tag @security, @contributor if the issue poses a risk to funds
4. Provide details about what you observed and why you initiated the halt

Step 3: Wait for Solution

While the chain is halted:

1. Create a thread in #mainnet to oordinate with other Node Operators and the development team
2. Explain what you observed
3. Provide logs if you can or if relevant
4. Wait for guidance from the development team or consensus from other Node Operators

Step 4: Validate Solution

Before resuming operations:

1. Confirm the fix has been applied or the external issue resolved
2. Ensure a supermajority of nodes are synced to the chain tip
3. Verify solvency checks pass
4. Review dashboards for any anomalies:
   • thorchain.network - Comprehensive THORNode dashboard
   • thorchain.net/nodes - Monitor node status and chain tip sync
   • runescan.io - THORNode information within block explorer

Step 5: Re-enable the Chain

Follow the resumption procedure below to safely restore operations.

Resuming Operations

Once the issue is resolved:

Step 1: Verify Resolution

1. Confirm the fix has been applied or the external issue resolved
2. Ensure a supermajority of nodes are synced to the chain tip
3. Verify solvency checks pass

Step 2: Vote to Resume (In Order)

Re-enable signing first to clear any stuck swaps:

make mimir
=> Enter THORNode Mimir key: HALTSIGNING<CHAIN>
=> Enter THORNode Mimir value: 0

Verify orderly processing of outbounds, then re-enable trading:

make mimir
=> Enter THORNode Mimir key: HALT<CHAIN>TRADING
=> Enter THORNode Mimir value: 0

To delete your vote entirely, use -1:

make mimir
=> Enter THORNode Mimir key: <KEY>
=> Enter THORNode Mimir value: -1

Step 3: Monitor Recovery

• Watch for any backlog of transactions that need processing
• Monitor chain sync status
• Verify normal trading operations resume

Quick Reference

Action	Mimir Key	Value
Halt trading	HALT<CHAIN>TRADING	1
Halt signing	HALTSIGNING<CHAIN>	1
Halt chain (severe)	HALT<CHAIN>CHAIN	1
Resume trading	HALT<CHAIN>TRADING	0
Resume signing	HALTSIGNING<CHAIN>	0
Resume chain	HALT<CHAIN>CHAIN	0
Delete vote	<KEY>	-1

Common Mimir Key Mistakes

{% hint style="warning" %} Beware of typos! The following are examples of incorrect vs correct Mimir keys:

Incorrect (Bogus)	Correct
HALTRADING	HALTTRADING
HALTETHSIGNING	HALTSIGNINGETH

Always double-check your Mimir key format before submitting a vote. {% endhint %}

Correct formats:

• Trading: HALT<CHAIN>TRADING (e.g., HALTETHTRADING, HALTBTCTRADING)
• Signing: HALTSIGNING<CHAIN> (e.g., HALTSIGNINGETH, HALTSIGNINGBTC)
• Chain: HALT<CHAIN>CHAIN (e.g., HALTETHCHAIN, HALTBTCCHAIN)

Related Resources

• Emergency Procedures
• Network Halts
• Mimir Abilities
• Security

Notes

• A node's vote is valid as long as they are active, removed if they are not, and operational Mimir resets on churn
• A node can change their vote at any time, other nodes can override
• For network-wide emergencies, use make pause instead (see Emergency Procedures)