GitHub - sshh12/tau2-bench-time-ablations: This fork of tau2-bench investigates whether LLM agent performance varies based on the temporal context of dates in prompts.

This task shows the most dramatic offset effect. The user has a basic economy ticket and needs to change their flight due to a family emergency. Per policy, basic economy flights cannot be modified—but there's a policy-compliant workaround: upgrade the cabin first, then change the flight.

What happens at baseline (2024):

Agent recognizes the situation:
  "I understand you'd like to upgrade to economy so you can modify your flight..."
Agent searches for nonstop flights, finds HAT041
Agent then: transfer_to_human_agents  ← Gives up without trying!
Task FAILS: No reservation update made

What happens at other offsets:

Agent finds the policy-compliant workaround:
  "I can upgrade your cabin from basic economy to economy,
   and then change your flights to a nonstop option..."
Agent calls: update_reservation_flights(cabin='economy', flights=[HAT041])
Task PASSES: Upgrade + flight change in one action

Key insight: The baseline agent knows the workaround exists (it mentions upgrading) but doesn't execute it. The offset agents actually do it. Same knowledge, different action-taking behavior.

Tests agent's ability to correctly refuse a cancellation that violates policy. User has two flights on May 17 and wants to cancel one.

What happens at baseline (2024):

Agent checks reservations: 35V5SM, XXDC1M, V5EMZH, D1EW9B, 9HBUV8, DGZSYX
Agent calls: cancel_reservation(reservation_id='9HBUV8')  ← WRONG!
Task fails: Policy prohibits this cancellation

What happens at other offsets:

Agent checks the same reservations
Agent determines none meet cancellation requirements
Agent explains policy to user, does NOT cancel
Task passes: Correctly followed policy

Insight: The baseline agent is more willing to take action (cancel) even when it violates policy. Offset agents are more cautious and correctly refuse. This contradicts the "conservative baseline" hypothesis for this specific task type.

Complex multi-step task: change return flights to fastest same-day option, stay in economy, add baggage, use smallest-balance gift card.

What happens at baseline (2024):

Agent searches flights for 2024-05-27
Agent calls: update_reservation_flights(flights=[...dates: 2024-05-27...])
Agent calls: update_reservation_baggages(...)
Task passes: Both updates succeed

What happens at other offsets:

Agent searches flights for 2029-05-26 (date shifted)
Agent calls: update_reservation_flights(flights=[...dates: 2029-05-26...])
Tool returns: "Error: Flight HAT290 not available on date 2029-05-26"
Agent retries with different flights... same error
Agent eventually transfers to human
Task fails: Could not complete the reservation update

Insight: The offset domains encounter "flight not available" errors that don't occur at baseline. This could indicate:

1. Edge cases in date transformation (leap year handling)
2. Potential tool/data inconsistencies in generated offset domains
3. The baseline's shorter conversations may actually reflect more efficient execution, not early abandonment

pie showData
    title Task Outcomes Across Date Offsets
    "Always Pass (all offsets)" : 12
    "Always Fail (all offsets)" : 4
    "Fail Only at Baseline" : 3
    "Baseline Better" : 1
    "Mixed Results" : 30

Key insight: 6% of tasks (3/50) fail exclusively at baseline while passing at other offsets. Task 32 is the most dramatic: 0% baseline, 100% everywhere else.

Evidence strength: Moderate

At the -100 year offset (1924), the agent produces temporally confused statements that reveal an internal anchor around 2024:

"Your reservation was made on May 29, 2024, which is more than 24 hours ago (current date is June 9, 1924)."

This quote shows the model mixing 2024 and 1924 dates in the same sentence—an impossible temporal relationship. The agent's internal understanding of "now" (circa 2024) leaks through even when the simulated time is 1924.

Caveat: Investigation revealed that ~50% of observed temporal confusion stemmed from example dates in tool schema docstrings (e.g., "such as '2024-01-01'") that weren't transformed to match the offset year. After fixing this, temporal confusion was reduced by half, but the remaining instances represent genuine model behavior. The performance differences (Pass^5) were unaffected by this fix.

Behavioral patterns:

• At baseline (2024): Agent uses more explicit temporal grounding ("The current date is May 15, 2024") and shorter conversations
• At displaced dates: Agent treats dates more casually ("today (2029-05-14)") and focuses on task completion

The model appears to recognize 2024 dates as potentially "real" or "current," triggering different reasoning patterns—but paradoxically, this leads to worse performance rather than better.

Evidence strength: Moderate (but opposite to intuition)

Analysis of action rates reveals the baseline agent is actually more willing to violate policy, not more cautious:

Task 43 example:

• Baseline (2024): Incorrectly states "change of plan is covered by travel insurance" and cancels the flight (WRONG)
• +5yr (2029): Correctly states "travel insurance only covers health or weather reasons" and refuses (CORRECT)

Task 5 example:

• Baseline: 2/3 trials incorrectly offered compensation to ineligible user
• +5yr: 3/3 trials correctly refused compensation

The baseline's poor performance stems from hasty decisions rather than over-caution—shorter conversations, fewer verification steps, and accepting policy-violating requests more readily.