The Timezone Bug That Cost Us $50K (And Nearly My Job)

The 3 AM Wake-Up Call

My phone exploded with notifications at 3:17 AM on a Tuesday. Slack messages. PagerDuty alerts. A missed call from our CEO. My stomach dropped before I even read the first message.

"Why are customers being charged twice?"

I was the senior backend engineer who'd shipped our new billing system two weeks earlier. It had been running flawlessly. Until it wasn't.

The Perfect Storm

Here's what happened: Our SaaS platform charged customers at midnight in their local timezone. Seems reasonable, right? We had customers across 40+ countries, and we wanted to be respectful of their business hours.

The system worked beautifully in testing. It worked perfectly in production for 13 days. Then came daylight saving time.

On that Sunday night, when clocks "fell back" in most of the United States, our billing system entered a time loop straight out of a sci-fi nightmare. Between 1:00 AM and 2:00 AM happened twice. And our system? It charged customers twice.

By the time I silenced the alerts, we'd double-charged 1,247 customers. The refunds, processing fees, support costs, and goodwill credits totaled $50,000. But the real cost was trust.

The Code That Betrayed Me

Here's the innocent-looking code that caused the chaos:

# ❌ THE DANGEROUS WAY
from datetime import datetime
import pytz

def schedule_billing(customer):
    customer_tz = pytz.timezone(customer.timezone)
    now = datetime.now(customer_tz)

    # Schedule for midnight tonight
    billing_time = now.replace(hour=0, minute=0, second=0, microsecond=0)

    if now.hour >= 1:  # Already past midnight, schedule for tomorrow
        billing_time += timedelta(days=1)

    schedule_task(billing_time, charge_customer, customer.id)

The problem? During DST transitions, "midnight tonight" is ambiguous. When clocks fall back, there are literally two midnights. The replace() method doesn't handle this—it just picks one, and our scheduler hit both.

The Lesson: Never Trust "Now"

After three sleepless nights, countless coffees, and one very uncomfortable board meeting, here's what I learned:

1. Store everything in UTC. Everything.

Your database should only know UTC. Timezones are a display concern, not a storage concern.

2. Use timezone-aware datetimes religiously.

Naive datetimes (without timezone info) are bugs waiting to happen.

3. Never schedule based on local time.

Schedule in UTC, convert for display only.

The Fix

Here's how we rebuilt the system:

# ✅ THE SAFE WAY
from datetime import datetime, timezone
from zoneinfo import ZoneInfo  # Python 3.9+

def schedule_billing(customer):
    # Always work in UTC
    now_utc = datetime.now(timezone.utc)

    # Calculate next midnight in customer's timezone
    customer_tz = ZoneInfo(customer.timezone)
    now_local = now_utc.astimezone(customer_tz)

    # Get next midnight in their timezone
    next_midnight_local = (now_local + timedelta(days=1)).replace(
        hour=0, minute=0, second=0, microsecond=0
    )

    # Convert back to UTC for storage/scheduling
    next_midnight_utc = next_midnight_local.astimezone(timezone.utc)

    # Store UTC timestamp with customer ID
    schedule_task(next_midnight_utc, charge_customer, customer.id)

    # Add idempotency key to prevent duplicate charges
    idempotency_key = f"{customer.id}:{next_midnight_utc.date().isoformat()}"
    return idempotency_key

Key improvements:

• UTC everywhere: All scheduling happens in UTC
• Explicit timezone conversion: We convert to local time only to calculate the target, then immediately back to UTC
• Idempotency keys: Even if the scheduler glitches, we won't charge twice
• DST-aware: astimezone() properly handles DST transitions

The Idempotency Safety Net

The real hero was adding idempotency:

def charge_customer(customer_id, idempotency_key):
    # Check if we've already processed this charge
    if redis_client.exists(f"charge:{idempotency_key}"):
        logger.info(f"Skipping duplicate charge: {idempotency_key}")
        return

    # Set key with 48-hour expiration
    redis_client.setex(f"charge:{idempotency_key}", 172800, "1")

    # Now safe to charge
    process_payment(customer_id)

This meant even if our scheduling logic failed again, we'd never double-charge.

The Takeaways

For you, the developer reading this at 2 AM before your deployment:

1. Test DST transitions explicitly. Set your system clock forward and backward. It's tedious. Do it anyway.

2. Timezones are not offsets. UTC+5 isn't a timezone—it's an offset. Timezones have names, histories, and political baggage.

3. Use ISO 8601 for everything. 2024-03-10T08:00:00Z is unambiguous. 03/10/2024 8:00 AM is a lawsuit waiting to happen.

4. Add idempotency to financial operations. Always. No exceptions.

5. Monitor for duplicates. We now alert if the same customer is charged twice within 12 hours.

The Aftermath

I kept my job. Barely. We kept most of our customers. Barely. But I gained something invaluable: a healthy paranoia about time.

Now, whenever I see datetime.now() in a code review, I break out in a cold sweat. And I ask: "Is this UTC? Are we handling DST? What happens during a leap second?"

My teammates think I'm paranoid. But I'm the guy who cost the company $50K because I trusted time to be simple.

Time is never simple.

---

Have your own datetime horror story? We should start a support group. Meeting time: TBD (timezone complications).