Debugging Delayed Notifications in a High-Volume Scheduling System
Tracing missed clock-in and clock-out delays to an out-of-memory scheduled job, then bounding memory with SQL pagination and adding Grafana visibility.
At a Glance
When I Work sends missed clock-in and missed clock-out notifications based on each customer's configured timing. A scheduled PHP job queried shift and time-entry data to decide what needed to be sent.
Customers reported that those notifications were missing or arriving late. During peak periods, most notifications did not go through; those that did could arrive 5–30 minutes after the configured threshold. The issue could not be reproduced in lower environments, which did not have production-scale data volume.
Production logs showed out-of-memory errors clustered around common shift start and end times. Rather than raising the process memory limit, the job was changed to paginate in ~10,000-record batches, with additional logging and a Grafana dashboard for failures.
After pagination, notifications were generally processed within a minute of the configured threshold. Memory usage stayed bounded during peak windows, and the team gained visibility into scheduled-job failures instead of learning about them from customer reports.
The Problem
At When I Work, customers began reporting that missed clock-in and missed clock-out notifications were either not arriving or arriving well after the timing they had configured. During peak periods, most notifications did not go through. Those that did arrive could be 5–30 minutes past the configured threshold.
Those alerts are how managers learn someone did not punch as expected. When they are late or missing, it is harder to catch attendance issues while they can still be corrected. Failures concentrated around high-volume shift start and end times.
The workflow ran as a scheduled PHP cron job: query shift and time-entry data, decide which notifications were due, then send them. The same job looked fine in lower environments. Those environments did not have production-scale data, so the failure never appeared there.
Investigation
Production logs were the only reliable signal. Successful executions looked normal. Occasional out of memory errors were easy to miss if you only sampled healthy runs.
Searching specifically for those errors and graphing them over time made the pattern obvious. Failures clustered around common shift start and end times—roughly 8:00, 8:30, 9:00, and 9:30 in the morning, and 4:00 and 4:30 in the afternoon.
That is when large numbers of employees begin or end shifts. During those windows the job was querying a much larger set of shifts and time entries. The process would run out of memory, terminate, and restart, which explained why most notifications in those windows did not go through, or arrived late.
Investigation path
Production logsDuring peak windows, most of these notifications did not go through.
Lower environments lacked the data volume that triggered the failure.
Healthy runs hid the signal; plotting OOM errors showed a time-of-day pattern.
Spikes lined up with common start and end times and unbounded query results.
Out-of-memory errors by time of day
IllustrativeRelative frequency only—not production counts. Peak bars mark common shift start and end windows.
Root Cause
The cron job loaded the full query result into memory before processing. Memory usage scaled with the size of that result. Peak shift windows pushed the process past its available memory.
When the process crashed and restarted, notification work in those windows was dropped or delayed. From the customer's side it looked like a notification bug. Underneath, a background job could not survive its busiest hours.
The Solution
Raising the PHP memory limit would have postponed the crash without changing the scaling behavior. The next traffic peak would have found the new ceiling.
The processing strategy changed instead. The job retrieves about 10,000 records, processes that batch, then fetches the next until the full result set is handled. Memory stays proportional to one batch, not the entire matching set.
Load the full shift and time-entry set, then process. Peak windows spike memory until the process crashes and restarts.
Fetch a batch, process it, fetch the next. Memory stays predictable even when total matching rows grow.
Batched notification job
Illustrative flowPHP cron kicks off missed clock-in / clock-out evaluation.
Paginated SQL reads the next ~10,000 matching records.
Determine which notifications are due for that slice of data.
Deliver due alerts, then repeat until the result set is complete.
Once the paginated job was stable, notifications were generally processed within less than one minute of the configured threshold. The improvement came from changing the scaling behavior of the job rather than simply increasing available memory.
During peak shift start and end times. Those that arrived late were 5–30 minutes past the configured threshold.
Reliable delivery during peak periods after the job was stabilized.
Improving Observability
The investigation also showed the team did not have enough visibility when this scheduled job failed. Successful runs looked healthy. Crashes were easy to miss until customers reported missing notifications.
Additional logging covered job execution and each paginated batch, along with error-rate visibility. A Grafana dashboard made it possible to watch failures during peak periods and confirm the batched job was behaving.
The goal was to catch similar failures in the job itself—before they showed up as customer tickets.
What I Learned
The customer-facing symptom looked like a notification problem. The actual issue was scalability and observability in a background process.
- Follow the evidence. Sampling successful runs hid the failure. Graphing the errors revealed the shape of the problem.
- Volume changes the system. Production behavior can diverge sharply from test environments when the difference is data size, not business logic.
- Background jobs need the same scrutiny as APIs. A silent cron restart is still a production incident.
- Fix the scaling characteristic. More memory would have delayed the next crash. Batching kept usage bounded as shift volume grew.
- Trace through layers. Senior work is often staying with a customer symptom until the real constraint—here, unbounded query results—is visible.
What I Would Improve Today
The dashboard closed the original visibility gap. Today I would make failure modes even harder to ignore:
- Explicit alerts on job failures, not only a dashboard that someone has to open
- Execution duration and record counts / throughput per run, so a slow or stalled batch is obvious
- Alerts on abnormal behavior—error-rate spikes, sudden drops in processed records, unusually long runs
- Stronger production-like load testing for scheduled jobs that only fail when result sets reach peak size
Interested in discussing this architecture?
I'm currently open to Senior Frontend, Product Engineering, and Tech Lead opportunities. Let's talk about technical planning, high-concurrency systems, or frontend craftsmanship.