Skip to content
Expert O'clock

Python

Scheduling Python scripts safely

How to schedule Python reporting scripts safely: pick a scheduler, make runs repeatable, lock against overlaps, log failures, and test.

Talha Ansari··5 min read

Key takeaways

  • Decide the consequence of failure and name an owner before automating.
  • Pass the reporting period as a parameter so retries process the right data.
  • Make the job idempotent and guard against overlapping runs.
  • Test the schedule in the real environment and document how to rerun it.
On this page

Start with the consequence of failure

A scheduled script is a small production system. Before choosing a scheduler, decide what happens if the run fails, finishes late, or produces a wrong number. A daily sales email that must arrive by 09:00 needs different monitoring from a monthly archive nobody reads.

Name the owner, the deadline, and the acceptable recovery time. If nobody would notice a failure for a week, fix ownership before adding automation. A job with no owner is not automated; it is unattended.

Use a scheduler the team already operates

The best scheduler is the one the organisation can monitor, restart, and hand over. A cron entry on a server is simple and transparent, but it has no built-in alerting or dependency management. A workflow orchestrator adds retries, history, and visibility, at the cost of another system to learn.

Whichever you choose, run the job with the same interpreter, environment, and working directory every time. A job that works in a terminal can still fail under a scheduler because one of those settings differs.

Scheduler options and their trade-offs
OptionGood fitWatch out for
cron or systemd timerA few jobs on one serverNo alerting by default; environment differs from your shell
Windows Task SchedulerA Windows reporting stackCredentials, working directory, and interpreter path
Cloud schedulerJobs already running in a cloud platformTimeouts, permissions, and log retention
Workflow orchestratorDependent jobs and backfillsSetup cost; unnecessary for one simple script

Make the run repeatable

A scheduled job should produce the same result from the same input, and it should be safe to run twice. Pass the reporting period as a parameter rather than reading the current date inside the script. If a run fails and is retried the next day, it must still process the intended period.

Write outputs to a temporary file and replace the target only after the write succeeds. Where the output is a database table, delete and insert the affected period or use a key-based upsert inside a transaction. A partially loaded table is harder to detect than a missing file.

Guard against overlapping runs

A job that takes longer than its interval will start again while the previous run is still working. Use the scheduler's concurrency setting where it exists, and a lock file as a second line of defence. The lock should fail fast with a clear message rather than queue silently. The example uses fcntl, which is available on Linux and macOS; on Windows, use an exclusive lock file or the scheduler's concurrency control.

Exit when another run already holds the lockPython
import fcntl
from pathlib import Path

lock_path = Path('run/monthly_report.lock')
lock_file = lock_path.open('w')

try:
    fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
    raise SystemExit('Another run is still active')

try:
    run_report()
finally:
    fcntl.flock(lock_file, fcntl.LOCK_UN)
    lock_file.close()

Log enough to diagnose a failure

Write a log line at the start and end of the run with the period, input count, output count, and duration. Log validation failures with enough context to identify the offending record. A stack trace alone rarely tells the business owner what to do.

  • Start and finish timestamps, plus the reporting period processed.
  • Input and output counts, with any difference explained.
  • Validation failures, quarantined rows, and the source file.
  • The output path or table name, so the result can be found.
  • A non-zero exit code whenever the run is not safe to publish.

Test the schedule, not only the script

Run the job manually with the same user, environment, and working directory the scheduler uses. Check permissions on the output directory, the availability of the interpreter, and the location of configuration files. A script that works in a terminal but fails under cron usually depends on an interactive shell setting.

Test a failure: remove the input file or point the script at an empty folder, then confirm the alert arrives. Document how to rerun a period, where the logs live, and who to contact. The runbook is part of the automation.

Frequently asked questions

Is cron still appropriate for reporting jobs?+

Yes for simple jobs on a single server, provided you add logging, alerting, and a lock. Move to an orchestrator when jobs depend on each other, need backfills, or must be visible to people who do not use the server.

How do I stop a job running twice?+

Use the scheduler's concurrency setting and a lock file, and make the output idempotent so a second run cannot duplicate rows. The lock should fail fast rather than wait for the first run to finish.

How should I handle time zones and daylight saving?+

Schedule in a fixed, documented time zone, store timestamps in UTC, and check how the scheduler behaves at a daylight-saving transition. Avoid scheduling critical jobs in the repeated or skipped hour.

What should happen if a run fails?+

Exit non-zero, alert the owner, keep the previous good output in place, and record the failure in the log. Rerun the intended period after the cause is fixed rather than letting the next scheduled run overwrite the gap.

Need help applying this?

Improve your data cleanup & pipelines.

Review the service scope or tell us what you are trying to improve. We will reply with a clear next step.

Discuss this serviceView service
Free KPI definition template

Related reading