Outcome being modeled
Enterococcus concentration on the log scale. The site's median historical log level acts as the baseline, and the rain model predicts the residual above or below that baseline.
Prediction Methodology
How NYC Water Check estimates bacteria levels after rain
This page documents the current model, the historical backtests behind it, and the rainfall-source validation work used to decide how repaired local gauge-derived fields should drive the prediction pipeline.
The goal here is not marketing copy. This is a working methods note for a public-facing prediction system built on real historical shoreline bacteria and rainfall data.
Dataset snapshot
- Samples with 7-day rain
- 14,582
- Sites in model set
- 159
- Best test RMSE
- 0.7349
- Best threshold accuracy
- 64.9%
Overview
Current prediction logic
The production estimate starts with each site's own typical bacteria level, then adjusts that baseline using rainfall features learned from the historical record. Missing rain is excluded rather than treated as zero, and every lag follows calendar dates rather than the order in which samples were collected. It is a pooled citywide model rather than a fully custom coefficient set per site, because fully site-specific fits overfit the data.
Core rain windows
The model separates the most recent 48 hours, days 2–3, and days 4–6. Where continuous daily rain is available, it also builds exact calendar-week totals for the preceding four to eight weeks so a gap in sampling can never masquerade as a one-week lag.
Model family
Ridge regression with log-transformed rain features. This reduces instability from large storm days and keeps the coefficients from swinging too far on sparse patterns. Candidate models are compared on the same future-only holdout rows, then the winner is refit on all eligible history.
Backtest
Historical model performance
These backtests use chronological train/test splits so the model is always evaluated on future samples, not shuffled history. Lower RMSE and MAE are better. Threshold accuracy measures whether the model correctly places a sample below or above the 35 MPN swim threshold.
| Candidate | Features | Test RMSE (log) | Test MAE (log) | Threshold accuracy |
|---|---|---|---|---|
| extended-memory | recentRain, midRain, lateRain, priorWeeksRainMemory, dryWeeks, seasonSin, seasonCos | 0.7349 | 0.5659 | 64.4% |
| long-memory | recentRain, midRain, lateRain, lagWeek1, lagWeek2, lagWeek3, lagWeek4, lagWeek5, lagWeek6, lagWeek7, lagWeek8, dryWeeks, seasonSin, seasonCos | 0.7352 | 0.5585 | 64.7% |
| 7d-distributed | recentRain, midRain, lateRain | 0.7379 | 0.5712 | 64.9% |
| 48h | recentRain | 0.7420 | 0.5765 | 63.8% |
| baseline | site baseline only | 0.7872 | 0.5999 | 63.3% |
Best numeric fit
extended-memory had the lowest held-out RMSE at 0.7349, which is only a modest gain over the simpler pooled variants.
Best threshold result
7d-distributed remained the best rain-only threshold model we tested at 64.9%.
Main modeling lesson
Fully per-site models looked attractive but performed worse overall. Pooled models with site baselines generalize better on future samples.
Decision Metric
What happens if we optimize for the swim threshold directly?
We also tested simpler strategies that predict only the safe / unsafe threshold decision. These are useful comparisons, but they optimize a different outcome from the production regression, which is selected for numeric bacteria error and then translated into condition bands.
Best rain-only threshold performer we tested across the held-out historical set.
Per-site rain cutoff using only the most recent 48 hours.
Simple bucketed rule based on seven-day rain rather than a regression.
Rainfall Inputs
Which rainfall source appears more trustworthy?
We compared the stored site rain fields against the assigned NOAA daily station records and did the same for Open-Meteo historical forecast data. The validation window covers March 23, 2021 onward. DOHMH beaches, whose source dataset contains no rain, use a single consistent ERA5 reanalysis series backfilled at each beach location through Open-Meteo.
| Source | Compared rows | Correlation to NOAA | MAE | RMSE | Within 0.10 in |
|---|---|---|---|---|---|
| Raw stored gauge-derived rain fields | 30,469 | 0.505 | 0.138 in | 0.418 in | 76.7% |
| Open-Meteo historical forecast | 30,469 | 0.321 | 0.167 in | 0.502 in | 75.0% |
Result: the raw stored gauge-derived fields are materially closer to NOAA station observations overall, so the evidence does not support replacing the entire rainfall source with Open-Meteo. Across sites, the raw stored fields beat Open-Meteo at 69 sites versus 37 for Open-Meteo. The model metrics above use the repaired version of those stored fields and drop incomplete seven-day windows instead of silently converting missing values to dry days.
Outlier Review
Where the rain fields were repaired or still look suspect
The main risk is not that the entire stored rainfall source is bad. The current model now repairs the clearest bulk-filled or misaligned sample-date clusters before building rain features, while less certain cases stay on the review list.
`precipitationDayOf = 0.10 in` and `precipitationPreviousDay = 7.13 in` were swapped before feature generation.
`precipitationPreviousSat = 4.45 in` was removed from the prior-week rain window.
`precipitationPreviousMon = 2.54 in` was removed after NOAA and Open-Meteo both showed a near-dry day.
Raritan sites shared `precipitationPreviousTue = 0.28 in`; this stays on the watchlist until it can be confirmed site by site.
Operational conclusion: keep the stored gauge-derived rainfall as the primary source, apply targeted repairs for confirmed sample-date + field problems, and leave ambiguous clusters out of the override set until they can be checked against station-level evidence.
Next Steps
What would make the prediction system stronger?
The current rain-only model is useful but not definitive. With the clearest rain outliers repaired, the most promising next gains are broader QA automation and richer predictors, not just more rain windows.
Expand rain QA coverage
Turn the manual outlier checks into a recurring station-level validation pass so future imports can be flagged before they affect predictions.
Train directly for the threshold
If the product goal is a safe / unsafe call, the next model should optimize that decision directly rather than optimizing only numeric bacteria error.
Add more physical context
Tide timing, recent prior bacteria readings, and station-level weather context are the likeliest variables to improve the model beyond the current rain-only ceiling.