I build Phytotron, an open benchmark for greenhouse climate controllers (github.com/growmatics/phytotron, AGPL). It runs on GreenLight-Gym2, a 28-state ODE model of a tomato greenhouse. Every result it reports is simulated. I calibrated the model against a measured season and I am not satisfied with my own method. I would like to hear how you would do it.
The data: Autonomous Greenhouse Challenge 2019, Wageningen, CC0. Six glass compartments of 96 m2, cherry tomato, December 2019 to May 2020. Indoor climate and actuator states every 5 minutes. Heat only as daily totals. Each compartment was run in closed loop by a different controller, five by teams' algorithms, one by growers.
What I did first was open-loop replay: the logged actuators go into the model, nothing corrects it. The simulated climate is compared with the measured one. Eight multiplicative factors on ten parameters (cover convection, screen air exchange, floor conductivity and thickness, soil conductivity, leakage, vent discharge, pipe exchange), bounded least squares on 5-minute residuals of air temperature, RH and CO2, scaled by 2 C, 5 points and 150 ppm. Twelve windows of 11 days over three compartments, 3 days spin-up each. The fourth compartment held out for its whole season.
What came out. Held-out air temperature MAE 2.17 C before, 1.81 after. Night bias -2.33 to 0.04 C. RH got worse, 5.97 to 7.16 points. Four factors ended on their bounds.
What I checked afterwards, and what it showed.
- Naive floor: yesterday's value at the same time predicts the room to 0.66 C. The model in open loop does 1.81.
- Identifiability (finite-difference Jacobian, Brun's collinearity index): one factor, floor conductivity, has no measurable effect and is collinear with floor thickness. The set of eight sits at 9 to 11 on Brun's index.
- A fit of only the four identifiable factors does better on the hold-out than the ten-parameter fit. Fewer parameters won.
- Absolute humidity instead of RH in the residual moves the solution to a different point at the same cost. The surface is flat.
- Thresholds, which are what the benchmark actually scores: the measured room spent 809 hours above 90% RH, mostly in winter. The calibrated model gives 177 hours, mostly in May. The room was above 30 C for 33 hours; the model givesĀ 300. Lower MAE, worse at the thresholds.
Where I think it is weak.
Closed-loop data. The controller hid the model's errors during the experiment. Inputs are correlated with the disturbances. I used simulation-error least squares with no noise model. Is there an identification method for a fixed nonlinear ODE structure on closed-loop records that you would trust more? Prediction-error with a noise model, multiple shooting, MCMC on the factors, something else?
The fitness metric. MAE is not what a crop cares about. I ended up with hit rate and false-alarm rate on threshold crossings, and hours above threshold per month. Is there a standard statistic for "does the model reproduce exceedance hours"?
Input uncertainty. Heat is a daily total spread by pipe temperatures. Fogging was not logged. How would you treat errors in the inputs rather than pushing them into the parameters?
The selection rule. I chose the fit on the training windows and the hold-out only reports. Now the hold-out says a smaller model is better. A rotating hold-out across the four compartments is my next idea. What would you do?
The water side. The vapour content error is 2.4 g/m3 under every variant, twice the naive floor. Night transpiration, condensation on the screen, unlogged fogging, or leakage?
Sugestions are more than welcome. And if you want to contribute. The benchmark is public and takes any controller through a small interface. The calibration scripts and records are on a branch under review and will follow. Issues and discussions are open on the repo. I would also take a pull request that replaces my fit with a better one, and I will publish the comparison either way.
