Page 2 of 3 < 1 2 3 >
Topic Options
#971867 - 08/03/2011 15:54 Re: Rainfall prediction modelling [Re: camtang]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
Yes but it still needs some work. I hope to post something here before too long.

Top
#973546 - 12/03/2011 10:19 Re: Rainfall prediction modelling [Re: Keith]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
Might need more caution comparing results here. Bellenden's rainfall for February is now showing 2298mm. I'm positive I didn't misread anything before. It's not been quality-controlled.

If that figure is correct then I probably picked a bad time..after all, it's been the biggest wet season ever in the tropics.

As for Sydney's, it's not responding to my 'treatment'. For now, anyone interested in Sydney's or its suburbs' rainfall should look at Page 1 of this thread.

Top
#977527 - 21/03/2011 19:21 Re: Rainfall prediction modelling [Re: Keith]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
After a lot of experimenting, I've come up with the following outlook for Eastern Sydney:



The data are a composite of several of Sydney's eastern and northern suburbs, certainly not all the reporting rainfall stations. The dark blue trace is the actual data, the other trace, the predicted data going out to March 2012. The criteria for inclusion were that there had to be less than 5% of data missing.

I'm going to see if I can refine it further so I'll avoid my usual long dry (excuse the bad pun) commentary. I'll just say that the model explained over 63% of the variation in the rainfall, which I analysed as a monthly series from January 1950 to December 2010.

This modelling is not primarily designed to predict specific months' rainfalls so much as to identify or predict broader scale variations. I therefore think that I have a reasonable result given that La Nina is expected to wane to some degree, before possibly returning later in 2011. The model suggests a fairly dry winter ahead.

When I've done some more work I'll probably post here or simply write up something and post it on my website.

Stay tuned.


Top
#977706 - 22/03/2011 11:05 Re: Rainfall prediction modelling [Re: Keith]
CoastalStorm22 Offline
Weatherzone Addict

Registered: 10/07/2006
Posts: 2679
Loc: Lane Cove, Sydney, NSW
Hi Keith,

I always look forward to the results of your modelling, however I never seem to be able to view the graphs you post?

Top
#977742 - 22/03/2011 12:55 Re: Rainfall prediction modelling [Re: CoastalStorm22]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
Is anyone else having issues with my graphs? If so I'll revisit the security options but I thought I fixed that before.

[EDIT] I've just had a look at them anyway. Please anyone let me know if there's still an issue. Thanks.

Top
#977779 - 22/03/2011 14:20 Re: Rainfall prediction modelling [Re: Keith]
davidg Offline
Weatherzone Addict

Registered: 01/06/2008
Posts: 2204
Loc: Glenbrook/Penrith
Hi Keith, no probs veiwing for me.

Im liking your predictions of a dry Winter though. This bodes well for some decent cold spells as a dry winter suggests a predominantly westerly regime. The ESNO state returning to cool-neutral after such a strong Lanina with such huge falls is probably the perfect setup for a decent snow season IMO as it means plenty of moisture but weaker easterlies, of course mother nature will do its best to prove me wrong im sure lol. Will be following with baited interest, keep up the good work!

Top
#977781 - 22/03/2011 14:25 Re: Rainfall prediction modelling [Re: davidg]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
Thanks David, knowing my luck, my prediction will go off the rails..but anyway I'm refining the torture I'm dishing out to my muddle oops model so it will hopefully not require further drastic surgery laugh. Your thoughts on the westerlies were exactly mine.

Top
#977789 - 22/03/2011 14:48 Re: Rainfall prediction modelling [Re: Keith]
CoastalStorm22 Offline
Weatherzone Addict

Registered: 10/07/2006
Posts: 2679
Loc: Lane Cove, Sydney, NSW
All good at my end now Keith. Westerly dominated winters here are about as exciting as watching paint dry, so my hope is that we have a really wet winter with lots of ECL's and cold pools, however if El Nino manages to get a foot hold then it could indeed be a rather dry winter. This might be a stupid question, but dose your model take into account the potential state of the ENSO moving into winter Keith?

Top
#977814 - 22/03/2011 15:36 Re: Rainfall prediction modelling [Re: CoastalStorm22]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
Hi CS22, firstly, I'm glad that's working for you now.

No, that's not a stupid question by any means. Yes, the models do take account of projections. I have derived those using neural network forecasting. This turned out to be the most successful (though more so than others in some cases) among various techniques. This chart:



shows movements of the recent NINO34, NINO4 and Southern Oscillation indexes, with predictions out to March 2012 which were derived as above. You see that the SOI looks like dropping quite a bit this year before increasing later. The 2 NINO indicators vary accordingly. However I would have expected NINO to go down as the SOI goes up (see mid-2011 onwards). I guess this means either that there is a lag in the NINO data (the SOI is based on air pressure differences), or the prediction becomes more inaccurate the further out you go (which I would expect to be the norm). It seems from the chart that the SOI is more variable month to month than the NINO indexes, which wouldn't be surprising due to ocean heat storage..historically the NINO indexes look somewhat sluggish in comparison. It seems also that the (roughly 3 year) long-term cycles in NINO look like peaking within the forecast period and should start dropping around mid-2012.

Top
#978453 - 24/03/2011 14:08 Re: Rainfall prediction modelling [Re: Keith]
Seira Offline
Meteorological Motor Mouth

Registered: 27/08/2003
Posts: 7103
Loc: Adelaide Hills.
There is the possibility of a strong relation between the NINO indices and the ENSO one, which may or may not be reflected in the graph above...it is interesting what you have presented nonetheless smile . I am actually trying to understand the rainfall forecast potential of Southern Ocean phenomena (for SA), which might link in with what you have been presenting in this thread, but I'm not entirely sure yet.

Perhaps understandably, it is not always easy to find relations in a predictive sense (e.g. ENSO with the Southern Ocean and Southern Australia), however I do think what you are doing is very encouraging smile .

Top
#978460 - 24/03/2011 14:29 Re: Rainfall prediction modelling [Re: Seira]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
Hi Cosmic, it certainly would link in for SA because all I have to do is run the model for the rainfall there, although as you suggest, maybe not exactly tailored to aspects of your research.

Top
#979177 - 26/03/2011 18:57 Re: Rainfall prediction modelling [Re: Keith]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
I'm making some progress hopefully, in refining the modelling..having found multiple correlations between various sets of regressors. But one thing stands out apart from all that, and that is, on a 'birds eye' view of charts, the gradual rise in sea surface temperatures from 1950. My suspicions of everything that emanates from the climate science fraternity's data makes me wonder whether there's been tampering with the numbers...although..a statistical (Mann Kendall) trend test didn't disclose anything significant.

Top
#982435 - 06/04/2011 11:27 Re: Rainfall prediction modelling [Re: Keith]
Seira Offline
Meteorological Motor Mouth

Registered: 27/08/2003
Posts: 7103
Loc: Adelaide Hills.
To expand on my last comment, which is probably relevant to your modelling, there is the likelihood the NINO (SST) and SOI (pressure) indices are related concurrently.

Preliminary Regression (months):

Multiple R 0.707079
Observations 300

Top
#982457 - 06/04/2011 13:22 Re: Rainfall prediction modelling [Re: Seira]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
I have all the data for your location back to 1882 (except a few isolated missing observations). What particular NINO indices did you use? Did you use actual pressure differences or the SOI itself?

On running a regression with the SOI and the months I only get a multiple R of 0.021359. I used data from January 1985 to December 2009.

The result was much better with a number of other regressors including SSTs for various blocks of latitude/longitude.

Top
#982470 - 06/04/2011 14:26 Re: Rainfall prediction modelling [Re: Keith]
Seira Offline
Meteorological Motor Mouth

Registered: 27/08/2003
Posts: 7103
Loc: Adelaide Hills.
I have to be careful that this is not just a red herring and waste of time, put this basically what I did:

Downloaded NINO 1+2, 3, 4 and 3.4 (1/1950 to 8/2010), and corresponding anomalies.

http://www.cpc.ncep.noaa.gov/data/indices/sstoi.indices

Downloaded SOI from BoM or same site:

http://www.cpc.ncep.noaa.gov/data/indices/soi

The raw SOI.

Performed a regression analysis using all anomalies (1+2, 3, 4 and 3.4).

The 300 observations are less than half that available.

Determined which parameters were reasonable (below 0.05 confidence threshold in Excel – P-value)…removed those indices which were above this accordingly…re-processes regression, checking the signs of the parameters (whether positive or negative), for a physical mechanism:

http://www.bom.gov.au/climate/glossary/soi.shtml

…Where the SOI is essentially a normalised pressure difference across the Pacific.

I used ANOM (1+2) and (4).

It’s best that the intercept is removed in this analysis because the P-value is not statistically significant.

As a note of benefit, anything that is a SST that gets regressed against NINO indices will give a high correlation…because that’s what the NINO indices are based on. However, if you have SSTs from a different region (not a NINO region) or a different time (lag/lead) then there might be something in them.

Top
#982501 - 06/04/2011 16:28 Re: Rainfall prediction modelling [Re: Seira]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
OK thanks for that.
The intercorrelation between NINO and SST was something I observed from the start that needed care..however I may not have dealt with that adequately. I had hoped that the regression program would reveal multicollinearity between the variables, which, in any case, can be overcome by principal components which makes the variables perpendicular to one another in space (at least I think that's the general idea). Apparently ridge regression is the standard tool for dealing with that problem. Excel won't remove such problems; it relies on the user to do it by other means.

I've used lags of the regressors extensively also, as well as monthly dummy variables.

I'll do some more work and post back...hopefully I will get similar results.

Top
#982527 - 06/04/2011 18:26 Re: Rainfall prediction modelling [Re: Keith]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
I have rerun my regression for data from 1985 to 2009 (300 data points) but this time using the above NINO data.

From what I have read in various texts it's not advisable to remove the constant unless Y (in this case the rainfall) =0 when X=0. Some of the NINO parameters including the SOI fall below zero. So I'm not sure if it should be removed despite any insignificance it might have.

On running the above, the residuals were highly unstationary so I transformed the rainfall with a Box-Cox power transform. This removed the stationarity. The residuals had no autocorrelation.

In Excel, with the intercept set at zero and the data untransformed, Multiple R was 0.957793. NINO 1+2 and its anomaly were insignificant. On removal of these and rerunning, Multiple R was 0.957712.

After inclusion of the constant and rerunning, multiple R was 0.684203. The intercept was significant with 97.5% confidence. The other regressors were significant with well in excess of 99% confidence.

This is a very profitable and informative discussion. The rainfall is apparently highly dependent on NINO 3 and 4 (I just now realised I left out the SOI).

Top
#982528 - 06/04/2011 18:38 Re: Rainfall prediction modelling [Re: Keith]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
Further thought (edit timed out): the NINO regressors are highly correlated with one another. I suspect that's going to invalidate my regression.

Top
#982556 - 06/04/2011 20:49 Re: Rainfall prediction modelling [Re: Keith]
Seira Offline
Meteorological Motor Mouth

Registered: 27/08/2003
Posts: 7103
Loc: Adelaide Hills.
A high correlation between the NINO indices might be expected. They are all SST anomalies. The highest correlation seems to be between ANOM 3 and 3.4: 0.94, the lowest 0.45 between ANOM 1+2 and 4: 0.45 – period 1/1950 to 8/2010.

If the P-value (Excel) is less than 0.05 (95% confidence) it implies the relationship between the NINO index and the SOI is greater than between the NINO indices themselves (you can check this with NINO 1+2 against NINO 4), however it is still possible for the relationship to be coincidental/typical (i.e. not insightful), hence my “red herring” comment.

I removed the Y-intercept manually just to see what effect, if any, it had on the trend I produced (SOI and estimated SOI)…the effect was negligible. The intercept P-value was 0.91, way over 0.05.

The P-values for the ANOM 1+2 and 4 were in scientific notation (very small). The easiest way to do the regression is to take all the ANOMs (1+2, 3, 4 and 3.4), average them...

(i.e. ((1+2) + 3 + 4 + 3.4)/4)

...and regress that average against the SOI (1/1950 to 12/1989 – 480 obs). From a preliminary trial of that method I obtained a P-value of 4.96e-72 (Y-intercept not removed, in Excel) and Multiple R of 0.70 for 480 obs (months), which is ok, but not particularly high. Increasing the length of a NINO series used apparently seems to increase the R-value.

Top
#982565 - 06/04/2011 21:15 Re: Rainfall prediction modelling [Re: Seira]
Keith Offline
Meteorological Motor Mouth

Registered: 16/12/2001
Posts: 6453
Loc: Kings Langley, NSW
I can see what you mean by a red herring..to me it looks like a statistical trap. The SOI is of course just an index, the standardised difference in the air pressure between Tahiti and Darwin (X 10), reflecting a pressure gradient. But if that pressure gradient results from SSTs in the NINO areas then I think we have exposed the said trap. The SSTs would drive the formation of the various pressure systems that result in the pressure differences, though of course they aren't the only influences. The other thing that worried me was the use of anomalies as well as the data they are anomalies of, in the same regression. I usually use either the anomalies, or the data, but never both in the one calculation. I think they should be mutually exclusive.

I typically find that with regression problems, as you add more 'ingredients' into the 'soup', one or more other regressors become insignificant and you can end up solving the issue of multicollinearity through sheer attrition. Many of my tests are coming up with quite a large number of significant regressors in addition to the usual NINO indices. Seasonality of the SSTs I think has to be factored in as well...I assume this is why NOAA etc use 'standarised' anomalies. It would be interesting to remove the trend through decomposition of the data and use the residuals as the regressors, then add the seasonal/trend components back to the forecasted residuals to get an estimate.

Have you tried a Monte Carlo simulation of the variables to get a theoretically longer series? This of course would only mean extending the statistical properties of the series so maybe that wouldn't mean a better R-value..though one could try playing with the data by 'hindcasting' an observable trend...something I'm not sure I want to attempt.

I might try a simulation myself tomorrow and see what comes out in the wash.

Top
Page 2 of 3 < 1 2 3 >


Who's Online
6 registered (Wave Rider, Markus, DDstorm, willitrainagain, aussiestormfreak, 1 invisible), 299 Guests and 4 Spiders online.
Key: Admin, Global Mod, Mod
Today's Birthdays
annair, marestails, Quasar, typhoontim
Forum Stats
29389 Members
32 Forums
23700 Topics
1465941 Posts

Max Online: 2925 @ 02/02/2011 22:23
Satellite Image