I actually hadn’t even considered that prospect. I assume one would have to find the derivative of the equations in question with respect to the variable presented. The average soil temperature would first have to be measured (this is actually done locally in the Adelaide Hills out at Charleston, only I haven’t investigated much further).

The damping depth (as defined in document) could be calculated from the thermal diffusivity (given in square metres per second and calculated from the air density (in kilograms per cubic metres, which is dependent on the measured temperature and MSLP), the thermal conductivity (in Watts per metres per degree Kelvin, which can be calculated from the change in surface Enthalpy of moist air with respect to measured change from maximum to minimum temperature)). The other component of the damping depth (the frequency of temperature measurements) simply depends on the time interval between temperature readings (again, as indicated in the document).

Once the damping depth is calculated, we then need the depth at which soil measurements are taken. We also need to amplitude for soil temperature fluctuations, which is defined in the document for years. One would simply need to change the calculation to suit the required temporal scale (for any timescale this would mean taking half the difference between the average maximum and average minimum temperature over the period of interest). Then it is simply a matter of applying the differential equation given under “Assumptions and Simplifications.”

Much of this stuff is simply off the top of my head and I haven’t actually tested the model put forward Nofziger, however the graphs shown in the document present favourably for interpretations of changes in soil temperature with time.

I actually think that Nofziger’s ideas might be relevant to how much moisture is able to infiltrate a soil surface. There is an idea I have been testing with daily rainfall data which, at the moment, is pretty simple, but does seem at a glance to capture some of how much rainfall on a given day is able soak into the soil, and the rate of evaporation, only it probably needs some sort of damping depth (as described in the document) as a datum so the idea is not free-floating actually has a physical connection. The shortfall in this approach, however, is twofold: lack of knowledge of soil structure and lack of knowledge about soil moisture content, other than that derived from soil temperature.

What’s really interesting about Nofziger’s main equation is it’s potential application for forecasting daily temperature changes because of its temporal dependency (dependence of temperature on time).

Thanks Naz, I'm glad it's off the top of your head because although I think I can grasp the principles behind it all it's mostly way over the top of my head.

From what you say, it looks like an (artificial) neural network forecast would take account of the variables, assuming they have a significant input to the observations.

I am presently experimenting with an 'ANN' software package to see if I can estimate future rainfalls given various possible effects from ENSO as well as cycles in the data. The trouble is, it's very very hard to eliminate the insignificant regressors (money market data is so much easier).

I actually prefer using regression statistics/functions above a threshold if 80% accuracy. I have never actually dealt with ANNs apart from in reading journal articles.

I’m also rather interested in the type of climate data (daily, weekly, monthly; and the variables) and the length of the records that Surly has, as I wonder whether comparisons can be made across the continent with the records I have, covering the period 1/1979 to 11/2004.

I have tried to predict daily temperature variations (for maximum and minimum temperature, dewpoint, surface vapour pressure and MSLP using the 1st law of thermodynamics), only it is not straightforward as I first thought, because the equation for temperature variations needs to be time-dependent for the iterations to work properly.

Log of Smoothed Maximum Temperature and Rainfall
Last 3 years at Manilla

This simple graph shows the most basic climate trends since July 2007. Now that data for May 2010 are available, full smoothing extends to the end of spring 2009.



Maximum Temperature
The days with extreme high temperature in late November 2009 clearly relate to a big Quasi-biennial Oscillation (QBO) shown here. Smoothed temperature anomalies rose, almost without interruption, from a very low value of -1.61 degrees in February 2008 to a very high value of +1.35 degrees in November 2009. That was a rise of 2.96 degrees in 21 months: 0.14 degrees per month.
Before February 2008, the temperature had been falling rapidly for a year; after November 2009, data points (not yet fully smoothed) suggest that the temperature is again falling rapidly. It may be below normal already.

I find it very strange that quasi-biennial oscillations as large as this have not been mentioned in public. For comparison, I have included the 60-year trend of rising temperature in Australia that we are (rightly) advised to be concerned about. That trend rises at 0.0014 degrees per month, only one hundredth the rate seen in this current QBO!

Extreme Values
Lines near the top and bottom of the graph show the extreme values of both smoothed Maximum Temperature and smoothed Monthly Rainfall found in my 11 years of data. It happens that the extremes of Maximum Temperature occur within the last three years. Those of Monthly Rainfall do not.
As a rule, monthly rainfall anomalies exceeding +/-20 mm and Maximum Temperature anomalies exceeding +/- 1.0 degree are equally rare.

Monthly Rainfall
Smoothed monthly rainfall anomalies were very small for the first year, then peaked at +16 mm in October 2008. They fell to -16 mm by July 2009, and stayed there for at least four months and perhaps ten months. This is a persistent mild rainfall drought.
The last three years have shown little of the common pattern, which is for temperature anomalies to follow close behind rainfall anomalies, but with the opposite sign. That pattern prevailed only for nine months from November 2008 to July 2009, at which time the rainfall anomaly stopped decreasing, but the temperature anomaly kept rising to an extreme.
Less smoothed data points since November 2009 suggest a rapid fall in temperature anomaly with no matching rise in rainfall anomaly.

I have not followed the detail of changes that should have been expected in recent months due to El Nino, etc. I would be obliged to anyone willing to put my observations in that context. From what I have seen in other threads, surely someone could confidently say: "Yes, your graph shows just what we expected to see, because..." "And what we expect to happen next is...".

Surly, it has been discussed and pretty much proven in other threads that the rainfall increase you note in your graph that corresponds with a temperature fall is fairly typical. Certainly the further nth you go into the tropicals of Australia. The simple explanation is that the temperature falls with an increase in humidity and this is tied to moist air. Late Novemeber into December is typically dryer here and as such it is hotter.

The hottest temperatures in the past one hundred years have all been tied to elnino events. Due entirely to the drier air that typically is accompanied by westerly winds so therefore comes out of our arid regions. The single biggest friend to lower temperatures in summer is cloud cover up here and if the monsoon arrives on time it is significantly cooler (in temperature at least) by mid January albeit much more humid and sticky.

The central regions of NSW are sometimes subjected to a monsoonal influence and sometimes not. So modelling would be inconsistent.Your worst droughts are evident when the cold fronts move south of the continent and the monsoon rains stay across the top end or move out into the pacific. Because NSW is subjected to both of these variations it typically suffers the worst.

Your graph currently plots a sharp drop in temperatures in coming months and a corresponding rise in rainfall. It's fair to say that is the most likely outcome as the lanina is forecast to develop in that time frame. Currently models are showing a reasonably well defined lanina by summer.

Whilst many in here have looked for strong links between the IOD ,pacific oscillation soi and others ,the fact remains that the climate is far more complex. There is certainly a link between ENSO and rainfall and we know what enso is and even when it is forming one way or the other. What we don't understand fully is its timing. Why do 40 percent of elnino typically end with the formation of a lanina? Why do the other 60 % not?

What was going on in the period from 1950 to 1979 to make it the wettest 30 year period in the past 330 years? Are we really going back into another wet 30 year period? The past 3-4 years indicate we may be. Why was the timing of the formation of the past 3 enso events out of whack? Was it the heralding of a change to a 30 year wet period? Or a trigger if you like. There is still alot clearly that our very brief (in historic terms) study of the climate has left unanswered.

Much of the theory on GW originate from studies done in the 90's as we headed towards a super elnino. Since that heat was released the data has become flawed somewhat.

There have been plenty of graphs entered into the ENSO thread that have had a certainty attached only to be dismissed soon after. The climate is massively variable and that vast body of water to our east holds a lot of the secrets. Its relationship with the sun is undeniable.

If we ever do work out exactly how it works it will become boring.

congratulations on a great reply Cold front.I really enjoyed it.

lol, was trying to discuss this in a thread on ski.com but didn't have much of a response. Have you tried using a Kalman filter? It may be more effective for what your trying to do as you are only trying to estimate a single outcome (rainfall).

An ANN would be a much more effective package over a Kalman filter when you have many forecast variables as it not only has the ability to "learn" and recognise patterns, it can assigned weighted outcomes to very complex "paths" or forecast outcomes. ANN's are still in their infancy but greatly reduce the computing power needed to run something such as a climate model. When we programmed a Kalman filter and an ANN to perform a simple forecast for the movements of a robot based on inputs to its differential, the Kalmsn filter took around 12hs to complete 300 iterations while the ANN completed it in about 1h.

From what I remember at uni, in order to program an ANN it must first "see" the pattern initially, thus building the path, much like the human brain. Once the path is created it is then weighted accordingly depending on the number of times that particular patter is recognised. This makes it difficult to use as a climate model as it can take many years to program unless you have detailed and accurate data sets to feed it. This is especially the case of a climate model with an almost infinite number of outcomes. I would assume current data records (I.e. ssts, temp trend etc) would simply be insufficient to program such a network if the desired outcome was a forecasting tool for say the Eastern Seaboard of Aus.

It has been a long time since i have done anything with this type of stuff so forgive me if ive gotten something wrong there. Would be very interested to here more about your rainfall model though Keith. Ive always thought ANN's could be very effective forecasting tools. What program are you using by the way? Matlab?

Hi David,

What I am trying to do (and it looks like I'm getting there), is to predict future monthly rainfalls using ENSO variables as independent regressors. In other words I'm not trying to predict multiple climate variables. So I'm sorry if I confused anyone in the original post.

For the purpose of this, I run a correlation analysis to see which ENSO variables are significantly correlated with the rainfall, then use those significant ones as inputs in the ANN. I used principal components analysis to narrow down over 220 sea surface temperature variables into groups. I also used cycles in the rainfalls (Fourier trig functions).

On the question of filters, I've used a Hodrick Prescott filter in other work but it's a bit subjective and probably better suited to economic data. I've heard of Kalman but haven't gone into it.

I did use multiple regression with the variables at first but ran into problems in the cycles (where too many of them were linear combinations of others). So I resorted to the correlation analysis and weeded out the insignificant variables as described above. I notice that correlation analysis has been used in certain peer-reviewed papers on ANNs to which I had referred for preliminary help. The ANN software detects the contribution of each input variable, thus allowing removal of anything of a low order, but in the case of rainfall I thought this was counterproductive especially when regression and correlation was delivering totally different and at times, contrary results. It thus seemed to me to be better to eliminate the insignificant regressors before running an ANN. In the end it seems that one has to make value judgements of the data as the software doesn't necessarily know all the finer attributes of them.

I also use ARIMA for modelling but it's not much good for forecasting beyond a bare minimum period so it's mainly just for comparison or 'academic' interest.

I use a program called Statgraphics Centurion for the initial heavy duty stats stuff (PCA etc), then another program called Forecaster XL which is an add-in for Microsoft Excel spreadsheets. It will do a forecast of the entire rainfall dataset according to how it thinks it has 'learned' the patterns in the data, and also can be used to forecast months or even further ahead, based again on those same patterns. Certain parameters are customisable but for now I'm just letting the add-in decide the best ones. Of course, one would not want to project the forecast too far into the future.

Presently I'm experimenting with rainfalls in the tropical north coast of Queensland. The forecasted modelling thus far is picking up the seasonal and other patterns in the data and looks promising.

As soon as I can get certain automation of data (copying and manipulating using Excel VBA macros) going properly I hope to be able to post some results and write up a bit of an article. I've lost count of how many times changing a row or column number in the VBA code has stuffed up the whole thing, forcing me to debug it all again.

Sorry for a long reply but it's become quite a complex process.

Thanks for your comments, ColdFront.
It is good to get some input after a long gap. I appreciate your general discussion of the dynamics of the climate.
It does seem that I must learn more about climatic controls myself. Using one person's data and another's theory does not seem to work very well.

I am puzzled that you saw things in my last graph that I don't see there. I did not plot any coming months, and the last months I did plot do not show a rise in rainfall. Indeed, I am anxious to learn why the rainfall deficit simply became static last July, while the temperature peaked in November and then fell (if we trust unsmoothed values).

Here are a couple of charts I have made from my artificial neural network modelling:



These are for Cairns, on the north tropical Queensland coast, showing its monthly rainfall since January 1948 (the earliest date of record).

The top chart shows the actual rainfall received to December 2008 and the darker blue trace on the right is the projected rainfall from the model out to December 2013.

The lower chart is more of a 'closeup', showing the actual rain from January 2000, the projected rainfall as before, and, superimposed as a thick green trace, the actual rain from January to December 2009.

The projections in both plots are based on the entire data series. As is clear, the modelling for 2009 doesn't do well at all.

In the top chart, generally the projection seems to have detected not only the main summer cycles but smaller cycles as well. These are better seen in the lower chart.

So I don't really know what can be made of all of this. Certainly December 2009 would have been wildly off course! It may be that I have to include a cyclical component for extreme rainfalls.

The original rainfall data were not filtered to omit the 16 months or so over the entire range in which zero rain was recorded. I did try the modelling without those items but there wasn't much difference. So I thought it better to leave them there; after all, they aren't corrupt data but rather are a real component of the climate in winter in that area. A power transform of 1/12 was applied as this seemed to give the best correlation between the modelled rainfall and what actually fell (R^2 ~70%). There were 43 significantly correlated variables (out of 77). These 77 included factor scores derived from a principal components analysis on sea surface temperatures covering all available data on the NOAA site, divided into averages over 10 degree grids of all longitudes, out to latitudes 40°S and 50°N. I also included several standard ENSO variables and the average temperature, relative humidity, outgoing longwave radiation, surface pressure and precipitable water for each month of the period, for the Barron North Coast district, in which Cairns is located.

So I hope this doesn't scare people away! Food for thought at least, and constructive comments welcomed, of course.

Wow nice stuff Keith. Impressive to say the least. The data analysis sounds even more complicated than the actual ANN haha. An analyist in a past life perhaps? Agreed a bit of a way to go but its clearly showing seasonal variation. I would imagine monsoonal falls will be quite difficult to predict due to the unpredictable nature and timing of monsoonal events. Have you tried using seasonal forecasts from your model rather than monthly? Using ENSO climate indicators may make it difficult to increase the resolution of your model output because they are generally seen as seasonal forecasting parameters.

Thanks David, good point about the seasonality. I will try that and see what comes out in the wash.

I've been doing more work on these rainfall data, and this time decided I'd switch to Sydney (just for a change, and to see how it shaped up). This is a chart for one of Sydney's wettest suburbs:



In an earlier post I referred to using the Hodrick Prescott filter and decided to do that in this latest effort. The chart shows the original monthly rainfall data for Turramurra from January 1950. The dark blue wavy line is the trace of the HP filter using the recommended transform for monthly data.

The red line at the end is what the ANN predicted on the filter.

I know that HP filters are meant to extract trends from data series, so this is a prediction of a trend in the series.

Now, there's one problem: Is it feasible to invert the filtered data, then add back the residuals (the difference between the original and fitted data)? If so, how is the inversion performed?

I think I'm going about all this the wrong way somehow.

I didn't have much luck with seasonal data ie 3 months at a time but did try a 3 month moving average; probably that's where I'm going to end up with this.

Manilla Smoothed Monthly Anomalies of Climate Variables
Parametric Plots
Update for June 2010

New data for June 2010 allow updating with more smoothing applied to all months back to December 2009, which is now fully smoothed. A new commentary to replace the one in Post #866367 is not yet justified. New trends may be evident in two months time, when the summer months of 2009-10 will all be fully smoothed.




If you click the image you can get quite good resolution.

Just a very quick update on the analysis work I referred to a few posts back..if it's any update at all, it's that I cannot get these networks to make reliable step-ahead predictions. Although concurrent forecasts and historical data correlate well (>98% in some cases), the forecasting ahead is worse than atrocious.

Not sure if and when I will post on this aspect again..unless some form of new approach makes itself apparent.

I look forward to anything you can come up with, Keith. Predictions about the future are the most difficult kind.

Your station data was running close to mine in smoothed seasonal trends a while ago. Is it still doing that? Like the way there was a sharp temperature peak last spring, but scarcely a trough in the rainfall that you would notice.
I'm using a big Gaussian smoothing function (from Excel) on monthly values now, but the (1:2:1)/4 seasonal smoothing we both used before gives much the same result.
I had access to Wagga data a few months ago. A lot of the anomaly pattern is the same there, but some of it is different.

I'd have to revisit what I was doing at the time..but what is the Gaussian function you are using? Is it the NORMSINV or one of those similar ones (I'm not up to scratch on some of these Excel stats functions)?

And I'm not sure if what I am now going to try will be any use either, anyway I'm getting hold of some additional data that presumably have a direct effect on rainfalls. These include temperature, pressure/thickness values, humidity and wind at various heights (as well as the surface). I knew about these influences before of course, but there's so much data out there in so many different places with so many confusing descriptions (to the amateur) that I was put off from it for a while.

I am using one of NOAA's reanalysed datasets, but (and this is going to look funny, because it probably is), just now after getting over a dozen variables into a spreadsheet, ready to assault this issue with rigor and despatch, I noticed that the temperatures in July were higher than they were in December, at all levels.

Only one reason for this. Brain in gear..I had extracted the northern hemisphere's data and not the southern. Reason: I didn't put a minus sign in front of the latitude coordinates! Confused by online instructions? Probably, but I have to admit fault..just didn't read them carefully enough.

So work is progressing. Predict or perish.

I think I have it.

Here is the result of many further long hours of rainfall modelling:



I was going to start a new thread but as I had already posted on it here I thought this is where I should continue. So now for a bit of explanation.

The chart is the result of 2 different models:a neural network and a regression model.It covers monthly rainfall from June 1950 to May 2009, with projections to December 2012.

The data itself is the average of the monthly rainfalls of certain locations in western and eastern Sydney, the upper and lower central tablelands, the Hunter district and the mid-north coast (all in NSW, of course).

Locations within each of these districts were restricted to those which had a minimum of 95% of the months with data. Others were discarded. Before modelling, a power transform of 1/12 was applied. This was mainly to force the neural network to a higher correlation with the original rainfalls. For consistency the same transform was used in the regression model. In all cases the predicted rainfalls were back-transformed by inversion of the original.

The motivation for modelling was that it was clear that rainfalls are dependent on a highly complex mix of climatic variables. This varies from one location to another. In the end I considered 245 different independent variables, including sea surface temperature anomalies in 2.5° blocks of latitude from 50°N to 40°S, across all longitudes. There were also things like wind strength, humidity, specific humidity, temperatures and precipitable water values from the surface upward to the 500mb constant pressure level. In the end, this narrowed down to just 10 variables. I'll skip detailed statistical comment (whew! thank goodness for that, I hear you say).

The chart shows the original average rainfall across the 4 climate districts and the historical rainfall predicted by the models for the same period. I have only graphed from January 2000 to make it easier to compare. The faint dotted lines on the right of the chart are the limits within which we would have 95% confidence of the forecast rainfall occurring (as measured by the regression model). In other words we would be 95% confident that the forecast rainfall would lie between the two dotted lines (as read off from the vertical axis on the left). You will notice that most of the neural network model rainfall projection falls within these confidence intervals, indicating that by and large, the models seem to be at least reasonably useful.

The 2 models performed rather differently to each other and I thought that a reasonable guess could be had by averaging them. This is what the bright red line represents. Individual model projections are also shown. The neural model seemed to be much more sensitive to extreme rainfalls than the regression model.

The models seem to suggest that they have predicted the occurrence of La Nina in the latter half of 2010, with very wet weather expected in November. I found however that individual ENSO predictors such as NINO34 and NINO4 were insignificant in the regression model. It may be that it has failed to take account of those variables. It would be instructive to rerun the whole thing from a more recent time, say 1980, and compare results. However I've not done that at this stage as this has been more of a learning exercise so far.

I hope people find this of interest (and not too heavy!).

Keith, your graph appears as just a little icon which won't make an image for me. Maybe it is just my tired computer.