Storm Prediction Center
D. Greg Harmon
NWS Sioux Falls SD
As part of a presentation at the NWS/Texas Tech Severe Storms Conference (Lubbock, 9 Feb 1998), the presenter/authors conducted an exercise where individuals in the audience were asked to assign F-scale damage ratings based on photos displayed on a scene-by- scene basis. Multiple photos were provided for each scene along with verbal descriptions, to aid the rating process. Each site was numbered in red in the upper right corner of the slides. The audience members rated the damage on multiple-choice worksheets, with room in each damage scene's section for justification and comment. This exercise since has been done at the National Severe Weather Workshop (Norman OK, Feb 2003) and St. Louis University (Nov. 2003) as well. Here were the slides used, to refresh the memories of those present...
DAMAGE EXERCISE SCENES: Click on each thumbnail for full size slide. Scene numbers are in red in the upper right corner.
GENERAL EXPERTISE OF RESPONDENTS:The Lubbock and Norman audiences were each composed of a wide variety of people with at least enough interest in the damage rating process for them to desire to attend. Many were NWS forecasters with greatly varying experience levels in damage surveying. Like NWS forecasters as a whole, few had personally witnessed F5 damage, in order to have a mental benchmark for the full spectrum of tornado damage. [Spencer was officially rated F4.] There was a considerable contigent of storm chasers and spotters and, at Lubbock, several professional wind engineers. At St. Louis, the audience was les diverse, composed of university meteorlogy faculty and graduate and undergraduate students. Besides the wind engineers, only a few of the audience members in any of the venues had any formal training in damage analysis techniques, outside rudimentary illustrative memoranda that some of the NWS personnel had read on-the-job. No NWS personnel in attendance had formal structural or architectural engineering education, to our knowledge. One audience member in Lubbock was a professional engineer with a Master's degree in meteorology and extremely extensive survey experience. For many others, this was the first time they had ever attempted to rate damage. So, as is collectively true with those who are commonly tasked to survey wind damage events, the spectra of expertise and experience were immense.
SURVEY RESULTS:The following images summarize some statistics from each workshop location based on the multiple-choice responses. They are stacked for ready comparison and contrasting.
As expected, there was substantial disagreement among participants on the F Scale rating of each damage scene. In Lubbock, at least 3 scales (scene 6, the apartments) and as many as 5 scales (scene 5, the fire station and water tower) were assigned in a scene. Although the fire station and water tower (Scene 5) generated the biggest range of F-scale ratings, the relatively large standard deviation of Scene 3 indicates people had the most overall difficulty rating the shed. People seemed to agree most closely about the damage in scene 6 (the apartments), with the smallest standard devation of ratings from any scene. [This agreement, however, was not as close as that for Scene 1 from teh Norman and St. Louis audiences.] Though more people rated scene 4 (fire station and water tower) F5, several also rated it as low as F1; so the scene with the highest average damage rating ended up being Scene 2 (non-anchored house swept away).
The Norman workshops yielded the largest number of participants, at over 100; and the audience was similarly diverse in experience and expertise as at Lubbock. As with the original Lubbock exercise, the greatest confusion or uncertainty about the application of the F-scale appeared to be with the anchored metal shed (Scene 3), which is no surprising considering its dissimilarity with any standard or common examples in damage survey materials. All six levels of the F-scale were used, meaning someone rated the scene F0 and someone else rated it F5. Scene 3 accordingly produced a standard deviation of nearly one whole F-scale (.93). The next most difficult scene for the Norman audience to rate was the fire station and water tower (Scene 5). people seemed to agree most closely on Scene 1 ("slider" house and anchored mobile home).
In St. Louis, the audience of mainly college students and faculty still had the most difficulty with Scene 3 (the metal shed). As in Lubbock and Norman, the standard deviation approached a whole F rating (.93); and that scene tied for the largest number of different F-scale values assigned to the damage therein. Next hardest to rate appeared to be the obliterated but unanchored frame house (Scene 2 with a standard deviation of .81). Though this group of participants was only about a quarter the size of the Norman collective, the house and mobile home (Scene 1) also was the easiest to rate, with the same standard deviation of responses (.42 of an F scale). Audience members, in the mean, rated this damage example close to F1.
Though the psychology of rating tornado damage can be argued ad nauseum, one thing is very clear from these results: rating damage using the F Scale is highly subjective and variable, as asserted by Doswell and Burgess (1988).
Some people chose not to assign ratings at certain scenes, or to assign "F-unknown." This accounts for the variation in sample size among scenes; for such entries were not computable. Where respondents circled two adjacent ratings, uncertainty and/or the intent to assign borderline ratings were assumed. Though real-world damage rating is done in whole F-scale integers only, we allowed for some uncertainty by allowing entries where two adjacent ratings were circled. For statistical purposes only, such choices were assigned the half-integer level between circled ratings (e.g., 3.5 where F3 and F4 were both circled). Also, due to time constraints, no attempt was made to categorically sort responses by any measure of expertise or experience. These stats are provided for informational purposes only and do not represent a formal analysis!
SELECTED WRITTEN RESPONSES
Here are some justifications and comments provided by respondents, after the rating each assigned to the scene. No endorsement or refutement of these comments is implied by the presenters, nor do we claim any of them are "right" or "wrong." These comments are provided as-is (edited only for spelling and capitalization), mainly to illustrate the tremendous subjectivity of the current system for rating damage. [Some of the comments might prove rather entertaining as well.] The above stats for each scene are provided on its page also.
Doswell, C.A., and D.W. Burgess, 1988: On some issues of United States tornado climatology. Mo. Wea. Rev. 116, 495-501.