The Significance of Significance
Finding significance in educational environments may not be as “significant'” as it is cracked up to be. What we are saying is that because there are so many variables and because the statistics behind educational research allows up to a 5% error term, we are really not “proving’ anything. There are those (i.e. Ellenberg, 2014) who suggest that what we most often are looking for, especially in non-precise instances, simply to be playing the percentage game and not always trying to be right but, conversely it might be enough to say we are not wrong. In this case, probability and predictability are our friend. In other words, we want to view assessment in terms of how far removed we are from a simply coin toss. Estimating outcome assessment is not very precise (nor does it need to be) and can be viewed as a continuum:
|At zero, virtually no learning is taking place||
< — 10% – 25% – 50% – 75% – 100% –>
At 100, we have “proof” that learning goals are met
Fifty percent represents a virtual “coin toss”
This models the same valuation that we assign to the concept of reliability… i.e., anything over 50% is considered approaching reliability. We almost never reach 100% but are relatively satisfied. In imprecise learning situations (those in which we do not or do not wish to administer testing), we always want to be better than a coin toss and are also relatively satisfied when we can demonstrate it. Granted, this view is an overly simplified version and a lot more statistics are involved but that is the subject of our evaluation and analyses courses. What we are demonstrating here is a simplified estimate of internal validity.
How do we approach internal consistency/validity without testing?
One effective way to approach internal validity is to base your design on as many complementary instructional design principles/premises as possible and essentially ‘cobbling’ them together … the newly formed composite is what provides the ability/power to make some predictions… (the whole is greater than the sum of its parts) Following is a list of design processes/premises that we have already introduced in this course that will assist in the development of an assessment plan. The more that are utilized the better. It is not a complete list, but is a starting point.
PrinciplesThe bottom line is that learning most likely is best viewed as a process and not an end result. Sometimes we need to be able to re-frame the question to get to the answer were are looking for (that is where design thinking comes into play and why we introduced it to you earlier in the term).