Sue Bauer/ April 18, 2019/ Defining Who We are…

Assessment
vs.
Evaluation
Probability
and
Sampling
The Significance
of
Significance
Classic
vs.
Official View

The Significance of Significance

Finding significance in educational environments may not be as “significant'” as it is cracked up to be. What we are saying is that because there are so many variables and because the statistics behind educational research allows up to a 5% error term, we are really not “proving’ anything. There are those (i.e. Ellenberg, 2014) who suggest that what we most often are looking for, especially in non-precise instances, simply to be playing the percentage game and not always trying to be right but, conversely it might be enough to say we are not wrong. In this case, probability and predictability are our friend. In other words, we want to view assessment in terms of how far removed we are from a simply coin toss. Estimating outcome assessment is not very precise (nor does it need to be) and can be viewed as a continuum:

At zero, virtually no learning is taking place

< — 10% – 25% – 50% – 75% – 100% –>

At 100, we have “proof” that learning goals are met

Fifty percent represents a virtual “coin toss”

This models the same valuation that we assign to the concept of  reliability… i.e., anything over 50% is considered approaching reliability. We almost never reach 100% but are relatively satisfied. In imprecise learning situations (those in which we do not or do not wish to administer testing), we always want to be better than a coin toss and are also relatively satisfied when we can demonstrate it. Granted, this view is an overly simplified version and a lot more statistics are involved but that is the subject of our evaluation and analyses courses. What we are demonstrating here is a simplified estimate of internal validity.

How do we approach internal consistency/validity without testing?

One effective way to approach internal validity is to base your design on as many complementary instructional design principles/premises as possible and essentially ‘cobbling’ them together … the newly formed composite is what provides the ability/power to make some predictions… (the whole is greater than the sum of its parts) Following is a list of design processes/premises that we have already introduced in this course that will assist in the development of an assessment plan. The more that are utilized the better. It is not a complete list, but is a starting point.

Principles

The Principle of Vicarious Learning

This link will explain it fairly concisely.. and provides ties to some studies that help make the point and add validity to the premise. It ties very closely with the next principle (Social Learning)

Social Learning

The social learning most of us know about is most often tied to Albert Bandura and is founded on several psychological principles. In the museum setting we cannot overlook the visitor’s one’s peers/classmates. In a museum more often than not it is the family members that matter, as well as the cultural implications. .. or the father who makes the child read all the labels or participate in all the exhibits. Motivation based on shared cultural beliefs and language (meaning that the designer needs to be able to know/predict the learner). The exhibit, then, has to be a bit open ended.

Theoretical Foundations

Both Vicarious Learning are theoretical in nature. The ones we discussed earlier are also presented with a theory into practice focus.

Using a Rubric (such as RETAIN)

The focus of this particular rubric was originally construed as a basis for assessing the instructional efficacy of video games. Based on the similarities of gamifying a lesson and other forms of informal learning we suggest it also be extrapolated it to utilize in the design of non-game based instruction using tools such as micro-credentialing and badging.

Establishing a Solid Research Basis

What we refer to here is the idea behind educational design research, as proposed by McKenney & Reeves.

tumblr_kxw7y5PmUu1qz4t8no1_1280

The idea behind this is that the research is formative and iterative in nature… the research participates at the development stage (the second “D” in ADDIE, and does not wait until the end (the “E”). Evaluation is continuous and participatory using feedback at each stage of development.

The Principle of Constructionism

Constructionism is a constructivist learning theory and theory of instruction. It states that building knowledge occurs best through building things that are tangible and “sharable” (Ackerman et al., 2009: 56). “ Constructionism (in the context of learning) is the idea that people learn effectively through making things. The assessment part is qualitative in nature in that you assess/evaluate the learning via an observation of the artifact that is constructed.

Layout of the Environment/Visitor Studies

The major allure of museum learning is the setting. Many have researched and analyzed that the actual layout of the physical plant of the museum plays a major role in the learning. Visitor studies as to the popularity of certain exhibits play a role in deciding the efficacy of the activity. Studies have correlated location to that popularity. While this is less precise than other forms of analyses, it does it appears, along with other constructs, play a significant part. The construct of the exhibit, of course is also important. The design should (as most well run exhibits generally do) mirror the way people learn most of the time. Well constructed exhibits are dynamic, multidimensional, and complex to reach out the the visitor in a holistic way. The main claim to fame, as it were is that learning is not delimited by space and time (as is with traditional classrooms). Visitors generally take as much time as they wish to be engaged. Time on task is a two-edge sword. To some degree it measures engagement, but it also could imply being overly complex. Is it possible to predict at what point the so-called value proposition of the exhibit flattens? (i.e., once usage flattens, that the learning has peaked).. this can be measured over time to see where in the exhibit’s presentation most visitors stop and move on… An important consideration is that of expected value… also a probability theory concept that utilizes data to be meaningful.. on a much simpler scale it relates to knowing the visitor/learner and what they will value in terms of knowledge acquisition… Fran Liebovitz often wrote about how she would work until she made enough money to pay the rent and make a car payment, then would quit. At what point will your visitor/learner make that same personal assessment about their learning? While oyu may know know this in advance, accurate data collection will assist you over the long run and the ability to collect that kind of data is an important design decision.

The bottom line is that learning most likely is best viewed as a process and not an end result. Sometimes we need to be able to re-frame the question to get to the answer were are looking for (that is where design thinking comes into play and why we introduced it to you earlier in the term).

 

Share this Post