Carver recommends using a tiered research approach to iteratively refine course definitions of deep understanding, instructional and assessment methods, and research designs to yield progressively deeper explanations of effective teaching/learning processes. Courses should have cognitive models of their learning objectives which include specification of the knowledge representation and organization (from individual facts to conceptual networks), processing skills (procedures and strategies for performance…
Researchers should also consider including metacognitive knowledge and skills for planning, monitoring, and controlling learning as well as attitudes and motivation.
Learning sciences researchers should consider including specific experimental manipulations within design experiments or running concurrent experiments to establish clear causal explanations that can then serve as foundations for other design decisions. Numerous critiques of design-based investigations advocate including more experimental methods and using both design and analysis strategies to minimize researcher bias (Dede 2004; Kelly 2004).
By using multiple methods with differing sensitivity to deep understanding plus a tiered approach ranging from a broad sample over a period of years to a single case study, Clark and Linn were able to identify the specific role of instructional time in the learning process. In addition, they were careful to identify and address possible research confounds.
Cobb et al. (2003) advocated multiple levels of analysis within design experiments to handle the processes and learning of students, the classroom community, the professional teaching community, and the school and/or district.
Carver lists basic quantitative and qualitative data recording techniques (McAfee et al. 2004) and states that cognitive science researchers extend this basic list to include rich observations of think-aloud protocols, classroom discourse, human-computer interaction logs, and researchers continue to develop data mining and analysis tools to help manage the unwieldy volume of data collected via these methods (Cobb et al. 2003). In addition to decision tree models and other deterministic data gathering tool…
Basically, the more you narrow your focus to the heart of the matter, the clearer you will be about where to invest the time and energy to do the deepest investigation and analysis.