Research and Evaluation

The NSTA Learning Center desires to showcase the real impact teachers and students achieve when experiencing our high quality resources and experiences as part of their personalized learning, whether as an individual or as part of a blended professional development experience at the district or state level. Below you will find links to several peer-reviewed publications, a third-party independent evaluation, and several recent presentations that share our research and story:

• Peer-reviewed Publications
• Third Party Evaluation Reports
• Recent Learning Center Presentations

• Simple and Computational Heuristics for Forum Management in the NSTA Learning Center: A Role for Learning Analytics in Online Communities of Practice Supporting Teacher Learning. Hawaii International Conference on System Sciences, January, 2012, Maui, Hawaii. Perez-Lopez, K; Cambridge, D; Byers, A.
  
  Abstract: Prior research has demonstrated that the quality of moderation and management in online communities of practice is key to their successful support of learning. However, as communities grow in size and complexity, it becomes increasingly difficult for unaided experts to fully understand and take action in response to the activity of participants within them. Learning analytics has the potential to provide the support that community of practice leaders need to improve their performance. The National Science Teachers Association and U.S. Department of Education’s Connected Educators project are exploring three approaches for managing forums to make them accessible and to synthesize the knowledge they generate: archiving, summarizing, and reorganizing. This paper describes manual heuristics for the first two of these, as well as the use of social network analysis to help develop algorithms to automate the third, community forum reorganization.
  
  
• First steps towards a social learning analytics for online communities of practice for educators. Learning Analytics and Knowledge 2012, Vancouver, Canada.
  
  Abstract: Learning analytics has the potential to provide actionable insights for managers of online communities of practice. Because the purposes of such communities and the patterns of activity that might further them are diverse, a wider range of methods may be needed than in formal educational settings. This paper describes the proposed learning analytics approach of the U.S. Department of Education's Connected Online Communities of Practice project, and presents preliminary applications of social network analysis to the National Science Teachers Association Learning Center as an illustration. Authors: Cambridge, D., Perez-Lopez, K.
  
  
• Developing a Web-Based Mechanism for Assessing Teacher Science Content Knowledge Journal of Science Teacher Education, 22 (1), March 2011. Authors: Byers, A.; Koba, S.; Sherman, G.; Scheppke, J.; Bolus, R.
  
  Abstract: The National Science Teachers Association (NSTA) recently launched a comprehensive electronic professional development (e-PD) online portal, the NSTA Learning Center. This support site for educators currently includes over 4,000 e-PD resources and opportunities available on-demand, as well as various tools designed to help educators maximize the effectiveness of using NSTA resources. One tool, the PD Indexer, helps teachers identify their own areas of content strengths and weaknesses by selecting content-specific assessments. Individual NSTA resources are recommended based on assessment outcomes. This paper presents a detailed description of the procedures employed by NSTA to develop valid and reliable PD Indexer content-specific multiple-choice assessment items. Authors: Byers, A.S., Koba, S., Sherman, G., Scheppke, J., Bolus, R.
  
  
• Electronic portfolios in the professional development of educators. In D'agustino Steven (Ed.), Adaptation, Resistance, and Access to Instructional Technologies: Assessing Future Trends in Education (pp. 429-444). Sherman, G., & Byers, A. (2010).
  
  Abstract: This chapter presents an overview of current electronic portfolio options available to practicing educators, emphasizing the different roles portfolios play within the professional development process. Available features within a variety of free and subscription web-based portfolio services are compared and the use of specific portfolio options within different professional development environments such as university graduate programs are profiled. Using a case study-like approach, the chapter details the use of the National Science Teachers Association’s (NSTA) Learning Center, a collection of resources available to teachers that includes a web-based professional development plan and portfolio tool. The chapter concludes with a reflection on the different ways in which professional organization resources like the NSTA’s PD Plan and Portfolio Tool can be used in the near future to continually improve the professional practice of educators.
  
  
• Evaluation of online, on-demand science professional development material involving two different implementation models. Journal of Science Education and Technology, 17(1), pp. 19-31. 2008.
  
  Abstract: This report presents pilot-test results for a science professional development program featuring online, on-demand materials developed by the National Science Teachers Association. During the spring 2006 semester, 45 middle school teachers from three different school districts across the United States participated in a professional development program designed to facilitate content knowledge and skills in the area of Newtonian force and motion. Participants from one of the school districts experienced a full-day instructor-led workshop along with two web-based seminars with a content-area expert. This was followed by a 4-week period of time in which they had access to self-directed, online, on-demand instructional materials that included activities, information, simulations, examples, and practice with immediate feedback over the targeted outcomes. Participants from the two other school districts only had access to the online materials with no instructor-led experience. This report documents positive gains in achievement as well as levels of confidence in teaching the material within all of the professional development groups. Data about the use of specific features within the online material are included, as well as completion rates and attitude survey results. Recommendations for future study are also included. Authors: Sherman, G., Byers, A., Rapp, S.
  
  

Houston Independent School District SciPack Study
Report on Outcomes from the Study of a SciPack Deployment in the Houston Independent School District during School Year 2009-2010. Edvantia, Inc., PO Box 1348, Charleston, WV 25325-1348. Download the full report.

Abstract: Edvantia conducted a two pretest-posttest delayed-treatment control group design study involving random assignment of two of NSTA’s SciPacks (online web modules focused on Earth’s Changing Surface and Force and Motion). This study examined the extent to which middle school teachers in the Houston Independent School District used two of the Learning Center’s SciPacks, and the SciPacks’ impact on teacher science teaching efficacy, teacher and student content knowledge in earth science and force and motion, and teacher instructional practices.

Survey results indicate that teachers’ sense of efficacy for teaching science increased over the course of the study, which suggests positive effects of the online professional development experience. Further, teachers who completed at least two SciPacks significantly increased their perceived preparedness to teach earth science over the course of the study. Teachers’ content knowledge of earth science and force and motion was examined by embedded SciPack final assessments, and pre- and post-assessments. Assessment scores indicated that teachers significantly increased their content knowledge in both content areas. Teachers improved their scores over time almost equally for both earth science and force and motion. Further, treatment teachers achieved higher gain scores than control teachers in both content areas.

Students of participating teachers did, indeed, improve their performance in earth science or force and motion. Fifth-grade students in the treatment teachers’ classrooms scored significantly higher on an earth science assessment than did those in control teachers’ classrooms, suggesting that the SciPacks had a positive impact on student performance in earth science. In addition, 6th- and 8th-grade students of treatment group teachers had gain scores that were significantly larger than the gain scores of students in control teachers’ classrooms, again suggesting that teacher SciPack completion had a positive influence on their students’ knowledge of force and motion. Authors: O’Connor, C. L., Chadwick, K. L., Samanta, D., Gore, J. N.

Integrating NASA Digital Educational Assets (IDEA) Project Report
Summary report and slide presentation highlight the accomplishments and third-party evaluation findings of the first two years of the IDEA project. Download the summary report. Download the slide presentation.

Abstract: This NASA grant supported middle and high school teachers representing 13 school districts via a blended model of teacher professional development combining both synchronous and asynchronous experiences via the NSTA Learning Center with local face-to-face PD efforts. Goals include increasing teacher content knowledge and pedagogical effectiveness and increasing student interest in Earth/Space Science and STEM career opportunities, as well as exploring which features of blended PD affordances best supported teacher learning preferences. After completing pre/post knowledge tests, teachers’ scores significantly increased more than 20 percentage points. Teachers confirmed the value of online community engagement to support their learning. Students who participated in project activities were much more likely to agree with the item, I am likely to take extra science, engineering, or math courses in high school that are beyond what is required.

	Blending Online and Onsite Professional Development: Applying What the Research Says for Effective and Sustainable Learning Communities Virginia Science Education Leadership Association, Williamsburg, Virginia, November 7, 2012. Invited Keynote: Byers, A.
	Directed Online Teacher Professional Development: Applying the Research for Effective and Scalable Learning Communities Educational Technology Leadership Conference, Roanoke, Virginia, November 5, 2012. Invited Spotlight Session Keynote: Byers, A.
	Digital Resources to Support Science Instruction National Association for Research in Science Teaching, National Conference Symposium, Indianapolis, Indiana, March, 2012. Milne, C.; Schwartz, R; Recker, M.; Byers, A.; Dorsey, C.; Reichsman, F.; Goldenberg, L.; Sicker, T.; Anderson, A.; Pasquale, M.; Wiebe, E.
	Social Network Analysis of Affiliation Networks to Promote Online Communities of Practice for Science Education International Network for Social Network Analysis, Sunbelt XXXII Social Networks Conference, 2012, San Diego, California, March, 2012. Perez-Lopez, K.; Cambridge, D.; Byers, A.; Booth, S.
	Blending Onsite and Online Professional Development: Applying the Research for Effective and Scalable Learning Communities National Science Education Leadership Association, Professional Development Institute, Indianapolis, Indiana, March, 2012. Byers, A.; Mendez, F.; Nickerson, L.A.; Le, D.
	Research Insights into Online Communities of Practice and Teacher Learning Online: The NSTA Learning Center National Science Teachers Association, Indianapolis, Indiana, March, 2012. Byers, A.; Cambridge, D.; Sherman, G
	The NSTA Learning Center: A Tool to Develop Preservice Teachers National Science Teachers Association, Indianapolis, Indiana, March, 2012. Byers, A.; Mendez, F.; Mohr, C.; Bailey, B.; Odell, M.
	Online Communities of Practice: What Works? Florida Education Technology Conference, Orlando, Florida, January, 2012. Invited Panelist: George, M.; Beach-Nussbaum, S.; Byers, A.; Levy, A.; Mageau, T.
	Online Professional Development: Applying What the Research Says for Effective Learning Next Step Institute ─ Navigating the Landscape of STEM. Dover, DE. May 2011. Invited presenter: Byers, A.
	Online Professional Development: Research on Teacher Perceptions, Learning Preferences, and Learning Outcomes for Self-directed NSTA Web Courses National Science Teachers Association, National Conference, San Francisco, CA. March, 2010. Presenters: Byers, A., Chadwick, K., Sherman, G.
	The NSTA Learning Center: A Tool to Develop Preservice Teachers National Science Teachers Association, National Conference, San Francisco, CA. March, 2011. Presenters: Byers, A., Sherman, G., Bailey, B., Odell, M.
	Online Professional Development: Applying what the research says for Effective Learning National Science Teachers Association, National Conference, Research Development Conference, San Francisco, CA. March, 2011. Presenters: Byers, A., Sherman, G., Chadwick, K., Mendez, F.
	Developing Online Learning Communities at Scale National Science Foundation-sponsored STEM Summit, Longwood University, Farmville, VA. February, 2011. Invited Presenter: Byers, A.
	Building Scalable and Sustainable Learning Centers Always Learning Web Seminar Series with host Bryan Chapman. July, 2010. Online Archive. Invited presenter: Byers, A.