Research and Evaluation

The NSTA Learning Center desires to showcase the real impact teachers and students achieve when experiencing our high quality resources and experiences as part of their personalized learning, whether as an individual or as part of a blended professional development experience at the district or state level. Below you will find links to several peer-reviewed publications, a third-party independent evaluation, and several recent presentations that share our research and story:

Publications and Conference Proceedings:
  • Gamrat, C., Zimmerman, H. T., Dudek, J., & Peck, K. (2014).
    Personalized workplace learning: An exploratory study on digital badging within a teacher professional development program. British Journal of Educational Technology, 45(6), 1136-1148. doi: 0.1111/bjet.12200

    Abstract: To provide customized workplace learning opportunities, a digital badge system was designed by a university, governmental agency and national professional association to support teachers’ implementation of professional development (PD). Teacher Learning Journeys (TLJ) is an approach that allows for teachers to customize their PD experience to their workplace and make decisions about what PD they need based on their expertise and interests. The digital badging provided and assessed experiences in online PD. Using a theoretical framework that focused on decision making and customization as part of personalization, researchers conducted a theory-driven thematic analysis on teachers’ TLJ artifacts (goal statements, interviews and reflective activity logs). Data came from 36 teachers who completed 154 PD activities over a 3-month period. A case study was developed from an in-depth analysis of eight teachers’ artifacts and interviews. Using TLJ as a PD tool, teachers made decisions when selecting learning goals they identified as personally relevant. Teachers customized the level of assessment and the specific content depth to personalize the PD training for workplace constraints. This project informs future research aiming to understand how personalized learning activities support teachers and other professional learners in a variety of workplaces.

  • Sociotechnical Capital and Situated Knowledge in Online Communities of Practice. The Eighth Annual INGRoup Conference, Atlanta, GA, July 11-13, 2013. Interdisciplinary Network for Group Research: Using research to advance the understanding of group behavior, dynamics, and outcomes, Straus, Susan G.; Karam, Rita; Bikson, Tora K.; Byers, Al.

    Abstract: Teaching, unlike many other professions, is a largely isolated occupation in that teachers spend the majority of their work day instructing classes on their own, with limited time available to interact with their peers and share and develop knowledge. In urban areas, competing demands on teachers leave little time to develop a sustaining learning community within a school. In rural areas, teachers' opportunities to collaborate with colleagues are additionally constrained by limited numbers of peers and high turnover. We expect that feelings of isolation are intensified for K-12 science teachers, as their number within a school tends to be smaller than other content area teachers (e.g., math and language arts), and there may be few or no colleagues within the same science domain. The need for interaction and collegiality among science educator peers might, in principle, be met through attendance at in-person professional society meetings or PD events, but science teachers report lacking adequate local opportunities for such activity. Online communities of practice (CoPs) appear to afford promising alternatives for overcoming the absence of collocated peers in science education. This study explores how web-based collaboration technologies are used to build distributed communities of practice (CoP) among science educators. We focus on CoPs among science educators due to increasing national attention on science, technology, engineering, and math (STEM) education, the potential for web technologies to support teachers who often lack physically co-located colleagues, and the very limited research on CoPs in this domain. This presentation addresses how participation in virtual CoP activities leads to the generation of situated knowledge and sociotechnical capital and factors that are associated with participation in these communities. Data analysis is in progress.

  • Simple and Computational Heuristics for Forum Management in the NSTA Learning Center: A Role for Learning Analytics in Online Communities of Practice Supporting Teacher Learning. Hawaii International Conference on System Sciences, January, 2012, Maui, Hawaii. Perez-Lopez, K; Cambridge, D; Byers, A.

    Abstract: Prior research has demonstrated that the quality of moderation and management in online communities of practice is key to their successful support of learning. However, as communities grow in size and complexity, it becomes increasingly difficult for unaided experts to fully understand and take action in response to the activity of participants within them. Learning analytics has the potential to provide the support that community of practice leaders need to improve their performance. The National Science Teachers Association and U.S. Department of Education’s Connected Educators project are exploring three approaches for managing forums to make them accessible and to synthesize the knowledge they generate: archiving, summarizing, and reorganizing. This paper describes manual heuristics for the first two of these, as well as the use of social network analysis to help develop algorithms to automate the third, community forum reorganization.

  • First steps towards a social learning analytics for online communities of practice for educators. Learning Analytics and Knowledge 2012, Vancouver, Canada.

    Abstract: Learning analytics has the potential to provide actionable insights for managers of online communities of practice. Because the purposes of such communities and the patterns of activity that might further them are diverse, a wider range of methods may be needed than in formal educational settings. This paper describes the proposed learning analytics approach of the U.S. Department of Education's Connected Online Communities of Practice project, and presents preliminary applications of social network analysis to the National Science Teachers Association Learning Center as an illustration. Authors: Cambridge, D., Perez-Lopez, K.

  • Developing a Web-Based Mechanism for Assessing Teacher Science Content Knowledge Journal of Science Teacher Education, 22 (1), March 2011. Authors: Byers, A.; Koba, S.; Sherman, G.; Scheppke, J.; Bolus, R.

    Abstract: The National Science Teachers Association (NSTA) recently launched a comprehensive electronic professional development (e-PD) online portal, the NSTA Learning Center. This support site for educators currently includes over 4,000 e-PD resources and opportunities available on-demand, as well as various tools designed to help educators maximize the effectiveness of using NSTA resources. One tool, the PD Indexer, helps teachers identify their own areas of content strengths and weaknesses by selecting content-specific assessments. Individual NSTA resources are recommended based on assessment outcomes. This paper presents a detailed description of the procedures employed by NSTA to develop valid and reliable PD Indexer content-specific multiple-choice assessment items. Authors: Byers, A.S., Koba, S., Sherman, G., Scheppke, J., Bolus, R.

  • Electronic portfolios in the professional development of educators. In D'agustino Steven (Ed.), Adaptation, Resistance, and Access to Instructional Technologies: Assessing Future Trends in Education (pp. 429-444). Sherman, G., & Byers, A. (2010).

    Abstract: This chapter presents an overview of current electronic portfolio options available to practicing educators, emphasizing the different roles portfolios play within the professional development process. Available features within a variety of free and subscription web-based portfolio services are compared and the use of specific portfolio options within different professional development environments such as university graduate programs are profiled. Using a case study-like approach, the chapter details the use of the National Science Teachers Association’s (NSTA) Learning Center, a collection of resources available to teachers that includes a web-based professional development plan and portfolio tool. The chapter concludes with a reflection on the different ways in which professional organization resources like the NSTA’s PD Plan and Portfolio Tool can be used in the near future to continually improve the professional practice of educators.

  • Evaluation of online, on-demand science professional development material involving two different implementation models. Journal of Science Education and Technology, 17(1), pp. 19-31. 2008.

    Abstract: This report presents pilot-test results for a science professional development program featuring online, on-demand materials developed by the National Science Teachers Association. During the spring 2006 semester, 45 middle school teachers from three different school districts across the United States participated in a professional development program designed to facilitate content knowledge and skills in the area of Newtonian force and motion. Participants from one of the school districts experienced a full-day instructor-led workshop along with two web-based seminars with a content-area expert. This was followed by a 4-week period of time in which they had access to self-directed, online, on-demand instructional materials that included activities, information, simulations, examples, and practice with immediate feedback over the targeted outcomes. Participants from the two other school districts only had access to the online materials with no instructor-led experience. This report documents positive gains in achievement as well as levels of confidence in teaching the material within all of the professional development groups. Data about the use of specific features within the online material are included, as well as completion rates and attitude survey results. Recommendations for future study are also included. Authors: Sherman, G., Byers, A., Rapp, S.

Third Party Evaluation Reports:

Houston Independent School District SciPack Study
Report on Outcomes from the Study of a SciPack Deployment in the Houston Independent School District during School Year 2009-2010. Edvantia, Inc., PO Box 1348, Charleston, WV 25325-1348. Download the full report.

Abstract: Edvantia conducted a two pretest-posttest delayed-treatment control group design study involving random assignment of two of NSTA’s SciPacks (online web modules focused on Earth’s Changing Surface and Force and Motion). This study examined the extent to which middle school teachers in the Houston Independent School District used two of the Learning Center’s SciPacks, and the SciPacks’ impact on teacher science teaching efficacy, teacher and student content knowledge in earth science and force and motion, and teacher instructional practices.

Survey results indicate that teachers’ sense of efficacy for teaching science increased over the course of the study, which suggests positive effects of the online professional development experience. Further, teachers who completed at least two SciPacks significantly increased their perceived preparedness to teach earth science over the course of the study. Teachers’ content knowledge of earth science and force and motion was examined by embedded SciPack final assessments, and pre- and post-assessments. Assessment scores indicated that teachers significantly increased their content knowledge in both content areas. Teachers improved their scores over time almost equally for both earth science and force and motion. Further, treatment teachers achieved higher gain scores than control teachers in both content areas.

Students of participating teachers did, indeed, improve their performance in earth science or force and motion. Fifth-grade students in the treatment teachers’ classrooms scored significantly higher on an earth science assessment than did those in control teachers’ classrooms, suggesting that the SciPacks had a positive impact on student performance in earth science. In addition, 6th- and 8th-grade students of treatment group teachers had gain scores that were significantly larger than the gain scores of students in control teachers’ classrooms, again suggesting that teacher SciPack completion had a positive influence on their students’ knowledge of force and motion. Authors: O’Connor, C. L., Chadwick, K. L., Samanta, D., Gore, J. N.

Integrating NASA Digital Educational Assets (IDEA) Project Report
Summary report and slide presentation highlight the accomplishments and third-party evaluation findings of the first two years of the IDEA project. Download the summary report. Download the slide presentation.

Abstract: This NASA grant supported middle and high school teachers representing 13 school districts via a blended model of teacher professional development combining both synchronous and asynchronous experiences via the NSTA Learning Center with local face-to-face PD efforts. Goals include increasing teacher content knowledge and pedagogical effectiveness and increasing student interest in Earth/Space Science and STEM career opportunities, as well as exploring which features of blended PD affordances best supported teacher learning preferences. After completing pre/post knowledge tests, teachers’ scores significantly increased more than 20 percentage points. Teachers confirmed the value of online community engagement to support their learning. Students who participated in project activities were much more likely to agree with the item, I am likely to take extra science, engineering, or math courses in high school that are beyond what is required.

Recent Presentations:
Blended Professional Learning for Pre-Service and In-Service Science Educators: The NSTA Learning Center. National Science Foundation Online Teacher Professional Development (OTPD) Summit, University of Massachusetts, Boston, MA, November 15, 2014. Two day summit focused on reviewing proposed Harvard Press book chapters from leading scholars on current and emerging models for OTPD that address effectiveness, scale and sustainability. The NSTA Learning Center is one invited chapter for review. Chapters are reviewed against specific protocols and framework. Byers, A., and Mendez, F.
Developing Large Scale Effective Online Teacher Learning Communities at the National Science Teachers Association. Ninth Annual International Conference on Networked Learning, Edinburgh, United Kingdom, March 2-8, 2014. Invited "Hot Seat" online expert panelists: Byers, A. Mendez, F.
A Tool to Develop Science Pre-service Teachers. Association of Science Teacher Educators Annual Conference, San Antonio, Texas, January 17, 2014. Two hour panel and hands-on workshop. Byers, A., and Mendez, F. ,NSTA; Baird, K., Indiana University-Purdue University Columbus; Blunk, S., University of Maryland Baltimore County; Mohr, C., Dominican University; Odell, M., University of Texas at Tyler; Sparrow, K., Florida International University; Veal, W., College of Charleston.
Making it Count-Integrating Formal and Informal PD. US Department of Education Connected Educators Month (CEM) Invited Kickoff Panel.
October 2, 2013. The Archive of the live web seminar session is available from CEM web site.
Developing Large Scale Effective STEM Teacher Learning Communities at the National Science Teachers Association. Invited Closing Keynote for online GlobalSTEMx Conference sponsored by Hewlett Packard. The Archives of this and all the web seminar sessions are available from the GlobalSTEMxCon web site.
Developing Large Scale Effective Teacher Learning Communities at the National Science Teachers Association. ISTE, San Antonio, Texas, June 24, 2013. Invited speaker: Byers, A.
Blending Online and Onsite Professional Development: Applying What the Research Says for Effective and Sustainable Learning Communities
Virginia Science Education Leadership Association, Williamsburg, Virginia, November 7, 2012. Invited Keynote: Byers, A.
Directed Online Teacher Professional Development: Applying the Research for Effective and Scalable Learning Communities
Educational Technology Leadership Conference, Roanoke, Virginia, November 5, 2012. Invited Spotlight Session Keynote: Byers, A.
Digital Resources to Support Science Instruction
National Association for Research in Science Teaching, National Conference Symposium, Indianapolis, Indiana, March, 2012. Milne, C.; Schwartz, R; Recker, M.; Byers, A.; Dorsey, C.; Reichsman, F.; Goldenberg, L.; Sicker, T.; Anderson, A.; Pasquale, M.; Wiebe, E.
Social Network Analysis of Affiliation Networks to Promote Online Communities of Practice for Science Education
International Network for Social Network Analysis, Sunbelt XXXII Social Networks Conference, 2012, San Diego, California, March, 2012. Perez-Lopez, K.; Cambridge, D.; Byers, A.; Booth, S.
Blending Onsite and Online Professional Development: Applying the Research for Effective and Scalable Learning Communities
National Science Education Leadership Association, Professional Development Institute, Indianapolis, Indiana, March, 2012. Byers, A.; Mendez, F.; Nickerson, L.A.; Le, D.
Research Insights into Online Communities of Practice and Teacher Learning Online: The NSTA Learning Center
National Science Teachers Association, Indianapolis, Indiana, March, 2012. Byers, A.; Cambridge, D.; Sherman, G
The NSTA Learning Center: A Tool to Develop Preservice Teachers
National Science Teachers Association, Indianapolis, Indiana, March, 2012. Byers, A.; Mendez, F.; Mohr, C.; Bailey, B.; Odell, M.
Online Communities of Practice: What Works?
Florida Education Technology Conference, Orlando, Florida, January, 2012. Invited Panelist: George, M.; Beach-Nussbaum, S.; Byers, A.; Levy, A.; Mageau, T.
Online Professional Development: Applying What the Research Says for Effective Learning
Next Step Institute ─ Navigating the Landscape of STEM. Dover, DE. May 2011. Invited presenter: Byers, A.
Online Professional Development: Research on Teacher Perceptions, Learning Preferences, and Learning Outcomes for Self-directed NSTA Web Courses
National Science Teachers Association, National Conference, San Francisco, CA. March, 2010. Presenters: Byers, A., Chadwick, K., Sherman, G.
The NSTA Learning Center: A Tool to Develop Preservice Teachers
National Science Teachers Association, National Conference, San Francisco, CA. March, 2011. Presenters: Byers, A., Sherman, G., Bailey, B., Odell, M.
Online Professional Development: Applying what the research says for Effective Learning
National Science Teachers Association, National Conference, Research Development Conference, San Francisco, CA. March, 2011. Presenters: Byers, A., Sherman, G., Chadwick, K., Mendez, F.
Developing Online Learning Communities at Scale
National Science Foundation-sponsored STEM Summit, Longwood University, Farmville, VA. February, 2011. Invited Presenter: Byers, A.
Building Scalable and Sustainable Learning Centers
Always Learning Web Seminar Series with host Bryan Chapman. July, 2010. Online Archive. Invited presenter: Byers, A.