This Week in Learning Analytics (January 3 – 9, 2015)

I took this picture of the contents of a garbage can that I sat next to at UC Davis last week. A mass grave for marshmallow snowmen.
I took this picture of the contents of a garbage can that I sat next to at UC Davis last week. A mass grave for marshmallow snowmen.


Student Success

New Study Reveals that PhD Holders are not Absorbed into Employment as Easily as Previously Thought
Holly Elisa | Times Higher Education | 3 January 2015

A recent paper published by Heidi Skovgaard Pederson in the Journal of Higher Education Policy and Management looks to the European Commission’s Careers of Doctorate Holders surveys on PhD labour market outcomes and observes that “Several policies have been implemented to promote the production of PhDs and support their labour market outcomes. However, the latter has received relatively little attention empirically”. She concludes the paper by commenting that “There is a need to increase knowledge within the area to understand mobilisation patterns, to ensure continued attractiveness of doctoral education in the longer run and provide a research strategy to assist policymakers in their decision-making.”

Three Eastern Kansas Universities Seek to Identify Struggling Students More Quickly and HelpThem to Graduate
Associated Press | The Republic | 5 January 2015

The University of Kansas, Ottawa University, and Haskell Indian Nations University are each using systems to mine student data to increase student success, as part of a national push to improve retention and graduation rates. The University of Kansas in particular makes use of a Starfish330 integration with Blackboard Learn to flag at-risk students and facilitate interventions.


The University of Kansas is also one of 11 universities in the University Innovation Alliance, which aims to share knowledge about predictive analytics and intensive advising strategies to improve educational outcomes for all students, regardless of background.

AVID Students Stick with College and Outpace Peers
Maureen Magee | U-T San Diego | 6 January 2015

The National Student Clearninghouse has found that high school graduates from 2010 and 2011 who participated in AVID (Advancement Via Individual Determination) persisted through their freshman and sophomore years of college at a higher rate than their counterparts who were not in the program. the AVID program promotes academic success in middle and high school cultivates success through intensive tutoring and by fostering independent study and organizational skills. Now in its 35th year, AVID began in San-Diego, and is currently used by more than 5,000 schools in 44 states and 16 countries.


Auditors Find that the Georgia Department of Education Needs Better Analysis of Student Data
Walter C. Jones | Athens Banner-Herald | 4 January 2015

A report by the Department of Audits and Accounts observes that the Georgia Department of Education lacks a comprehensive assessment of data collected. Such a comprehensive, systematic analysis would allow the department “to identify significant trends or anomalies that could indicate weaknesses in local school systems’ data collection and reporting processes, particularly as it relates to the Full-Time Equivalent data collection.” The Department of Education apparently does a fine job of ensuring the cleanliness of its data, but does little in the way of analysis that would contribute to actionable insights.

Irish Minister of Education considers retaining student data until students turn 30 | 7 January 2015

Irish Minister of Education, Jan O’Sullivan, has said that she would consider a plan to adopt a Primary Online Database (POD) that would function as a central repository for student information, with a data retention policy that wold allow data to be retained until students reach the age of 30. Critics are concerned about government overreach.


Smart Classroom Market Growth Driven by Learning Analytics and its Globally Increasing Adoption
PR Newswire | 5 January 2015 has added an 83 page research report on education and smart classroom industry to the IT & Telecommunication intelligence section of its online business research and data library. “Global Smart Classroom Market 2015-2019” discusses the impact of learning analytics on online teaching practices and its usefulness in gaining insights into how students understand smart courseware courseware. The report observes that market drivers like increased broadband adoption, increased interest in flexible learning models, and increased investment in smart education systems are likely to see increased investment in learning analytics over the next 3-4 years.

InfoSnap Signs Student Privacy Pledge
InfoSnap | 6 January 2015

InfoSnap, a leading provider of cloud-based registration management solutions, has signed the K-12 School Service Provider Pledge to Safeguard Student Privacy sponsored by the Software & Information Industry Association (SIIA) and The Future of Privacy Forum. In signing the pledge, InfoSnap joins the likes of Knewton, Microsoft, Renaissance Learning, Shutterfly, and others in their commitment:

  • Not sell student information
  • Not behaviorally target advertising
  • Use data for authorized education purposes only
  • Not change privacy policies without notice and choice
  • Enforce strict limits on data retention
  • Support parental access to, and correction of errors in, their children’s information
  • Provide comprehensive security standards, and
  • Be transparent about collection and use of data

Language-Learning App Duolingo Introduces Program for Classrooms
Kif Leswing | GIGAOM | 8 January 2015

The language learning softwre company, Duolingo, has just launched Duolingo for Schools. The program, which is already used by many teachers, will give teachers more centralized control over the program, as well as data on how students are doing and where remediation might be required


University Of Phoenix Finds Payoff In Private Cloud
Charles Babcock | GIGAOM | 9 January 2015

The Apollo Education Group (parent organization of the University of Phoenix) has modernized its IT infrastructire to better support the collection and analysis of student data. After exploring public cloud options like Amazon’s EC2 cloud, Apollo found that, economically, a private cloud solution (based on VMware’s vCloud Suite and Vblocks from the VMware-Cisco-EMC consortium, VCE) was the best approach.


Tools for Evidence-Based Action (TEA) Workshop
Timothy D. Harfield | | 6-7 January 2015

Hosted by the UC Davis iAMSTEM Hub and generously funded by the Helmsley Trust, Tools for Evidence-Based Action (TEA) aims to disseminate new tools and methodologies for classroom observations and visualizing student pathways. Its inaugural workshop brought together delegates from all over the United States for two days of intense sharing and discussion of data visualization tools. I had the geat pleasure of being invited to the event, which I documented in the blog posts below:


MUST READ When Small Words Foretell Academic Success: The Case of College Admissions Essays
James W. Pennebaker, Cindy K. Chung, Joey Frazee, Gary M. Lavergne, David I. Beaver | PLoS ONE | 9(12)

The smallest and most commonly used words in English are pronouns, articles, and other function words. Almost invisible to the reader or writer, function words can reveal ways people think and approach topics. A computerized text analysis of over 50,000 college admissions essays from more than 25,000 entering students found a coherent dimension of language use based on eight standard function word categories. The dimension, which reflected the degree students used categorical versus dynamic language, was analyzed to track college grades over students’ four years of college. Higher grades were associated with greater article and preposition use, indicating categorical language (i.e., references to complexly organized objects and concepts). Lower grades were associated with greater use of auxiliary verbs, pronouns, adverbs, conjunctions, and negations, indicating more dynamic language (i.e., personal narratives). The links between the categorical-dynamic index (CDI) and academic performance hint at the cognitive styles rewarded by higher education institutions.

NOTE: A summary of this research was recently published by Scott Jaschik for Inside Higher Ed here: New Study Links Certain Application Essays and College Success



Student Needs Should Trump Interests of Adults
Wes Smith | U-T San Diego | 2 January 2015

Smith (Executive Director of the Association of California School Administrators) observes that defining and measuring educational quality is complex, but that this complexity is no excuse to stall efforts to advocate for high-quality education and equitable access to effective educators. He observes that data collected about student performance and teacher effectiveness are rarely tied to student learning. In response, the Association of California School Administrators is working on an educator effectiveness bill which emphasizes learning over performance, and sees responsibility of high-quality education extend beyond the teacher.

A Big Data-Powered ‘GPS’ Helps Colleges Boost Graduation Rates: Predictive Analytics Help Schools Identify and Aid Struggling Students
D. Frank Smith | EdTech Magazine | 7 January 2015

A very nice summary of the successful work being done at Georgia State University, under the direction of Timothy Renick, to develop a predictive analytics system that functions in support of an intensive advising program. Georgia State praised by President Obama at a recent White House summit on strategies to increase higher education graduation rates.

MUST READ Learners, Teachers, and Technology: Personalization in 2015 and Beyond
Kristen DiCerbo | Wired Magazine | 7 January 2015

DiCerbo helpfully addresses a common source of equivocation by clearly distinguishes three ways in which the term ‘Personalized Learning.’ In her view, the use of analytics for the sake of personalization can put one of three agents in the position of decision-maker: (1) The Student, (2) Technology, and/or (3) The Teacher. She insists that personalization needs to involve a sophisticated blending of learner, teacher and technology, Decisions about WHEN to privilege a particular perspective over others are difficult but important. Reflection on these roles, responsibilities, and the conditions under which they are adopted are crucial to the use of personalization to improve student outcomes, and will richly inform the future of technology in the space.


Education Predictions for 2015
NPR Ed Team | National Public Radio (NPR) | 3 January 2015

Elana Zeide (Privacy Research Fellow, Information Law Institute, New York University) predicts that, in response to increasing public concern over student data becoming permanent records, 2015 will see increased regulation by government as well as public commitments to student data privacy on the part of private industry. Jose Vilson (author of This Is Not A Test: A New Narrative on Race, Class, and Education) expects to see more nuanced kind of data in schools. In particular, he expects to see a return to anecdotal evidence, and increased concern for issues of diversity in teacher evaluation.

What Schools Could Use Instead of Standardized Tests
Anya Kamenetz | National Public Radio (NPR) | 6 January 2015

Kamenetz observes that, in spite of strong criticism of standardized testing, there is nonetheless little in the way of discussion of what might replace it. She suggests four (non-mutually exclusive) possibilities

  1. Sampling
  2. Stealth Assessment
  3. Multiple Measures
    • Social and Emotional Skills Surveys
    • Game-Based Assessments
    • Performance or Portfolio-Based Assessments
  4. Inspections

Tools for Evidence-Based Action (TEA) Meeting – Day Two

The second day of the Tools for Evidence-Based Action (TEA) Meeting opened with a continuation discussion from yesterday, about features and feature requests for the two tools introduced yesterday: the Ribbon Flow Tool and the General Observation and Reflection Protocol (GORP) Tool.

Feature requests for the Ribbon Flow Tool included adding more stops and a csv to json convertor, but also the ability to customize color-codes and tool-tips. Several security concerns were raised in discussion yesterday, which developer Matt Steinwachs resolved by the morning. Nevertheless, general consensus was that comprehensive documentation is necessary in order to gain the institutional approvals necessary before hosting any kind of student data, aggregate or otherwise, on an external server.


As far as the GORP tool is concerned, Matt Steinwachs presented a kind of development roadmap that would see the app implemented in a way that was fully customizable, allowing users to record observations according to any observation protocol or variation. In future, users will be able to fully customize the buttons presented (including text, images, colors, and position). Also customizable will be time frame (i.e. interval or real time logging). This is clearly going to be a powerful and flexible tool for recording classroom observations, but what really excites me is its potential as an active learning tool. I imagine, for example, customizing the app for use by students so that they can record affective and/or learning states during the course of a class period. The simplest example of this would be to strip down the app to include only a single ‘bored’ button and asking students to push it when they lose interest during a class period. Data collected from a course (especially from a large course) could produce a heatmap on a lecture recording, in order to associate classroom activity patterns with lags in student engagement.

In the afternoon, Mary Hueber and Pat Hutchings, Evaluators for the Bay View Alliance, talked about the organization’s latest Research Action Cluster (RAC): Using Academic Analytics to Support and Catalyze Transformation. Led by Marco Molinaro and Chris Pagliarulo (UC Davis), the project will produce three case studies about three institutions at different stages in their learning analytics program implementations. The emphasis of these case studies will be on data-driven intervention effectiveness. UC Davis has been selected as the first institution to be evaluated. The others (which are likely to include one other school from the Bay View Alliance, and one from outside) will be selected in the coming months.

The day concluded with discussions of how the TEA community might be sustained and grow, and how participants might bring insights back to their home institutions and promote tool adoption. The Helmsley Charitable Trust has committed to funding another TEA meeting in a year. I look forward to continuing to inform the development of the UC Davis learning analytics toolkit, and to continuing my active involvement in this initiative. Big thanks are due to Marco Molinaro, Chris Pagliarulo, Alan Gamage and Margie Barr for their hard work in organizing this event, and catalyzing this community of interest in approaches to evidence-based action.

Tools for Evidence-Based Action (TEA) Meeting – Day One

Today was the first day of the Tools for Evidence-Based Action (TEA) Meeting, hosted by the UC Davis iAMSTEM Hub. Organized by Marco Molinaro (Assistant Vice Provost for Undergraduate Education, UC Davis), with assistance from Alan Gamage and Margie Barr, the TEA meeting is part of an effort supported by the Helmsley Trust to disseminate new tools and methodologies for classroom observations and visualizing student pathways.

Day one introduced the group of invited participants (representing colleges and universities from all over the United States) to three primary tools: Trellis, a Ribbon Tool, and a GORP Classroom Observation Tool.

Trellis Social Networking and Collaboration Platform

Trellis is a social network in development by the American Association for the Advancement of Science (AAAS). Trellis is still in development, but is currently being beta-tested among a number of communities, including TEA. When it goes live, Trellis is meant to provide a multidisciplinary space for anyone in science to engage in the communities to which they belong.


As a social network, Trellis has a lot of potential. The UX is clean and, supported by AAAS, is uncluttered by advertisements. It is also free for scientific communities that choose to use it. A colleague observed that Trellis might function as a ‘Facebook for nerds,’ but only if all of the organizations and societies to which we belong were to choose a common social network. Short of this, Trellis is simply another social network among social networks.

In addition to its basic social networking functionality, Trellis also promises to provide a rich set of collaboration tools, to simplify document preparation, annotation, and sharing. In all this, however, it continues to be unclear to me what is supposed to distinguish Trellis from other social networks and collaboration platforms that are already such a central part of most of our lives (especially LinkedIn and Google+). Most importantly, however, I hesitate to fully support a new network in the absence of a clear revenue model, and without a firm commitment to data privacy.

Ribbon Tool

The cloud-hosted (via Amazon Web Services) Ribbon FLOW Tool that is being developed at UC Davis by Matt Steinwachs filters and visualizes change in group membership over time. In the demonstration, the tool was used to visualize change in declared major upon admission compared to graduation. Accepting data in json format, the tool is flexible enough to visualize any kind of data flow.

UC Davis Ribbon FLOW Tool

The tool is still in early development, but in future releases and before being made available more widely, Matt has promised to add the following

  • The ability to upload/download data in csv format (a must for usability among non-technical users)
  • Multiple stops, so that changes in group state that take place between start and end can be represented
  • Comprehensive documentation (including security model and privacy policy)

Once completed, the goal is to make the tool open source, and I would love to see it used by the Learning Analytics Initiative or as part of some other similar open learning analytics toolkit.

General Observation and Reflection Tool (GORP)

The General Observation and Reflection Tool (GORP) classroom observation tool is being developed by Matt Steinwachs for Chris Pagliarulo (Director of Instruction & Assessment, iAMSTEM, UC Davis). The web-based tool facilitates data entry for ‘agents’ (student coders) observing classes according to the Classroom Observation Protocol for Undergraduate STEM (COPUS). The tool seeks to solve the problem of having to manually enter data collected by coders at two minute intervals, by automating the collection and storage process.

GORP Classroom Observation Tool


The day concluded with a “Bring Your Own Data and Tools” session, which provided TEA Meeting participants with an opportunity to share tools that they had developed and/or had been using with great success. A few highlights include:

  • Marilyne Stains (University of Nebraska-Lincoln) presented the results of work that she and her co-investigators have been doing with the Classroom Observation Protocol for Undergraduate STEM (COPUS). Using the GORP tool, she has been able to identify coding redundancies (ex. Students listening’ and ‘Instructor Lecturing’ represent the same classroom state). A cluster analysis has identified several basic class types.
  • Andrew Feig (Wayne State University) demonstrate the use of Circos to represent individual student course pathways from semester to semester.
  • Erin Soloman (Washington University in St. Louis) presented a flexible classroom observation protocol (OPAL) and iOS app that records and timestamps every coding event. The advantage of this approach over others (which aggregate all events at 2 minute intervals) is that it allows educational analysts to correlate spikes in student engagement with specific moments in a recorded lecture.