Monday, March 19, 2012

Research Projects for Open Ed Researcher Badge

The final step that I need to complete for the open ed researcher badges is to outline research studies for each of the three topics that I selected for research.  In previous posts I have fulfilled the requirements to identify scholarly literature and other available resources relevant to the topic.  Those posts were on the topics of:

  1. OpenCourseWare (click to see the post)
  2. Open Source Software (click to see post)
  3. Open Educational Resources (click to see post)
This post will outline research studies for each of these three topics per the requirements to earn the Open Ed Researcher badge.

Research Project #1 - OpenCoursWare
A prevalent issue confronting proponents of Open CourseWare is the financial  sustainability of efforts to produce and maintain open courses.  Previous studies identified or proposed different approaches to addressing this sustainability concern.  OpenCourseWare fits with the traditional pattern of initiatives needing funding at institutions of higher education and other organizations seeking to provide a public benefit at minimal cost.  Traditional sources of funding for such initiatives and organizations include: government taxpayer support through appropriations and grants, sponsoring institution support, endowments, philanthropic grants, foundation grants, patron donations, add-on sales, and licensing.  Some of these sources feed the parent institution sponsoring which then allocates funding from institutional budgets to fund these efforts.

MIT OpenCoursWare has experienced funding from the institution, philanthropists, foundations, and is now soliciting patron support on its website.  Other initiatives to provide open courses (particularly outside of the U.S.) rely on government support for their efforts through institutions established for open learning and as open universities.  Some institutions that have entered the OpenCoursWare space have exited due to the lack of ongoing funding.

My proposed research project is to explore an approach that may demonstrate a strategic reason for higher education institutions to consider funding open courses from their regular budgets.  Policies are emerging from governments and accrediting bodies that are attempting to hold institutions more accountable for the achievement of specific learning outcomes by their students.  These policies and standards require an emphasis on measurement and reporting.

I contend that OpenCourseWare can be a valuable tool to insure that the design of instruction is tied to specific learning outcomes and covers the entire scope of curriculum in the academic period in which it is to be delivered.  The design of online courses can enforce discipline in the design of instruction to meet all learning objectives.  Such completeness of coverage may be sacrificed when limited to the contact hours in the classroom as faculty often need to adjust due to the limits of time.  Having a well-designed curriculum available online promotes thorough coverage of learning objectives, even if the students have to be referred to the online materials for some of the instruction that could not be delivered in the classroom due to time constraints.  This is one advantage of creating an identical version of the course materials and instruction that can be provided through OpenCourseWare.

Another benefit that could make OpenCourseWare an institutional strategy priority is the possibility that open courses could be an always-available supplement to the instruction provided in the classroom.  The research study I am proposing attempts to document if such a benefit exists, as indicated by a correlation between accessing the open course and the grades students achieve when using the open course as a supplement to classroom instruction.

Research Setting
The ideal research setting for the project would be a University that offers sections of a specific course taught in the classroom to admitted students, and also provides the same course (equivalent or same artifacts, lectures, media, etc.) as an OpenCoursWare option.  Ideally the same professor who designed the course would be the instructor in the classroom, and the principal involved in developing (or supporting) the OCW version of the course.  The purpose of the study is to use the open version of the course as a supplement to classroom instruction and see if there is a correlation between student performance on critical assessments and the time students spent interacting with the open course.

The participants in the study would be the faculty member teaching the course and all of the students enrolled in at least one section of the course in at least one semester.  Two or more periods may provide more robust data that could lead to better analysis.

Data Collection
The professor would include reference to the open version of the course in the syllabus and class discussion with the students the first day of class.  The students would be told that the course materials match those that would be used in the instruction in the classroom and that they can refer to the open course as often as they would like for supplement to their learning of the materials.  The professor would regularly mention the open course supplement throughout the semester.

The students would be asked to keep a record of the amount of time, or the number of times, that they use the open course to aid them in their learning.  The raw test scores students achieve on critical assessments throughout the course would be recorded for data analysis.  At the completion of the course the students will complete a short, non-anonymous survey, that collects qualitative and quantitative data.  The data will include the hours, or number of times, that the student accessed the open course during the semester.

Another piece of data that might be interesting would be to consider the possibility that some students will use the open course as a substitute for class attendance.  Asking about the number of times students actually attended class might also interact with the open course access in considering effects on grade achievement.

Analysis Methodologies
The data would be analyzed by correlating the raw scores on the assessments with the amount of engagement with the open course.  I would see if there is a significant correlation between the use of the open course and the scores students achieved.

If I include the classroom replacement data, we would run multiple regression to determine if there was a significant interaction between class attendance, open course as a replacement, and the accesses to the open course on the achievement of the students.

The report would indicate whether a practice of providing an open version of the course would enhance student performance as measured on critical assessments.  We would also review any qualitative data in hopes that such data that would convey information about the open course itself, the student experience with the open course, and information about the strategies that the students used.

The report would provide analysis for an institution to consider as it's leadership evaluates a possible strategic priority for funding open courses.

Research Project #2 - Open Source Software
Me research on the open source topic revealed numerous studies that considered the motivations for those who participate in open source projects.  One of the basic tenets of the open source movement is stipulated by Raymond in The Cathedral and the Bazaar as what he calls Linus' Law: Given enough eyeballs, all bugs are shallow.

One of the critical benefits of open source projects that is required is to have a large enough base of interested developers that there is a sustainable life for the development project.  Raymond comments that one of the critical principles for success of an open source project is that there is a hand-off when a developer moves away from a project that he or she has been shepherding.  If that happens, there needs to be a new developer who will assume ownership when the incumbent moves on or the project will die.

This challenge prompts me to consider just how effective the community is at developing and sustaining projects once the initial developer has "scratched his/her itch" by proposing (and programming) a project that satisfies their individual need in the open source community.  Do these projects receive life and vibrancy once that proposer of the project leaves?  That is not to say that the project may not be useful in the form at which its continued development stops.  But it would be interesting to know how many of the projects take on an ongoing life with support for upgrades and updates from an interested and organized effort by developers in the open source community.

I propose a research project to find out how the community may respond and support efforts when there is a limited gateway to meter the flow of projects and channel them appropriately.  Who will take up the charge to keep a project current and recruit assistance as needed to update the application?

Research Setting
My proposal is to use the data available at the open source project site  This site accepts projects for open development and interested developers can create logins and then participate in projects.  The site has hosted more than 300,000 projects and has more than 2 million registered users.  I propose that we set our research in an analysis of data from that web site.

The participants who would be included in the project would be those who are associated with the sourcefourge projects.  There would be minimal involvement of these participants, limited to the possibility of outreach by survey to a sample of contributors.

Data Collection
The data collection would be to access data on projects by mining the data available at the site.  The main data to be analyzed would be collected through this mining effort.  A possible addendum to the data collection would be a survey of participants (perhaps voluntary from a notice posted to the site or a random sample to which we reach out).

Analysis Methodologies
The mined data would be used to calculate the number of participants per project and any other useful information that might show how active the community is in support of the projects on the site.  It is assumed that those with very few participants would be in danger of extinction without more developers participating in the project.  We could also look for indications of long lapses of time since the last contribution, or between contributions.  The output would be a profile of projects in the sourceforge database to include duration, participant developers, and any other metrics that might be meaningful in looking at the health and sustainability of the projects.

If we could accumulate survey data we would focus on projects with few developers per project, or long lapses of time between contributions, to determine the reasons that the project that they are developing was suggested, if that need has been fulfilled, if that need continues to exist in a significant installed base, and if there are reasons to update and maintain the code that has been developed?  The intent would be to understand the genesis and life of a project and propose why the observed data about the project are what they are.

Ultimately, we would want to identify a meaningful report structure that would verify the health of open source projects as a movement, or if the movement is restricted to a few superstar projects and then tens of thousands of others that fill a small niche or are not actively developed.  There may be some follow-on research that will tie the motivation research to the data from this study.  Perhaps this can be used to inform those proposing projects on ways to encourage greater and more frequent participation.

Research Project #3 - Open Educational Resources (OER)
One of the most important characteristics of OER is the reduction in the costs to provide educational opportunities to impoverished or less-developed communities.  There have been projects that have opened content and shared copyrighted products using Creative Commons and other open licenses.  Some projects have been undertaken to experiment with the production of open textbooks and schools have been created where the entire curriculum uses OER.  Such efforts have focused on the applying the lower costs of these resources to save education budgets and re-direct funds to other priorities.

My research on this topic led to several studies that were case analyses of the use of OER in different settings.  The research project I would propose for OER is to actually work with educators in less-developed countries where they can participate in an OER-based curriculum and determine whether the suppositions about lower-cost opportunities can be realized.  I would also use the findings as a basis for communicating with interested parties proposals about how an effective implementation of OER could be structured in realizing these benefits.

Research Setting
I would work with international organizations interested in improving educational opportunities for less-developed communities to identify a handful of sites where we might engage the educators, administrators, and the government education agencies to design and implement curriculum using OER.

Study Participants
The teachers, administrators, and government agencies would participate in the study.

Data Collection
The first step in the collection of the data would be to collect information on the costs currently incurred to provide materials, training, ongoing inservice and support, procurement, and general administrative costs associated with the curriculum and instruction.  This cost would be inclusive of all direct and indirect costs, reasonably and rationally allocated.

Once that cost data is accumulated, the next step would be to consider the curriculum design that would be the ideal aspirations of the participants that best fit the culture and infrastructure in the community.  The same costs that were captured in the first step would be estimated using existing materials and services that would be required without an aggressive OER implementation.  This cost would be inclusive of all direct and indirect costs, reasonably and rationally allocated.

The final cost analysis would be to work on an aggressive OER based plan to restructure the curriculum according to the ideal aspirations.  All of the costs would be accumulated, with the appropriate substitutions of the current costs from step one and the traditional education model from step two with the equivalent costs that would replace them in step three.  For example the procurement costs and overhead may be replaced with identification of OER appropriate for instruction and production of the media that would be used.

Analysis Methodologies
The analysis of the data would consider the first two calculations against the actual expenditures experienced in step three.  This analysis would compare the costs of the OER strategy against the existing strategy and the implementation of a new strategy using traditional educational resources.  The study would inform the experience of these communities in the use of OER.  There may be evidence that the cost reduction can allow additional opportunity that would not otherwise be available without the costs savings that the OER strategy delivered.

The purpose would be to align the findings against the proposition that use of OER may provide better educational opportunities for impoverished or developing communities.  I could also formulate proposals for implementing an OER strategy that learns from this research, identifying ways that the expenditures in step three of the project could be modified or eliminated to decrease the funding required.


  1. I think the first proposal is the best.

    The second presumes that if an open source project doesn't attract a large and active developer base that the project is a failure. Is this really true? If the original developer succeeds in scratching his own itch, do we call that scratch a failure just because no one else had a similar itch? I think there is more nuance in the notion of "success" of an open source project than we might realize. It would be great to discuss this more and unpack some of the nuance.

    The third is framed as a "can this work?," yes-or-no kind of question. I think it might be better reframed as "HOW can we make this work?" that might want to simultaneously explore several models in several pilot contexts. But I still like what you've done with it.

  2. David,

    I agree that the second and third projects are not as well thought or structured as the 1st, we obviously know why that is. But, taking into account your comments above, I have reworded some of my thoughts about the last two projects to better reflect my intentions.

    I have changed the wording in the second and third projects so that the presumption that you have made is not as apparent. There is no doubt that this can be construed, but the real intent of the project is to look at the nature of the projects that are on the site and then determine if the data reveals patterns or approaches that developers who might want an enduring project, with a lot of interested participants, might consider placing their work and communicating with participants.

    We may find that the vast majority of the projects on the site were one-offs with the need to scratch a one time itch and no further updating or development will be needed. Knowing that may allow us to recommend structures for these project sites that segregates these projects from those where the developer is seeking a longer life with ongoing programming to keep the application current.

    I agree with the observations about the third project and think that the simultaneous model is best.