Accessibility Skip to Global Navigation Skip to Local Navigation Skip to Content Skip to Search Skip to Site Map Menu

Take our website usability survey

The 2013 LMS Review

January 2013

Prepared for the Committee of Advancement of Learning and Teaching (CALT)

Project Team:

  • Tracy Huntleigh-Smith (ITS)
  • Jenny McDonald (HEDC)
  • Judy Fisher (Library)

Table of Contents

Corrections

This paper was circulated to the ‘List of Interviewees’ for feedback on 14 December 2012. The authors are grateful for the feedback received and have incorporated the feedback into this final version of the report.
N.B. Typographical and grammatical errors (typos) have been noted and corrected where they were immaterial to the paper’s overall meaning.
The following material feedback has been received, and has been listed here for clarity:

^ Top of Page

1. Summary

The Committee for the Advancement for Learning and Teaching commissioned this paper to review the use of Learning Management Systems (LMS¹). A Project Team with representation from the Library, HEDC and ITS was established to undertake the review. The team undertook a comprehensive assessment of the available literature, as well as in-depth consultation with a wide variety of University of Otago academics. We also consulted with vendors and industry analysts.

Several key points arose in the development of the document. These points are highlighted in the document itself and summarised here.

Should the University Change its Learning Management System?

We found that questioning the use of an LMS is common and probably related to unrealistic initial expectations of the technology. However, changing a LMS Platform is a costly exercise and careful analysis is required before proceeding. Any new tool or system is subject to the same ‘hype cycle’, and as such it should be carefully evaluated in-class before being adopted - even on a small scale.

We also found that Blackboard and Moodle mirror each other’s functionality - and that vendors change their products frequently to maintain this parity. Therefore, functionality, features or ‘look and feel’ are no longer adequate reasons for changing an LMS platform. Industry analysts also advised that it is unwise to invest in changes to LMS at this stage of the technology as new models are emerging on the horizon.

The philosophical debate between open source and commercial software is not a valid basis for selecting a LMS platform. Each model has costs involved, and choosing an open source product is not necessarily always the lower cost option.

From an educational perspective there is no convincing evidence to recommend one LMS over another, or indeed to recommend any LMS at all. This review finds that University’s focus is better directed to supporting the use of tools to support learning outcomes – and supporting teachers to innovate in the classroom.

What Changes are Appropriate at Otago?

Our review found that overall, University of Otago satisfaction with Blackboard is reasonably high. Although there are parts of the organisation that still want more functionality; many teaching staff at University of Otago report that Blackboard meets their needs of a LMS and they would not welcome change. Despite the reasonably high levels of satisfaction, however, there are issues with the use of Blackboard that need to be resolved.

Blackboard at Otago is efficiently supported by a very small team which is set up to provide a single system used in a standard and simple manner across a wide customer base. This team is simply not resourced to provide a wider and more comprehensive service. In the course of the review, we found a few excellent instances of Learning Management System use; and that the consistent element in the provision of the excellent service was appropriate resourcing and funding. Regardless of the system used (whether Blackboard, Moodle or Ocean Browser) a superior and tailored service is possible where additional resource is provided.

There are clearly gaps in the service provided at University of Otago. There is a need for:

  • More support for teaching staff who wish to use the advanced features of Blackboard or any other LMS.
  • More support for teaching staff who wish to use alternative tools to support their teaching practice.
  • Better information about the services available to academic departments – i.e., an integrated consistent message which combines information about the Library, ITS, and HEDC.

The Use of New Tools

The use of new tools should be coordinated by a centralised service to ensure that innovations are made available across the wider organisation. In terms of new tools on the horizon we noted that:

  1. The future of LMS will be in smaller and more targeted ‘apps’ rather than large scale platforms.
  2. eTextbooks or Personal Learning Experiences (PLEs) are digital products and services that engage students through interactivity and offer teaching staff choice in content, platforms, devices and learning tools. This is an emerging technology giving students access to their course materials anytime, anywhere – on their desktops, laptops, tablets or mobile phones.
  3. Learning analytics is an emerging field which offers a great deal of potential for Universities. Integration between LMS and the Student Management System will be critical to leveraging these analytics.

Conclusion and Recommendations

This review concludes that there is no evidence to support a change of LMS at University of Otago. Indeed with the LMS market in a state of rapid change it would seem advisable to postpone any new investments in this area until the technology environment is more stable. Current indications are that there is a move away from large platform LMS towards a tool box approach and the investment in specific tools for specific purposes may be more sustainable.

The review has found that the University can and should do more to support the use of current systems. There is a need to provide more information to academic departments about the services available to them, and to provide a consultancy service which supports their use of these tools in their teaching practices.

Based on the above conclusions, it is therefore recommended that:

  1. The University of Otago retains Blackboard as the central Learning Management System.
  2. A group is established in 2013 to review:
  • a. Emerging technologies and how these could be deployed at University of Otago.
  • b. The support arrangements for Learning Management tools, and the use of these tools and technologies in teaching and learning at the University of Otago.

¹ In writing this report we use the acronym LMS to refer to Learning Management Systems in both the plural and singular in order to avoid the ungainly use of LMSs.

^ Top of Page

2 Introduction/Background

This paper looks at the use of Learning Management Systems (LMS²). We analyse the current opinion and case studies of LMS use within other institutions and comment on the use of LMS within the University of Otago.

The paper was requested by The Deputy Vice Chancellor Academic on behalf of The Committee for the Advancement of Learning and Teaching (CALT). The intended audience for the paper is CALT members, although it has been written in a way that can also be circulated to other interested parties in the University Community.

This paper is produced by a team representing the Library, HEDC and ITS.

² In writing this report we use the acronym LMS to refer to Learning Management Systems in both the plural and singular in order to avoid the ungainly use of LMSs.

^ Top of Page

3. Methodology

In order to undertake this review, the Project Team first took a look at the existing literature about LMS. We consulted reports and studies which document: LMS adoption, LMS reviews, and changes of LMS.
We consulted the three internal University of Otago reviews which have also been documented.

We interviewed over twenty University of Otago staff who have been involved with LMS. We asked these people about their experiences with different LMS, and for their feedback about using LMS at Otago. The full list of interviewees is included as Appendix One.

As we developed the concepts in this paper we have discussed the findings with many of the people initially interviewed to test the robustness of the review’s findings.

^ Top of Page

4. General Dissatisfaction with Learning Management Systems

A brief review of the available literature reveals that Otago is not alone in questioning our use of Learning Management Systems (LMS).

4.1. Reviewing LMS is Common

Looking at documented reviews of University LMS suggests that not only is it common to review the use of a LMS, in the past eight years it is also fairly common to re-review or to expect to be re-reviewing its use in the near future. The following reviews are a sample of those published:

This trend to review LMS was so prevalent that in 2011, EDUCAUSE (a non-profit association whose mission is to advance higher education through the use of information technology) published a framework on how to best conduct this kind of review (EDUCAUSE Learning Initiative (ELI), 2011).

The Gartner Hype Cycle

A possible explanation for the general dissatisfaction and review of LMS could well be that institutions set too high expectations at the introduction of the technology. The Gartner Research Hype Cycle (Gartner, 2012) (a tool developed to provide a source of insight in the deployment of new technologies), provides a useful context for this trend. The hype cycle reflects the way technologies tend to evolve in use, and usefulness, over time.

Fig 1 The Gartner Hype Cycle

Fig 1 The Gartner Hype Cycle

In early 2000 LM systems (and Blackboard in particular) were widely adopted in the ‘Peak of Inflated Expectations’, where according to the model “early publicity produces a number of success stories”(Gartner, 2012). Within this phase of high expectation, LMS were heralded as a disruptive technology that would change the way that teaching was delivered.

There were also high expectations of cost savings and increased effectiveness. For example, in 2006 when QUT adopted Blackboard to replace the existing in-house systems, the expected benefits were quoted as:

  • “Increased functionality (i.e. communication tools, content management, tracking, student tools, etc.).
  • Collaborative opportunities for development of tools, content and user services.
  • Easy integration with existing systems, i.e., Turnitin, Mediasite, Peoplesoft, etc.
  • Development and integration of innovative tools without compromising the system’s core.
  • Maximising skills of staff and students coming from other universities.
  • Proven scalability, stability and robustness as experienced at other universities with the opportunity to learn from others.
  • Semi-automated content migration from existing systems.
  • Over time, reduced costs in maintaining and supporting the system”
    (Regina Obexer & QUT Teaching and Learning Support Services, 2006).

By 2009 however, the market was becoming more cautious. California State University conducted a review of the state of the LMS market (Delta Initiative & California State University, 2009); and this review suggests the technology was entering Gartner’s ‘Trough of Disillusionment’ phase. Within this phase of the model, Gartner suggests that:

“Interest wanes as implementations and experiments fail to deliver. Producers of the technology shake out or fail. Investments continue only if surviving producers improve their products to the satisfaction of early adopters.” (Gartner, 2012).

The California State University review commented that “there are no silver bullets to your LMS concerns” reinforcing the fact that by 2009, many LMS were failing to live up to expectations. Common themes in the reviews around this period are the realisation that:

  • All LMS experience problems when scaling up to high course numbers,
  • Teaching staff still require significant technical support in using the technology, and
  • Teaching staff need support to adapt their teaching style to get maximum value from the new media.

The AUT LMS review in 2011 stated that:

“LMS haven't lived up to the hype. Probably because these simply automated current processes and did not support the teachers. Steps need to be taken to ensure the quality of instruction is maintained and enhanced throughout the process - it doesn’t just happen by itself” (Bennett, 2011).

It is important to note that the Gartner model highlights eventual disillusionment with technologies or products as part of a cycle, which for the most part is related to inflated initial expectations. Long term value and productivity enhancements are a feature of the more mature use of a product, which is better understood, better integrated and better supported. While it is tempting to replace a technology or product during the Trough of Disillusionment stage in the hype cycle, this effectively prevents the possibility of longer term value and maturity. Indeed the new replacement products are as likely to be subject to the same cycle and are just as open to eventual frustration.

Key Point:

Dissatisfaction with LMS is common and probably related to unrealistic initial expectations of the technology.

New Tools Also Susceptible to Similar Hype Cycles –and Transition Costs

The 2011 report by AUT notes that there are more than 90 different types of LMS available (Pina, 2010). Although definitive data are very difficult to access, the AUT research found that “of 33 universities across New Zealand and Australia 29 are currently using a version of Blackboard (including WebCT), 12 are using a version of Moodle, one is using Sakai and one is using Desire2Learn” (Bennett, 2011). The LMS landscape is shifting and in addition to evolving products such as Blackboard’s hosted CourseSites, there are also new products being touted as innovators in this space such as Epsilen. It is important that researchers in HEDC and ITS staff keep a watching brief on these. It is however, equally important to recognise that any new tool or system is subject to the same hype cycle. As such it should be carefully evaluated in-class before being adopted - even on a small scale - and certainly before any decision is taken for adoption as a University central service.

The costs of conversion are significant and need to be factored into any decision to change a LMS. Massey University is currently transitioning from Blackboard to Moodle and the initial business case included a projected cost of $4-5 million.

Key Point:

Changing a LMS Platform is a costly exercise and careful analysis is required before proceeding.
Any new tool or system is subject to the same hype cycle. As such it should be carefully evaluated in-class before being adopted - even on a small scale.

4.2. This is Wider than Blackboard versus Moodle

This debate is often polarised as a question of whether an institution should move from Blackboard to Moodle. This came up frequently in our discussions with University of Otago staff.

In the Blackboard / Moodle debate, (The ACODE Educational Technology Survey found that 60% of institutions were investigating moving to Moodle (Lambert, 2011)), the two primary arguments that seem to drive this question are:

  • whether Blackboard is as innovative as Moodle,
  • whether an open source product is better than a commercial model, and
  • that Blackboard was a difficult company to deal with.

These issues are discussed in the following sections.

Innovation/Functionality

We found a common perception that Moodle is a more innovative tool than Blackboard. Nevertheless, this perception was far from universal and a number of our interviewees were quite comfortable with Blackboard – they know how to use it and it (more or less) does what they need it to do. The literature review suggests a simple Moodle versus Blackboard analysis is not straightforward.

Rather than considering Moodle as the innovator in the LMS competitive landscape, Industry analysts such as Gartner and Ovum report that Moodle and Blackboard are becoming increasingly similar, and ‘hitting a plateau’ in the features that they offer. The AUT review (Bennett, 2011) concurs stating:

“Increasingly, reviewers agree that there is little difference between the most recent versions of Blackboard and Moodle’s LMS (Feldstein, 2010; Momani, 2010; R Obexer & Bakharia, 2005)). For example, commenting on a recent review of Blackboard 9.1 and Moodle 2.0.1 undertaken at the Ruhr University in Bochum Germany (Otto, 2011), Thibault states: Blackboard Learn 9.1 (SP3) and Moodle 2.0.1 are extremely similar in their capabilities in about 95% of the features and tools. With a few extensions and additional plugins ... the LMS can mirror each other in 100% of the functionalities they set out to provide." (Thibault, 2011)

Ovum discourages institutions from choosing a LMS strictly on features (or look and feel) because of this similarity and because the underlying technology is quickly changing. A consistent theme in all the literature is that: the way the system is implemented, and how it is integrated into teaching is a more significant indicator of satisfaction with the system. As the AUT review states “How you use the LMS seems to be the key” (Bennett, 2011).

Key Point:

Blackboard and Moodle mirror each other’s functionality and the platforms change frequently. Functionality, features or ‘look and feel’ are no longer adequate reason for changing LMS platform.

Open Source / Cost of Licencing

Another polarising debate is whether an open source product is preferable to a commercially provided system. Both points of view have valid arguments. Open Source products are growing in popularity, although they are clearly not a panacea, and it is important to view early successes with caution.

It is however important to understand that open source technology is not ‘free.’

“The open-source philosophy aligns with higher education's preference for freedom and flexibility. … Nevertheless, Ovum reminds institutions that both approaches require system configuration, training, maintenance, and integration, all of which involve significant financial and human investment. … Ovum warns against the misperception of open source as "free" software, and emphasizes the importance of investing in the necessary planning and support, and ultimately, understanding that control and flexibility have their price as does the cost to support open source” (Ovum Analyst, 2010).

Ovum (Tsai, 2010) notes that many organisations see open source products as a way to avoid the expense of pre-packaged solutions that often contain more functionality than they use or need. However, they also go on to state that “as many institutions lack the technology resources necessary to maintain an open source project independently, their reliance on open source support consultants is equally costly”.

However, many institutions lack the technology resources necessary to maintain an open source project independently. Open source vendors like Moodlerooms and rSmart offer solutions and support that help institutions enjoy the advantages of open source without bearing the burden of platform upgrades, maintenance and solution customizations. In a sense, these affiliates serve as an intermediary step in the open source movement (Tsai, 2010).

A local example of this is the adoption of the Otago developed open-source media-sharing application, UniTube by Te Wananga o Raukawa (TWoR). While University of Otago made UniTube freely available under an open-source licence; TWoR commissioned the services of Catalyst IT to support them with their local implementation of UniTube.

University of Otago staff also commented that Blackboard licencing is expensive and suggested that this money could be diverted into better internal support if an open source product was used. The annual licencing cost for Blackboard is less than NZD$120,000. Based on experiences at other universities this amount would easily be absorbed by third party intermediaries integrating an open source option.

Key Point:

The philosophical debate between open source and commercial software is not a valid basis for selecting a LMS platform. Each model has costs involved, and choosing an open source product is not necessarily always the lower cost option.

Blackboard Vendor Relationship

In our conversations with University of Otago staff, there was a view that Blackboard is a difficult company to deal with. Staff were concerned that Blackboard had been particularly unhelpful in dealing with problems associated with a recent upgrade. The ITS staff who have responsibility for this vendor relationship also agreed that this was a particularly challenging time. They noted however that all vendor relationships tend to have challenging elements to them, and that the wrong account representation can often detract from a good product. Emerson Pratt (Manager Teaching and Learning Facilities) reported that in the last two years Blackboard has undergone significant organisational restructuring. From an Otago account perspective, Blackboard has subsequently replaced all the account staff involved in this poor upgrade interaction and the eLearning Team are finding the new account staff are more responsive.

4.3. University of Otago Reviews Consistent With External Findings

University of Otago commissioned three formal reviews dated 2001, 2009 and 2010, and these reviews are consistent with both the Gartner Hype Cycle and the external industry findings.

In 2001, after the introduction of Blackboard’s precursor CourseInfo at Otago, a review found that the system had been well implemented and was generally well accepted by staff and students (Gunn, Peddie, & Auckland UniServices Ltd, 2001). (It is however, interesting to note that this review was already highlighting that there was a risk in the lack of system & support resources.)

In 2009, a review supported by CALT and conducted by OUSA, HEDC and ITS found that material on Blackboard was often poorly organised and not well linked to the overall course objectives. Course providers were however effectively using the lower level functions of Blackboard such as the announcements, discussion boards and course documents.³ (Aman et al., 2009).

In 2010, the University of Otago Quality Advancement Unit (QAU) also reviewed the use of Blackboard and found that while most of the previous issues were still occurring, the level of satisfaction with Blackboard was “reasonably high” with 70-87% staff members broadly satisfied with the system (Mirosa, 2010).

A fourth subsequent review in 2011 looked at the use of Blackboard in specific courses within Health Sciences. The feedback from Health Sciences is notable in considering the QAU review – Health Sciences perspective is that the reported 70-87% of teaching staff indicating broad satisfaction with Blackboard overlooks the fact that the smaller numbers of dissatisfied teaching staff represents much larger numbers of students (Blyth, 2011).

In our most recent interviews with Faculty of Medicine staff, it should be noted that one of the key reasons for the adoption of Moodle was the flexibility of its calendaring feature compared to Blackboard. This, coupled with the Faculty hosting and management of Moodle has provided significant gains that may not have been readily achieved through the centrally supported Blackboard implementation. Nevertheless, the specific requirements of the Medical Faculty for flexibility with respect to calendaring, organising large numbers of clinical staff and small group of students do not necessarily translate across the University; and in fact the need was not mirrored in our discussions with those involved in, for example, the large First Year Health Sciences courses.

Key Point:

Overall University of Otago satisfaction with Blackboard is reasonably high although there are parts of the organisation that still want more functionality.

³ The 2009 review also identified some technical issues e.g., limited bandwidth which were causing Blackboard to ‘crash’ however the Manager of Teaching and Learning Facilities advised that under more detailed analysis this was later found to be incorrect.

^ Top of Page

5. The Industry is in a State of Flux

In 2012, all New Zealand universities attended a workshop hosted by Gartner Research which considered the emerging trends in LMS (Harris, 2011). The industry analysts’ view was that the LMS market is in a state of flux, and that it would be short-sighted for LMS customers to make significant changes within this stage of change. Major vendors appear to be re-evaluating their approach to market as well as their core infrastructure.

It was considered that (as noted in earlier sections of this paper) most LMS providers have focused on adding more and more functionality to their platforms in order to compete with the functionality provided elsewhere in the market. The result of this process for tertiary institutions who use the LMS is that services are duplicated across a number of providers. For example several systems provide an email add on – despite the fact that most tertiary institutions already provided their own email platform and that both Google and Microsoft were offering free cloud based email services for students.

Additionally, as systems have developed over time, even open source products which were previously considered as innovators are having to grapple with legacy issues, i.e., parts of their original build now conflict with the emerging services they wish to provide.

Key Point:

Industry analysts advise against making changes to LMS infrastructure in the short term.

5.1. Learning Stacks / Tool Box Approach

Moving to a tool box approach is an emerging theme. The AUT Review questions the appropriateness of using LMS to serve as a top-down distribution vehicle for academic resources while simultaneously expecting innovation and collaboration in this space from the bottom-up.

Teaching and learning are highly context-dependent endeavours. If teaching staff are to be supported to provide the most appropriate learning environment for their context and their students then that support needs to be highly flexible. The AUT Review suggests the notion of 'simpler' tools that do only one or two things but which can be combined in a variety of ways.

This is consistent with the current trend towards 'apps' both on mobile devices and computers. Certainly Blackboard and Moodle are already embracing this trend with for example Blackboard Mobile and Moodle Mobile. These 'apps' provide access to resources hosted on the LMS in an accessible and simpler mobile format.

The Gartner industry analyst’s view supported the AUT concept of small pieces loosely combined. The view was that in the short to medium term, LMS would evolve from large standardised platforms to a tool box environment. There is a strong expectation of interlinked applications which would be unique to departments, and potentially even a tool box which would be unique to the level of individual instructor (Lowendahl, 2011).

5.2. Personal Learning Experiences (PLEs) and eTextbooks

Personal Learning Experiences (PLEs) and eTextbooks go beyond the current concept of an eBook, a course delivery platform or Learning Management System to deliver a personal, interactive and device agnostic suite of products and services that engages students through interactivity and offers teaching staff choice in content, platforms, devices and learning tools.

This is an emerging category of technologies which gives students access to their course materials anytime, anywhere – on their desktops, laptops, tablets or mobile phones.

Key Point:

The future of LMS will be in smaller and more targeted apps rather than large scale platforms.

^ Top of Page

6. Pedagogical Review

This section begins with a plea quoted directly from a 2005 review conducted by Prof. Richard James and colleagues at the Centre for the Study of Higher Education at the University of Melbourne:

“Decisions about university teaching and learning should not be restricted to checklist evaluations of technical and organisational factors. It is vital to maintain the educational perspective rather than emphasise any technological determinism which takes specific characteristics of online systems or teaching for granted.” (Coates, James, & Baldwin, 2005)

There has been little research into the relationship between LMS and the quality of student learning outcomes. (See for example: Coates et al., 2005; Kember, McNaught, Chong, Lam, & Cheng, 2010; McGill & Klobas, 2009).
By contrast, there is a plethora of reports and studies which compare technical and administrative features of LMS and which document LMS adoption (McGill & Klobas, 2009). Since the late 1990s the adoption of LMS by tertiary institutions worldwide has been both swift and comprehensive. According to Coates et al., by 2005 70% of universities in the UK, Australia and Canada have licenses for one or both of Blackboard and WebCT (WebCT has been bought out by and integrated with Blackboard more recently). In 100% of NZ universities in 2012 it is safe to say that there is widespread use of LMS, typically Blackboard or Moodle.

(Coates et al., 2005) suggest six key drivers that have driven the wholesale adoption of LMS by the university sector:

  • Increasing efficiency through for example, reducing course management overheads, and physical space demands.
  • The promise of enhanced student learning through the provision of access to a larger range of resources, increasing the opportunities for interaction with teaching staff and peers, and by mediating automated and/or adaptive formative assessment.
  • Student expectations for advanced technologies.
  • Competitive pressure between tertiary institutions at both the national and increasingly international level.
  • Response to increasing demands for greater access to higher education.
  • Control and regulation of teaching.

Arguably only the second of these drivers provides a strong educational perspective and even here the promised affordances could be seen to be largely related to ease of access and administration rather than being grounded in strong educational principles.

The last point is probably the most controversial but Coates et al., argue that the perceived order created by LMS may be one of the most persuasive reasons for their uptake in an environment which increasingly is obliged to respond to the demands of quality assurance and control.

All of which begs the question, how do we maintain the educational perspective promoted by our colleagues in Melbourne, with whom we would certainly agree, in the face of drivers which arguably have little to do with education?

In the absence of many studies which specifically investigate the educational impact of LMS and which allow us to generalise, we will digress briefly to see what research into educational technologies more broadly over the last thirty or forty years tells us about their impact on student learning outcomes.

The brief version of the impact of technologies when considered in isolation from other educational interventions or practices is that there is in fact little impact. Large scale meta-analyses, for example Hattie (2009) and Tamim, Bernard, Borokhovski, Abrami, and Schmid (2011) conclude that the average effect size from technology alone is small (0.2-0.3) and is below 0.4 which is the average effect size for all factors investigated⁴. It is difficult to see why LMS would fare any differently, on average, than any other technology. This could hardly be called a compelling educational argument for their wholesale adoption.

Among the factors which make the most difference to student learning outcomes are feedback, instructional quality, direct instruction, students' prior ability and disposition to learn. Nevertheless, it is important to set these 'high-impact' factors in context too and to remember that the effect sizes reported in Hattie's analysis are averages. If technologies, including LMS, are used wisely to support educational strategies which we know make a difference, then this surely is no bad thing.

LMS are with us in a variety of forms, although they are, as noted elsewhere, becoming increasingly alike. The challenge facing the University of Otago and indeed every other tertiary institution is to support effective teaching and learning with the technology tools of the day. From an educational perspective there is no convincing evidence to recommend one LMS over another or indeed to recommend any LMS at all. This is not to say we should rush to abandon our investment in the existing University LMS, far from it. The ensuing disruption to established practices, especially in large-class settings would likely have significant impact on teaching and learning in the short-term. Rather we should focus on supporting teachers and students to use the most appropriate tools and technologies available to achieve their desired teaching and learning outcomes. In addition we should be supporting teachers to innovate and evaluate in the classroom, including with the use of contemporary technologies.

Key Point:

From an educational perspective there is no convincing evidence to recommend one LMS over another or indeed to recommend any LMS at all. The University should focus on supporting the use of tools to support learning outcomes – and supporting teachers to innovate in the classroom.

This is a significant challenge, but through further developing co-operation between HEDC, ITS and the University Library to enhance effective teaching and learning through research-informed support, we believe it is a challenge that can be met.

A recent local example of a case-study where LMS resources are carefully integrated with the curriculum in the context of 5th year medical teaching is reported in the latest issue of AJET (Stebbings, Bagheri, Perrie, Blyth, & McDonald, 2012).

⁴ The Hattie study noted average effect sizes between about 0.2-0.3 (the range of effect sizes for factors which might be included as technology was between 0.09 - for example, web-based learning to 0.41 - for example, hypermedia). Tamin et al. found the average effect size ranged between 0.30 – 0.35. As Hattie has pointed out and as reiterated by Prof. Tom Reeves during his visit to Otago in 2011, an average effect size of less than 0.4 is, in general, not necessarily meaningful as the average effect size across a wide range of educational factors is around 0.4. Tamin et al. also issue a note of caution in interpreting average effect sizes because of the wide range of variability between individual effect sizes and uncertainty about the factors which may have produced the variability.

6.1. Learning Analytics

Learning analytics is a developing area of research in Higher Education.

Information captured about student actions or input in the LMS or any other online teaching tool can help us to understand student conceptions and can therefore be used to inform teaching practice and the development of teaching resources. There is also the potential for information relating to student performance to be married with the Student Management System (SMS) in order to derive a comprehensive view of student performance (academic analytics).

A recent review from the Open University, Knowledge Media Institute, adopts the definition in the call for papers of the first Learning Analytics and Knowledge conference and points out that this definition assumes that large digital data sets are involved and techniques are employed which would not normally be practical without computers (Ferguson, 2012).

“Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs.” (LAK '11 Call for Papers, 2011).

Van Barneveld, Arnold, & Campbell (2012), in a recent Educause review, suggest that the focus on students and their learning behaviours in learning analytics distinguishes it from business analytics and academic analytics, where the focus is at the institutional level.

Closer to home, The Australasian Council of Open, Distance and e-Learning, of which the University of Otago is a member, is running a workshop on Learning Analytics in Perth, in November 2012 and is actively seeking examples of learning analytics from member institutions.

Key Point:

Learning analytics is an emerging field which offers a great deal of potential for Universities. Integration between LMS and Student Management Systems will be critical to leveraging these analytics.

^ Top of Page

7. Current Situation at University of Otago

There are a number of different LMS in use at the University: the centrally supported LMS is Blackboard, there are several Moodle implementations, and Ocean Browser is a commercial system which was initially designed for the Ophthalmology department.

This current review team have found anecdotal evidence that Blackboard is still primarily being used in a standard and basic manner. The “reasonably high” level of satisfaction reported by University of Otago Quality Advancement Unit (QAU) reflects this standard and basic use of the system.

Despite the reasonably high levels of satisfaction evidenced in the QAU report, there are issues with the use of Blackboard that need to be resolved. To understand why this is an issue, it is helpful to look at the different ways of engaging with the LMS within the University.

Key Point:

Despite the reasonably high levels of satisfaction, there are issues with the use of Blackboard that need to be resolved.


7.1. Varying Levels of Engagement with a LMS by University Users

When discussing the current state of LMS in use at University of Otago, it is important to recognise that our teaching staff are a heterogeneous group, i.e., different teachers have distinctly different requirements and ways of engaging with the LMS that they use.

During the course of this review we found four general types of engagement with a LMS and these are outlined in the following diagram.

Fig 2 Varying levels of engagement with an LMS

Fig 2 Varying levels of engagement with an LMS

Figure 2 depicts the use of the LMS by course provider; as the course provider (rather than the students) is largely responsible for the choice of LMS used, the way that it is used, and if an LMS is used at all.

It is notable that some teaching staff choose not to use a LMS. At the beginning of each semester, the ITS Service Desk fields many enquiries from students who think the system is faulty because a paper in which they have just enrolled can’t be found on Blackboard. These students are redirected to the course provider who will have a different mechanism for distributing course materials.

Efficient Central Support for Standard Blackboard Use

The support structure for Blackboard at University of Otago is designed to support the bulk of teaching and support staff who use the system in a basic and habitual manner. A small technical team (approx. 3 FTE) manages upgrades, responds to help calls, and supports teachers through peak times, e.g., rolling over papers etc. ITS can support this standard use of the system with very few resources because there is little customisation for individual needs. Training can be provided efficiently, i.e., standard classroom-based training modules as part of the internal training schedule. This is overall, an efficient support model which supports the use of a single system used in a standard and simple way.

The generally positive feedback reported in the QAU survey (Mirosa, 2010) largely maps to the first two types of users. For those with infrequent, basic or habitual use of LMS functionality, the centrally provided Blackboard service works well. The main value for these teachers is to have one place where students can go to find information. Blackboard is largely used to share content using standard features. These teaching staff do not welcome change, either in terms of learning new functionality or potentially having to re-write class materials for a different system.

Key Point:

Many teaching staff at University of Otago report that Blackboard meets their needs of a LMS and they would not welcome change. The system is efficiently supported by a very small team which are set up to provide a single system used in a standard and simple manner across a wide customer base.

Little Support Available for Cutting-Edge Use

There is, however, no central resource assigned to support advanced use of LMS, or those teachers who prefer to use alternative systems.

Fig 3 Central support for Blackboard caters to standard use

Fig 3 Central support for Blackboard caters to standard use

Centres of Excellence

Some departments have resources to support LMS customisation, (either Blackboard or alternative LMS), and these departments also show high levels of satisfaction with their LMS - regardless of the system.

We found three examples of excellence. These areas have tailored their site to their own specific needs and have provided excellent support for their teaching staff and students. The areas are:

  • The Anatomy Department Moodle Implementation
  • The Centre for Post Graduate Nursing Research Implementation of Blackboard.
  • The Ophthalmology Department Implementation of Ocean Browser

In these instances, the additional departmental staff provides a level of service and customisation that would be difficult to replicate at an enterprise-wide level. For example, it is clear that the Anatomy Department’s IT staff have a deep knowledge of the department’s subject matter and processes. With similar levels of resourcing to that of the ITS eLearning team (which supports the central Blackboard implementation across the University as well as the eResearch portfolio), the Anatomy team can provide tailored services which are highly valued by the department. In a similar vein, the Post Graduate nursing implementation of Blackboard is recognised in the department as a highly useful tool which contributes value for staff and students and assists in improving retention rates. This has been achieved by employing a specific staff member to tailor and manage the LMS specifically for their unique needs.

Key Point:

Regardless of the system used - a superior and tailored service is possible where additional resource is provided.

Key Areas Where Current Support is Insufficient

Dissatisfaction with Blackboard tends to be reported by departments which want to operate at the cutting edge use of Learning Management, but which have to rely on limited central resources to support their use of Blackboard. These teaching staff typically need to:

  • support large classes,
  • use advanced features such as adaptive release testing, or
  • use the more interactive features of the system.

These features cannot easily be supported by the limited central resources without either disadvantaging other users, or providing more resource to the central support team.

There is a need to consider the specific needs of teaching staff related to their specific circumstances, whether this is distance learning or on-campus, large or small classes and to provide the specific functionality they need such as:

  • lab manuals,
  • lecture slides & PDFs,
  • links to lecture podcasts,
  • learning module(GLM) PDF files,
  • ImageViewer electron micrographs,
  • study group worksheets,
  • studysmart,
  • online tests for lab assessments,
  • exit tests,
  • adaptive release modules,
  • discussion boards,
  • course announcements,
  • grade centre to provide confidential advice to students of test/exam results.

This wide variety of tools is unlikely to be excellently provided by a single system.

Key Point:

There is clearly a need for more support for teaching staff who wish to use the advanced features of Blackboard or any other LMS.

Little Support for Experimenting/Innovation with Learning Management Tools

There is also limited resource available to assist teaching staff in experimenting with new tools whether these are embedded within an LMS (for example online tests) or external to it (for example the ImageViewer application developed by HEDC). In our interviews we came across teaching staff who wanted to try out new systems; either because they had become frustrated with Blackboard, or because they were simply interested in using new tools. This highlighted that while HEDC have a role to assist staff in identifying and experimenting with new tools, this is not consistently understood across the University.

Key Point:

There is clearly a need to more support for teaching staff who wish to use alternative tools to support their teaching practice.

We found that the existing use of alternative systems has largely been driven at a departmental level. The risk of this situation is that the knowledge about the innovation is siloed within departments and not centralised for dissemination to other parts of the University; essentially the innovative departments become hubs of knowledge about the new tool.

It was even suggested by some of the people we interviewed, that these departments should go on to become service providers to other departments wanting to use the innovation. This would of course add to the duplication of services within the University as well as detracting academic departments from their core purposes of teaching and research.

A preferable solution would be to establish a process involving HEDC, ITS and the Library whereby these innovations can go on to be supported centrally supported if they are to be extended to other parts of the University.

Key Point:

Once an innovation has been identified, a centralised service should ensure that the innovations are made available across the wider organisation.

The Role of Internal Service Providers in Supporting LMS

When interviewing internal support staff, we found variations in teams’ view of their own roles. We suggest the following breakdowns are the most commonly held views but that this is not uniformly agreed.

Fig 4 Breakdown of views - Library vs ITS vs HEDC

We also found varying understanding of the service group’s roles by teaching staff throughout the University.

A lack of awareness about what help was available and where and how to get the assistance they needed was a common feature of those people who cited dissatisfaction with the LMS. Few departments saw providing a LMS as their key function, and may not have proceeded down that path if there was a clearer link to central services.

Key Point:

There is a need to provide more information to academic departments about the services available to them. This information needs to be an integrated consistent message which combines information about the Library, ITS, and HEDC.

^ Top of Page

8. Conclusion

In conclusion, this review has not found any evidence to support a change of LMS at University of Otago. Indeed with the LMS market in a state of rapid change it would seem advisable to postpone any new investments in this area until the technology environment is more stable. Current indications are that there is a move away from large platform LMS towards a tool box approach and the investment in specific tools for specific purposes may be more sustainable.

The review has found that the University can and should do more to support the use of current systems. There is a need to provide more information to academic departments about the services available to them, and to provide a consultancy service which supports their use of these tools in their teaching practices.

^ Top of Page

9. Recommendations

Based on the above conclusions, it is therefore recommended that:

  1. The University of Otago retains Blackboard as the central Learning Management System.
  2. A group is established in 2013 to review:
  • a. Emerging technologies and how these could be deployed at University of Otago.
  • b. The support arrangements for Learning Management tools, and the use of these tools and technologies in teaching and learning at the University of Otago.

^ Top of Page

Bibliography

Aman, K., Butson, R., Epere, A., Hussein, R., Holdsworth, S., & Thomson, C. (2009). Blackboard@Otago: A student perspective. Report to the Committee for the Advancement of Learning and Teaching (CALT). Dunedin: University of Otago. Dunedin: University of Otago.

Bennett, S. (2011). AUT University: Learning management systems: A review. Auckland: Auckland University of Technology.

Blyth, P. (2011). eLearning Review Document prepared for Faculty of Medicine Curriculum Committee. University of Otago, Faculty of Medicine. Dunedin.

Carvalho, A., Areal, N., & Silva, J. (2011). Students' perceptions of Blackboard and Moodle in a Portuguese university. British Journal of Educational Technology, 42(5), 824-841.

Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of learning management systems on university teaching and learning. Tertiary education and management, 11, 19-36.

Croy, M., Smelser, R., & McAlpin, V. (2009). Report to the provost from the learning management system evaluation committee (pp. 1-38). Charlotte: University of North Carolina.

Delta Initiative, & California State University. (2009). LMS Webinar: The state of learning management in higher education systems Retrieved 5th May, 2012, from http://www.deltainitiative.com/archived-webinars/new-choices-the-state-of-learning-management

EDUCAUSE Learning Initiative (ELI). (2011). 7 things you should know about LMS evaluation. Retrieved from http://www.educause.edu/library/resources/7-things-you-should-know-about-lms-evaluation

Feldstein, M. (2010). The evolving LMS market, part ii. eLiterate: What we are learning about online learning...online. Retrieved 6th June, 2012, from http://mfeldstein.com/the-evolving-lms-market-part-ii/

Ferguson, R. (2012). The State Of Learning Analytics in 2012: A Review and Future Challenges. Technical Report KMI-12-01, . UK.: Knowledge Media Institute, The Open University.

Gartner. (2012). Research methodologies: Hype cycles Retrieved 5th May, 2012, from http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp

Gunn, C., Peddie, R., & Auckland UniServices Ltd. (2001). Evaluation of the implementation of the CourseInfo project: Final Report prepared for the University of Otago. Auckland, NZ: University of Auckland.

Harris, M. (2011). Gartner Presentation: Building learning stacks for an evolving learning environment: Gartner Inc.

Hattie, J. (2009). Visible learning: a synthesis of over 800 meta-analyses relating to achievement. New York: Routledge.

Johal, N. (2012). Learning Management Systems: the Competitive Landscape. OVUM. Retrieved from http://ovum.com/research/learning-management-systems-the-competitive-landscape/

Kember, D., McNaught, C., Chong, F. C. Y., Lam, P., & Cheng, K. (2010). Understanding the ways in which design features of educational websites impact upon student learning outcomes in blended learning environments. Computers & Education, 55(3), 1183-1192.

LAK '11 Call for Papers. (2011). LAK '11: 1st International Conference on Learning Analytics and Knowledge 2011 Retrieved 12 October, 2012, from https://tekri.athabascau.ca/analytics/call-papers

Lambert, S. (2011). ACODE 2011 Educational Technology Survey: ACODE: the Australasian Council on Open, Distance and E-Learning.

Lowendahl, J.-M. (2011). Gartner Industry Research: Hype Cycle for Education. G00214466, 2012, from http://www.gartner.com/id=1755718

McGill, T. J., & Klobas, J. E. (2009). A task-technology fit view of learning management system impact. Computers & Education, 52(2), 496-508.

Mirosa, R. (2010). Blackboard Support Survey Report. Dunedin: Quality Advancement Unit, University of Otago.

Momani, A. M. (2010). Comparison between two learning management systems: Moodle and Blackboard. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1608311

Obexer, R., & Bakharia, A. (2005). OLT System Review: Environmental Scan Report. Brisbane: Queensland University of Technology.

Obexer, R., & QUT Teaching and Learning Support Services. (2006). OLT System review recommendation Brisbane: Queensland University of Technology.

Otto, R. (2011). Feature Comparison Moodle-Blackboard. Bochum, Germany: Ruhr-Universitat.

Ovum Analyst. (2010). 2011 Trends to Watch: education technology. Consumer preferences are shaping the new education landscape.

Pina, A. A. (2010). An Overview of Learning Management Systems. In Information Resources Management Association (Ed.), Virtual learning environment: concepts, methodologies, tools and applications (pp. 33-51). Hershey, PA: Information Science Reference.

Randall, B., Sweetin, J., & Diane Steinbeiser, M. A. (2010). North Carolina Community College System Office: Learning Technology Systems Learning Management System Feasibility Study. Part II of the Open Source Collaborative Moodle Assessment Report: North Carolina Community College.

Sclater, N. (2008). Large-Scale Open Source E-Learning Systems at the Open University UK. Educause Centre for Applied Research, Research Bulleting, 2008(12).

Stebbings, S., Bagheri, N., Perrie, K., Blyth, P., & McDonald, J. (2012). Blended learning and curriculum renewal across three medical schools: The rheumatology module at the University of Otago. Australasian Journal of Educational Technology, 28(7), 1176-1189.

Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What Forty Years of Research Says About the Impact of Technology on Learning: A Second-Order Meta-Analysis and Validation Study. Review of Educational Research, 81(1), 4-28. doi: 10.3102/0034654310393361

Thibault, J. (2011). Blackboard vs Moodle 2.0: 100+ metrics compared head to head by @MoodleRUB. Moodle News. Retrieved from http://www.moodlenews.com/2011/blackboard-vs-moodle-2-0-100-metrics-compared-head-to-head-by-moodlerub/

Tsai, J. (2010). Manage the New Higher Education Environment with LMS (Analyst Insight). Retrieved from Ovum website: http://ovum.com/ (subscription needed)

University of Canterbury. (2008). Learning Management Systems review: Final report and recommendations.

van Barneveld, A., Arnold, K. E., & Campbell, J. P. (2012). Analytics in higher education: Establishing a common language. Educause Learning Initiative, 1, 1-11.

Wheatley, M. (2004). Open Source: The Myths of Open Source. CIO: Technology Topics. Retrieved from http://www.cio.com/article/32146/Open_Source_The_Myths_of_Open_Source

^ Top of Page

Appendix 1: List of Interviewees

Bill Anderson
Phil Bishop
Phil Blyth
Amanda Clifford
Jo Forrester
Kirsten Franklin
Phil Handcock
Katherine Harris
Mike Harte
Andrea Howard
John Kaiser
Phil Kelly
Wing Lai
Beverley Lawrence
Eileen McKinlay
Ross Marshall Seeley
Paul Muir
Emerson Pratt
Craig Rodger
Gordon Sanderson
Terry Scott
Annemaree Senior
Shane Sturgeon
Will Sweetman
Steve Woods
The Health Sciences First Year Operational Committee Meeting

^ Top of Page