NEEDS The National Engineering Education Delivery System:
Brandon Muramatsu and Alice M. Agogino
1.0 IntroductionThe rapid growth of the Internet and global network systems for the development, presentation and transfer of digital information is causing educators to rethink the method in which they provide information. The advent of low cost, widely accessible computers and high bandwidth networks has opened the door for new methods to enhance and transform the traditional education process. Throughout the education community, the quantity of material available digitally is increasing daily as the "publishers" of this new era are making many of their materials available on the World Wide Web. Digital courseware, computer-based tools, large datasets, and electronic supplements to learning are becoming widely accessible by faculty, students and learners in general.
NEEDS The National Engineering Education Delivery System (see www.needs.org) is the distributed architecture developed by Synthesis: A National Engineering Education Coalition (see www.synthesis.org) to enable new pedagogical models based on Internet-mediated learning environments [1, 2]. The emergence of the World Wide Web (WWW) in the early 1990's as a viable means for national and international sharing and re-use of education materials fundamentally changed our view of the way education and learning can be delivered. Internet-mediated learning environments provide mechanisms for the learner to be anyone, anywhere, at anytime. NEEDS has expanded its scope and currently catalogs courseware and other instructional software developed nationally and internationally to provide a resource where both instructors and learners can search, access, and download educational materials over the World Wide Web. In addition, NEEDS supports a multi-tier courseware evaluation system including a national award competition the Premier Award for Excellence in Engineering Education Courseware .
NEEDS' vision of what a digital library for undergraduate engineering education should be is more than just a traditional academic library in digital form. The digital library of the future will be a community of learners encompassing faculty, students, and life-long learners. With our current system, NEEDS has taken the first steps to developing this "new" digital library; we are driven by recent advances in research on education and student learning. We distinguish ourselves from commercial, Web-based search engines by providing focused, value-added services to our community. NEEDS is in the process of supplementing our current services by adding features such as user reviews and attachments, user registration, and discussion tools.
This article will provide a description of the NEEDS project. Following a brief history, we will examine the current NEEDS system. We will then outline our current thrusts in engineering and outline the new features we are deploying. Throughout the article we outline some of the challenges in developing a digital library of educational materials and describe how some of our current thrusts address these challenges.
2.0 Brief History
Synthesis: A National Engineering Education Coalition, with funding from the National Science Foundation and industrial partners, was established in 1989 to reform undergraduate engineering education. This broad, nationally distributed coalition of eight diverse universities (including California Polytechnic State University, San Luis Obispo; Cornell University; Hampton University; Iowa State University; Southern University; Stanford University; Tuskegee University; and the University of California, Berkeley) embraced instructional technologies as a means of assisting our curricular goals in the reform of undergraduate engineering education. Courseware, whether it is stand-alone and CD-ROM based, or networked and accessible over the Internet, provides methods to enhance the traditional learning process and address multiple learning styles. Key in the early stages of the Coalition was a vision to enable the sharing and reuse of materials developed within the Coalition.
Synthesis developed NEEDS as the mechanism to support the development, use and more importantly re-use of these instructional technologies. The nationally distributed nature of the Coalition and our interest in cross/multi-disciplinary education proved to be extremely important. Our experiences in dealing with a wide variety of institutions, technologies, and disciplinary and multidisciplinary content provided us with a broad view of how a national resource can be used, who can be its target audience, what instructional technologies can be made available through this resource, and what's needed to describe them.
2.1 NEEDS Infrastructure
NEEDS has developed a robust infrastructure built around a distributed server architecture. We maintain strong ties to the best of state-of-the art research in databases, information retrieval, and digital libraries. These ties to research coupled with a production focus have allowed us to evolve NEEDS in sync with the rapid changes occurring with information technologies in order to meet the needs of our users. From its debut in the early 1990's as a "text-based search engine over a centralized library catalog with pointers to digital course material"  accessible via telnet to its transition to the Web in 1994 to the current system of 1999, NEEDS has undergone three major updates in response to users' needs collected through surveys of faculty and students in engineering education. NEEDS is ultimately about providing a service to our user community; we apply the appropriate technology to meet our users' needs. Along the way, we have developed and implemented key elements in a reliable, distributed, scalable system architecture, including: redundant Web servers; an integrated database with WWW-based searching and downloading, utilizing state-of-the-art database and information technologies; and an indirection system to support multiple, nationally-distributed archive servers for courseware download.
NEEDS continues to grow and evolve based upon international standards for data description (USMARC and emerging metadata descriptors) and data access (HTTP and the World Wide Web), because they allow access to all users [4, 5]. NEEDS uses a robust structured query language (SQL) relational database to store and index courseware records. The underlying framework for these courseware records is based upon a standard library format for indexing and storing documents (USMARC) [4, 5]. The USMARC standard lets us apply a wealth of cataloging experience gained in the library community. Key to continued success is evolution, which currently includes adoption of the metadata descriptors developed by the Educause Instructional Management Systems (IMS) project .
Materials indexed and searchable through NEEDS are diverse content ranges from single topics that can be covered in a few minutes to fully integrated, term-long courses. One of the most powerful concepts supported by NEEDS is courseware modularity, in some cases NEEDS catalogs courseware as well as the individual elements (e.g., images, videos, and text) that comprise the courseware. Courseware elements provide a vehicle for continued re-use of content material beyond the lifespan of any particular courseware module. These elements can be used as is, or distilled from multiple sources and joined together to create new, customized courseware. The modularity supported within NEEDS is seen as a major enabling technology for fostering educational material adaptation and re-use.
Courseware cataloged in NEEDS is acquired through a Web-based interface. Authors and other submitters are able to add materials to NEEDS in the same fashion that learners can access those materials, anytime, anywhere. Consistent with library cataloging standards, we perform a basic review of courseware as it is cataloged. We provide this review to insure that the courseware record has certain required fields e.g., a title, author, publisher and platform (e.g., PC, Mac, or WWW); to perform a standard check for viruses; and to verify that the program can be operated on the intended platform(s). We use this basic functionality check in place of forced editorial oversight before courseware archiving; we have learned that it is counterproductive to introduce too many barriers to the cataloging process. Instead we have developed review systems to evaluate courseware once archived in NEEDS.
2.2 NEEDS A Use Scenario
Perhaps the best way to understand the features of NEEDS is through an illustrative use scenario.
3.0 Quality Review of Courseware
Courseware on NEEDS, as with any scholarly effort, requires review to assure quality and provide recognition . The Quality Review of Courseware effort was developed to advance the field of technology-enhanced learning review as well as continually raise-the-bar for excellent courseware. The first step in developing a review procedure was to examine existing schemes of software, courseware and paper review and adopt those aspects that were deemed most appropriate for a rapidly changing environment of courseware creation, in particular, and multimedia technology, in general.
3.1 Evaluation Criteria
Since 1994, NEEDS has worked with numerous experts nationwide including students, engineering educators, instructional designers, cognitive scientists, and learning theory experts [7, 8], to develop and implement a quality review procedure for engineering education courseware. This process, led by Professor Pamela A. Eibeck, Chair and Professor of Mechanical Engineering at Northern Arizona University, has resulted in evaluation criteria and a multi-tiered review system for courseware. Based on our initial study , NEEDS has developed and has developed a comprehensive, tiered peer review including: non-reviewed, endorsed, and premier courseware.
The key ingredient to the peer review process was the development of evaluation criteria for engineering education courseware. In developing the criteria, NEEDS has taken a learner-centric outlook on the educational experience. Many of our beliefs about education are interwoven into the criteria we have chosen as well as how we view courseware that meet the criteria. The criteria builds upon some of the latest thinking about higher education and value constructivist learning. We value courseware that includes learning goals and instructor's guides that prescribe pedagogical applications of the courseware.
There are a seemingly infinite number of ways to set forth criteria for evaluating courseware. In fact, in the process of developing the set presented below, we did keep rearranging the individual criteria. However, in the end we assigned the criteria to three categories Engineering Content, Software Design, and Instructional Design because we felt that this division mirrored the evolution of courseware design and development by our target audience, engineering professors. (See http://www.needs.org/premier/1999/criteria.html for additional information about the criteria.)
Table 1 Evaluation Criteria for Engineering Education Courseware
3.2 Premier Award for Excellence in Engineering Education Courseware
Our initial focus has been on developing the highest level of scholarly review described above, the Premier Award for Excellence in Engineering Education Courseware. NEEDS, Synthesis, and John Wiley & Sons, Inc. (see www.wiley.com) developed the Premier Award to "recognize high-quality, non-commercial courseware designed to enhance engineering education." The Premier Award competition results in courseware being named Premier Courseware. The winning instructional courseware packages are chosen through a rigorous application and review process. They represent excellence in the use of computer-based instruction to enhance engineering education. It is important to recognize that the Premier Award recognizes more than just educational software but rather the entire learning experience. The Premier Courseware have distinctive strengths, representing a breadth of styles, sophistication, pedagogies, and use of multimedia that should encourage incorporation of the courseware by others into their learning environments. Through the first two years of this national competition, NEEDS has developed the structure necessary to ensure the Premier Award becomes an established and respected aspect of the engineering community.
NEEDS solicits courseware from engineering educators via our website, conference and workshop presentations, ads in ASEE Prism magazine, and by word of mouth. The submission procedure (see http://www.needs.org/premier/1999/submit.html) is designed to help the judges better evaluate the courseware. The procedure requires submitters to address the review criteria directly and to respond to the following:
NEEDS has conducted two Premier Award competitions with the Inaugural competition held in 1997. In the recently completed 1998 competition, the following three outstanding, high-quality courseware were named the Premier Courseware of 1998:
As part of the Premier Award, NEEDS and Wiley have committed to widely publicizing and distributing the Premier Courseware. NEEDS distributed 1,300 CD-ROM's of the Premier Courseware of 1998 at FIE 1998 and to the Deans of Engineering. Over the last two years, NEEDS has also distributed approximately 2,300 3CD-ROM sets of the Premier Courseware of 1997 at FIE 1997 and 1998, at ASEE 1998, at the International Conference on Engineering Education 1998, and to the Deans of Engineering. The Premier Award and Premier Courseware have been highlighted in NEEDS and Wiley exhibit booths at FIE and ASEE.
4.0 Current Thrusts Expanding our Engineering Core
NEEDS has been awarded a three year grant from the NSF Engineering Action Agenda program to provide services to the engineering education community . Our goals are:
The success of NEEDS as a service depends not only on the development of a critical mass of content, but also on the development of a critical mass of viewership. The development of an on-line community consisting of developers, adapters, adopters, interested parties and learners of the content made available through NEEDS provides the means of sustaining NEEDS as a resource. The content continually draws the user back, and stimulates discussion among community members regarding adoption and adaptation of existing courseware, leading to new courseware development and courseware acquisitions.
The community can address many of the concerns and challenges in cataloging courseware (e.g., inadequate courseware description or lack of support material) by becoming a user-based support and evaluation system. Many of the new features we plan to add to NEEDS are focused at strengthening our present methods of review for all courseware within the system in order to help our user community evaluate whether to adopt or adapt a particular piece of courseware.
For the last four years we have relied on standard usage data generated by Web servers (e.g., browser-type and where the browser is located) to guide system development and features offered. However, Web usage logs do not allow us to easily couple information about what queries (searches) are performed with the results of that query (i.e., which courseware was presented), which courseware was downloaded, whether it was successfully downloaded, who downloaded it and why it was downloaded. To do this, we are instituting a transaction-based system to track queries and their results. This Extended Usage Tracking system will allow us to provide authors with attempted download statistics and better understand our users' search patterns.
We also instituting a User Registration system for that allows us to map 'who' uses courseware to 'where' and 'how' they are using it. This in turn facilitates longitudinal tracking and dissemination efforts for authors, and provides them with greater user feedback. We can track 'successful' downloads with a follow-up survey to ascertain the adoption or adaptation of courseware; find similar or related courseware; and receive feedback for continuous system refinement. In addition, we will allow registered users to develop User Profiles through which we will proactively identify newly cataloged courseware of interest to the user. NEEDS thus becomes demand-driven by carefully targeting acquisition efforts toward identified content needs.
We are expanding our concept of bibliographic record to allow User Reviews and supplementary Support Materials. User Reviews will provide a wealth of community-based support for courseware. They may include information on a user's experience with the courseware and how a user employed the courseware potentially expanding upon the author's original intent. A good example of User Reviews can be found at Amazon.com where book buyers can read reviews, written by other consumers, to better judge a book before making a purchase decision. Support Materials extend the catalog record by attaching user-provided links to related information.
We are well into the third year of the Premier Award for Excellence in Engineering Education Courseware. As we do so, we expect to raise the expectation of what represents quality in courseware. We have issued a 'Call for Submissions' and expect to see a wide range of high-quality courseware. Submissions for the 1999 Premier Award are due July 31, 1999 (see http://www.needs.org/premier/1999/submit.html).
In addition to a traditional Peer Review system, we are developing Feature Reviews similar to book reviews found in journals and newspapers. Finally, we are adding "off the shelf" threaded Discussion tools to further transform the static courseware record into a dynamic, living entity. These services extend the courseware record by providing a forum for users to discuss the courseware and its application.
5.0 Lessons Learned and Future Directions
The NEEDS project has had close to a decade of experience running a digital library of courseware for engineering education. We have greatly benefited from continuous feedback from our users, through focus groups and questionnaires, and from our advisory groups. Our ties to the latest digital library research have allowed us to periodically update our system to meet the growing needs of our user community and quickly deploy advanced information technologies. The decision to use hardened relational database technology as a back-end has allowed us to efficiently manage large datasets and quickly update the user interface. As an example, we were able to move from a X-windows interface to an HTML interface with a single day of coding in 1994.
We believe the decision to build on evolving library standards in archiving and indexing is a great strength of our system. Our original use of MARC records for cataloging has allowed us to easily move to IMS standards. The library standard of maximizing access led us to create interfaces that are accessible via the WWW, across all platforms and that require minimal levels of technical expertise. In addition to providing a minimum level of presentation standards, such as title and author information, we perform virus checking and verify platform requirements before archiving courseware on NEEDS. In order to allow greater author control and to scale the number of records we could catalog, we moved away from employing a full time library cataloger to a system of author-assisted cataloging. The author provides the initial information for his or her courseware through a Web-based cataloging form. Nationally-distributed Internet catalogers will then review the records, perform a minimal level of acceptance testing and communicate with the author if changes or more information is needed.
We are expanding our collection development to the interface between engineering and science and math. In doing so, we hope to break down the artificial barriers imposed by a strict disciplinary view of education. This expansion is a natural evolution of our work because physics, chemistry and math are the prerequisite lower division classes that can ultimately culminate in an engineering degree. The NEEDS infrastructure is robust enough to allow us to catalog, archive, and search for courseware across the boundaries between engineering, physics, chemistry, and math.
The frenetic pace of change in information technologies has exacerbated the age old problem of finding information. With everyone a "publisher" in this new information age, the volume of materials is drastically increasing. Even if we "just" limit ourselves to engineering education, the volume is enormous. NEEDS is developing a new vision of a digital library to serve the engineering education "community." By forming a community of learners, we can leverage the personal expertise of its participants to help the community-at-large organize, describe and locate the appropriate resources. The addition of user reviews, support materials and discussion tools add a strong, community-based support system to further promote the adoption and adaptation of courseware.
About the Authors
Alice M. Agogino is the NEEDS Principal Investigator and a Professor of Mechanical Engineering. She is also Associate Dean in the College of Engineering at UC Berkeley in charge of the College's Center for Underrepresented Engineering Students (CUES) and Instructional Technology and Distance Learning Programs. She served as Director of Synthesis from 1994-1998. She conducts her research in the Berkeley Expert Systems Technology (BEST) lab and the C3AD (Concurrent, Collaborative, Computer-Aided Design) instructional lab.
Brandon Muramatsu is the NEEDS Project Director. He is a lecturer in multimedia at UC Berkeley, directs the Berkeley Instructional Technology Studio and is Chair of the San Francisco Section of the American Society of Mechanical Engineers.
NEEDS is a team effort and we wish to acknowledge the ongoing contributions of: William H. Wood III, Assistant Professor of Mechanical Engineering at the University of Maryland, Baltimore County; Gregory Paschall, NEEDS Lead Programmer, University of California at Berkeley; Flora McMartin, NEEDS Evaluation Director, University of California at Berkeley; George Toye, President of WITHInc and Consulting Associate Director of the Stanford Learning Laboratory; Pamela A. Eibeck, founding NEEDS Editor and Chair and Professor of Mechanical Engineering at Northern Arizona University; and Joseph G. Tront, NEEDS Editor and Assistant Dean of Engineering Computing at Virginia Tech. We also wish to acknowledge those that contributed to the NEEDS foundation: Steven Bradley, University of California at Berkeley; Arvide Eide, Iowa State University; Larry Genalo, Iowa State University; Jim Harris, Cal Poly San Luis Obispo; Jeff Huston, Iowa State University; Jaya Krishnagapolan, Tuskegee University; Dave Martin, Iowa State University; Adebisi Oladipupo, Hampton University; John Saylor, Cornell University; Robert Thomas, Cornell University; the members of the NEEDS Quality Review of Courseware Committee; and Standards Study Group.
NEEDS has been supported by the National Science Foundation (EEC-9872570, IIS-9817406, EEC-9625456, EEC-9053807) and by our industrial partners including Apple, IBM, John Wiley & Sons, and Sun Microsystems.
 Agogino, A. M. and W. H. Wood III, "The Synthesis Coalition: Information Technologies Enabling a Paradigm Shift in Engineering Education," Keynote talk, in Hyper-Media in Vaasa '94: Proceedings of the Conference on Computers and Hypermedia in Engineering Education (Ed., M. Linna and P. Ruotsala, Vaasa Institute of Technology) pp. 3-10.
 Agogino, A. M. and Muramatsu, B., "The National Engineering Education Delivery System (NEEDS): A Multimedia Digital Library of Courseware," International Journal of Engineering Education, Vol. 13, No. 5, pp.333-340, 1997.
 Eibeck, P., "Criteria for Peer-Review of Engineering Courseware on the NEEDS Database," IEEE Transactions on Education, Special Issue on the Special Issue on the Application of Information Technologies to Engineering and Science Education, Volume 39, Number 3, pp. 381-387, August 1996.
 Synthesis Coalition, "Standard Study Report 1.0," Ithaca, NY, March 1993.
 Wood III, W. H. and Agogino, A. M., "Engineering Courseware Content and Delivery: the NEEDS Infrastructure for Distance-Independent Education," Journal of the American Society for Information Science, Vol. 47, No. 11, pp. 863-869, 1996.
 Instructional Management System, "The IMS Metadata Specification," March 21, 1998, URL: http://www.imsproject.org/md_overview.html.
 Muramatsu, B., Eibeck, P. A., Stern, J. L., and Agogino, A. M., "Effective Processes to Give Engineering Educators Easy Access to Quality-Reviewed Electronic Courseware," Invited Presentation, NSF Engineering Education Innovators' Conference, Washington, D.C., April 8, 1997.
 Synthesis Coalition, "Quality Workshop," Synthesis Coalition: San Luis Obispo, CA: May 1995.
 Agogino, A. M., "Expanding the National Engineering Education Delivery System as the Foundation for an On-Line Engineering Education Community," NSF Grant No. EEC-9872570, 1998.