Some trends in highly cited articles published in the journal IRRODL from 2008 to mid 2013
Introduction
Journal International Review of Research in Open and Distance Learning (IRRODL) was released for the first time in 2000. Since then, the journal has been published quarterly and is a focal point in ODL discipline, on implementation projects; it also presents viewpoints and perspectives from experts in different fields and also encourages peer-reviewed articles providing a more detailed treatment of the various aspects of ODL, its objectives and its contexts.
Citation analysis
Citation analysis has been used in the social sciences for investigating the research contributions of individuals, institutions and academic journals (Kinshuk, Huang, Sampson, & Chen, 2013). It allows researchers to examine how frequently a document has been cited by other authors, providing one measure of the influence and importance of a writer or of a specific article. Kinshuk et al. (2013) analyzed the research type, research topic, first author’s country, international collaboration, participant levels, learning domain, research method, and frequently appearing keywords among top 20 highly cited articles in the Educational Technology and Society journal (ET&S) during 2003-2010.
According to Chiu and Ho (2007) cited by (Kinshuk et al., 2013), the impact or visibility of an article can be identified by the number of citations. Previous studies have reviewed the highly cited articles in different fields (Lee, Driscoll, & Nelson, 2004; Allen, Jacobs, & Levy, 2006; Blessinger & Hrycaj, 2010; Rezaei, Navidi, Rokni, & Pourmand, 2012; Kinshuk et al., 2013). However, little research has been conducted regarding the analysis of the characteristics of the highly cited articles in the field of Open and Distance Learning. The present study provides insight to the research trends and basic citation trends of the highly cited articles published in the Journal IRRODL.
Moreover, studies on using author keywords to analyze the research trends have shown that this method can effectively predict the research (Ozcinar, 2009; Kinshuk et al., 2013). The purpose of author keywords analysis is to reveal the frequencies of the different words that appear in a piece of text. Word clouds can present what the most common words are with their size reflecting their frequency. There are Wordle similar tools such as TagCrowd, MakeCloud and the used in this work: Tagul.
Google Scholar
The release of the beta version of Google Scholar (GS) (http://scholar.google.com) in November 2004 generated much media coverage and academic commentary (Giustini, 2005). GS searches retrieve results that include scholarly literature citations as well as peer-reviewed publications, theses, books, abstracts, and other articles from academic publishers, professional organizations, and preprint repositories, universities, and other scholarly organizations (Gehanno, Rollin, & Darmoni, 2013). The coverage of GS is increasing and, despite the fact that it is said to be not exhaustive, is it exhaustive enough for the studies that are considered of enough quality or relevance for systematic reviews (Beckmann & Wehrden, 2012).
Google Scholar does not offer the authority structure or transparency of coverage that librarians and bibliometricians expect from a scientific information resource. However, it might well be of considerable use for individual academics interested in citation analysis, as well as higher level bibliometric analyses such as citation analysis. (Torres-Salinas, Ruiz-Pérez, & Delgado-López-Cózar, 2009)
Some of the features of Google Scholar have suffered constant criticism; however, it has been shown that the journals listed in Google Scholar are more likely to have a higher number of citations in comparison to the ones received in Web of Science. The most common criticism comes from the lack of standardization and/or duplication of retrieved articles. That’s why GS is preferably used to conduct bibliometric studies, it encourages openness and use of this tool for free access journals.
Sample and Methods
The data were based on the highly cited articles published in the IRRODL from 2008 to mid 2013. The IRRODL was chosen because of the reputations as one of the most prominent and recognized journals in the field of open and distance learning (Zawacki-Richter, Bäcker, & Vogt, 2009). The IRRODL, a quarterly journal, began publishing referred journal articles in 2000. All the journal’s articles are freely accessing online at http://www.irrodl.org. The journal have published 447 articles from 2008 to 2013, this statistics can be consulted at the journal site and it is provided in detail in Appendix 2.
The search period was set during 2008 to mid 2013. Identifying the characteristics of the highly cited articles published in the IRRODL during the past five years will provide a macroscopic examination for readers to have a holistic and precise understanding of the research influence in the field of open and distance learning.
All articles published between 2008 and mid 2013 (Vol 9 to Vol 14 N2) in the journal indexed by GS were retrieve (N = 402). The criterion for selecting the highly cited articles (N=33) was those articles published in the IRRODL and cited at least 30 times, was taken into account that the h-index in the query result IRRODL in GS was 30.
For the analysis of the citations in Google Scholar was used the software Publish or Perish (www.harzing.com/pop.htm) version 4.4.6. by Journal Impact Analysis option, taking into account the time range between 2008 to mid 2013, were searched using the exact title of the journal “Journal International Review of Research in Open and Distance Learning”.
Publish or Perish is a software developed by Tarma Software Research (www.tarma.com) that retrieves and analyzes academic citations provided by Google Scholar and Microsoft Academic Search, presenting their results in a friendly format that can be exported to any application Windows or other formats for further analysis (Harzing & van der Wal, 2008).
In addition, the authors used keyword clouds to be a supplementary research tool to support traditional content analysis methods. Word clouds reveal the frequencies of the different words within the body of text. The more frequent the word, the more important is the concept. Since word clouds deal with each word as the unit of analysis, the authors used this supporting tool to validate the finding of research topics obtained from manual coding (McNaught & Lam, 2010).
Finally, a Ranking of Total Galley Views (all formats: PDF, HTML, EPUD and MP3) was carried out with internal data provided by IRRODL and it is compared with the results. This rank can be consulted in Appendix 1. It is important to explain that these total views are a historical data, which means they take into account views since 2006, when Open Journal System was implemented to support the journal. This additional gauge is introduced as a way to contrast the results, so the articles views suggest relevance for readers.
Results
The results of the citation analysis of the top highly cited articles from the years 2008 to mid 2013 is detailed in a list of the highly cited articles in rank order by total number of citations that can be found in the Appendix 1. The query yielded 402 articles of such 33 with more than 30 cites for the periods 2008 to mid 2013.
The total number of different authors who contributed to the 33 articles in this study was 64 and come from 11 different countries. Figure 2 shows the frequencies of all authors’ countries over time interval. Based on author’s country, the following countries were identified in the highly cited articles: United States (24), United Kingdom (12) and Canada (11) with more than 10 authors, followed by Germany, Israel, Turkey, Norway, Italy, Denmark, Bahrain and Australia. There was 5 international co-authored articles conducted by researchers from: United Kingdom and Canada (2008); Canada, Germany and United States (2009); Denmark and Norway (2009); Turkey and Canada (2009); and Bahrain and United Kingdom (2009). The leaders contributors with 2 articles in the selected 33 papers was David Wiley from United States, Olaf Zawacki-Richter from Germany and Rita Kop from United Kingdom.
Figure 2. Frequencies of author’s country in highly cited
The table 1 shows the 4 articles with more than 100 cites, highlighting Connectivism: Learning theory of the future or vestige of the past? by Rita Kop and Adrian Hill with 134 cites. This articles received 49048 (rank 11), 20812 (rank 53), 39574 (rank 18), 43515 (rank 14) views respectively.
Table 1. Articles with more than 100 cites.
Cites
|
Title
|
Authors
|
Country
|
134
|
Connectivism: Learning theory of the future or vestige of the past?
|
Rita Kop
Adrian Hill
|
United Kingdom
Canada
|
127
|
Open content and open educational resources: Enabling universal education
|
Tom Caswell
Shelley Henson
Marion Jensen
David Wiley
|
United States
United States
United States
United States
|
121
|
Three generations of distance education pedagogy
|
Terry Anderson
Jon Dron
|
Canada
Canada
|
114
|
Technology-enhanced learning in developing nations: A review
|
Shalni Gulati
|
United Kingdom
|
Figure 3 shows that the years with the highest number of articles within the 33 selected were 2009 with 14 and 2008 with 10, which represents 72% of the total, followed by 2011 with 5 and 2010 with 4, representing 27 %. It is noted that no provision of the 2012 and 2013 appears within the most cited, this can be given by its life shorter and therefore less query time.
Figure 3. Frequencies of years in highly cited articles.
Figure 4 shows different research types in the highly cited articles over time interval, this classification is provide in metadata information by IRRODL site. Among highly cited article was found case study (12) representing the 33%, 8 literature review, 6 theoretical studies; survey, mixed methods and historical analysis with two; and Technical Evaluation Report with one. Interestingly, the number of literature review plus theoretical studies (14) with the 42% of total affirm this types of studies receive a high number of cites.
Figure 4. Frequencies of research types in highly cited articles.
According to Mao, Wang, and Ho (2010), author keywords analysis provides researchers with the information of research trend. The author used word clouds to present the frequently used author keywords obtained from the highly cited studies during 2008 to mid 2013.
The Tagul (http://tagul.com/) program automatically generated the word clouds after the authors introduce all the keywords provide by paper’s authors (Figure 5). The results obviously indicated that “learning”, “education” and “online” were the top keywords that appeared in the studies over the time interval. It is important to note that the Tagul program shows single words, and in this analysis, those words were combined as per the keywords provided by the authors in order to obtain meaningful analysis. The others frequently appeared keywords were: distance, open, model, access, social, theory, design, mobile, blended, connectivism and resources.
Figure 5. Word clouds of the keywords listed by authors
Conclusion
Performing systematic reviews is a complex and time consuming task, because of the body of literature to be searched and the high number of databases that must be used, considering that no one of them is considered exhaustive. The use of GS is increasing, as well as its coverage, and we wanted to assess if this coverage is high enough to be used alone in systematic reviews.
Using the h index, the 33 most frequently cited articles published between 2008- mid 2013 (see Appendix) and their authors were identified; the citation counts of these articles ranged between 134 and 30. According to Shih, Feng and Tsai (2008), “articles with more citation frequencies are usually those that are better recognised by others in related fields. They probably present more fundamental ideas about the issues for future research” (p.960).
Researchers are growingly involved in international collaborative projects. The collaboration in this study is express in 5 papers, representing the 15%. According to Zawacki-Richter (2009), this happening has accelerated during the last decades through the massive production of communication technologies and the expansion of innovative web tools, like social software or Web 2.0 applications, which facilitate collaborative work, social interaction, and negotiation of meaning. The results of the present research show that 73% of authors come from the UK, USA and Canada, this confirm the findings of Zawacki-Richter (2009) “more than 80% of all articles were contributed by authors from only five countries: USA, Canada, UK, Australia, and China” (p.44).
Our experience in using Tagul to analyses keywords have led us to suggest that word clouds can be a useful research tool to aid educational research. They can allow researchers to quickly visualize some general patterns in text. Is important to mention that some issue like costs, productivity, and management are not presents in the keywords, so that could means the authors avoid that subjects or the relative paper published had have less relevance to the community.
The comparison of highly cited articles with Total Galley Views confirm the relevance of this papers to the community because they are included in the most historically viewed papers of the journal. This comparison acquire additional importance so the highly cited articles are matched with all articles publish from 2006 in the journal.
Acknowledgment
Advice and comments from Dr Tony Bates, President and CEO of Tony Bates Associates Ltd, is acknowledged with appreciation.
We want to thanks to editors of the IRRODL journal for provide the internal data of articles statistics and especially to Prof. Terry Anderson for his comments and motivation.
Appendix 1.
Top 33 highly cited IRRODL papers (by citation counts in total, as of October 10, 2013) during 2008-2013 query)
Cites
|
Authors
|
Title
|
VR*
|
134
|
R Kop, A Hill
|
Connectivism: Learning theory of the future or vestige of the past?
|
11
|
127
|
T Caswell, S Henson, M Jensen, D Wiley
|
Open content and open educational resources: Enabling universal education
|
53
|
121
|
T Anderson, J Dron
|
Three generations of distance education pedagogy
|
18
|
114
|
S Gulati
|
Technology-enhanced learning in developing nations: A review
|
14
|
84
|
CS Cavanaugh, MK Barbour, T Clark
|
Research and practice in K-12 online learning: A review of open access literature
|
30
|
74
|
Y Park
|
A pedagogical framework for mobile learning: Categorizing educational applications of mobile technologies into four types
|
12
|
73
|
R Kop
|
The challenges to connectivist learning on open online networks: Learning experiences during a massive open online course
|
143
|
69
|
J Brindley, LM Blaschke, C Walti
|
Creating effective collaborative learning groups in an online environment
|
4
|
64
|
A Fini
|
The technological dimension of a massive open online course: The case of the CCK08 course tools
|
38
|
63
|
JL McBrien, R Cheng, P Jones
|
Virtual spaces: Employing a synchronous online classroom to facilitate student engagement in online learning
|
136
|
54
|
R Williams, R Karousou, J Mackness
|
Emergent learning and learning ecologies in Web 2.0
|
94
|
51
|
S Motlik
|
Mobile learning in developing nations
|
65
|
51
|
C Dalsgaard, MF Paulsen
|
Transparency in cooperative online education
|
81
|
48
|
T Wilson
|
New Ways of Mediating Learning: Investigating the implications of adopting open educational resources for tertiary education at an institution in the United Kingdom …
|
201
|
47
|
JH Valk, AT Rashid, L Elder
|
Using mobile phones to improve educational outcomes: An analysis of evidence from Asia
|
3
|
46
|
O Zawacki-Richter, E Baecker, S Vogt
|
Review of distance education research (2000 to 2008): Analysis of research areas, methods, and authorship patterns
|
84
|
42
|
F Bell
|
Connectivism: Its place in theory-informed research and innovation in technology-enabled learning
|
62
|
40
|
D Wiley, J Hilton III
|
Openness, dynamic specialization, and the disaggregated future of higher education
|
171
|
40
|
Z Akyol, DR Garrison, MY Ozden
|
Online and blended communities of inquiry: Exploring the developmental and perceptional differences
|
244
|
39
|
T Müller
|
Persistence of women in online degree-completion programs
|
138
|
37
|
P Parrish, J Linder-VanBerschot
|
Cultural dimensions of learning: Addressing the challenges of multicultural instruction
|
6
|
35
|
M Abbad, D Morris, C De Nahlik
|
Looking under the bonnet: Factors affecting student adoption of e-learning systems in Jordan
|
152
|
35
|
N Friesen
|
Open educational resources: New possibilities for change and sustainability
|
245
|
33
|
R Berenson, G Boyles, A Weaver
|
Emotional intelligence as a predictor of success in online learning
|
52
|
33
|
S Gokool-Ramdoo
|
Beyond the theoretical impasse: Extending the applications of transactional distance education theory
|
126
|
33
|
A Ravenscroft
|
Dialogue and connectivism: A new approach to understanding and promoting dialogue-rich networked learning
|
195
|
32
|
P Gorsky, I Blau
|
Online teaching effectiveness: A tale of two instructors
|
87
|
32
|
K Precel, Y Eshet-Alkalai, Y Alberton
|
Pedagogical and design aspects of a blended learning course
|
89
|
30
|
B Robinson
|
Using distance education and ICT to improve access, equity and the quality in rural teachers' professional development in western China
|
50
|
30
|
O Zawacki-Richter
|
Research areas in distance education: A Delphi study
|
55
|
30
|
L Barnard, V Paton, W Lan
|
Online self-regulatory learning behaviors as a mediator in the relationship between online course perceptions with achievement
|
90
|
30
|
S Uzuner
|
Questions of culture in distance learning: A research review
|
99
|
30
|
T Elias
|
Universal instructional design principles for mobile learning
|
162
|
*VR: Ranking of Total Galley Views provided by IRRODL
Appendix 2.
IRRODL Journal statistics from 2008 to 2013
2008
|
2009
|
2010
|
2011
|
2012
|
2013
|
|
Issues published
|
3
|
6
|
3
|
7
|
5
|
5
|
Items published
|
57
|
77
|
70
|
111
|
77
|
55
|
Total submissions
|
98
|
155
|
138
|
182
|
207
|
229
|
Peer reviewed
|
65
|
111
|
85
|
103
|
151
|
125
|
Accept
|
29 (45%)
|
44 (40%)
|
41 (48%)
|
52 (50%)
|
56 (37%)
|
54 (43%)
|
Decline
|
36 (55%)
|
67 (60%)
|
44 (52%)
|
51 (50%)
|
95 (63%)
|
71 (57%)
|
Days to review
|
89
|
69
|
74
|
56
|
61
|
48
|
Days to publication
|
168
|
152
|
182
|
159
|
177
|
120
|
References
Allen, M., Jacobs, S., & Levy, J. (2006). Mapping the literature of nursing: 1996-2000. Journal of the Medical Library Association, 94(2), 206-220.
Aylward, B. S., Roberts, M. C., Colombo, J., & Steele, R. . (2008). Identifying the classics: An examination of articles published in the Journal of Pediatric Psychology from 1976-2006. Journal of Pediatric Psychology, 33(6), 576-589.
Beckmann, M., & Wehrden, H. (2012). Where you search is what you get: literature mining – Google Scholar versus Web of Science using a data set from a literature search in vegetation science. J Veg Sci, 23(6), 1197–1199.
Blessinger, K., & Hrycaj, P. (2010). Highly cited articles in library and information science: An analysis of content and authorship trends. Library and Information Science Research, 32, 156-162.
Bornmann, L., & Daniel, H. (2008). What do citation counts measure? A review of studies on citing behavior. J Doc, 64(1), 45-80.
Garfield, E. (1955). Citation indexes for science: A new dimension in documentation through association of ideas. Science, 22, 108-111.
Gehanno, J., Rollin, L., & Darmoni, S. (2013). Is the coverage of google scholar enough to be used alone for systematic reviews. Medical Informatics and Decision Making, 13(7). Recuperado a partir de http://www.biomedcentral.com/1472-6947/13/7
Giustini, D. (2005). How Google is changing medicine. BMJ, 33(1), 1487–1488. Recuperado a partir de http://www.biomedcentral.com/1472-6947/13/7
Harzing, A., & van der Wal, R. (2008). Google Scholar as a new source for citation analysis. Ethics in Science and Environmental Politics, 8(1), 62-73.
Kinshuk, Huang, H.-W., Sampson, D., & Chen, N.-S. (2013). Trends in educational technology through the lens of the highly cited articles published in the journal of educational technology and society. Educational Technology and Society, 16(2), 3-20. Recuperado a partir de http://www.ifets.info/journals/16_2/2.pdf
Klein, J. (1997). ETR&D - Development: An analysis of content and survey of future direction. Educational Technology Research and Development, 45(3), 57-62.
Lee, Y., Driscoll, M., & Nelson, D. (2004). The past, present, and future of research in distance education: Results of a content analysis. The American Journal of Distance Education, 18(4), 225-241.
MacRoberts, M., & MacRoberts, B. (1989). Problems of citation analysis: A critical review. J AM SOC INF SCI TEC, 40(5).
Mao, N., Wang, M. H., & Ho, Y. S. (2010). A bibliometric study of the trend in articles related to risk assessment published in Science Citation Index. Human and Ecological Risk Assessment, 16, 801-824.
McNaught, C., & Lam, P. (2010). Using Wordle as a Supplementary Research Tool. The Qualitative Report, 15(3), 630-643.
Ozcinar, Z. (2009). The topic of instructional design in research journals: A citation analysis for the years 1980-2008. Australasian Journal of Educational Technology, 25(4), 559-580.
Rezaei, E., Navidi, I., Rokni, M., & Pourmand, M. (2012). Assessing the Effect of Highly Cited Papers on the Impact Factor of Journals in the Field of Public Health. Iranian Journal of Public Health, 41(12), 84-85.
Shih, M., Feng, J., & Tsai, C. C. (2008). Research and trends in the field of e-learning from 2001to 2005: A content analysis of cognitive studies in selected journals. Computers & Education, 51, 955-967.
Torres-Salinas, D., Ruiz-Pérez, R., & Delgado-López-Cózar, E. (2009). Google Scholar como herramienta para la evaluación científica. El profesional de la información, 18(5).
Tsai, C., & Wen, L. (2005). Research and trends in science education from 1998 to 2002: A content analysis of publication in selected journals. International Journal of Science Education, 27, 3-14.
Zawacki-Richter, O., Bäcker, E., & Vogt, S. (2009). Review of Distance Education Research (2000 to 2008): Analysis of Research Areas, Methods, and Authorship Patterns. International Review of Research in Open and Distance Learning, 10(6).
No hay comentarios:
Publicar un comentario