This study analyzed usage metrics and citation counts for articles published in 47 Croatian open access journals available in the HRCAK repository. It found that while full-text downloads and article visits were strongly correlated with each other, they showed only low correlations with Scopus and Web of Science citation counts. There were also significant differences observed between usage, downloads, and citations across six subject areas. The study concludes that multiple metrics provide a more comprehensive view of scientific work and impact than any single metric alone.

1. Multiple Bibliometric
Indicators Approach to
Croatian Open Access
(OA) Journals
Franjo Pehar, Jadranka Stojanovski
Department of Information Sciences, University of Zadar
Libraries in the Digital Age 2014

2. “Better to understand a little than to
misunderstand a lot.”
Anonymous writer

3. Rise of metrics
● importance and usage of citation metrics has
increased substantially in recent years
● the most popular metric indicator is still JIF
● ...initially intended as a way to help libraries
to select the journals to be acquired…
● ...and to help academics to ‘filter content’
● used to measure academic achievement
o at individual, project (grant) and institutional level

4. ...
● academic career
● research project assessment
● institution ranking
● funding decisions
● …
● new situation has drawn significant criticism
o practices that encourage conformity in research
o “salami-slicing” or incremental publishing
o unequal treatment of researchers from low-cited
disciplines

5. Eugene Garfield at LIDA 2004
“JIF is not
created for all the
purposes that
some people
want to use them
for. A lot of
people who are
using JIF are
doing it for
advertising
purposes.”
http://www.garfield.library.upenn.edu/
papers/medicalwritingv8(1)1999.html
“It is crucial to
know how
citations are
connected, how
they are related.”
“The real problem is when you
start using JIF as the
substitute for citation
analysis of individuals. I have
always said that you should
not use JIF to evaluate a
person or department. If I
manage to get my paper
published in Nature, does that
guarantee that it is a great
paper? Even if the paper is
never cited?”

6. http://researchimpact.files.wordpress.com/2011/10/research-impact-cartoon.jpg

7. What is a citation?
● simple line in the reference list (not) pointing
to the paper
● paper which act as a foundation for our own
work
● paper which help us to broaden our
knowledge
● tool to confirm that we prepared well prior to
our own research
● tool to flatter someone
● tool to “self-cite” at different levels ...

8. Citation as a metric tool
● many factors can influence the rate of citation
o age of article, type of research conducted, the subject
area
o objectivity of citations? real benefit? (erroneous
references with hundreds, thousand citations)
● tools for evaluation and assessment are
evolving constantly, giving rise to new metrics
o different aspects of citation and publication behaviour

9. Journals
● fee and free journals
● „OA journal” as a synonym for APC model
● journals? - or more portals
● very prominent journals = high JIF journals =
very for-profit journals
● „international journals” and „local journals”
● „regional journals”

10. Journals
● journal as an old concept from the printed
world
● journal as (still) main channel of scholarly
communication
● journals slowing down a possible
development of science (?), not
implementing ICT advantages and
possibilities…
● ...or implementing it at very slow pace

11. What’s about journals from the
scientific semi-periphery?

12. Narrower context
● Croatia
o 4.5M population, 11k scholars, 7 universities, 25
research institutes, 25k papers per year and 1300
islands
● Croatian journals
o challenges for small language and semi-peripheral
science context

13. Croatian journals in numbers

14. Academic career policies in Croatia
“papers must be published in journals with JIF above average for the specific
subject category”
“at least two-thirds of the papers must be published in the journals listed in SCIE”
“paper is rated regarding ratio of the JIF and median of the subject category, and
the minimal value must be 0.5
“the value of the qualitative criteria Q is calculated as a sum of the ratios of JIF
and median JIF of the subject category, multiplied by the factor Fd of the
candidate”
“Example: Journal of Engineering Mechanics in 2011 has JIF 0.99, subject
category is Engineering – Mechanical with median JIF 0.743 …”
“in the A1 category are the papers published in journals with JIF in Q1, Q2, i Q3
of the subject category ...in the A2 category are papers from the journals in
Q4…”
“in the A1 category are the papers published in journals indexed by Web of
Knowledge...if the JIF falls in the first 50% of the subject category...then the
paper is counted as two papers”

15. Hmmmm….
are Croatian journals important?
is it reasonable just to transfer the Croatian
journal funds to APC or fee based prominent
and popular “international” journals?
what justification do we have to publish
Croatian scholarly journals?

16. HRCAK - improved visibility,
readability...popularity...citations

17. HRCAK by disciplines

18. Objective of the study
● Which aspects of journal usage metrics differ
from citations metrics
● In particular, following issues were be
addressed:
o Croatian OA journals usage
o Relationship of full-text paper downloads and
citation counts
o Comparison between full paper downloads and
citations counts relative to different subject areas

19. Materials and methods
● HRCAK usage statistics
o Cover journal's web page visits
o Issue web page visits
o Article (bibliographic records) web page visits
o Full-text (PDF) downloads
o Unique visits from the same IP address during the
period of 30 minutes

20. ...
● For the purpose of the study
o Usage data for 115,458 full-text papers
(documents) acquired from 340 unique OA
journal titles
 March 2007 to the May 1st 2014
 Article page visits
 Full-text downloads

21. 674 downloads
1293 visits

22. Research sample
● 47 OA journals (28,121 papers) available in
HRCAK repository and indexed by
o Web of Science™ Core Collection (Thomson
Reuters)
 28,725 indexed papers
o Scopus (Elsevier)
 28,953 indexed papers

23. Preprocessing challenges
● Assembling a single data source based on three
data sets
o lack of (P)ID’s for matching documents from
different data sets
● Titles were reduced to the same form
● Levenshtein distance algorithm to determine
similarity between paper titles (2 iterations)
o 15,023 (53%) papers in Web of Science and 16,592
(57%) papers in Scopus matched with HRCAK titles
o 13,127 (ca. 46%) unique papers merged from all
three data sets
● 2 journals dropped-out

24. ...
● Initial cleaning of usage data
o validity, reliability, transparency ...
● Missing data
o HRCAK page visit & download dates/years, more
reader demo-geographic information
● Matching/merging procedure
o two journal titles completely lost
o further improvements possible > expected matching
rate ca. 85% (cat(ti ... jn … is … vl … pg)

25. Results
● download and citation frequencies (March
2007 - May 2014)
Matched
articles
HRCAK
visits
HRCAK
Downloads
Scopus
citations
WOS
citations
13,127 2,084,174 4,174,888 38,106 30,324

26. Visits vs Downloads
Article page visits: 1-250=84%;
Downloads: 1-250=59%, 251-500=22%

27. cor(log(1+HRVisits, HRDown),
method=”spearman”)

28. Possible reasons …

29. ...

30. ...

31. Scopus vs Web of Science citations
Scopus: 0=41%, 1-5=44%;
Web of Science: 0=47%, 1-5=42%

32. cor(log(1+HRDown, ScopusCit,
WoSCit), method=”spearman”)

33. Visits, downloads and citations
relative to different subject fields
Subject area No. journals % No. articles %
Biomedicine and
health
10 22.2% 5856 44.6%
Biotechnical
sciences
5 11.1% 1420 10.8%
Humanities 7 15.6% 527 4.0%
Sciences 8 17.8% 2037 15.5%
Social sciences 6 13.3% 1365 10.4%
Technical sciences 9 20.0% 1922 14.6%
Total 45* 100.0% 13127 100.0%

34. Median number of visits and
downloads & Mean number of citations
Median number of HRCAK visits and
downloads relative to six subject areas
Mean number of Scopus and Web of
Science citations relative to six
subject areas

35. Key results
● Different usage (reader) and impact (author)
characteristics
● Very strong, strong and low correlations
o Scopus and WoS citation counts
o article page visits and downloads
o downloads and citation counts
● But, with significant differences between and
within six subject areas

36. Conclusion
● Important to have different (MULTIPLE) but
complementary measures for better
reflecting the multifaceted and
multidimensional character of scientific work
(output, impact, activities …)

37. Acknowledgment
• HRCAK team (University Computing Centre)
• Tomislav Jagušt (Faculty of Electrical
Engineering and Computing, University of
Zagreb)

38. Bieber’s response to yesterday's
accusation