An Empirical Study on the Maturity of the Eclipse Modeling Ecosystem

Flickr/BenNuttall
An Empirical Study on the Maturity of
the Eclipse Modeling Ecosystem
Javier L. Cánovas Izquierdo, Valerio Cosentino, Jordi Cabot
flickr/Kansasphoto

Maturity Model
ISO 25000 ISO 9126
OPM3 CMMI PRINCE
Technology
Readiness
Level
Polarsys
Maturity
Model
McCabe Ayalew et al. …

Maturity Model
DIMENSION ATTRIBUTE QUESTION Metric ID INF.
Ecosystem
Activity
Developer Activity eco_avg_commits_developer ↗
SCM Activity
eco_num_commits ↗
eco_num_contributors ↗
eco_avg_commits_month ↗
eco_avg_commits_week ↗
eco_avg_commits_last_year ↗
Diversity Developer Activity
eco_ratio_outsiders ↗
eco_ratio_commits_top_committers ↘
eco_ratio_casuals ↗
Product
Analysability
Code Size prod_lines_code ↘
File Types prod_num_extensions ↘
Code Complexity
prod_class_complexity ↘
prod_functions_complexity ↘
prod_file_complexity ↘
Changeability
Code Smells prod_code_smells ↘
File Types (see metrics defined in analysability attribute) -
Code Complexity (see metrics defined in analysability attribute) -
Reliability
Number of Issues prod_open_issues ↘
Code Complexity (see metrics defined in analysability attribute) -
Reusability
Code Documentation prod_comment_density ↗
Technical Debt prod_technical_debt ↘

Research
Questions
flickr/Despeja

Research Questions
Are modeling projects as mature as non-modeling projects?RQ1
Is the incubation status of a project an important factor when comparing
modeling and non-modeling projects?
RQ2
Is the incubation status a valid indicator of the maturity of a modeling
project?
RQ3

Research Questions
RQ2
project?
RQ3
PROJECT TYPE
PROJECTSTATUS

Building
the Dataset
&
Calculating
Metrics
flickr/Clement127

Dataset Construction
Metric Calculation

Building the dataset
INPUT DISCARDED OUTPUT
PROJECTS REPOS PROJECTS REPOS PROJECTS REPOS
WEB SCRAPING 216 1,710
FILTERING 216 1,710 49 1,361 167 349
CLONING 167 349 22 28 145 321
IMPORT 145 321 0 0 145 321
SQL-BASED 145 321 0 0 145 321
SONARQUBE-BASED 145 321 9 47 136 274
FINAL 136 274

FILTERING 216 1,710 49 1,361 167 349
CLONING 167 349 22 28 145 321
IMPORT 145 321 0 0 145 321
SQL-BASED 145 321 0 0 145 321
SONARQUBE-BASED 145 321 9 47 136 274
FINAL 136 274
Inactive repositories
Companion repositories
eclipse.org
gerrit

FILTERING 216 1,710 49 1,361 167 349
CLONING 167 349 22 28 145 321
IMPORT 145 321 0 0 145 321
SQL-BASED 145 321 0 0 145 321
SONARQUBE-BASED 145 321 9 47 136 274
FINAL 136 274
DECEMBER 2016
M T W T F S S
1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31

FILTERING 216 1,710 49 1,361 167 349
CLONING 167 349 22 28 145 321
IMPORT 145 321 0 0 145 321
SQL-BASED 145 321 0 0 145 321
SONARQUBE-BASED 145 321 9 47 136 274
FINAL 136 274 master
Gitana
https://github.com/
SOM-Research/gitana

Calculating metrics
FILTERING 216 1,710 49 1,361 167 349
CLONING 167 349 22 28 145 321
IMPORT 145 321 0 0 145 321
SQL-BASED 145 321 0 0 145 321
SONARQUBE-BASED 145 321 9 47 136 274
FINAL 136 274

Dataset descriptive statistics

Answering our
Research Questions
flickr/AftabUzzaman

Research Questions
modeling and non-modeling projects
RQ2
Project?
RQ3

PROJECT TYPE
PROJECTSTATUS
RQ1
Are modeling projects as mature as non-
modeling projects?RQ1
M ¬Mvs.

PROJECT TYPE
PROJECTSTATUS
RQ1
Are modeling projects as mature as non-
modeling projects?RQ1
M ¬Mvs.
Our mission:
Assess whether the distributions of a
given metric are different…
Student’s t-test
Behrens-Fisher test
Mann-Whitney-Wilcoxon
for modeling
vs.
non-modeling

RQ1
ECOSYSTEM METRICS
METRIC DIFF. INF.
eco_avg_commits_developer  ↗
eco_num_commits  ↗
eco_num_contributors *** ↗
eco_avg_commits_month  ↗
eco_avg_commits_week  ↗
eco_avg_commits_last_year  ↗
eco_ratio_outsiders  ↗
eco_ratio_commits_top_committers  ↘
eco_ratio_casuals ** ↗
PRODUCT METRICS
prod_lines_code - ↘
prod_num_extensions ** ↘
prod_class_complexity  ↘
prod_functions_complexity  ↘
prod_file_complexity  ↘
prod_code_smells  ↘
prod_open_issues  ↘
prod_comment_density *** ↗
prod_technical_debt * ↘

RQ1
Are modeling projects as mature as
non-modeling projects?
RQ1
ECOSYSTEM METRICS
METRIC DIFF. INF.
eco_avg_commits_developer  ↗
eco_num_commits  ↗
eco_num_contributors *** ↗
eco_avg_commits_month  ↗
eco_avg_commits_week  ↗
eco_avg_commits_last_year  ↗
eco_ratio_outsiders  ↗
eco_ratio_commits_top_committers  ↘
eco_ratio_casuals ** ↗
PRODUCT METRICS
prod_lines_code - ↘
prod_num_extensions ** ↘
prod_class_complexity  ↘
prod_functions_complexity  ↘
prod_file_complexity  ↘
prod_code_smells  ↘
prod_open_issues  ↘
prod_comment_density *** ↗
prod_technical_debt * ↘
Some metrics present significant differences in
their distribution
Modeling projects achieve lower maturity
levels than non-modeling ones but that the
overall difference is small.

Research Questions
RQ2
Project?
RQ3

PROJECT TYPE
PROJECTSTATUS
RQ2
Is the incubation status of a project an
important factor when comparing modeling
and non-modeling projects?
RQ2
MI
M¬I
¬MI
¬M¬Ivs.
vs.

PROJECT TYPE
PROJECTSTATUS
RQ2
Is the incubation status of a project an
important factor when comparing modeling
and non-modeling projects?
RQ2
MI
M¬I
¬MI
¬M¬Ivs.
vs.
Our mission:
ANOVA
T
for modeling vs.
non-modeling
&
incubation vs.
non-incubation
~

RQ2
ECOSYSTEM METRICS
METRIC MI vs. ¬MI M¬I vs. ¬M¬I INF.
eco_avg_commits_developer   ↗
eco_num_commits   ↗
eco_num_contributors  ** ↗
eco_avg_commits_month   ↗
eco_avg_commits_week   ↗
eco_avg_commits_last_year   ↗
eco_ratio_outsiders - - ↗
eco_ratio_commits_top_committers   ↘
eco_ratio_casuals  * ↗
PRODUCT METRICS
prod_lines_code ***  ↘
prod_num_extensions ***  ↘
prod_class_complexity - - ↘
prod_functions_complexity - - ↘
prod_file_complexity - - ↘
prod_code_smells ***  ↘
prod_open_issues ***  ↘
prod_comment_density *** *** ↗
prod_technical_debt *  ↘

RQ2
Is the incubation status of a
project an important factor
when comparing modeling and
non-modeling projects?
RQ2
Numerous significant differences in product
metrics between modeling and non-modeling
projects when they are incubation ones
Differences fade down once the projects
leave the incubation status
The incubation status of a project affects the
maturity comparison between modeling and
non-modeling projects
ECOSYSTEM METRICS
METRIC MI vs. ¬MI M¬I vs. ¬M¬I INF.
eco_avg_commits_developer   ↗
eco_num_commits   ↗
eco_num_contributors  ** ↗
eco_avg_commits_month   ↗
eco_avg_commits_week   ↗
eco_avg_commits_last_year   ↗
eco_ratio_commits_top_committers   ↘
eco_ratio_casuals  * ↗
PRODUCT METRICS
prod_lines_code ***  ↘
prod_num_extensions ***  ↘
prod_code_smells ***  ↘
prod_open_issues ***  ↘
prod_comment_density *** *** ↗
prod_technical_debt *  ↘

Research Questions
modeling and non-modeling projects
RQ2
project?
RQ3

PROJECT TYPE
PROJECTSTATUS
Project type and status
Is the incubation status a valid indicator of
the maturity of a modeling project?RQ3
MI
M¬I
vs.
Our mission:
ANOVA
T
for modeling vs.
non-modeling
&
incubation vs.
non-incubation
~

RQ3
ECOSYSTEM METRICS
METRIC M¬I vs. MI ¬M¬I vs. ¬MI INF.
eco_avg_commits_developer  * ↗
eco_num_commits  ** ↗
eco_num_contributors ** *** ↗
eco_avg_commits_month  *** ↗
eco_avg_commits_week  ** ↗
eco_avg_commits_last_year  *** ↗
eco_ratio_commits_top_committers  *** ↘
eco_ratio_casuals  *** ↗
PRODUCT METRICS
prod_lines_code  *** ↘
prod_num_extensions  *** ↘
prod_code_smells  *** ↘
prod_open_issues  *** ↘
prod_comment_density   ↗
prod_technical_debt   ↘

RQ3
ECOSYSTEM METRICS
PRODUCT METRICS
Is the incubation status a valid
indicator of the maturity of a
modeling Project?
RQ3
No remarkable differences in the metrics for
modeling projects according to its incubation
status

RQ3
ECOSYSTEM METRICS
PRODUCT METRICS
Is the incubation status a valid
indicator of the maturity of a
modeling Project?
RQ3
No remarkable differences in the metrics for
modeling projects according to its incubation
status
The incubation status makes a difference
only for non-modeling projects

Extensions
Beyond a Git-centric
Approach
Collaboration as a key
Maturity Dimension
Collaboration Beyond
the Code
Why not consider other
supporting tools?
(e.g., forums, instant messages, etc.)
Pay attention to the community as one
of the cornerstones in Open Source
Let’s combine previous extensions and
analyze collaboration networks

Collaboration as a Key Maturity Dimension

Collaboration Beyond the Code
JDT Papyrus

Conclusion and Further Work
• First studies on the maturity of modeling tools
• Comparison of over a hundred of modeling and non-modeling Eclipse
projects
• There are differences, though less than expected
• Extensions to increase the scope of our maturity model
What we have shown
Much more research needs to be done to really understand the technical and
social factors that make both families of projects different

Maturity vs. Adoption
flickr/crwr
Vallecillo, A (2014). On the Industrial Adoption of Model Driven Engineering, Is you Company ready for MDE? . International
Journal of Information Systems and Software Engineering for Big Companies (IJISEBC), Vol. 1, Num. 1, pp. 52-68

Maturity vs. Adoption
flickr/crwr
…are they related?
Vallecillo, A (2014). On the Industrial Adoption of Model Driven Engineering, Is you Company ready for MDE? . International
Journal of Information Systems and Software Engineering for Big Companies (IJISEBC), Vol. 1, Num. 1, pp. 52-68

Why modeling has not
succeeded yet?
flickr/TimNorris

Thanks!
Replication Package
http://hdl.handle.net/20.500.12004/1/C/MODELS/2017/218
Javier L. Cánovas Izquierdo
jcanovasi@uoc.edu
@jlcanovas
Valerio Cosentino
vcosentino@uoc.edu
Jordi Cabot
jordi.cabot@icrea.cat
@softmodeling
Modeling projects achieve lower maturity
levels than non-modeling ones but that the
overall difference is small.
RQ1
The incubation status of a project affects the
maturity comparison between modeling and
non-modeling projects
RQ2
The incubation status makes a difference
only for non-modeling projectsRQ3
Take-home messages:

An Empirical Study on the Maturity of the Eclipse Modeling Ecosystem

Recommended

Recommended

More Related Content

Similar to An Empirical Study on the Maturity of the Eclipse Modeling Ecosystem

Similar to An Empirical Study on the Maturity of the Eclipse Modeling Ecosystem (20)

More from Javier Canovas

More from Javier Canovas (20)

Recently uploaded

Recently uploaded (20)

An Empirical Study on the Maturity of the Eclipse Modeling Ecosystem

Editor's Notes