This document discusses programming languages and modeling. It notes that programming languages are not expressive, high-level, abstract, domain-specific, or modular enough. It proposes addressing this by using modeling with higher-level, domain-specific concepts and notations, and code generation from models. However, modeling and programming tools have traditionally been separate worlds. The document envisions a future where modeling and programming are integrated by mixing models and programs, and developing languages and tools that support programming at different levels of abstraction from different viewpoints. Enabling technologies could include advanced parser generators and projectional editing, while available tools mentioned include Eclipse Xtext and JetBrains' Meta Programming System for developing domain-specific languages.

1. Programming from to Modeling …and back again Markus Voelter Independent/itemis voelter@acm.org

2. A Journey…

3. „Houston, we have a Problem!“

4. Programming Languages Erlang C# C++ Python Java Ruby Groovy Fortran C

5. Programming Languages are not expressive enough. Erlang C# C++ Python Java Ruby Groovy Fortran C

6. Programming Languages are not high-level enough. Erlang C# C++ Python Java Ruby Groovy Fortran C

7. Programming Languages are not abstract enough. Erlang C# C++ Python Java Ruby Groovy Fortran C

8. Programming Languages are not domain-specific enough. Erlang C# C++ Python Java Ruby Groovy Fortran C

9. Programming Languages are not enough. Erlang C# C++ Python Java Ruby Groovy Fortran C

10. Programming Languages Formats Erlang C# C++ Python Java Ruby Groovy Fortran json C HTML CSS yaml XML

11. Programming Languages Formats Frameworks Erlang C# C++ Python Java Django Ruby Groovy WPF Fortran json C JEE JMS HTML CSS Rails yaml XML lift

12. Programming Languages Formats Frameworks enough. are not Erlang C# C++ Python Java Django Ruby Groovy WPF Fortran json C JEE JMS HTML CSS Rails yaml XML lift

13. not enough? for what?

14. not enough? for what?

15. not enough? for what?

16. So?

17. Modeling … Higher Level … Domain Specific Concepts & Notations … Code Generation Interpretation

18. Modeling … Higher Level … Domain Specific Concepts & Notations … Code Generation Interpretation Solvesthe Problem!

19. But:

20. Different Worlds Programming Tools != Modeling Tools

21. Different Worlds Modeling Tool != Modeling Tool

22. Different Worlds Mix Models and Programs

23. Different Worlds Mix Models and Programs AST Navigation & Query

24. Different Worlds Mix Models and Programs AST Navigation & Query Integration of 3GL code

25. Different Worlds Mix Models and Programs AST Navigation & Query Integration of 3GL code Code Constraints

26. pare Com

27. pare Com

28. Domain Specific Notations and Abstractions Flexible! Limited! Frameworks Libraries (Fluent) APIs

29. Graphical Textual Forms Tables Limited! Flexible! Textual Trees

30. Customize Generator or Interpreter Limited! Flexible! Reflection Meta Programs Open Compilers

31. Define custom Query or Navigate or Transform Limited! Flexible! AST APIs Static Analysis Regex

32. Custom Validation or Error Checks Limited! Flexible! IDE plugins Static Analysis Open Compilers

33. Different Representations and Projections Limited! Flexible! Text is Text Code Folding Tree Views Visualizations

34. Mixing and Composing Languages Limited! Flexible? Python-to-C-like Internal DSLs Embed-As-String Specific: LINQ

35. Scalable Usable IDE Support Mature! Brittle! Modeling Tools…!?

36. Debugging Refactoring Testing Mature! Brittle! ?

37. Versioning Diff, Merge Branching Mature! Brittle! for some tools…

38. Gets the Gets some Job Done! Jobs done. some people doubt that… everybody agrees…

39. Why the difference?

40. History?

41. Modeling Programming

42. Modeling Programming … (Mostly) Textual Notations … Concrete Syntax Storage … (Fancy) ASCII Editors … Read-Only Visualizations

43. Modeling Programming … (Mostly) Graphical Notations … Abstract Syntax Storage … Projecting Editors … Different editable views for model … (Mostly) Textual Notations … Concrete Syntax Storage … (Fancy) ASCII Editors … Read-Only Visualizations

44. Why the difference?

45. It is time for …

46. … a Different Perspective

47. Programming the waywe do Modeling? Modeling the waywe do Programming?

48. Modeling Programming == == Programming Modeling

49. We don‘t want to model, we want to program!

50. We don‘t want to model, we want to program! … at different levels of abstaction … from different viewpoints … integrated!

51. We don‘t want to model, we want to program! … with different degrees of domain-specificity … with suitable notations … with suitable expressiveness

52. We don‘t want to model, we want to program! And always: precise and tool processable

53. Where do we go from here?

54. A vision for  programming

55. A vision for  programming Enabling  Technologies

56. A vision for  programming Enabling  Technologies Available  Tooling

57.  A vision for Programming 

58. Programming Languages are not MODULAR enough.

59. Programming Languages are not COMPOSABLE enough.

60. Programming Languages are not CONFIGURABLE enough.

61. Programming Languages are not ADAPTABLE enough.

62. Programming Language Syntax is not FLEXIBLE enough.

63. Big Language? withmany first class concepts!

64. Small Language? with a few, orthogonal and poweful concepts

65. a b c d e f g h i j k l Modular Language with many optional, composable concepts

66. Modular Language Likeframeworksandlibraries,

67. Modular Language Likeframeworksandlibraries, but withsyntaxand IDE support

68. Not a new idea…

71. Growing A Language (Guy L Steele)

72. Language Workbench (Martin Fowler)

73. Language Workbench (Martin Fowler) Freely define languages and integrate them

74. Language Workbench (Martin Fowler) use ? persistent abstract representation

75. Language Workbench (Martin Fowler) language ::= schema editors + generators +

76. Language Workbench (Martin Fowler) ? projectional editing

77. Language Workbench (Martin Fowler) persist incomplete or contradictory information

78. Language Workbench (Martin Fowler) powerful + editing testing refactoring debugging groupware language definition implies IDE definition

79. Language Workbench (Martin Fowler) + support for „classical“ programming and „classical“ modeling

80. Syntax primarily textual

81. Syntax primarily textual withmoresymbols think: mathematics

82. Syntax primarily textual sometimes box&line style

83. Syntax primarily textual sophisticated visualizations

84. Viewpoints

85. Viewpoints suitable abstractions and notations foreach

86. Viewpoints Integrated via symbolic references and seamless transitions

87. Viewpoints Business custom purpose-built create/include

88. Viewpoints Business Custom Notations real business expert integration

89. Viewpoints Business Custom Notations real business expert integration But that‘s another talk…

90. Viewpoints Technical predefined library configure

91. Example Languages

116. (Seemingly) Simple Example Adding matrices to C in an embedded environment.

117. Currently:

118. Currently: Matrices not supported in XML format and generator

119. Currently: Tool team would have to do the extension … a lot of work … busy … one central tool

120. Currently: No real compiler support in the resulting C code … type checks … operator overloading … generics (matrix<int>) … matrix syntax?

122. Better Solution

123. Better Solution generic matrix and vector types

124. Better Solution real matrix and vector literals

125. Better Solution syntax highlights for vectors and matrices

126. Better Solution operator overloading

127. Better Solution operator overloading static optimization … symmetrical matrices … identity matrix … diagonal matrices

128. Better Solution math notation

129. Better Solution a separate language module used only by those who really need it

131. In addition: PLE Variability annotating variability expressions to arbitrary regions

132. In addition: PLE Variability annotating variability expressions to arbitrary regions statically checked

133. In addition: PLE Variability annotating variability expressions to arbitrary regions project variant

135. more later… (tool examples)

138.  Enabling Technologies 

139.  Enabling Technologies  Advanced Parser Generators

140. Modeling as Programmig … (Mostly) Textual Notations … Concrete Syntax Storage … (Fancy) ASCII Editors … Read-Only Visualizations

141. Custom Syntax Graphical Textual Symbolic++

142. IDE Support Teamwork Debugging Custom Editors

143. Complete Symbolic Integration Goto Def Find Refs Refactoring

144. Xtext-like Tools provide editor support Custom Editors Teamwork Goto Def Find Refs Refactoring

145. Limited to Unicode how to handle non-character symbols

146. Graphics != Text two worlds… separate editors … per syntax/viewpoint … models can still be ref integrated

148. What if! combine languages same seq<Char> as different tokens?

149. Ambiguous! Not Parsable. Bad Luck.

150. Traditionally: Language ::= Token* | Rule*;

151. Traditionally: Language ::= Token* | Rule*; Token ::= Character*;

152. Traditionally: Language ::= Token* | Rule*; Token ::= Character*; Rule ::= Token* | Rule;

153. Traditionally: Language ::= Token* | Rule*; 2 Phases: 1. Global Tokenization (aka Scanning) 2. Grammar Recognition (aka Parsing)

154. Modern: Language ::= Rule*; Rule ::= Char* | Rule*; 1 Phase: Grammar Recognition directly on char*

155. Scannerless Parsing No Ambiguity when combining languages Always Parsable. (well, almost. There are some corner cases…)

157.  Enabling Technologies  Projectional Editing

158. text … to tree … to text Parser-based

159. Projectional tree … to text-lookalike (editor) … to other trees … [*] … to text

160. Programming as Modeling … (Mostly) Graphical Notations … Abstract Syntax Storage … Projecting Editors … Different editable views for model

161. Programming as Modeling … (Mostly) Graphical Any kind of Notations … Abstract Syntax Storage … Projecting Editors … Different editable views for model

162. Language Composition There‘sno parsing. Unique Language Element Identity. Unlimted language composition.

163. Flexible Notations Textual like ASCII treated the same } Graphical can be mixed box & line Semi-Graphical mathematical

164. Automatic IDE Extension tool support is inherent for languages build with projectional tools language definition implies IDE definition

165. Multiple Notations … for the same concepts e.g. in different contexts or for different tasks

166. Partial Projections … different views … for different roles/people … only a particular variant

167. Storage != Schema … store arbitraty meta data change log conflicting information variability annotations … independent of language schema! … „aspects“, overlay

168. Live Programs think: spreadsheet a change to one part of program can lead to (dependent) changes in other parts

169. Tree Editing … is different from editing text … try to make it feel like text … takes some gettingused to but: for more flexible notations a more general editing paradigm is needed

170. Infrastructure Integration … storage is not text … diff/merge must be in tool … existing text tools don‘t work

171. Proprietary Tools … no standards … no interop

173.  Available Tooling 

174.  Available Tooling  Eclipse Xtext

175. http://eclipse.org/modeling

176. http://eclipse.org/xtext

177. Eclipse Xtext Building Textual Editors

178. Eclipse Xtext Building Textual Editors

179. Xtext: Specify Grammar

180. Xtext: Gen. Meta Model

181. Xtext: Constraints

182. Xtext: Generated Editor

183. Xtext: Generated Editor Code Completion

184. Xtext: Generated Editor Syntax Coloring Custom KeywordColoring

185. Xtext: Generated Editor RealtimeConstraintValidation

186. Xtext: Generated Editor Customizable Outlines

187. Xtext: Generated Editor Code Folding

188. Xtext: Generated Editor Goto Definition Find References Cross-File References Model as EMF

189. Xtext: Generated Editor

191. PlatformArchitecture DSL

192. As you understand anddevelopyour PlatformArchitecture…

193. Develop a language to express it!

194. Language resembles architectural concepts…

195. Express the applications with the language.

199. Another Story… Architecture As Language

201. DEMO I Building DSLs with Eclipse Xtext

203.  Available Tooling 

204.  Available Tooling  Jetbrains‘ Meta Programming System

205. also do… IntelliJ IDEA Resharper

206. released in Q3 2009 currently 1.1 RC1

207. licensed under Apache 2.0

208. Build new standalone DSLs

209. Build new standalone DSLs Build DSLs that reuse parts of other languages

210. Build new standalone DSLs Build DSLs that reuse parts of other languages Java++ (MPS comes with BaseLanguage) extend base language

211. Build new standalone DSLs Build DSLs that reuse parts of other languages Java++ (MPS comes with BaseLanguage) extend base language build DSLs that reuse parts of BaseLanguage

212. Language Extension Example Java + Extension

213. Language Extension Example Old Java Java + Extension ReadWriteLock l = … l.readLock().lock(); try { //code } finally { l.readLock().unlock(); }

214. Language Extension Example Old New Java Java + Extension ReadWriteLock l = … l.readLock().lock(); try { //code } finally { l.readLock().unlock(); } ReadWriteLock l = … lock (l) { //code }

215. Structure  Editor  Typesystem  Generator

216. Structure  Editor  Typesystem  Generator

217. Structure  Editor  Typesystem  Generator

218. Structure  Editor  Typesystem  Generator

219. Structure  Editor  Typesystem  Generator

220. Structure  Editor  Typesystem  Generator

221. Structure  Editor  Typesystem  Generator

222. Structure  Editor  Typesystem  Generator

223. Structure  Editor  Typesystem  Generator

224. Language Extension Example Result behaves like a native base language construct

225. Language Extension Example Result behaves like a native base language construct

226. Language Extension Example Translated to regular Java code based on the generator package jaxdemo.sandbox.sandbox; import java.util.concurrent.locks.Lock; public class DemoClass { private Lock lock; public DemoClass() { try { this.getLock().lock(); SharedResouce.instance().doSomething(); } finally { this.getLock().unlock(); } } private Lock getLock() { return this.lock; } }

228. Example Languages UI Language

229. Example Languages HTML Templates

230. Example Languages Persistent Classes

232. DEMO I Building DSLs with JetBrains MPS

234.  Available Tooling 

235.  Available Tooling  Intentional Software Domain Workbench

237. Version 1.0 released in Dec 15, 2008 currently at 1.4

238. Commercial Product

239. Eval available upon request

240. Statemachine Example Native Projection

241. Statemachine Example Tabular Projection

242. Statemachine Example Textual DSL Projection

243. Statemachine Example Java Projection

244. Statemachine Example Ruby Projection

245. Statemachine Example Two projections side/side

246. Statemachine Example Two projections side/side

248. Pension Workbench Example Text Editing Domain

249. Pension Workbench Example Insurance Mathematics Domain

250. Pension Workbench Example Pension Contract Rules Domain

251. Pension Workbench Example All in one Document

252. Pension Workbench Example Symbolically integrated

254. A Journey… Arewethere?

255. A Journey… .coordinates web email skypexinglinkedin www.voelter.devoelter@acm.orgschogglad http://www.xing.com/profile/Markus_Voelter http://www.linkedin.com/pub/0/377/a31