Netflix Nebula - Gradle Summit 2014
•
22 likes•7,824 views
Netflix has open sourced many of our Gradle plugins under the name Nebula. These plugins are there to lend our expertise and experience to building responsible projects, internally and externally. This talk will cover some of the ones we've published, why we want to share these with the community, how we tested and published them, and most importantly how you can contribute back to them. Nebula started off as a set of strong opinions to make Gradle simple to use for our developers. But we quickly learned that we could use the same assumptions on our open source projects and on other Gradle plugins to make them easy to build, test and deploy. By standardizing plugin development, we've lowered the barrier to generating them, allowing us to keep our build modular and composable.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (17)
Similar to Netflix Nebula - Gradle Summit 2014
Similar to Netflix Nebula - Gradle Summit 2014 (20)
Recently uploaded
Recently uploaded (20)
Netflix Nebula - Gradle Summit 2014
- 1. @quidryan@quidryan http://www.slideshare.net/quidryan Netflix Build Language Justin Ryan <jryan@netflix.com>
- 3. Why Netflix Uses Gradle • Better Dependency Management story • Flexible lifecycle • Groovy
- 4. How Netflix Uses Gradle • JVM Languages • Resolution • Code Quality • Publishing • Deployment Orchestration
- 5. How Netflix Sets up Gradle • Patched Gradle • Custom Distribution • Custom Wrapper
- 6. apply plugin: ‘nebula’! apply plugin: ‘java’! ! nebula {! readyForJava7 = true! }! ! dependencies {! compile ‘netflix:platform:latest.release’! } build.gradle
- 8. Unite two builds • Model a responsible project • Componentize via plugins
- 9. nebula-plugins • Infrastructure • Use Github • Use CloudBees • Use Bintray • Mailing List • nebula-plugin-plugin
- 10. Github Repositories nebula-* or gradle-*-plugin
- 11. Ensuring Github • https://github.com/nebula-plugins/ensure • Ensure repository has • Description • Web Hooks • Ensure “contrib" team has all repositories in it • Ensure a contrib team exists for every repository
- 12. Continuous Integration • Job DSL to create jobs • Per branch • Snapshot job • Release job • Lock job
- 13. CloudBees jobs Release, snapshot, and pull request per branch
- 14. Job DSL
- 15. Bintray Packages
- 16. Ensuring Bintray • https://github.com/nebula-plugins/ensure • Ensure every repository has a package • Ensure package has the description as Github • Ensure license is set to Apache 2.0 • Ensure labels are “gradle" and “nebula”
- 18. nebula-test • ProjectSpec • PluginProjectSpec • IntegrationSpec • Thanks to Marcin and Luke • Runs with Tooling API or GradleLauncher
- 19. class PluginExampleSpec extends PluginProjectSpec {! @Override! String getPluginName() { return 'plugin-example' }! ! def ‘run task’() {! when:! project.plugins.apply(PluginExample)! ! then:! def t = project.tasks.get(‘example’)! ! when:! t.run()! ! then:! new File(projectDir, ‘build/example.txt’).exists()! }! } PluginProjectSpec
- 20. ! def 'setup and run build'() {! buildFile << '''! apply plugin: 'java'! '''.stripIndent()! ! when:! writeHelloWorld('nebula.hello')! ! then:! fileExists('src/main/java/nebula/hello/HelloWorld.java')! ! when:! def result = runTasksSuccessfully(‘build’, ‘-v’)! ! then:! fileExists(‘build/classes/main/nebula/hello/HelloWorld.class')! result.wasExecuted(':compileTestJava')! def output = result.standardOutput! output.contains('Skipping task ’:compileTestJava'')! } IntegrationSpec
- 21. nebula-core • Collection of tasks • Download • Untar • Unzip • AlternativeArchiveTask • CopySpecHelper • GradleHelper
- 22. class GradleHelper {! ! def beforeEvaluate(Closure beforeEvaluateClosure)! ! def getTempDir(String taskBaseName)! ! def addDefaultGroup(String defaultGroup)! }! ! class CopySpecHelper {! ! def visitCopySpec(CopySpecInternal copySpec, Closure closure)! ! def findCopySpec(CopySpecInternal delegateCopySpec, closure)! }! ! class ClassHelper {! ! String findSpecificationVersion(Class clazz)! ! Manifest findManifest(Class clazz)! ! def findManifestValue(Class clazz, String key, defaultValue)! } nebula-core helpers
- 23. nebula-publishing-plugin • Artifact plugins • nebula-javadoc-jar • nebula-source-jar • nebula-test-jar • Publishing plugins • resolved-ivy • resolved-maven • nebula-sign
- 24. apply plugin: ‘nebula-maven-publishing‘! apply plugin: ‘nebula-source-jar'! apply plugin: ‘nebula-javadoc-jar'! apply plugin: ‘nebula-test-jar'! apply plugin: ‘nebula-sign'! apply plugin: 'java' nebula-publishing.gradle my-plugin-1.12.0-javadoc.jar! my-plugin-1.12.0-javadoc.jar.md5! my-plugin-1.12.0-javadoc.jar.sha1! my-plugin-1.12.0-javadoc.jar.asc! my-plugin-1.12.0-sources.jar! my-plugin-1.12.0-sources.jar.md5! my-plugin-1.12.0-sources.jar.sha1! my-plugin-1.12.0-sources.jar.asc! my-plugin-1.12.0-tests.jar! my-plugin-1.12.0-tests.jar.md5! my-plugin-1.12.0-tests.jar.sha1! my-plugin-1.12.0-tests.jar.asc! my-plugin-1.12.0.jar! my-plugin-1.12.0.jar.md5! my-plugin-1.12.0.jar.sha1! my-plugin-1.12.0.jar.asc! my-plugin-1.12.0.pom! my-plugin-1.12.0.pom.md5! my-plugin-1.12.0.pom.sha1! my-plugin-1.12.0.pom.asc
- 25. gradle-info-plugin • Collects meta data • ‘info-java' • ‘info-ci' • 'info-scm' • Reports in key/value pairs • ‘info-jar' • ‘info-props'
- 26. buildscript {! ! repositories { jcenter() }! ! dependencies { ! ! ! classpath 'com.netflix.nebula:gradle-info-plugin:1.12.+'! ! ! classpath ‘org.eclipse.jgit:org.eclipse.jgit: 3.2.0.201312181205-r'! ! }! }! apply plugin: 'info' info.gradle
- 28. gradle-contacts-plugin • Express people involved in the project • Make people and roles available to other plugins • contacts-manifest • contacts-pom
- 29. apply plugin: 'contacts'! contacts 'minnie@disney.com', ‘mickey@disney.com'! ! contacts {! 'club@disney.com'! 'bobby@company.com' {! roles 'notify', 'owner'! }! 'billy@company.com' {! role 'techwriter'! }! 'downstream@netflix.com'! role 'notify'! }! }! contacts.gradle
- 30. gradle-scm-plugin • Attempt to provide SCM abstraction for other plugins • E.g. gradle-dependency-lock-plugin and gradle-info- plugin
- 31. gradle-dependency-lock-plugin • Developer declare their ideal situation • Save resolved version • If tests pass, commit to SCM
- 32. {! "com.github.townsfolk:gradle-release": { ! ! ! "locked": "1.2", "requested": "1.2" },! "com.jfrog.bintray.gradle:gradle-bintray-plugin": { ! ! ! "locked": "0.3", "requested": "0.3" },! "com.netflix.nebula:nebula-project-plugin": { ! ! ! "locked": "1.12.0", "requested": "1.12.+" },! "com.netflix.nebula:nebula-test": { ! ! ! "locked": "1.12.0", "requested": "1.12.+" },! "org.codehaus.groovy.modules.http-builder:http-builder": { ! ! ! "locked": "0.7.1", "requested": “latest.release" },! "org.jfrog.buildinfo:build-info-extractor-gradle": { ! ! ! "locked": "2.2.4", "requested": "2.2.+" }! } apply plugin: ‘gradle-dependency-lock'! lock.gradle ./gradlew generateLock
- 33. nebula-integtest-plugin • Sets up integTest source set • Adds integTestCompile and integTestRuntime configurations • Creates integrationTest task
- 34. nebula-project-plugin • Pull together other plugins • Responsible projects
- 35. nebula-plugin-plugin • Used by plugins • Strong opinions on how to publish • Force nebula-project- plugin on projects
- 36. gradle-ospackage-plugin • Merging ubuntu-packager-plugin and gradle-rpm- plugin • Uses CopySpec definition • Via just Java, generates RPMs and DEBs
- 37. ospackage {! ! os = LINUX! ! into '/opt/foo'! ! from ('dist') {! ! ! user 'builds'! ! ! exclude '**/*.md'! ! }! ! postInstall file('scripts/postInstall.sh')! }! ! buildRpm {! ! requires('bar', '2.2', GREATER | EQUAL)! ! from (‘build/metadata.properties’)! ! link(‘/etc/init.d/foo’, '/opt/foo/bin/foo.sysv',)! }! ! buildDeb {! ! link('/etc/init/foo', '/opt/foo/bin/foo.upstart')! } ospackage.gradle
- 38. gradle-override-plugin • Take command line arguments • Intelligently apply in afterEvaluate • E.g. -Nfindbugs.enabled=false
- 39. Internal Plugins • netflix-repos • nebula-ospackage • nebula-grails • nebula-findbugs, etc • ivyimport • nebula-fixexcludes • nebula-intellij
- 40. buildscript {! ! repositories { jcenter() }! ! dependencies { ! ! ! classpath ‘com.netflix.nebula:nebula-plugin-plugin:1.12.+'! ! ! classpath ‘org.eclipse.jgit:org.eclipse.jgit: 3.2.0.201312181205-r'! ! }! }! ! description ‘Example Plugin'! apply plugin: ‘nebula-plugin'! ! contacts {! ‘jryan@netflix.com’ {! moniker 'Justin Ryan'! github 'quidryan'! }! } plugin.gradle
- 41. curl -s get.gvmtool.net | bash! gvm install lazybones! cd ~/Projects/github/nebula-plugins! lazybones create nebula-plugin <name-of-project>! cd <name-of-project>! git init! git remote add origin git@github.com:nebula-plugins/<name-of- project>.git! ./gradlew clean build! git add -A! git add -f gradle/wrapper/gradle-wrapper.jar! git commit -m "Initial template”! git push --set-upstream origin master Making a plugin https://github.com/nebula-plugins/nebula- plugins.github.io/wiki/New-Plugins
- 42. Getting it out the door • Let “ensure” run • Run <name-of-project>-release • Link package to jcenter • Link package to gradle-plugins
- 45. Gotchas • NamedDomainObjectSet • Debugging Tooling • File as @Output • afterEvaluate
- 46. Outstanding • CloudBees permissions • Bot to create repositories
- 47. Participating • Use individual plugins • Get on nebula-plugins Google Group • Move your plugin to nebula-plugins • Start a new plugin in nebula-plugins
- 48. @quidryan@quidryan http://www.slideshare.net/quidryan We’re hiring Justin Ryan <jryan@netflix.com> HOUSE of GRADLE
Editor's Notes
- I work on the Engineering Tools team at Netflix, where we build tools to support the developers. Today I’m here to talk about Netflix Build Language, or as we like to call it, nebula.
- And I’m here to talk about how we try to exist in the open source space with plugins.
- We had to transition out of a Ant/Ivy. Primarily a JVM shop, there are a few players in this space for build. A lot of conventions were in place and changing them to fit Maven’s model wasn’t an option.
- Look after the users, and where we can add value.
- Just couldn’t get the experience as smooth as we wanted it in our enterprise environment. Still haven’t. Patched Version of Gradle - Patch to cover a bug in Ivy. Patch to expose status/statusScheme. Custom Distribution - So we can embed some init.d scripts, e.g. add our repository servers and add our plugin’s jar. Custom Wrapper - To force some variables getting set, like memory defaults or GRADLE_USER_HOME. Plans on customizing to support for re-downloading of static URL
- End user sees apply plugin nebula. Nebula extension block to hold other extensions via @Delegate annotation.
- NetflixOSS pre-dated our internal work, but it was going down the Maven route. And we had a goal that we wanted to move to Gradle. We also had concerns about how to integration POMs and our internal Ivy files. That was done in a way to make the Gradle work really obvious, i.e. minimal work hidden in plugins, no custom DSL. It’s hideous at this point and based on Gradle 1.6. We had to rush it, to pre-empt projects going out with Maven. github/cloudbees/bintray project plugin forthcoming.
- NetflixOSS fell way behind our internal development. Majority of our work was not Netflix specific, but turned out to just be what a responsible project needs. Also had growing pains with our plugin, so we wanted to decouple them. Wanted to make new plugins easy, with CI and SCM. We’re build people, right?
- Found that many plugins we wanted to use also suffered from basic release engineering practices, which is clearly ironic given the space we’re in. Group plugin with SCM, CI and Deployment. Chosen of familiarity, I wish I had looked at more options. Common patterns to give the apply plugin nebula like experience, enabled via nebula-plugin-plugin. Lot of hard coding.
- Exists as a dedicated Organization, with project names following a pattern.
- We learned from the @NetflixOSS work that GitHub doesn’t maintain itself.
- We have to eliminate the manual work. And Jenkins is a great place to do that, except that setting up Jenkins is manual work. I don’t want to fight Jenkins, so we have a script to manage this.
- The nebula prefix for highly opinionated plugins. While the gradle prefix is for plugins that have general applicability. Many of them are used in the plugin-plugin since they’re so helpful.
- Not really a plugin, but integrated into the Gradle ecosystem. Meant to be used in tests. * ProjectSpec has its limitation, since you can only really apply plugins. Though you can run a hidden evaluate(). Really fast and recommended when ever possible PluginProjectSpec is a small addition that just applies your plugin by name, tries it twice and tries it in multi-module project. IntegrationSpec run a real Gradle build.
- All the nebula-plugins use these Test classes, so there’s plenty of examples.
- Helper methods to create a project, to run a project, and to evaluate the effects of a execution. Will get it’s own directory to run in. GradleLauncher is necessary for some in-memory checks or debugging.
- Not really a plugin, has tasks and Helper method. Meant for runtime. AlternativeArchiveTask is to provide an implementation of AbstractArchiveTask, which is not also a CopyTask, since those are greaten special.
- beforeEvaluate is actually before after evaluates getTempDir gives a build directory for a task addDefaultGroup lets you set a group, optionally. Hard otherwise because project.getGroup() will provide the parent project’s name as the group.
- Finally a real plugin. Though docs are the worst of the bunch, this is the most important. Actually a bunch of plugins. All meant to get a publication looking just right, down to the signing. Artifact plugins attempt to make the jars (which we’d all expect). Publishing plugins attempt to make resulting Ivy/POM file cleaner, primarily by using resolved variables and including excludes. nebula-sign looks for magical properties and conditionally signs, unlike other approached. To support these, we needed to use Publish 2.0, which isn’t fully baked. We were unable to use as is, so we made our CustomComponent. Allows any plugins to contribute artifacts, with more control of the resulting dependencies and confs (more important in Ivy).
- Two plugins are available for ivy vs maven, nebula-ivy-publishing and nebula-maven-publishing.
- ‘info-java&apos; version of java being used ‘info-ci&apos; tells about CI system and current build ‘info-scm&apos; derives info about current SCM ‘info-jar&apos; injects into the Jar’s MANIFEST.MF ‘info-props&apos; creates file with values ‘info-jar-props&apos; puts property file in jar for later retrieval All go through a broker that you can listen to.
- contacts-manifest puts people into the JAR manifest contacts-pom puts people into developers section of POM Example of how we believe some plugins can talk to each other without a DSL in the way.
- Then we can send emails on release by the notify role.
- We believe in a reproducible build for every developer all the time. Dynamic versions make that hard. We can’t imagine manually editing versions. We’ve all had the experience of a new dependency get published and everyone is broken. We also have cases where we want to tweak a specific module, for a patch, which can be done via the command line. Inspired by systems like Bundler.io. Let some automated job update your dependencies when they’ve proven valid.
- Properly sets up classpath.
- Does the things we’d expect in a responsible project. Even if they don’t like it. E.g. doesn’t fail on Javadoc error, maintains status even though the java plugin wants to walk all over it.
- Adds OJO and Bintray, with publishing. Plugin Portal additions. Needs to be extracted out for different orgs, different package names. Very meta, since it applies itself.
- Most popular plugin, even though we haven’t advertised them at all. Takes the CopySpec idea and applies it to redline and jdeb.
- buildRpm and buildDeb are implicitly created, but can still be configured if needed.
- Proposed.
- Anytime we find a problem internally, we make a plugin to test the problem and fix it. Roll it out and get happy users. Many times, we’re wrapping another plugin and configuring them with our defaults.
- https://github.com/nebula-plugins/lazybones-nebula-plugin-template
- A few manual steps to finalize release.
- Lots of withType or .all {} calls Ability to create tasks in reaction Also try to abstract out logic out of Task, so that it can be called sequentially.
- Sometimes we even got a third level Configure tasks
- Reacting to the user before the task graph, forces afterEvaluate’s NamedDomainObjectSet configuration comes after events, so no ability to re-act, except by name @Outputs have to be Files, not in-memory String. Evaluation of File names for outputs can be tricky. Can’t debug tests through Tooling API