vendor/github.com/hashicorp/go-plugin/README.md

   1 # Go Plugin System over RPC
   2
   3 `go-plugin` is a Go (golang) plugin system over RPC. It is the plugin system
   4 that has been in use by HashiCorp tooling for over 3 years. While initially
   5 created for [Packer](https://www.packer.io), it has since been used by
   6 [Terraform](https://www.terraform.io) and [Otto](https://www.ottoproject.io),
   7 with plans to also use it for [Nomad](https://www.nomadproject.io) and
   8 [Vault](https://www.vaultproject.io).
   9
  10 While the plugin system is over RPC, it is currently only designed to work
  11 over a local [reliable] network. Plugins over a real network are not supported
  12 and will lead to unexpected behavior.
  13
  14 This plugin system has been used on millions of machines across many different
  15 projects and has proven to be battle hardened and ready for production use.
  16
  17 ## Features
  18
  19 The HashiCorp plugin system supports a number of features:
  20
  21 **Plugins are Go interface implementations.** This makes writing and consuming
  22 plugins feel very natural. To a plugin author: you just implement an
  23 interface as if it were going to run in the same process. For a plugin user:
  24 you just use and call functions on an interface as if it were in the same
  25 process. This plugin system handles the communication in between.
  26
  27 **Complex arguments and return values are supported.** This library
  28 provides APIs for handling complex arguments and return values such
  29 as interfaces, `io.Reader/Writer`, etc. We do this by giving you a library
  30 (`MuxBroker`) for creating new connections between the client/server to
  31 serve additional interfaces or transfer raw data.
  32
  33 **Bidirectional communication.** Because the plugin system supports
  34 complex arguments, the host process can send it interface implementations
  35 and the plugin can call back into the host process.
  36
  37 **Built-in Logging.** Any plugins that use the `log` standard library
  38 will have log data automatically sent to the host process. The host
  39 process will mirror this output prefixed with the path to the plugin
  40 binary. This makes debugging with plugins simple.
  41
  42 **Protocol Versioning.** A very basic "protocol version" is supported that
  43 can be incremented to invalidate any previous plugins. This is useful when
  44 interface signatures are changing, protocol level changes are necessary,
  45 etc. When a protocol version is incompatible, a human friendly error
  46 message is shown to the end user.
  47
  48 **Stdout/Stderr Syncing.** While plugins are subprocesses, they can continue
  49 to use stdout/stderr as usual and the output will get mirrored back to
  50 the host process. The host process can control what `io.Writer` these
  51 streams go to to prevent this from happening.
  52
  53 **TTY Preservation.** Plugin subprocesses are connected to the identical
  54 stdin file descriptor as the host process, allowing software that requires
  55 a TTY to work. For example, a plugin can execute `ssh` and even though there
  56 are multiple subprocesses and RPC happening, it will look and act perfectly
  57 to the end user.
  58
  59 **Host upgrade while a plugin is running.** Plugins can be "reattached"
  60 so that the host process can be upgraded while the plugin is still running.
  61 This requires the host/plugin to know this is possible and daemonize
  62 properly. `NewClient` takes a `ReattachConfig` to determine if and how to
  63 reattach.
  64
  65 ## Architecture
  66
  67 The HashiCorp plugin system works by launching subprocesses and communicating
  68 over RPC (using standard `net/rpc`). A single connection is made between
  69 any plugin and the host process, and we use a
  70 [connection multiplexing](https://github.com/hashicorp/yamux)
  71 library to multiplex any other connections on top.
  72
  73 This architecture has a number of benefits:
  74
  75   * Plugins can't crash your host process: A panic in a plugin doesn't
  76     panic the plugin user.
  77
  78   * Plugins are very easy to write: just write a Go application and `go build`.
  79     Theoretically you could also use another language as long as it can
  80     communicate the Go `net/rpc` protocol but this hasn't yet been tried.
  81
  82   * Plugins are very easy to install: just put the binary in a location where
  83     the host will find it (depends on the host but this library also provides
  84     helpers), and the plugin host handles the rest.
  85
  86   * Plugins can be relatively secure: The plugin only has access to the
  87     interfaces and args given to it, not to the entire memory space of the
  88     process. More security features are planned (see the coming soon section
  89     below).
  90
  91 ## Usage
  92
  93 To use the plugin system, you must take the following steps. These are
  94 high-level steps that must be done. Examples are available in the
  95 `examples/` directory.
  96
  97   1. Choose the interface(s) you want to expose for plugins.
  98
  99   2. For each interface, implement an implementation of that interface
 100      that communicates over an `*rpc.Client` (from the standard `net/rpc`
 101      package) for every function call. Likewise, implement the RPC server
 102      struct this communicates to which is then communicating to a real,
 103      concrete implementation.
 104
 105   3. Create a `Plugin` implementation that knows how to create the RPC
 106      client/server for a given plugin type.
 107
 108   4. Plugin authors call `plugin.Serve` to serve a plugin from the
 109      `main` function.
 110
 111   5. Plugin users use `plugin.Client` to launch a subprocess and request
 112      an interface implementation over RPC.
 113
 114 That's it! In practice, step 2 is the most tedious and time consuming step.
 115 Even so, it isn't very difficult and you can see examples in the `examples/`
 116 directory as well as throughout our various open source projects.
 117
 118 For complete API documentation, see [GoDoc](https://godoc.org/github.com/hashicorp/go-plugin).
 119
 120 ## Roadmap
 121
 122 Our plugin system is constantly evolving. As we use the plugin system for
 123 new projects or for new features in existing projects, we constantly find
 124 improvements we can make.
 125
 126 At this point in time, the roadmap for the plugin system is:
 127
 128 **Cryptographically Secure Plugins.** We'll implement signing plugins
 129 and loading signed plugins in order to allow Vault to make use of multi-process
 130 in a secure way.
 131
 132 **Semantic Versioning.** Plugins will be able to implement a semantic version.
 133 This plugin system will give host processes a system for constraining
 134 versions. This is in addition to the protocol versioning already present
 135 which is more for larger underlying changes.
 136
 137 **Plugin fetching.** We will integrate with [go-getter](https://github.com/hashicorp/go-getter)
 138 to support automatic download + install of plugins. Paired with cryptographically
 139 secure plugins (above), we can make this a safe operation for an amazing
 140 user experience.
 141
 142 ## What About Shared Libraries?
 143
 144 When we started using plugins (late 2012, early 2013), plugins over RPC
 145 were the only option since Go didn't support dynamic library loading. Today,
 146 Go still doesn't support dynamic library loading, but they do intend to.
 147 Since 2012, our plugin system has stabilized from millions of users using it,
 148 and has many benefits we've come to value greatly.
 149
 150 For example, we intend to use this plugin system in
 151 [Vault](https://www.vaultproject.io), and dynamic library loading will
 152 simply never be acceptable in Vault for security reasons. That is an extreme
 153 example, but we believe our library system has more upsides than downsides
 154 over dynamic library loading and since we've had it built and tested for years,
 155 we'll likely continue to use it.
 156
 157 Shared libraries have one major advantage over our system which is much
 158 higher performance. In real world scenarios across our various tools,
 159 we've never required any more performance out of our plugin system and it
 160 has seen very high throughput, so this isn't a concern for us at the moment.
 161