]>
Commit | Line | Data |
---|---|---|
1 | # Go Plugin System over RPC | |
2 | ||
3 | `go-plugin` is a Go (golang) plugin system over RPC. It is the plugin system | |
4 | that has been in use by HashiCorp tooling for over 4 years. While initially | |
5 | created for [Packer](https://www.packer.io), it is additionally in use by | |
6 | [Terraform](https://www.terraform.io), [Nomad](https://www.nomadproject.io), and | |
7 | [Vault](https://www.vaultproject.io). | |
8 | ||
9 | While the plugin system is over RPC, it is currently only designed to work | |
10 | over a local [reliable] network. Plugins over a real network are not supported | |
11 | and will lead to unexpected behavior. | |
12 | ||
13 | This plugin system has been used on millions of machines across many different | |
14 | projects and has proven to be battle hardened and ready for production use. | |
15 | ||
16 | ## Features | |
17 | ||
18 | The HashiCorp plugin system supports a number of features: | |
19 | ||
20 | **Plugins are Go interface implementations.** This makes writing and consuming | |
21 | plugins feel very natural. To a plugin author: you just implement an | |
22 | interface as if it were going to run in the same process. For a plugin user: | |
23 | you just use and call functions on an interface as if it were in the same | |
24 | process. This plugin system handles the communication in between. | |
25 | ||
26 | **Cross-language support.** Plugins can be written (and consumed) by | |
27 | almost every major language. This library supports serving plugins via | |
28 | [gRPC](http://www.grpc.io). gRPC-based plugins enable plugins to be written | |
29 | in any language. | |
30 | ||
31 | **Complex arguments and return values are supported.** This library | |
32 | provides APIs for handling complex arguments and return values such | |
33 | as interfaces, `io.Reader/Writer`, etc. We do this by giving you a library | |
34 | (`MuxBroker`) for creating new connections between the client/server to | |
35 | serve additional interfaces or transfer raw data. | |
36 | ||
37 | **Bidirectional communication.** Because the plugin system supports | |
38 | complex arguments, the host process can send it interface implementations | |
39 | and the plugin can call back into the host process. | |
40 | ||
41 | **Built-in Logging.** Any plugins that use the `log` standard library | |
42 | will have log data automatically sent to the host process. The host | |
43 | process will mirror this output prefixed with the path to the plugin | |
44 | binary. This makes debugging with plugins simple. If the host system | |
45 | uses [hclog](https://github.com/hashicorp/go-hclog) then the log data | |
46 | will be structured. If the plugin also uses hclog, logs from the plugin | |
47 | will be sent to the host hclog and be structured. | |
48 | ||
49 | **Protocol Versioning.** A very basic "protocol version" is supported that | |
50 | can be incremented to invalidate any previous plugins. This is useful when | |
51 | interface signatures are changing, protocol level changes are necessary, | |
52 | etc. When a protocol version is incompatible, a human friendly error | |
53 | message is shown to the end user. | |
54 | ||
55 | **Stdout/Stderr Syncing.** While plugins are subprocesses, they can continue | |
56 | to use stdout/stderr as usual and the output will get mirrored back to | |
57 | the host process. The host process can control what `io.Writer` these | |
58 | streams go to to prevent this from happening. | |
59 | ||
60 | **TTY Preservation.** Plugin subprocesses are connected to the identical | |
61 | stdin file descriptor as the host process, allowing software that requires | |
62 | a TTY to work. For example, a plugin can execute `ssh` and even though there | |
63 | are multiple subprocesses and RPC happening, it will look and act perfectly | |
64 | to the end user. | |
65 | ||
66 | **Host upgrade while a plugin is running.** Plugins can be "reattached" | |
67 | so that the host process can be upgraded while the plugin is still running. | |
68 | This requires the host/plugin to know this is possible and daemonize | |
69 | properly. `NewClient` takes a `ReattachConfig` to determine if and how to | |
70 | reattach. | |
71 | ||
72 | **Cryptographically Secure Plugins.** Plugins can be verified with an expected | |
73 | checksum and RPC communications can be configured to use TLS. The host process | |
74 | must be properly secured to protect this configuration. | |
75 | ||
76 | ## Architecture | |
77 | ||
78 | The HashiCorp plugin system works by launching subprocesses and communicating | |
79 | over RPC (using standard `net/rpc` or [gRPC](http://www.grpc.io)). A single | |
80 | connection is made between any plugin and the host process. For net/rpc-based | |
81 | plugins, we use a [connection multiplexing](https://github.com/hashicorp/yamux) | |
82 | library to multiplex any other connections on top. For gRPC-based plugins, | |
83 | the HTTP2 protocol handles multiplexing. | |
84 | ||
85 | This architecture has a number of benefits: | |
86 | ||
87 | * Plugins can't crash your host process: A panic in a plugin doesn't | |
88 | panic the plugin user. | |
89 | ||
90 | * Plugins are very easy to write: just write a Go application and `go build`. | |
91 | Or use any other language to write a gRPC server with a tiny amount of | |
92 | boilerplate to support go-plugin. | |
93 | ||
94 | * Plugins are very easy to install: just put the binary in a location where | |
95 | the host will find it (depends on the host but this library also provides | |
96 | helpers), and the plugin host handles the rest. | |
97 | ||
98 | * Plugins can be relatively secure: The plugin only has access to the | |
99 | interfaces and args given to it, not to the entire memory space of the | |
100 | process. Additionally, go-plugin can communicate with the plugin over | |
101 | TLS. | |
102 | ||
103 | ## Usage | |
104 | ||
105 | To use the plugin system, you must take the following steps. These are | |
106 | high-level steps that must be done. Examples are available in the | |
107 | `examples/` directory. | |
108 | ||
109 | 1. Choose the interface(s) you want to expose for plugins. | |
110 | ||
111 | 2. For each interface, implement an implementation of that interface | |
112 | that communicates over a `net/rpc` connection or over a | |
113 | [gRPC](http://www.grpc.io) connection or both. You'll have to implement | |
114 | both a client and server implementation. | |
115 | ||
116 | 3. Create a `Plugin` implementation that knows how to create the RPC | |
117 | client/server for a given plugin type. | |
118 | ||
119 | 4. Plugin authors call `plugin.Serve` to serve a plugin from the | |
120 | `main` function. | |
121 | ||
122 | 5. Plugin users use `plugin.Client` to launch a subprocess and request | |
123 | an interface implementation over RPC. | |
124 | ||
125 | That's it! In practice, step 2 is the most tedious and time consuming step. | |
126 | Even so, it isn't very difficult and you can see examples in the `examples/` | |
127 | directory as well as throughout our various open source projects. | |
128 | ||
129 | For complete API documentation, see [GoDoc](https://godoc.org/github.com/hashicorp/go-plugin). | |
130 | ||
131 | ## Roadmap | |
132 | ||
133 | Our plugin system is constantly evolving. As we use the plugin system for | |
134 | new projects or for new features in existing projects, we constantly find | |
135 | improvements we can make. | |
136 | ||
137 | At this point in time, the roadmap for the plugin system is: | |
138 | ||
139 | **Semantic Versioning.** Plugins will be able to implement a semantic version. | |
140 | This plugin system will give host processes a system for constraining | |
141 | versions. This is in addition to the protocol versioning already present | |
142 | which is more for larger underlying changes. | |
143 | ||
144 | **Plugin fetching.** We will integrate with [go-getter](https://github.com/hashicorp/go-getter) | |
145 | to support automatic download + install of plugins. Paired with cryptographically | |
146 | secure plugins (above), we can make this a safe operation for an amazing | |
147 | user experience. | |
148 | ||
149 | ## What About Shared Libraries? | |
150 | ||
151 | When we started using plugins (late 2012, early 2013), plugins over RPC | |
152 | were the only option since Go didn't support dynamic library loading. Today, | |
153 | Go supports the [plugin](https://golang.org/pkg/plugin/) standard library with | |
154 | a number of limitations. Since 2012, our plugin system has stabilized | |
155 | from tens of millions of users using it, and has many benefits we've come to | |
156 | value greatly. | |
157 | ||
158 | For example, we use this plugin system in | |
159 | [Vault](https://www.vaultproject.io) where dynamic library loading is | |
160 | not acceptable for security reasons. That is an extreme | |
161 | example, but we believe our library system has more upsides than downsides | |
162 | over dynamic library loading and since we've had it built and tested for years, | |
163 | we'll continue to use it. | |
164 | ||
165 | Shared libraries have one major advantage over our system which is much | |
166 | higher performance. In real world scenarios across our various tools, | |
167 | we've never required any more performance out of our plugin system and it | |
168 | has seen very high throughput, so this isn't a concern for us at the moment. |