]>
Commit | Line | Data |
---|---|---|
bae9f6d2 JC |
1 | # Go Plugin System over RPC |
2 | ||
3 | `go-plugin` is a Go (golang) plugin system over RPC. It is the plugin system | |
4 | that has been in use by HashiCorp tooling for over 3 years. While initially | |
5 | created for [Packer](https://www.packer.io), it has since been used by | |
6 | [Terraform](https://www.terraform.io) and [Otto](https://www.ottoproject.io), | |
7 | with plans to also use it for [Nomad](https://www.nomadproject.io) and | |
8 | [Vault](https://www.vaultproject.io). | |
9 | ||
10 | While the plugin system is over RPC, it is currently only designed to work | |
11 | over a local [reliable] network. Plugins over a real network are not supported | |
12 | and will lead to unexpected behavior. | |
13 | ||
14 | This plugin system has been used on millions of machines across many different | |
15 | projects and has proven to be battle hardened and ready for production use. | |
16 | ||
17 | ## Features | |
18 | ||
19 | The HashiCorp plugin system supports a number of features: | |
20 | ||
21 | **Plugins are Go interface implementations.** This makes writing and consuming | |
22 | plugins feel very natural. To a plugin author: you just implement an | |
23 | interface as if it were going to run in the same process. For a plugin user: | |
24 | you just use and call functions on an interface as if it were in the same | |
25 | process. This plugin system handles the communication in between. | |
26 | ||
27 | **Complex arguments and return values are supported.** This library | |
28 | provides APIs for handling complex arguments and return values such | |
29 | as interfaces, `io.Reader/Writer`, etc. We do this by giving you a library | |
30 | (`MuxBroker`) for creating new connections between the client/server to | |
31 | serve additional interfaces or transfer raw data. | |
32 | ||
33 | **Bidirectional communication.** Because the plugin system supports | |
34 | complex arguments, the host process can send it interface implementations | |
35 | and the plugin can call back into the host process. | |
36 | ||
37 | **Built-in Logging.** Any plugins that use the `log` standard library | |
38 | will have log data automatically sent to the host process. The host | |
39 | process will mirror this output prefixed with the path to the plugin | |
40 | binary. This makes debugging with plugins simple. | |
41 | ||
42 | **Protocol Versioning.** A very basic "protocol version" is supported that | |
43 | can be incremented to invalidate any previous plugins. This is useful when | |
44 | interface signatures are changing, protocol level changes are necessary, | |
45 | etc. When a protocol version is incompatible, a human friendly error | |
46 | message is shown to the end user. | |
47 | ||
48 | **Stdout/Stderr Syncing.** While plugins are subprocesses, they can continue | |
49 | to use stdout/stderr as usual and the output will get mirrored back to | |
50 | the host process. The host process can control what `io.Writer` these | |
51 | streams go to to prevent this from happening. | |
52 | ||
53 | **TTY Preservation.** Plugin subprocesses are connected to the identical | |
54 | stdin file descriptor as the host process, allowing software that requires | |
55 | a TTY to work. For example, a plugin can execute `ssh` and even though there | |
56 | are multiple subprocesses and RPC happening, it will look and act perfectly | |
57 | to the end user. | |
58 | ||
59 | **Host upgrade while a plugin is running.** Plugins can be "reattached" | |
60 | so that the host process can be upgraded while the plugin is still running. | |
61 | This requires the host/plugin to know this is possible and daemonize | |
62 | properly. `NewClient` takes a `ReattachConfig` to determine if and how to | |
63 | reattach. | |
64 | ||
65 | ## Architecture | |
66 | ||
67 | The HashiCorp plugin system works by launching subprocesses and communicating | |
68 | over RPC (using standard `net/rpc`). A single connection is made between | |
69 | any plugin and the host process, and we use a | |
70 | [connection multiplexing](https://github.com/hashicorp/yamux) | |
71 | library to multiplex any other connections on top. | |
72 | ||
73 | This architecture has a number of benefits: | |
74 | ||
75 | * Plugins can't crash your host process: A panic in a plugin doesn't | |
76 | panic the plugin user. | |
77 | ||
78 | * Plugins are very easy to write: just write a Go application and `go build`. | |
79 | Theoretically you could also use another language as long as it can | |
80 | communicate the Go `net/rpc` protocol but this hasn't yet been tried. | |
81 | ||
82 | * Plugins are very easy to install: just put the binary in a location where | |
83 | the host will find it (depends on the host but this library also provides | |
84 | helpers), and the plugin host handles the rest. | |
85 | ||
86 | * Plugins can be relatively secure: The plugin only has access to the | |
87 | interfaces and args given to it, not to the entire memory space of the | |
88 | process. More security features are planned (see the coming soon section | |
89 | below). | |
90 | ||
91 | ## Usage | |
92 | ||
93 | To use the plugin system, you must take the following steps. These are | |
94 | high-level steps that must be done. Examples are available in the | |
95 | `examples/` directory. | |
96 | ||
97 | 1. Choose the interface(s) you want to expose for plugins. | |
98 | ||
99 | 2. For each interface, implement an implementation of that interface | |
100 | that communicates over an `*rpc.Client` (from the standard `net/rpc` | |
101 | package) for every function call. Likewise, implement the RPC server | |
102 | struct this communicates to which is then communicating to a real, | |
103 | concrete implementation. | |
104 | ||
105 | 3. Create a `Plugin` implementation that knows how to create the RPC | |
106 | client/server for a given plugin type. | |
107 | ||
108 | 4. Plugin authors call `plugin.Serve` to serve a plugin from the | |
109 | `main` function. | |
110 | ||
111 | 5. Plugin users use `plugin.Client` to launch a subprocess and request | |
112 | an interface implementation over RPC. | |
113 | ||
114 | That's it! In practice, step 2 is the most tedious and time consuming step. | |
115 | Even so, it isn't very difficult and you can see examples in the `examples/` | |
116 | directory as well as throughout our various open source projects. | |
117 | ||
118 | For complete API documentation, see [GoDoc](https://godoc.org/github.com/hashicorp/go-plugin). | |
119 | ||
120 | ## Roadmap | |
121 | ||
122 | Our plugin system is constantly evolving. As we use the plugin system for | |
123 | new projects or for new features in existing projects, we constantly find | |
124 | improvements we can make. | |
125 | ||
126 | At this point in time, the roadmap for the plugin system is: | |
127 | ||
128 | **Cryptographically Secure Plugins.** We'll implement signing plugins | |
129 | and loading signed plugins in order to allow Vault to make use of multi-process | |
130 | in a secure way. | |
131 | ||
132 | **Semantic Versioning.** Plugins will be able to implement a semantic version. | |
133 | This plugin system will give host processes a system for constraining | |
134 | versions. This is in addition to the protocol versioning already present | |
135 | which is more for larger underlying changes. | |
136 | ||
137 | **Plugin fetching.** We will integrate with [go-getter](https://github.com/hashicorp/go-getter) | |
138 | to support automatic download + install of plugins. Paired with cryptographically | |
139 | secure plugins (above), we can make this a safe operation for an amazing | |
140 | user experience. | |
141 | ||
142 | ## What About Shared Libraries? | |
143 | ||
144 | When we started using plugins (late 2012, early 2013), plugins over RPC | |
145 | were the only option since Go didn't support dynamic library loading. Today, | |
146 | Go still doesn't support dynamic library loading, but they do intend to. | |
147 | Since 2012, our plugin system has stabilized from millions of users using it, | |
148 | and has many benefits we've come to value greatly. | |
149 | ||
150 | For example, we intend to use this plugin system in | |
151 | [Vault](https://www.vaultproject.io), and dynamic library loading will | |
152 | simply never be acceptable in Vault for security reasons. That is an extreme | |
153 | example, but we believe our library system has more upsides than downsides | |
154 | over dynamic library loading and since we've had it built and tested for years, | |
155 | we'll likely continue to use it. | |
156 | ||
157 | Shared libraries have one major advantage over our system which is much | |
158 | higher performance. In real world scenarios across our various tools, | |
159 | we've never required any more performance out of our plugin system and it | |
160 | has seen very high throughput, so this isn't a concern for us at the moment. | |
161 |