]>
Commit | Line | Data |
---|---|---|
bae9f6d2 JC |
1 | # go-getter |
2 | ||
3 | [![Build Status](http://img.shields.io/travis/hashicorp/go-getter.svg?style=flat-square)][travis] | |
4 | [![Build status](https://ci.appveyor.com/api/projects/status/ulq3qr43n62croyq/branch/master?svg=true)][appveyor] | |
5 | [![Go Documentation](http://img.shields.io/badge/go-documentation-blue.svg?style=flat-square)][godocs] | |
6 | ||
7 | [travis]: http://travis-ci.org/hashicorp/go-getter | |
8 | [godocs]: http://godoc.org/github.com/hashicorp/go-getter | |
9 | [appveyor]: https://ci.appveyor.com/project/hashicorp/go-getter/branch/master | |
10 | ||
11 | go-getter is a library for Go (golang) for downloading files or directories | |
12 | from various sources using a URL as the primary form of input. | |
13 | ||
14 | The power of this library is being flexible in being able to download | |
15 | from a number of different sources (file paths, Git, HTTP, Mercurial, etc.) | |
16 | using a single string as input. This removes the burden of knowing how to | |
17 | download from a variety of sources from the implementer. | |
18 | ||
19 | The concept of a _detector_ automatically turns invalid URLs into proper | |
20 | URLs. For example: "github.com/hashicorp/go-getter" would turn into a | |
21 | Git URL. Or "./foo" would turn into a file URL. These are extensible. | |
22 | ||
23 | This library is used by [Terraform](https://terraform.io) for | |
24 | downloading modules, [Otto](https://ottoproject.io) for dependencies and | |
25 | Appfile imports, and [Nomad](https://nomadproject.io) for downloading | |
26 | binaries. | |
27 | ||
28 | ## Installation and Usage | |
29 | ||
30 | Package documentation can be found on | |
31 | [GoDoc](http://godoc.org/github.com/hashicorp/go-getter). | |
32 | ||
33 | Installation can be done with a normal `go get`: | |
34 | ||
35 | ``` | |
36 | $ go get github.com/hashicorp/go-getter | |
37 | ``` | |
38 | ||
39 | go-getter also has a command you can use to test URL strings: | |
40 | ||
41 | ``` | |
42 | $ go install github.com/hashicorp/go-getter/cmd/go-getter | |
43 | ... | |
44 | ||
45 | $ go-getter github.com/foo/bar ./foo | |
46 | ... | |
47 | ``` | |
48 | ||
49 | The command is useful for verifying URL structures. | |
50 | ||
51 | ## URL Format | |
52 | ||
53 | go-getter uses a single string URL as input to download from a variety of | |
54 | protocols. go-getter has various "tricks" with this URL to do certain things. | |
55 | This section documents the URL format. | |
56 | ||
57 | ### Supported Protocols and Detectors | |
58 | ||
59 | **Protocols** are used to download files/directories using a specific | |
60 | mechanism. Example protocols are Git and HTTP. | |
61 | ||
62 | **Detectors** are used to transform a valid or invalid URL into another | |
63 | URL if it matches a certain pattern. Example: "github.com/user/repo" is | |
64 | automatically transformed into a fully valid Git URL. This allows go-getter | |
65 | to be very user friendly. | |
66 | ||
67 | go-getter out of the box supports the following protocols. Additional protocols | |
68 | can be augmented at runtime by implementing the `Getter` interface. | |
69 | ||
70 | * Local files | |
71 | * Git | |
72 | * Mercurial | |
73 | * HTTP | |
74 | * Amazon S3 | |
75 | ||
76 | In addition to the above protocols, go-getter has what are called "detectors." | |
77 | These take a URL and attempt to automatically choose the best protocol for | |
78 | it, which might involve even changing the protocol. The following detection | |
79 | is built-in by default: | |
80 | ||
81 | * File paths such as "./foo" are automatically changed to absolute | |
82 | file URLs. | |
83 | * GitHub URLs, such as "github.com/mitchellh/vagrant" are automatically | |
84 | changed to Git protocol over HTTP. | |
85 | * BitBucket URLs, such as "bitbucket.org/mitchellh/vagrant" are automatically | |
86 | changed to a Git or mercurial protocol using the BitBucket API. | |
87 | ||
88 | ### Forced Protocol | |
89 | ||
90 | In some cases, the protocol to use is ambiguous depending on the source | |
91 | URL. For example, "http://github.com/mitchellh/vagrant.git" could reference | |
92 | an HTTP URL or a Git URL. Forced protocol syntax is used to disambiguate this | |
93 | URL. | |
94 | ||
95 | Forced protocol can be done by prefixing the URL with the protocol followed | |
96 | by double colons. For example: `git::http://github.com/mitchellh/vagrant.git` | |
97 | would download the given HTTP URL using the Git protocol. | |
98 | ||
99 | Forced protocols will also override any detectors. | |
100 | ||
101 | In the absense of a forced protocol, detectors may be run on the URL, transforming | |
102 | the protocol anyways. The above example would've used the Git protocol either | |
103 | way since the Git detector would've detected it was a GitHub URL. | |
104 | ||
105 | ### Protocol-Specific Options | |
106 | ||
107 | Each protocol can support protocol-specific options to configure that | |
108 | protocol. For example, the `git` protocol supports specifying a `ref` | |
109 | query parameter that tells it what ref to checkout for that Git | |
110 | repository. | |
111 | ||
112 | The options are specified as query parameters on the URL (or URL-like string) | |
113 | given to go-getter. Using the Git example above, the URL below is a valid | |
114 | input to go-getter: | |
115 | ||
116 | github.com/hashicorp/go-getter?ref=abcd1234 | |
117 | ||
118 | The protocol-specific options are documented below the URL format | |
119 | section. But because they are part of the URL, we point it out here so | |
120 | you know they exist. | |
121 | ||
122 | ### Checksumming | |
123 | ||
124 | For file downloads of any protocol, go-getter can automatically verify | |
125 | a checksum for you. Note that checksumming only works for downloading files, | |
126 | not directories, but checksumming will work for any protocol. | |
127 | ||
128 | To checksum a file, append a `checksum` query parameter to the URL. | |
129 | The paramter value should be in the format of `type:value`, where | |
130 | type is "md5", "sha1", "sha256", or "sha512". The "value" should be | |
131 | the actual checksum value. go-getter will parse out this query parameter | |
132 | automatically and use it to verify the checksum. An example URL | |
133 | is shown below: | |
134 | ||
135 | ``` | |
136 | ./foo.txt?checksum=md5:b7d96c89d09d9e204f5fedc4d5d55b21 | |
137 | ``` | |
138 | ||
139 | The checksum query parameter is never sent to the backend protocol | |
140 | implementation. It is used at a higher level by go-getter itself. | |
141 | ||
142 | ### Unarchiving | |
143 | ||
144 | go-getter will automatically unarchive files into a file or directory | |
145 | based on the extension of the file being requested (over any protocol). | |
146 | This works for both file and directory downloads. | |
147 | ||
148 | go-getter looks for an `archive` query parameter to specify the format of | |
149 | the archive. If this isn't specified, go-getter will use the extension of | |
150 | the path to see if it appears archived. Unarchiving can be explicitly | |
151 | disabled by setting the `archive` query parameter to `false`. | |
152 | ||
153 | The following archive formats are supported: | |
154 | ||
155 | * `tar.gz` and `tgz` | |
156 | * `tar.bz2` and `tbz2` | |
157 | * `zip` | |
158 | * `gz` | |
159 | * `bz2` | |
160 | ||
161 | For example, an example URL is shown below: | |
162 | ||
163 | ``` | |
164 | ./foo.zip | |
165 | ``` | |
166 | ||
167 | This will automatically be inferred to be a ZIP file and will be extracted. | |
168 | You can also be explicit about the archive type: | |
169 | ||
170 | ``` | |
171 | ./some/other/path?archive=zip | |
172 | ``` | |
173 | ||
174 | And finally, you can disable archiving completely: | |
175 | ||
176 | ``` | |
177 | ./some/path?archive=false | |
178 | ``` | |
179 | ||
180 | You can combine unarchiving with the other features of go-getter such | |
181 | as checksumming. The special `archive` query parameter will be removed | |
182 | from the URL before going to the final protocol downloader. | |
183 | ||
184 | ## Protocol-Specific Options | |
185 | ||
186 | This section documents the protocol-specific options that can be specified | |
187 | for go-getter. These options should be appended to the input as normal query | |
188 | parameters. Depending on the usage of go-getter, applications may provide | |
189 | alternate ways of inputting options. For example, [Nomad](https://www.nomadproject.io) | |
190 | provides a nice options block for specifying options rather than in the URL. | |
191 | ||
192 | ## General (All Protocols) | |
193 | ||
194 | The options below are available to all protocols: | |
195 | ||
196 | * `archive` - The archive format to use to unarchive this file, or "" (empty | |
197 | string) to disable unarchiving. For more details, see the complete section | |
198 | on archive support above. | |
199 | ||
200 | * `checksum` - Checksum to verify the downloaded file or archive. See | |
201 | the entire section on checksumming above for format and more details. | |
202 | ||
203 | ### Local Files (`file`) | |
204 | ||
205 | None | |
206 | ||
207 | ### Git (`git`) | |
208 | ||
209 | * `ref` - The Git ref to checkout. This is a ref, so it can point to | |
210 | a commit SHA, a branch name, etc. If it is a named ref such as a branch | |
211 | name, go-getter will update it to the latest on each get. | |
212 | ||
213 | * `sshkey` - An SSH private key to use during clones. The provided key must | |
214 | be a base64-encoded string. For example, to generate a suitable `sshkey` | |
215 | from a private key file on disk, you would run `base64 -w0 <file>`. | |
216 | ||
217 | **Note**: Git 2.3+ is required to use this feature. | |
218 | ||
219 | ### Mercurial (`hg`) | |
220 | ||
221 | * `rev` - The Mercurial revision to checkout. | |
222 | ||
223 | ### HTTP (`http`) | |
224 | ||
225 | None | |
226 | ||
227 | ### S3 (`s3`) | |
228 | ||
229 | S3 takes various access configurations in the URL. Note that it will also | |
230 | read these from standard AWS environment variables if they're set. If | |
231 | the query parameters are present, these take priority. | |
232 | ||
233 | * `aws_access_key_id` - AWS access key. | |
234 | * `aws_access_key_secret` - AWS access key secret. | |
235 | * `aws_access_token` - AWS access token if this is being used. | |
236 | ||
237 | #### Using IAM Instance Profiles with S3 | |
238 | ||
239 | If you use go-getter and want to use an EC2 IAM Instance Profile to avoid | |
240 | using credentials, then just omit these and the profile, if available will | |
241 | be used automatically. | |
242 | ||
243 | #### S3 Bucket Examples | |
244 | ||
245 | S3 has several addressing schemes used to reference your bucket. These are | |
246 | listed here: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro | |
247 | ||
248 | Some examples for these addressing schemes: | |
249 | - s3::https://s3.amazonaws.com/bucket/foo | |
250 | - s3::https://s3-eu-west-1.amazonaws.com/bucket/foo | |
251 | - bucket.s3.amazonaws.com/foo | |
252 | - bucket.s3-eu-west-1.amazonaws.com/foo/bar | |
253 |