vendor/github.com/hashicorp/go-getter/README.md

   1 # go-getter
   2
   3 [![Build Status](http://img.shields.io/travis/hashicorp/go-getter.svg?style=flat-square)][travis]
   4 [![Build status](https://ci.appveyor.com/api/projects/status/ulq3qr43n62croyq/branch/master?svg=true)][appveyor]
   5 [![Go Documentation](http://img.shields.io/badge/go-documentation-blue.svg?style=flat-square)][godocs]
   6
   7 [travis]: http://travis-ci.org/hashicorp/go-getter
   8 [godocs]: http://godoc.org/github.com/hashicorp/go-getter
   9 [appveyor]: https://ci.appveyor.com/project/hashicorp/go-getter/branch/master
  10
  11 go-getter is a library for Go (golang) for downloading files or directories
  12 from various sources using a URL as the primary form of input.
  13
  14 The power of this library is being flexible in being able to download
  15 from a number of different sources (file paths, Git, HTTP, Mercurial, etc.)
  16 using a single string as input. This removes the burden of knowing how to
  17 download from a variety of sources from the implementer.
  18
  19 The concept of a _detector_ automatically turns invalid URLs into proper
  20 URLs. For example: "github.com/hashicorp/go-getter" would turn into a
  21 Git URL. Or "./foo" would turn into a file URL. These are extensible.
  22
  23 This library is used by [Terraform](https://terraform.io) for
  24 downloading modules, [Otto](https://ottoproject.io) for dependencies and
  25 Appfile imports, and [Nomad](https://nomadproject.io) for downloading
  26 binaries.
  27
  28 ## Installation and Usage
  29
  30 Package documentation can be found on
  31 [GoDoc](http://godoc.org/github.com/hashicorp/go-getter).
  32
  33 Installation can be done with a normal `go get`:
  34
  35 ```
  36 $ go get github.com/hashicorp/go-getter
  37 ```
  38
  39 go-getter also has a command you can use to test URL strings:
  40
  41 ```
  42 $ go install github.com/hashicorp/go-getter/cmd/go-getter
  43 ...
  44
  45 $ go-getter github.com/foo/bar ./foo
  46 ...
  47 ```
  48
  49 The command is useful for verifying URL structures.
  50
  51 ## URL Format
  52
  53 go-getter uses a single string URL as input to download from a variety of
  54 protocols. go-getter has various "tricks" with this URL to do certain things.
  55 This section documents the URL format.
  56
  57 ### Supported Protocols and Detectors
  58
  59 **Protocols** are used to download files/directories using a specific
  60 mechanism. Example protocols are Git and HTTP.
  61
  62 **Detectors** are used to transform a valid or invalid URL into another
  63 URL if it matches a certain pattern. Example: "github.com/user/repo" is
  64 automatically transformed into a fully valid Git URL. This allows go-getter
  65 to be very user friendly.
  66
  67 go-getter out of the box supports the following protocols. Additional protocols
  68 can be augmented at runtime by implementing the `Getter` interface.
  69
  70   * Local files
  71   * Git
  72   * Mercurial
  73   * HTTP
  74   * Amazon S3
  75
  76 In addition to the above protocols, go-getter has what are called "detectors."
  77 These take a URL and attempt to automatically choose the best protocol for
  78 it, which might involve even changing the protocol. The following detection
  79 is built-in by default:
  80
  81   * File paths such as "./foo" are automatically changed to absolute
  82     file URLs.
  83   * GitHub URLs, such as "github.com/mitchellh/vagrant" are automatically
  84     changed to Git protocol over HTTP.
  85   * BitBucket URLs, such as "bitbucket.org/mitchellh/vagrant" are automatically
  86     changed to a Git or mercurial protocol using the BitBucket API.
  87
  88 ### Forced Protocol
  89
  90 In some cases, the protocol to use is ambiguous depending on the source
  91 URL. For example, "http://github.com/mitchellh/vagrant.git" could reference
  92 an HTTP URL or a Git URL. Forced protocol syntax is used to disambiguate this
  93 URL.
  94
  95 Forced protocol can be done by prefixing the URL with the protocol followed
  96 by double colons. For example: `git::http://github.com/mitchellh/vagrant.git`
  97 would download the given HTTP URL using the Git protocol.
  98
  99 Forced protocols will also override any detectors.
 100
 101 In the absense of a forced protocol, detectors may be run on the URL, transforming
 102 the protocol anyways. The above example would've used the Git protocol either
 103 way since the Git detector would've detected it was a GitHub URL.
 104
 105 ### Protocol-Specific Options
 106
 107 Each protocol can support protocol-specific options to configure that
 108 protocol. For example, the `git` protocol supports specifying a `ref`
 109 query parameter that tells it what ref to checkout for that Git
 110 repository.
 111
 112 The options are specified as query parameters on the URL (or URL-like string)
 113 given to go-getter. Using the Git example above, the URL below is a valid
 114 input to go-getter:
 115
 116     github.com/hashicorp/go-getter?ref=abcd1234
 117
 118 The protocol-specific options are documented below the URL format
 119 section. But because they are part of the URL, we point it out here so
 120 you know they exist.
 121
 122 ### Checksumming
 123
 124 For file downloads of any protocol, go-getter can automatically verify
 125 a checksum for you. Note that checksumming only works for downloading files,
 126 not directories, but checksumming will work for any protocol.
 127
 128 To checksum a file, append a `checksum` query parameter to the URL.
 129 The paramter value should be in the format of `type:value`, where
 130 type is "md5", "sha1", "sha256", or "sha512". The "value" should be
 131 the actual checksum value. go-getter will parse out this query parameter
 132 automatically and use it to verify the checksum. An example URL
 133 is shown below:
 134
 135 ```
 136 ./foo.txt?checksum=md5:b7d96c89d09d9e204f5fedc4d5d55b21
 137 ```
 138
 139 The checksum query parameter is never sent to the backend protocol
 140 implementation. It is used at a higher level by go-getter itself.
 141
 142 ### Unarchiving
 143
 144 go-getter will automatically unarchive files into a file or directory
 145 based on the extension of the file being requested (over any protocol).
 146 This works for both file and directory downloads.
 147
 148 go-getter looks for an `archive` query parameter to specify the format of
 149 the archive. If this isn't specified, go-getter will use the extension of
 150 the path to see if it appears archived. Unarchiving can be explicitly
 151 disabled by setting the `archive` query parameter to `false`.
 152
 153 The following archive formats are supported:
 154
 155   * `tar.gz` and `tgz`
 156   * `tar.bz2` and `tbz2`
 157   * `zip`
 158   * `gz`
 159   * `bz2`
 160
 161 For example, an example URL is shown below:
 162
 163 ```
 164 ./foo.zip
 165 ```
 166
 167 This will automatically be inferred to be a ZIP file and will be extracted.
 168 You can also be explicit about the archive type:
 169
 170 ```
 171 ./some/other/path?archive=zip
 172 ```
 173
 174 And finally, you can disable archiving completely:
 175
 176 ```
 177 ./some/path?archive=false
 178 ```
 179
 180 You can combine unarchiving with the other features of go-getter such
 181 as checksumming. The special `archive` query parameter will be removed
 182 from the URL before going to the final protocol downloader.
 183
 184 ## Protocol-Specific Options
 185
 186 This section documents the protocol-specific options that can be specified
 187 for go-getter. These options should be appended to the input as normal query
 188 parameters. Depending on the usage of go-getter, applications may provide
 189 alternate ways of inputting options. For example, [Nomad](https://www.nomadproject.io)
 190 provides a nice options block for specifying options rather than in the URL.
 191
 192 ## General (All Protocols)
 193
 194 The options below are available to all protocols:
 195
 196   * `archive` - The archive format to use to unarchive this file, or "" (empty
 197     string) to disable unarchiving. For more details, see the complete section
 198     on archive support above.
 199
 200   * `checksum` - Checksum to verify the downloaded file or archive. See
 201     the entire section on checksumming above for format and more details.
 202
 203 ### Local Files (`file`)
 204
 205 None
 206
 207 ### Git (`git`)
 208
 209   * `ref` - The Git ref to checkout. This is a ref, so it can point to
 210     a commit SHA, a branch name, etc. If it is a named ref such as a branch
 211     name, go-getter will update it to the latest on each get.
 212
 213   * `sshkey` - An SSH private key to use during clones. The provided key must
 214     be a base64-encoded string. For example, to generate a suitable `sshkey`
 215     from a private key file on disk, you would run `base64 -w0 <file>`.
 216
 217     **Note**: Git 2.3+ is required to use this feature.
 218
 219 ### Mercurial (`hg`)
 220
 221   * `rev` - The Mercurial revision to checkout.
 222
 223 ### HTTP (`http`)
 224
 225 None
 226
 227 ### S3 (`s3`)
 228
 229 S3 takes various access configurations in the URL. Note that it will also
 230 read these from standard AWS environment variables if they're set. If
 231 the query parameters are present, these take priority.
 232
 233   * `aws_access_key_id` - AWS access key.
 234   * `aws_access_key_secret` - AWS access key secret.
 235   * `aws_access_token` - AWS access token if this is being used.
 236
 237 #### Using IAM Instance Profiles with S3
 238
 239 If you use go-getter and want to use an EC2 IAM Instance Profile to avoid
 240 using credentials, then just omit these and the profile, if available will
 241 be used automatically.
 242
 243 #### S3 Bucket Examples
 244
 245 S3 has several addressing schemes used to reference your bucket. These are
 246 listed here: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro
 247
 248 Some examples for these addressing schemes:
 249 - s3::https://s3.amazonaws.com/bucket/foo
 250 - s3::https://s3-eu-west-1.amazonaws.com/bucket/foo
 251 - bucket.s3.amazonaws.com/foo
 252 - bucket.s3-eu-west-1.amazonaws.com/foo/bar
 253