Skip to content

Commit 947db55

Browse files
authored
Merge pull request #205 from sapcc/tls-cert-key
`fromEnv` syntax for Swift options and TLS client auth
2 parents 4199ada + 2279c16 commit 947db55

File tree

5 files changed

+114
-60
lines changed

5 files changed

+114
-60
lines changed

CHANGELOG.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,12 @@
1+
# v2.10.0 (2023-02-28)
2+
3+
New features:
4+
- Add support for specifying TLS client certificate and key file.
5+
- Add `{ fromEnv: ENV_VAR }` syntax support for all Swift options.
6+
7+
Changes:
8+
- Updated all dependencies to their latest versions.
9+
110
# v2.9.1 (2022-05-25)
211

312
Bugfixes:

README.md

Lines changed: 30 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
# swift-http-import
22

33
* [Why this instead of rclone?](#why-this-instead-of-rclone)
4-
* [Do NOT use if...](#do-not-use-if)
4+
* [Do NOT use if\.\.\.](#do-not-use-if)
55
* [Implicit assumptions](#implicit-assumptions)
66
* [Installation](#installation)
77
* [Usage](#usage)
8+
* [Specifying sensitive info as environment variables](#specifying-sensitive-info-as-environment-variables)
89
* [Alternative authentication options](#alternative-authentication-options)
910
* [Source specification](#source-specification)
1011
* [Yum](#yum)
@@ -106,8 +107,13 @@ jobs:
106107
object_prefix: ubuntu-repos
107108
```
108109
109-
The first paragraph contains the authentication parameters for OpenStack's Identity v3 API. Optionally a `region_name`
110-
can be specified, but this is only required if there are multiple regions to choose from.
110+
The first paragraph contains the authentication parameters for
111+
OpenStack's Identity v3 API. Optionally a `region_name` can be specified, but this is only
112+
required if there are multiple regions to choose from. You can also specify the `tls_client_certificate_file` and `tls_client_key_file` for creating a TLS client.
113+
114+
You can use the `fromEnv` special syntax for the `to.container`, `to.object_prefix`, and
115+
the Swift fields (options under the `swift` key).
116+
See [specifying sensitive info as environment variables](#specifying-sensitive-info-as-environment-variables) for more details.
111117

112118
Each sync job contains the source URL as `from.url`, and `to.container` has the target container name, optionally paired with an
113119
object name prefix in the target container. For example, in the case above, the file
@@ -124,6 +130,21 @@ ubuntu-repos/pool/main/p/pam/pam_1.1.8.orig.tar.gz
124130
125131
The order of jobs is significant: Source trees will be scraped in the order indicated by the `jobs` list.
126132
133+
### Specifying sensitive info as environment variables
134+
135+
For some config fields, instead of specifying the value as plain text, you can use the
136+
special `fromEnv` syntax to read the respective value from an exported environment
137+
variable.
138+
139+
For example, instead of specifying the `swift.password` as plain text, you can use the
140+
following syntax to retrieve the password from the `OS_PASSWORD` environment variable:
141+
142+
```yaml
143+
swift:
144+
password: { fromEnv: OS_PASSWORD }
145+
...
146+
```
147+
127148
### Alternative authentication options
128149

129150
Instead of password-based authentication, [application credentials][app-cred] can also be used, for example:
@@ -139,13 +160,6 @@ jobs: ...
139160
140161
[app-cred]: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/application-credentials.html
141162
142-
Instead of providing your secret credentials as plain text in the config file, you can use a special syntax for the
143-
`password` field or the `application_credential_secret` field to read the respective password from an exported
144-
environment variable:
145-
146-
```yaml
147-
password: { fromEnv: ENVIRONMENT_VARIABLE }
148-
```
149163
150164
### Source specification
151165
@@ -254,8 +268,8 @@ Since GitHub's API rate limits the number of requests per IP therefore it is
254268
field. Refer to the [GitHub API docs](https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting)
255269
for more info on its rate limiting. This field is **required** if the repository is hosted on a Github Enterprise instance instead of `github.com`.
256270
Instead of providing your token as plain text in the config file, you can use the
257-
`fromEnv` special syntax for the `jobs[].from.token` field. See [Alternative
258-
authentication options](#alternative-authentication-options) for more details.
271+
`fromEnv` special syntax for the `jobs[].from.token` field. See
272+
[specifying sensitive info as environment variables](#specifying-sensitive-info-as-environment-variables) for more details.
259273

260274
If a repository publishes GitHub releases using different tags, e.g. server components at
261275
`server-x.y.z` and client at `client-x.y.z`, and you only want to get releases whose tag
@@ -301,6 +315,10 @@ jobs:
301315
object_prefix: ubuntu-repos
302316
```
303317

318+
For defining Swift options, you can use the `fromEnv` special syntax for all the fields
319+
under the `from` key instead of specifying these fields as plain text. See
320+
[specifying sensitive info as environment variables](#specifying-sensitive-info-as-environment-variables)
321+
304322
### File selection
305323

306324
#### By name

pkg/objects/config.go

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ import (
2828
"github.com/majewsky/schwift"
2929
yaml "gopkg.in/yaml.v2"
3030

31+
"github.com/sapcc/go-bits/secrets"
3132
"github.com/sapcc/swift-http-import/pkg/util"
3233
)
3334

@@ -74,10 +75,11 @@ func ReadConfiguration(path string) (*Configuration, []error) {
7475
//across all Debian/Yum jobs.
7576
var gpgCacheContainer *schwift.Container
7677
if cfg.GPG.CacheContainerName != nil && *cfg.GPG.CacheContainerName != "" {
78+
cntrName := *cfg.GPG.CacheContainerName
7779
sl := cfg.Swift
78-
sl.ContainerName = *cfg.GPG.CacheContainerName
80+
sl.ContainerName = secrets.FromEnv(cntrName)
7981
sl.ObjectNamePrefix = ""
80-
err := sl.Connect(sl.ContainerName)
82+
err := sl.Connect(cntrName)
8183
if err == nil {
8284
gpgCacheContainer = sl.Container
8385
} else {
@@ -277,7 +279,7 @@ func (cfg JobConfiguration) Compile(name string, swift SwiftLocation) (job *Job,
277279
errors = append(errors, fmt.Errorf("missing value for %s.segmenting.segment_bytes", name))
278280
}
279281
if cfg.Segmenting.ContainerName == "" {
280-
cfg.Segmenting.ContainerName = cfg.Target.ContainerName + "_segments"
282+
cfg.Segmenting.ContainerName = string(cfg.Target.ContainerName) + "_segments"
281283
}
282284
}
283285

pkg/objects/github.go

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -38,11 +38,11 @@ import (
3838

3939
type GithubReleaseSource struct {
4040
// Options from config file.
41-
URLString string `yaml:"url"`
42-
Token secrets.AuthPassword `yaml:"token"`
43-
TagNamePattern string `yaml:"tag_name_pattern"`
44-
IncludeDraft bool `yaml:"include_draft"`
45-
IncludePrerelease bool `yaml:"include_prerelease"`
41+
URLString string `yaml:"url"`
42+
Token secrets.FromEnv `yaml:"token"`
43+
TagNamePattern string `yaml:"tag_name_pattern"`
44+
IncludeDraft bool `yaml:"include_draft"`
45+
IncludePrerelease bool `yaml:"include_prerelease"`
4646

4747
// Compiled configuration.
4848
url *url.URL `yaml:"-"`

pkg/objects/swift.go

Lines changed: 65 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
package objects
2121

2222
import (
23+
"crypto/tls"
2324
"fmt"
2425
"io"
2526
"net/http"
@@ -39,18 +40,20 @@ import (
3940
// SwiftLocation contains all parameters required to establish a Swift connection.
4041
// It implements the Source interface, but is also used on the target side.
4142
type SwiftLocation struct {
42-
AuthURL string `yaml:"auth_url"`
43-
UserName string `yaml:"user_name"`
44-
UserDomainName string `yaml:"user_domain_name"`
45-
ProjectName string `yaml:"project_name"`
46-
ProjectDomainName string `yaml:"project_domain_name"`
47-
Password secrets.AuthPassword `yaml:"password"`
48-
ApplicationCredentialID string `yaml:"application_credential_id"`
49-
ApplicationCredentialName string `yaml:"application_credential_name"`
50-
ApplicationCredentialSecret secrets.AuthPassword `yaml:"application_credential_secret"`
51-
RegionName string `yaml:"region_name"`
52-
ContainerName string `yaml:"container"`
53-
ObjectNamePrefix string `yaml:"object_prefix"`
43+
AuthURL secrets.FromEnv `yaml:"auth_url"`
44+
UserName secrets.FromEnv `yaml:"user_name"`
45+
UserDomainName secrets.FromEnv `yaml:"user_domain_name"`
46+
ProjectName secrets.FromEnv `yaml:"project_name"`
47+
ProjectDomainName secrets.FromEnv `yaml:"project_domain_name"`
48+
Password secrets.FromEnv `yaml:"password"`
49+
ApplicationCredentialID secrets.FromEnv `yaml:"application_credential_id"`
50+
ApplicationCredentialName secrets.FromEnv `yaml:"application_credential_name"`
51+
ApplicationCredentialSecret secrets.FromEnv `yaml:"application_credential_secret"`
52+
TLSClientCertificateFile secrets.FromEnv `yaml:"tls_client_certificate_file"`
53+
TLSClientKeyFile secrets.FromEnv `yaml:"tls_client_key_file"`
54+
RegionName secrets.FromEnv `yaml:"region_name"`
55+
ContainerName secrets.FromEnv `yaml:"container"`
56+
ObjectNamePrefix secrets.FromEnv `yaml:"object_prefix"`
5457
//configuration for Validate()
5558
ValidateIgnoreEmptyContainer bool `yaml:"-"`
5659
//Account and Container is filled by Connect(). Container will be nil if ContainerName is empty.
@@ -63,16 +66,16 @@ type SwiftLocation struct {
6366

6467
func (s SwiftLocation) cacheKey(name string) string {
6568
v := []string{
66-
s.AuthURL,
67-
s.UserName,
68-
s.UserDomainName,
69-
s.ProjectName,
70-
s.ProjectDomainName,
69+
string(s.AuthURL),
70+
string(s.UserName),
71+
string(s.UserDomainName),
72+
string(s.ProjectName),
73+
string(s.ProjectDomainName),
7174
string(s.Password),
72-
s.ApplicationCredentialID,
73-
s.ApplicationCredentialName,
75+
string(s.ApplicationCredentialID),
76+
string(s.ApplicationCredentialName),
7477
string(s.ApplicationCredentialSecret),
75-
s.RegionName,
78+
string(s.RegionName),
7679
}
7780
if logg.ShowDebug {
7881
v = append(v, name)
@@ -88,6 +91,15 @@ func (s *SwiftLocation) Validate(name string) []error {
8891
result = append(result, fmt.Errorf("missing value for %s.auth_url", name))
8992
}
9093

94+
if s.TLSClientCertificateFile != "" || s.TLSClientKeyFile != "" {
95+
if s.TLSClientCertificateFile == "" {
96+
result = append(result, fmt.Errorf("missing value for %s.tls_client_certificate_file", name))
97+
}
98+
if s.TLSClientKeyFile == "" {
99+
result = append(result, fmt.Errorf("missing value for %s.tls_client_key_file", name))
100+
}
101+
}
102+
91103
if s.ApplicationCredentialID != "" || s.ApplicationCredentialName != "" {
92104
//checking application credential requirements
93105
if s.ApplicationCredentialID == "" {
@@ -99,7 +111,7 @@ func (s *SwiftLocation) Validate(name string) []error {
99111
result = append(result, fmt.Errorf("missing value for %s.user_domain_name", name))
100112
}
101113
}
102-
if string(s.ApplicationCredentialSecret) == "" {
114+
if s.ApplicationCredentialSecret == "" {
103115
result = append(result, fmt.Errorf("missing value for %s.application_credential_secret", name))
104116
}
105117
} else {
@@ -124,7 +136,7 @@ func (s *SwiftLocation) Validate(name string) []error {
124136
result = append(result, fmt.Errorf("missing value for %s.container", name))
125137
}
126138

127-
if s.ObjectNamePrefix != "" && !strings.HasSuffix(s.ObjectNamePrefix, "/") {
139+
if s.ObjectNamePrefix != "" && !strings.HasSuffix(string(s.ObjectNamePrefix), "/") {
128140
s.ObjectNamePrefix += "/"
129141
}
130142

@@ -154,16 +166,16 @@ func (s *SwiftLocation) Connect(name string) error {
154166
s.Account = accountCache[key]
155167
if s.Account == nil {
156168
authOptions := gophercloud.AuthOptions{
157-
IdentityEndpoint: s.AuthURL,
158-
Username: s.UserName,
159-
DomainName: s.UserDomainName,
169+
IdentityEndpoint: string(s.AuthURL),
170+
Username: string(s.UserName),
171+
DomainName: string(s.UserDomainName),
160172
Password: string(s.Password),
161-
ApplicationCredentialID: s.ApplicationCredentialID,
162-
ApplicationCredentialName: s.ApplicationCredentialName,
173+
ApplicationCredentialID: string(s.ApplicationCredentialID),
174+
ApplicationCredentialName: string(s.ApplicationCredentialName),
163175
ApplicationCredentialSecret: string(s.ApplicationCredentialSecret),
164176
Scope: &gophercloud.AuthScope{
165-
ProjectName: s.ProjectName,
166-
DomainName: s.ProjectDomainName,
177+
ProjectName: string(s.ProjectName),
178+
DomainName: string(s.ProjectDomainName),
167179
},
168180
AllowReauth: true,
169181
}
@@ -173,9 +185,22 @@ func (s *SwiftLocation) Connect(name string) error {
173185
return fmt.Errorf("cannot create OpenStack client: %s", err.Error())
174186
}
175187

188+
transport := &http.Transport{}
189+
if s.TLSClientCertificateFile != "" && s.TLSClientKeyFile != "" {
190+
cert, err := tls.LoadX509KeyPair(string(s.TLSClientCertificateFile), string(s.TLSClientKeyFile))
191+
if err != nil {
192+
return fmt.Errorf("failed to load x509 key pair: %s", err.Error())
193+
}
194+
transport.TLSClientConfig = &tls.Config{
195+
Certificates: []tls.Certificate{cert},
196+
MinVersion: tls.VersionTLS12,
197+
}
198+
provider.HTTPClient.Transport = transport
199+
}
200+
176201
if logg.ShowDebug {
177202
provider.HTTPClient.Transport = &client.RoundTripper{
178-
Rt: http.DefaultTransport,
203+
Rt: transport,
179204
Logger: &logger{Prefix: name},
180205
}
181206
}
@@ -199,7 +224,7 @@ func (s *SwiftLocation) Connect(name string) error {
199224
}
200225

201226
serviceClient, err := openstack.NewObjectStorageV1(provider, gophercloud.EndpointOpts{
202-
Region: s.RegionName,
227+
Region: string(s.RegionName),
203228
})
204229
if err != nil {
205230
return fmt.Errorf("cannot create Swift client: %s", err.Error())
@@ -220,20 +245,20 @@ func (s *SwiftLocation) Connect(name string) error {
220245
return nil
221246
}
222247
var err error
223-
s.Container, err = s.Account.Container(s.ContainerName).EnsureExists()
248+
s.Container, err = s.Account.Container(string(s.ContainerName)).EnsureExists()
224249
return err
225250
}
226251

227252
// ObjectAtPath returns an Object instance for the object at the given path
228253
// (below the ObjectNamePrefix, if any) in this container.
229254
func (s *SwiftLocation) ObjectAtPath(path string) *schwift.Object {
230255
objectName := strings.TrimPrefix(path, "/")
231-
return s.Container.Object(s.ObjectNamePrefix + objectName)
256+
return s.Container.Object(string(s.ObjectNamePrefix) + objectName)
232257
}
233258

234259
// ListAllFiles implements the Source interface.
235260
func (s *SwiftLocation) ListAllFiles(out chan<- FileSpec) *ListEntriesError {
236-
objectPath := s.ObjectNamePrefix
261+
objectPath := string(s.ObjectNamePrefix)
237262
if objectPath != "" && !strings.HasSuffix(objectPath, "/") {
238263
objectPath += "/"
239264
}
@@ -247,7 +272,7 @@ func (s *SwiftLocation) ListAllFiles(out chan<- FileSpec) *ListEntriesError {
247272
})
248273
if err != nil {
249274
return &ListEntriesError{
250-
Location: s.ContainerName + "/" + objectPath,
275+
Location: string(s.ContainerName) + "/" + objectPath,
251276
Message: "GET failed",
252277
Inner: err,
253278
}
@@ -265,17 +290,17 @@ func (s *SwiftLocation) getFileSpec(info schwift.ObjectInfo) FileSpec {
265290
var f FileSpec
266291
//strip ObjectNamePrefix from the resulting objects
267292
if info.SubDirectory != "" {
268-
f.Path = strings.TrimPrefix(info.SubDirectory, s.ObjectNamePrefix)
293+
f.Path = strings.TrimPrefix(info.SubDirectory, string(s.ObjectNamePrefix))
269294
f.IsDirectory = true
270295
} else {
271-
f.Path = strings.TrimPrefix(info.Object.Name(), s.ObjectNamePrefix)
296+
f.Path = strings.TrimPrefix(info.Object.Name(), string(s.ObjectNamePrefix))
272297
lm := info.LastModified
273298
f.LastModified = &lm
274299

275300
if info.SymlinkTarget != nil && info.SymlinkTarget.Container().IsEqualTo(s.Container) {
276301
targetPath := info.SymlinkTarget.Name()
277-
if strings.HasPrefix(targetPath, s.ObjectNamePrefix) {
278-
f.SymlinkTargetPath = strings.TrimPrefix(targetPath, s.ObjectNamePrefix)
302+
if strings.HasPrefix(targetPath, string(s.ObjectNamePrefix)) {
303+
f.SymlinkTargetPath = strings.TrimPrefix(targetPath, string(s.ObjectNamePrefix))
279304
}
280305
}
281306
}
@@ -323,7 +348,7 @@ func (s *SwiftLocation) GetFile(path string, requestHeaders schwift.ObjectHeader
323348
// The given Matcher is used to find out which files are to be considered as
324349
// belonging to the transfer job in question.
325350
func (s *SwiftLocation) DiscoverExistingFiles(matcher Matcher) error {
326-
prefix := s.ObjectNamePrefix
351+
prefix := string(s.ObjectNamePrefix)
327352
if prefix != "" && !strings.HasSuffix(prefix, "/") {
328353
prefix += "/"
329354
}

0 commit comments

Comments
 (0)