f2cc97a44c
Improve MQTT token registration retry ( #8397 )
2019-10-15 19:39:14 -07:00
d48fd6fde9
Remove unusued params and functions ( #8399 )
2019-10-15 18:35:41 -07:00
b7ee0bbbc9
madmin: Migrate to provable secure channel construction ( #8395 )
...
This commit replaces the currently used channel construction
for en/decrypting config data with a provable secure scheme.
2019-10-15 02:36:04 +05:30
c33bae057f
Fix build issues with event target logger
2019-10-11 19:07:09 -07:00
d7060c4c32
Allow logging targets to be configured to receive minio ( #8347 )
...
specific errors, `application` errors or `all` by default.
console logging on server by default lists all logs -
enhance admin console API to accept `type` as query parameter to
subscribe to application/minio logs.
2019-10-11 18:50:54 -07:00
bd10640846
Fix logger and audit http endpoint config lookup
2019-10-11 18:33:25 -07:00
175b07d6e4
Fix queueStore stops working with concurrent PUT/DELETE requests ( #8381 )
...
- This PR fixes situation to avoid underflow, this is possible
because of disconnected operations in replay/sendEvents
- Hold right locks if Del() operation is performed in Get()
- Remove panic in the code and use loggerOnce
- Remove Timer and instead use Ticker instead for proper ticks
2019-10-11 17:46:03 -07:00
f16df2a4e7
Avoid crash in PopulatePublicKey() by re-initializing the args ( #8388 )
...
This is to avoid nil pointer dereference when method by pointer
reference and method by value reference are implemented.
Fixes #8387
2019-10-12 03:37:08 +05:30
290ad0996f
Move etcd, logger, crypto into their own packages ( #8366 )
...
- Deprecates _MINIO_PROFILER, `mc admin profile` does the job
- Move ENVs to common location in cmd/config/
2019-10-08 11:17:56 +05:30
3b8adf7528
Move storageclass config handling into cmd/config/storageclass ( #8360 )
...
Continuation of the changes done in PR #8351 to refactor,
add tests and move global handling into a more idiomatic
style for Go as packages.
2019-10-07 11:20:24 +05:30
002ac82631
S3 Select: Add parser support for lists. ( #8329 )
2019-10-06 07:52:45 -07:00
589e32a4ed
Refactor config and split them in packages ( #8351 )
...
This change is related to larger config migration PR
change, this is a first stage change to move our
configs to `cmd/config/` - divided into its subsystems
2019-10-04 23:05:33 +05:30
74008446fe
CPU hardware info ( #8187 )
2019-10-03 20:18:38 +05:30
fb1374f2f7
Rename iam/validator -> iam/openid and add tests ( #8340 )
...
Refactor as part of config migration
2019-10-02 03:37:20 +05:30
c1a17c2561
S3 Select: Aggregate AVG/SUM as float ( #8326 )
...
Force sum/average to be calculated as a float.
As noted in #8221
> run SELECT AVG(CAST (Score as int)) FROM S3Object on
```
Name,Score
alice,80
bob,81
```
> AWS S3 gives 80.5 and MinIO gives 80.
This also makes overflows much more unlikely.
2019-09-27 16:12:03 -07:00
1c5b05c130
S3 select: Fix output conversion on select * ( #8303 )
...
Fixes #8268
2019-09-27 12:33:14 -07:00
2a2ff96ee1
change ReadPerf into ReadThroughput in NetPerfInfo. ( #8316 )
...
Previously `ReadPerf` was in time.Duration is changed to `ReadThroughput` in uint64.
2019-09-27 00:01:18 +05:30
fd53057654
Add InfoCannedPolicy API to fetch only necessary policy ( #8307 )
...
This PR adds
- InfoCannedPolicy() API for efficiency in fetching policies
- Send group memberships for LDAPUser if available
2019-09-26 23:53:13 +05:30
be313f1758
S3 Select: Workaround java buffer size ( #8312 )
...
Updates #7475
The Java implementation has a 128KB buffer and a message must be emitted before that is used. #7475 therefore limits the message size to 128KB. But up to 256 bytes are written to the buffer in each call. This means we must emit a message before shorter than 128KB.
Therefore we change the limit to 128KB minus 256 bytes.
2019-09-26 04:56:20 +05:30
2e02e1889b
Cleanup ResponseWriter function for audit and trace ( #8283 )
2019-09-24 02:04:28 +05:30
4925bc3e80
log server startup messages to admin console api ( #8264 )
2019-09-22 13:54:32 +05:30
520552ffa9
S3 select: flush when reaching limit ( #8279 )
...
Add missing flush when reaching select limit.
2019-09-20 11:00:17 -07:00
dc0dce9beb
Check for connection errors during event transit in elasticsearch and webhook ( #8273 )
2019-09-19 08:23:43 -07:00
456ce4cc92
Add rootCAs support to Kafka & MQTT ( #8236 )
...
Fixes #8211
2019-09-18 23:43:04 +05:30
dac1cf5a9a
S3 Select: Parsing tweaks ( #8261 )
...
* Don't output empty lines.
* Trim whitespace from byte to int/float/bool conversions.
2019-09-17 17:21:23 -07:00
c9b8bd8de2
S3 Select: optimize output ( #8238 )
...
Queue output items and reuse them.
Remove the unneeded type system in sql and just use the Go type system.
In best case this is more than an order of magnitude speedup:
```
BenchmarkSelectAll_1M-12 1 1841049400 ns/op 274299728 B/op 4198522 allocs/op
BenchmarkSelectAll_1M-12 14 84833400 ns/op 169228346 B/op 3146541 allocs/op
```
2019-09-17 05:56:27 +05:30
017456df63
Wait clearing the close channel ( #8250 )
...
Close channel should not be nilled before goroutines have exited.
Fixes potential hang on closing.
2019-09-16 16:18:01 -07:00
9ac12cf898
Remove unusued Set/GetConfigKeys API ( #8235 )
2019-09-13 16:34:34 -07:00
ddea0bdf11
Concurrent CSV parsing and reduce S3 select allocations ( #8200 )
...
```
CSV parsing, BEFORE:
BenchmarkReaderBasic-12 2842 407533 ns/op 397860 B/op 957 allocs/op
BenchmarkReaderReplace-12 2718 429914 ns/op 397844 B/op 957 allocs/op
BenchmarkReaderReplaceTwo-12 2718 435556 ns/op 397855 B/op 957 allocs/op
BenchmarkAggregateCount_100K-12 171 6798974 ns/op 16667102 B/op 308077 allocs/op
BenchmarkAggregateCount_1M-12 19 65657411 ns/op 168057743 B/op 3146610 allocs/op
BenchmarkSelectAll_10M-12 1 20882119900 ns/op 2758799896 B/op 41978762 allocs/op
CSV parsing, AFTER:
BenchmarkReaderBasic-12 3721 312549 ns/op 101920 B/op 338 allocs/op
BenchmarkReaderReplace-12 3776 318810 ns/op 101993 B/op 340 allocs/op
BenchmarkReaderReplaceTwo-12 3610 330967 ns/op 102012 B/op 341 allocs/op
BenchmarkAggregateCount_100K-12 295 4149588 ns/op 3553623 B/op 103261 allocs/op
BenchmarkAggregateCount_1M-12 30 37746503 ns/op 33827931 B/op 1049435 allocs/op
BenchmarkSelectAll_10M-12 1 17608495800 ns/op 1416504040 B/op 21007082 allocs/op
~ benchcmp old.txt new.txt
benchmark old ns/op new ns/op delta
BenchmarkReaderBasic-12 407533 312549 -23.31%
BenchmarkReaderReplace-12 429914 318810 -25.84%
BenchmarkReaderReplaceTwo-12 435556 330967 -24.01%
BenchmarkAggregateCount_100K-12 6798974 4149588 -38.97%
BenchmarkAggregateCount_1M-12 65657411 37746503 -42.51%
BenchmarkSelectAll_10M-12 20882119900 17608495800 -15.68%
benchmark old allocs new allocs delta
BenchmarkReaderBasic-12 957 338 -64.68%
BenchmarkReaderReplace-12 957 340 -64.47%
BenchmarkReaderReplaceTwo-12 957 341 -64.37%
BenchmarkAggregateCount_100K-12 308077 103261 -66.48%
BenchmarkAggregateCount_1M-12 3146610 1049435 -66.65%
BenchmarkSelectAll_10M-12 41978762 21007082 -49.96%
benchmark old bytes new bytes delta
BenchmarkReaderBasic-12 397860 101920 -74.38%
BenchmarkReaderReplace-12 397844 101993 -74.36%
BenchmarkReaderReplaceTwo-12 397855 102012 -74.36%
BenchmarkAggregateCount_100K-12 16667102 3553623 -78.68%
BenchmarkAggregateCount_1M-12 168057743 33827931 -79.87%
BenchmarkSelectAll_10M-12 2758799896 1416504040 -48.66%
```
```
BenchmarkReaderHuge/97K-12 2200 540840 ns/op 184.32 MB/s 1604450 B/op 687 allocs/op
BenchmarkReaderHuge/194K-12 1522 752257 ns/op 265.04 MB/s 2143135 B/op 1335 allocs/op
BenchmarkReaderHuge/389K-12 1190 947858 ns/op 420.69 MB/s 3221831 B/op 2630 allocs/op
BenchmarkReaderHuge/778K-12 806 1472486 ns/op 541.61 MB/s 5201856 B/op 5187 allocs/op
BenchmarkReaderHuge/1557K-12 426 2575269 ns/op 619.36 MB/s 9101330 B/op 10233 allocs/op
BenchmarkReaderHuge/3115K-12 286 4034656 ns/op 790.66 MB/s 12397968 B/op 16099 allocs/op
BenchmarkReaderHuge/6230K-12 172 6830563 ns/op 934.05 MB/s 16008416 B/op 26844 allocs/op
BenchmarkReaderHuge/12461K-12 100 11409467 ns/op 1118.39 MB/s 22655163 B/op 48107 allocs/op
BenchmarkReaderHuge/24922K-12 66 19780395 ns/op 1290.19 MB/s 35158559 B/op 90216 allocs/op
BenchmarkReaderHuge/49844K-12 34 37282559 ns/op 1369.03 MB/s 60528624 B/op 174497 allocs/op
```
2019-09-13 14:18:35 -07:00
8700945cdf
Handle connection failures on webhook/url pings ( #8204 )
...
Properly handle connection failures while replaying events
Fixes #8194
2019-09-12 16:44:51 -07:00
6ba323b009
Add ability to test drive speeds on a MinIO setup ( #7664 )
...
- Extends existing Admin API to measure disk performance
2019-09-13 03:22:30 +05:30
53e4887e02
Simplify and cleanup metadata r/w functions ( #8146 )
2019-09-11 22:52:12 +05:30
a7be313230
Start using new errors package ( #8207 )
2019-09-11 22:51:43 +05:30
18fedc67d5
friendly prompt for s3select MalformedXML error ( #8171 )
...
partly fix #7911
2019-09-09 21:33:27 -07:00
b52a3e523c
Avoid using fastjson parser pool, move back to jsoniter ( #8190 )
...
It looks like from implementation point of view fastjson
parser pool doesn't behave the same way as expected
when dealing many `xl.json` from multiple disks.
The fastjson parser pool usage ends up returning incorrect
xl.json entries for checksums, with references pointing
to older entries. This led to the subtle bug where checksum
info is duplicated from a previous xl.json read of a different
file from different disk.
2019-09-06 04:21:27 +05:30
428836d4e1
Ignore "connection reset" errors while initializing the targets with queue store enabled ( #8185 )
...
Fixes #8178
2019-09-05 13:55:48 -07:00
810a44e951
KMS Admin-API: add route and handler for KMS key info ( #7955 )
...
This commit adds an admin API route and handler for
requesting status information about a KMS key.
Therefore, the client specifies the KMS key ID (when
empty / not set the server takes the currently configured
default key-ID) and the server tries to perform a dummy encryption,
re-wrap and decryption operation. If all three succeed we know that
the server can access the KMS and has permissions to generate, re-wrap
and decrypt data keys (policy is set correctly).
2019-09-05 01:49:44 +05:30
8a71b0ec5a
Add admin API to send console log messages ( #7784 )
...
Utilized by mc admin console command.
2019-09-03 23:40:48 +05:30
ec9bfd3aef
speed up the performance of s3select on csv ( #7945 )
2019-08-31 00:07:40 -07:00
fa3546bb03
Add NetPerfInfo() API in madmin ( #8112 )
2019-08-31 08:27:53 +05:30
83d4c5763c
Decouple ServiceUpdate to ServerUpdate to be more native ( #8138 )
...
The change now is to ensure that we take custom URL as
well for updating the deployment, this is required for
hotfix deliveries for certain deployments - other than
the community release.
This commit changes the previous work d65a2c6725
2019-08-28 15:04:43 -07:00
c6f86d35d3
Rectify Examples and README.MD ( #8141 )
2019-08-28 10:56:02 -07:00
d65a2c6725
Implement cluster-wide in-place updates ( #8070 )
...
This PR is a breaking change and also deprecates
`minio update` command, from this release onwards
all users are advised to just use `mc admin update`
2019-08-27 11:37:47 -07:00
c38ada1a26
write() to disk in 4MB blocks for better performance ( #7888 )
2019-08-23 15:36:46 -07:00
7bf1caa0fe
Fix broken link to go install docs ( #8090 )
2019-08-15 16:00:50 -07:00
bf9b619d86
Set the policy mapping for a user or group ( #8036 )
...
Add API to set policy mapping for a user or group
Contains a breaking Admin APIs change.
- Also enforce all applicable policies
- Removes the previous /set-user-policy API
Bump up peerRESTVersion
Add get user info API to show groups of a user
2019-08-13 13:41:06 -07:00
0ed6daab59
fix : #8051 so that stale DNS entries are cleaned-up ( #8053 )
2019-08-13 08:49:26 -07:00
e96f19c867
Use default ports for the schemes when dialing ( #7960 )
...
Fixes #7946
2019-08-13 16:43:11 +05:30
aaef18b1a3
Fail for disks which overflow upon usage calculation ( #8056 )
...
Fixes #8035
2019-08-13 02:58:43 -07:00
1ce8d2c476
Add bucket lifecycle expiry feature ( #7834 )
2019-08-09 10:02:41 -07:00
Harshavardhana