Commit Graph

12279 Commits

Author SHA1 Message Date
Ramon de Klein e5b18df6db Fix checksum error during startup when minio is loaded via PATH environment variable (#20509) 2024-10-01 15:13:18 -07:00
Anis Eleuch 0abfd1bcb1 heal: Use etag as quorum when none found for modtime (#20500) 2024-10-01 08:19:10 -07:00
Harshavardhana 6186d11761 handle the locks properly for multi-pool callers (#20495)
- PutObjectMetadata()
- PutObjectTags()
- DeleteObjectTags()
- TransitionObject()
- RestoreTransitionObject()

Also improve the behavior of multipart code across
pool locks, hold locks only once per upload ID for

- CompleteMultipartUpload()
- AbortMultipartUpload()
- ListObjectParts() (read-lock)
- GetMultipartInfo() (read-lock)
- PutObjectPart() (read-lock)

This avoids lock attempts across pools for no
reason, this increases O(n) when there are n-pools.
2024-09-29 15:40:36 -07:00
Poorna e8b457e8a6 Change delete marker proxy test to use distributed setup (#20494) 2024-09-27 18:02:26 -07:00
Harshavardhana afea40cc0f fix: keep locks based on the first pool, first EC set (#93)
multi-object deletion may or may not compete with locks
granted for other callers, causing concurrent operations
to succeed on each other.

A continuation of the PR https://github.com/minio/minio/pull/20356
2024-09-27 03:41:37 -07:00
Aditya Manthramurthy 402b798f1b fix: allow all console actions with custom authZ (#20489)
When custom authorization via plugin is enabled, the console will now
render the UI as if all actions are allowed. Since server cannot
determine the exact policy allowed for a user via the plugin, this is
acceptable to do. If a particular action is actually not allowed by the
plugin the call will result in an error.

Previously the server was evaluating a policy when custom authZ is
enabled - this is fixed now.
2024-09-26 23:44:44 -07:00
Klaus Post 4759532e90 Fix PPC cgroup memory limit (#20488)
The "unlimited" value on PPC wasn't exactly the same as amd64.

Instead compare against an "unreasonably big value".

Would cause OOM in anything using the concurrent request limit.
2024-09-26 10:07:10 -07:00
Harshavardhana 7f1e1713ab use absolute path for binary checksum verification (#20487) 2024-09-26 08:03:08 -07:00
Poorna b2c5819dbc hold on to batch job stats till cleanup (#20480)
This PR also fixes job stats not available after restart
2024-09-24 14:50:11 -07:00
Anis Eleuch 2b0156b1fc Add TTFB to all APIs and enable for responses without body (#20479)
Add TTFB for all requests in metrics-v3 in addition to the existing
GetObject. Also for the requests that do not return a body in the
response, calculate TTFB as the HTTP status code and the headers are
sent.
2024-09-24 10:13:00 -07:00
Harshavardhana f6f0807c86 cleanup existing part.N's before renamePart() (#20466)
this is a safety-net to avoid any unexpected parts to show up.
2024-09-24 04:26:41 -07:00
Harshavardhana 0c53d86017 remove the list from 'mc stat' from testing via '--no-list' (#20468)
avoid 'listing' as it may get incorrect results for
these tests, we are only interested in 'mc stat' as
in HEAD object here.
2024-09-24 01:03:58 -07:00
Minio Trusted 6a6ee46d76 Update yaml files to latest version RELEASE.2024-09-22T00-33-43Z 2024-09-23 19:42:46 +00:00
Klaus Post 974cbb3bb7 Limit jstream parse depth (#20474)
Add https://github.com/bcicen/jstream/pull/15 by vendoring the package.

Sets JSON depth limit to 100 entries in S3 Select.
2024-09-23 12:35:41 -07:00
Harshavardhana 03e996320e upgrade deps pkg/v3, madmin-go/v3 and lz4/v4 (#20467) RELEASE.2024-09-22T00-33-43Z 2024-09-21 17:33:43 -07:00
Taran Pelkey 78fcb76294 Add ListAccessKeysBulk API for builtin user access keys (#20381) 2024-09-21 04:35:40 -07:00
Ramon de Klein 3d152015eb Use MinIO console v1.7.1 (#20465) 2024-09-20 18:18:54 -07:00
jiuker ade8925155 fix: add default http timeout for audit webhook (#20460) 2024-09-20 09:02:50 -07:00
Klaus Post 05a6c170bf Fix PutObject Trailing checksum (#20456)
PutObject would verify trailing checksums, but not store them.

Fixes #20455
2024-09-19 05:59:07 -07:00
Ramon de Klein e1c2344591 Log an error when calculating the binary checksum failed (#20454) 2024-09-18 20:48:32 -07:00
Ramon de Klein 48a591e9b4 Ensure proper stale_uploads_cleanup_interval is used at all times (#20451) 2024-09-18 10:59:26 -07:00
Anis Eleuch fa5d9c02ef batch: Set a default retry attempts and a prefix (#20452)
A batch job will fail if the retry attempt is not provided. The reason
is that the code mistakenly gets the retry attempts from the job status
rather than the job yaml file.

This will also set a default empty prefix for batch expiration.

Also this will avoid trimming the prefix since the yaml decoder already
does that if no quotes were provided, and we should not trim if quotes
were provided and the user provided a leading or a trailing space.
2024-09-18 10:59:03 -07:00
Shubhendu 5bd27346ac Added iam import tests for openid (#20432)
Tests if imported service accounts have 
required access to buckets and objects.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Co-authored-by: Harshavardhana <harsha@minio.io>
2024-09-17 09:45:46 -07:00
Taran Pelkey 3c82cf9327 Fix behavior of AddServiceAccountLDAP for non-admin users (#20442) 2024-09-16 16:04:51 -07:00
Harshavardhana 70d40083e9 remove windows CI/CD for now (#20441)
windows has decided to be a community support
only and source compile-friendly.
2024-09-16 13:46:53 -07:00
Klaus Post 8a30967542 Limit S3 Select JSON documents to 10MB (#20439)
Closes #20430

Limit allocations from badly formed documents.
2024-09-16 09:59:03 -07:00
Harshavardhana 229f04ab79 fix: the permissions one more time on /usr/bin
fixes https://github.com/minio/operator/issues/2319
2024-09-15 16:10:23 -07:00
Minio Trusted 1123dc3676 Update yaml files to latest version RELEASE.2024-09-13T20-26-02Z 2024-09-14 13:18:19 +00:00
Harshavardhana 5bf41aff17 hold granular locking for multi-pool PutObject() (#20434)
- PutObject() for multi-pooled was holding large
  region locks, which was not necessary. This affects
  almost all slowpoke clients and lengthy uploads.

- Re-arrange locks for CompleteMultipart, PutObject
  to be close to rename()
RELEASE.2024-09-13T20-26-02Z
2024-09-13 13:26:02 -07:00
Anis Eleuch e47d787adb tier: Add force param to force tiering removal (#20355)
Currently, it is not possible to remove a tier if it is not accessible
or contains some data, add a force flag to make the removal successful
in that case.
2024-09-12 13:44:05 -07:00
Anis Eleuch 398ffb1136 Enable compression with encryption in CopyObject API (#20411)
When the encryption and compression are both enabled, the
the server will avoid compressing the data for no apparent reason

This commit will enable it and update unit tests.
2024-09-12 13:10:44 -07:00
Shubhendu 5862582cd7 IAM import test with missing entities (#20368)
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-09-12 08:59:00 -07:00
fumoboy007 e36b1146d6 Download static cURL into release Docker image for all supported architectures (#20424)
Download static cURL into release Docker image for all supported architectures.

Currently, the static cURL executable is only downloaded for the `amd64` architecture. However, `arm64` and `ppc64le` variants are also [available](https://github.com/moparisthebest/static-curl/releases/tag/v8.7.1).
2024-09-12 08:45:19 -07:00
Harshavardhana c28a4beeb7 multipart support etag and pre-read small objects (#20423) 2024-09-12 05:24:04 -07:00
Sveinn 15ab0808b3 making sure we don't panic if globalReplicationStats have not been set (#20427) 2024-09-12 04:39:51 -07:00
Sveinn 3bae73fb42 Add http_timeout to audit webhook configurations (#20421) 2024-09-11 15:20:42 -07:00
Harshavardhana bc527eceda handle the actualSize() properly for PostUpload() (#20422)
postUpload() incorrectly saves actual size as '-1'
we should save correct size when its possible.

Bonus: fix the PutObjectPart() write locker, instead
of holding a lock before we read the client stream.

We should hold it only when we need to commit the parts.
2024-09-11 11:35:37 -07:00
Anis Eleuch b963f36e1e fix: Add missing grid handler of clearing upload-id from the cache (#20420) 2024-09-11 09:09:13 -07:00
Poorna cdd7512a2e use rename() safety for in-place 'xl.meta' updates (#20414) 2024-09-11 09:08:51 -07:00
Frank Wessels a6d5287310 Reenable SVE support for Graviton 4 (#20410) 2024-09-10 08:35:12 -07:00
Minio Trusted 22822f4151 Update yaml files to latest version RELEASE.2024-09-09T16-59-28Z 2024-09-10 00:47:02 +00:00
Shubhendu 0b7aa6af87 Skip non existent ldap entities while import (#20352)
Dont hard error for nonexisting LDAP entries instead of logging them
report them via `mc`

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
RELEASE.2024-09-09T16-59-28Z
2024-09-09 09:59:28 -07:00
Harshavardhana 8c9ab85cfa Add multipart uploads cache for ListMultipartUploads() (#20407)
this cache will be honored only when `prefix=""` while
performing ListMultipartUploads() operation.

This is mainly to satisfy applications like alluxio
for their underfs implementation and tests.

replaces https://github.com/minio/minio/pull/20181
2024-09-09 09:58:30 -07:00
Klaus Post b1c849bedc Don't send a canceled context to Unlock (#20409)
AFAICT we send a canceled context to unlock (and thereby releaseAll). This will cause network calls to fail.

Instead use background and add 30s timeout.
2024-09-09 08:49:49 -07:00
Harshavardhana fb24bcfee0 fix: set audit/logger webhook retry interval to maximum 1m (#20404) 2024-09-09 02:36:47 -07:00
Harshavardhana 8268c12cfb Add support for audit/logger max retry and retry interval (#20402)
Current implementation retries forever until our
log buffer is full, and we start dropping events.

This PR allows you to set a value until we give
up on existing audit/logger batches to proceed to
process the new ones.

Bonus:
 - do not blow up buffers beyond batchSize value
 - do not leak the ticker if the worker returns
2024-09-08 05:15:09 -07:00
Sveinn 3f39da48ea fix: retries and failed message counter (#20401) 2024-09-07 17:13:57 -07:00
Klaus Post 9d5cdaa2e3 Limit Response Recorder memory (#20399)
Disable body recording for...

* admin inspect
* admin metrics
* profiling download

Also, if the recorded body is > 10MB, drop it.
2024-09-07 12:16:04 -07:00
Taran Pelkey 84e122c5c3 Fix duplicate groups in ListGroups API (#20396) 2024-09-06 17:28:47 -07:00
Praveen raj Mani 261111e728 Kafka notify: support batched commits for queue store (#20377)
The items will be saved per target batch and will
be committed to the queue store when the batch is full

Also, periodically commit the batched items to the queue store
based on configured commit_timeout; default is 30s;

Bonus: compress queue store multi writes
2024-09-06 16:06:30 -07:00