Commit Graph

1161 Commits

Author SHA1 Message Date
Minio Trusted 27d94c64ed Update yaml files to latest version RELEASE.2022-02-16T00-35-27Z 2022-02-16 05:34:56 +00:00
Pierre Kancir d0862ddf86 doc: add console-address on all example (#14307)
--console-address ":9001" is  missing on docker example for regular user.
2022-02-15 09:26:04 -08:00
Harshavardhana f088e8960b docs: turn-on more markdown rules and fix them (#14301) 2022-02-14 08:50:42 -08:00
Anis Elleuch 1f92fc3fc0 Always check for root disks unless MINIO_CI_CD is set (#14232)
The current code considers a pool with all root disks to be as part
of a testing environment even if there are other pools with mounted
disks. This will result to illegitimate writing in root disks.

Fix this by simplifing the logic: require MINIO_CI_CD in order to skip
root disk check.
2022-02-13 15:42:07 -08:00
Minio Trusted f71b114a84 Update yaml files to latest version RELEASE.2022-02-12T00-51-25Z 2022-02-13 19:31:37 +00:00
Harshavardhana e3e0532613 cleanup markdown docs across multiple files (#14296)
enable markdown-linter
2022-02-11 16:51:25 -08:00
Harshavardhana 97b5bf1fb7 update gateway docs to indicate code-freeze 2022-02-10 11:39:41 -08:00
Poorna ed3418c046 Refactor replication resync to be an active process (#14266)
When resync is triggered, walk the bucket namespace and
resync objects that are unreplicated. This PR also adds
an API to report resync progress.
2022-02-10 10:16:52 -08:00
Harshavardhana a2230868e0 remove all stale old docs about 2018 releases 2022-02-10 09:54:27 -08:00
Krishnan Parthasarathi 0ee2933234 Export tier metrics via Prometheus (#13413)
e.g
```
minio_cluster_ilm_transitioned_bytes{server="minio3:9000",tier="S3TIER-1"} 1.36317772e+08
minio_cluster_ilm_transitioned_bytes{server="minio3:9000",tier="S3TIER-2"} 2892
minio_cluster_ilm_transitioned_bytes{server="minio3:9000",tier="STANDARD"}
1.3631488e+08

minio_cluster_ilm_transitioned_objects{server="minio3:9000",tier="S3TIER-1"} 1
minio_cluster_ilm_transitioned_objects{server="minio3:9000",tier="S3TIER-2"} 0
minio_cluster_ilm_transitioned_objects{server="minio3:9000",tier="STANDARD"} 1

minio_cluster_ilm_transitioned_versions{server="minio3:9000",tier="S3TIER-1"} 3
minio_cluster_ilm_transitioned_versions{server="minio3:9000",tier="S3TIER-2"} 2
minio_cluster_ilm_transitioned_versions{server="minio3:9000",tier="STANDARD"} 1
```
2022-02-08 12:45:28 -08:00
Minio Trusted 2480c66857 Update yaml files to latest version RELEASE.2022-02-07T08-17-33Z 2022-02-07 09:19:24 +00:00
Minio Trusted 570670be8c Update yaml files to latest version RELEASE.2022-02-05T04-40-59Z 2022-02-05 18:33:46 +00:00
Minio Trusted 9417fd933e Update yaml files to latest version RELEASE.2022-02-01T18-00-14Z 2022-02-01 23:37:07 +00:00
Harshavardhana 74faed166a Add quota usage as part of prometheus metrics (#14222)
Bonus: pass caller context when needed to all bucket metadata handling calls.
2022-01-31 17:27:43 -08:00
Harshavardhana dbd05d6e82 remove FIFO bucket quota, use ILM expiration instead (#14206) 2022-01-31 11:07:04 -08:00
Minio Trusted c7df5fb119 Update yaml files to latest version RELEASE.2022-01-28T02-28-16Z 2022-01-28 05:12:35 +00:00
Minio Trusted a66071099c Update yaml files to latest version RELEASE.2022-01-27T03-53-02Z 2022-01-27 04:47:45 +00:00
Aditya Manthramurthy 7dfa565d00 Identity LDAP: Allow multiple search base DNs (#14191)
This change allows the MinIO server to lookup users in different directory
sub-trees by allowing specification of multiple search bases separated by
semicolons.
2022-01-26 15:05:59 -08:00
Bingchang Chen affe51cb19 docs: add casdoor as identity provider (#14185) 2022-01-26 09:49:10 -08:00
Minio Trusted f556a72fe2 Update yaml files to latest version RELEASE.2022-01-25T19-56-04Z 2022-01-25 20:42:46 +00:00
Harshavardhana cd7a5cab8a update docs for Decommission 2022-01-25 11:56:04 -08:00
Poorna 295730408b Disallow delete replication for tag based rules (#14167) 2022-01-24 15:22:20 -08:00
Harshavardhana f30afa4956 docs: add decommission docs about pool removal (#14159) 2022-01-24 09:47:06 -08:00
Harshavardhana 8fb4ae916c update decommission docs 2022-01-21 18:34:06 -08:00
Klaus Post 07df9eecda Update xl.meta docs (#14150) 2022-01-21 08:47:46 -08:00
Harshavardhana 9d588319dd support site replication to replicate IAM users,groups (#14128)
- Site replication was missing replicating users,
  groups when an empty site was added.

- Add site replication for groups and users when they
  are disabled and enabled.

- Add support for replicating bucket quota config.
2022-01-19 20:02:24 -08:00
Poorna 54a98773f8 fix: replication of tag removal (#14056)
Currently tag removal leaves replication state as `PENDING` 
because the `HEAD` api returns just a tag count but not the 
actual tags, and this is treated as a no-op
2022-01-10 19:06:10 -08:00
Harshavardhana 76b21de0c6 feat: decommission feature for pools (#14012)
```
λ mc admin decommission start alias/ http://minio{1...2}/data{1...4}
```

```
λ mc admin decommission status alias/
┌─────┬─────────────────────────────────┬──────────────────────────────────┬────────┐
│ ID  │ Pools                           │ Capacity                         │ Status │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Active │
│ 2nd │ http://minio{3...4}/data{1...4} │ 329 GiB (used) / 421 GiB (total) │ Active │
└─────┴─────────────────────────────────┴──────────────────────────────────┴────────┘
```

```
λ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
Progress: ===================> [1GiB/sec] [15%] [4TiB/50TiB]
Time Remaining: 4 hours (started 3 hours ago)
```

```
λ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
ERROR: This pool is not scheduled for decommissioning currently.
```

```
λ mc admin decommission cancel alias/
┌─────┬─────────────────────────────────┬──────────────────────────────────┬──────────┐
│ ID  │ Pools                           │ Capacity                         │ Status   │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Draining │
└─────┴─────────────────────────────────┴──────────────────────────────────┴──────────┘
```

> NOTE: Canceled decommission will not make the pool active again, since we might have
> Potentially partial duplicate content on the other pools, to avoid this scenario be
> very sure to start decommissioning as a planned activity.

```
λ mc admin decommission cancel alias/ http://minio{1...2}/data{1...4}
┌─────┬─────────────────────────────────┬──────────────────────────────────┬────────────────────┐
│ ID  │ Pools                           │ Capacity                         │ Status             │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Draining(Canceled) │
└─────┴─────────────────────────────────┴──────────────────────────────────┴────────────────────┘
```
2022-01-10 09:07:49 -08:00
Minio Trusted f394313fee Update yaml files to latest version RELEASE.2022-01-08T03-11-54Z 2022-01-08 05:53:59 +00:00
Harshavardhana b7c5e45fff heal: isObjectDangling should return false when it cannot decide (#14053)
In a multi-pool setup when disks are coming up, or in a single pool
setup let's say with 100's of erasure sets with a slow network.

It's possible when healing is attempted on `.minio.sys/config`
folder, it can lead to healing unexpectedly deleting some policy
files as dangling due to a mistake in understanding when `isObjectDangling`
is considered to be 'true'.

This issue happened in commit 30135eed86
when we assumed the validMeta with empty ErasureInfo is considered
to be fully dangling. This implementation issue gets exposed when
the server is starting up.

This is most easily seen with multiple-pool setups because of the
disconnected fashion pools that come up. The decision to purge the
object as dangling is taken incorrectly prior to the correct state
being achieved on each pool, when the corresponding drive let's say
returns 'errDiskNotFound', a 'delete' is triggered. At this point,
the 'drive' comes online because this is part of the startup sequence
as drives can come online lazily.

This kind of situation exists because we allow (totalDisks/2) number
of drives to be online when the server is being restarted.

Implementation made an incorrect assumption here leading to policies
getting deleted.

Added tests to capture the implementation requirements.
2022-01-07 19:11:54 -08:00
Aditya Manthramurthy 0a224654c2 fix: progagation of service accounts for site replication (#14054)
- Only non-root-owned service accounts are replicated for now.
- Add integration tests for OIDC with site replication
2022-01-07 17:41:43 -08:00
Minio Trusted e420a1de4d Update yaml files to latest version RELEASE.2022-01-07T01-53-23Z 2022-01-07 05:57:36 +00:00
Aditya Manthramurthy 62dc0f7698 Update site replication docs (#14044)
- Now OpenID external IDP is also supported for site replication.

- IAM users/groups are also replicated
2022-01-06 17:53:23 -08:00
Minio Trusted f4c03e56b8 Update yaml files to latest version RELEASE.2022-01-04T07-41-07Z 2022-01-04 23:01:29 +00:00
Minio Trusted 5dd40b9377 Update yaml files to latest version RELEASE.2022-01-03T18-22-58Z 2022-01-04 00:12:07 +00:00
Harshavardhana a60ac7ca17 fix: audit log to support object names in multipleObjectNames() handler (#14017) 2022-01-03 01:28:52 -08:00
Minio Trusted c9e49f4366 Update yaml files to latest version RELEASE.2021-12-29T06-49-06Z 2021-12-29 07:38:12 +00:00
Yoann Guillerme 3e28af1723 docs: update TLS doc use -keyout instead of -key (#14001) 2021-12-28 12:51:38 -08:00
Minio Trusted 6aa0574a53 Update yaml files to latest version RELEASE.2021-12-27T07-23-18Z 2021-12-27 19:52:13 +00:00
Harshavardhana 7e3a7d7044 add healing for invalid shards by skipping the blocks (#13978)
Built on top of #13945, now we need to simply skip the
shards and its automated.
2021-12-23 23:01:46 -08:00
Harshavardhana 416977436e rename MINIO_CACHE_.._MASTER_KEY to MINIO_CACHE_.._SECRET_KEY
fixes #13975
2021-12-22 12:11:07 -08:00
Harshavardhana 0e3037631f skip inconsistent shards if possible (#13945)
data shards were wrong due to a healing bug
reported in #13803 mainly with unaligned object
sizes.

This PR is an attempt to automatically avoid
these shards, with available information about
the `xl.meta` and actually disk mtime.
2021-12-21 10:08:26 -08:00
Minio Trusted f3602d7d08 Update yaml files to latest version RELEASE.2021-12-20T22-07-16Z 2021-12-20 23:13:48 +00:00
Harshavardhana 0b21734571 update helm chart link 2021-12-20 13:20:59 -08:00
Minio Trusted cfd66ab8c3 Update yaml files to latest version RELEASE.2021-12-18T04-42-33Z 2021-12-19 22:19:30 +00:00
Krishnan Parthasarathi 3ddb501190 Add docs for NewerNoncurrentVersions (#13944) 2021-12-17 19:32:29 -08:00
Harshavardhana 4fa250a6a1 add hash-set debugging currently supports SIPMOD (#13911)
fixes a bug in s3-check-md5 for single part
uploaded multipart objects.
2021-12-15 12:07:15 -08:00
Harshavardhana 88ad742da0 fix: error handling cases in site-replication (#13901)
- Allow proper SRError to be propagated to
  handlers and converted appropriately.

- Make sure to enable object locking on buckets
  when requested in MakeBucketHook.

- When DNSConfig is enabled attempt to delete it
  first before deleting buckets locally.
2021-12-14 14:09:57 -08:00
fpaupier 40dbe243d9 update: keycloak binding documentation (#13894) 2021-12-13 18:23:16 -08:00
Harshavardhana 109c927dad docs/debug: log any corruption and continue debug tool 2021-12-13 17:45:34 -08:00