Skip to content

HDDS-13063. Support splitting json output to multiple valid json files in a directory#10604

Open
prathmesh12-coder wants to merge 1 commit into
apache:masterfrom
prathmesh12-coder:HDDS-13063
Open

HDDS-13063. Support splitting json output to multiple valid json files in a directory#10604
prathmesh12-coder wants to merge 1 commit into
apache:masterfrom
prathmesh12-coder:HDDS-13063

Conversation

@prathmesh12-coder

Copy link
Copy Markdown

What changes were proposed in this pull request?

ozone debug replicas verify currently prints all verification results as one large JSON to stdout.
This PR adds support for writing output to a directory as multiple valid JSON files, similar to ozone debug ldb scan.

below options are added:

  • --out / -o: output directory. If it does not exist, it is created. If it already exists, it is reused.
  • --max-records-per-file: maximum number of keys per file.
    When set, output is split into files like dirName.0, dirName.1 etc. Each file is valid JSON with a keys array.

If --out is not used, the command still prints JSON to stdout as before.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13063

How was this patch tested?

  • Added integration tests in TestOzoneDebugReplicasVerify:


    • testSplitOutputToNewDirectory — verifies the command creates a non-existent output directory and splits the output across the expected number of valid JSON files, 
with each file respecting --max-records-per-file.
    • testSplitOutputToExistingDirectory — verifies the same behaviour when the output directory already exists.

  • Verified split files are valid JSON, respect --max-records-per-file, and contain all keys.

  • Manually tested in local docker-compose Ozone cluster: 


ozone debug replicas verify <vol>/<bucket> --checksums --all-results -o <dir path> --max-records-per-file <no. of keys per file>


confirmed the directory is created, the output is split into numbered valid JSON files, and the run summary is printed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant