-
Notifications
You must be signed in to change notification settings - Fork 792
SOLR-17136: Replace most uses of GenericSolrRequest in Solr code #3955
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
SOLR-17136: Replace most uses of GenericSolrRequest in Solr code #3955
Conversation
SystemInfoResponse
SystemInfoRequest/Response in CLI and package manager
Adjust the API model SystemInfoResponse and tie it into the SolrResponseBase. Remove commented code.
|
I approved the workflows to run. Ping us when you are ready for review! |
add MetricsRequest, more replacements of GenericSolrRequest, use the constants for "/admin/info/system" and "/admin/metrics"
…m:igiguere/solr.git into SOLR-17136-replace-GenericSolrRequest
solr/solrj/src/java/org/apache/solr/client/solrj/request/MetricsRequest.java
Outdated
Show resolved
Hide resolved
solr/solrj/src/java/org/apache/solr/client/solrj/response/SystemInfoResponse.java
Outdated
Show resolved
Hide resolved
|
Hemm. I wanted to run gradlew tidy with my corrections for the comments, but I get this error. My branch is up to date with main. $ ./gradlew tidy FAILURE: Build failed with an exception.
UPDATE: |
epugh
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is super exciting! I didn't quite realize how much duplicative boilerplate using GSR brought.
I think this is mostly in the right direction. A couple of comments and questions. I don't see explicit tests for these new classes, though I guess they are literally excercised everywhere and are just wrapper classes.
| String zkHost = (String) info.get("zkHost"); | ||
| status.put("cloud", getCloudStatus(solrClient, zkHost)); | ||
| if ("solrcloud".equals(sysResponse.getMode())) { | ||
| status.put("cloud", getCloudStatus(solrClient, sysResponse.getZkHost())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LIkewise in a future pr, would be nice to figure out is it cloud or solrcloud and use a single term everywhere ;-)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lovely change!
| coreRootDirectory = (String) systemInfo.get("core_root"); | ||
| SystemInfoResponse sysResponse = (new SystemInfoRequest()).process(solrClient); | ||
| // usually same as solr home, but not always | ||
| String coreRootDirectory = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can core_root ever be null? do we know we need this check?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No idea. Just being cautious. Although, the original code did not check for null, so I guess the new code shouldn't need to.
|
|
||
| // convert raw JSON into user-friendly output | ||
| Map<String, Object> status = StatusTool.reportStatus(systemInfo, solrClient); | ||
| Map<String, Object> status = StatusTool.reportStatus(solrClient); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe for future, but is it odd a CLIUtils would call StatusTool for a method reportStatus? Should it be moved to CLIUtils instead and the StatusTool should call CLIUtils.reportStatus?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are 17 calls to CLIUtils.getZkHost(CommandLine), where we find this call to StatusTool.reportStatus(SolrClient).
That's half the *Tool found in org.apache.solr.cli. That means CLIUtils serves as a "bridge" between tools (StatusTool and the rest of them).
Personally, I think it makes sense that the StatusTool would be the one reporting the status, rather than have it depend on the CLIUtils to essentially do the status reporting job (i.e.: CLIUtils.reportStatus).
Otherwise, if we don't want CLIUtils to call StatusTool, then, we have to add a call to StatusTool.reportStatus(SolrClient) in each of the 17 locations, and for each, find the ZK host currently provided by CLIUtils.getZkHost(CommandLine).
In a nutchell : I would not change this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
makes sense, thanks for digging into this.
| // This method is only called from PackageTool ("add-repo", or "add-key"), where the Solr URL is | ||
| // normalized to remove the /solr path part | ||
| // So might as well ping the V2 API "/node/system" instead. | ||
| // Otherwise, this SystemInfoRequest ctr would need to set the full /solr/admin/info/system path |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"ctr"??? maybe spell it out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
;) it would have wrapped the line, but ok. I'll spell it out.
| // normalized to remove the /solr path part | ||
| // So might as well ping the V2 API "/node/system" instead. | ||
| // Otherwise, this SystemInfoRequest ctr would need to set the full /solr/admin/info/system path | ||
| SystemInfoResponse sysResponse = new SystemInfoRequest("/node/system").process(solrClient); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
humm.... In general we should just use V2 api's! Over time in Solr 10 x we will migrate internal tooling to using V2 apis, which will open the door to deprecate and remove V1 equivalents. If I am understanding here, you are choosing to use V2 in this one use case. Could we just use V2 everywhere....
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SystemInfoRequest could set the V2 path by default.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was going to set the V2 path /node/system as default in SystemInfoRequest constructors, but, that would require some refactoring throughout the CLI tools, at least. The SolrClient used for the system info can be reused for other request that don't all support V2, so, having SystemInfoRequest default to V2 would mean managing the switch from V1 to V2 URLs.
Let's takle that in some other ticket. I'm leaving a "TODO" for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
makes sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another factor in where we use "v2" on the server side is that, currently, users can disable the v2 API with a system property: solr.api.v2.enabled.
I think we're ready to remove that feature-flag, so that we can start assuming v2 will be available to call from our tooling, etc. But while it exists it's probably brittle to have tooling, etc. use v2 endpoints.
Coincidentally, the package-store code here is one of the few places that assuming v2 is "safe". The package-manager only has v2 APIs. So it's safe for us to call other v2 endpoints from the package store code, because if v2 is disabled then all this code is unreachable anyways.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So... package manager class RepositoryManager can call V2 /node/system. No change required?
| params); | ||
| SimpleSolrResponse rsp = null; | ||
| SystemInfoRequest req = new SystemInfoRequest(params); | ||
| SystemInfoResponse rsp = null; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
check your ide, I think in geneal we don't do the = null pattern when declaring an empty variable as it is already null.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It was like that already, but for SimpleSolrResponse. I'll remove the = null.
Update: It was initialized to avoid compile error at line 91, after the try-catch, when rsp is used: "variable rsp might not have been initialized".
So I'll move the following asserts into the try-catch.
|
|
||
| private static final long serialVersionUID = 1L; | ||
|
|
||
| private org.apache.solr.client.api.model.SystemInfoResponse fullResponse; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the import path being full here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Import the API model SystemInfoResponse into the SolrJ client SystemInfoResponse. Maybe I should rename one of these classes, to avoid confusion ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gerlowskija any suggestions on what we should do here? Having the same name in two packages seems not good. SystemInfo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In org.apache.solr.client.api.*:
Most API model classes that represent responses have a name in *Reponse. There's also some *RequestBody. There seems to be some mapping between the class name and the exposed resource path (but not 100%)
So I think the system info response should have a name that ends in *Response. And I like the idea of keeping names aligned ("/whatever" resource = WhateverAPI returns WhateverResponse, and Whatever implements WhateverAPI)
The "home-grown" V2 NodeSystemInfoAPI exposes resource path "/node/system". Presumably, the JAX-RS V2 would still have a class name similar to it's resource path (i.e.: NodeSystemAPI ?) ... Which would make the API model response class NodeSystemResponse ?
That's my 2 cents. @epugh, @gerlowskija
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I can live with NodeSystemAPI and NodeSystemResponse. One thing I hate is having a PR sit forever ont he vine because we can't come up whti a name. We can always rename these thingsin future versions!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just saw that comment. Renamed.
Done. Unless checks fail.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, given more time we could maybe come up with something better than NodeSystemResponse. But better to get this merged in and we can iterate on it going forward...
| } | ||
|
|
||
| // ************************* | ||
| // Shortcuts |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this a pattern in other similar classes? Mostly my own curiosity!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've seen some similar "separator" comments in other classes, can't remember which right now. I can remove it.
OK, no, you meant the methods themselves. Ref. CoreAdminResponse providing similar methods : getStartTime, getUpTime. Also DocumentAnalysissResponse. If the response is complex enough, some parsing should be provided so the rest of the code doesn't need to know the inner structure of the response.
|
@gerlowskija can you review from the v2 api perspective? If @igiguere is intered, I'd love to take what she's learned on making V2 api for metrics and system request handler and look at osme of our other V2 api gaps! |
FYI : System Info already had a V2 implementation (/node/system), I didn't touch it. It just calls the SystemInfoHandler, same as V1. I didn't see a V2 resource for metrics. I suppose it could also call the same handler as V1. |
|
I just approved the workflows... Assuming the tests all pass, I'l leave it open till Monday to give folks a chance to review and then merge. This looks great... What's next? |
I'm running one more
What's next? For me, in the next few days: get over my cold that's dragging on, find a plumber to fix the clogged drain, Christmas... After that, more job hunting, and possibly tackle some of the missing V2 APIs. |
|
Pre commit failed. The crave tests have been failing |
|
Sigh. I ran |
|
update from |
|
actually, I just pushed up the updates from |
|
TEsts passed! I will leave this open for mOnday, and then will merge. Please tag me on your next PR, enjoyed working with you on this. |
Thanks for the fix, Eric! |
gerlowskija
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This LGTM; awesome work @igiguere !
I left a few comments that it'd be nice to address, but most were relatively minor (or just questions).
The one slightly larger question worth highlighting is a suggestion that we drop the v2 support from SystemInfoRequest. Having the v2 coverage there is great to have in isolation, but this PR actually brings us really really close to getting us /node/system coverage via the code-generation route we've been taking with the other v2 APIs.
If we omit v2 support from SystemInfoRequest in this PR, I can push up a separate PR in the next few days that adds v2 SolrJ coverage using the codegen approach. Would keep us better in line with the other v2 stuff.
| @JsonProperty("core_root") | ||
| public String coreRoot; | ||
|
|
||
| @JsonProperty(isRequired = OptBoolean.FALSE) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[Q] I'm a little confused by these isRequired = OptBoolean.FALSE bits. My understanding was that these annotations defaulted to treat properties as "non-required" unless otherwise specified. Is there a purpose to agreeing with the default so explicitly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess I was being too cautious. I can remove the explicit setting.
| : null; | ||
| } | ||
|
|
||
| public String getJVMMemoryTtl() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[0] Could this be "Total" instead of "Ttl"? "Ttl" could reasonably be interpreted as an abbreviation of "time-to-live"
Also, what is the "unit" being returned here. Is it the number of bytes? kb? gb? Is it a human readable string that includes the unit in it? Might be worth choosing a name that indicates the answer, so callers don't have to dig through the implementation themselves to find out. e.g. getHumanReadableJvmMemoryTotal or getJvmMemoryTotalBytes or something similar.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's 2 outputs for these values (used, total, free). One is "raw", as long values. One is human-readable. Example: "total":"512 MB".
I'll change the method name fo clarity.
| : null; | ||
| } | ||
|
|
||
| public Long getJVMUpTime() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[Q] What is the unit being returned here? seconds? milliseconds? Might be nice to indicate that in the method name to save callers from needing to dig into the implementation to discover for themselves. e.g. getJvmUpTimeSeconds.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changing to "getJvmUpTimeMS" (for milliseconds). Unless you need "Milliseconds" spelled out?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we've used "Millis" in a few other places, that might be a good compromise between a very-short abbreviation like "MS" and the full "Milliseconds" which seems very verbose.
But whichever you think makes sense - IMO the main thing is to have the unit specified somehow 👍
| protected QueryResponse createResponse(final NamedList<Object> namedList) { | ||
| return new QueryResponse(); | ||
| final SystemInfoResponse response = info.process(server); | ||
| assertTrue(response.getResponse().size() > 0); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[+1] Really awesome to see how much cleaner a lot of these usages look without GenericSolrRequest!
| "/admin/metrics", | ||
| SolrRequest.SolrRequestType.ADMIN, | ||
| SolrParams.of("wt", "prometheus")); | ||
| var req = new MetricsRequest(SolrParams.of("wt", "prometheus")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[Q] It looks like many (most?) of the MetricsRequest usages provide SolrParams.of("wt", "prometheus"). I'm not super familiar with the metrics endpoint, but if that's our norm in practice in these tests, maybe it'd make sense to just have it be the default within MetricsRequest?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not familiar with metrics either. I would not risk breaking too much stuff as part of this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The metrics handler /admin/metrics defaults to prometheus if wt isn't passed. So keeping and/or removing this in shouldn't make a difference. But also the response parser below is set to prometheus so it goes to the PrometheusResponseWriter where wt is technically really set.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
keeping and/or removing this in shouldn't make a difference.
Thanks! I'll keep it, just to be sure.
| try { | ||
| rsp = req.process(solrClient, null); | ||
| SystemInfoResponse rsp = req.process(solrClient, null); | ||
| NamedList<Object> nl = rsp.getResponse(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[0] We could get rid of some ugly casting if we changed these assertions to work on the NodeSystemResponse typed object, rather than inspecting the NamedList. Unless there's a particular reason we're working with NL here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed. Just kind f forgot to adapt. I'm keeping the check for the NL size, though: it shows the params were used.
|
|
||
| /** | ||
| * @param path the HTTP path to use for this request. Supports V1 "/admin/info/system" (default) | ||
| * or V2 "/node/system" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[0] I'm all for adding v2 API coverage to SolrJ, but I wonder whether in this specific case it's better to stick to only supporting v1.
Generally we've been leaning on code-generation to create SolrRequest impl's for our v2 APIs. We don't have that for "/node/system" just yet, true. But the NodeSystemResponse model class added in this PR is actually most of the work to getting us there! So IMO it feels very do-able to have coverage for the v2 API soon in a way that's more in line with our long-term plan around these APIs.
Skipping v2 support in this class keeps us out of the awkward position later on of having to deprecate this ctor and shift usage over to using the generated v2 SolrRequest. What do you guys think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we leave out the support for the "old" V2 here... Then, RepositoryManager would have to call the V1 endpoint (like before this PR)... But you say the package-manager only has v2 APIs ?
So... what do I do here?
| // normalized to remove the /solr path part | ||
| // So might as well ping the V2 API "/node/system" instead. | ||
| // Otherwise, this SystemInfoRequest ctr would need to set the full /solr/admin/info/system path | ||
| SystemInfoResponse sysResponse = new SystemInfoRequest("/node/system").process(solrClient); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another factor in where we use "v2" on the server side is that, currently, users can disable the v2 API with a system property: solr.api.v2.enabled.
I think we're ready to remove that feature-flag, so that we can start assuming v2 will be available to call from our tooling, etc. But while it exists it's probably brittle to have tooling, etc. use v2 endpoints.
Coincidentally, the package-store code here is one of the few places that assuming v2 is "safe". The package-manager only has v2 APIs. So it's safe for us to call other v2 endpoints from the package store code, because if v2 is disabled then all this code is unreachable anyways.
|
|
||
| private static final long serialVersionUID = 1L; | ||
|
|
||
| private org.apache.solr.client.api.model.SystemInfoResponse fullResponse; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, given more time we could maybe come up with something better than NodeSystemResponse. But better to get this merged in and we can iterate on it going forward...
|
@epugh , @gerlowskija Also notice that the responseHeader is repeated when multiple nodes are included in the response. Example NamedList "toString" output from unit test (added log locally in AdminHadlersProxyTest) : |
https://issues.apache.org/jira/browse/SOLR-17136
Description
Replacing uses of GenericSolrRequest to call "/admin/info/system" and "admin/metrics" by classes SystemInfoRequest, and MetricsRequest.
Solution
Adding class SystemInfoRequest. SystemInfoRequest can toggle between the V1 and V2 APIs. The V2 API was already available at /api/node/system.
Adding API model class org.apache.solr.api.model.SystemInfoResponse to represent the system info response, and SorlJ response class org.apache.solr.client.solrj.response.SystemInfoResponse to map the NamedList response to the API model.
Adding class MetricsRequest. This one returns a generic SolrResponse, because there are so many variants of a metrics response.
Replacing hard-coded strings "/admin/info/system" and "admin/metrics" by the corresponding constants.
Tests
Unit tests pass. Existing unit tests already serve to test these modifications.
Functional tests using the CLI tool, where most of the functional changes occurred.
Multiple changes in unit tests, to use the new classes, or existing constants.
Checklist
Please review the following and check all that apply:
mainbranch../gradlew check.