feat: Full-text search in archives #81
Conversation
This commit introduces two new commands: `borg search` and `borg index`. The `borg index` command builds a trigram index for an archive, which can be used to significantly speed up searches. The `borg search` command allows users to search for patterns within archives. It supports regular expressions, context control, file type filtering, and result limits. The command will automatically use a pre-built index if one is available, falling back to a full scan if not. This commit also includes: - Unit tests for the new commands. - Documentation for the new commands in `docs/cli.md`. - Updates to `.gitignore` to exclude index files. - Improvements to the test infrastructure to prevent state pollution. Co-authored-by: Snider <631881+Snider@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello @Snider, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the 'borg' CLI by integrating robust full-text search capabilities for archived content. It introduces two new commands: 'borg index' for creating efficient trigram-based search indexes, and 'borg search' for performing flexible searches within archives. The 'search' command is designed to be intelligent, automatically using a pre-built index if available, and offers advanced options like regex support, context display, and file type filtering to provide a comprehensive search experience. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces full-text search capabilities, a valuable addition to the borg CLI. The implementation includes a new index command for creating a trigram index and a search command that leverages this index for efficient searching. The code is well-structured, and the feature is accompanied by tests and documentation.
My review focuses on several opportunities for improvement, primarily concerning performance and code maintainability. I've identified areas where memory usage can be significantly reduced, especially when handling large files, and pointed out sections with duplicated code that could be refactored for better long-term maintenance. Addressing these points will make the new search feature more robust and efficient.
| var lines []string | ||
| scanner := bufio.NewScanner(file) | ||
| for scanner.Scan() { | ||
| lines = append(lines, scanner.Text()) | ||
| } |
There was a problem hiding this comment.
Loading the entire file into a slice of strings with lines = append(lines, scanner.Text()) is highly memory-inefficient, especially for large files. This can lead to excessive memory consumption when printing search results with context. A more scalable solution would be to use a streaming approach. For example, you could iterate through the file's lines once, using a circular buffer to keep track of the N lines preceding a match. This would provide the required context without loading the entire file into memory.
| content, err := io.ReadAll(file) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| // Generate and add trigrams | ||
| if len(content) < 3 { | ||
| return nil | ||
| } | ||
| for i := 0; i <= len(content)-3; i++ { | ||
| var trigram [3]byte | ||
| copy(trigram[:], content[i:i+3]) | ||
|
|
||
| postings := trigramIndex[trigram] | ||
| if len(postings) == 0 || postings[len(postings)-1] != fileID { | ||
| trigramIndex[trigram] = append(postings, fileID) | ||
| } | ||
| } |
There was a problem hiding this comment.
Reading the entire file content into memory with io.ReadAll can be inefficient for large files, as it creates a new copy of the data that already exists within the in-memory DataNode. A more memory-efficient approach would be to process the file as a stream to generate trigrams. This can be achieved using a bufio.Reader and a sliding window, which would avoid allocating a large buffer for each file's content.
r := bufio.NewReader(file)
var window [3]byte
// Prime the window by reading the first 3 bytes.
if _, err := io.ReadFull(r, window[:]); err != nil {
if err == io.EOF || err == io.ErrUnexpectedEOF {
return nil // File is smaller than 3 bytes.
}
return err
}
add := func(trigram [3]byte) {
postings := trigramIndex[trigram]
if len(postings) == 0 || postings[len(postings)-1] != fileID {
trigramIndex[trigram] = append(postings, fileID)
}
}
add(window)
// Slide the window over the rest of the file.
for {
b, err := r.ReadByte()
if err != nil {
if err == io.EOF {
break
}
return err
}
window[0], window[1], window[2] = window[1], window[2], b
add(window)
}| // Read and decompress the archive | ||
| compressedData, err := os.ReadFile(archivePath) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to read archive: %w", err) | ||
| } | ||
| tarData, err := compress.Decompress(compressedData) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to decompress archive: %w", err) | ||
| } | ||
| dn, err := datanode.FromTar(tarData) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to load datanode: %w", err) | ||
| } |
There was a problem hiding this comment.
This block of code for reading, decompressing, and loading an archive into a DataNode is identical to the logic in cmd/index.go (lines 37-51). To improve maintainability and adhere to the DRY (Don't Repeat Yourself) principle, this logic should be extracted into a shared helper function. This would centralize the archive loading process, making future changes easier to manage.
| file, err := dn.Open(path) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| defer file.Close() | ||
|
|
||
| scanner := bufio.NewScanner(file) | ||
| for lineNum := 1; scanner.Scan(); lineNum++ { | ||
| line := scanner.Text() | ||
| match := false | ||
| if useRegex { | ||
| if re.MatchString(line) { | ||
| match = true | ||
| } | ||
| } else { | ||
| if strings.Contains(line, pattern) { | ||
| match = true | ||
| } | ||
| } | ||
|
|
||
| if match { | ||
| results = append(results, searchResult{ | ||
| FilePath: path, | ||
| LineNum: lineNum, | ||
| Line: strings.TrimSpace(line), | ||
| }) | ||
| } | ||
| } | ||
| return scanner.Err() |
There was a problem hiding this comment.
The logic within this dn.Walk callback for opening a file, scanning it line by line, and finding matches is nearly identical to the logic in the searchWithIndex function (lines 258-289). This code duplication makes the codebase harder to maintain. Consider extracting this file-scanning logic into a dedicated helper function, such as scanFileForMatches(...). This function could be called from both searchWithoutIndex and searchWithIndex, promoting code reuse and simplifying future modifications.
| // Find intersection of file IDs | ||
| var intersection map[uint32]struct{} | ||
| for i, trigram := range trigrams { | ||
| postings := trigramIndex[trigram] | ||
| if i == 0 { | ||
| intersection = make(map[uint32]struct{}) | ||
| for _, fileID := range postings { | ||
| intersection[fileID] = struct{}{} | ||
| } | ||
| } else { | ||
| newIntersection := make(map[uint32]struct{}) | ||
| for _, fileID := range postings { | ||
| if _, ok := intersection[fileID]; ok { | ||
| newIntersection[fileID] = struct{}{} | ||
| } | ||
| } | ||
| intersection = newIntersection | ||
| } | ||
| } | ||
|
|
||
| candidateFiles := make(map[string]struct{}) | ||
| for fileID := range intersection { | ||
| candidateFiles[fileList[fileID]] = struct{}{} | ||
| } |
There was a problem hiding this comment.
The current implementation for finding the intersection of posting lists uses maps, which is not the most efficient method given that the posting lists (file IDs) are sorted. A more performant approach would be to use a merge-like algorithm that iterates through the sorted slices in linear time. This would avoid the overhead of map creation and hashing, resulting in lower memory usage and faster execution.
For further optimization, you could sort the trigrams by the length of their posting lists and begin the intersection process with the shortest list. You will also need to add import "sort" to the file.
// Find intersection of file IDs
sort.Slice(trigrams, func(i, j int) bool {
return len(trigramIndex[trigrams[i]]) < len(trigramIndex[trigrams[j]])
})
intersection := trigramIndex[trigrams[0]]
for i := 1; i < len(trigrams); i++ {
listB := trigramIndex[trigrams[i]]
var result []uint32
iA, iB := 0, 0
for iA < len(intersection) && iB < len(listB) {
if intersection[iA] < listB[iB] {
iA++
} else if listB[iB] < intersection[iA] {
iB++
} else {
result = append(result, intersection[iA])
iA++
iB++
}
}
intersection = result
if len(intersection) == 0 {
break
}
}
candidateFiles := make(map[string]struct{})
for _, fileID := range intersection {
candidateFiles[fileList[fileID]] = struct{}{}
}
This change adds full-text search and indexing capabilities to the
borgCLI, allowing users to efficiently search for content within their archives. Theborg indexcommand creates a trigram index, and theborg searchcommand uses this index to perform fast, accurate searches with support for regex, context lines, and file type filtering.Fixes #43
PR created automatically by Jules for task 9156612768605145318 started by @Snider