[WIP] RF-39249 Add AI Test Generation and Regeneration Commands #500
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds AI-powered test generation capabilities to the Rainforest CLI through a new generate command. The feature allows users to create tests from natural language prompts via the Rainforest API, with support for customization options and optional RFML download.
Changes:
- Added
generatecommand withgenandaialiases for creating tests from natural language prompts - Implemented API client methods (
CreateTestWithAI) for AI test generation with request/response structures - Added comprehensive validation for AI-supported browsers and input parameters
- Included test coverage for the new functionality and updated documentation
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| rainforest-cli.go | Adds the generate command definition with flags for title, start-uri/url, platform, environment-id, credentials, and download options |
| tests.go | Implements generateAITest handler with validation logic, API integration, and optional RFML download functionality |
| rainforest/tests.go | Defines AITestRequest, AITestResponse structs and implements CreateTestWithAI API method with browser validation |
| tests_test.go | Adds comprehensive unit tests for generateAITest covering various scenarios including missing params, URL options, platforms, and credentials |
| rainforest/tests_test.go | Adds unit tests for IsAISupportedBrowser and CreateTestWithAI API method including validation tests |
| README.md | Documents the new generate command with usage examples and explanation of options |
| CHANGELOG.md | Records the new feature in version 3.7.0 |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
sebaherrera07
force-pushed
the
RF-39249-support-ai-test-generation
branch
from
178126b to
a9c1140
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
sebaherrera07
changed the title
generate command for AI test generationgenerate command for AI test generation
sebaherrera07
force-pushed
the
RF-39249-support-ai-test-generation
branch
2 times, most recently
from
7119044 to
7fa7b22
Compare
sebaherrera07
force-pushed
the
RF-39249-support-ai-test-generation
branch
from
7fa7b22 to
4be863d
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
sebaherrera07
changed the title
generate command for AI test generation
sebaherrera07
force-pushed
the
RF-39249-support-ai-test-generation
branch
from
4be863d to
c179fac
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
sebaherrera07
force-pushed
the
RF-39249-support-ai-test-generation
branch
from
c179fac to
5ca079e
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
rainforest/tests.go
Outdated
| // CreateTestWithAI creates a new test using AI generation | ||
| func (c *Client) CreateTestWithAI(request *AITestRequest) (*AITestResponse, error) { | ||
| // Validate that exactly one browser is provided (API only uses the first one) | ||
| if len(request.Browsers) == 0 { | ||
| return nil, errors.New("At least one browser is required for AI test generation") | ||
| } | ||
| if len(request.Browsers) > 1 { | ||
| return nil, errors.New("AI test generation only supports one browser at a time") | ||
| } | ||
|
|
||
| // Validate that the browser is AI-supported | ||
| browser := request.Browsers[0] | ||
| if name, ok := browser["name"].(string); ok { | ||
| if !IsAISupportedBrowser(name) { | ||
| return nil, fmt.Errorf("Browser '%s' is not supported for AI test generation. Supported browsers: %v", name, AISupportedBrowsers) | ||
| } | ||
| } else { | ||
| return nil, errors.New("Invalid browser format in request") | ||
| } | ||
|
|
||
| // Ensure type is "test" | ||
| if request.Type != "test" { | ||
| return nil, errors.New("Type must be 'test' for AI test generation") | ||
| } | ||
|
|
||
| // Ensure prompt is provided | ||
| if request.Prompt == "" { | ||
| return nil, errors.New("Prompt is required for AI test generation") | ||
| } | ||
|
|
||
| // Ensure at least one of StartURI or FullURL is provided | ||
| if request.StartURI == "" && request.FullURL == "" { | ||
| return nil, errors.New("Either StartURI or FullURL must be provided for AI test generation") | ||
| } | ||
|
|
||
| // Ensure StartURI and FullURL are mutually exclusive | ||
| if request.StartURI != "" && request.FullURL != "" { | ||
| return nil, errors.New("Only one of StartURI or FullURL may be provided for AI test generation") | ||
| } | ||
|
|
||
| // Validate mutually exclusive login parameters | ||
| if request.PromptCredentials != "" && request.LoginSnippetID > 0 { | ||
| return nil, errors.New("Cannot specify both prompt_credentials and login_snippet_id. Choose one login method") | ||
| } | ||
|
|
||
| // Prepare request | ||
| req, err := c.NewRequest("POST", "tests", request) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| // Send request and process response | ||
| var response AITestResponse | ||
| _, err = c.Do(req, &response) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| return &response, nil | ||
| } | ||
|
|
||
| // RegenerateTestWithAI regenerates an existing AI test with a new or modified prompt | ||
| func (c *Client) RegenerateTestWithAI(testID int, request *AIRegenerateRequest) (*AITestResponse, error) { | ||
| // Validate that exactly one browser is provided (API only uses the first one) | ||
| if len(request.Browsers) == 0 { | ||
| return nil, errors.New("At least one browser is required for AI test regeneration") | ||
| } | ||
| if len(request.Browsers) > 1 { | ||
| return nil, errors.New("AI test regeneration only supports one browser at a time") | ||
| } | ||
|
|
||
| // Validate that the browser is AI-supported | ||
| browser := request.Browsers[0] | ||
| if name, ok := browser["name"].(string); ok { | ||
| if !IsAISupportedBrowser(name) { | ||
| return nil, fmt.Errorf("Browser '%s' is not supported for AI test generation. Supported browsers: %v", name, AISupportedBrowsers) | ||
| } | ||
| } else { | ||
| return nil, errors.New("Invalid browser format in request") | ||
| } | ||
|
|
||
| // Ensure StartURI and FullURL are mutually exclusive | ||
| if request.StartURI != "" && request.FullURL != "" { | ||
| return nil, errors.New("Only one of StartURI or FullURL may be provided for AI test generation") | ||
| } | ||
|
|
||
| // Validate mutually exclusive login parameters | ||
| if request.PromptCredentials != "" && request.LoginSnippetID > 0 { | ||
| return nil, errors.New("Cannot specify both prompt_credentials and login_snippet_id. Choose one login method") | ||
| } | ||
|
|
||
| // Prepare request | ||
| req, err := c.NewRequest("PUT", fmt.Sprintf("tests/%d/regenerate", testID), request) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| // Send request and process response | ||
| var response AITestResponse | ||
| _, err = c.Do(req, &response) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| return &response, nil | ||
| } |
There was a problem hiding this comment.
There is significant code duplication between CreateTestWithAI and RegenerateTestWithAI. The browser validation logic (lines 584-600 and 645-661), the StartURI/FullURL mutual exclusivity check (lines 617-620 and 663-666), and the login parameters validation (lines 622-625 and 668-671) are duplicated. Consider extracting this shared validation logic into a common helper function to improve maintainability and reduce the risk of inconsistencies when making future changes.
| if c.Bool("download") { | ||
| testDirectory := c.String("test-folder") | ||
| absTestDirectory, err := prepareTestDirectory(testDirectory) | ||
| if err != nil { | ||
| return cli.NewExitError(err.Error(), 1) | ||
| } | ||
|
|
||
| test, err := api.GetTest(response.TestID) | ||
| if err != nil { | ||
| return cli.NewExitError(fmt.Sprintf("Failed to download generated test: %v", err), 1) | ||
| } | ||
|
|
||
| testIDPairs, err := api.GetTestIDs() | ||
| if err != nil { | ||
| return cli.NewExitError(err.Error(), 1) | ||
| } | ||
| testIDCollection := rainforest.NewTestIDCollection(testIDPairs) | ||
|
|
||
| err = test.PrepareToWriteAsRFML(*testIDCollection, false) | ||
| if err != nil { | ||
| return cli.NewExitError(err.Error(), 1) | ||
| } | ||
|
|
||
| paddedTestID := fmt.Sprintf("%010d", test.TestID) | ||
| sanitizedTitle := sanitizeTestTitle(test.Title) | ||
| fileName := fmt.Sprintf("%v_%v.rfml", paddedTestID, sanitizedTitle) | ||
| filePath := filepath.Join(absTestDirectory, fileName) | ||
|
|
||
| file, err := os.Create(filePath) | ||
| if err != nil { | ||
| return cli.NewExitError(err.Error(), 1) | ||
| } | ||
|
|
||
| writer := rainforest.NewRFMLWriter(file) | ||
| err = writer.WriteRFMLTest(test) | ||
| file.Close() | ||
| if err != nil { | ||
| return cli.NewExitError(err.Error(), 1) | ||
| } | ||
|
|
||
| log.Printf("Downloaded test to %v", filePath) | ||
| } |
There was a problem hiding this comment.
The download functionality in this section (lines 419-460) duplicates logic from the downloadTests function (around lines 829-942). The code for downloading a test, preparing it as RFML, creating the file path, and writing it is nearly identical. Consider extracting this into a shared helper function to reduce code duplication and improve maintainability.
| func (c *Client) CreateTestWithAI(request *AITestRequest) (*AITestResponse, error) { | ||
| // Validate that exactly one browser is provided (API only uses the first one) | ||
| if len(request.Browsers) == 0 { | ||
| return nil, errors.New("At least one browser is required for AI test generation") |
There was a problem hiding this comment.
The error message "At least one browser is required for AI test generation" is misleading because the very next check (lines 588-590) requires exactly one browser, not "at least one". Consider changing this message to "Exactly one browser is required for AI test generation" to better communicate the constraint to users.
| return nil, errors.New("At least one browser is required for AI test generation") | |
| return nil, errors.New("Exactly one browser is required for AI test generation") |
rainforest/tests.go
Outdated
| func (c *Client) RegenerateTestWithAI(testID int, request *AIRegenerateRequest) (*AITestResponse, error) { | ||
| // Validate that exactly one browser is provided (API only uses the first one) | ||
| if len(request.Browsers) == 0 { | ||
| return nil, errors.New("At least one browser is required for AI test regeneration") |
There was a problem hiding this comment.
The error message "At least one browser is required for AI test regeneration" is misleading because the very next check (lines 649-651) requires exactly one browser, not "at least one". Consider changing this message to "Exactly one browser is required for AI test regeneration" to better communicate the constraint to users.
| return nil, errors.New("At least one browser is required for AI test regeneration") | |
| return nil, errors.New("Exactly one browser is required for AI test regeneration") |
Add support for generating tests using AI through the CLI. The new `generate` command (with `gen` and `ai` aliases) accepts a natural language prompt and creates a test via the Rainforest API. Features: - Generate tests from natural language prompts - Support for --start-uri or --url to specify the starting point - Optional --title flag (defaults to prompt text) - Optional --download flag to save generated test as RFML - Optional --test-folder flag to specify download location - Optional --environment-id flag for test environment - Support for authentication via --credentials or --login-snippet-id (mutually exclusive) - Platform selection with --platform (defaults to windows11_chrome) - Validation for AI-supported browsers (windows10_chrome, windows11_chrome, windows11_chrome_fhd) - Comprehensive validation and error handling Example usage: rainforest generate "Log in with valid credentials" --start-uri /login rainforest generate "Add item to cart" --url https://example.com --download Files changed: - rainforest/tests.go: Add AITestRequest, AITestResponse structs, AISupportedBrowsers list, IsAISupportedBrowser helper, and CreateTestWithAI API method - rainforest/tests_test.go: Add comprehensive unit tests for AI generation - tests.go: Add generateAITest and parseAndValidateBrowser helper functions - tests_test.go: Add tests for generateAITest with various scenarios - rainforest-cli.go: Add generate command definition with all flags - README.md: Add documentation and examples for the new command
Update version constant and add CHANGELOG entry for the new `generate` command feature.
sebaherrera07
force-pushed
the
RF-39249-support-ai-test-generation
branch
from
5ca079e to
22f31f0
Compare
Introduces AI-powered test generation to the Rainforest CLI, allowing users to create tests using natural language prompts.
New Commands
rainforest generate- Generate a new test from a natural language promptrainforest generate "Log in with valid credentials" --start-uri /loginKey Features
--start-urior--url(mutually exclusive)--title,--download,--environment-idflags--credentialsor--login-snippet-id(mutually exclusive)--platform(defaults towindows11_chrome)windows10_chrome,windows11_chrome,windows11_chrome_fhdImplementation Highlights
CreateTestWithAI()API methodDocumentation
Version: 3.7.0