From c0afbd53bd967c247c4b0900e718325d75121fe0 Mon Sep 17 00:00:00 2001 From: "Parker (bot)" <257644329+parker-ekwis-bot@users.noreply.github.com> Date: Tue, 27 Jan 2026 22:54:22 +0000 Subject: [PATCH 1/2] docs: update README quickstart --- README.md | 220 ++++++------------------------------------------------ 1 file changed, 23 insertions(+), 197 deletions(-) diff --git a/README.md b/README.md index 822b918..0da6019 100644 --- a/README.md +++ b/README.md @@ -1,219 +1,45 @@ # Salesforce Python Template -A robust Python template designed for seamless interaction with Salesforce. This project enables bulk data operations, SOQL queries, CSV processing, interactive field mapping, and data enrichment via web scraping. +A concise template project for building Salesforce data workflows in Python. It includes scripts to upload CSVs via the SObject Collections API, run SOQL queries to CSV, and an example enrichment flow that scrapes public webpages and updates records after confirmation. -## Features - -- **Salesforce Authentication:** Utilises environment variables for secure access -- **Bulk Data Operations:** Supports insert, update, delete, and upsert operations -- **SOQL Query Execution:** Run queries against Salesforce and export results to CSV -- **Interactive Field Mapping:** Map CSV columns to Salesforce fields during upload -- **Data Enrichment:** Enhances Salesforce records by scraping additional company information -- **Customisable Configuration:** Easily adjust settings via a YAML configuration file -- **Comprehensive Logging and Error Handling** -- **Unit and Integration Tests:** Ensuring high code quality with pytest - -## Project Setup - -### 1. Clone the Repository -```bash -git clone -cd Salesforce-Python-Template -``` - -### 2. Create and Activate a Virtual Environment +## Quickstart ```bash -python3 -m venv venv +python -m venv venv source venv/bin/activate - -# Confirm Python version (3.9 or higher required) -python --version -``` - -### 3. Install Dependencies -```bash pip install -r requirements.txt -``` - -### 4. Set Up Environment Variables -Create a copy of the environment file and update it with your Salesforce credentials: -```bash cp .env.example .env -``` - -Open the `.env` file in your preferred editor and configure the following variables: -```plaintext -SALESFORCE_USERNAME=your_username -SALESFORCE_PASSWORD=your_password -SALESFORCE_SECURITY_TOKEN=your_token -SALESFORCE_DOMAIN=login # Use 'test' for sandbox environments -``` - -## Configuration - -Review and modify the configuration file located at `config/config.yaml` to suit your requirements. - -### Key Configuration Areas: -- **Salesforce API Settings:** API version, batch size, and timeout -- **Logging:** Log level, format, and file destination -- **CSV Processing:** Encoding, delimiter, and directories for input and error files -- **Data Enrichment Fields:** Specify which fields to update for different Salesforce objects - -### Example Configuration: -```yaml -api: - version: '57.0' - batch_size: 200 - timeout: 30 - -logging: - level: INFO - format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s' - file: 'errors/app.log' - -csv: - encoding: 'utf-8' - delimiter: ',' - error_directory: 'errors' - input_directory: 'input' +# edit .env with your Salesforce credentials -enrichment: - fields: - Account: - - Phone - - Website - - BillingStreet - - BillingCity - - BillingState - - BillingPostalCode - - BillingCountry - Contact: - - Phone - - Email - - MailingStreet - - MailingCity - - MailingState - - MailingPostalCode - - MailingCountry - Lead: - - Phone - - Email - - Street - - City - - State - - PostalCode - - Country +python scripts/query_data.py "SELECT Id, Name FROM Account" output/accounts.csv ``` -## Usage - -### 1. Upload Data Script (`scripts/upload_data.py`) - -This script uploads CSV data to Salesforce with interactive field mapping. - -#### Supported Operations: -- insert -- update -- delete -- upsert - -#### Features: -- **Field Mapping:** Interactive CSV headers to Salesforce fields mapping -- **Batch Processing:** Records processed in configurable batches (up to 200 records per call) -- **Flexible Operations:** Support for all major bulk operations - -#### Command-Line Options: -| Option | Description | Required | -|--------|-------------|----------| -| `csv_file` | Path to the CSV file | Yes | -| `object_name` | Salesforce object name (e.g., Account, Contact) | Yes | -| `--operation` | Operation type (insert, update, delete, upsert) | No (default: insert) | -| `--config` | Path to configuration file | No (default: config/config.yaml) | -| `--external_id_field` | External ID field for upsert | Only for upsert | -| `--batch_size` | Number of records per API call | No | - -#### Example Usage: +Run tests (requires Salesforce credentials and will create/delete Case records): ```bash -source venv/bin/activate -python scripts/upload_data.py path/to/input.csv Account --operation insert +pytest ``` -### 2. Query Data Script (`scripts/query_data.py`) +## Using this template +- Create a new repository from this template or clone it and rename the directory. +- Update `README.md`, `config/config.yaml`, and `.env.example` for your org and workflow. +- Commit and start building your Salesforce integration. -Execute SOQL queries and export results to CSV. +## Scripts +- `scripts/upload_data.py`: Upload CSV data with interactive field mapping; supports insert, update, delete, and upsert (SObject Collections). Writes results to `results/` and errors to `errors/`. +- `scripts/query_data.py`: Run SOQL and write results to CSV. +- `scripts/manipulate_data.py`: Enrichment example that finds public company info via web scraping and updates Account/Contact/Lead fields after confirmation. -#### Command-Line Options: -| Option | Description | Required | -|--------|-------------|----------| -| `soql` | SOQL query (in quotes) | Yes | -| `output_file` | CSV output file path | Yes | -| `--config` | Path to configuration file | No | - -#### Example Usage: +### Examples ```bash -source venv/bin/activate +python scripts/upload_data.py path/to/input.csv Account --operation insert python scripts/query_data.py "SELECT Id, Name FROM Account" output/accounts.csv +python scripts/manipulate_data.py 0016g000009yzfpAAA Account --fields Phone Website ``` -### 3. Data Enrichment Script (`scripts/manipulate_data.py`) - -Enrich Salesforce records with additional data through web scraping. - -#### Command-Line Options: -| Option | Description | Required | -|--------|-------------|----------| -| `record_id` | Salesforce Record ID | Yes | -| `sobject_type` | Object type (Account, Contact, Lead) | Yes | -| `--config` | Path to configuration file | No | -| `--fields` | Specific fields to update | No | - -#### Example Usage: -```bash -source venv/bin/activate -python scripts/manipulate_data.py 0016g000009yzfpAAA Account --fields Phone Website BillingStreet -``` - -## Testing - -Run the complete test suite: -```bash -pytest tests/ -``` - -Generate test coverage report: -```bash -pytest --cov=src tests/ -``` +## Configuration +- `config/config.yaml` controls API version, logging, CSV settings, and default enrichment fields. +- `.env` holds Salesforce credentials (see `.env.example`). ## Troubleshooting - -### Common Issues and Solutions: - -1. **ModuleNotFoundError for 'src'** - - Run scripts from project root - - Adjust PYTHONPATH if needed - -2. **Authentication Failures** - - Verify Salesforce credentials in `.env` - - Confirm API access is enabled - -3. **API Errors** - - Check API version in `config/config.yaml` - - Verify user permissions in Salesforce - -## Contributing - -We welcome contributions! Please follow these steps: - -1. Fork the repository -2. Create a feature branch -3. Submit a pull request with detailed descriptions - -## License - -This project is licensed under the MIT License. - ---- - -Thank you for using the Salesforce Python Template. We hope this tool streamlines your Salesforce data operations and enrichments. For any issues or further assistance, please open an issue on GitHub. \ No newline at end of file +- `ModuleNotFoundError: src`: run scripts from the project root. +- Authentication errors: confirm `.env` values and Salesforce API access. From d910baca04ad458efb20f3033ef7a789c682708a Mon Sep 17 00:00:00 2001 From: "Parker (bot)" <257644329+parker-ekwis-bot@users.noreply.github.com> Date: Tue, 27 Jan 2026 23:01:30 +0000 Subject: [PATCH 2/2] docs: expand README and fix env example --- .env.example | 11 +++++++---- README.md | 54 ++++++++++++++++++++++++++++++++++------------------ 2 files changed, 42 insertions(+), 23 deletions(-) diff --git a/.env.example b/.env.example index a513fca..a12c173 100644 --- a/.env.example +++ b/.env.example @@ -1,4 +1,7 @@ -SALESFORCE_USERNAME=username@domain.com -SALESFORCE_PASSWORD=password -SALESFORCE_SECURITY_TOKEN= -SALESFORCE_DOMAIN=login / company.my \ No newline at end of file +# Salesforce credentials +SALESFORCE_USERNAME=you@example.com +SALESFORCE_PASSWORD=your_password +# Security token (required unless your IP is whitelisted) +SALESFORCE_SECURITY_TOKEN=your_security_token +# Domain: "login" (production), "test" (sandbox), or your My Domain subdomain (e.g. "acme") +SALESFORCE_DOMAIN=login diff --git a/README.md b/README.md index 0da6019..0eff71b 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,9 @@ # Salesforce Python Template -A concise template project for building Salesforce data workflows in Python. It includes scripts to upload CSVs via the SObject Collections API, run SOQL queries to CSV, and an example enrichment flow that scrapes public webpages and updates records after confirmation. +A concise template for Salesforce data workflows in Python. It includes: +- CSV upload via the SObject Collections endpoint (batched up to 200 records). +- SOQL query to CSV. +- An enrichment demo that scrapes public pages and updates records after confirmation. ## Quickstart ```bash @@ -14,32 +17,45 @@ cp .env.example .env python scripts/query_data.py "SELECT Id, Name FROM Account" output/accounts.csv ``` -Run tests (requires Salesforce credentials and will create/delete Case records): -```bash -pytest -``` - -## Using this template -- Create a new repository from this template or clone it and rename the directory. -- Update `README.md`, `config/config.yaml`, and `.env.example` for your org and workflow. -- Commit and start building your Salesforce integration. +## Configuration overview +- `.env` (see `.env.example`) + - `SALESFORCE_USERNAME`, `SALESFORCE_PASSWORD`, `SALESFORCE_SECURITY_TOKEN` + - `SALESFORCE_DOMAIN` is `login` (prod), `test` (sandbox), or your My Domain subdomain +- `config/config.yaml` + - `api`: API version, default batch size, timeout + - `logging`: log level and log file (default `errors/app.log`) + - `csv`: encoding/delimiter and error/input directories + - `enrichment`: default fields per object for `scripts/manipulate_data.py` ## Scripts -- `scripts/upload_data.py`: Upload CSV data with interactive field mapping; supports insert, update, delete, and upsert (SObject Collections). Writes results to `results/` and errors to `errors/`. -- `scripts/query_data.py`: Run SOQL and write results to CSV. -- `scripts/manipulate_data.py`: Enrichment example that finds public company info via web scraping and updates Account/Contact/Lead fields after confirmation. - -### Examples +- `scripts/upload_data.py` + - Uses SObject Collections (composite/sobjects), not the Bulk API. + - Interactive field mapping; insert/update/delete/upsert. + - Upsert requires `--external_id_field`. + - Writes per-record results to `results/` (success and error CSVs). +- `scripts/query_data.py`: runs SOQL and writes a CSV. +- `scripts/manipulate_data.py`: looks up a record, scrapes public pages, and proposes updates for confirmation. + +### Usage examples ```bash python scripts/upload_data.py path/to/input.csv Account --operation insert +python scripts/upload_data.py path/to/input.csv Case --operation upsert --external_id_field CaseNumber python scripts/query_data.py "SELECT Id, Name FROM Account" output/accounts.csv python scripts/manipulate_data.py 0016g000009yzfpAAA Account --fields Phone Website ``` -## Configuration -- `config/config.yaml` controls API version, logging, CSV settings, and default enrichment fields. -- `.env` holds Salesforce credentials (see `.env.example`). +## Salesforce client helpers +`src/salesforce.py` includes Bulk API helpers (`bulk_insert`, `bulk_update`) and a `query` helper. The upload script does not use these bulk helpers; it batches records through the SObject Collections REST endpoint. + +## Testing +- Unit tests: `tests/test_salesforce.py`, `tests/test_bulkification.py`, `tests/test_manipulate_data.py` (with mocked HTTP). +- Integration tests: `tests/test_upload_operations.py` runs live against Salesforce and creates/deletes Case records. `tests/test_manipulate_data.py::test_full_integration` is marked integration but skipped by default. + +Run all tests (requires Salesforce credentials and will create/delete Case records): +```bash +pytest +``` ## Troubleshooting - `ModuleNotFoundError: src`: run scripts from the project root. -- Authentication errors: confirm `.env` values and Salesforce API access. +- Authentication errors: confirm `.env` values, domain (`login`/`test`/My Domain), and API access.